How not to make yourself redundant with AI

I was asked to do this talk internally to lessen a bit of anxiety within the team. And that was the first slide I decided to start with this. Because nothing lessens anxiety more than threat of redundancy. But yeah, actually the talk is on how not to make yourself redundant. And this is mostly about my personal experience.

So the goal of this talk is to make all of you to think how IdentityAI can be your little helper and not your direct competitor. So let's start with a very important question that I get asked regularly. What does head of engineering do? The answer to this question depends on context and depends on the company. Personally, I joined relevance to help build and scale some of the company processes. For example, let's look at the example.

Here are the some of the processes I was supposed to introduce, improve, or streamline. All of this sounds pretty cool, at least in my mind, and this is not even a full list, but we start there. The problem is this is too much change. This is a lot of changes for people, teams and stakeholders to get behind. This gets us to the next very important question.

How do you know if you're succeeding in all of those changes that you promised to your stakeholders? As an industry, we actually already invented a lot of things that we can measure. A lot of them are time related, like time to resolve, time to triage, cycle time, lead time, time to address, time to identify, time of PR reviews.

All of those amazing things we talk about a lot in management. So those are metrics. Metrics are those magical things that we apply to other magical things to see how magical things are doing. And I am personally a very metric driven person. This presentation is a proof of that.

You'll see. So next slide. We are not going to talk about random engineers. We are going to talk about me. This is my average week in hours. I do a lot of one on ones and mentoring sessions. I have one on ones with all of my engineers. I have one on ones with my product people and with my design people. A lot of those meetings require prep.

So I need to look at how they're doing. I need to figure out what the business is doing so I can have those very important direct conversations that allow me to build connections with all of my people. Then there are team meetings where I have to be. There are leadership meetings where I have to be. And I am one of those people who sits there with like dollar count for every minute of this meeting in their mind.

But that's fine. I do interviews. I do admin. And because I just showed you a list of all the things that I am going to change in the company that I joined, I do a lot of metric checks and review aggregations. We talked a bit about objective metrics. There are also subjective metrics. For example, love. Love is a subjective metric.

And I love some of those things and I really don't love other ones. I really enjoy one on ones. I really enjoy my mentorship and prep. Team meetings, leadership meetings, I have to be there. I don't enjoy them that that much. Interviews and admin is just part of my job. I have to do this. But then there are metrics checks and reports aggregation.

That's where you endlessly look at Jira or linear, whatever you use. You look at the PR reviews. You calculate all of this. You try to figure out what is happening in terms of metrics in your business. So what's the value of those meetings? I personally value my one on ones very highly.

They're very important. It is about this human connection that we mentioned a couple of times here. It is very important for me to talk to my people, to see how they're doing, to hear their opinions, to give them a chance to give me feedback. Team meetings are also quite important. Interviews are important. Admin is important.

The problem is that those metric checks and reports aggregation is also very important. But I really, really don't like them, and they take a lot of time. It is high value, but very low enjoyment. So I really don't enjoy looking at linear for hours, so let's fix it with robots. So how are we going to fix it with robots?

Again, as a metric driven person, I sat down and made this list. Time and hours that I spend in a week, a value that I assign it, and enjoyment that I assign it. And based on those metrics, decided, no, I am not using AI for my one on ones. Those are the most important things. I am not delegating this.

I cannot delegate leadership meetings. I need to convince people to put more money here or hire people there. Don't know if I can be as convincing as I am. Hopefully not. I can partially delegate some of the interviews and some of the admin, but those metric checks and reports aggregation, oh, they can go to robots any day.

So what is it underneath if you look at robots working for me? It's actually 75% scripting. We're all engineers. We all know how to write scripts. So you write a script that's potentially a couple of API calls grabbing all the data, packaging all the data. Then you form your prompt, and you send it to LLM.

LLM looks at it and gives you an answer, which is agentic. LLM has access to additional tools if it needs to, etcetera, etcetera. And 5% is the manual check afterwards. Because sometimes I may not share the theory that LLM has about why this ticket took so long. So I need to check it manually, maybe not every time, but from time to time, just to like keep my level of trust into a little amount put high.

So I'm going to introduce you to my agentic army. I have a lot of little agents. I'm actually working on organizing all of them into a workforce. So instead it's like a proper analytical little things giving me information that I need about my team, what my team is doing, what the business is doing, and how can I help?

Yes, they all have names. And we're going to use those names because they are adorable. So first we have Belle. This is an example of Belle's output. She gives me a cycle time, lead time. She also looks at potential outliers and builds me theories around why those outliers exist. Why do I need her?

Those weekly reports, even though they may not have a lot of value on their own, they actually allow me to correlate a lot of internal stuff that's happening within the company to the engineering output. For example, at relevance, we're very agile. We quite often spin up a new team, pull people from different teams to work on this specific project.

And I can immediately see that when I took those three engineers and moved this them into this new team, the cycle time on the previous team actually went up. Oh my god. What does it mean? Can I have those conversations with my engineers now? So it's definitely informative because it allows me to ask questions. It allows me to dive a bit deeper into what's happening within the company and how the actions that we as leadership make actually affect people and affect the metrics, the numbers.

So Bell tells me what's happening. Then I have Beta. We use Beta Stack for alerting. So she generally just pulls in all the alerts that we had in a specific time frame and then sends them back saying, this is the number of alerts. This is how much time passed before we acknowledge them.

This is how much time passed before we resolve them. Again, it is very important because it allows me to see what are we actually doing. Are we is our product becoming more stable? Was this last release a crazy release that woke up my engineers at 3AM every single night? Generally, I can also see if they were awake at 3AM every night just on their faces, but it's still very helpful because it gives me the data that I need.

I have Doctor. Kirby who is looking at our back situation. Again, pulls all the tickets, looks at those tickets, journey throughout our linear, and tells me what is happening, what's the triage time, cycle time, lead time. This one is actually quite interesting because when I joined Relevance, we started changing our bug triaging process.

So I went to leadership. I said, let's be honest. I love those low priority bugs. None of them ever getting fixed. This is just not happening. Let's be honest about it. Let's change the process. Let's delegate one person every week to look at high value bugs, just high. If it's not high, they can send them to relevant teams, they can close them, whatever they want to do.

If they're, for some reason, passionate about this medium bug, they can address it. But their focus should be on high. And then PM started coming to me like, Inger, I feel like our back to your person is not doing anything. Why my backlog is so full of medium tickets? And I was like, here is the data.

We're actually sending only 30% of the tickets to the teams and we're actually dealing with 60% of the tickets. So I respect your feelings, but data does not support them. So this allows you to have those conversations based on data, not on feelings, not on, I guess, ideas that people have.

That's why I love data so much. Again, another agent that I have, GitHub metrics. Pause all the PRs that we had, calculates the time, how much time we spent before we reviewed it, how much time it spent in the pipeline before we merged it, how many reviews it gotten. Again, it's pretty important data that everyone potentially should have, but why it's helpful for me?

We did anyone say vibe coding on the stage before? I'll be the first one. Okay. So at Relevance, we are encouraging our PMs and our designers to do a bit of vibe coding because it allows them to prototype faster. Sometimes when there are some bugs that really, really bother them, they can go and try and fix them.

But obviously, when this idea was raised, the first question was, will it put additional load on engineering? Because engineering still has to review all of those PRs. And if we merge them, we're making them our problem. So what we did? We just build an agent, an agent that looks at how many wide PRs from our designers and PMs we had, how many reviews those PRs required, how big those PRs were, how much time we spent actually addressing issues in those PRs.

So we can see if in the long run it's going to be a problem for us or not. So I think it also helps us to do a bit of PR review celebration because we're trying to do this cultural shift to make sure that engineers value doing reviews a lot more than value actually writing code. Because LLMs are getting good at writing code.

But reviewing, understanding, does this actually solve the problem that we're trying to solve? Is it going to be a nightmare to maintain? All of those things are picked up in review. So we're trying to celebrate our reviewers a bit more. We have like weekly competition with leaderboards around people doing most reviews, people reviewing specific reports, etcetera, etcetera.

Again, this is data where we use a genetic AI to actually change the culture within the company. And I think this is pretty amazing. I have Donna. Donna functionality is very simple. She just looks at linear pulls all the tickets that were completed and tickets that are in progress, and allows me to have this overview of what's happening in engineering.

What are we currently building? We also use LLM to sort of package them so I can have intelligent conversations about here are all the generic front end improvements that went out last week. This is very helpful. It is not super detailed, but again it allows me to understand what my team, is a pretty big team now, is doing and how. And this is the vibe.

This is exact engineer who analyzes all of my PRs and gives me all the data around those PRs. We only have him for the last four weeks, so we'll see how that's going. But for now, the results are actually pretty good. Designers are super passionate about their PRs. They're doing their best. They're learning so much from engineering.

I can see them like paying on reviews with my engineers. It's absolutely adorable. And I have a secret agent. So this is the agent that I don't share much. A story of this agent goes like this. It starts with a very passionate manager, really wants to be helpful to people she manages.

The problem is that it's 15 people. So even if she tries really, really hard and works twenty hours a day, there is no way for her to actually manually go through all the PR reviews and give people valid, helpful, technical feedback. So what this agent does, it actually analyzes all the PR reviews that specific engineer received in a specific time frame and then tries to find, are there any specific recommendations that are repeated from PR review to PR review?

Is this something that this junior should learn? Is this something that this mid level engineer should pay more attention to? Obviously, don't fully trust that I check. I also have like peer reviews. I also talk to people. I hear ten hours of one on ones in every week. So but it's still very helpful for me to see what people are doing and have very important conversations with them.

So you just met my agents army. I'm still building more agents. I actually find it very interesting. I really enjoyed it. Relevance has those like little pixelated agents that run around your screen and do things. It's adorable. Sometimes you just have an animation in the background. Also, you don't have to use relevance for this. For this talk specifically, I actually went and rebuilt all of my agents just in scripting.

You package it. You send it to LLM. You call the tools. A lot of those tools have MCPs now, so it's pretty easy. You can do it right now. You can use Corsair to write code for this for you as long as it solves the problem that you have. But the approach that I recommend is a bit different.

Don't allow the industry to tell you how to use LLMs. Look at what you do. Look at what annoys the hell out of you. Look at what makes you a bit less productive or maybe more bored and try to find a way to delegate those things to a giant TKI. Because if if that's what makes you 10x engineer or 10x manager, your level of enjoyment from your work is actually very important. So do more things that you enjoy and find a way for a giant TKI or LLMs do the rest for you.

So build your agents if you want to. I highly advise to do it. Also, maybe use relevance platform for it. Or write scripts, whatever works. But definitely look at them as tools. Use them as tools that exist to make your life easier. Don't give them the full control. Don't give don't give them the things that you love doing.

Like I love my one on ones. I enjoy building connections with people around me, people that I work with. And can LLM be a better psychotherapist than I am? It probably can, but I'm not going to allow it because I also have business context and we all know this 200,000 tokens context window. I can keep more. I'm already better than LLM.

And that's it. That's all I had for you today. Follow me on LinkedIn. Follow me on Instagram for Corgi Photos because I have a lot of those. Thank you so much.