Why AI coding tools might not make the slightest difference

Quick microphone check. Yes. I can hear myself. Alright. Good afternoon, everyone. I'm gonna start us off with a little bit of a history lesson, and I'm using the same timeline. Oh, we've got a bit of feedback. Sure we're controlling that. Same timeline as Jeff this morning going all the bay way back to 2024, which I'm declaring here as the year of GitHub Copilot.

I don't know if you can cast your mind back to 2024, but at least for us, that was kind of the predominant tool to a certain extent. It was the only game in town if you're an enterprise. So we were rolling this out. Adoption of that was going up. It started off obviously with the the enthusiasts trying it out, started rolling through the organization.

It was really easy to roll out. We're already on GitHub. It cost $19 a user a month. Easy decision to make. Easy. Good. 2025, declaring the year of Cursor. So people got excited about Cursor as the front runner, and we roll out Cursor at Seq. Adoption for AI coding tools in general kept going up for us, so got around to the 60%.

Price went up a bit, still fairly controllable, $65 user per month. Don't have to think too hard about that. 2026, so far, year of Claude code. Certainly, over the the holidays, sentiment seemed to turn really sharply towards it with Claude Opus 4.6, And we've seen our adoption keep going up. We've seen our costs keep going up.

You might consider that kind of a a low baseline. We're at the start of the year. It seems to be trending in the uppers direction still. So these are all good numbers. It's good to see graphs go up into the right, and you'd expect a lot of productivity out of that. That 10% number comes from DX. We did a longitudinal study of PR throughput increase over the last year and they saw between 515% productivity uplift.

Now if you're at a large company, a lot of engineers, maybe that's okay. Maybe you're okay with a 10% uplift. That's quite a bit compared to how much you're paying for these tools at the moment, but it starts to get a bit uncomfortable. I'm here to talk about why aiding AI coding tools might not make the slightest difference.

And if you saw Jeff's talk this morning, the keynote, you might think we're worlds apart. We're probably not, but I'll be explaining that through the lens of the theory of constraints. So I'm not gonna give a full overview of theory of constraints. That's a talk in and of itself, but I can tell you the key claim of the theory of constraints is that for any goal orientated system at any point in time, there is a constraint in the system. So I've got a little toy simulation here.

We can imagine those particles of little bits of work going through the system. So if you're in a situation where we're calling it implementation here is your bottleneck, then the thing you should do to get more throughput out of the system is to work on the bottleneck to make increase the throughput of implementation. And then once you do that, the constraint moves somewhere else.

So I'm declaring here the throughput's going from a 100 to a 145. They're arbitrary numbers. And in this case, the constraint has moved to solution design. Again, don't think too hard about whether this model is your system. This is a toy example. If you were to increase the throughput of implementation in this system, throughput remains at a 145 because you have not increased the throughput of the bottleneck. And I think we can maybe map that onto that history. So maybe you're in the situation in 2024, GitHub Copilot.

It was good. It kinda lifted the floor of productivity. That's how our model for it. We brought on cursor. It's had a different model. You can do more with a single prompt. As we're bringing on Claude, we're certainly spending more. It's not clear that we're getting more throughput out of the entire system. So when you're constraint is coding, so this May was maybe me with ideas I'd have on the weekend circa 2018.

Coding was the hard bit. Finding time for it, you'd end up doing a lot of research. Now it's more like this where coding for me is not the bottleneck on the weekend for my weekend ideas. So where constraint where the constraint is coding, AOS effect is absolutely large and obvious. That doesn't mean the constraint has gone away or your system is constraintless.

So maybe on the weekend, you're running out of weekend ideas or you can't be bothered marketing it or something else is stopping you from progressing that. Is coding your bottleneck? Show of hands, who thinks coding is their bottleneck at their workplace? Oh, one person. Excellent. Okay. So let's look at where coding fits in at a large organization. So here's a extremely simplistic model of what an engineer is doing. So there's feature work coming in and it gets coded.

And maybe from a long distance away, that's what it looks like. At least for myself, when I code, tend to introduce bugs, so I have to debug those. So our task expands. We have PR review. You've gotta read other other people's PRs. You gotta make sure that we're shipping the right things. Feature work doesn't come in fully formed.

You've gotta plan it. You've gotta schedule it. You've gotta think about how it fits into the system architecture and design. Feature work is not the only work that comes in. There's security and compliance work. There's maintenance work, which I dare say is gonna keep going up as we keep shipping more code. An engineer's week will also be filled with a certain amount of learning new technology. Here's the here's the bigger picture.

So coding is not implementation, and it's not engineering. People are taking photos, so I'll pause for a moment. This is maybe more what an engineer's work week looks like, and these numbers are from a Microsoft study called time warp, and the numbers are, you know, a bit aggregated. So if you look at that paper, you won't see exactly those numbers. I've had to squish them up to together a bit, but it's representative. So if you're kind of rolling out AI coding tools and that's all you're doing for enablement, the the thing that they are going to affect most is coding and if coding is 15% of your time, even if it's making you 10 times faster at coding, you might still just be 10% more productive overall because that's the part of the system you're optimizing.

So these other activities I claim have something in common which is failure avoidance. Your org has to avoid failure. So do a little thought experiment. Think of if you had someone in your organization who had complete autonomy, had all of the keys to the castle in the organization, how much damage could they do in forty five minutes versus how much benefit could they have to the coming in forty five minutes.

So there is very good reasons that your company probably has a lot of review processes, different ways to avoid failure. I think to Knight Capital which was a financial services company in 2012 bankrupted itself in forty five minutes due to a bad deploy, $440,000,000 out the door. So there's these kind of fat tail events that you've got to avoid. And often those sorts of failure avoidance activities involve a lot of coordination which is where things can really slow down.

So the the need for this failure avoidance is very real at large organizations. There is maybe a tendency to not clean up after these activities and they tend to accumulate. And so often you're in this process and you're not quite sure why this thing exists. It probably emerged out of some near miss or something like that. But it's it's going to get in the way of getting that 10x improvement and it probably should.

So where we are so far, every system has exactly one constraint. If your constraint is coding, aiO's effect is large and obvious and your constraint probably isn't coding. What is your constraint? Can you look at a cute little model like this and work out what your constraint is? No, you cannot. Reasons are you don't have one of these.

I'm assuming. I'm assuming you don't have one of these. If you do, please let me know and if you do have a accurate map like this, I want to know how long it took you to get that accurate map and I want to see it, but you don't need them. I would I suggest that you can make some educated guesses on where your constraint actually is.

I would suggest that if you talk to 20 leaders at your organization, get them to say the top three things that are getting in our way from shipping, you'll probably triangulate on what your actual constraints are pretty quickly. You can then experiment on some interventions in those areas. I think in talking to those same people, they'll have some good suggestions. You can measure the results of that, and you should.

So we use PR throughput as a proxy measure for activity in the organization. We don't put targets around it. We just measure it. And that's an indicator of do you have healthy teams If your PR throughput is in a reasonable range, happy to talk to people. I know that can be a little controversial.

We're happy to talk to people about that and keep repeating that process. Where could your constraint be? So I'll I'll flash up some suggestions of areas you might like to look. Your build and test experience, how much capacity do your engineers or builders have for deep work, How confident are you in changes? Cross team collaboration, decision making, documentation, ease, release, incident handling, incremental delivery, etcetera. These are some areas you might like to look, none of which are coding or like typing speed.

These are all productivity drivers. This is what the company DX calls productivity drivers. We use DX as our engineering intelligence platform And I'm gonna focus on just a few of these being the build and test experience, decision making, and cross team collaboration, which are ones I'll explore in a moment and and ones we've been focusing on.

So build and test experience, just as an example, that minus 24 is because we use DX, we get industry comparisons, and that is how far we're off on our build and test experience for the industry comparison. So that was our biggest gap. And again, not how fast we can type.

I'll get into how we improve that situation in a moment. I will say, so I've gone from make some educated guesses, talk to people, to a full engineering intelligence platform. The engineering intelligence platform gives us these industry comparisons which are very helpful, but it was also just the highest voted issue.

So if you've just taken the votes, which is a much easier thing to start with, doing a survey and saying what's your top three best voted issues, this is where I think you'll find there is kind of obvious and low hanging fruit that engineers have been whinging about for a few years. As a bit of a side note, if you're looking to make investment cases around these things, so DX and I'm sure other engineering intelligence platforms make claims around the there's a developer experience index score that you get from summing up or aggregating those productivity drivers.

And DX makes claims around they found correlation between how much engine engineering time is saved and that score, which means you can do some fun maths like this. Like, if you have 600 engineers and you think you can maybe uplift five DXI points and you apply some numbers to how much an engineer costs as a fully loaded thing, you could potentially save $3,000,000. That's not the profit and loss.

I'm sorry. That is regained productivity. So for the engineers you're already paying for, but it does help in making the case for these things. Let me talk through a few of the interventions we've made. So the first one I'll be talking about is build wait time. So evidence for this being constraint was that we had a very large gap between us and the benchmark, which I showed, and that was the most voted issue.

I didn't see the ten minute marker. Saw the five minute marker, so we're good. The intervention was a lot of things that the the team probably had on the backlog, but this put a real impetus behind that investment. And some of it is quite involved, but it was things like developing the the warm pool of build agents, a bit of predictive agent scaling so that we're scaling up in anticipation of build jobs coming in, a lot of caching.

We found that the median build wait time has gone down 15% over the last year and it's also gone from our most voted issue to our number four most voted issue. So engineers are noticing the difference. Was this an AI thing? Can I help? It's not the main game. We we've got some nice skills around helping people optimize their builds, but it wasn't the main game.

It was a lot of traditional engineering. We're going up in our use of AI now. So AI assisted project inception. This relates to decision making. So evidence for projects inception or decision making in general being a constraint was that we had a 12 gap with our industry comparison here. Important projects would take weeks to coordinate on details.

We ran some as experiments, we ran AI assisted initiative kickoffs with the help from our friends from AWS. They have a strange name for these things. They call them unicorn gyms, but we've quite liked that practice and we're we're rolling it through the organization. And where AI is helping with that is it kinda gets you to the 80% of an integration plan.

So if you're kicking off a big initiative, it involves maybe a dozen teams, maybe as many systems, working out the interaction points, the contracts between those systems would be something that traditionally would kind of get passed around a lot. You'd have to coordinate on a lot of meetings. If you can get off the blank page quicker, we find that we got to consensus much quicker.

Last ones that I'll talk about is scale localization, introducing a new language. So when we had to introduce a new language to the platform and across our dozens of systems, it would take a lot of teams and a lot of coordination. And we decided to experiment with just not doing all that coordination and getting one team to do the work across 40 repos, across thousands of translation files.

And the result of that is that it's very near completion in matter of weeks rather than months, and the engineers involved in that has spent a few thousand dollars in tokens in the process. So they're they're at the top of our spend for the month, but we we don't mind at all when they're having this kind of scaled effect. And this is an example of something that would not have happened without AI.

So some overall results. So we did go up three DXI points which I've referenced before. That was we've draw an equivalence there to about a high $1,500,000 for our team. Our PI throughput went up 20%, so we're we're beating the averages. And we we had uplifts in these DXi productivity drivers. We'll likely focus on these things next.

So why AI coding tools might not make the slightest difference? One is that coding isn't your constraint, probably. Another is you don't know what your constraint is, and a third is that AI might not fix your constraint. So the response to the first one is don't expect weekend results. Don't expect to just introduce Claude code and you're done.

That's where you'll get the 10%. If you don't know where your constraint is, make some educated guesses, measure and repeat. And with AI potentially not fixing your constraint, remember to use AI as an ingredient and not the whole recipe. So you definitely shouldn't be ignorant of AI. You should be curious about it. You should be working out where it can help you.

There's plenty of areas it can help you, but don't expect to just roll it out and for everything to get better. That's me. That is talk resources. Thank you very much.