Scaling Coding Agents (without breaking your dev team)

Good afternoon. I'm AJ Fisher. I'm fractional CTO as John explained, and primarily, I work at the intersection of tech, business, and experience. And mostly, what I do is work on performance systems for teams, around how do you do transformation and innovation at scale.

And as you can imagine, AI has now very firmly become a big part of that sort of conversation and the sort of stuff that I work on with my teams across various organizations. It's been a pretty wild couple of years within the transformation space. We've seen all these talks today, which have been great, and you can just see how much movement is kinda happening and how teams need to kinda adapt very rapidly to be able to scale up to start to use this technology in anger.

So all of these tools have gone from not very good, let's face it, a couple of years ago, and expensive and not very good, through to becoming good enough, but now they're becoming cheap enough and reliable enough that we can start to deploy them at scale across the entirety of our teams and potentially out into the organization, as well. And this changes things quite a lot because for the first fifty odd years of software engineering as a practice, dev time has been the most scarce, precious commodity that we held within the wider team. It was something to be guarded, and suddenly, we're going from this position of famine to flood, and that changes things a lot because this flood of capacity brings with it all these new things that we have to deal with.

And what can we do with it? But this is not the first time this happens. You know, regularly within the span of human history, we bring new technologies to bear, and it changes things from scarcity to abundance. And you only need to look back at the advent of mechanization during the industrial revolution to see how these kind of changed things pretty conclusively, but also, it's got a lot of parallels to what's going on in our world now.

So press press press. Before mechanization, goods were artisanal. They were heavily crafted. They were kind of scarce, precious commodities that were bought and sold. Output was completely linear. It was proportional to the amount of skilled labor that you could bring to bear to actually make something, and how big a problem could you actually solve as a result.

As we brought mechanization into the process, this created new abundance, but it also created chaos. Factories, all of a sudden, springing up, started creating these shoddy products that went out into market. We had cities explode in size overnight, as people rushed into the city to get these new jobs.

Professions collapsed and changed. Accidents, injured workers, as they were going through this stuff. It was fast, it was messy, and it was frightening for everyone who was involved in it. This kind of starts sounding pretty familiar to what's happening with our work practices. We're getting more merge conflicts, we're getting code sprawl, we've got agents going rogue and deleting production databases that you might have seen. It's the same chaos transported to a different technology and a different century, but ultimately, it's the same thing.

But out of that chaos came new systems. So we got supply chains, we got labor laws, we got standardized parts, quality assurance systems, management science, all of these things that were created in order to tame the chaos that came about from abundance. And so as we think about the scarcity of developer time, our systems that we've developed over the last fifty years of software engineering have evolved to protect that super scarce resource across our teams.

So we filter, We have business cases that you've all got to do to get work done. We have return on investment scenarios. We have planning committees. We have prioritization. We do more prioritization, then we change the prioritization. All of this is to kind of say no to development more than we say yes because it's such a scarce resource.

But now that scarcity is starting to change, and those systems become less relevant, and we're starting to unleash more work to be done. So we need to redesign some of our systems and the way that we work collectively so that we can start to capitalize upon this abundance. So that's really the focus of what I wanna talk about today. And so really, you know, thinking about these kind of three big areas. So how do we start to bring context so it's cheap and fast to kind of get up and going?

How do we set up our guardrails so that at scale, we can deliver with confidence and securely? And then how do we start to orchestrate the activity working across not just our agents that we're deploying, but also across our human teams as well? So how did that work in conjunction? So let's crack in. So over the last twenty years or so, our code bases have exploded in size and complexity, and a lot of that is because our applications have got more complex.

They're bigger. They kind of do more things. So what once took a day or so to onboard now takes weeks. Jacob talked about this as well where, you know, we used to kinda, you know, be able to just get up and going really quickly. Now if I'm a mid level developer joining a team, taking a few weeks to a month to onboard onto a code base and get up to speed, if I'm gonna be working on that for the next year, that's okay.

You know, that's not a bad problem. You know, I've got time. We'll get the payoff over time. That's okay. But when I start to deploy agents, we need to get that context fast. You've got ten seconds before the context that's loaded into that agent either works or starts sending it off down a rabbit hole.

So can your systems deliver context that fast? That's the key. Because if you don't, if you don't get them context instantly, then they're gonna start spinning their wheels, they're gonna go down rabbit holes, they're gonna burn tokens, they're gonna burn time, they're gonna produce junk, and now your team has to clean up that mess as a result.

So we need to build context faster, but we also need to build it accurately. So there's a couple of ways that I can think we can try to do this. So one is about how we're arranging our code. What's our code base actually look like to be able to provide that context, and how are we getting documentation layered into that so that every step, we're getting good information about the code that's actually being worked on. Right.

So the first thing we need to understand is that agents, particularly when you deploy them into a private repo, they're really bad at the moment of spanning multiple repos, especially if they're private. So those, scenarios where you might have had helper repos, like they've got a whole bunch of utilities and libraries and stuff like that, if they're private, that means it's not in the agent's training data, which basically makes them invisible as far as it goes.

And so like a dev sort of sitting down looking at an import statement and a whole bunch of function calls, sitting there going, what the hell is this? Where did this come from? Where's the docs for this? Where's this who even wrote this repo? You know? You're basically forcing the agent to do the same thing, and so consequently, it's gonna just go off and hallucinate and kinda come up with its own guess about what it's actually doing.

So I'm not necessarily gonna get into the pros and cons of kind of monorepos. That's a whole other talk, there's been lots of those given, but think about it through the monorepo mindset, if you like. So we start to co locate the stuff that matters together, but as we do so and we start to bring in stuff maybe that was sitting in these helper libraries and all that sort of stuff into our kind of application repositories, We need to make sure that we're kind of setting it up nice and well structured.

So we need to think about clear app delineation, component level delineation, all that sort of stuff, but also where do our shared libraries live, where do our utilities live, all of that sort of thing, and set it up so it's nice and clear. Also, as we start to bring more stuff potentially within to the repository context, we need to watch file sprawl.

Now, I do a lot of work in the JS land, in particular, and we are are really kind of like, you know, we do a lot of this. Right? We have code files everywhere. Right? So a single component might end up with, you know, a file for the component, another one for its routes, another one for its types. We've got classes somewhere else for its styles.

There's stuff everywhere. Now, as humans, it's annoying, but it's manageable, and our IDEs kind of enable us to do this to a large extent. But for agents, that becomes confusing. Right? So we might need to start bringing some of that stuff together and putting it into one place. And, yeah, okay, as a human engineer, looking through it, maybe I've got to actually use the scrolling wheel on my mouse.

But for an agent, now it's starting to provide that snapshot of context. So rule of thumb you might want to apply here is if you have to open up multiple tabs to kind of see just the picture of how a change might manifest and you're doing that routinely, maybe it's time to start refactoring it and start to bring it together a little bit more.

Alright? So we've talked about our structures, so now let's think about documentation. Jacob touched on this as well, which was great. So, you know, who loves good documentation? Come on. You all should. Right? Who loves writing good documentation? One, two. So we all love it.

It's great when it's there, but we all hate doing it. Right? You know, but the purpose is to zero us in on where the change needs to be. Now, the way that I've been thinking about this more recently is that, you know, good documentation that exists all the way down through our code acts like a ladder and it kinda allows us to explore and kinda get gain access to context deeper and deeper and deeper into the code base.

So at the very top, we might have our kind of our readme file that kinda sits in the root of our repository, and this is really our run sheet. It's telling us what is this thing? What's the structure of my repo? How's it all kinda set up? And then it's linking down into the kind of the the sublevel kind of readme's that are kind of talking about apps and components, and this is really the how to's of how those things work.

How do I run certain commands? How do I kind of do particular activities within this sort of thing? What are my utility functions? All of that sort of stuff. And then right right down at the code layer, you know, we start to bring back things like JSDOT, PYDOT, you know, documentation in line of the code. And the beauty of this is that this layering means that you can only you only need to take a thin little strip of the whole repo to start getting understanding about what's actually going on, which is perfect for an agent because you don't flood its context window by dumping the whole repo into it.

Right? And so now you can keep it on task more effectively. Now I've just kind of talked about readme files. Now, the elephant in the room is that every tool maker under the sun wants you to have an agents file. Right? You know, and that's fine. That's you know, they are useful. Right? But if you remember, agents and LLMs, well, LLMs really, were trained on human repos.

Right? And we have trained, all of us have trained each other to go and look in a readme file first. So where's the LLM looking first? Right? It's looking in the in the readme file. Right? That's its primary source of knowledge. So good readmes that kind of extend all the way down through your codebase provide that anchor for humans and your agents alike, and that's a good starting platform.

Then if you're noticing exceptions, you know, you've got a particular, you know, you're using a particular code agent and it's it's consistently going off in a particular direction that you don't like, use your agent's files to kind of deal with the exceptions. Nudge it back into kind of like, you know, behavior that you wanna see. But it really should just be for the exceptions, not the kind of, you know, the everything of documentation for the agents.

Alright. So a couple of things that you can start to do straight away, with all of this is obviously assess the quality of your documentation, but you can start to use particularly, you know, all the modern code agents to help you out with producing this stuff. Right? Look for the gaps and then kinda just task an agent to go away and produce it.

Right? It'll look through your code and kinda summarize it and do a really good job, and then you can layer on top of that. If you start using regularly things like Pydoc or JSDoc or that sort of stuff, even just a few files, you know, a component or whatever, your code assistants will see that and they will start suggesting it when you're working on your other parts of your code base. So very cheaply, you can start to produce this stuff, even in mature code bases.

And once you've started to get all of this in place, use an agent to kind of sanity check what's actually going on there. You know, set it a task. You know, can you assess all of this kind of section, this component, whatever? Do the documents still make sense? They can flag Drift. They might even make suggestions for kind of things that they can fix up, or you can apply your expertise, and you start to link all of that together, and you're building this asset that grows and compounds over time in terms of everybody's usefulness of it.

Alright. So now we've got all this set up, let's think about tool use and automation. So how often in your teams does someone forget to use a pretty standard kind of behavior like conventional commits happen? Yeah. A few hands. Or, you know, forget to run a build tool that kinda, you know, makes all the schemas for your API validation stuff.

Yeah. That was me causing problems in one of my teams last week. It happens all the time. Right? And this stuff happens because a lot of these behaviors are not codified. Right? This is where documentation gaps exist, but also, it's because it's really about team norms and culture. Right? And this is the stuff that happens in team meetings.

It happens over a coffee when you're discussing a particular problem. It's like, oh, yeah. We should totally do it like that. And no one writes it down, but everyone starts doing it. Right? Or it exists in Slack messages or Teams messages that now agents have got no visibility over that stuff. So as we start to bring more agents into the fray, they don't have culture. They don't have the team norms that you've established just by, you know, verbal kind of hand down of knowledge, so you need to start to codify this stuff.

So we need systems that consistently drive these types of behaviors because as we get that consistency, we start to get repetition. If we can do repetition reliably, then that allows us to scale. We start getting that flywheel effect of kind of getting that, you know, high speed repetition. So the areas that we can think about here are, you know, being able to deliver automations, doing hooks for quality, and then starting to use reference patterns, as well.

Alright. So as John said, I'm an old man, So this should be about as hard as it should be in 2025 to spin up a dev environment, you know, locally on your machine. It really should not take anything more than this. Alright? Docs are a suggestion. Scripts become a guarantee.

If 20 grads join your team tomorrow, you could hand them a Confluence page and say, hey. Go and do this. This is how you get set up, or just give them a script that works, and you won't be dealing with a week of, cleanup afterwards. And the same thing goes with agents. You need them to just get going really quickly.

Alright. Every common action that you take within your repos, you need a script or a CLI command to be able to kind of deliver on it really effectively, and then add it back into your readme and your documentation so developers know about it, agents know about it, they know it exists, how to use it, and when, to use it.

As we're starting to build out code, and Jacob touched on this as well, we need to start building guardrails in as well so that that way we know that the code that we're producing is secure, it's trusted, it's got high validity. So we can add things in where we're kinda creating code in whatever container that's kinda running in.

So we add the usual things like links and, you know, unit tests and all that sort of stuff, and that all happens at the commit level. But we need defense in-depth so that we're not marking our own homework, essentially, and that goes for developers as well as for agents. We need external validation that what we're producing is secure and that it's gonna work.

So when we start to push things up, we do things like static analysis, vulnerability scanning, risk analysis, all of this sort of stuff, so that as we're kind of producing code, and particularly if we're dealing with situations where we are producing code in no touch or light touch review processes, that we can flag risk really effectively.

So looking for things like new packages, changes to URLs or hosts that we might be using, new shell commands, that sort of stuff. Because as we start to scale up, we start to get more and more kind of code flowing through, reviewers need tripwires to set off alarms, not go hunting for Easter eggs after the fact. The final part of this really is about saying, you know, what does good look like?

So we can start to build reference designs that we can use to kind of nudge our agents in the right direction so we can kind of get them to say, okay, here's how we do things around here, and that stops this kind of, you know, an agent coming up, spinning up for a second, and then just coming up with its own way of doing something that doesn't really fit with what you're doing.

So this is a form of style transfer, which LLMs are really, really good at, and it's handy for new team members as well. And so this starts to build that kind of consistency play. Skeletons, boilerplates, templates, you can have tools that can actually generate this stuff, but ultimately, they're saving time, they're reducing variance, and they're making your reviews faster and easier, because people can use their pattern recognition to make it work.

So a couple of things that you might wanna get on and do is, you know, build out some scripts for, your common tasks, set up those hooks and get that all the way through your code pipelines, and then build out those boilerplates. And, you know, pro tip, agents are really good at building this stuff, you know, because it's small tasks that they can get on done with. Alright. So let's start thinking about how we bring this together in terms of our work and how we bring it together with our people. So every dev is about to become an agent herder, and it's really nothing like managing people because that agent didn't exist yesterday, and it's not gonna exist tomorrow.

You're asking developers to manage a school of super intelligent goldfish with the lifespan of a fruit fly. They're brilliant, they're forgetful, they're relentless, they're ephemeral, but they're pieces of tech. And if everyone starts spinning them up willy nilly, then you're gonna drown in chaos and, you know, have conflicts everywhere, and you're just gonna create more work for the team rather than less, which ultimately is what we wanna do.

So we need orchestration to tame the chaos here. So we can think about how we're tasking, how we're running stuff in parallel, and then also how we're coordinating across our humans and agents together. So if we think about the agents, they've got very, you know, relatively small context windows compared to us and no memory. So you wanna keep tasks really tightly defined.

We've talked about before, if you give them big, vague jobs, they start spinning off and they kinda go off doing, all sorts of wacky stuff. You know, I gave Gemini CLI a task the other week, and it was just Yeah. It went way, way off paced, and started building stuff I didn't even ask for. So the fix is small scope, clear outputs, clear definitions of done, all of that sort of stuff, and fast iteration loops so that you can see and give feedback.

Basically, treat them like a junior. Now once we've got our microtasks, we can start to run them in parallel, and this is where we start to get our flywheel. So different agents, different tasks, and we can start to, you know, really start to get the things moving, as a result of that. We wanna converge those tasks though into a consolidation branch, so you might need to start thinking about how have devs got environments that they can use to support this sort of stuff.

So think about transient environments and all that sort of thing. Because as you start building more stuff, you're gonna need to test more stuff, and validate that it's actually working. How much parallelism makes sense? This is more of a field question. As you use the tools and you start working with them, you'll start to get a vibe of kind of like, you know, okay, I can make this highly parallelized.

I can make this one smaller, you know, etcetera. But now we're starting to get into the point where our teams can start producing more work more effectively, you know, as individuals. But then we need to start coordinating all of that work together across our whole team, and the goal here is to really, like, take that goldfish and put them into a school that's all swimming together in the right direction, and not bumping into each other and causing problems. So those macro tasks is still where our humans need to spend time coordinating the activity.

Right? The engineers then go and do the microtasks and the parallelism and kind of everything else underneath, but our teams need to spend that time together, working together to plan out. And so some of my teams I've started seeing, they're spending a little bit more time on sprint planning and kind of stand ups stuff like that, because they're orchestrating the work between them because they know that they're gonna be using agents and they don't want to tread on each other's toes, as they go.

So to kind of wrap up, we've touched on these kind of three areas. So how do we get context quick and cheaply so that agents can spin up in no time? How do we set up those guardrails so that things are not going out of control and we can understand that we're making stuff correctly every single time?

And how are we orchestrating the activity of our agents and our humans together very effectively? So as I said at the top, we're moving from scarcity to abundance, and that abundance is gonna force us to redesign a lot of the systems that we use for work. It's not about replacing engineers in my mind, not even slightly.

It's about building systems so agents and humans can work more effectively together than they could on their own. If we build this well, then it's gonna help us tackle the bigger challenges that we have across our organization so we can tackle more serious problems, but also deliver better experiences to our customers, which is really what we should all be here for.

So here's my details. If you wanna connect with me, there's all the details there. That URL is also what the QR code goes to. There are heaps of references that I've kind of got for all of this sort of stuff that I've talked about. There's probably a couple of dozen in there that span everything from books to kind of blog posts and YouTube videos, all sorts of different things. So if you wanna kind of dive into any of these topics in a lot more depth, then that's the URL for you. Other than that, come and have a chat to me because I love talking about this stuff because, yeah, I love organizing teams.

So thank you very much.