Legacy Software + Agentic Discovery

Awesome. Thank you, team. An existing code base is a crime scene. You arrive after the incident, it's been tampered with sometimes thousands of times, there are many witnesses, most are no longer around and the ones that are around, maybe they're not that great at communicating. There's millions of clues, every product decision, business decision, architectural compromise, they're all leaving evidence. But what exactly is the truth?

The working software is the truth. I can see the truth. I click a button on the screen and I can visually see what happens. But I only see the external effects. It's not clear that that button is only there because I'm in this user permission. I also don't understand that when I click it, something behind the scenes happens invisibly.

We can't understand exactly how software systems work solely from interacting with them. Now we know that code is the source of truth and it doesn't matter if that code is messy, gross, over engineered or scattered across thousands of different code files that you

would love to

refactor. As engineers, we've spent our careers learning to wield these mystical programming languages so we can communicate to computers using code. But most people in your organizations cannot read code. This made engineers the interpreters or the gatekeepers of that truth. Now to understand what happens when I click this button, it might seem, you know, easy but it could be deceptively complex.

It might involve hundreds of invocations across front end, APIs, back end, maybe the database logic, microservices if you went down that route. And whether that's a developer manually reading and navigating these files or an AI agent traversing via many tool calls, understanding the truth of how this feature works just from reading the files and functions every time is a really cumbersome and splintered process.

But if we roll up that complex call graph of files and functions, we can build a feature definition, a readable layer that explains in plain language that everyone can understand what this feature does. It doesn't replace the code by any means, but it augments it with traceability back to the exact files that implement this feature and how.

Now features don't live in isolation. Understanding the relationships between software features is just as important as understanding their individual functions. One feature might navigate to another feature. It might contain child features or it might include or reference functionality in global features. For maintaining knowledge, this content needs to be centralized and linked, not repeated again and again.

Duplication is the slow death of documentation. Now you can ask Claude Code or Codex to document what happens when I click that button. And for a small system, that might be perfect. But as Matt mentioned in the last talk, like whilst amazing at coding, these coding harnesses often stop and fall short just as it gets to good enough.

It might miss like tracing every conditional branch and pathway to the very end. And when we're talking about perhaps gigantic complex legacy software, the devil is in those details. And this is what we've been tackling at UserDoc with a feature called Code to Docs where we reverse engineer code, whether that's COBOL or React or everything in between, into those detailed software definitions.

Over the next fifteen minutes, I'll take you through the journey of this process, what works, didn't work, lessons learned, all that stuff. Hopefully you guys can find some repeatable patterns that you can take away to your own work. So version one of this and I'll just say version one was started about fourteen months ago. And I think in AI years, that's about seventy two years ago.

It was a 30 step LLM pipeline where each one of these individual steps in the chain was its own complex subsystem. For a large code base, we might be talking tens of millions of LLM calls with a token cost of, you know, 30 to $40,000. And we were optimizing for quality over cost. I wanted this to be self serve so you could rock up, connect your code, answer a few questions, and depending on the size of the code base, you know, between two hours or twenty hours, you would have your detailed software specs.

As this was just version one, I'm not going to go through every step, but I'll give you a high level overview of how we initially tackled this. First thing, we turned every code file in the system into a single language agnostic JSON representation called a codenalysis. It contained a brief overview of what it did, the functions, the arguments, the data flow, categorization to metadata, etcetera.

As future steps in this pipeline could get ridiculously expensive, this gave us a very token efficient artifact to work with. We then built a code map, which is essentially like Yellow Pages to quickly look up file dependencies and relationships. We could we could leverage static analysis tools like TreeSitter or Roslyn if it's dot net.

But importantly, this needed to work across architectural boundaries. So if a front end TypeScript component calls a URL in the back end for an API, we needed to map what API endpoint resolved that data. If that API then wrote to a database table and that triggered a database trigger on the SQL layer, we needed to map that dependency as well.

We used vector embedding to then group those artifacts into high level themes by business domain. So in a fictional system, this could be authentication, ordering, invoicing, etcetera. We grouped every code file into those themes, so essentially slicing a large piece of software into small little domain specific code bases. This was primarily for context window limitations.

But even still, with like 1,000,000 context windows we have today, a large code base can be 1,000,000,000 tokens. So we're talking ginormous amount of information here. These themes often were actually really useful as well as a first human in the loop checkpoint so customers could take an initial look to make sure that, you know, it was covering everything their system had.

For every theme, we did thousands of LLM passes to basically tease out and uncover the features. So the features are submit order, add to cart, process order, receive notification, etcetera. The tricky part here is if you ask 10 people in a room what a feature is, you'll get 10 different responses. It's not a scientific unit. Is it a screen?

Is this individual address lookup component, should that be factored into its own feature? It's actually super complex. So correctly sizing and splitting those features into like maintainable documentation is a real art. And we spend a lot of time working with beta customers to refine that. As I mentioned, features have relationships. So not just that feature a links to feature b, but also that, you know, once we process a giant code base, we might find out that there are cross cutting features.

By that, I mean very repeatable bits of information such as soft deletes or like handling, you know, notifications that are referenced from thousands of features. So very very importantly, we want them linked and not repeated. Basically, after all these 30 complex steps and human and loop sign offs, we had a detailed knowledge graph that we could then synthesize into that feature definition that I showed you.

All those upstream steps were simply in place to make sure that this was accurate and trustworthy enough for customers to use moving forward. They might want to do a modernization. They might want to rebuild. But either way, they needed to be able to trust the artifact. And version one worked. We did rigorous testing over six months, evaluations across different languages, frameworks, and five very large open source code bases.

These were manual testing. There is no way we could really actually get another LLM to be a judge there. So we spent a lot of time, yeah, doing a lot of business analysis essentially. We hit our goal of 85 accuracy. And that, you know, that doesn't sound a 100% perfect, but we figured out quite quickly from feedback that humans are around 85% accurate when looking at a very large complex code base.

And this provided significant value to early customers. We got five customers very fast in in the next couple of months of launch, and we saved them man months of manual documentation and and did it in hours. It worked until it didn't work. Along came a customer with a piece of software with 52,000 files.

It was 12,000,000 lines of a thirty year old Visual Basic system. They were about to embark on a mammoth rebuild over the coming years, and they, first of all, needed to understand what was the current state. No one in their organization knew all the features and how it all worked, and they certainly didn't have accurate documentation. A new legacy systems had skeletons in the closet, but this was next level. There were single files which were well over 1,000,000 tokens.

Their inheritance changed like a 100 levels deep, and core logic mainly lived in deeply nested stored procedures in the database. I nearly closed down my business and ran away, but we got it done. Instead, we worked, full time for two months, many meetings with the customer and their subject matter experts, traversing the code base, finding those skeletons. We ultimately either replaced or highly customized nearly every step in that 30 step pipeline just to work on this system.

But it was a success. We delivered seven and a half thousand detailed feature definitions where the customer came back and said they estimate that that would have taken them around five years to do internally. But it certainly couldn't scale. Our next couple of customers had equally weird, gigantic software in a completely different flavor of weirdness.

It was those unknown unknowns which couldn't scale. You know, we'd spent around $250,000 building out version one and testing it. And whilst those learnings were priceless, we couldn't keep adding steps to it like a state machine workflow to deal with the unique complexities of software. So probably by the name of the talk, it's not surprising to know that agents are the answer.

I knew that with the right resources and setting that agents can deal with those uncertainties. I tried initially letting an agent just orchestrate the version one pipeline, but it just wasn't it was too rigid. And agents need agents thrive with a different kind of communication mechanism. So in early February this year, I made the decision to throw away version one and completely start version two from scratch as an agent first CLI. And this is how I did it.

If you picture software as a sphere on the surface, on the outside of the parts that other systems and people touch. You might have a user interface, screens for human, systems integrations from other systems or a process or batch job for like scheduled tasks. Now not all code, not all files in a code base are equal. Everything inside this sphere exists to support that surface.

It's the plumbing, the logic, the persistence layer. So first, we send agents on a discovery mission, map the surface of the sphere and report back what they find. It's a preflight mission that also does discovery and can is used in these later steps. From each of the entry points that have been found, agents then mine the execution tunnels inwards to the sphere, finding both direct and indirect file dependencies and figuring out those relationships as they move through it. Whilst mining the tunnels, they report any of the high level features that they're finding to an overarching orchestration agent, and they're taking into account those feature sizing and slicing rules I mentioned, again, as they work. This orchestration agent allows, you know, the communication between agents, and importantly, it can remove duplication as it goes.

Each high level feature is then deeply excavated, finding the data lineage throughout the system and building the relationships between features. Agents don't often need to read the entire file. We can use static analysis and, you know, LSPs to pluck individual functions and and variables so it can get what it needs in a token efficient manner. And the output of those three steps I showed you is that same detailed feature definition that we got from the 30 steps in version one. And we hit parity.

So with tens of thousands of agents using those deceptively simple steps, we hit 85% accuracy on our eval code bases. I call these missions as opposed to steps because in a pipeline, obviously, an agent might do like a 120 tool calls to actually get something done. But importantly, this didn't solve those unique weirdnesses I I spoke of. For that, we had to give the agents the ability, with three custom tools.

So the first one, like any AI, it will make it will make assumptions if it doesn't have context. And in a code base, this would be unresolved references, logic outside the repo, maybe something important in a CSV file on some shared drive that it can't read. So we want the agents to stop and ask questions instead of assuming. Those questions are initially to other agents in the swarm.

If they don't have the correct answers, it bubbles up to our team, and we can actually bubble that up to subject matter experts on the customer side. Those questions and their authority of answers then become shared context for the rest of the agents. Some discoveries are expensive via tool calls, and they're reusable. So resolving an API endpoint string might involve 20 tool calls across routes, middlewares, inheritance, etcetera.

Once an agent figures something like this out, it can, drop a hint for other agents which they can then use to upskill. So instead of repeating the same boring nuance again and again, the swarm gets smarter as it progresses. As thousands of agents are doing this through a code base, deterministic patterns can sometimes emerge. They might find something that shouldn't be left up to AI.

For example, say all the user interface text comes from a giant XML file of translations. Right? Agents have to keep on grepping again and again. To solve this, we allow agents to actually write deterministic hooks. So they can write a hook, for example, that when the other a next agent requests a user interface file, the hook fires and the translations are automatically replaced in line.

So that a Next agent doesn't even need to know about this complexity. It's automatically being solved. Those hooks are approved by a human. And as we go, the weirdness becomes the infrastructure. The agentic rebuild worked with those three separate missions and those agent tools. It was faster, significantly cheaper, and importantly, more scalable.

We hit 92% accuracy on our eval benchmarks. In the last five weeks, we've processed over 7,000,000 lines of code through this agentic rebuild. The last one was a forty seven year old programming language that I'm sure no one in this room has heard of. And that system alone probably would have taken again another two months of custom work on that original pipeline.

The customer again said it would have taken multiple team members, multiple years to get that result. And this is a very low resolution image of what the customers end up with. So this is the UserDoc platform. We can see the code organized by their business domains. You can see the individual features and their relationships between each other.

You can click and dig down into those features, seeing their detailed definitions with links to their supporting features. And you can also see the code files that back up and implement those features and how they do so. Customers then use the broader UserDoc platform to collaborate, make version control changes, use AI to make new features, generate test cases, and ultimately integrate that with AI coding tools or other AI tools.

The research development iterations of v one took a bit over twelve months, about fourteen months for for two people. The AgenTic rebuild took me just under four weeks from scratch. Part of that is because in 2026, an experienced software developer armed with a state of the art coding model and a few grand in tokens can do amazing things.

But I think the main reason is that when agents are using the right harness, the right model, they have the correct tools. They can ask each other questions and other people questions. They can then share that knowledge, and they can importantly identify and build deterministic hooks that this is the future of software engineering. And something that is significantly less code can do amazing things.

That's the journey. That's the adventure I've taken over the net the last fourteen months. If you guys are familiar with spec driven development, you might think that, you know, this sounds a lot like specs. I think specs are kind of future historically facing. The word documentation also sounds a bit historic. So I like the term software definitions, but irrespective of what we call it, I don't think these should be trapped inside a code repo just for developers.

My general belief is a shared accurate definition of what your software actually does readable by technical, nontechnical teams alike and your AI agents is gonna be a huge organizational asset for every business moving forward. It's my story. Thanks, team.