The Death of Documentation

Okay. Is this thing on? Yes. Fantastic. Well, I hope to break the trend of finishing early, but I cannot guarantee that. I think I ran this talk in the hallway, and it was about twelve minutes. So let's let's waffle a little bit. Let's waffle a little bit. Yes. So documentation, a room full of engineers.

Is it your favorite thing? Hands in the air. Favorite thing? Anybody? Anybody? Favorite thing? Yes. Yes. Yes. A couple of you. Right. It's my favorite thing up until the point where it's not. Right? And I guess that's that's part of what this talk's about. So yes, I want to make the claim that documentation is dead. And admittedly, that sounds maybe slightly bigger than what it actually is.

And by the end of this talk, I mean, hopefully, we'll have a bit of a better understanding of where documentation sits within our engineering discipline and how we can move forward with documentation in mind in a post AI era. So that's the spiel. It all starts with a model, I think, that

we

inherited for the for the majority of computer history. And I think that model we inherited from traditional engineering. Right? In many ways, we've built software in the same way that we've built skyscrapers. Right? Blueprints first, steel second. Because obviously, in that environment, mistakes are very, very costly.

But in the early days, admittedly, like writing software was also not a cheap endeavor. Right? Like back when we were dealing with mainframe computers and punch cards, like a mistake in the source code of one of those programs, was catastrophic in many ways. And recovering from that was months, I don't know, ages. It's a little bit before my time, so I probably shouldn't talk to it with too much authority.

However, anecdotally, it sounds like it was a pretty terrible time to just YOLO code out into the, into the universe, which we're all becoming very accustomed to. Hey, vibe coding. I'm a big I'm a big advocate. Anyway, so yes, we we wrote things down because writing was cheap. It was a good way to get our ideas across, collaborate with our peers, thinking, all that sort of good stuff.

It's a cheap medium. We could put it down on paper, agree on the outcomes, build the thing, iterate on the thing, rinse and repeat, right? And so, I mean, that's the theory, right? But in practice, as we're all aware, you know, documentation is unfortunately the first thing to drop when the deadline looms, right?

When push comes to shove, we sort of prioritize the other aspects of our discipline, right? Shipping the feature is quite important. And taking a look at ourselves in terms of making sure that we document everything, all the shortcuts we took, all the architectural decisions we made on the fly, that wasn't obviously a priority.

It has become less of a priority in my experience at least. Now, I've been sort of building software for about fifteen years now. And I've seen it countless times. I remember the company I first worked with, we had our own knowledge base, and it was just a dumpster fire. Like, you you were pointed at it when you joined the company, and you you go in there, it's like, wow, what is all this crap? And you quickly close it, and you just sort of defer to the code if you ever wanna figure out what's going on, or the domain expert that happens to be in the room next to you. And so, like I say, that tends to be, in my experience, the first thing that goes, the documentation.

Right? And we put a ticket in the backlog. Right?

We say, yeah, we'll get

back to it. We'll definitely prioritize that next sprint. Right? And then that sprint goes by, and we made some made some more changes to the to the product, and they went documented. And so documentation continues to be neglected, right? And unfortunately so. But this is, at least in my experience, the world we have lived in.

And it's not getting any better with AI. So a phrase I keep coming back to is documentation is often a rumor about how a system used to work. Right? It's a record that we made at a point in time when this is the thing that we were doing. And we, as as the humans in the room, mostly tolerated this, right, as I mentioned earlier, because you could usually just go and tap the person on the shoulder who has their name against the git commit, you know, like, what what's to do with this?

Stupid crap? Why why are we doing it this way? And of course, you know, you have that conversation. The domain knowledge grows within an organization through the people within the organization. Alright? But guess what? I mean, I don't think it's any surprise. But there's now a third player in the a second player in the game, and it is in fact AI.

And AIs, they have a really hard time differentiating between fact and fiction. So when they read something and they see something else, they have to spend a whole bunch of time like figuring out what exactly is is the most likely in between for these two conflicting opinions. Right? And so from my perspective, this is the new liability.

Right? And it is in fact the new failure mode for these systems. Right? When you feed your Kodi agent with stale documentation along with your source code, the agent, as I said, sees these contradictions. Right? The docs say one thing, the code says the other. And so we burn tokens trying to figure out figure out what what actually is going on.

And so that's the worst case, right? We just spend a bunch of money, a bunch of time, a bunch of tokens. But obviously, like the worst worst case is production goes down, right? Because you ship something that maybe didn't do the best job at reviewing, but hey, you're low. And we blame the blame the agent because that was the problem.

Right? A silly thing, should have asked me. Definitely ask me all the time whenever you are unsure of anything is the prompt we've all sent to our agents recently, I'm sure. How many times has it come back and said, hey, I'm unsure about this thing? It's like, no, I'm definitely confident about this thing. I'll go build it.

Anyway, so in my opinion, obviously, stale docs, they're not just inconvenient. They're poison for the context window of these entities. And so the stopgap, the obvious first move is to basically automate away the bits of the documentation authoring process that we sort of often neglect. Right? So at Prefactor, we've adopted a bit of a system at the moment, much like this, where we point an LLM at linear. Right? And it goes and reads what tickets are in progress, what tickets have been done.

It goes and cross references that work and the conversations that happened there with the PRs that were closed in GitHub. Again, conversation that happens there accumulates all of this knowledge and helpfully generates a markdown document at the other end, right, which we use, actually as the way that we publish our public documentation website.

Right, so anyone building on top of Prefactor is building on top of documentation that we ourselves have not written, but have overseen as it goes through this pipeline. And it works. Right? It works. Like, if you go to our documentation site, you'll see some documentation. But I I don't think it's enough.

I don't think it's enough. And I think it solves the wrong problem. Right? Ultimately, I think it solves the wrong wrong problem. And here's why. Right? The inputs, commit history, source code, API schemas, right, these are highly structured deterministic sets of data. Right? And we're taking this high resolution input, and we're compressing it into natural language. You know?

And then we're expecting these machines to sort of take that natural language and turn it into something kind of useful for themselves to operate on. Right? So, you know, high fidelity to low fidelity, back to high fidelity, hopefully. And hopefully nothing gets lost along the way, but unfortunately, this is a lossy compression algorithm, right? As you go to a to a low resolution state, you can sure expand the the JPEG backup to be you know, four k, but it's gonna look pretty crunchy.

And so I believe that's the case with a lot of these issues we're seeing with AI agents is we're just kind of feeding them the wrong data or we're feeding it at the wrong resolution, right? And the example here is, you know, imagine tasking someone with recreating a CAD model from a a sketch you've taken and rendered from another CAD model, and being like, build the same thing, or like, you know, tweak it.

And it's like, okay, cool. Thanks for that. So I think the medium in this case is the bottleneck regardless of who's writing it. And so once you accept that the the medium is the bottleneck, the next move is obvious. Right? Let's just let the machines read the code.

Right? Here's a simple example to illustrate my point. Minified JavaScript in all its glory. Who here has had to have the privilege of debugging minified JavaScript in production? Alright. We've all done it. It's not impossible. It's incredibly taxing though.

And it's it's by no means enjoyable. Right. And I think in part, there is almost and this is a hypothesis, so, you know, throw tomatoes if you want. But I think that to a to a large language model, the compressed JavaScript on the right, left, right, yes, is indistinguishable from the plain language specification on the left.

Right? It sees this string of lovely compressed symbols, and it infers correctly that a holds the value that was retrieved from a database table. It then decreases that value by some transaction value. The result is then sent to a payment gateway, and there's a guard that rejects negative values, right?

It infers that A is in fact a user's account balance, right? I mean, how of how many of you would have picked it up in the time it took me to sort of explain that if I hadn't hadn't written the bit on the side? Right? I think you would have been like, oh, what

is this

crap? What is this crap? JavaScript. So so if the model can read code directly, the next question is, what's the cleanest possible input we can give it? Right? And my answer to this question, I believe, is deterministic specifications. Right? Written. Written in code.

Ideally, the code itself in its uncompressed glory. But so in this case, you know, an open API document, it's a service contract. Right? It defines the required fields, the types, the error codes, every action the system can take. And for our tool calling agents, like this is the ideal format from my perspective.

And because the spec is deterministic, right, when combined with the source code, minified or not, it becomes a translation layer. Right? This translation layer can be remixed on demand. Right? A Python SDK, a Go client, some CLI tool for some obscure operating system you've never heard of or used, all the way through to the onboarding documentation for the hire that just started yesterday. Right?

On demand, fresh as it comes, and accurate, hopefully. And so an interesting anecdote actually from yesterday, someone came up to our booth and we were having a discussion, what is pre factor, x y z? Come and find us at the booth later if you wanna know what pre factor is. I'm not gonna talk about that too much in this talk.

But they asked us, what is pre factor? And, you know, the discussion went the way that it did, and they said, well, do you have an integration for this specific agent harness that I'm using? And without naming names, it's a fairly new one, Open Claw esque, but not quite. And so we're like, no, obviously, we don't. We don't. We don't have time to build all these all these integrations. But we put a challenge to this individual.

We said, hey, look, if you are so inclined, maybe just ask it to, like, build the integration itself. Here's our open source doc, not only our documentation, but here is the source code for the SDKs that we have officially supported. Go and see if it will we'll build it for you. And I'm sure it's no surprise to the people in this room, But two hours later, he came back and said, hey, my agent is like talking to your product.

How great. I'm like, awesome. That's great. I think there'll be a lot more of that in the future. And so in that world, I believe that the combination of the deterministic specs, the code, now documentation goes from this rigid static thing to fluid surface that can be rendered just in time in whatever format the customer, the consumer actually needs.

So loosely speaking, I'm calling this just in time synthesis. Right? Effectively, same question in, same code base, same access to the specifications, right? But a different answer shaped shaped purely by who who is asking it, right?

So two examples here. We've got a new starter on day one, asks the question, how does the billing system talk to the email service? Right? They get a high level walkthrough. Not only do they get a high level walkthrough, but as someone that's new to the company, they can ask all the silly questions to this thing. And it'll be like, yeah, I'll figure it all out. No worries. And when they do get stuck, they can probably come and talk to a human.

But like the first the first cut is always, I believe, going to be this this culmination of your source code, your specifications delivered to the individual in exactly the format they want. And depending on which agent they're using, it can even be in the language that they care to consume content in. Right? Like maybe they're a visual learner.

Why not let them like actually build their own, like render their own diagrams, right, that are in the same, you know, whatever format they like. Anyway, that's one example. Obviously, another example is that a security order comes in and says the same exact question. Instead of an onboarding flow, they get literally a breakdown of where all of these systems talk to each other, what data they're accessing, how much PIs in all these transactions, and sort of where the boundaries are of this system in terms of like compliance and risk.

Right? And so as a result, there's nothing there's nothing for us to maintain. We just keep building the system, engineering the system. And the artifact that falls out the other end of that in this world is is, again, the documentation. And so this leads to the obvious conclusion. Drop the docs entirely.

Right? Don't even bother about it. Engineer, high quality code, high resolution inputs, LLMs to summarize the whole thing, Bob's your uncle. And so I argue that if the system can describe itself at any moment, yeah, the documentation isn't just redundant.

It becomes a liability. But as with everything, there's a catch. While a large language model can reverse engineer the heck out of some minified JavaScript, and given enough time, it will tell you everything you wanna know about this minified payload of React Peppered goodness. What it cannot do, regardless of how much compute you do in fact throw at it, it cannot answer the why.

Right? So code captures the outcome of a decision, It does not capture the decision itself. So here's a chronicle example, and this is kind of loosely based on my own experience at another company. But effectively, you ask the AI about some code in your code base, and it detects correctly that you're using MySQL, and you're using it in a kind of suboptimal way given the problem space, and that Postgres would be a much better solution to this, and hey, if you want, I'll go and just move everything over to Postgres, cool, fantastic. However, it does not know that we have signed, you know, a five year vendor contract that only supports MySQL, right?

It does not know about the CTO's decision to kill the migration to Postgres for cost reasons, and it also doesn't know about the legal restrictions around how we are storing this data that somehow makes Postgres MySQL, the most appropriate database for this particular problem. Kind of convoluted toward the end, but you get the picture.

Which leaves us with the job that the machines cannot do, and that's writing down the why. Right? So the industry standard for this is architecture decision records, ADRs for short. They don't aim to explain how the code works, they instead explain that on a given date, options a, b and c were considered, that b was chosen despite despite being slower, and that the reason was specific to compliance constraints and costing.

That's the record. That's the institutional knowledge. So the new division of labor looks something like this. Right? Machines trace execution paths, generate diagrams, write SDKs, and answer questions about the system on demand. Human's own, strategy, business context, the politics of the decision, and the the the constraints imposed by an organization.

The machine the machine handles the how, we handle the why. And so my conclusion, my dear friends, documentation as we knew it, the static relic of a time before AI, I think that's gone. Right? But documentation as a dynamic service, rendered just in time from code and context, that's just the beginning.

The practice isn't over, it's just different. Documentation is dead, long lived documentation. Thank you.

Thank you, Josh. I I did have a couple of quick questions for you myself, if you don't mind. Well, they they are out, but, you know, I was just curious for myself, and maybe others are. One was where are you storing the ADRs? In the repo, like, alongside everything else, just like one one kind of long file. And I've just forgotten what my other question was, So maybe you are off

the hook

after all. But I will come find you at the booth.

Absolutely. Come and find me at the booth. Yep. I'll be I'll be in the hall later today. So I'll be wearing this obnoxious shirt shirt. You'll know where to find me. Alright. Thank you all so much.