Fully Automated Luxury Gay Space Engineering

There we go. Okay. So, you heard from Ali yesterday and Ben earlier today, both from Style, about how we're thinking about, you know, automating specific parts of our workflows. I wanna zoom out a bit, and I wanna really start thinking about how this technology potentially changes not just the way we individually do our jobs, but the whole way that our businesses work and how we build software.

So what is cool about coding agents? We've all been really excited about them. We're all using them. We've all got our favorites, Claude, Codex, whatever floats your boat. Well, it's kinda cool that they think a bit like a human does. So you can give them a fuzzy task, poorly specified, and it kinda works out what you mean.

It goes and reads the docs, looks at the code, pulls up the context. It can help you do your job, which is great. But put your, like, business hat on. They cost some money. They make engineers a bit faster. But, really, like, if we wanted to do a lot more software engineering, we had the ability to do that before.

We'd just hire more engineers, hire more managers. Google's got what? Like, 50,000 engineers or something? Like, this is a solvable problem. If you want to ship more software, you can just do it. And where we are now is that you are the limiting factor on how fast your coding agent can work. The context that you can keep in your head, how quickly you can switch between tasks, how well you can communicate with everyone else in your organization, that is rapidly becoming the limiting factor.

What's really cool about coding agents is that they think a bit like humans, and they scale the way computers scale. So what you're seeing up here is a list of BuildKite jobs, each of which is running on a AWS instance in the cloud, running an instance of Claude code wrapped in our, agent harness, which is just some Rust wrappers.

And these are all running right now even though half our engineering team is sitting in the audience with their laptops closed. These are all contributing to our code base. If you take one thing away from my talk today, it's remove the laptop. Stop thinking about coding agents as something that help you do your job, and start thinking about them as something that helps your business produce value.

Once you're running your coding agents in the cloud, you can set them up to run automatically on triggers. None of this is new. This is all stuff that a lot of people are playing with today. Automatically review PRs. We've got a whole suite of specialized critic agents, basically personas with extra wrapper code that are looking for certain things on every change that goes into our code base.

Security is a really obvious one. And what's great is these are more diligent, and more careful about checking the code than any human reviewer typically is. We've actually seen fantastic results with documentation. We've got agents that are checking that your code follows the documented best practices, but even better, that the documentation is up to date with the change to the code that you just made.

And that means that every time you write a doc, it automatically becomes a linting rule. And people are actually writing documentation and keeping it up to date now because Claude reads the docs even if your colleagues don't. When you've got coding agents running in the cloud, every anytime you push up a change, if you, broke some test over here, failed to handle some edge case, Claude can just fix that for you.

And it can do it without you having to context switch back to the work that you were doing. You write up a feature, you push it, you go for lunch, you come back, and now it's passing tests. Maybe it wasn't passing tests when you pushed it, but it is now. The vast majority of the time, your intent is clear from the code you wrote, ideally, the commit messages and the comments that you put into that code, and the kinds of things that need to be fixed in order to make it pass all the tests, pass all the code review, pass all the automated checks, they're kinda trivial, and they shouldn't need your time as a human engineer to go and fix them.

It can automatically resolve tickets. A lot of the tickets we all deal with are small. Support problems, minor bugs, styling issues, parsing issues, random errors in production. These are all well within the scope of Claude to fix, and you can just assign Claude to go and fix tickets as they come in. What we've built here is essentially an agent orchestrator.

This is our version of Gastown or pick your favorite hot new orchestration framework today. But, again, it doesn't run on a laptop. It doesn't run on a single server. It's built across our entire CI system, our ticketing system. It uses all of the existing tools that our engineers are used to using and that everyone out of in our company is used to using. And that means that it's got all the user interfaces, all the integrations, all the metrics, everything that we're used to interacting with that makes it really accessible and really observable.

And this is all working well. This is a graph showing the number of pull requests that we're shipping over the last twelve months. Our headcount's been approximately flat during that time. Humans, you'll see, are now shipping about twice as much code. Our AI agent is now shipping more code than the humans ship.

So that's a three to four increase three to four x increase in throughput over the last twelve months. But that's not the coolest part here. What if we aren't just automatically fixing issues, but automatically creating issues? We were actually doing this before coding agents were a thing because it's kind of easy to do.

Just scroll through your logs, find errors, find unexpected service dates, find things that are too slow, create a ticket for them. Put all the context from the logs of the request, what the user was doing into that ticket. That's great for a human engineer. When they pick it up, they can see exactly what was going wrong. It's also great for Claude.

And with that sort of context, Claude is solving about 50%. That was a couple of months ago. It's probably higher now of the issues that we encounter in production automatically. Most of the time now, a human picks up an a bug ticket that gets assigned to them. They look at the code that Claude's written, and they say, yep.

Looks good to me. They don't have to context switch into the code base. They don't have to go read the history. They don't have to understand what the author was thinking when they wrote this code. Claude's already done that work for them. And so this is how work is increasingly getting done at style. Our teachers and students use our platform.

Things go wrong. They always do. We collect those errors, aggregate them, gather the right context, turn them into tickets. Claude writes code to fix the problems. Humans review that code, and then it gets pushed to production. This is a closed feedback loop. What that means is that to make this system more reliable, you need to stress it more.

You need to push more information through this loop faster and faster. We need more users. We need load testing. We need fuzz testing. We need penetration testing. The platform will auto heal itself as you hit it harder. We've had two instances now where human security researchers, pen testers that we paid, found a way to make our service generate a 500 error. And before they could find a way to do anything useful with that unexpected service date, it had been patched.

And it really annoyed them, and it made me really happy. But humans are still in this loop. The slowest part of this process by far, the thing that slows down our bugs getting patched in production is the fact that humans are reviewing code and humans are checking that the problem is actually fixed in production. We are working very hard to try and solve that problem. We're not there yet, But we're thinking very carefully about what kinds of problems are humans still catching in code review that could be caught by the machine.

How can we automatically check that the original intent of a ticket is now achieved in production? Ideally, if your ticket's well specified, that's quite easy. And once you remove humans from this picture, this is where we end up. So what do we all do now?

Like, what's the job? I'm sorry to say it, but you're all managers now. Basically, everyone has their boss's job. If you're an engineer, you are managing a team of coding agents. You're managing a system for code being written. You are no longer writing code. If you're a manager, if you're a team leader, if you're a tech lead, you're really doing the job of an executive.

Your job is to be looking at what's coming down the track, seeing how to adapt your company's approach. Your job is to work out where the bottleneck is in the system. There's always a bottleneck. And as the technology advances, as we change how we build things, that bottleneck moves. I think we've all had the experience over the last two months of the bottleneck going from writing code to reviewing code. Soon, the bottleneck is going to be coming up with good ideas, designing code, designing new features.

You need to own the quality of the outcomes even though you are not doing the work yourself. That's a really hard transition to make. It's something that managers have to go through and often really struggle with. I'm responsible for my team's output, but I didn't literally do it. How am I supposed to be responsible? Joe did it. Well, too bad.

You're responsible. Work it out. Your job is to work out when the system is failing. Notice when it's failing. Fix it. And critically, this is really easy to miss. Collect the data about what went wrong. If you're collecting the data, then you can do trend analysis, classic, you know, SAT reliability engineering work, work out why the system broke, and go back and fix it.

But when you think about it, all of these are actually pretty rote processes too. We're starting to think about how you can build feedback loops on feedback loops. I've got engineers now looking through all of the comments that humans have left on PRs in the last month and saying, what are the themes? How can we build a loop that takes all of the things that humans had to identify that bots didn't and turn those into new kinds of specialized critics or updated rules for our existing critics so that they get caught automatically next time?

How can we do incident analysis automatically? Look at the ways that our system failed in production. Look at the way that our coding systems are failing in production and turning those into tickets that Claude can solve. It's all about feedback loops. This is your job now. You have to think about the whole business, where the information lives, how the users interact with it, where the value flows, how that information is connected back, and how you can make those feedback loops shorter and tighter.

Where does this all end? Why don't we just make Claude CEO and go out for drinks? I think the honest answer is the models are not ready for that yet. There's two big limits I see. One, context. Everyone knows about this. This has been half the talks over the last couple of days. How do you get the right context in place across the whole organization? That's basically impossible.

There will probably be a technical breakthrough there in the next few years, maybe one year, maybe five years, maybe ten years, maybe twenty. It's easy to argue about that, but, like, it doesn't matter that much. It's gonna be within the working lifetime of pretty much everyone in this room. So you need to be thinking about how you're gonna operate in that sort of world.

The other limitation right now is executive function. Their judgment just isn't that good. It's actually pretty good about how to write code, how to make a technical decision, how to handle an error. It's not that good about, like, should we go into this market? Should we adopt this new product line? Do our customers want feature x or feature y?

It's not that good yet. Maybe it'll get there. And good engineering is actually more essential than ever. Coding, that's done. That's not a job today. But what is a job is systems analysis, identifying feedback loops, tightening them, finding new kinds of ways that we can automate work in the business. And I think what's really underrated is looking ahead and seeing what's going to be possible because the reality is that the rate of change out there in the world is increasing really rapidly.

We're all at a conference talking about how to use tools effectively that weren't good enough for most of us to use, like, six months ago. Right? That pace of change is new, and it's only accelerate accelerating. It's not good enough now as an engineer to be saying, oh, what are the established tools and practices? How can I take them?

How can I apply them in my business? We're at this really, really exciting time, like one of those times you read about in the textbooks where people were discovering what electricity is and that you could use it to make light. We're there with computers now. Everyone here is in that position where you can go and work out new ways to apply this technology to do things that no one's ever thought of before.

The ideas are lying there on the ground. And I think one of the most important insights here is pick low hanging fruit. A lot of things are easy now. It's not worth committing to a twelve month mega project when that project is going to take half the time if you wait twelve months before starting. Do the things that are easy now that were hard last year and take the things that were easy last year and automate them and never think about them again.

Your job as an engineer is to step back and design the machine that builds the code. Your job is no longer to be a cog in that machine. And I think there's a real obligation on all of us to think about how this applies to our colleagues, to our friends and family who aren't engineers. It's coming for us first.

We will not be the only ones. It's clearly coming for designers. If you draw things for a living, if you lay things out, AI is going to do that job in the same way that it does for writing code. If you're a writer, like, it's not a great writer yet, but most writing in the world is not that great.

Most writing that people get paid for is actually not that great. We you know, we're building a literacy product now. Previously, our first drafts of chapters for this literacy textbook were done by a third party vendor. They cost around half $1,000,000 and took five weeks. We took our best pedagogical experts in the company, sat down with a couple of engineers, built a system for Claude to do that drafting. It takes ten minutes, costs a dollar 25, and the result is better.

Right? It's clearly coming for every area of the economy. The robotics is a bit behind, but how much behind? Five years, ten years, not that far. And what really separates people who are having a great time now from people who are struggling and worried is an abundance mindset. If you're worried that the old way of working is gone, that things are changing, you don't know what's going on, it's gonna be a difficult time, and I'm sorry.

But if you see what's possible now, that you can do 10 times as much as you could previously, that you can tackle projects that you previously didn't even consider starting because they were just too hard, you can now do them in a weekend. That's exciting. If that, like, makes you wanna get out of bed in the morning and tackle 10 different new projects that you don't even know how to start, come work for us.

We are looking for the best engineers in the world to come work for us here in Melbourne and build science lessons, literacy lessons, and maths lessons that are helping the next generation of critical thinkers. Thanks.

How long oh, yeah, we have got a bit of time for questions. I've actually got a question, if you don't mind. So we're trying to sort of adopt a similar type of thing. And so the wider team can ask questions in Slack that will then get turned into tickets, and it will go and do the work. Seems to be fine if it's, oh, can you do this copy change or, like, for certain things.

But some one example we have was, oh, we want to add a search bar

But

when we kind of looked at it from a product perspective, it's, well, a search bar the solution here. If it was given to Claude, Claude would run off. Would do a great job of doing the search bar and fixing it. But actually, it was using a card pattern where there was hundreds of rows. Probably should have been a table in the first place with filters and whatever else.

How do you or I mean, do you have, like, similar type of things that are coming through?

Do Like, one of the scarcest things now, because execution is cheap, building the right thing is hard. So, you know, we're trying to spend more time in schools, more time talking to teachers, more time stepping back and thinking about what our product ought to be. Like, forget how hard it is. Forget where we are now. What is the way this product should work if we had all the time in the world to make it work that way, and then run-in that direction.

If you're too constrained by where you are now and what would be an easy small increment over that, it's very easy to get trapped in local optima. And I think a huge threat for all existing companies, especially bigger, slow moving ones, is, there are a whole wave of AI startups that are greenfield. They've got no tech debt, no legacy.

They can build really quickly. And in fact, your best engineer, your best product person, your best salesperson can go and recreate what you've got very quickly. So I think one of the most valuable things today is spending a lot of time thinking about how would I want this product to work for me. Go and find your best customers and hire them and get them to tell you. Our staff, like, between a half and a third teachers, which has always been a huge asset for us.

The more the more people you can find who can help you do that work, the better.

Totally agree. We're we're similar. We're a small start up doing legal tech. Yep. Obviously, you mentioned earlier, three engineers, three lawyers. Perfect. There are sales staff because we figured it's easier to teach a lawyer to sell than it is a salesperson law, or at least that's the logic.

Absolutely.

Anyone else have a question for Daniel? Yep. In the front.

Hi. Nice talk. I'm also on a mission to do something similar to our team. And I wonder what what specific task or what specific issues are do you find that agents still have trouble auto fixing, like, where humans still need a lot of review or need more review?

I mean, the biggest thing is where there's either product or technical architectural questions, as I was just saying, like, where what you actually want isn't clearly defined. Another obvious one is when the information isn't written down anywhere. There's no context. We've always had a mono repo, and now we're just trying to put as much into that as possible because it makes it really easy for Claude to go and suck all the information out of context.

So, really, places where information only lives in people's heads, where it lives somewhere else entirely separately, where the task isn't well specified and we didn't really know what we wanted, those are the most difficult cases. So doing that work to break problems down, to clarify them, to make sure the information's accessible, that's one of the sort of organizational engineering jobs we're spending a lot of time on these days.

Amazing. Great answer. Okay. Yeah. So one more one more quick question. Yep.

I think I was kind of interested in, like, how, like, heavily still you're relying on, like, ticketing as well. One of the things I I asked because, like, one of the things I've been doing with my organization as well is, like, working directly with, like, sometimes the the CTO and, you know, generating, like, products on the fly Mhmm.

You know, generating, like, your full interfaces and then, you know, shipping that, like, one, you know, one big feature. Yep. And I think that one of something I didn't used to do. But but I think one of the thing I was wondering, do you still find yourself putting, like, a lot of a lot of things into little little tickets as to go through your process?

Yeah. Mean, huge new features. Like, they really require someone to sit down and and build something interactively just because it it's an exploration process to work out what you want. We're actually finding our UX designers end up doing a lot of that work. Like, our UX designer built the first version of our most recent, like, whole new page feature, write all the code.

We've done some minor tune ups to it after the fact, but we're really trying to shorten the loop there by shortening the handoffs between people and teams in the organization. But a lot of what a lot of the work that ends up happening is like, oh, something went wrong with this system. Let's change it this way. We want this extra feature over here.

Let's change it this way. And that's a really good solution for a ticketing system. But for big new features, we don't bother usually.