Building Frameworks Building Systems

Cool. So There's a content warning. There we've got YAML, and we've got YACS in here. So if you're afraid of those, I can't recommend. Style last year was recently acquired by Curriculum Associates, an ed tech company from The States.

It came with a whole bunch of changes, some more subtle than others, more projects, different customers, scaling out our features, so on. One of those changes is expanding our reach from primarily science based classroom audience to also maths. That's meant we've needed to expand our collection of interactive games in our lesson lessons for students and teachers in classrooms.

There's a pretty nifty catch, though. We need to scale out the production of Sims or interactive games from just four a year, a few years ago, to 20 last year, to a promised 200 for a potentially few million students. Oh, but we can't expand our engineering team to do it, and we need to migrate our existing backlog and to get on top of all the accessibility issues with them and then serve all the new interactives.

Enter me. I'm pretty self centered. I love sticking my fingers in with every pie under the sun. This talk is basically entirely about the one time I was told to hold my fingers back for a minute. Those same pies just came flying back at my face. This is some question content in one of our lessons.

You've got a couple questions. They're pretty dry on their own. By contrast, these are some interactive games. This is one science one, and this is one maths one. They're obviously interactive, but I'm not risking any live demos. You'll just have to take my word for it. The best way to think about an interactive in the context of a science or maths lesson is, for the most part, is that it serves a supplementary purpose of the core content itself, its practice or application. And that's that's sort of changing.

Building interactives from the ground up isn't a trivial ask right now, so we really needed a reframe of the problem getting building 200 a year. Really, we don't care about the code itself, we care about being able to maintain them with as little effort as possible. We have a lot of machinery for that in our mono repo already. We just we also have to care about the aesthetics and the accessibility. They have to be representative of our brand, and they have to be inclusive.

Therefore, we have a series of constraints we can improve over time, and the artifact is the code and thus the interactive itself. We essentially encode the code as a form of content, form of data. So Lambdas at Home is a project that I've been working on in my spare time.

It runs on a really simple premise. I want a schema that lets me define a function that can be run wherever I want it, in the CLI, on a server, wherever. Don't care how it runs, when it runs, who runs it. I just need it to be secure. You'll notice that in this YAML, the bulk of the schema isn't actually the code. It's everything else around it.

It's the metadata, the executor, the arguments. It's everything that defines a contract. You can even add tests to it. There's some basic tests here. And once you have that, you actually just realize there's little need to write or read the code yourself here. You sort of just care about the inputs and the outputs, verification, and then you just replace the implementation with anything else.

Personally, on my home server with way too much RAM before RAM's exploded, I run GBT o OSS and a flavor of Kimi k 2.5. So it runs pretty fast and it runs pretty reliably. But given the right scaffolding, security, automatic verification, you get the thing out. It's you're essentially defining invariants for a very specific task in a very specific context.

So we took that concept and applied it to our interactives. In particular, we need to run the interactive. We'll find a way to do that. We need to dockerize it. We need to automatically smoke test it and deploy it and then test it and improve it over time. And this pipeline ought to be able to run essentially any blob of arbitrary JavaScript in HTML.

You can sort of think about this or each step. Analogously is like a a boundary value problem in however many dimensions. As long as you're not asking too much of an LLM by way of constraints and your input isn't too extreme, what you get out on the whole won't be either. Coming back to this diagram though, you might think it's a bit odd that we test it after we deploy it.

Well, not in stand up mode. So start up mode. That's actually because this this this flow is split into two parts. The bootstrapping part and the improvement part. In the bootstrap side of things, we do run discovery.

How do we run the simulation? Punch it into an LLM, some kind of instructions. We verify that runs, opens, returns a port and host, and that there are no errors on launch. Then we gotta migrate it. We gotta make it standalone. If it has references to the SIM outs, everything outside itself, we can't do anything with it.

We gotta copy it in so we can migrate it. And in this regard, we we we do extensive testing to make sure that is standalone. Then we can go set set up standard CI and deployment infra. Upload all the assets because we're moving stuff to the CDN, and then we test out whether it can be deployed.

The entire point of this pipeline is to make as few changes as possible to the target in order to preserve its integrity with the exception of the asset uploading. So this is what it kinda looks like, and this is what the loop looks like. It's probably pretty familiar, although it is Rust. Sorry. The prompt, you got your checks, whatever, reprompting, loop, that kind of thing. The second part of this process is the improvement pipeline, which I'll skip through because it's worth another talk.

We're gonna explore the interactive. We're gonna build a plausible state machine representation as a student. I mean, it's basically evaluating all the available interactive features, interact with the most obvious feature that the interactive is insisting we should, and repeat until some stopping condition. Call the final state, repeat by backtracking, and then repeat the entire process some number of times.

What we then get is a whole bunch of states that we can write tests automatically based on. Use a cheap model. You can use a good model. Doesn't matter. You have a definitive state. Then once you do that, you can run write modifications according to some other critic frameworks that we've got internally.

And what you get out is a fully accessible interactive game with minimal inter intervention. Screen reader support, keyboard support, all that kind of stuff. I didn't have that before. There's just like one really, really big glaring problem with all of this. It's not the one that you'd expect out of a theoretically token burning, hands off self driven pipeline does genuinely perform pretty well and fairly efficiently. It's just like everything else. See, that's the problem.

Everyone else is the problem. I'm perfect. Not really. So problem number one or finger in pie number one. How do I minimize the drift of all the boilerplate that's driving the testing of of the interactives? Some background, our build system is coordinated using a piece of homegrown software called Armchair General.

It's a get back solution that solves the question of, given something that I've got that I need to build, what other things and what files do I need to build it? These things are called products, which can represent a Docker image, a dependency, or test shaped stuff. Again, more YAML, not sorry.

Just got some metadata and some dependencies and some files. The idea is that you chain them and all the file system dependencies together for a fully deterministic build graph depends only on the state of the file system. So the solution here, because this isn't enough, is we build a new concept into it that creates them on the fly, normal YAML, except there is, using safely vibe coded scripts.

Hurrah. This is what that looks like. It's a pretty simple one by comparison. You've got an action script, include some files in order to know how the script runs. The most time consuming thing about implementing this wasn't actually implementing it.

It was talking to everyone who had a better idea and better opinions than I did on the system. And we have some really good SMEs. So that's one bottleneck down. But problem number two or a yak and a shave is my PRs aren't building. They're hanging around awaiting build cat agents, spot instances forever. And, honestly, the solution to this is kind of the engineer's dream. We rewrote the autoscaler in a day and named it Rasputin because I discovered free will.

So that's what that looks like. Previously limited to about 800 at a time. If that sounds mildly dissatisfying, though, that's okay. Naturally, we found other bay bugs in the agents, so we wrote a zombie killer to compensate. So when we had identified what engineers had been thinking about this problem for a long time than I longer than I had and when we decided we'd had it about up to here waiting for builds, And we identified all the problems with the existing autoscaler.

We had targeted fixes that we just wanted to make, so we made them. Clawtrust gave us the ability to do that, like I said, in a day. And what's even better, having written our own auto scaler now is we get all access to logs, tracing, metrics, observability by default because we had the good foundations for them. So we can understand why it's broken.

And we can do more parallelism, which everyone loves. So problem number three, well, that one pie writing yak is, well, I've done. Cool. My PR is building cleanly.

That's great. But my PR is taking hours to merge. Days. And so is everyone else's now that I've locked 50 interactive code PRs over the fence for no one to review, by the way, but it's now in the queue. Surely, it's a transitory problem. Right? Nope. Engineers were also waiting hours and days consistently for anything to merge. So I sketched out an idea, ran it past a few others who had, again, more experience in the area, and it was a resounding yes.

So for some background, we home brewed our own merge we home brewed our own merge queue. Batchbrew. It's a genuinely brilliant piece of software, but it hasn't really been scaling to our increasingly multifaceted, monorepo esque needs for a variety of reasons. So I I vibrated a simulator for it, one which integrated Git, mocked BuildKite, and GitHub, and allowed us to conduct some Monte Carlo simulations on new strategies to evaluate their effectiveness.

Importantly, I didn't wanna make changes to our merge queue in production. I I didn't wanna piss off the other engineers when I inevitably broke it. So developing a good testing ground was incredibly important. And then using the power of armchair general, as I explained before, I parallelized batch brewer. Technical details will be somewhere else, I'm sure, by group grouping mutually exclusive overlapping PRs and being smart about when we throw away work.

And you can see the results here. That's my maniacal laughter. Ignore. We got to the end of our dependency backlog. Some happy people. We got to the end of our complete like, got to the back of our backlog. Stuff merged in ten minutes, and my interactive PRs merged.

And so that's one more bottleneck down. For anyone who likes graphs, this is our the red line, the red smooth line is our ninetieth percentile time to merge in minutes. That's sixteen hours for ninetieth percentile PRs to merge. We got that down to less than three.

So you should start seeing a pattern here. We've used AI for bolstering our ability to make changes responsibly, safely, and consistently using the patterns that I've shown that lift up the entire engineering team, not just my interactive stuff. And then they let us ship to soon to be millions of of students. The bottlenecks are moving back and back.

And, ultimately, what we unlocked through these changes is many, many interactive games using an artifact shipping framework. We've migrated and deployed 250 interactives, a number of which are maths and some of which are our existing backlog. Oh, sorry. Most of them are our existing backlog. It wouldn't been wouldn't have been possible to do in the same time frame and as few engineers, me.

Otherwise, without this without the systems and frameworks that we already had and improved on and built. They effectively built themselves here. But but but but but it's far too easy to sell ourselves short here. We must not seed good engineering practices to locking out AI generated code left, right, and center unless you intend for no one to review them.

And now more than ever, there's increasing incentive to use it to serve our own colleagues, ultimately consumers in ways that not only benefit them now, but pave their pathways to do more good too if we let them. If any of these problems looked interesting to you, there are so, so many more where they came from and a lot more to them that I've not jumped into here.

And we're hiring. A handful of us from Style are here, and I promise we don't bite much. Thank you.