Stop vibing your agents to production: applying ML discipline to agent development

So we're gonna talk about vibing and machine learning engineering, which are possibly the two most extreme topics that we've seen today. But I guess we've to start at vibing first. So in our team, vibing was when me when I joined the team, I saw teams trying to build LM experiments by standing up a full front end, back end stack, just to prove a concept whether if LLMs are fit for purpose.

So we first we started, you know, messing around with the SDK, and that's not a bad place to start. I mean, a year ago, maybe two years ago, it was just chat completions. Today, there's responses. Anthropic messages, cloud providers have their own. Same goes for the eval libraries, tracing, all these stuff you have to figure out if you want to do it all yourself.

So some teams go one step higher, elevate, pick a framework, pick an abstraction. And maybe that's an abstraction or an SDK that your cloud provider provides, which is all great. So, know, you don't have to solve a lot of this stuff lower level. But at that point, maybe just think, and we almost did this. Let's just hand it over to our cloud provider, like, runtime, evals, telemetry. And they're also asking for our CICD.

Just let that go. And what that meant for our team there we go. Is we tried to do six to eight experiments. Some of them agentic. Some of them, you know, just simple hitting the LLM API. And all of that 60 to 70% effort wasted on that infrastructure led us to take roughly ten to twelve weeks, three months, just a proven experiment. Through four sheer sheer force of will, though, we were able to ship two of them to production.

But one thing that emerged that was sort of becoming a bit of an anti pattern was my team became the bottleneck. And we felt like we had to fix that at some point. So my background is platform engineering. I build platforms. I focus a lot on developer tooling and experience. And one of the main goals, or if the only goal of developer experience, is to make it easier for software engineers to make the right choice, write the right code, performant code code that you can ship in the easiest way possible.

So that's developer experience for me, which becomes way harder with distributed systems. So I've built, you know, tooling platforms to make distributed systems problems a bit easier to do, distributed telemetry across service boundaries, distributed transactions and workflows, cap theorem stuff, distributed databases. And we have to be a bit opinionated.

And and to those who've worked with platform teams, you'll see that, you know, they're so opinionated. This is how you do async. This how you do sync. But that's just the reality for large engineering organizations. And finally, internal developer platforms, which goal is to make it easier for dev teams to onboard themselves, their services, their teams.

So with all that, I thought, yeah, maybe I've got something good enough in my experience to maybe build a platform for building agents. But then tried to build it, and I think there's something missing. And there was one talk yesterday. I think it was Kareem Abdul who talked about your r and d and your app team should no longer be separate.

So your r and d team is usually your machine learning engineers, the people who train models, regression models, maybe neural networks. Then you've got your application team, which, you know, serve APIs, you know, serve actual product. And I think that makes a lot of sense. And I'm not a machine learning engineer by trade, by the way, but I mess around, you know, train my own convolutional neural networks just to make a classifier for Pokemon types.

And GPT two on my machine, fine tuned it through NanoGPT from Andre Garpati. Really recommend that one. But what I learned from doing all that stuff is everything is configuration. When you put together your model, you don't build a transformer block from scratch if you use PyTorch. You pick the transformer block, put a feed forward, pick the activation function. And you version all of that.

Right? Obviously, that's the sole version through code, through Git. And then when you run your training runs, another set of variables and parameters come in. So these are your learning parameters, learning rate, your Adam W optimizer beta one, beta two parameters, things that all you treat as configuration as well. And then finally, you treat everything as a hyperparameter.

So we don't manually tune all these parameters. I mean, you you think a a naive way is to do grid search. So you run multiple training runs at a time with different parameters, and you pick the best performing one. And then once you've, you know, you've settled, you define what's good enough, you deterministically ship that to production.

The most important bit is the loss function. So loss function is what allows you to know to determine what your training gradient is. How do you converge? Which direction does your machine learning model converge? Just a bit of trivia. If I can get this to work. Anyone know what the loss function is for a autoregressive transformer model?

Might buy you some beer. It's cross entropy. Other loss functions are mean squared error for, you know, standard regression type regression types. So with that playbook and, you know, with the platform engineering stuff under my belt, there's his how we built it.

We don't have a good name for it. I'm not we're not good at naming names in the government, as you may know. But at the bottom, obviously, there was our cloud provider, inference provider, hosting, data. We were stuck with Azure to start. We're lucky to have AWS inference just recently. I'm so happy we I get to access Quad finally.

We are a dot net shop, fortunately or unfortunately. And we also build, obviously, front end view in TypeScript. So what we've picked in sort of that layer is semantic kernel, which is a year ago. It's probably the only mature framework out there. Today, there's agent framework. The next one, I don't know. Maybe not sure yet.

Maybe we won't pick a framework. Then for front end agent communication, AG UI, which is basically just servers and events. And at the top layer, which is the layer our AI engineers deal with on a daily day to day basis, is where they write their agents, their experiments, the configuration, the evaluation, the test cases in YAML, driven through, you know, whatever the harness that we picked is, and then fed the output to the Microsoft at extensions.ai Evaluations library, which was quite opinionated.

So they came in. It was just LLMs as judges. So we had to extend it to have more determinism in. Implemented METR, GLUE, BLUE, which are translation metrics, which are quite good, especially with free text and if you've got a good ground dataset. Then once you're done and after you've defined what good enough is, we then use those same artifacts to ship that through source generation.

So to any dot net folks here, source generation is a cool thing. We're not using AI to generate the code. It's deterministic code generation. Ship that to an API that you can ship to production.

Cool.

Some trade offs. So obviously, we picked config. Agent as config, experiment as config. Geoff had a great point during the keynote that every AI engineer, for those who don't want to be one, needs to be able to write an agent loop themselves. And that's true. But in a large organization, you don't really need to rewrite that agent loop every single time.

Next is self hosted. Obviously, self hosting is very important to us. It's a systemic risk to be dependent on a cloud provider. And the last bit, which is well, not the last bit, but the other thing that I hinted on earlier was you might not actually need another agent framework. And and my point around here is, you know, framework versus harness, which one do you pick?

I would say harnesses today are probably some of the more battle tested agent loops you can use. So Cloud Agent SDK, OpenAI Agent SDK, OpenCode SDK for a more open source option. And the last bit was the evals. So with the evals, we defaulted to LLM to judge first.

But like I said, we extended it. We had to invest in more deterministic graders. And I use the term graders because I we our team got inspiration from Anthropic's great blog post, which is over there at the bottom, demystifying agent evals for AI agents. So elements as judges, expensive, costs money like your end to end integration tests. But, obviously, at the top, you still need that domain expert so they can help you write your ground truths for your deterministic tests.

So this is how it led to us. We were able to do experiments faster, focus more on the use case, focus more on the stuff that will allow us to take its production faster. The most most important bit is now we're no longer the bottleneck. Other teams are now starting to experiment. And really, that's the goal of platform engineering in any large organization, which is to share the wealth, share the expertise.

It was an open source thing, so anyone could contribute to it. Inner source, not open source. Now I'd love to talk more about our use cases, but I've they're gonna ship me off to a goal if I do. So I did the next best thing that I could with Claude, distill everything that I learned in an open source project called Holodeck.

It's Python. It's a CLI. So if you're gonna build agents using agents, what better tool can you use or can you give them rather than a CLI? It's all in YAML, you know, same kind of agent as config, eval as config, and allows you to ship stuff to a Docker container, which you can host anyway. So I'll be doing a demo.

I'll just take a rest over there. This seemed like a very good idea, so I could take a break. So just hang on, and hopefully, you can hear it properly.

For today's demo, we will show financial assistant built on Holodo. We used the ConFinD QA dataset for its grounding data and most importantly, its validation set. It's a research dataset that aims to study and evaluate complex numerical reasoning. The study and the dataset is free for anyone to use.

And here's the raw dataset, and here's the documents used in the conversations. Holodeck agents are configured in YAML. We have the model configuration, system instructions, evaluation settings, and tools. You can see it has three tools, two math tools, and a separate search tool. The test cases fed in the eval run is in a separate YAML file, which you'll see here.

This is the original config QA dataset from earlier. Let's run evaluation. The eval set has hundreds of test cases and each having multiple turns. So we'll limit it to 10 for now. It is now spinning up the agent, feeding the test cases, then running the configured eval metrics when it finishes.

The eval harness captures everything that the agent emits. Tool calls, both arguments and results, its reasoning output, and the final responses. Now that the eval run has finished, let's look at the results. The run we did shows a 50% pass rate for the 10 test cases we did. Drilling down to that run shows the agent configuration snapshot, the detailed transcript return including tool call arguments, and the results.

Each turn is subject to the same eval metrics configured, but it can be overwritten. We can see the turn zero has passed while turn one has failed. You can also assert which tools must be called per turn. And at the bottom, there are the aggregates. Let's look at a test case that passed. Same thing, detailed transcripts, metric result aggregation.

Hollerbeck ships a dashboard where you can view historical agent runs. Let's say we're done with evals and want to test our agent with an app. We'll use Holodeck serve to expose an API that implements AG UI. For the front end, we're using a TypeScript library called CopilotKit, which implements the AGUI protocol.

Let's warm up the agent. Then let's ask it its tools. And then let's take one of the test cases and feed it into the agent. AG UI is a native agentic protocol which uses server sent events that standardizes on the events passed between UI and agents. Agents. This includes conversation start and tool call use, results and arguments, and then finally the final response.

The UI side captures these events and renders them accordingly. As the agent was running during test and serve, we collect telemetry. Here we can see the app logs, but most importantly, the traces and spans which are tagged with GenAI specific metadata. Now let's say we're happy with how the app works.

We'll now use the Holodeck deploy commands to pack the agent in a container. That includes the Holodeck runtime plus the agent configuration. Now let's push the generated image to our container registry. Now that the image is pushed, let's use the holodeck deploy run command to push it up to Azure Container Apps.

In the agent YAML file, there are settings on how to configure your agent to be deployed. This is purely optional though, and you may choose to use your own deployment mechanisms. Let's have a look at Azure and the deployed agent. Azure Container Apps is just one of the supported native deployment targets.

We plan to support AWS AppRunner and GCP Cloud Run. This is purely optional, and you can choose your own container runtime infrastructure to run your agent with your own tools. We've reconfigured the Copilot kit app to point to the agent endpoint in Azure. So let's give it a go by feeding in another test case.

Just taking a peek in the log so we know it's live. And the final response, thanks for watching this demo and give Holodeck a try. Thank you.

That one better than expected. I was able to take a break, so I'll do that next time again. It was a good tip from John. So there's one more thing. We did say we wanted to use machine learning engineering principles. And it's not machine learning if there's no stochastic grad student descent. Sorry. Stochastic gradient descent.

You got the joke. Awesome. So for basically, we want to be able to have a process to automatically tune agents. There's two things that normally we tune. You've got your hyper parameters. So think about your temperature. What else? If you've got search tools, vector search tools, you've got top k. You've got minimum similar similarity scores, things you can tune numerically. There's all obviously, the text prompt as well.

So for the numbers, Bayesian optimization for hyperparameters. For text prompts, TextGrad style critic TextGrad style critic prompt optimization agents. Still building that, testing that out as well in in the organization, but I'll let you know how it goes.

Takeaways. Own your life cycle, at least for developing agents. And I think the analogy here is you wouldn't outsource your application logic. You wouldn't outsource your unit test framework. So why should you do that for agents if you're going to do use them for anything material in your business?

Treat evals as the loss function. You won't be able to do machine learning style optimization without a loss function. So treat the evals as loss. Start with deterministic graders, which is a bit of a theme that I've picked up across the two days. And finally, if you got an organization that's large enough, maybe this is what you ask your platform engineering team to build.

Maybe. So that's all I have. I've got a QR code there for the GitHub repo, my LinkedIn, and all the others and the sample app sample experiment as well. Have a look. Give it a try or reach out to me. Have a great day, and have a good rest of your afternoon.