Optimising GenAI at Runtime with Experimentation and Guardrails

Excited to chat with the this audience, which we will fill in later using AI. Alright. So, yeah, as mentioned today, we're gonna explore how teams are optimizing AI systems in production and not just deploying them, but continuously improving them and and using experimentation and guardrails. And I both of those things are very key to what we're gonna talk about. The use of AI is accelerating software development, and it's at a really incredible pace.

But it introduces a whole bunch of problems. And one of those well, not one of those. The chief among them is how do you control AI behavior, once it's in production? So, you know, LaunchDarkly has been around a pretty long time. We have over 6,000 customers across the world over every vertical. We do 50,000,000,000,000 flag evaluations per day.

Those are not AI evaluations. Those are flag evaluations in our feature flag delivery network, but they're very similar in lot of ways, which we'll

get into a little bit.

And, 84% of our customers said that they release code faster and more frequently and reduce bottlenecks by using LaunchDarkly. So this is this is great. This is exactly what we've been doing for over a decade. It also means that we have a large pool of folks to go and talk to about AI development. And what we found was surprising.

Right? AI is it is changing how we build software, but what we found was that 94% of the folks that we talked to said that AI increased velocity, but it did not increase the confidence in this code that they shipped. Right? 81% of them shipped code with unresolved risks in the code base, And 91% of them, automate code creation but not control.

In other words, there there's delivery pressure that forces compromise there. Should we be surprised by this? Was this what was supposed to happen? I think that we all kinda thought that as AI grew and these kinds of features grew around them, that this would get really easy, but it seems like it actually hasn't gotten easier at all.

In fact, maybe it's gotten harder. Quick aside here. Rachel, Enterkin. I'm gonna I'm butchering her name. She recently ran the Kokodana two fifty. It's an ultramarathon, and she had the fastest time ever of men or women. She beat it by miles and miles. So it's a huge accomplishment.

She's great. Super cool lady. And she had this really interesting quote I just heard a couple days ago. When something has never been done before, don't expect it to be easy. I think this is where we're at with with AI right now. Right? We're kind of doing things that have never been done before, and we're also expecting it to be easy, but that doesn't really make any sense. It's actually just gonna be harder and harder until we finish our ultramarathon, and we're at the very beginning.

Alright. So there's a major runtime control gap, right, which is that AI does not behave like traditional software development. Right? We've solved for speed, not for control. Traditional systems behave predictably. AI systems don't. They are, by their nature, unpredictable. And so once they're in production, controlling that behavior becomes super difficult.

And we're seeing this amongst our customers. And the interesting part is we've built tools around controlling behavior of your code for, like I was saying, over a decade, and we really want our customers to apply that to the way that they use AI as well. So what are the ways that it's not, you know, the same as traditional software? There's a ton.

But I think these are kind of the four major categories that they fall into. One is that the outputs are variable. Right? So we have these nondeterministic systems, means that the output that you get is just not always gonna be the same. You know, when you do evals offline, this is interesting and novel. When you do it in production, it is scary and costs you money. The other is that validation is continuous.

The thing that you validated before you put it in production does not mean that it will continually validate the same way. Right? So you can't prove everything up front. Once you put things into production, things actually change over time. Also, quality is contextual. So what is good for some folks is not good for everybody, And that is something that depends on the user, depends on the intent, and it's not a single universal expected result.

And then finally, the change is continuous. So everything about AI is this way, but I think when we think about it in the way that we, are releasing models, data, and you're doing retrieval in production, this is something that we are putting out there into the world that will just continue to change over time naturally, and it's kind of in an organic way.

So we need to learn from real usage, but not only that, you need to have the ability to intervene in real time if something goes wrong. So this is our proposal, that AI generates at scale, but testing validates at scale. In other words, this is where ongoing monitoring and optimization for application becomes really crucial.

We're talking about taking learnings from your real users at runtime and then validating your assumptions for your AI based on applications with, like, a rigorous and systematic approach. And this doesn't just help you make decisions faster. There's also an aspect of this is that it lets you make changes faster, and in fact, almost in real time when things go wrong with minimal, you know, problems.

Okay. So what needs to be tested? I mean, kind of everything needs to be tested. There's a curse of abundance here, which is that we have, more than we've ever wanted. We can generate more code than we've ever wanted. We can get more interactions and evaluations we ever wanted. But that means that there's a ton that needs to be tested.

So we kind of break this down into three categories here. And when you're using AI to accelerate code development, you may wanna take additional steps to safeguard your customers from being exposed to unexpected errors. So what does that look like? One, companies that are ahead of, competitors shipping AI product features with experiment with an experimentation lens, are gonna continue to do this.

Right? And then every single product feature could be turned into an experiment at that point. And then three, the challenge there becomes to determine which ones of these should be an experiment, and more often than not, most of them should be. So many of you out here may already be using feature flags. You may be using LaunchDarkly.

You may be using something else. You may be using Home Run System, even. And, the feature flags allow you to toggle things on and off. And when you start to consider who should see a feature, when people should see a feature, this is where you begin to use things like feature flags to do that. But this is also the same kind of mechanism that you want for AI in production.

And that's why we built tools around this that leverage the feature flag delivery network that we've had for over a decade. So each of these decisions will also contribute to understanding which product feature you'd like to experiment on, the same way that you consider, how will I create feature flags for the features that I'm creating? So from runtime to deployment, we have this optimization pattern, right, which is that we have these new ideas, and, we build the feature.

We put it behind a flag. We measure feature success with an experiment. And when I say experiment, I really mean this in the scientific sense. We're gonna measure two things against each other and look for a statistical significance or a high probability that this is successful. Then we move on to conduct advanced analysis, you know, downstream metrics, downstream effect. And then we ship the best version of that feature, and we start the process over because this doesn't really ever end.

Right? That's what gets us all jobs anyway.

Right? So this is good.

We apply this to the idea of AI, much of this is the same. The components of it change. Here you can see how it breaks down within LaunchDarkly's ecosystem. The LaunchDarkly SDK lives inside your application and delivers a lot, an AI agent configuration. Right?

This is the model, the prompt, the parameters, the tools, all the things attached to that. Your application then takes that config and makes the model call directly to your LM or gateway. And, you know, LD is never really in the data path there. We're an abstraction above it, but with the ability to make changes in real time. And then LaunchDarkly has no visibility into the actual inference or request that way.

This is safer for you and safer for us. When you get down to actually putting things into production, though, what you'll find is there are many components here which can be experimented on. And this is why we built agent control around those parameters.

Right? So you have models, you have model parameters themselves, and you have prompts. And this is the basis for the way that you do experimentation in the agent control world. So we've really separated our product into two different pathways. One is code control. This is classically what people have done in the past. And the other is agent control.

And this focuses on these three types of experiments and combinations of them. So when shipping with, AI, you're adding this layer of complexity. It's not just what was the intent of the feature or, you know, what is the outcome by the users, but also these additional parameters which can drive cost, efficiency, and, even things like vendor lock in. So any of these elements could become candidates for an experiment.

Even when things are operating as they should, we may want to reduce token consumption, or we may want to, as a company, for legal reasons, move to another model. The challenge is then determining which configuration works best, but we need to do this not just offline in evaluations, but with real users in real production environments.

So I've been kind of saying this over and over, but that is the real, challenge here, which is how do we change things in real time? How do we have runtime control? This is something that a lot of the folks here who are building different tools around agents have never had to tackle before.

If you think of these things in a lab environment, in other words, if you think about how do I compare models against each other offline, then this is relatively easy. But if you think about things like, how do I limit the amount of risk I'm exposing myself to when I put this into production, but I also need to learn from it in real time.

Then you need a much bigger infrastructure that's much more mature to enable you to do this. That's what we actually offer, and we built it on top of what we were already doing. And I'll tell you a funny part about this is that we did it because our users wanted us to and because they were trying to do it themselves.

So nearly a year ago now, we looked at flags that our users were creating and found that they were putting the prompts that they were using for their LMs into those feature flags. And that's a clear sign that they wanted something from us that we were not yet offering, and we built this around that. But we used the existing architecture that we had because it already allowed us to make changes in real time to things that are already in production globally using our flag delivery network, which became the backbone for what we're already doing now with agent control.

So, you know, which model performed best? Which prompt drives better outcomes? These are all questions that you can now answer in production while minimizing the risk to your users. And, I keep mentioning that we're minimizing the risk to the users, and I say that, I really mean we're minimizing the risk to ourselves. And we do this by putting up guardrails.

So guardrails are not just the indicators that you get from the models themselves, but the indicators that show you the health of your system. So progressive rollouts reduce the blast radius of things. We'll creep up the amount of people that see this over time. Evaluation outputs in production show us what real users are doing and give us intent and also outcomes.

We also give you the ability to detect issues like toxicity or hallucinations, and then automatically roll things back before too many users are impacted and also so that we can sleep through the night. There's no reason for us to wake up at two in the morning to hit a switch to roll something back when we could just allow LaunchDarkly to roll it back for us.

But you have to recognize that when you release with the AI, you are creating a system of continuous experimentation. Experimentation really gains value when it reaches a certain net volume. In other words, the more experiments you can run, the more you can learn, and the more you can learn, the more you can make these really marginal improvements over time.

This is a cheat code when it comes to AI because AI allows you to do this very rapidly. So, you know, in my eyes, we're really thinking about which releases don't need to be experiments because most of them do. And we also get the ability to do things like this much faster than we typically have done them before.

So every release really could and should be an experiment unless there's an exception. So we can test models, we can test prompts, we can test the parameters all in production, and every change becomes a measured hypothesis, not just a guess. So, you know, earlier we spoke about AI breaks some of our assumptions about traditional software development, basically.

And instead of shipping once and sort of validating before production, what we do now is continuously configure benchmark, release, observe, and iterate. When AI behavior changes constantly, you know, every model changes constantly, this is the new control gap that we're closing. We experiment with models and prompts, we get real user interactions, we apply these guarded rollouts, and then we continually optimize.

I'm gonna take a side step here, just a side note real quick about I'm using two words kind of interchangeably, and I do wanna talk about that real quick. There's optimization and there's experimentation. So we also have something called agent optimization. And agent optimization is an iterative idea. In other words, we look at something like the prompt and we iterate over that, perhaps online, perhaps offline.

And this allows us to improve that element of the prompt, but it's not a statistically rigorous approach to that. This is why experimentation in production is still really important. And so we actually offer both of these features. And in fact, the outcome, the results screen, the way that we build experiments, it's the same for code control or for agent control within LaunchDarkly. Agent optimization within LaunchDarkly is its own component so that we can optimize for maybe the best prompt and then compare it to the one that's already running safely in production as an experiment to to see that it is truly better and then roll that out to everybody. This allows us to minimize the amount of time that we put this potentially dangerous outcome into production just to learn something from it, when we already have something there.

And this mechanism is different than traditional AB tests, But we have both, and we'll actually need both in this toolkit. Alright. Closing up here. I really hope that you have these kind of four takeaways from this talk. One is that you really can't validate everything upfront.

Doing stuff in the lab in a closed environment or with your your golden dataset is not the same as the data that comes in the future. Right? So, an AI system can't be fully validated before it's released. The other is that we we also want to optimize in production and, you know, real improvement happens where real users are.

The other is that we should be doing guarded rollouts. In other words, we should always have really specific things that we're looking at to make sure that we're not seeing a regression as opposed to an improvement. And then finally, what we have the opportunity to do that's actually been very difficult for a lot of folks for many years is to really experiment at scale.

Many of us did not have the opportunity to do this, or it was very difficult. And actually, what we have in front of us now is an opportunity to do this. And experimentation is how you really validate AI behavior, but it's also how you make real gains. They take time, and they're marginal at first, but they add up over time. Okay.

Thank you. I hope you enjoyed it. Come down to our booth and see this live in action. We can show you exactly how this works, and, I hope you enjoyed the talk. We have ten minutes.

Yeah. Thanks, need a bite?

Yeah. We've got ten minutes till the next one, so have some downtime. Get some refreshments if you want. That's great. Thank you.