Testing GenAI Applications: Patterns That Actually Work

I work at Tetra. It's like a network type company, gateway service mesh, etcetera. But I've been spending a lot of time in open source for the last fifteen years or so. Actually, that's how I met met Mick a long long time ago on on J Cloud's compute stuff. And so, I do have a bias to open source and that will show through. And I did wanna do a quick recap on agent stuff on kind of like a CLI basis.

And the way I think about it is that, you know, we have this thing like the LLM, the MCP, right? So, the agent stuff is usually this handoff between the brain and the hands. And so, these are actually executable commands you could use to like invoke an actual MCP, in this case to list all the tools available in GitHub.

So, I'll post the slides later if you're interested in just whacking around in the shell. Hello. So, this is my take. I think there's a lot of different takes that we can have as far as the testing struggles. And, I mean, probably what's more explored because you don't even have to write software to know things are not very deterministic and that that has some knock on effects on testing and how you you can, you know, basically embrace that.

And, there are some like non technical things that that play out technically. So, know, I'll get into like the yellow cloud effect where people are like pushing products into public services without really always thinking about what they're doing and making decisions and undocumented things all the time and that that also plays out in our apps.

Then we have our users ourselves where we are basically in this kind of insecure and fast moving space directed by our primal emotions like fear, range and all these things. And, you can see in Reddit, all over the place people getting quite upset about pricing hikes or, you know, just this fear of missing out thing where everyone's like looking for like the latest model on the leaderboards. And then we have to add agentic stuff on top of all of it.

So, so this is sort of like just context setting, but there are there are things that we have, you know, going on and and they're not I don't think they're getting easier and I did want to share some experience on that. So, I have some tools and these these are none none of these are golden, you know.

So, it's just like a few few things have helped along the way, like how to do some stabilization, how to look at this like eval space for those of you who are interested enough to figure out like how to actually, you know, make sense of of what some of the responses are doing. And then, when we do have complex agentic scenarios, there are ways to break down tasks to the point where you can reason with them, and that was really nice to have the presentation right before about these, like, sub agents and whatnot.

I think that's a very valuable way to think. So, let's talk about yellow clouds. And this is some anecdotes here. Basically, when when we when I talk about like flaky CI, I'll do that first, that's easier. And so, flaky CI in the case LLMs is just that you're going get a different response each time.

Maybe you'll be very surprised if you do get the same response each time. And so, oftentimes, if folks are like trying to deal with this situation, their first instinct is just to retry five times and and hope hope hope for the best. Right? So, maybe if I just lower case this enough or I do an insert instead of equals, maybe I'll I'll survive.

But really, the flake is still there. Like, it's just you're trying to kind of extend that bomb to the next person. And, the thing that you can do for any any kind of weird service that you don't control, whether it's like a web service API way before LLMs or or anything, as long as it's an http backed system, you can record the requests or rather the responses to a request.

And that way you get the determinism. And there's some interesting side effects of that in AI, especially as they change their APIs in incompatible ways all the time. So, this idea of a VCR, which some of us will remember of using to like basically, like, take everything that's been sent in and then, you know, capture that basically in a YAML file so that your your test can can replay that.

Obviously, you wanna do some scraping of the the credentials so they don't they don't get posted. And so, the main idea really doesn't is not a thing about LLMs. It's basically anything. It's about getting rid of the randomness, but at least having at one point where the request was live. So so, I think that's a pretty like interesting intersection between integration tests and unit tests that like you you had at some point it it was this.

And and it can capture these undocumented changes for the same reason. So, for example, I'll get into a a case study in this, but basically responses are are can be quite different depending on library choices. And and then, of course, it's not necessarily about your cost, but it's about how many times you have to put a key somewhere and whether that key is going to expire or someone else is going to hit its rate limit or anything weird like that.

And this VCR thing, just a quick anecdote on that, is that it only works if all of your test is repeatable. So, if you have some library that's not interceptable and you're making like a service call to an LLM and then this and that, If there's other things are not actually recorded, then it's not really gonna work out. You'll end up with a flake again.

So, this would only work in a in a controlled like kind of unit testy type of environment. So, I I work on the Envoy AI gateway, is basically like a proxy that can route on the front end from like OpenAI's API to like back end Bedrock or whatever you might be using. And part of that to do like the hardening, I noticed that we didn't have like a lot of scenario tests for like all of the weird things in OpenAI.

And if you're doing things like text to audio or image to text and all the directions of it, sometimes the behavior will be quite wonky. And, like, for example, if you would enter nothing like, I I had misrecorded an audio wave and then I had put like accidentally there was no sound. And, it 500 the entire OpenAI platform.

I mean, not was my request, but if you but I found it, I was like, is this just me? And then, was like, my friend, I was like, can you just send like no audio to it? Yep. It crashes it. So, when you're dealing with something where you have like a vast of things that your agent might do, and you're dealing with portability, which is maybe not everybody's concern here, but it would be for some, make sure you capture like the weird requests.

And that will also allow you to, like, follow-up later. So, if it is quite weird and it's broken, like, maybe they'll fix it one day, and then you can just yank the cassette and rerecord it. And this this was a very interesting one for the same project. So, it's a it was Envoy AI Gateway is is actually was born out of a Bloomberg internal need to, like, kind of wrangle their, like, 100 different LLM proxies into one.

And, I had a very interesting conversation with one of the Bloomberg engineers and she's like, oh, wait, I wanna use like the the right library, the OpenAI Go library, the one that's quite good. And then, I'm like, well, it doesn't actually keep up with its issues list and it's not necessarily representing I'm like, no, no, no, it's totally right. And then, was like, okay.

As soon as I tried the GPT five nano model, it started injecting response fields that were not documented, or were not documented for the behavior that we were actually having. And then, she's like, okay, we'll just open the issue. That issue remains open today. So, this sort of thing is what I would call like YOLO clouds, meaning that people are iterating on products faster than their quality process. And you have lots of gaps between open API schemas and actual behavior. And so I do think that this is quite relevant even though the same approaches work for any endpoint.

Does play out here. And then, because I was working on like quite a large feature, because actually I was doing observability for the AI gateway, whenever you're doing any large feature, and this kind of goes back into this sort of context problem, is that when you do get to that compaction or otherwise known as destruction of your whole day, the point is that you don't want everything very very important to be something you trust the server to remember, or their side rather.

So, if they're going through their like special hook to do compaction and whatever, like, there are there are tools that you can get, like, there's MCP memory tools and there's other things you can get quite fancy. But, one thing I find is just as important as like your GooseHence or your agent dot m d file is if you're doing something quite big, literally just write it down in a in a markdown because that will not I mean, they may change it, but commit it temporarily at least.

And then, just every time it forgets, just say like reread this thing, do it again. That's what the up arrow is for in all these CLIs. I think it's because they know they just forget all the time. So, you just like hit backwards to the last time you told it to read the test plan and do it again and just do it again. Because it's a pragmatic part of the reality, and and we just have to remember that just like people, you know, often take shortcuts, so do LLMs because they were trained by our text anyway. Yeah. So, now let's get into like this rage and and FOMO and how that how that plays out.

So, I I did like some audit of like just since April, all of the craziness that's happened in the Gen I landscape that that I either hadn't hit or or felt like was like in my feed all the time. One thing that was like for me as a back end person was that at some point there was this like like we had like, wow, like there's reasoning models.

I was like, oh, that's neat. And then, of course, then you had a choice, well, do I choose the reasoning one or or the non reasoning one? And then you're like, okay, well, one's really slow, one's not. And then, of course, then there's a toggle, like, okay, well, then it can it can be toggled to to reason or not.

And, that broke a lot of tools. You have like things with like XML thinking tags going through on tools that haven't been upgraded yet and just trying to figure out whether they can do dynamic changes. So, there's some things that are playing out quite recently actually. MCP, which we hear a lot, is actually not that old, but it kind of went blasty.

I would say, like, there's there were moments. I mean, Goose was one of them. They're a founding member of that protocol, even if not necessarily always cited as so. But, I would say like GitHub putting like a MCP server out there was was one of those things that was quite quite a moment. It was also going remote, so like trusting enough to put it on a public endpoint.

And, since June, there's actually been an explosion in that. And, that's sort of changing my relationship with how I think about servers because before it's been like, okay, well, you're gonna run npx or some docker container on your on your host or like throw it into Kubernetes somehow. And then, there's some things that just would never have made sense in the first place, like, why would you launch a two gigabyte, you know, node container just to call a URL, like, it just doesn't make any sense at all.

And so, I think that that sense conversion will happen, we'll have some other new types of blips that are coming through, new types of rate limits to hit. The pricing rage, which which just seems to be ongoing, where where a lot of us who have been using something like Claude will have a 10 x, and then then you you pay the money and then they degrade your service right after that.

Leaderboard races, this is something that's that I'll get into a bit technically later, but, you know, in tech we usually say, okay, well, no one's doing testing. And now it's like, they're still not doing testing, but everybody is doing leaderboards. Like, there's like 100 different leaderboards that everyone is competing with and it's like, and then the co produced is usually untested.

So, I think we'll hopefully get the quality up at the same time as the hype, but they're never converged. Right now, think that we're very much in sort of like a who's the best model type of thing. And, some of these have interesting knock ons like GLM four or five and the Chinese models in general who are who are, like, kind of changing our relationship with, like, where thought leadership is coming from. And and I I just love these, like, snafus, like the g p d five deleting the g p t four o model, though it was a production model, and then quickly rolling it back.

And, I'm sure that caused a lot of people some bad days, as as it was obvious. And then, of course, these these pricing wars, which I think is related to the leader boarding. It's, you know, you have very, very distinctly different types of models, they have different capacities, and yet they're sort of like saying, oh, this one's 48 times cheaper than the other.

Is that true? Is it possible? Like, I don't know. Like, can you do the same commands? That's why we have these leader boards to try to assess, but there's so many of them it's hard to reason So, one way you can do that is like this thing called LLM evals. And so, this is just like a big old network diagram, but just say like the parts on one side I don't know if this is the laser one, but the purple y one is the gateway. And then, if that intercepts your traffic, then it can like throw out some formatted tracing data, OpenTelemetry stuff, which would have the request and responses recorded in such a way you could use another open source tool called Phoenix, which is from a LLM observability company.

Then they have basically a way to decorate your production traces with either LLM as a judge or human in the loop or even like thumbs up, thumbs down feedback from the app directly, so that you can get some sort of like aggregate score on on some of those things. And so, that's that's kind of just like a call out where you where you can hand over as long as you can get the data in the right shape.

So, it's kind of interesting. I think of QA and testing is like, you know, basically the same thing. It's just how are you doing it and who you're optimizing for. And basically, the end of the day, it's just a cron job usually because these are after the fact, you don't want to hit the performance of your app by making someone wait two minutes for the eval to finish.

So, there would be some sort of a way that you would have in your script to decide like how to sample what you want to actually evaluate. And then, what are the different metrics like, you know, instruction following, hallucination or things people tend to talk about. But, there are actually some interesting blogs out there that say that like, these are really made for the frontier models and and what they're going for and that just normal error analysis is the best way for you as a site to kind of figure out what you should be doing.

And so, there are programs at the end of the day, so you can write your own evals. I have a link there because I made a workshop on this when I was working at Elastic, and it has an example on how to do that. So, you know, but but when the LLM stuff was happening and Arise is a new company and LMs are not necessarily new, I mean, it's dog ears.

Right? So, if this is the year of the agents, like LMs and tools end up being together in this space and like, so we're moving from, you know, auto completing text to auto completing actions and and things like that. And, the knock on effects are pretty interesting. We already heard people talk about the session in the context management stuff, but also like tool calls themselves, where are you getting your tools from? Are they enough to poison your whole context?

And then definitely if you're looking at some of the expensive models, you want to be careful with how many tokens they use up. And, I want to just reintroduce, I know this goose has been talked about a few times, especially he's born here in Sydney, so should be. Forgive the link, but we give you a $10 credit if you want to use Envoy AI gateway as a service.

But, basically, I got involved with Goose last year as a community member when it was in Python. It was rewritten to Rust. It's actually quite crazy how much has changed in a year's time. And, it has all sorts of interesting features, and I'll talk about some of those and how they affect and impact our testing processes.

So, one of the things that's very interesting to think about with an open source project, especially one that's not like vendor as in like a tech vendor backed, is that it's open all the way and there are some impacts to how that plays out. Also, it's consumer focused, like sometimes these tools are focused on coding, and I think that's actually a fairly smart way to make an agent product because you're going to self select into a crowd who are going to be able to know some of the ways that this could have fallen down.

And, if you have a directly consumer facing agent, you actually have more problems because you have basically to accept like all types of humans into your support group. And, I think that's a very It was amazing when I saw Goose go out there because I was like, are you sure you want to do this? Like, especially if you don't, you know, if it's if it's basically just a service for folks.

And, the other thing that was quite daring was Goose was the first like MCP app I've ever heard of. They were a founding member of that, the architecture is MCP all the way down. So, it's it actually is an interesting architecture, as well as the decoupling of the LLM providers. So, they never really had too much of a dog in the race. Although, you will find some biases towards like the best model is is gonna win in these things.

Because I'm a c o l person, I I do like to use the command line tools and I do appreciate that Goose will let you do one shots and things, and and Mick himself has done some work on like GitHub actions and things, you could just roll your own quality checks using Goose as a backend for that.

So, do take a look at it. It can be in different parts of your pipeline. And, the model change impacts that affect you are gonna be some, you know, mythology about features like if you have a model card and we'll say it does this and this and this, there may be some gaps about like whether they work or whether they work in the environment that that you use, as well as just different differences that come up on the tool side, like how a how a schema is is described by a particular tool may not work well with a certain model or a certain agentic library.

And so, there's some interesting things that that I found in Goose's issues list, where people were like dutifully going through, you know, trying to do some troubleshooting with end users about how to solve some of the crises. And, I think that actually you could actually see a straight line of how that that led to changes of Goose itself, so that it could become a more efficient troubleshooting tool.

And so, like, this is basically the difference in my opinion between testing something that's quite easy to do, put in a question, get the answer back, like, it's not too complicated. Agentic stuff is complicated. So, I'm going to take the LLM eval to sort of like Goose eval, but it's the same thing applied to, whatever.

And there is a couple of things that are Goose specific, so I'll get into that. So, recipes are a relatively new feature of Goose, and in fact, I would have done this just for unit tests. But, basically, you can you can put together a combination of your prompt and the tools that it should use into a YAML and then the schema around it.

And that way you can kind of like package up a reusable task. And then you could, of course, you can make these things with cascade and such. But what I like about it is it does allow you to kind of get to the brass tacks of a particular problem space, and then you can run it against a different model and see if like, okay, well, this in fact does work really crap in model you're using, but, you know, it's it's gonna be okay if you try this one.

Why don't you give it a try? So, a same page type of thing as well as just a social sharing thing. Like, you do want to be able to teach people how to write these prompts better. So, this is, you know, is much more impactful to see like known working states than to just, like, watch someone talk on YouTube.

And so, this is just an example that you can Goose has the ability to have multiple different providers you can select from, you can pre configure them. So I can say, like, I wanna use Olama on my on my laptop, which would be free and slow versus, like, g p t four one mini, which is cheap and fast, and say, like, you know, a particularly simple thing.

Actually, I think this is one of mixed pull requests from yesterday. But you can get quite fancy with this, and there's a really interesting blog about using this recipe as like a meta generator, and they use that to kind of create scenarios to figure out, like, if you can make MCP tools, like, more token efficient.

It's brilliant. I I never would have thought to do it. Goosebench is another thing built into Goose. And so, people say benchmarks, usually they just mean like a series of tests that often have a non binary result, but but we do tend to use binary results because LLMs can reason with them better. Instead of having like five choices or saying it's like 75%, like sometimes the score will be just zero or one or like correct or incorrect. But in this, it's quite interesting because like, while cleverly named like Vibes, like there's some directories in there, there's actually a test in there about making a Flappy Bird app.

So, when you have a group that has diverse consumer base, there's all sorts of things that are going be in there. It's not just going be about writing Python. Right? So, when you keep going, you can say, okay, well, that's nice because you can test out certain features in Goose and how it's supported. You can also say like, well, how does Goose work in comparison with other agents that have similar tasks?

And I found this to be fantastic. It's a very interesting way to like interact, like basically you end up with as long as the agent, in this case Goose, can be driven by MCP, which it can, then you can actually have you can actually fork it over tasks. Like, these tasks in this benchmark would be like things like write a sorry, compile Linux kernel or do this data processing, like, a whole bunch of tasks.

And it was formerly only like a 100 tasks, now it's already up to 229. But you do have some hazards, and the most notable hazard is the speed at which a benchmark rating is obsolete, and that has to do with the frequency of models being cranked out by these frontier model providers, as well as the advances in the agents and how they work with them.

So, like, basically, if you see a ranking in some blog, and the blog was a month ago, just trash it and go look at the results again for this morning. But, are also some reasons why it's also quite difficult and you need a both dynamic agent supplier as well as a dynamic benchmark provider because there's leaks.

Like, there's some people put features into their tests to try to make it, like, obvious if something's being retrained on it. But but there's both a hazard of, like, teaching to the test and also accidentally tuning over it. And and then finally, there's an interesting hazard which is that we're getting very interesting clever model obfuscations, like a meta model that's actually dispatching right to another frontier model, and you might not know exactly what's in use at the time.

But but, yeah, let's come up for a second. So, this is a test practice. I hope you enjoyed this ride through things, but I do understand that I'm not a normal person. I don't think most people are writing an API gateway or maybe you're not writing an agentic library. So, what could you think about here? Like, one of them I would just say is like, you can see in general that flakiness is a pattern.

There are ways to dealing with flakiness. I hope that part is useful. And then, even if you're not necessarily using Goose itself to write your apps, which you can, you can still use reuse these type of thinking practices about, like, segmenting the parts of of whatever your agent, even if it's a chatbot, it's probably going to be using tools at this point.

And and so that you can evaluate those parts independently and just remember that the agility is there. Can You only lock yourself into a model for so long before the cloud might actually delete it. So, and then and then beyond that, if you if you're ending up with anything, you know, tests are one thing, but you also probably want to, like, take a look at its production side effects, tracing and things are a good way to do that.

So, hope you enjoyed it. There are some of the things that I dropped there, and yeah, have a good lunch.