Orbital Lasers vs For Loops: Economically Matching Models to Tasks

Thank you, mate. Cheers. Alright. Thank you very much for everyone being here. Now, I will say when I chose the talk title, I did just wanna use a big giant orbital laser on the title because that makes it look really cool. Now, as we get into this, I do wanna do a bit of audience participation as a bit of a show of hands.

Now, at the end of the day, don't worry. It's not gonna be too onerous. And I can't really see too far in, so don't worry. Everyone's hidden in there. The first thing I wanna talk about here is about different kinds of failure modes that we find when people are out using AI. Now, in the first instance here, how many of us here are using agentic coding agents to say, go do the thing?

How many of us are using something like that? Cool. I would expect nothing less in this room. Now, how many of us have ever hit a message that looks something like this, that we've hit our session limit? Excellent. There we go. We got some heavy users in the room. They would expect nothing less. Anyone who didn't put your hand up, you will be after this couple of days. I promise you that.

Now, when we look at that, we have a couple of options given to us here. Now, not normally all the options that we have here, but let's face it. It is 04:31. Damn, I'm a little early. So much for that joke. But this is one thing here. One of the main personas that we have around using AI is really at the moment for the developers who are coding with AI.

But that's not all. We also have people who are building solutions with AI, who are shipping AI in their product. We've heard some of these today. How many of us here are shipping AI in our product somehow? Oh, good. We love this. Now, here's a question. When we look at our graphs, we like graphs, we like metrics.

Have we ever seen a graph that looks something like this? This feel familiar to a few people? Hear a few giggles. Yep. Okay. Cool. When we see that our token spend outdoes our revenue, and that's kind of a bit of a permanent state for us. That's not too unusual, but I do want to talk through this as there's a couple of things here that we can talk about, starting out really with that developer persona.

Now, there is a saying here that quite often, we default to things. We default to things. Now, again, we have a lot of experienced people in the room. I'm not gonna tell you that the most, you know, powerful model is the most expensive. Yes. We know this. We have the options for the models, the expensive ones, the cheaper ones, or not using AI at all.

Plenty of talks on that too here today. Realistically, most of the things that we find that we end up coming back to really are using those most powerful foundation models for a lot of things. Now I will say, for a lot of people in this room, you're probably a little bit more experienced than that. But it does surprise me how often I talk to people out in different companies, even solo developers, who do hit their limits every single time and never really look at things like their model selection, and not just from the model selection, see if it's capable, but what we can actually do with that as well.

Now, if we look at this here, just as a bit of an idea, this is kind of what I'm talking classification of different models. This is just the conceptual framework for this talk. Right? Frontier models, big and powerful. Mid range models, nice and reliable. Fast models go really quick. Sometimes we can even run them local. That's just a bit of a conceptual basis for us here.

Now, when I first planned out this talk, I had a bit of a plan for this around different models that excel at different things. Because about six months ago, we did have a bit of a case where things like Gemini was very good at doing UI. We had cases where some of the codex models were doing very well at code, while Claude was still doing better at documentation.

It was a weird time. What I realized when I came back about a month ago, and it's like I actually have stopped putting the deck together, is, oh, all of that's changed again. All of that's irrelevant. Does that feel pretty irrelevant to people like, you know, things just changed so very quickly? Yes. I see a few nodding heads.

Remember, when Ralph loops were a thing? No. That was only a couple of months ago. It wasn't even six months ago. Wild how fast things change. But in general, one thing I think that comes back to this is offering that framework at the start. So for me, this is generally a high level of how I tend to approach it.

And again, I think this is an important one for us to take back into our organizations. The key thing here is remembering the fact that LLMs at the end of the day are all about natural language. They're natural language models. Therefore, if we're doing things that do require some of that nuance in natural language, that might be our documents, our architecture, especially complex code. And remember, with complex code, it's also about the reasoning aspect as well. So even if the code itself might not seem complex, taking advantage of the reasoning is still a powerful thing.

Likewise, we're doing some of that, you know, general code that we're kind of working on, you know, some of that stuff that really ties things together. Things aren't incredibly trivial, but still there. Mid range models, I'll admit I love Sonnet 4.6 as much as anyone else, but it's not always going to be the answer for things. And quite often, it's amazing to me just how much we can get away with local models as well or even some of the faster models cheaper at the end of the scale.

Now, I won't say local models are there yet. They're certainly not. As far as I believe anyway, and again, I'm on the stage. Please don't shoot me. But looking at this one here, you know, the idea of this is that we get a bit of a sense to where the models different work for our different tasks. And when we think about this, you know, I'll give a bit of an example here.

And when we look at this, we'll have a bit of an example here of setting up, say, let's say, a GraphQL server that is going to put a front end for a SQL database. Cool. Pretty straightforward, not incredibly complex, but a bit more than the bash script. I don't think it's going to surprise anyone that would we use a foundation model?

No. Way too expensive. Would we use a nice little fast local model? No. Because they're gonna hallucinate and spin themselves in circles and not do anything. Unsurprisingly, yes. We would look at that for the best fit models down the middle. The interesting thing here though is that there are cases where those mid range models on whatever we're working might not actually meet the needs. They might not meet the needs.

So what do we do then? We can still retry on that higher level. And when we retry with that higher model, I think sometimes I talk to developers who are little bit hesitant for this because I've just wasted my time. I've just wasted my tokens. When you're talking about like a 33% saving if it actually works, quite often, running that thing on the mid level is gonna be faster. And it amazes me, like for all of the people here in this room, you probably know this quite well.

It amazes me when we go back to our organizations and we talk to people who still think, you know, that Copilot thing is kinda done. Isn't it you know, you can, like, complete sentences. It's awesome. You can complete your code. It amazes me how much people don't think about this and how much quite often we've kind of identified this into our brains.

Other people haven't yet. But if we look at this as well, I'm gonna say that when I build these things out, I tend to get about 70% of my value out of those mid range models. Again, this is probably about right. Does this sound about right to people's experience, about 70% from the mid range sort of models like Sona, etcetera?

Cool. The one that was interesting to me as I started thinking about, and I actually did some looking at things over the couple of months when I was looking at this, is how much I actually got out of some of those like faster models, some of the cheaper models, especially the Kimi models, Kimi K 2.6. Wild what that thing can accomplish at such an incredibly low cost.

Now there is a bit of a thing here when it comes to the cost as well. Again, when I was putting these slides together, I was thinking to myself, is that too many dollar signs? Is that really just being a bit overblown? Turns out, it's actually not. It really is pretty close to the thing. Right? Now when we look at this, I want to make the point here that when we're retrying with more capable models, it's still going be cheaper than just over committing on the frontier models.

And that's a really important part because we are getting to the point where tokens are costing more, subscriptions are costing more, and we need to be thinking about the efficiency upfront. To that point, this is how it works for me. I get 70% out of those mid range models. By day and nature, I'm usually a cloud architect.

I'm not a machine learning engineer. I'm not a deep embedded systems developer or anything like that. For me, this works quite well. Possibly, this won't work the same for a lot of other people. So when we think about this, there's the whole thing that you need to experiment with your workflows. Find what works for you and for your teams as well.

Because ultimately, as much as there is a sense that you can pretty much download Claude code, now type some stuff in, and you can develop stuff, I would say there's a difference between a Vive coder and a software engineer. Because part of the engineering discipline that we have to have as engineers, regardless of what your title may be, if you're here, you're some kind of engineer, you treat this as an engineering practice, very serious thing, is it's all about for us knowing which tool to use and when, knowing how to experiment to find the best tool for the job, and adapting to the myriad of new changes because everything in this sector changes all the time.

You know, in thinking about that, I was talking to a friend actually this morning because this has actually come up a bit more recently. Anyone remember what happened with a GitHub Copilot two days ago? Yeah, when they moved to usage based billing, and suddenly it's like, oh, these things actually cost a lot of money. Like, plans aren't exactly as useful as they used to be.

Shocking, I know. Now, we can all point and laugh at GitHub Copilot because, again, please nobody shoot me. I hope they're not a sponsor in this room. GitHub Copilot is probably not the current bleeding edge. So you could look at that and say, yeah, it doesn't matter to us. Realistically, if one company does something, it's only a matter of time before more follow suit.

Is that going to be soon? Who knows? But we all know at some point, things are going to start getting more expensive. So that is an important point with this. Because the most important thing isn't in terms of the cost of our models. It's actually the velocity. Now, I'm pretty sure in this room, everyone here would probably end up costing more to their company.

Like, your salary would cost more than the AI tools you consume. Probably about right? Hopefully. So realistically, that velocity is very important because when we hit that usage limit, it's not just that we end up not being able to keep on vibing some stuff along. Our velocity effectively drops to zero.

Asterisk, we can still code manually. Yes. And we still should do that. That's still definitely a thing. But by comparison to what we have, when the system is working, we do drop. We drop quite heavily. Now bigger plans and more tokens can help push these further. If we want to get, you know, a $30 plan and we're regularly hitting our limits, sure.

Get the $200 plan. Right? That's a good thing. Like, I have no arguments with that. But at the same time, if we're finding that we're hitting these limits very frequently, it is sometimes the case, should we be thinking about how we're using these as well? Because even those big plans will hit walls. And I'm going to say this, again, there's a big push at the moment for just spend more, just spend more, burn more money through the furnace. If you're just looking at this saying we can just always buy more tokens, that is fundamentally poor engineering practice.

We cannot just always spend more tokens. When it comes to the engineering discipline and the engineering practice that we have as being software engineers, we need to be mindful about how we're using our tools. And just saying always use the bigger tool is fundamentally no good. We do not want to be doing that because ultimately, cost is important. But when it comes down to it, if we want to use some of those faster models, like generally look, I'll be honest.

Some of those faster models, they're okay, but they're still pretty dumb, right, overall. But there's some things we can do with that to kind of speed them along, and that's why giving them a better context. I'll say one thing that has been fantastic in the last, I would say, year or so is I don't think I've had to write any infrastructure as code by hand because I've been able to get my models to do it for me.

Quite often, they're very cheap models. Now if you try and ask something like Haiku or something like that to write Terraform, questionable how good things will come, but you can achieve far better outcomes if you give it the context. If you plug it into the documentation, suddenly, those cheap models become a lot smarter. There is the counterbalance there.

If you add too much context, they bloat, then weird things happen, and they start driving themselves insane. But again, we can start thinking about how we want to approach that and how we want to use our tools a little bit more with that engineering discipline. Yeah? When we come back to this though, this is all about the development so far.

Let's talk about building solutions. And when we talk about these frontier models, we're talking about embedding in a product. This is a very different sort of approach because when we're embedding in products, you know, it's different because for development, we can optimize on the fly. I can change my settings. Can't do that if I'm suddenly embedding in a product.

So let's take a bit of a use case here. In this case, a customer support and reply assistant sort of agent. I will say, if you are looking at something around customer support and triage stuff, there's a good session on tomorrow about 12:00 by a fellow by the name of Dave Hall. He's actually done some really proper work in this space.

But there's a bit of an example here. If I have the three different types of models again, we can kind of expect what happens here. Frontier models, really good quality. They can get the subtext, they can get the emotion, the nuance, etcetera, and they're fantastic, but they're expensive. For the mid range models, they can get a lot of stuff, do quite well, but they might miss a few things.

Where for the fast models, they sound like a robot and it's quite terrible. So we could use them for the analysis and that makes sense. Right? But as we come across this, we start thinking about where should we start. Now after that last session, we might think that I'm gonna say, we'll start somewhere in the middle. I'm actually gonna throw that on its head.

When you're building a product, when you're embedding it, you should start at the top. You should start with the best sort of models you have available to prove the viability of the idea. Now there might be cases where you might find that the frontier model, well, isn't the best because you can expect there's a couple of things in there that would actually make an anti case.

That's fine. But overall, starting out to prove viability is the most important. Because on top of this, as a key part of this and a very functional requirement of this, we don't just want to start this out, build it, deploy, and we're done. That's not how software has ever worked. And again, if we go back to the historical way of building software, we have a patent for this.

We build it out at first. We run through an evaluation to get that assessment of how this thing is actually coming together. So we get the evaluation exactly as Abdul was mentioning before. We run our evaluation, test how it's doing, LLM as a judge, whatever you might like, and from there, we can then refine this. We can refine the model, assess it again, and deploy whichever one works best for us.

I want to talk a little bit more about refinement because this is important. Now, when we refine stuff, quite often people assume, well, take expensive model and make to less expensive model. Is good? Is bad? Thumbs up, thumbs down. Right? That's quite often where a lot of this refinement and optimization goes to die. I do find that that's often where things can fall short. Instead, you might change your prompt.

Make your prompt a little bit more on point because you have to add a little bit more into that to counteract the fact that the model isn't quite as smart. Likewise, we might add in examples. We might wanna give it a couple of shots to actually explain to it. Here's a couple of pairs of response and answers.

And on top of that, for tools, you know, searching the knowledge base, that's fine. Escalating to a human, that's fine. Do I want a slightly dumber model being able to be tricked into doing refunds? Nah. We'll remove that. So that's part of the refinement process is that we wanna consider not just the model, but the whole thing around that. The prompt, the context, the harness, the whole lot.

Because with that, we can engineer to actually make these things scale better. Right? And that comes back to the assessment criteria. Do we need identical levels of quality? Maybe, maybe not. Especially if we're talking about something here, like from Opus to Sonnet, that's a 33% cost saving. Now, this might sound like, okay, well, I spend about $20 on tokens a month in my solution.

Big whoop. What do I care? Spending $20,000 or $200,000, certainly that's a fairly significant jump. That's why I wanna wrap towards the end of this by coming back to something here about why production AI must be economical and why I don't think we hear about it as much. Now typically, when we hear about stuff in AI, we're talking about the blitzscaling startup. We want monthly active user growth at all cost.

That's what we track. On top of that, we care about our burn rate. We care about how long we have left. We're not generally too cared about our profitability as we're looking to really just make sure we can make to the next funding round or our exit. We can afford to move to burn tokens to move faster and that by burning tokens, generally, we look like we're doing pretty well. Counter intuitively, that's what the industry is at the moment. Now, for me, I tend to work I've worked in startups, but I also work in places like large enterprise, government, places like that. So this is the real world approach as to how quite often AI comes into it, and this is important for a lot of people in the room.

When we're looking at that kind of mature enterprise, we do have a bottom line and objectives to meet. We can't just burn money for the sake of it, which is kind of counter to how a lot of the traditional knowledge goes when it comes to AI. And with this, IT is often viewed as a cost center. The value metrics we often look at, you know, as something that needs to be expected there.

And at the end of the day with this one, we do have to have our cost discipline being quite well measured. At the end of the day, with this, the developers, you want to protect your quotas, protect your margins, start lower as a developer. As a solution developer, start with something big and optimize and tune as required. Because at the end of the day, cost is always a nonfunctional requirement. It is essential, always has been, always will be, everything that's old is new again.

Thank you very much. It's been a pleasure being here.