How Many Agents Are Too Many? The Hidden Cost of Multi-Agent Systems

So today, I'm going to tell you how just running two agents cost me the rent of a month in one single day. So I built a game. We had a similar event back in India. So I come from India, and we had an event of, you know, thousand developers roughly.

And I built this quite exciting game of finding the horcruxes. And if you're a Harry Potter fan, by the way, then you relate to this, where Harry has to find the seven horcruxes, Voldemort has hidden the seven horcruxes in different weird places. And Harry and his agents, if you might call, like Dumbledore and Ron, they go on finding and hunting these hockruxes and destroying them.

So pretty simple game, and I built this and made it live on the game day for that event. And that's when actually the reality hit. But before that, let me see let me show you how exactly this game is running. Pretty simple, we start the game, One of the opponent starts the starts hiding the hook ruxes.

Now, this is automated, so I'm just showing you the demo. The other agent or the other person, the opponent, tries to find the different hook ruxes that are there. There's a whole bunch of reasoning in the back end. Time ran out, the opponent failed, and Harry actually lost. This went on throughout the day, and at the end of the day, when I opened up my AWS console, the big bill of $1,847.

And not only the bill, but it also cost me developer experience and user experience. I got a lot of feedback. The game is slow. The agent Harry never wins. Is it broken? Sometimes, like, because it was timing out, or the reasoning was so long in the back end. It was taking time, and people were asking, is it broken?

Is it working? Why is it so slow? And these things happened without even giving a perspective to the user why it's happening. Right? So let's see what exactly happens in the back end. Now, is the RCA part of it, where we start with very simple things like tokens and how expanding and compounding context actually costs you. The first part is, I had 50 turns where Harry gets 50 turns, or the agent A gets 50 turns to find out all the hocuses and destroy them, along with his tools that he has searched, destroy, move, view, and other things.

And in first turn itself, you see there's 800 tokens that one agent is sending to an LLM, not only the locations, the details of the hockruxes that he's trying to find, the other details like maybe game state and other belief systems. This compounds in the next iteration.

And after 25 iterations, there were 3,500 tokens. And as the game was ending at the fiftieth interaction, there were 6,000 tokens each agent was sending to the LLM. That's not only context compounding, but it's memory overloading.

Also, we know that tokens cost each token cost, each LLM call costs. Now, if you are sending these many tokens in a production system, just imagine how much it costs you. And I did the math behind this. Like, if I ran thousand games a day, every day for a month, I would get a whopping $49,000 bill at the end of the month. And if I ran it for, let's say, maybe a year or something, I would get a $5,000,000 bill.

I could have brought a house somewhere in the suburbs of Sydney with that cost. Right? Not only does token cost, but also how many interactions and operations the agents are doing. We all talk about different protocols, different agent interactions.

But agents are not only interacting with each other, they have state calls, they have database retrievals, they call external APIs, they are doing web searches, a lot of things. Now in my game also, there were 12 operations for each and every turn. Now remember, there are 50 turns, and each turn has 12 operations. That's like 600 operations throughout the game.

And this total game time takes two minutes or more sometimes if you have more complex localizations, more complex reasonings behind the system, or maybe Voldemort does some trick and curses a particular location or tries to deploy a horcrux, which is not a real horcrux.

We have seen it in the movie. So then these kind of reasoning and the number of operations increase every turn. Right? So we saw there are four problems. The core of the problems, first is latency. The game is slow.

We know the reasons because of context compounding, number of operations increasing, a lot of reasoning that's happening in the back end. And also, every interaction is sequential, like once Voldemort hides, then only Harry can find. If Voldemort moves, then only Harry can go and relocate. So these things are sequential.

And a lot of these operations in real world production systems, even let's say, when you try to book a flight, when you try to maybe do a bank transaction, are all sequential. And if you deploy agents, there's latency behind that. Then comes quality. Like, how does agent a find out in a whole host of 15 different locations, the seven hockruxes, with number of active relocations, a different policy every time he tries to go, not the the same horcrux is not hidden in the same location every time we run the game in a different session.

So these things definitely need to be looked at if we want to ensure quality. And not only that, we also have the problem of cost as we saw that we are going on compounding the tokens and the cost is increasing. In the end, it costs customer trust and reliability because we are not able to orchestrate these agents properly.

So for these four problems, there are four simple solutions I devised, which I'll be talking now. The first one is bounding the problem space. Now, there are 15 locations we know of, that's deterministic. We know there are seven hoe cruxes, that's deterministic. But the actions and the reasoning is nondeterministic as we know.

So how do we bound the problem space for the different agents to work with confined guidelines? That's where defining and refining the prompts come in. So initially, if you see, before we were doing 90 iterations every game. Now with prompts, I can definitely tell agents like agent A where are these seven horcruxes hidden, but I definitely do not tell how to find them and how to actually destroy them.

Or I can, in fact, go the other way where I don't tell the locations, but I tell how you can actually destroy the hocuses. So that's pruning the constraints and limiting the search space for the agents to actually go ahead and reduce what's reduce the reasoning behind that.

The next is sorry. Yeah. The next is how you reduce the number of tokens. We saw at the end of the fiftieth interaction, there were 6,000 tokens. Now, how do you limit that? How do you limit the memory expanding? That only not not only cost not only cost you cost, but also cost you space, where you have to save all the context and retrieve it before each iteration.

So here, a concept of game theory comes from machine learning where we give it a zero sum game, and we create a belief map which works plainly on probability. Now what happens every interaction is the agent A does not have to search every location every turn of the game, even though, let's say, he has destroyed a horcrux in that particular location.

He now sees a belief system rather with 15 locations with probabilities. So for example, let's say in Hogwarts, one of the horcrux was hidden and he has already destroyed it. The next turn, the belief map shows him the probability of that location as zero because the horcrux is already destroyed in that location. How it relates back to the real world production system is, for example, you have RAG, you have a database, and you have, let's say, categories of canceling taking a simple travel problem, cancellations, rebooking, refunds, and other sort of things.

Now, how you bound this is, if you get a question, you do an intent mapping and you find out that question is from refunds. So you don't go search the whole space of cancellation, rebooking refund, you just redirect that question to get the data from your database only to relating to the category of refunds because your question relates to that. So that's how, with math, we optimize this.

The next part is how you reserve reasoning and the calls to LLM only when cases of ambiguity arise. Not for every interaction will you ask the LLMs to give you an answer, to give you a response. So what you do, you define if then else, which is a rule based system which we have ideally, in fact, our previous software engineering cycles. And then you have your AI responses through your LLMs.

So for example, if Harry is out looking for a horcrux, and Voldemort knows that Harry knows the location of that horcrux, so isn't it obvious that he will hide the hocrux or maybe move the hocrux from that location? So it's obvious. Hence, we don't take an LLM decision there. We keep the LLM decisions only when ambiguity arises.

Let's take an example where Harry now finds a hocus, but he does not know whether this is a real or a hoax hocus. So how does he find out? He calls the LLM to check the properties, to check the dimensions, to give it information about how to find the hockrocks and whether this is a real one or not.

So that's where ambiguity, if it arises, then you call the LLMs rather than calling LLMs for each and every interaction. And it also requires you to do the thresholding for each and every interaction so that you find out whether you need to call the LLMs or not. Finally, you separate the different layers of your application.

Now, how I did it is I created four simple layers. The one is the persistence layer, so I would definitely need it to save the game state, to save any data, any interactions, any multi turn conversations happening as a part of the game. I would need to process that, hence that application logic layer is required. And it will cost me, but it's a matter of dimes as and when we proceed with the game.

The next is the engine layer, which practically does not cost me anything. This is your logic layer. This is all localized. You don't go to the LLMs for any of the interactions. It's centralized where you have rule based systems. You know the deterministic outputs of those particular interactions, so you don't need to call an LLM.

The next is interface boundary. This is where, even if I have a lot of context, I won't pass every context to the LLM. I'll be compressing them, summarizing them. I know that Harry has already gone through five locations, he has destroyed five hockruxes, why to send whole of the context to the LLM? I can just keep it as a summary by giving it a simple map that these locations, these hook ruxes have been destroyed and pass it on to the next layer for processing.

The final layer is the agent layer where you actually need to do the reasoning, call the LLMs, get a response. And this is inevitable, but however, if you see, there are certain patterns where you actually find out when you can define these layers and what actually falls within these layers. Starting with I would say, start very simple, like how you define your agents, if they are specialized, if they are doing particular interactions, which definitely require LLM call, they fall in the first, the top layer of the agent layer.

Then comes something like, let's say, taking a very similar example of the travel domain, where you know there's a particular math which goes behind trying to figure out how much is the refund given to a customer when he or she cancels a ticket. That does not require an LLM call. It's a simple math. You put it in the engine layer.

So similar things can be done when you identify patterns within your application and you let the particular logics fit in the different layers. So now, with all these optimizations, I could actually make Harry win 52% of the times now.

There's always two sides of the coin when you flip, it could be a win or a loss for agent A, but now the chances have increased. Not only have the chances increased, but if you see, there's something called as the time. Game time has reduced from two minutes to forty three seconds or to one minute. And every turn now takes milliseconds, so you see a zero point zero seconds in the timer for each and every turn. So that is how you reduce not only the cost, but also the time for every interaction in your multi agent systems.

This is the math behind it, where it shows how the same agents which were running in game type A and game type B differ. So before, we were having 100 interactions, 90 interactions for each and every game. Now, with the different principles that I applied, I reduced it to 40 interactions, where only 40 times I call the LLM rather than calling it each and every interaction.

Similarly, the other benefits of how agent A can agent A win can increase, how the cost decreases, and how I optimize the latency for each and every turn of the game. Summing it all up, there are three things that you should always look out for any agentic apps you're building, be it a single agent or a multi agent system.

Especially for multi agent systems, is a mediator pattern. Don't just leave all your agents interacting with each other. Always have an orchestrator or a mediator, as we call it, to define the guidelines of how the other agents act. Right? In my case, it was game engine.

That was the orchestrator who was calling the different agents in the second state of the game and defining which agent needs to take what action. Then there's the command pattern. So every action should be treated as a command. And what you do is, before taking any action, you validate it.

Similarly, in your prompt, you might have given the positive ways of acting for an agent and the negative ways of acting for an agent. For example, if you find this information, summarize it and give it to me. If you don't find the information, don't go looking for the whole web. Just tell me you didn't find that information within my database.

So both ways of prompting and doing the validation checks on those particular actions are important. The final one is also around prompt constraints, but if you find something which is invalid state, you don't go ahead and call an LLM for that action.

If you know that a particular horcrux has been destroyed in a location, you again don't need to pass the same information back to the LLM to find out whether or to reason out whether you need to take any action or not. Similarly, in your multi agent system, let's say there's a particular action that you have done, and you found the faithfulness of that answer.

The faithfulness of that answer does not match your threshold. You do not go ahead and give it back that answer to the second agent to do a further processing. You break there. You fail fast and you fail safe. Rather than giving a wrong answer to the customer, you fail safe there. So when is multi agent systems worth?

If you have context isolation where your context does not expand beyond beyond thousand tokens, every interaction is ideally when you have multi agent systems. Even if you have used optimization techniques for memory, then if you require specialized agents for doing different interactions.

For example, in the flight industry, might have cancellation agent, you might have booking agent, but you definitely don't need an info agent which fetches flight informations. That could be a simple tool call over an agent call. And the final one is, you only have agents if your decision making is worth an LLM call.

That's when you actually build multi agent systems. With that, I rest my case, and thank you so much for listening to me today. Thank you.