AI Agents Are Distributed Systems

Thank you. AI agents aren't magic. They are distributed systems with better marketing. Now, before I start talking about our system and architecture, let me tell you how it all started, because it started with something genuinely exciting. I built a chatbot.

It talked to me, it answered my questions, I even shared my favorite coffee name with it. It's lactose free vanilla latte, to which it said, brilliant choice. Of course, it's a brilliant choice. The LLM loved me. My agent loved me. I was happy. I I felt so empowered, like I unlocked something.

I decided, let's deploy it. And that was easy too, just a few commands and done. I was flying because this was the future, and I was living in it. And then, the next day, I talked to my agent, and it forgot me.

It had no idea who I was, no memory, no session. It was like meeting a stranger who had my phone number but not my name. Now, when things break, I don't accept it because I'm a developer. I ask questions.

I go find reasons, what's happening. I talk about architecture. I like to think about what exactly is this problem. And this question about what kind of a problem is this, is this whole talk all about. Now, as developers, we think about scalability, latency, availability, fault tolerance.

And these aren't just buzzwords. These are the questions that separate your demo from a product. These are the questions that determine whether you have feature or infrastructure. And if you're not a developer, these are the questions worth asking your team. So before I tell you what exactly I built, I would like to acknowledge my FOMO.

This is my phone every single morning. There's new tools, new frameworks, new models, new protocols dropping every week, and you cannot possibly experiment with them all. At some point, you have to choose something and go deep. I am a Google Developer Expert and AWS Community Builder, so I decided to choose GCP and AWS.

Now this is not an endorsement. It is not a criticism. It is just something that I know and I am still learning. So I built a little postcard agent. What it does is it sends me a postcard for a weather in a US location along with a poetic weather summary.

This is what it looks like. That is the postcard of the weather of San Francisco and a little poetic summary on the weather of San Francisco. Cute, small, nice, simple. But this is exactly what it is. I have created this agent using agent development kit, and it currently calls four different MCP servers.

A weather MCP server that I have custom built, it takes the latitude and longitude of a location. But where does this get the latitude and longitude of the location? It gets it from the Google Maps remote MCP server. And then I'm using Google's Genmedia Imagine Go experimental project. So that exists, and I actually just deployed it.

And it actually helps me to create the postcard images using Imagine. And finally, I'm using a third party agent mail MCP server for sending emails. Now, you can obviously see from the diagram that the Weather MCP server, the Google Maps one, and the Genmedia one, they all can be called remotely via HTTPS, while the agent mail is running alongside my agent and can be called with Standard IO.

This is four tools, three deployment models, two transport protocols, and one agent. This stopped being an AI demo the moment I drew this diagram. Now, before I committed to this architecture, I definitely asked myself, why not use CLI?

Of course, it would help me save some money, but what I really wanted was something composable, independently deployable tools, which can be managed by different teams or have different SLAs, and MCPs gave me that edge. So for those who are watching those MCPs dead videos, I wouldn't really agree to it just yet.

So while I created this little agent, I really enjoyed it because I discovered some patterns. The first one being so the Google Maps MCP server, it's actually managed by Google. Weather MCP server, managed by me. Agent mail, managed by someone else. And these all have different owners, different failure modes, different update cycles.

This is something too similar to what we already know. Microservices. The model just happens to be there deciding which one to call. To that point, if you notice, the agent isn't the one that is generating the image.

It isn't the one that is finding me those coordinates. It is the one that is deciding which tool to call with what inputs and when. It is like a control plane. So we need to build it like one. It is not a smart chatbot. These are different set of engineering problems.

And finally, I don't know if you already noticed, but when you're talking to your agent and if there is a sub agent that is responding to you, a sub agent can directly talk to a user. However, a function, even though it is going to give you the response, it would actually relay it back to the agent and then to the user.

It wouldn't directly talk to you. The outcome on the screen looks very similar, but underneath, the flow is different, and it matters. It matters when you are debugging. I was so happy with my learning. I decided, let's evolve this.

Why not, instead of calling the postcard agent directly, I expose it to an orchestrator agent? I expose it via agent to agent protocol, and the orchestrator agent could discover it and call it via the orchestrator agent. So that, the postcard agent now becomes a tool or like a subagent. So a call an agent is calling an agent.

My control plane has a control plane. And I did not stop there. I thought, why not expose this Orchestrator agent as a API endpoint? This stopped being a demo, and it became a back end primitive, something that is front end agnostic.

Any client can call it, and that's infrastructure. And when infrastructure breaks, it doesn't look very pretty. So let's actually look at things that break and nobody show you in a demo. So I have this little postcard agent, and I thought, okay, give me the postcard for California, but send it on my work email. And it fails successfully.

I'm sorry. I'm unable to send the email. The recipient is blocked. Now that's a successful error. But is it helpful? Not really, because it doesn't tell me if it is my Gmail settings, is it my agent mail's sending domain, is it a spam filter, or there is a policy change.

The error tells me nothing, and there was nothing in the logs either. It is a side effect of distributed system. This is the pattern we already know. Something upstream changes, something you cannot control, something you do not own, and the agent fails silently or cryptically.

If MCP had a spec change, the same thing can happen to my tools. This is dependency drift in a distributed system. It happens in microservices, and it has happened here. So what do you need to do? You just need to version your tool contracts.

Let's check out the other one. So now I know it's a postcard agent and everything, but it can still generate images because I have not got any guardrails or anything. Alright? So I ask it, can you generate a photo of a cat holding a phone? And it fails, and it errors out on maximum token limit exceeded, whatever. Token exceeded? What?

Why? It took me a day to figure out that it wasn't my Imagine MCP server that was failing to create the image. It was the agent's model itself, which was maxing out on tokens when it was trying to generate or render not generate, render the image. So I decided to fix it.

Let's give it a URL to a storage bucket and ask it to store there. Guess what happened? It recreated the image. It's definitely stored it on the bucket, but never returned me the URL. How am I supposed to view? Right? So the point being, if the image generation fails halfway, do you retry from the start?

What if the email sends out but the bucket, right, hasn't finished? Does the user get broken attachment? No one knows. Where does the state live? Who owns the failure? This is idempotency, distributed transactions. We have solved this years ago in back end engineering, and yet here we are again solving it with a language model in the middle. Okay. Question for you guys.

Let's see who is awake. What do you think is the most underestimated problem? Anyone? No one? Okay. Authentication. Guys, authentication. That's the most underestimated problem. Because if you notice, we had four MCP server. That means we had four auth contexts.

So some could use API keys. Some could use OAuth. Agent mail can have its own token flow. And now I have a secrets management problem, a rotation problem, a which credential has just expired debugging problem. And when auth fails, the agent's error is rarely auth failed on tool x. It's usually something very, very less helpful.

And don't get me started on the cold starts. Every serverless tool, regardless of the cloud provider, would have this issue. While you can fix it with some time out settings, it's more of a bandit fix because you can never never exactly have your latency control there. Let's talk about the most evilest thing, hallucination.

I'm glad I haven't asked any food recommendations or about edible foods, at least. But hallucination in a multi tool agent isn't always as obvious. For example, in my case, when I was trying to create this postcard agent, I gave the weather MCP server URL and forget forgot the slash MCP at the end.

Rookie mistake, I know. But my agent never told me that. It tried all the random tool calls, only to hallucinate and tell me to try a different location. I found it I found the error buried in the logs somewhere in the terminal. That showed me the light.

So hallucination isn't always the model making things up. Sometimes it's also model confidently reasoning about the capabilities it doesn't really have right now. You can't really trust the model, and so you need observability. Log everything from day one, not as an afterthought.

And if you're not testing your agent's tool calling behavior systematically, my friends, you are flying blind. Now, there's so much that I would like to cover, and I can watch all these emerge in the documentation, in the agent development kit, and across the ecosystem.

And to be honest, I haven't built all of them. But but I can see their shape, and it looks like a distributed system pattern that we already know. By the way, both Google and AWS have announced managed platforms to solve exactly these problems, memory, auth, observability, evaluation, governance.

This list of uncharted territory is actually being charted right now by these cloud providers, which tells you everything about how real these problems are. Okay. All right. So what did I really build?

I didn't build three different things. I just built one thing and kept asking what it needed to be, a toy on localhost, an agent with agent to agent protocol, a service that is behind an API endpoint. And each step wasn't about the model getting smarter.

It was about the architecture getting more honest. The AI part is the easy part. Everything around it, the deployment, the state, the failures, the odds, the contracts between services, that's engineering, and that's really hard.

But we already know how to do it. So for all the engineers in the room, if you are building agents, you are building distributed systems. Treat them that way from day one. Version your tool contracts. Handle your failures explicitly. Log everything. Don't wait for the production architecture to find things.

And for everyone else, if someone shows you an agent demo and it looks magical and shiny, ask them what happens when tool number three fails. Ask them how they're doing the authentication. Ask them where the state lives, and the answers will tell you if they have a demo or a product.

My agent still forgets who I am. The memory problem, it's quite real. I tried Vertex AI engine. It's promising but complex. I tried Bedrock, powerful, but the role assumption across environments is generally tricky. I'm still working on it. But I now know what kind of problem it is.

I started with a toy that made me feel like a wizard. I still think it's magic. I just now know what parts are engineering. AI agents aren't magic. They're distributed systems with better marketing. Thank you.