Long-running Agents with ADK

Hello, everyone. Thank you for your time today. So I'm Nakul Galdra. I'm an AI AI and ML architect at Google Cloud, based in Melbourne. The session I'm gonna be talking about today is, how you can run agents that run for days, weeks, even months, right, without burning a lot of tokens, without spending a lot of money on compute, and how you can do that without any sort of hallucinations or any sort of, context dilution.

So before we dive into code and examples, just wanted to set the scene with what an AI agent is. I'm pretty sure you're familiar with it, but just to set the scene, an AI agent is an autonomous system that can act on behalf of users and complete tasks. So when we think of agents, there are a couple of things that you need to look look out for.

One is the orchestration. So this could be any sort of framework, open source framework that you have. Today, we'll be talking about ADK as the orchestration framework here. Then this framework is supposed to connect the LLM models to tools and have some sort of a harness in terms of skills, profiles, and instructions there. And all of this is underpinned by memory, other sessions which is short term, or long term facts which could be the long term memory there. So a quick, you know, hands up in the room if anyone has played around with the ADK before.

Yep. Some people have. So what ADK is a quick, TLDR of A1 ADKs. ADK is an open source, agent framework that Google has open sourced. So it is available in a couple of languages Python, Java, Go, etcetera. So if you're an AI engineer or developer, I highly recommend this framework, especially because it makes it really easy, because it's modular, it's LLM agnostic, so you can plug and play any sort of models that you want.

And it's got a bunch of different features, such as state management, session management, etcetera as well. So local testing, user scenarios, everything pretty much built into the framework. So you're not hand rolling your own and can take advantage of the framework to do the heavy lifting for you. And educate covers the entire agent lifecycle, right from defining the scope all the way to deployment and everything in between.

So tool calling, memory management, user testing, simulations, pretty much everything that you would need. And I come from a Python native background, so it is very, very easy as a Python developer to get started. There's not a lot lot of configurations that you need to do. It's Python native, so it makes it very easy to understand and debug as well.

Okay. So today, we'll talk about a scenario for our long running agent, which is an onboarding scenario. Right? So generally, when you think of onboarding, we typically think often weeks. Right? So when we want to onboard a new employee, we basically send out a welcome email and for them to sign any sort of documents that they have.

Then provision hardware, any sort of software. Then an employee would confirm that they received the hardware. And then we would basically set them up for the first day. So introductions, calendar invites, all of that stuff as well. So generally this whole onboarding process would be typically a few weeks and we want an agent to take care of this.

So we want the agent to understand where the employees are in the journey and help them throughout the journey until they get onboarded. So if you have to build an agent to meet this, it has to have certain criteria. It has to be long running because again the horizon task is about weeks. It has to be cost efficient.

We don't want this agent to be running all the time. It has to sleep. When I say sleep, it has to completely shut down and still remember the context there. The next one is it should be able to delegate to sub agents, like specialized agents as needed as well. So we shouldn't have a single agent that does everything.

We should be able to chop and change and provide multi agent systems for it to run as well. So you might have seen like, hey, chatbots on all of these agents as well. So this wouldn't be a great fit for our use case mainly because the chatbot sort of interfaces are, you know, they can have a lot of sessions there, which can lead to context pollution, right? So when we want to know the exact state of where the employees are on their employee journey, if they have a lot of interactions, that context can get polluted and the agent can hallucinate or think, hey, the employee is actually on a different state when they are not.

Next is the token cost. So we want this agent to actually shut down when it's not in use. We don't want any sort of token cost to be used up, especially if you have a lot of sessions, a lot of state. Putting all of that together for every single query becomes quite expensive. And the hallucination as well.

So we don't want the agent to understand or forget or think that employee's on a different state when they actually know. So what we're looking at here is a long running agent that we can build out using the Python ADK framework, where we have a durable session state. So this session is deterministic, meaning that hey, there is no sort of, hallucination, that can happen.

There is a durable schema as well. So this is a predetermined schema rather than any sort of output that LLM can generate. It is event driven, which which means that the agent shuts down. And as soon as an event gets triggered, the agent wakes up, completes the task, goes back to sleep. And this agent is a multi agent system as well.

So we have a HR coordinator agent, then we have an IT specialist agent. So they can coordinate with each other. And we want to do all of this in a way that is robust, is observable, we can put in evaluation criteria, etcetera. So we'll talk about the logical architecture of if you had to build this agent, how would this look.

So first is we have to initialize the details for the new employee, so their email address, their credentials, whatever they need. Next, we send out a welcome packet to this employee. Then we wait for this welcome pack and the documents to be signed. So the agent shuts down. We have a webhook, which basically can be triggered on an event.

So maybe there's a DocuSign, they sign something. Based on that, we trigger off an alert. Then this webhook is caught by this agent. Agent wakes up. It will run through the process for that welcome part of the email. Then again, goes to sleep. Next, the hardware or software is starting to get provisioned. Once the employee gets the hardware, they confirm that they have received this hardware. That triggers off an event.

Event comes back to the agent. The agent wakes up. It does its thing. Then again, goes back to sleep. So that's the sort of logical architecture we're looking to build. And if you had to put this into, hey, how would we do this with ADK? First, we would have to have a fast API gateway or webhook that basically sends inbound requests.

Then we have a root agent, which is our HR coordinator agent. It has a bunch of different tools for it to complete the task. Then we have a sub agent, is specialized for a specific task. As example, hardware provisioning, software provisioning, etcetera. It's got tools dedicated to do that as well. Then we have a persistent state management.

So this is a database. We have used Cloud SQL. But again, ADK supports any sort of database that you want, price grace, AlloDB, etcetera. So you can put in any sort of database that you want there as well. So this is the technical architecture. So you have again, why I put the development tools here is you don't have to hand roll your own ADK code.

We have a CLI called agent CLI, which can take care of all the scaffolding for you. So the Terraform scripts, creating even the framework for the ADK itself, you can use agent CLI to do that. You can create evaluations using the agent CLI. And you can even use that within your ID of choice. It could be anti gravity, etcetera.

Then we have the framework itself, which is ADK. Then we have the persistent store, which is Cloud SQL in this case. Or you can use for local development SQLite, for example. Then you need to have a runtime where this agent runs. You can use other Cloud Run. You can even use your own Kubernetes cluster if you wanted to.

It doesn't really matter. In our case, we have used, agent runtime, which is our Google native service that manages the agent there. Cool. So now we'll dive into the code and how this would work. So this is the schema that we have defined. So we have explicitly defined states. So the start state, welcome pack is sent, documents assigned, hardware is delivered, hardware is provisioned, and finally completed.

So what we're going to do is we're going to use these state variables and have two couple of state gates called as current step and pending signals. So this is how we will track where the employee's at on their journey. And why this is important as well is if a container goes down, if an agent goes down, we have the state so we can always rebuild from the state and carry on as well. Next what we're going to do is we're going to take these states and inject them into our system instructions.

So if a certain step is there, this is what you would need to do, etcetera. So it's either calling tools, it could be calling sub agents, it could be shutting down as well. Next, within ADK, have something called as a tool context. So what tool context provides is a way for you to access the state variable. So within the ADK, we have something called a runner. And a runner manages the state.

But this state can be accessed using something called the tool context. So for example, you can see where the state is at at a certain point. And based on that, the agent will understand, hey, what's the next pending state to do? Because we can manage and access the state, we can actually save it in a database as well.

Right? So for example, if an agent goes down, if something happens catastrophic to the agent, the state is saved at a user level and a session level. So you would understand if I have to reboot, you understand exactly where the flow of this agent task is. This is great. Now we have an agent working with manager state, but how can we trigger this agent when it goes to sleep?

So for this, we can use the fast API endpoints or webhooks. And what this basically does is as soon as an event is triggered, it will run, what we call as the onboarding resume handler. What this does is basically hydrates the state. So for example, the the agent is down, completely shut off. As soon as an event comes through, this handler gets triggered.

It will hydrate the state from our database. So it will update the current state, and then it will actually trigger off the agent run. So we have something called as a runner, as I mentioned, with an ADK, runner async. Basically starts up the runner. And then it will hydrate the state so it understands exactly where, the employee is on their onboarding journey.

Again, based on the state, we have the system instructions the agent runs. It was able to look at the state variable and able to pick up what it needs to do from that state onwards and carry on. So here, for example, document signed, it delegates to a sub agent. The sub agent is able to, you know, start the hardware as well as the software provisioning.

Cool. After that is done, again, the agent goes back. It's wait waits for the hardware to be delivered. Once the hardware is del delivered, an event gets triggered, event comes through, the resume handlers gets kicked off, the state gets hydrated, the agent continues on this journey. So that's how we can build an agent that can run for for days, weeks, months without actually having to burn a lot of token cost because the agent goes down.

You have a way to handle those state changes that is deterministic in nature. You're able to handle those events as well. Okay. Let's see this in action. So we have open sourced this as a GitHub repo, so you can actually go in and and play around with this as well. So again, this is meant as a a skeleton or a framework for you to build upon.

And the use case is about employee onboarding, but you can take this for customer support, especially when you're doing async sort of chatbots. It could be for sales motions, for example, if you've put in a marketing campaign and you wanna check how effective that campaign was, it will take a long framework to to get those events in as well. So in this scenario, you're seeing that we have a new employee that's about to start.

And if you go and have a look, you have a hiring packet with the employee in this a fictitious employee called Olivia. We have a welcome email. As as you can see, there's a pending signature. Right? So what I can do is I can go and again, I'm triggering this with a click, but again, this could be through an API as well.

Right? So I'm gonna sign the packet. As as soon as you sign the packet, as you can see, packet has been signed. And again, the state has been updated to document signed. Now it goes to the next state, which is hardware provisioning. Now the hardware has been sent, the employee receives the hardware and they confirm either through like FedEx or through an email.

That would be triggered and then we can trigger off the hardware received as well. Now the hardware is received. The agent now goes to the next state, which is hardware received. Now is the day one welcome and schedule. As you can see, the agent is able to schedule day one for this employee. Welcome, log in setup, the meeting and greeting the team. And as you can see, has gone through that as well.

So this is how you can you can sort of build an agent, that is dormant and gets triggered at a certain point in time. You might be wondering, is great. This is a a demo. How can we actually deploy this to production? So the next step is actually putting in a robust eval plan in place. So it could be like golden dataset for a long running agent. It gets triggered.

Does actually the API endpoint gets triggered and stuff like that as well. We have a system called user simulations built into ADK as well, so you can catch edge cases that might not be captured within your golden data set. Then we have online evaluations as well. So if your agent is running in production on Google Cloud, for example, then our agent observability platform, because of OpenTelemetry logging, is able to capture the user prompt, the response, the tool calls, the tool trajectory, everything that you would need.

And all of that becomes an adaptive rubric, which is a dynamic rubric that gets generated by the platform itself. So you can catch errors as they happen in real time. Next, obviously, you're putting this in the wild, you want to have some security guardrails, so service such as Model Armor. It's an API that you can plug in.

You can look at PII information. You can look at prompt injection prevention, all of that stuff as well. So you would want to plug that in. And obviously, you want to look at logging. ADK natively supports OpenTelemetry, not just the JSON, but also the native primitive data structures as well. So gen AI underscore something which OpenTelemetry supports.

So as OpenTelemetry updates this native schema, ADK will update as well. So which is, you you don't have to worry about trying to match OpenTelemetry feature feature state because ADK natively will ensure that. Cool. Thank you for your time. I just wanted to share the blog, which goes into a lot more detail about this session that I talked about with sample code.

And also, we have a great ADK community as well because it's open