Three Lanes Below One Millisecond: A Rust SDK for Gemini Live

Thank you so much. Let me know when you guys can hear me fine. So one of the key reasons why we wanna have this talk was to share a little bit from the trenches what we experienced when we started building full duplex voice agents with Python, and the sort of challenges that we had, and when we were scaling for countries like India where they were looking at cost effective means of potentially calling probably 10,000,000 users per week, or even high value customers where even the slightest glitch or pause would really ruin the voice user experience. And and these these are hard learned lessons, and there's a reason why we chose Rust.

I could literally complete this entire conversation by saying that, hey, Python's garbage collection is really bad. And when you start scaling to 500,000 users, you eventually see that the memory consumption goes haywire, and things start of pausing no matter how horizontally scalable your system is. And therefore, why Rust? Because I'm a sucker for punishment, and I like experimenting with new languages and its capabilities.

We wanted extremely strong controls on memory consumption, and therefore we chose Rust as the platform. So just a quick show of hands, how many of you have worked with or productionized speech to speech models? Amazing. So I'm sure there are some people who felt the same pains that I have. It's great to see some of my comrades in the group.

Not the STT plus LLM plus TTS pattern, which is known as half duplex, where you essentially fake the experience of a real voice to voice application, but a true speech in speech out full duplex system. Can I see a show of hands for those who've at least played around with or experimented with? Amazing. I see a few people there as well.

Look, the reason why this is so critical is the applications are really vast and wide. We could start from customer support to being a virtual receptionist for a small boutique practice. The kind of things that we could monetize this voice to voice capability is immense. However, the problem is that these voice to voice models trade granularity for raw expressive power.

While they give you a really great sense that you're talking to a human, or at least the fidelity is great, the fundamental challenge is there's something like session boundaries, you can only send a single instruction, and it usually tends to be massive. And it sucks at tool calling, and you can only define a static tool at the beginning of the session, or a collection of tools at the beginning of the session.

And you struggle when you have to dynamically inject tools. While they handle arbitrariness fantastically well, it's almost impossible to sort of get the determinism that we need by essentially having a conversational flow. What we need is a face machine that steers the conversation in a very specific way while not compromising on tool calling.

And at the same time, you cannot wait for these, say, complex state machines to compute. You'll need absolutely low computational footprint to make this work. So if you look at it, what we're proposing is that for full duplex streaming models, the transcript is the most valuable part that you can use as a control loop. Most people try to use the system prompt that you send to the native voice model as the control loop, is okay if you're doing a toy use case.

But for anything of economic consequence, you will have to operate on the transcript. And the transcript is a streaming transcript, and it fills at different phases. There are turn boundaries. The user speaks, the model speaks. You've got to constantly sort of do this tightrope dance of figuring out where the turn boundaries are, look at the transcript, compute state from that, and then use that state to compute a conversational phase, or essentially map that to a conversational phase, and then use that phase to control how the conversation flows through.

Right? Pretty simple. The idea is that, yeah, state machines and regex go really well together. You don't need an LLM to constantly operate on the transcript. You could essentially have a regex of a million odd patterns to essentially do what a small capable LLM can do. This is one of those things that we completely forgot.

Like, for example, if you want the customer to repeat after what you just said, as an example, you're collecting debt, you want to quote a regulation and say that, do you understand? You've got to repeat to consent. These are sorts of things that you can compute with regex in a conversational flow, and it gives ultra low latency.

You don't need to waste an LLM call to do that. Right? The second thing is you've got to chop, detect, and steer. What this means is you've got to put what you call state watches. Like a lot of conversations can be modeled by the changes in state, meaning that if state a went to state b, then trigger this, and so on and so forth.

Or it could be composite state where you say state a, state b, state c, they all change at the same time, or in the same turn, then trigger. So what you're really doing is you're using the raw text transcript to construct a structured output. And the structured output becomes your GPS to detect where you are in the global conversation state, and then you define how the conversation should go forward. So the problem is that this takes a lot of computation in memory.

And this happens every twenty milliseconds because audio frames and transcript arrive in streaming format. When do you decide how to recompute your state? How do you do this consistently? Like, how many loops can you run? These are sorts of questions that, you know, voice agents can make it extremely hard because the problem is if you await and trust an async process to run, there could be a glitch or a pause that you can't control, which you can feel in the voice conversation. Most people may have felt this awkward silence when sometimes you have a conversation with the model.

The awkward silence is horrendous. You can't necessarily process that because can't make up for a lag by processing ahead of time much faster because voice is sort of this continuous interaction that happens. So these are challenges that exist. The cost of a pause is really massive. The fundamental problem that we faced this has nothing to do with Google's ADK framework. It was built on Python.

We have a Go framework and JavaScript framework right now, but nothing comes quite close to rust when it comes to memory efficiency and how much control you can pin down on memory. But we don't want all of our customers or developers to become suckers for punishment by developing this from the ground up for rust. So we developed this and open sourced this package.

See, the the the first reality that we've got to account for is that audio frames arrive every 20 in a streaming audio. Whether it's GPT real time or Gemini live, the speech to speech models will keep streaming audio back in, And that's about 50 frames per second and one round trip per frame. It's not going to cut any slack for you.

Right? And then you've got to think about barge in, meaning that the user can essentially speak. While the model's speaking, you've got to make sure you pause and you listen, and then you understand what to do with the parts that you didn't play. For example, in some cases, the model may have said something of consequence and may have planned to say something on consequence and the user barged in. In those cases, you've got to repeat what you previously planned to say.

Or in this case, you might say, okay, the conversation went to a completely different state. Now I cannot talk about that topic. Now we're moving to a different topic. Think about this intuitiveness of how a human does this. They're computing all of this and deciding where they are at in this conversation phase, where you begin, where you are at, and where you end. So these are computations that you need to do, and you can't wait. So we decided to do a three layer crate build. So for those of you who worked on Rust, think of crates like packages, essentially.

And we said, Okay, for those of you that love working on the wire, who know how to work with WebSocket binary frames, who want to have even more control over how memory buffers and other things get allocated, go operate on the L0 layer. For those of you who are used to the agent way of doing things, which means that you're defining agent one, agent two, they communicate with each other in this way.

They pass this information. For those of you who built on Landgraf, CrewAI, or say ADK, this is the layer that we did. We rebuilt ADK on Rust so that we can do it. What we're trying to do is wrap this with a Python FFI so that you don't need to learn Rust, even at an SDK level, to take advantage of this.

So that's our next part of the project. And the top layer is the fluent layer. For those of you who understand what fluent operators are, any data engineers in the group here, people that write Apache Beam or Spark pipelines, they would realize that you operate most of this voice as a stream of data operations. So the transcript is a streaming text transcript, and you model your operations like a DAG or a directed acyclic graph.

So for those of you who are more comfortable with doing that, we created those verbs and nouns. And think of this like syntactic sugar. It uses the layers below to give you the same level of capability. So we're saying that a production voice agent should look like this, instruction, greeting, on audio, on text, on tool call.

Think of these as events, and you can write and compute state at all of these key events in the life cycle of a voice conversation. And we've created a three lane processor to make sure that the right things go to the right swim lanes. The idea is that we do not want to actually, you know, make the routers compute state so that there's no locks and no await that that breaks your overall computational loop.

The idea is that we created three lanes. One is a fast lane, which runs less than one millisecond. The second one is a control lane or an async lane. This is the place where you can't necessarily tell how long it will take to process it. For example, a transcript sending to an LLM can respond in one second, three seconds, or five seconds.

Right? But a regex, you can mostly pin it down. Right? Or if you write maybe an open source NLP model that you're using may not be transformer based, Their latency profile might be different. So you want to give a flexibility on the tool dispatch, face detection, control loops, and so on. And the last part is the telemetry line.

This is super critical because what you're looking at is the health of the WebSocket connection between the client and the server. You're looking at the number of frames that are going in and out. You're looking at token consumption. You're looking at how much context in the voice session has gotten filled to trigger certain things like summarization. Meaning that if you have to have a really long voice call, like most of these, models support maybe ten to twenty minutes of uninterrupted conversation, if you want to extend the life cycle of that call for much longer, say sixty minutes, if you're doing a selling pitch for an outbound SDR and you want the call to last for maybe sixty minutes, if you wanna do things like that, you've gotta dynamically trigger context compression and so on and so forth.

So we're creating another lane to make sure that those life cycle operations can stay away and stop interrupting with these. So the idea is we need something less than one millisecond. For control loops, it's async because you can't control the latency profile. For telemetry, you are okay with one millisecond as well, but it needs to come through a separate lane.

There are certain things that broke in the native Python ADK. I didn't wanna go into the details. I think I spoke about some of these things. While it may seem like we're rebuilding another agent building framework, we did not. The way we sort of want to model this is there's a shared state spine which operates on top of the transcript.

Every other person that consumes from it is a consumer, and they consume at different pace. They consume at different intervals. They recompute, some are reactive, some are synchronous. So think of these like a collection of utilities that can help you really steer and bring in a certain amount of determinism into a really capable model that can handle arbitrariness.

Right? So it needs to be deterministic where it must be. Example, if you want to model it as a DAG, let's say it needs to take these three steps before the fourth step needs to be taken, you need that kind of control loops. That's the sort of capabilities that we're building. No SDKs that these frontier model companies are building have this because they've not seen too many customers deploy this at this scale.

What we're doing is seeing the initial bunch of customers who are really pushing the limits of this. Eventually, you'll see these capabilities sort of absorb get absorbed back into the product, but right now, it's really the Wild West. So I'm not saying Python ADK is bad. It it is not suitable for a real time hot path. If you want to use production voice agents at scale with a millisecond budget, you know, you've gotta move to rust.

And the idea is that the latency budget is the architecture here for voice agents. You do not have any, you know, sort of, let's say, headroom for you to absorb, like text agent, where the model's probably thinking, the user's probably reading. You don't have any UX forgiveness. It's completely unforgiving. So therefore, we chose Rust and we rebuilt ADK, And we sort of were able to make a million calls a day outbound at scale, receive over 100,000 calls a day inbound, fairly demanding contact centers on Vernac languages in India.

But it's sort of a testimony to what sort of scale the system can sustain. That is it. I've shared the repo along with the crates link. Please do take it around for a spin. I have a contributing guide in there. I'm happy, and I'm looking for contributors from the community. We will not change the license type. It will be Apache two dot O, and it will continue to be Apache two DotO.

Thank you.