Building SDKs in the Agentic Era

Right. Thank you for the introduction. So I've been working in open source for a while now. I worked on some of the earlier Google Maps SDKs, worked on TensorFlow for a little while.

Keep talking.

We're good? Okay. More recently, worked on our original Gemini SDK. So this talk is about our Keep talking. Our, a kind of fairly niche problem that we had with I think it's now I've got the kid keyboard shortcuts. There we go. This is about a fairly niche problem that we have, where we needed Gemini to produce the right code to build using the Gemini SDK.

Now this talk is kind of about how we did this for Gemini. I do think it's kind of applicable more broadly beyond this to, SDKs that you might be maintaining or even using, for internal projects, or even for anything where you need to kind of update the models like internal knowledge or override what it thinks it knows.

So all LLMs, but Frontier LLMs in particular, have a knowledge cutoff, and this is because of the kind of fairly lengthy process it takes to train these models. So at the start, we have this like massive dataset curation and collection. We have a fairly lengthy pre training process. And then we have a post training process where models get their kind of capabilities and personalities.

And when that's done, we have like an evaluation and testing process where we want to make sure that the models do what we claim that they can do, and often these two can can cycle through for a bit. And what this means is that you end up with six to potentially twelve months from when a dataset is frozen to when the model is actually in your hands and and using it.

And oftentimes, these datasets are reused between generations as well, so they could they can get even longer. Compare this with something like software development where projects evolve daily. And for well established projects, so something like VIM or whatever that hasn't changed in a decade, this isn't really a big deal.

But when you're working with a more, like, bleeding edge library, this can become a problem where the models don't know how to use the latest versions of the of the libraries. This can show up in a few ways. So for example, if a model sorry. If a library releases a new major version, not a huge deal because the models can still kind of write code using the the previous version. It just means you're behind on it on a generation or maybe the model can't update like a a modern code base.

But for us, the problem was that we deprecated a library that was out there and replaced it with a a completely brand new one. And so the model was continuing to write code with our old SDK. This is what the timeline looks like. So typically, we we try and launch like things in lockstep due to, you know, major events or or or things we have planned.

Going back to 2023, we had our, Palm two models. This was our first open source LLM that we had available for the public to use, and we launched it with our Google generative AI Python SDK. Now, obviously, at the point that this model's launched, it has no idea about this SDK because it was not in the training data whatsoever.

We were still writing it at the point that it was shipped. Because of this, like, six to twelve month knowledge gap, it means that these models here are like, it's only until we get to kind of one point five and two point o that they've started to even see this SDK and kind of know how to write code with it. So obviously, by the time we get to two point o, it's starting to to understand this code.

So what we did was we we deprecated it and we released a brand new SDK. Not ideal, but, you know, these things happen. With this kind of timeline, though, it means that it's, like, potentially kind of early twenty twenty six to about now where the models are expected to actually know how to use this new SDK. And so, of course, at the end of last year, we went and launched a brand new API, which is a new way to interact, with the Gemini models going forwards.

This might kinda look like Google kind of not making its mind up and and and being all over the place, but this is is just what what innovation looks like. We need to try new things. We need to, throw away things that don't work. We need to move on to new ideas, especially in in a world of AI. Just to kind of illustrate what it looks like, back in 2024, we may a user may have written some code using our old SDK, Gemini two point o, build something that connects to the Gemini API.

Oops. The the code that comes out of the model looks the same. It's almost the exact same code. The only issue here is that it's using an old model ID, which isn't ideal, but, you know, it's easy enough to fix and often will actually work the first time. Compare that with code written, you know, a year or so later using a new SDK, Gemini 2.5 flash, a newer model, same prompt, it still produces the same old code.

So it uses the old SDK, but even worse it uses an old model and by this point, Gemini one point o was shut down and not available. So when the model generates this code, the first experience the user has is that it fails. And like this is not ideal. Like this is not what we want our developer experience to look like. So we need to fix this.

How we did this was we started with direct instructions. So we know that direct instructions, so your agents m d file, Gemini MD, Claude MD, whatever you're using, we know that this is the best and most effective way to get a model to do something. We know this more recently, Vercel has published research showing how well this works. We knew this in particular because we used a prompt like this in our own software.

So in our AI studio build interface, we had almost this exact prompt so that our own systems would generate the right code when people were were building. We can scroll through this just to show you what it looks like. We've got a lot in here. We have instructions that show what the SDK looks like, what the right thing and wrong things are.

And then we have, you know, sample code across a number of languages and lots of instructions and rules in here. The next question was, like, how do we get people to actually use this? The way that we tried to do this was by putting a cogen instructions m d file into the GitHub repository for each of the SDKs.

And this is good. It's publicly available, but it's very hard to kind of stumble on and, like, almost nobody knows about this and no one uses it. And then once you found it, you still had to copy and paste this entire file into your kind of your agents. Md file, wherever that was. The benefits is that it works really well.

Like, it works well with the current models. It works well with older models. The actual biggest problem is getting people to to use it and install it. But one other big problem is the file I just showed you is almost 6,000 tokens, and that's included with every single request that your model makes. So if you've been working on a on an app and you're updating the front end and the CSS, if it's in your agent's MD file, that same 6,000 tokens is being sent on every request.

And this is not ideal. Right? Like, tokens cost time and money, and we we don't want our users to have to to go through this. So we needed something better. The next thing we looked into was using llms dot text, which is a fairly well established pattern now. At the time, our docs didn't support it, so we updated our docs to serve markdown versions of all all of the content.

Benefit of this is that if you're pointing your agent directly at your, like, markdown live documentation, it means that it's getting fresh information on every request. Like, it doesn't have to have a stale copy of a of a markdown instruction file that may be out of date by six months. It's always gonna be correct on every request.

As a pretty straightforward, we have two files that are involved. We have our main LLMs dot TXT file that lives in like a project directory, and then this just contains the navigation. So it's just a list of links and the title of the page and then the model could go further into there as it needs to. And then each of our documentation pages on the website has a markdown version.

So we have this m d dot txt file, and then that contains just a markdown version of the of the docs themselves. And then users can paste this kind of directly into their their agents and have it pulled down, you know, on demand. It works pretty well for a simple setup if you just give it a URL. But in practice, the models need to kind of navigate a number of pages.

They like to read quite a few pages. They like to read API references and narrative guides, and these can take quite a few steps and take a bit of time. A lot of the documents are quite long, and so again, they can pull up quite a quite a few tokens. But also we needed, like, convince people to use it.

Not everybody knows about LLMs. TXT or thinks to even, like, look for it to add into their project, so it's not a very discoverable solution either. And to get it to kind of work effectively and efficiently, like, you your users really need to kind of add it to their instructions and add some kind of meta propting that explains how to use the, you know, the the documentation effectively.

So the next thing to kind of move on to from here was to find a way to package all of this up into like a simple tool that users can use in in just one one like one package. Fortunately, Context seven is available and does this. For those who haven't used it, it is a documentation MCP service.

They have an index of all of the popular SDKs, websites, GitHub repositories, and you can get access to this with just a single a single tool that you add to your model. One of the biggest benefits to this is that it already exists and there is a decent chance that your users are already using Context seven and so it enables this potential serendipity where if they've asked to build something in, you know, without using your product name, they can still stumble across your product anyway, which is which is great for discoverability. It's a complete and other end of the spectrum to than what we had before.

I think for for most users you can probably stop here and say, yep, look, context heaven is great. We'll use this. Being Google, having a third party service can make it more complex more complex to to ship integrations into things like anti gravity and onto our first party services. There's a lot more like, supply chain concerns we need to need to be considerate with.

And so we needed to do a bit more diligence to make sure there were, like, no other options we could exhaust. So by now, you're probably yelling at me to

say like, oh, why are

you using skills, you idiot? And in, like, June 2026, this this makes a lot of sense. However, when we did this at the start of the year end of last year, this hadn't been done a lot. This was the first skill that Google had written and published, and so we didn't really have any kind of internal, you know, previous work that we could we could lean back on.

So we we wrote something and and shipped it. Skills are good compared to, like the other techniques which can use thousands of, passive tokens. Skills only require this kind of 100 or so tokens in the first two lines here. And then when the model wants to use them, it activates it and gets the, you know, the further, like, one and a half thousand tokens in the rest of the file.

The way that we have our skill set up is that we have our a list of current model IDs, our SDKs, and then just a little hello world in each of the SDKs. And then finally at the end, we have, like, links out to specific key pages in our documentation so that the models can jump straight to them if they need to.

So this keeps them fresh. It keeps them keeps them fairly lean and works pretty well. For discoverability, Vercel has a pretty, amazing set of tools. This MPX skills tool that you can use. You can install skills directly from a GitHub repository, and, they also have a find skill skill, which enables this kind of serendipity if a user needs to like, how do I build with Gemini?

It will go and find the right skills and and prompt the user to install them into their into their repository. So with traditional software, we were able to unit test things and we can have confidence in the things that we ship based on our like automated testing. When you're working with markdown, things aren't as easy to do, so we wanted to make sure that not only did the skills actually work as well as they appeared to just from kind of manual inspection, but if we made a change, would it still have the right effect? Would it move us in the right direction?

So we did some evals. This is what it looks like. We we ran a set of prompts through all of these models both with and without the skills. I think the baseline numbers are interesting. If you look at like the 2.5 series models, because of the knowledge cutoff, they have no understanding of the SDKs at all and get a complete 0% success rate without adding skills.

And, you know, we've got a decent but not, you know, ideal 50% improvement from adding a skill on on on the 2.5 flash model. But the the newer models, the three series, got massive improvements from adding the skill. You can see that they have some understanding of, you know, how to use the SDK just because they're a bit more recent, but they really do need the skills to get across the line.

Now the this is just an example of what the prompts look like. Think I we had a 117 prompts in our in our set that we used to eval. This is a very simple one. Like, it just, you know, hello world builds the thing that we wanna build. This one is a little trickier. It has a specific model ID.

In the training data, this model ID is tied to the old SDK quite strongly, so this requires the model to really override it. And then a trickier one that requires, like, understanding how to handle an m p three file, which requires, a deep understanding of the SDK that you won't find on the, you know, the first text generation page. We ran these through each of the models and just kind of tested whether, they use the new SDK, the old SDK.

So it's a very, like, scrappy approach, not, you know, a fully robust eval framework, but this this works pretty well for our cases. The the first chance we had to, like, test whether this was really effective, was during the development of the skill. One of our PMs included this little block of text here, with good intentions. The idea was to always point agents back to our API spec. This is the source of truth for our SDKs, for the API itself, and should be able to, you know, resolve any kind of ambiguity that that might exist.

The problem here is that each of these URLs is our full API spec, and so each one is over 80,000 tokens. So, yeah, in in a lot of cases, the models will be pulling in both of these. And this is where you see some of the older models struggle because it's just polluting context with information they don't need.

So let's let's let's kill it and see what happens. The results here are pretty good. Almost everything either held the same or improved in performance. The only exception to this was our our three flash model, which dropped by just under 2%. It was using some of this this extra information. But to me, the big wins here are that the the 2.5 models, particularly Flash, jumped up to 70% success rate once we got rid of all this, like, extra information they weren't using.

So the lesson here is, like, keep your skills lean and, you know, be considerate of your your users' kind of context. And for us, this was this was good enough to kill this extra clause. So, hopefully, this has kind of shown that skills do work and that you should publish them for the libraries that you own and and care about.

One of the challenges that we see still is that skills can still get out of date, so they can end up being, like, suffering from the same problem that we are trying to fix here. The Versal skills tooling does have an update tool, but you need to run it. Like, there's nothing that tells you your skill is, you know, twelve months out of date.

You have to manually run this, so please, like, do this now if you've got your laptops with you. Otherwise, your skills can can get out of date and, you know, causing the problem to get even worse. As a quick bonus, when we did this research, it was kind of the start of the year. We had our three series models up to 3.1 available, and we tested with those. Last month, we launched Gemini 3.5 Flash, and so I just thought I'd show you some stats on how that's performing here. Interestingly, out of the box, it now gets over 70% success rate without adding any skills.

So, like, this is partially due to a, like, a more recent knowledge cutoff. But we've also started doing SFT post training using our SDK. This helps us get up to a high baseline, but it's not, you know, very helpful for everybody else in the world. The the interesting thing, is that on top of that, we still needed the skill to get it up to 90%.

And so to get like, the the performance is still comparable to what the the other three series models were, which I think is is encouraging and, like, everybody should be writing skills for this reason. Yeah. But just a bit of an insight as to how 3.5 Flash looks. So, thank you. I've been Mark. If you wanna, like, stay in touch and learn more about these, these are my links.

If you're building with the Gemini API, grab the skill and install it now and let me know how it works for you. But otherwise, my three kind of messages are one, like update your skills now. Second, start writing skills for things you care about. It doesn't have to be SDKs. It can just be any knowledge that you you notice that the models aren't, like, you know, adhering to.

And then, like, thirdly, if you're writing skills, test and email them. I'll be down at the DeepMind booth for the today if you wanna come and talk about any this stuff. So, otherwise, thank you.