What If You Never Needed an API Key Again? Building a Mesh LLM From Spare Compute

Thank you. Thank you. You can hear me okay? Great. Yeah. So, yeah, compute is all around us. It's in this very room with us. Quite a lot, actually. A lot of it's idle. Not all of it's idle. I can see a lot of people doing work or they've got things going on, but it's often underutilized as an asset.

So what if we could pull it together to help out with some of these new workloads where we've been struggling with? So the project to do this is called Mesh LLM. We didn't think too much about the name, but I'll go into that a bit later and hopefully teach a few things on the way. So inference, you've probably heard a lot this week is a bit of a bottleneck, especially LMs and agent workloads and things like that. It's really quite compute heavy.

Training is heavier but you train once and you do a lot of inference over and over and that's kind of where things are biting right now. It's getting expensive as people are starting to find. It's very, very resource heavy. It's not always the most efficient thing with the hardware that's used. And of course, you know, there's lots of angst about data center builds and waste and all that sort of stuff.

And very crude sort of chart based on some, you know, finger in the air data from the previous slide that has some credibility. But if you sort of group, you know, squint your eyes a bit and group all the consumer stuff together versus the deployed fleet of AI specialist stuff from 2025, the bar chart kind of looks like that. There's a lot of consumer stuff out there.

That's not like you're not comparing apples with apples, but it just gives you a taste of what's going on. Just the sheer power of compute that comes out of Apple devices and and other things out there that's just being bought every year renewed or just sitting there idle. It's been there for years. So why do we want to do that?

Well, John kind of talked about this a bit. Sovereignty has been talked about this week a bit. Sovereignty is just kind of giving you optionality of where things run. So you don't have to rely on a optic fiber that that goes out. That's the one that goes out from Bondi Bondi Beach and heads across to California.

Your data doesn't have to go there. So you've got, you know, national sovereignty and and for some reason in this diagram, it's going by New Zealand, which is chilling. There's a would you do things differently if you had zero marginal cost? Like if you had, you know, this is one of the things about local models or or, you know, personal stuff is you're not you're not paying for someone else's tokens or compute by the minute or whatever. Maybe you would do things differently.

Maybe that will give you more efficiency. And model choices and other ones. So you get that's related to sovereignty. You can, you know, you're not having things changed on you without you, kind of permission. So some more wires, so the Goose project that I was part of, we would encounter people that were interested in running local models, experimenting with maybe the people that worked in different industries, you know, chemical engineering, drug engineering, things like that.

And often they would have underpowered laptops and be trying to download and run things like with four gig. No no chance of doing anything out there. So I sort of started thinking, well, I've got some idle stuff at home. I could just share that over the web for them somehow or they could pull it with their friends.

Surely there's got to be a way to give them something. Also, I've always been a fan of peer to peer networking. It's more like a hobby for me. I don't know why. It just kind of fascinates me and it has been improving. There's a lot of innovations there. Networks are getting better, mobile networks, home networks, Starlink. And unified memory architectures are now pretty common for new PCs and laptops and things like that that happen to be quite good at these workloads. And I also think back to the '90s, like there's a bit of an analogy here.

Compute went from Unix machines running, you know, Solaris, SunOS, whatever, e 10 k's bespoke things, showing my age there, to commodity stuff running Linux, which and then, you know, the Internet as we know it took off from that. So is there an analogy here to birth kind of a new era? It's it's fascinating to see.

So, yeah, that's that's some links to the project. That's the a GitHub repo. It's called Michelle Lam. The importantly, the logo is a blue bottle jellyfish, which is actually not a jellyfish that animal nerds told me. It's actually a collective organism that is itself decentralized and distributed and has its tentacles touching each other. So I thought that was a funny accident.

Just to go at the high level into it, the experience is just can be this simple. You just run this one monolithic app command and by default, it will discover something, join, work out what computes there, does it can you contribute anything, can you use someone else's. And you get as a developer, you get a a standard sort of OpenAI or Anthropic compatible API on your desktop, the familiar API and a little little console to help you out.

So sort of the the surface. Sometimes it looks like this. This was like a snapshot of a public one this morning or something. It'll it'll give some stats like how much effective RAM is available. Sometimes there'll be thousands of things in there. Someone will be being either attacking it or being silly buggers and running it on a a fleet of thousands of machines, but it's sort of designed for it.

That's really just behind the scenes sort of look at of how the nodes that are talking to each other and what ones have direct connections. There's like a you know, everything has a chat interface. That's really just to demonstrate it's working. My wife did use it for a while. She had a thing against OpenAI and that's how she would use it. And so I'd get messages whenever it wasn't working.

She's number one customer. But it's really designed for developers to use. So there are challenges. And, you know, we do these things, but not because they're easy, but we thought that would be easy. So obviously, trust if you're sharing stuff. Obvious one, we can talk about that at length. Efficiency, of course, boring stuff. But the interesting stuff is like the the stuff that people say, oh, this can't be done.

And latency is a big one, especially during inference. Training has other challenges. No no time to go into that today. It's a subject to the future, but it's fascinating. And then finally, the hardware just varies so much. Like you you have someone who has a certain GPU and if it runs over a certain temperature, it just bricks, things like that.

There's a variety of operating systems, emerging chips, GPUs, architectures and, you know, a lot of compile targets basically. So just a little bit of a backgrounding just for those that don't have. So what we mean by inference time and what goes on. Inference time is sometimes called test time is is the term of art. I don't know why.

It annoys me. I just call it runtime, but test time it's called. And, you know, this is all hand drawn, not not an AI slop. This is all me doing it by hand. So that's me on the left. I put in my hopes and dreams as a prompt, maybe video, maybe voice, maybe text. Typically gets broken down into a a stack of things like that, the system prompt, schemas, and history, and traces, and all this sort of good stuff, the thing that you just asked and post to it. And that goes to Marvin, and then Marvin will use the tokenizer and do some embedding to find related things and attention and use a case to be efficient because fundamentally this Marvin is stateless.

He has no thoughtful mind. He's a there's no memory there really. And some other terminology that you probably heard thrown around tokenizing, taking input text images, whatever it is that and turning it into numbers that you can put in big fancy matrices or vectors. Embedding, which is finding a space, a high dimensional space that these numbers or groups of numbers can be a vector pointing to.

So something that mentions cat or feline will point to the same sort of area that helps with the models understanding what you're talking about and attention and all that sort of stuff. And then pre fill is like the early part of inference when your prompt goes in and has to like spread throughout the network and like populate the caches, go through all the layers.

And once that's done, then it can start doing the decode loop. And the decode loop is when it starts just making stuff up. And that's when you get the outputs back. And the KV is the cache that helps it along. So, you know, we've got caches, you know, and that's all inference. So normally, they're on big stonking boxes that have amazing interconnect, things like NVLink and that.

So you'll have a, you know, GPUs in a machine. It might be a a PC, like a games machine, you have multiple ones or it might be one of those big pro racks with two miles of copper interconnect. I believe it's copper because the speed of light is faster in copper than fiber and that matters. And the GPUs will have a bit of their own vRAM, sometimes it's called, and the CPU will have its own RAM and then l one and l twos and but all of these pipes here, again, hand drawn, like big fat things and low latency.

Latency meaning ten milliseconds is a lot of, sorry, ten microseconds is high latency. So really, really not light networks as we know them and that's how they do tensor scaling and parallelism and all this sort of stuff and burn a huge amount of power to do inference. Another architecture that's more like what's in this room and the consumer stuff is unified RAM, like all of Apple post Intel.

If anyone's still got an Intel Mac, then good on you that it's still running, but it doesn't do this. But, you know, they'll have a pool of GPUs sharing the same RAM more or less as the CPUs and the cores. And, yeah, that that gives you pretty good fast interconnect too. So it's it's different but but also similar.

But again, it's really high bandwidth, low latency, and even specialized devices like Cerebras that build giant dies where the all the GPU and memory is all interwoven with each other on the on the on the actual silicon is is like this as well. But what we wanna do in this mesh thing is we wanna use the Internet, which is not really known for its latency and reliability. In fact, it was built for things not to be reliable.

And we wanna use commodity networking and hardware, so we're gonna use things like IP and UDP and QUIC, QUIC being the the protocol that came from Google a while ago. So you can have all these little boxes of Internets talking to each other in ad hoc ways. So to do this, we need to use tricks.

So we need to build on the strengths and avoid the weaknesses. So the strengths is there's lots of compute around, and we can do lots of things in parallel if they're happy to do it in parallel and step away from the weaknesses, which is weird, diverse hardware. It's not gonna be the same hardware. The networks are gonna be, you know, variable.

Things will come and go. Latency can be high. And that means using certain ways of spreading out models and not others. So pipeline parallelism is one, not tensor parallel. Sometimes the terms get mixed. So pipeline is you take the layers of a model and you break up the layers and you can split them apart onto different machines if you want.

You can also split up a model into different experts. A lot of the bigger models that you probably use, if not maybe all of the frontier ones or near frontier, are mixtures of experts. So they have a bunch of experts and at any one time, you're only using a subset of them. And so if you work out the subset, experts, if you like, you can group them together with the with the trunk.

And then you have a single node that can do a good chunk of that model and you just route traffic to it and you don't have to worry about latency. Mixture of agents is something that's there were some papers I found, and I've done a version of that in the mesh where you it's using different models. It's called MOA, mixture of agents, but it's really a mixture of models, where it'll use a diverse pool of of models to get better and more correct answers and tool calls than one big model.

And it's less latency tolerant, more failure tolerant. And, yeah, as I said, embracing the high latency and then do things like speculative decoding and stuff like that, racing different ones to see who comes first because otherwise it's idle. There's all sorts of tricks. And of course, if you want to host like a frontier class model, like something larger than you would typically consider a local model, Because remember all these nodes, I'm talking about a commodity, they're not going to be super powerful on their own.

There's a few nice facts we can exploit like prefill is very friendly to parallel work. So that's great. It's not going to be too sensitive to latency. So we can spread our work out and take the pressure off memory bandwidth that way. So that means we can do work in parallel. We can also use these networking technology, peer to peer stuff to detect the topology and finding low latency connections from node to node and also make sure that they don't have to funnel through any centralized point. They can just talk direct and lay out the network that way. We split the model by pipeline, not by tensor, as I mentioned. So that means we split up the pipeline.

We precalculate it into stages and we upload those. We're all kept on hugging face. So there's lots of little files. So each node only needs download the bit it needs to serve because these things can get really big and downloading can be a real pain. And each node then only needs to contribute what it can do. It doesn't have to hold the whole model or any part of the whole model.

It takes a whole lot of pressure off the energy and memory bandwidth and so on. And then of course, we use drafting, which I'll show you in a bit. So to do this, we took Alana CBP and at first, it was a bit of a fork because it had some of the bits of it. And then we decided we wanted to build on it with some patches but build our own layer over the top called Skippy, as in Skippy is a bush kangaroo.

I don't know why we thought of that name. But roughly, it works like this. You've got a these are like four nodes, which we call stages, and a stage is made up of any number of layers of the model. And the mesh will gossip amongst itself and work out a a layout of what who can cope with how many layers using a heuristic, agree how to split.

And maybe if the flow will typically be left to right. So if there's any latency to group it appropriately, if it can, to see what takes what load. And then the stage is made up of one or more layers and there's many layers, hundreds in a typical model. So when inference starts, you talk to the mesh via your local little app that you saw before from your API, connects you to the mesh, and prefill happens.

So it'll start talking at least first stage zero. And then pretty quickly, it fans out across the other stages, and they all start in parallel doing their prefill, populating their KV cache or reusing it, managing it, doing all that sort of stuff on their own without having to talk amongst each other during the prefilled stage. So that's the first part of inference going on.

Now decode loop starts, which is where the actual hallucination and nonsense generation machine begins. So it's generating new stuff. So it's got everything set up. It's good to go. And it's now gonna go, I'm gonna guess what the next token is. And I've got a pretty good idea it's this or this set of probabilities. And then it go the activations go pop to the next stage, and it's layer five to whatever.

The first one is layer zero to four and so on and so forth. You know, it just pops out each one to the end, and a single token pops out, and it goes back around the loop and it all starts again. But as you can see, these latencies add up, which is not ideal. But if they're small, like on an Ethernet network, you know, you get millisecond latency, you can still have a fair number of layers and get a pretty good ceiling floor limit to how many tokens per second that's quite useful.

And it's also whilst these things are going on, these nodes, which are computers, are free to do other stuff. They could do their regular computation. They could run a bit of Minecraft or you could do batches in parallel where other users have requests coming in. So it's not like you you it's a it's a busy wait when it's when it's done.

It's it's freeing stuff up as things flow through, but we can do better. So speculation or drafting, so instead of one token, why not try eight or something? There's I don't know if people follow all of the excitement online. There's always new algorithms for things like compressing the KV store, TurboQuan, and there's also stuff to do with algorithms for predicting tokens.

So basically, you use some kind of a very fast, small, efficient model near late stage zero. So instead of the big model, you take a really stripped down dumb one that's fast but often wrong, and you just ask it to generate a small amount of tokens or, you know, activations, all that sort of information.

And so everything happens before. It's generating tokens and activations are passing through the network peer to peer. But along with that, there's a bunch of other information for the other from the from the other predicted ones up to eight or more. And so as it goes through, not only are the stages doing the computation to predict what the next token is, they're also in parallel validating what the draft model has said, I think these are the next tokens.

And they can look statistically and go, actually, yeah, all the way up to six of those, that's pretty good. Seven is not so good or all eight is great. And that passes it on to the next stage. And if everything goes great, you get to which often it does for certain workloads, especially longer prompts, agentic things. It'll get to the end and instead of one token popping out, you get multiple. So you've massively reduced your number of round trips.

And if it was wrong and it was just a stupid model predicting, not really any harm done. There's no time wasted, just a little bit of compute. So it's very similar to CPU branch prediction in that sense. So as I mentioned before, many frontier models are mixtures of experts. So we can pre calculate what the hot experts are, hot experts in your area, and group them together with a trunk.

That still can be pretty big, but for smaller models, can actually split them up pretty nicely that way. And then you use, well, either sticky session assignment, session affinity, or you do a little bit of processing at the head and go, oh, I think you're talking about cars. You can go to the car one. And then from then on, everything happens inside that node, no latency.

And it works, but the major downside is it can inexplicably get really stupid for like, it just because it doesn't have access to the other experts that it would normally think with full interconnect. And this feels like something you might experience with models day to day because they might be using routing and doing things like this behind the scenes. So it doesn't just fail.

It will just fail in strange ways. It will forget what the capital of France is, for example. So it's not an ideal solution. So a better trick is to, you know, again, leaning on the strength of the mesh is we can have many nodes run standalone, smart, specialized models if you like. Smaller models, so instead of big ones run many smaller ones, but then find a way to combine them into a collective agent intelligence.

And in the literature, it's known as mixture of agents. So this is a diagram I had it draw. This is not hand drawn. This is pure slop. But there is a ingress and gateway, you know, from that little API thing. And then depending on the nature of the crest, is it a tool result because tool results mean different to just a prompt. It'll fan things out to a worker pool.

It might race different models to find who can get an answer quickly or maybe one of the nodes is down. And then it will use arbitration to go, well, I sent out three requests, two came back, they agree, that's enough, we can go ahead. If they disagree, then something else will use a stronger model and form it into a response, basically using a pool of models that are out there that can come and go to give a sum greater than the whole.

And this is less sensitive or not really sensitive to latency in terms of it doesn't have the activations hopping all around. But, yeah, it's it's always trade offs. Some other things we do, which is I I guess other people could do this, but because we've got our hooks deep into the inference engine that we've got.

So we can monitor runtime performance as it's doing prefill or just as it's about to start predicting predicting things or to go loop. And we can there's different signals we look for, like a spike in entropy, which is and signals like that that might indicate that the model is about to, like, hallucinate, just talk nonsense or it's lost. It doesn't know what to do.

But models don't like to not answer, so they just make stuff up. But there's often statistical signals you can use that bits in bit of C plus plus code. We hook into that and it can call out to the network and go, it'll just pause, slows down for a minute and ask for help from someone else. And so can you check my work here?

And that that can incorrect, you know, stop it hallucinating, stop it going into a doom loop of repetition and it can also make it, you know, just get things factual, factually more correct. And this all shows up as a virtual LLM, an API, but just one simple API. You don't necessarily have to configure this to get that.

So some findings we had. We were able to run, you know, sort of a half trillion parameter class model. I guess that would probably be in Frontier a year ago, that's, you know, ages ago. On commodity hardware, just with Ethernet, like not even an exotic Ethernet or anything like that. And their throughput was really great and it also left CPU and GPU to spare. And that in an office network or in a home network, that would be great.

If I stretch that across interstate networks, then the tokens per second drop a lot. You can still do it. And of course, the mixture of agents approach can span as far as you want. So there's some early proof points there, but it's still very much an emerging and experimental project with a bunch of us collaborating from all over different companies.

Part of this is there's a free public mesh, which is really just for fun. And you you log in here. You it's discovered, and you can use it. It's not there's no guarantees. It's really just for demo purposes. You can set up a private one. You can join an existing one, and you get an API. And that's it. So there's no incentives for the public one.

This is not a scam. This is not an attempt or anything. It's really just a brief point. So yeah, that's where we are. So thanks again everyone for having me. This is the technology we use and this is some links you can follow-up. So thanks.