Democratizing Frontier LLMs: Cloud-cluster Scale Intelligence Running on Any Desktop PC
The AI Revolution: From Transformers to Mixture of Experts
The speaker traces the arc from the 2017 'Attention Is All You Need' paper through ChatGPT's viral launch in 2022 to the current explosion of over ten frontier AI labs building models with 100 billion to one trillion parameters. The segment then explains why this scale is possible: Mixture of Experts (MoE) architecture, where a routing mechanism selects only about 8 of 256 expert subnetworks per token rather than activating every parameter, reducing active compute to roughly 5% of a dense model of equivalent size.
The Memory Problem: Model Scale vs. Hardware Reality
Despite its compute efficiency, MoE is not memory-efficient — the entire model must still reside in memory, and models like Qwen3 235B (250 GB), Llama 4 (400 GB), and DeepSeek V4 Pro (1.6 TB) vastly exceed the 80 GB capacity of a single H100 GPU. The speaker explains the two phases of inference — compute-bound prefill, which activates nearly all experts, and memory-bandwidth-bound decode, which represents 80% of inference at only 5% expert activation — then contrasts multi-GPU data center clusters costing tens of thousands of dollars against a typical desktop PC to frame the core challenge.
Monad LLM: Hierarchical Caching and Topology-Aware Inference on Consumer PCs
The speaker introduces Monad LLM, a research-grade inference system that exploits MoE sparsity to run frontier models on standard consumer hardware costing $5,000–$25,000. The architecture ranks experts by activation frequency and distributes them across a three-tier hierarchy — GPU RAM, CPU RAM, and NVMe SSDs in parallel (up to 60 GB/s per card) — then uses layer prefetching and a topology-aware scheduler to overlap loading with computation and mask latency. A live demo on a single RTX 5070 running Qwen3 30B with 53% of weights offloaded to SSD shows generation speed climbing from 7 to 21 tokens per second as the cache warms, validating that cloud-scale intelligence does not require brute-force hardware scaling.
Let's rerun the clock back to 2017, where a small team of researchers at Google releases a paper with an unusual title, attention is all you need. At the time, few people could have predicted that this would fundamentally reshape the landscape of artificial intelligence. The transformer architecture that they released, laid the groundwork for everything we now call large language models or LLMs.
Fast forward to November 2022, where ChatGPT goes viral and the world meets conversational AI for the first time. Suddenly, mothers, fathers, students, teachers, and even toddlers are having conversations with a machine. It felt like magic. It felt like the future had finally arrived.
Then in December 2024, DeepSeek r one was released, which marked the first frontier scale open source model with mixture of experts. And presently, we're in an explosion of artificial intelligence with over 10 frontier AI labs racing to build the most powerful LLMs ranging from over 100,000,000,000 parameters to to 1,000,000,000,000 parameters.
Now you might be wondering, how is this even possible? How can we serve such massive models at scale to hundreds of millions of users without requiring colossal amounts of compute? Well, the answer to that lies within the mixture of experts architecture itself. So in a traditional dense model, you have all of your parameters will be activated for every single token token processed.
Whether your AI is researching quantum mechanics or perhaps writing you a poem, all of its hundreds of billions of parameters, the same set will still be activated, which is very computationally expensive as this gets looped thousands and thousands of times. In contrast, mixture of experts uses a different approach where the model divides into specialized subnetworks called experts.
And when you're when it generates, when processing a token, a router a routing mechanism selects only the relevant experts, which is typically only eight out of 256 using DeepSeek as an example. So this, instead of instead of activating all the parameters, only activates about 5% of the parameters. So this architecture enables us enables us to run 100 b to over 1,000,000,000,000 parameter models very efficiently and gives us frontier scale intelligence with that with only requiring a 5% compute cost.
Sounds perfect. Right? Well, there is a catch. Whilst mixture of experts isn't is very compute efficient, it is absolutely not memory efficient because traditional inference technology still requires that you have the entire model in memory for compute. So how large are these? Well, here's an example. Quin three two thirty five b is about 250 gigabytes in size.
Lama four by Meta is 400 b, four four four hundred gigabytes in size. The new DeepSea V4 Pro, about 1,600,000,000,000 parameters, 1.6 terabytes in size. So for con for for reference, advanced g data center GPUs like NVIDIA's h 100 only have 80 gigabytes of memory on board. So you need to shard this model across several of those for it to work, which is rather expensive.
Now let's consider what happens when you interact with one of these models. There are two phases to AI inference. The first is called prefill, and this is where the model reads your entire prompt all at once. And during this phase, you effectively end up activating 100% of your model's experts, which is extremely compute compute intensive or called con compute bound. The second phase is called decode, and this is where most of most of inference happens, about 80%.
And during this phase, the model is, the model generates a response one token at a time order aggressively. During this phase, only about 5% of the model's experts will be activated, which is very efficient, compute wise, but shifts the bottleneck to a different place called memory memory bandwidth bound, which is effectively how fast you can move data from the memory into the processes.
Now let's consider the stark contrast between a data center configuration and your desktop. On in a data center, you can have GPUs of about 80 gigs of HBM, high bandwidth memory, and interconnected with NVLink about 400 gigabytes per second. It's very elegantly networked, but it also costs as much as several luxury cars put together. And in contrast, your desktop PC probably has about a CPU, two GPUs perhaps, connected over PCIe five with about 64 gigabytes of bandwidth costing about eight k.
So the question is, can we design inference technology, inference software that can be memory footprint efficient enough to run frontier models directly on consumer PCs? And the answer is yes. So our architecture starts with caching, which is the process of identifying and retaining frequently accessed data in faster memory to improve performance.
Your typical desktop has the hierarchy of GPU RAM, GPU DRAM, CPU RAM, and SSD storage. In order to obtain the best performance, we start by ranking all the experts in order of their activation frequency. So the most frequently activated experts will reside in the CPU in the GPU RAM, which offers the highest the highest compute and memory capability, but also has the smallest capacity.
So then our second our second subset of experts will be located in the CPU RAM, which offers moderate speed and moderate capacity. And the remaining subset of experts will be retained in the SSD storage, which offers massive capacity and is extremely cost effective, about 1000x less per gigabyte versus HBM, GPU HBM.
We need to parallel multiple storage devices together to achieve an adequate data access bandwidth. So then the next question becomes is how can we make this work without introducing noticeable latency? And the answer to that is more layer prefetching. So whilst computing on the current layers experts, our system predicts the next layers experts and loads them from a slower memory tier into a faster memory tier, effectively overlapping compute with loading transfer to mask latency. So now the question is, in order to obtain the aggregate bandwidth, the bandwidth required, we can leverage NVMe SSDs using off the shelf components like this Quad M.
Two SSD card. So running SSDs in parallel could give us 60 gigabytes per second per card. Because PCs come in a variety of different shapes or a variety of different configurations, it's it's it's critical that our software understands the exact hardware topology that it's running on in order to in order to maximize performance. So if this includes noting noting how much GPU and CPU memory there is, how fast the processors are, how fast the interconnects are, how fast the aggregate SSD bandwidth is, etcetera.
And we use this, to obtain near optimal performance from our, Topology Aware scheduler. And this is an overview of our architecture where the predictor and cache manager work together using Topology Aware Scheduling. So remarkably, this system, only cost about 5,000 to $25,000 worth of regular off the shelf hardware and should be compatible with every single, well, most there there is a baseline of PCs, and we call it Monad LLM. It's still very much research grade.
Yeah. So in this, this is one of our earlier demos. You can see the technology readiness level is about three, using the NASA TLR standards. Here, we're using one RTX 5,070, and we're running a QUIN330B A3 Mo. We're offloading so we are retaining over 53% on the SSD with 53 gigabyte bandwidth, and we're using a 10 gig GPU CPU cache.
What's interesting about this is that, initially, because the cache is empty, as the cache gets, sorted correctly, the speed actually speeds up, starting from, like, 10, you'll see seven tokens per second, speeding up to 21 tokens per second, and it even gets faster. Yes. So this, in conclusion, we've demonstrated that most sparsity can be further exploited to, to enable memory efficient inference of cloud scale models on standard PCs, and that we don't need to brute force scale the hardware, continuously in order to
Technologies & Tools
- HBM
- Llama 4
- Monad LLM
- NVIDIA H100
- NVLink
- NVMe SSD
- Qwen 3 235B
- RTX 5070
Standards & Specs
- NASA TRL
- PCIe 5
Concepts & Methods
- Decode
- Layer Prefetching
- Mixture of Experts
- Prefill
- Topology Aware Scheduling
- Transformer Architecture
Organisations & Products
- ChatGPT
- DeepSeek R1
- DeepSeek V4 Pro
- Meta
Works
- Attention Is All You Need
Explore the engineering breakthroughs behind running state-of-the-art, ultra-large Mixture-of-Experts models (100B-600B+ parameters) entirely offline on consumer-grade desktop hardware. This session dives deep into our novel storage-centric inference engine, developed by Monadd-AI and a research partner, that leverages dynamic expert offloading, hardware-topology-aware scheduling and prefetching algorithms to run Frontier LLMs directly on PCs. Learn how we run cloud-computing scale AI models without cloud-computing scale resources, just on a Desktop PC, with a live demonstration of a massive MoE model running locally.
