All right. Hi, everyone. So we already got the introduction, but I'll probably start off with a quick question. So how many of you are today machine learning engineers and primarily building for the edge? Can we have a raise of hands? So I see only a few folks here, so hopefully by the end of this presentation you'll get a good understanding of why we are heading towards edge AI and what are the different techniques using which you can become better at making your models run efficiently on the edge. So the first point is why edge AI is really hard. Because you're not talking here about huge GPU cluster nodes where you're running inference with the latest frontier model that is out there, whether it's open source models or even closed source models.

But you're talking about physical devices which are either IoT devices like Raspberry Pis or Edge where you're running OpenCV or you could talk about laptops or phones. Right? So these are much more constrained because you have limited RAM, you have power constraints because you have limited sort of battery, and then of course the thermal limits as well.

Right? Because you are shrinking down the CPU and the GPU size to the size of just like a paper thin wafer and you're transmitting it inside of your really smart in your smartphone. And you also have strict latency concerns over here because ideally you want the response to be as fast as possible. And one of the biggest constraints is it also should run offline because a lot of privacy focused applications, for example, if you are in the medical industry or in the insurance industry, you don't want your private information of your customers to go to the servers.

So you want that offline first processing of that model and should run completely offline. Of course, there are a lot of talks happening that talk about local first LLMs that you can do using Olama or there are multiple such open source projects like Docker Model Runner. But those are again running larger models on your commercial grade laptop hardware. But I'm like talking even going shrinking it even smaller and smaller for a Raspberry Pi or smartphone.

So what are some of the design considerations and thought process that we have to keep in mind when trying to deal with models? And the four constraints, as I kind of spoke in the last slide, are accuracy. So what you want is that as you try to shrink the model size to be able to fit on device, there should be good accuracy.

So there should not be a drop in accuracy because normally, as a rule of thumb, if you look at the normal model sizes, it's a 32,000,000,000 parameter model or a 16,000,000,000 parameter model. And you can go all the way down to a 500,000,000 parameter model. So as a rule of thumb, if you look at the performance, especially, you know, back if you go back 2022, 2024, generally the models that were 500,000,000 parameter or 1,000,000,000 parameter, they were really really bad at being able to get responses done.

They would either hallucinate or they would just give you random garbage, right? So we have really improved on that front. And latency is also equally important because you want to be able to get quick responses with the resource constraints of a small GPU or a CPU. And, of course, you have limited amount of RAM, so it should be also be able to fit inside of the limited eight or 16 gigabytes of RAM.

I mean sometimes if you scale down your, IoT devices even smaller than it is all always like even smaller RAM, like two or three gigabytes, and power as well. Right? So these are the four optimizations. Now one thing to keep in mind is that there's not really a magic recipe that will allow you to, you know, take a small model and it will, you know, be able to like capture all of these four different benchmarks.

Sometimes there might be a use case where, you know, maybe you go for accuracy and latency but it does, you know, take up more power. So it's always going to be a mix. You have to find that balancing act to be able to figure out what is optimized for your use case and try to optimize all four of these.

So the bigger question is for folks wondering here, where does the model size even actually come from, right? So normally if you have had a good understanding of LLMs, you'll be aware that there are like these parameters, so the actual model weights. And they kind of contain the bulk size of the static memory, all of the different trainings that we have done.

So most of your you know, storage is coming from the model weights. And then you have activations. So these are if you talk about a transformer architecture as you're going through the pre token phase, tokenization, and then you have activations, and finally you have the decode phase. So activations also have some amount of memory. And then you have the KV cache.

So KV cache is primarily used for being able to like manage your context window. So as your conversation with the model increases, so does the KV cache size as well. And there are like a lot of techniques using which you can also optimize your KV cache. And sometimes there might be some intermediate tensors during your mathematical compute that's happening while the transformer is actually making some prediction.

So those are generally like the main things that are kind of capturing what, you know, the model weights are. So and one more thing towards the right of the screen that you'll see here is when you talk about the same model, so imagine the model could be a Gemma four model, The way that you can visualize this is that you can basically represent the same model weights in different precision like so we have FP 32 with this full precision 32 bits and what you see is that generally that model, this x model, will occupy four gigabytes of space when it is full precision 32 bit.

But if you as you reduce the amount of, width, right, so for example, you can represent the model in full precision 16 bit, so that time it will be in two gigabytes in size. If you go down and down in int eight, so eight bits or integer four bits, then you would see that how gradually the model size actually reduces.

So the reason for talking about this, right, like full precision 32, full precision 16 is the fact that we are talking about quantization. And that is going to be the more important topic or one of the most efficient ways of being able to reduce the model size. So in very simple Lehman terms, what quantization really is, it's being able to take the model's data, so like whatever the dataset that you have trained it on, so in the weights, if you're first basically representing it in full precision 32 bit, you can reduce it down to eight bit integer.

So the same information is being shrunk down. Now normally, the rule of thumb is that if you're shrinking the model size and you're trying to represent the same data or the information in a smaller quantization technique, so like let's say four bit integer, there will be a loss or there will be a reduction there will be a loss in the accuracy.

But then like what you have to find is a middle ground where you can find that what is the most preferred optimization or quantization technique depending on your use case. Because depending on what type of data you have trained the model on or what the model is, there are different quantization techniques that might work better. So they're generally a middle ground where you might have to try and identify which quantization techniques has the lowest drop in accuracy which is acceptable for your organization.

But as a rule of thumb, as you shrink down the model, you get faster inference, you'll have lower memory bandwidth and you'll be able to fit the model in a smaller footprint. And I mean there are like multiple types of quantizations and of course today's presentation is not really to go into each one of them but these are some of the different quantizations which are again dependent on how the architecture of your model is.

So if you have a mixture of expert model architecture, then you might not have like so the idea is that there's not really one quantization technique that really fits the will. There are multiple quantization techniques and of course if you look at most of the popular machine learning libraries like Pytorch, Onyx, they all come with built in support for quantization.

So usually you just have to define the target model and the target type of quantization that you want to achieve. And I hope like this is a good visualization. So you know, we start with like the floating point model, so that is FP 32 and now, the idea is that the quantization really represents or the model represents both the weights and the activations.

So as a rule of thumb, you will get the best accuracy when you have more data. So of course, when the model is still being represented in the full precision 32 bit or 16 bit but as you kind of go towards the left, you'll see that, you have different techniques. So you have eight bit, weights and 16 bit activations and what will happen is that as you reduce the model in size, of course, you'll get much better performance because latency is reduced but then that will reduce your accuracy.

Right? So those are some of the key things to keep in mind when you're quantizing. And then also this high level idea of what is the difference between post training quantization and quant aware or quantization aware training. So the most common one is this post training quantization. So that means you get your model, you have already had a trained model and then you quantize it.

Basically you don't have to retrain the model and you just use a basic quantization and the only caveat is that you do have a drop in accuracy. But given the fact that if you want to have really good accuracy even post the quantization, then I will recommend using quantization aware training. So this essentially simulates quantization during the training process so the model learns to be robust with it, right?

So you retrain most of the accuracy but you need to have a proper training pipeline. And given the practical decision rule here would be that you should always try your post training quantization because it's relatively cheap. You don't have to set up an entire quantization or retraining pipeline which will require you a lot of compute resources.

But, let's say that if the accuracy drop that's happening within quantization is too much because you have very strict benchmarks for your evals that you're running, then you can go with the quant aware quantization as well. So here, you'll set your benchmarks and you'll have gradient that can basically, look at how, you know, during the training phase itself, how the quantization is affecting the results.

Now another important aspect is, so that's like number one, right? Quantization is your de facto standard of being able to get your models to be able to shrink them down enough so that you can run them on device. And generally the rule of thumb is that, by being able to incorporate your quantization technique, you will be able to like, let's say, get a laptop grade, model that's like, let's say, a Lama three, 4,000,000,000 parameter model to be able to reduce to, let's say, just five twelve megabyte or even up to two gigabytes, which is really good for most of the modern day smartphones that you typically run.

Now another technique that I'd like to talk about is model pruning. So of course, a lot of you might be aware of the English term pruning, but really what we are trying to do here is we are trying to remove some of the non essential components from an overparameterized deep learning model. So when you talk about pruning, there are primarily like two or three different types of pruning techniques.

So broadly characterized into two. One is a weight pruning and unit pruning, and depending on the level of pruning that you want to apply. So a weight pruning would essentially compress your network and it will you know help to eliminate any significant neural connections so within the different deep layers that you have. So you know if you're talking about like a transformer architecture you might have let's say four deep learning layers or hidden layers to just reduce some of those unnecessary weights or maybe the corrections between the different weights and whereas if you talk about unit pruning, would basically reduce the model size by actually reducing the number of neurons that are there within your deep learning model.

Or sometimes also if you have certain convolutional filters within the model, even those will be impacted. So generally that's the main idea and usually you'll find that this is probably not as efficient as compared to using quantization. So that is why the model pruning technique has also existed way before LLMs came into Pixlr.

So usually if you have a deep learning model, you know, all the way from AlexNet, so from the early twenty ten's, model pruning has been there. But usually what I've seen is, and at least at my work at Qualcomm, we have seen quantization as a much more efficient way of being able to get the models to work well. And another way is model distillation.

So in a nutshell, the core idea is that you take this big tethered model and it's essentially based on fine tuning. So fine tuning essentially means that you retrain certain weights of your model with structured data that you present usually in these lines. So essentially model distillation is a spin off of, you know, of fine tuning where you basically say that I have this larger data model and I will take a certain set of data set, get the responses out from you know the larger model and then use those responses as a way for the student model or the small model to mimic the kind of responses.

And also if you talk about statistical distribution, the kind of output distributions that allow the model but let the smaller model also understand that. So with that, what we essentially try to aim is that the distant model will try to think and give you a response like a larger model, but of course for a very domain specific use case. So we're not really trying to get all of the weights of a large model get shrunk into the small model.

We just do the transfer learning very specifically for those weights that are basically use case driven. And one thing to note on edge is that not every single hardware is equal, right? This is extremely important. So you have CPUs, you have GPUs, and you also have NPUs. So neural processing units are specialized hardware that are specifically designed for matrix multiplication and all of the high compute mathematics that goes within an Geniei workload like LLMs.

So they are optimized to run for edge devices as well. But the point is that you might have the same model but different hardware and you'll get completely different performance. So you not really have to optimize for the kind of target device that you're shipping for. And that's like one of the biggest differences between, you know, doing it with just the NVIDIA CUDA GPUs versus fine tuning or optimizing the model for different types of hardware because different hardwares have completely different architectures. And there are some other ways in which you can actually also optimize for when it comes to really small architectures.

So you know, because we know that edge inference is basically memory bound and not just compute bound, right? So there are a lot of other things that you could do. For example, you could do operator fusion where you basically collapse different operators. So you have you'll have like a convolutional operator, a bias operator, or a relu operator and you'll just collapse all of that into a single kernel.

So this basically means that you just are reducing the memory footprint for that particular mathematical operation or that ML operation. And then you can also do graph optimizations. So basically, if you think of, you know, the transformer architecture like when you're doing the training or doing the inference, you can basically get the entire compile type graph that gets generated when you're doing inference and you can constantly like fold it, you can remove some of the, you know, dead nodes which might be present within your entire, in in your model.

Or sometimes you might also want to fold your batch normalization, into your convolutional weights. So this is again a more, you know, ML specific term here but generally by doing all of that, working with batch normalization, you remove some of these layers which are not required in your graph and that reduces the total compile compilation time for the graph that gets generated.

And you can also do better memory planning. So for example, you might have activations which are short lived or tensors which are overlapping. So you can by basically like if you have better oversharing of these tensors, you can basically reduce the memory allocation space that they are taking up and it can basically fit on a smaller DRAM that you're running on your hardware.

And also of course kernel selection. So essentially one convolutional has multiple implementations. So if you talk about like there might be a direct, there might be a window grad, and looking at depending on what is your tensor shape and silicon, you might want to optimize and choose a better kernel. So end to end is you take the model training, you explore the model using CUDA but then you can distill it so that makes the larger model be able to like get some of those characteristics and then you can quantize it depending on the kind of hardware you want to run it on And you can work on some of these runtime optimizations on the kind of weights, activations or even the kernels that you're running and then you can go ahead and deploy.

And normally you will see that this is a balancing act as I mentioned but generally, you'll see that as you quantize and you use all the techniques that I just mentioned to you, you'll see a much faster footprint in terms of latency and also a smaller reduction of the model size as well. And these are, as I kind of finish my talk, these are some of the common mistakes that you should avoid.

So always remember that you should always profile your model, which means that always look at the actual compute and the CPU, GPU utilization that you're using. So always keep that in mind. So always profile first and guess later and always look at what your target device is and optimize it for that particular target device. And yeah, I mean we are getting better foundational models.

We are getting NPUs, hybrid architectures and really, I mean, this is the prevalent, like, Gemma four is one of the most frontier fundamental models that can run completely efficiently offline. So do definitely also try that out. It's one of the first models also with Apache two point o license. I really recommend all of you to try that out.

And with that, I'll conclude my talk. So thank you so much. And, I'll be around at the conference to respond to any questions. Thank you so much.