When a Small Language Model Beat Our LLM in Production

Thank you. So the beautiful picture that you see on screen with a beautiful fancy cup of tea, That is our production system. Now imagine that you hired a Michelin star chef, and all you are asking of from that chef is to prepare a tea for you. Only tea, multiple times in a day, but just one item.

You have hired an expert and you are asking them to do one single thing. Now that expert will do it, they won't deny it. Right? Now in this case, chef brings tea for you, but they can't be themselves. Right? So by the habit, what they will do is they'll present you a simple tea in such a fancy fashion.

There will be foam on the top, beautiful flowers, fancy spoon on the side, but all you needed was just a simple cup of tea. Now this was the same case with our production system. What we did was when we had to do a simple automation using AI, the default choice was we went ahead with the best in class model and the most expensive model assuming that it would be reliable and it will be easier to implement.

And it did. Right? So when we implemented, the results were fantastic. Right? But we came across some issues later and that's what I wanted to talk about today. Right? Now what happens with large language model is by default, it will work very well probably. Right? But on top of it, when it renders result, it adds sometimes its own creative layer.

Right? So at times in production where your use case has to be deterministic, very focused, very boring, repetitive use case, If you use a model that tries to be creative around it, right, instead of being simple and straightforward, it is another definition of a wrong output in my eyes. So let's see the use case.

Right? Before going into the actual use case, just a quick recap of SLM versus l SLM. LLM versus SLM. Right? When we hear the word AI model, by default, we will start thinking about the large language models, the bigger ones, the expensive ones, the popular ones which everyone is talking about.

And that's when when you start thinking about solutioning, your default muscle will automatically start gravitating towards that part that, okay, if we start using this best in class model, we will be able to leverage all the functionalities that model is providing, and we will be able to stay ahead in the game.

It's fair point, nothing wrong with that. But the real question that probably no one thinks about is, and that's what I did as well in past was, is this the right fit for the use case that we are trying to solve? So LLM is like your fancy chef. They will bring all the expertise that you need. Right? But sometimes it will also add its own creative variation and cost of course, which you don't need always.

And that's where SLM comes into picture. That is like your specialized tea cook in this case. Right? It will just give you tea when you are asking for tea. It will not add foam. It will not add fancy flowers. It will not give you fancy spoons along with it, but it will do the job that you were asking the model to do.

Now the use case. The client that I was working with was in logistic domain. So if you have ever visited any logistic company back office, you will notice that the office is filled with stack of documents. Automation use case was very simple that they want to get rid of all such manual documentation.

So whenever a shipment comes, instead of people having to actually read through the document, extracting the standard details they used to extract every single time and convert that into a tidy format, let's use AI and let's take off those manual overhead where human will only be responsible to review the results and all the heavy lifting will be done by AI. That is something that we have been hearing since past three, four years almost.

Right? Now, again, naturally, we gravitated towards the latest and best LLM model at that time. That was the right choice at time at that time as well because while using that LLM model, we didn't had to worry about training the model with a lot of documents because those large stack of documents that were there, that were all of very standard format.

So the best in class model that was there at that time was able to process it very fast and in an efficient way without having to provide too much training to that LLM model. And it was easy to justify as well. If someone asked that, okay, why did you select this model? It was an easy thing to say that, this is the best in class model. We will be able to spin up your POC in just a couple of weeks probably, not more than that.

And that happened too. Like, just within a couple of weeks, we were able to spin up demo, and it worked very beautifully. So everything, like, know, checked the boxes. It was not hard job in a way. Scenario itself was not complex. Just processed a document, the standard choice that we went ahead with, and it was the safe answer as well.

So then why I'm standing here if everything was right. Right? Now after things went to production, we started noticing couple of issues, and that's when we started thinking around, is this the right thing that we did at that time? Right? The very first issue, very, like, know, predictable, the cost.

When we did the POC, there were just handful of document that we had to process. But when the system went live, there were literally thousands of document that were being processed every single day. And the cost of LLM usage was getting high, so money, like, you know, very predictable issue that we came across. Second was the speed.

Now even though in theory, we might say that best in class LLMs would process document within couple of second, but as the volume increases, that backlog also increases because it's not just processing document. Automation includes the entire pipeline on updating or propagating those values to rest of the system.

So latency surfaced later on when we start using this system in production, but the third part was the most surprising one. What happened was even though we had added so many guardrails around the output, the model that we were using, the LLM model, sometimes it used to silently slip and bring its creative side.

And let's say if we were expecting a very specific JSON format, it would still go out of boundary and it will try to add some commentary around it or try to provide some justification around it. Now in most of the cases, it would work fine. But in edge cases, let's say if it's not following or complying to that contract or JSON format, then it might use to then it used to mess up the results in a way.

So down the line, the investment that the client had done in developing this solution, all these three came up as main problem areas that client was skeptical that, okay, should we really go ahead with the solution that we had developed? Or it's better doing, like, know, going back to the manual model.

Right? So that's where we get together in a room and things that we should have done previously before starting the project. Unfortunately, we started retrospecting about it and thinking deeper, and that's where we thought about balancing it out in a hybrid way. Right? So what we did was we bought the small language model because we thought that the documents that are there, which client is processing, which is huge in number, but it is a very predictable flow. It is a very predictable format and the standard format that client is supplying the documents with.

So we fine tuned the model to consume that format, and instead of using LLM by default, we switched to start using small language model by default. Next is this was more of refinement that had to be done. We logged the schema so that whenever SLM used to process the document, if the output is not complying to the schema that we had defined, it will immediately escalate and it will not try to be creative around the output.

So that is the second thing. And we didn't completely let go of the LLM in that case because there were certain age age cases where, let's say, was a new country that came into picture. It come with a lot of different types of formatting as well. At that time, if the locked schema is not compliant, it will escalate it to the LLM model.

So this was the hybrid flow that we had implemented. By default, go to SLM, make sure that it is complying the schema. If it is not, escalate to LLM and then try to process the document. So 98% of the document will go through SLM, whereas some edge cases will still go to the LLM as a backup. As a result of this, what used to happen is majority of chunk used to get processed via SLM, small language model.

So of course it was cheaper, it cut down cost a lot. Processing was faster as well because unlike LLM, where we used to just throw the document and let LLM do the rest of the stuff, we pre trained the model using a lot of good data that client already had. All the labeling was done. That helped us to reduce the processing time as well.

And the numbers of errors also decreased because our model was very focused to do very specific task. Right? Now, this this was the use case that worked out really well, but let's say if we have to replicate the success story in other use cases, what is the right dimension that will help us to identify that, okay, in this case, you should only use SLM, or in this case, SLM won't even work or where the hybrid model would work.

Right? So how do we identify that it's a T job or it's a five course meal? The very first thing is, generally, if there is an activity that you are doing over and over again, then it might probably a good candidate for SLM. Second is along with that parameter, if you are expecting answer in a very specific format, as in you don't have to be creative with answer. For example, if there is a customer case resolution agent, you might want to tweak the output that you are getting.

But if you have very specific output like document processing, then SLM might be a good candidate. Third is you have good training data, either it is labeled or you have support to train that data in a very specific manner. And last one is the volume is high. Last one is important because you don't want to invest so much of this when you are just processing up a couple of documents, then you can just offload it to LLM.

But you know that your volume is predictable, it is very high, and it also satisfies the other three things, then you might want to consider the hybrid model of SLM versus SLM and LLM, and not just offloading everything to LLM. Right? Now, this was the case of logistic where where main issue was speed, cost, as well as the surprises that we used to get. Right? For another use case that I also want to share, SLM was really helpful was, I just now talked about the customer sentiment analysis. Right? What happens in customer sentiment is you will get signals of customer sentiment from many different system.

It might collate into your CRM system. You might either process it from data warehouse that you have. Right? And you prepare a sentiment analysis model for customer facing. But when it comes to employee sentiment, things get a little bit more tricky because you do not want to end up being an organization that feels like employee might feel that there is a constant CCTV that is being watching over me.

So just consent is not enough in some cases. Employee sentiment is far more critical and crucial in terms of privacy, etcetera, when it comes to customer sentiment. In this use case, what we did was instead of, again, throwing everything to LLM, we processed the sentiment of employees based on their consent on their signals that they opted to provide locally on edge devices itself. And instead of passing all the output or passing all input back to the cloud model, we just passed and calculated the score to the cloud model.

So what I mean by that is, let's say if I have opted to process all my team's recording and I'm okay for my employer to process it for sentiment analysis model, what I'll do is I'll not even pass the input that is being processed to the cloud model. I'll process it locally, and I'll just output the scores of that processing to the cloud model.

That way neither of my information will go in any traceable way, as well as the free form text which is much more dangerous will also not pass, and we will only be passing the output score to the cloud model. So this was very successful because a, we were able to provide transparency, control was on employees' hands on what they want to pass it on and what they want to restrict.

B, we were able to, of course, reduce the cost because we don't have to use LLM every time for basic tasks like summary and classification. And lastly, it was speed efficient as well because we were not dealing with like no TBs of data on cloud directly. We were just dealing with the scores that was processed by the SLM model. So this was another interesting use case that we were implement that we had implemented and it went quite successful as well.

Therein, data doesn't go to model, but model goes to the data that we are processing. So, yeah, like now, as it was mentioned before, that by default, any time when we speak about AI model, it is always about large language model, and it seems like a safe choice in a way, but we don't have to always think about it.

We need to think about what is the level of repetition which we are doing with that task, what is the cost associated with it, and what is the volume that we are trying to process. However, SLM is not always the right choice. If there is some complex tasks that you want to achieve, LLM is still the best choice.

And let's say if you don't have very good example to train your SLM model with, then also you should not rely on SLM completely. And last one is don't do too much of an effort for SLM if you don't have that volume that you want to deal with. So, yeah, that was a couple of use cases that I want to share, mistakes that I had made but then rectified that someone might find helpful.

So that is about me. If you want to learn more about the technical architecture of which model were used or what was the difference that we noticed after the model were implemented, feel free to connect and we can discuss more. So, yeah. Thank you.