From Zero to Production: How 15 Engineers Shipped a Production LLM Product with AI Coding Tools

Hello, everyone. I'm enjoying the conference so far and I'm hoping that you enjoy my talk as well. Today, I'm going to share a story on how a small team of engineers at MLB build and ship an LM powered agentic chatbot within the MLB stack and the learnings we got when we build it. Before I start to share, I want to ask a questions.

Please raise your hand if you have worked with a financial report, like a balance sheet report, a profit and loss. Okay. We got a few, about 10. What about would you claim yourself an expert in those report? Please raise your hand if you claim an expert in those report.

Nice. We got one. I wouldn't. After several months of working this project, I wouldn't claim myself an expert in reading those report. And we found it is the same for a lot of our customers, The small and the medium business owners who are the users of MLB accounting and business management software, they may or may not be an expert in those report.

But they want to get useful information from the software, they want to get insight, they want to know how good or bad their business is performing, and they want to spend the least amount of time to get to that so that they can spend their precious time into solving their business problem. That's one of the reason we start this.

And also, there's the growing trend in the market that more and more people, more and more software users would just love to talk to their software through voice, through chat. This is a new interface, this is a new trend, and that's another reason we started to build this chat based AI assistant widget into the existing MLB stack.

Imagine this like a Gemini or ChatGPT, but that actually sit within the product suite, have the same look and field as the MLB product suite, and it can answer your financial related questions. The difference to a ChatGPT is that it's not trying to do a general knowledge bot. It will refuse to answer the questions such as what's the weather today. But the benefit of doing this way over the Manila ChatGPT without the connector is that the business owner are able to talk to their own books.

They can ask their questions like, what is my profit last month? How am I looking like? How long of the runway I still have? It's a natural language interface and people love using it. As of today, we have tens of thousands of users can use it. This feature is under a feature flag in the production and we spent less than six months building it with a small team.

One of the lessons we learned is that, especially at the beginning, we want to start narrow. There are two guiding principles when we first started. One is that we want to focus on the speed of delivery, and second is that we want to focus on the security and the safety of the product because we are in the regulated industry.

The price of going wrong is high. So those two things might sound contradictory to some extent. And how do we achieve that? There is something we need to compromise. Cost at the very beginning might be one thing, but the other thing we compromise at the very beginning is the initial scope of the release. We don't start at something really big.

We are not trying to build a general purpose co pilot. We were not even trying to do the agentic actions some of the user might be expecting this chatbot copilot do. What we started with is a very narrow scope. Read only chatbot can answer some of the users' questions with a decent evaluation framework around it.

Doing it this way allow us to ship to the market fast, And this fast iteration enable us to get feedback fast. One of the very important feedback we want to get from this is a very simple question. What the user want to know? What are they actually asking? So kind of like a chicken egg problem here, without actually shipping some products, especially a chatbot product to the market, we wouldn't know for sure what the user is asking.

And without knowing what the user is asking and without knowing what we can and cannot support it, it's very hard to guide the next iteration of the product. So that's one of the reason we choose the way of progressive rollout. At the very beginning, about only 10 user inside of the company use it. And then we grow, and then we grow, the user base become larger and larger.

Another reason is that we can control the risk of it. The same mistake happened at the earlier stage of the product. We'll have much smaller risk when comparing against a mistake happened at the later stage of the product. I'm not so proud to share this, but I did cause an outage about thirty minutes in the first month when the product is released, and I was responsible for it.

At that time, nobody saw it coming, but we got a very valuable license out of it. And the price we pay at that lessons is relatively small since only minimum user use it at that time. And if the same thing happened today, the lessons we pay, the the the the price we need to pay for the same lessons is much larger and there will be more damage cost by this one. That's another reason we choose the way for progressive rollout for experimental AI features like this.

While we are building and rolling out this chatbot feature, security and the safety of the product is always at its core architecture. We are not trying to cut corner over there. We're trying to make sure that we tick every box of the security checks. One way to do that is by pen testing.

Not just one pen testing, and we did it and we forget about it. It's never going to be the case for the progressive rollout and experimental features like this. We need not just two, but more pen testing. There is one on the way this Friday. We will kick off another one. So the pen testing itself become the milestone we need to get to as part of the rollout process.

We need to hit the milestone of the pen testing. We need to find vulnerabilities from the pen testing, and we need to remediate the findings from the pen testing. So that is how we ensure that we find all the possible vulnerability potentially sits within the product. And they always comes from different angle.

One angle, of course, is the traditional web application side of the attack, like DDoS attacks. We need to have some certain rate limiting. We need to have we need to make sure that we can withstand certain load. One part of the pen testing makes and the defense we have to make sure we have those.

But also very importantly, since this is a large language model based product, it's a chatbot, we need to make sure that we have the guardrail in place so the LLM doesn't suddenly start to write code in a financial product. The LLM doesn't start to say something it shouldn't be saying. It stay within its role definition. In the example of MOIOB, one guardrail we put in is that we want to make sure that we don't provide financial financial otherwise.

We may give valuable insight to the customer, but there's always a fine line there between valuable insight and the financial advice. And we tuned our guardrails repeatedly. We use good evaluation dataset to guide the tuning of the guardrails, and we use a combinational hybrid approach by combining the off off the shelf guardrail. In this case, it's the Adebas guardrail we have, and tuning of that Adabas guardrails, as well as some of the in product self build checks to make sure that the LM are saying what it should be saying and not touching the area like superannuation product recommendation, for example, not touching the area like giving actionable financial suggestion, for example.

Those are the defenses we put in place to make sure that we pass the security hurdle. Because the underlying foundational model we are using, it's probabilistic. However, our approach to the safety, it can't be. We need to ensure our user, this is a safe product to use. We need to gain the confidence from the leadership team that this is a safe product to launch.

One thing we start to do from day one is building a good EOL data set. We gather the data from the user on the actual question they are asking and we pass those question to our friend of accounting accountant and bookkeepers. We ask them one simple question.

If your client ask you the same question, what would your answer be? And using our product, how would you get to that answer? What's the steps? Show us, please. By using those information, we use the steps from our subject matter expert to devise the workflow of the agentic agent.

We use their golden answer as the north star and compare our agent answer against it, so that we can constantly iterate and improve on this. As you can imagine, the evaluation data set when building the product, we never built anything or everything from scratch.

We have to work with the existing software stacks. One of the sources we need to work with and the team we need to work with is the Ledger Data API team. That's the interface we get a lot of the source data from. Most of the time, it worked fine.

We get the answer from the data by two calling, MCP maybe, and then massage the data from it and give it back to the user. Happy story. But sometimes, we met a challenge there. When a user, after several round of interaction with the agent, especially asking data related question and start to ask a question of, please tell me more. Very vague questions.

And the agent decided to becoming in into a detective mode, kind of like doing a deep research, start to ask more than 10 APIs and get all the information they can. The agent definitely does its own hard work, and I respect that. But this caused potentially unnecessary delay to the user, which reduce the usability of the product, and they cause unnecessary load to the other part of the system, which the organization may not be ready for.

So this is a challenge we need to solve by partnering with the domain team together and solving the problem with the communication we have with them, with the transparency on what we are trying to do, and what's the problem we are facing. Can you help us? Of course, it's never a one team's job to to optimize on this one. It's a joint effort.

As the agent team, we put the right harness to the agent. We limit the depths of the reasoning loops. We make sure that the agent respond in a reasonable time frame. And on the other hand, we enabled the domain team, the API team to help us so that they can optimize the query behind the API. They can optimize the performance and make sure that the p 95, the p 99 response time of those APIs is within the reasonable time frame.

By doing that, all the harness work and all the optimization work, we greatly optimize the performance and the overall p 50 response time for the agent. That's a win win story within the organization. While we are doing all of that, the optimization of the agent, the evaluation framework, the coding assistant tools like cursor, Cloud Code is definitely a false multiplier.

When the team is powered with the right tool and the best practices of using those tools, they can ship things much more faster than before. One of the examples we have is multiple front end stacks we have due to the interface we need to ship the chat feature to. Sometimes we do need to duplicate the code in multiple interfaces.

And the coding sinking agent we have is definitely helping us to greatly accelerate that process, reducing the days of work into hours of work, and the the team member essentially become the reviewer of that process. Same thing happened when we are trying to rearchitecture the agent.

Our colleague actually here is at a conference as well. He led a charge of using SPECKit to rearchitecture the version one of the agent using lan chain framework and can only answer one single content intent at the same time to deep agent framework where multiple intent can be answered at the same time.

And that came out as a result of a successful spike, and now it's in its pathway for the production at this moment. We are actively trying to integrate that part of the code into the system and ship it to the production environment. I mentioned spike. So that's something we really like to do.

We always have the mindset of bias to action. We enable the team to action, to make decision, especially if it's like a team wide decision where they can own it, they can act on it, and they can find good result from the actions. One good things about doing a spike is that if we time box it, there won't be too much to lose even if the result is negative. Negative finding is a good finding.

If it turned out to be successful, then there is always a pathway to production in a team. So a takeaway I want you to have is that next time when an engineer in your team is asking for a spike of trying a new feature or experimenting something new, I think probably that's a good idea, and we should allow them to try the idea and see what they can come up with.

Hopefully, you find my sharing useful. This is Michael, and I'm hoping you are enjoying the conference. And I'm always love to share more about the story, about the learnings, and I'm keen to learn from all of you as well. Feel free to get in touch in the conference, and there are exciting opportunities at MLB as well.

Thank you so much.