Thank you everyone for joining this talk. As he said, my name is Muhammad Ali. I have a really good name to have. I'm from, I look after Langfuse, which is an LLM observability platform. How many here have heard about LengthView so I kind of get an idea? Right. And how many used, any type of LLM observability, platform today? Alright. Cool.

And how many are actively building agents? And how many of you think that it's hard? Oh, okay. It's been a few months. Yeah. Cool. Alright. So definitely, I'm on the same camp because it is it is hard. Sometimes you feel like you are making progress. Sometime you feel like you are running in circles.

And that's basically by design because circling or iterating over the AI application is exactly how you improve it. And this is the topic for today. Right. So the way I see it, as you were talking about circles, I feel like AI engineers, when they are working, are more like hamsters. We are running in circles without with every iteration where we're getting some feedback from our users, and we find out things that aren't working, then these things that are not working.

We collect that feedback, and we improve, and we ship it again. And that's basically the cycle that needs to happen over and over again. And it's basically how tight is your feedback loop is what makes a whole lot of difference in how fast you can improve your system and and turn that hamster wheel into kinda like a flywheel. What is important here to know is, like, you really need to gauge on if you're not regressing, and that's a very critical point. As you are circling through or you're iterating through your application, you need to be making sure that you're making progress forward and you're not regressing, and that's what we will talk about today.

So I'll describe it how actually I have seen in the wild it works. And of the time, the way we start is we have our clock code, we start typing our ideas or whatever the requirements we have, Clot start building, and we provide instructions and the prompts. With the help of Clot, we actually build that and embed that into our agent.

We test it out with our, you know, what we think are reasonable set of inputs, and then we get good responses from our agents. We feel like it's good to go. It's probably 70% there, and then you can run a pilot or internal user. As soon as you ship it out, that's when the reality hits. Basically, your users are asking for things to the agent that you probably have not never heard, never thought before, or things that you have not probably even scoped in your requirement.

And if you haven't got your observability or visibility into these applications right, you probably are flying blind. If you are using a traditional application observability or APM tool, you probably are still running pretty much blind because these tools like your Datadogs or New Relics of the world, they would be good at telling you the health or operations of the application.

For example, the latency looks good, but your output of the LLM is just rubbish. That's not what they are going to do. What you need is a purpose built LLM observability platform that tells you exactly how what was the input from the user. So you're you're meaningfully looking into the content. See the LLM APM tools would not look into the content of the applications. So LLM tools are designed to look into the content of this application. What was the input from the user?

What step did the agent take through its decisioning and you know, in in in using its tools. What are the tokens that were spent? And and also how the user session evolved. That is what the LLM tools will offer you.

Length use is one such tool. And of course, there could be other scenarios when you're shipping your application in blind, it could be cost blowouts or quality degradations that could happen. Now, when you embed or when you start using a tool like LankFuse, you get full visibility into the traces and the steps, and as I said, how the agent is flowing through the system. This is where you start getting ideas about how you can actually tighten this thing.

So you take that feedback and you create what we call evals. Evals are basically unit tests. They test some inputs, and then they have a defined corrected correct outputs. So, essentially, these are unit tests that validate our agents working. Agents, emails need datasets to work with.

And how do you get those datasets? In the beginning, you can use, an LLM like, you know, cord or anything that can synthetically generate these datasets. You can potentially, ask an SME in your organization to give you a good set of inputs or questions that users are going to ask to your agent, and what should be the agent's response at the correct output. And when you have these observability tool in place, you can gather the traces from real life input data from the users, and you can bring that back into the datasets as well.

And when you are bringing those datasets back into the evals, one thing you could potentially do is have a human in the loop, which is what also Lankviews offers you through the concept of queues and annotations. So you feed that data into queues, human reviews, they annotate, and whatever they think is worthy of going back, worthy of being part of the offline evals, you can actually feed your datasets from there as well.

Now when you launch like this, you have somewhat better confidence in your applications. You have a good visibility into the traces, how your applications are working, what the cost looks like, and then you have a mechanism to create data sets and evals. Now you start addressing some of those long tails or the edge cases that, you know, break your agents.

So and and you could be experiencing things like silent decay of quality, which is if you have a poisonous history of of, you know, in context, then agents might not behave. So you start addressing the low hanging or I wouldn't say low hanging fruit, hard problems that, you know, in the second iteration. Right, so that's where we have normally seen online evals would come into the picture. Online evals are evals that run on the live data. As your users are talking to the agents, those datasets are then run through the evals online, either through LLM as a judge or via your applications.

So your applications tool like Langfuse offers SDKs so you can potentially capture the flow of the agent. For example, agent was supposed to take on average 10 iterations to get to the answer, but it's taking 100, right? So now these things can be captured via the SDK and sent back to LangFuse so that these traces can be part of your datasets so you can tighten the prompt later.

Now there are a couple of types of evals that are recommended. One is the capability evals, which is basically saying what is agent capable of doing. And the second type is more regression. So whatever the capability that you have developed, you don't want the agent to lose that. So you don't want to regress. So your regression needs to be really tight, whereas your capability can start at the low stage and then you can climb up the hill from there. But you absolutely need to make sure that you don't backslide with the regression.

Now who here has heard of a Carpathi loop self improvement or recursive self improvement. Now when you have these feedback loop created through these observability tools, LLM observability tool, you could potentially get your agents to improve your prompts, your instructions of their agents to improve itself.

So when you have these evals in place, which are scoring your data, your metrics are defined, you can have the agents look into the queues instead of having a human looking into the queue, as I was saying before. With LankFuse, you could potentially build that whole chain, you can automate the whole cycle. So as your users are interacting with your agent or application, you capture those traces and you run online evals, as I was saying before.

Online evals is LLM as a judge, or as I said, your application could also be running evals while it's operating. Now when you capture those scores and there are things which are scored low, you can send them to annotation queues or you can send them to queues where an agent is looking at.

And this agent can potentially pick that up, these queues, these datasets from there, and then start improving the prompts and run offline eval on the same thing to see if those regressions or if those inputs or inputs that has drifted your online evals actually are fixed.

So essentially, we are getting an agent to look at those queues, fixing the prompt, updating the prompts, and then giving it back to the human for a review. So instead of you or as a developer, we are improving the prompts. We get the agents to look at it and improve the prompts by ourselves. And then agent human, we can be at the at the very end as a gateway to potentially approve the changes that agent's making. Now what tools you're going to need to establish such a mechanism And what Langfuse offers here as well is the first thing is the skills.

Skills would be two types. One is how your application works, the SME knowledge, so you have the domain knowledge skills. And then you also have this Langfuse skills that are available online. So you can get your quad to install those skills and know what are the best practices, what are the different concepts and features in Langfuse. That's the Langfuse skills.

Then Langfuse offer MCP. So when users or the agents are going to retrieve the data from those queues, it can use MCP to retrieve the data. It can use MCP to look into every trace and every step of the agents working. And it also has access to the prompt repository inside LangFuse.

So you don't code your prompts inside your application. You have them in LangFuse, version inside Langfuse. So agent is going to change the prompt, run the offline eval, score that, and see what works. In fact, Langfuse offers what is called experiments. So agents can potentially run three or four experiments together, making tweaks to the same prompt in different ways and check which version of the prompt has the best score.

All of this can be done automatically using the CLI. So CLI what a length of CLI allows you to run the experiments, which are basically offline evals. So it takes this data set, the item that we were collecting online or offline with the SME, feed that into those tests or eval harness and eval harness will run and Langfuse will have the new scores.

It will compare the scores and it will say whether it's a go or not. So this whole thing can be fully automated. Now besides that self improvement, what are the features that LangFuse as a tool offers? It offers you, like as I said, Traces is the basic fundamental concept in there. So you get like a full detailed view of how your agent is progressing through the tool chain and the whole workflow.

It captures the cost. So Langfuse comes out of the box with knowledge of the pricing of many popular models. So if you are using Gemini, if you're using Claude, it will already know how many tokens it has spent and what would be the cost of single agent call. It tracks users in session so you can actually see whole you can see all the traces inside a single session because there conversational agents that can drift as the conversation progresses.

That will tell you how to improve those session level sort of problems that you have. We can score live, we can detect the drifts. It's got a really powerful search engine built into it so you can really slice and dice your data. And then, of course, you have the dashboards. From the improvement layer, of course, the first beta is the datasets.

It gives you the ability to create the datasets and version the datasets. So you can run different datasets and different versions of datasets in there. Then you have annotation queues. That annotation queues can be assigned to humans so they can go back and review what is working, what is not working, or something that needs to be part of the offline evals.

Now, it's got LLM as a judge, so online scoring is also available. As I said, there are experiment support, which are eval runs. You can run AB testing. So prompt A, prompt B, run the same experiment on both the prompts, and you'll get the different scores, and you can decide which prompt is better. Another bit about Langfuse is it's fully open source.

So you can download it on your machine and start using it. It's built on top of ClickHouse. ClickHouse has acquired Langfuse, but it's continue the way it's structured so the country Langfuse is going to continue to operate as an independent sort of entity, which means it will continue to push things into open source. There's nothing changed in there with the ClickHouse acquisition.

In fact, that's the whole reason for Lancaste's popularity that they build in public. So they first ship into public and then offer it as a managed service. We do have a managed service, which is the Langfuse Cloud. Langfuse Cloud offers SSO and other enterprise grade features that you might require, mainly around security and compliance, But all the features that you have the enterprise will also be available in the open source.

It is model and framework agnostic. So you could be potentially building TrioAI Pydantic or any other framework. I built on Maestra and then there we can use that. It works with any type of language, so we have SDKs for TypeScript and Python, and then Go and other languages are also community supported. So the whole idea of, the point I was trying to make was it's basically, it's not how fast you move.

It's basically how fast you take that feedback and embed that into that loop. That is going to make a difference between progress or staying stuck in the same place. So these are like a couple of QR codes for you to sort of learn. So there's the Langfuse Academy that teaches all the concepts And there's a QR code for LengthFuse Cloud if you were to use that. And if you are sticking around, I might be able to show you what LengthFuse looks like.

So here is the UI for LengthFuse. You can see I have traces here. I've got an agent and it shows you exactly how the agent flow worked in there. You have prompts managed in here. So all my application prompts are managed in there and you can technically see the traces and which prompt it actually started from so you can get the tags from the prompt.

We do have evaluations, I'm running a bunch of online evals over here. I'll show you how it actually works. So this is an application which is a simple application that teaches kids maths, science and whatnot. And there's an agent down on the right, you can see it greeted the child. And this application is built using a master SDK.

And I can go and attempt an activity and answer. If I do, if I get it right, the agent tells me what's going on. Or I can start interacting with it. So I say, I want to learn fractions. And then it's going to look up the skill list that it has for the activities and it's probably going to offer the answer on that, hopefully.

Come on. And that's exactly what we want to see in what's happening in Langfuse. Alright. So if I go here, this is this is what is happening. Let me zoom in. As you can see, the whole I asked it the question.

It probably isn't the first one, is have to see what's going on. So this is when you start. This is the system prompt that says you are a tutor, you have to greet the child and basically this is where the greeting came from. But let's see I've got the, if we got that answer. So here you go, it came back with some activities.

So if I click, it will basically take me to the activity which doesn't exist. That is a problem that I need to solve. But the idea is you can actually have all the traces that tells you exactly what Yaro is doing as an agent. Here you can see I ask, I want to learn fractions and it went on and fetched something and it came out with the answer. Here what is important to know is the online emails.

Every single prompt that user provides, I am actually taking the output of the LLM through the safety check online and this is what is scoring my output, whether it's safe, it's helpful or not. And then the annotated or the prompts or the traces that are low scored, I take that back into my datasets and then run the evals again. So that's the whole cycle.

We do have a booth downstairs in the expo. I can show you the whole cycle where we take that input from this from the scored LLM prompt and take that feedback back into the evals. So thank you for attending the talk. And again, if you're looking for links, here's Awesome.