The Application Layer Is the New Research Lab

Thanks. A few years ago, the application layer produced the the application layer consumed all the research which was produced from a research lab. Now it is the other way around. The application layer actually produces a lot of research signals which can be utilized to improve this whole system.

This wasn't the case a few years ago and now it is totally the case in a lot of these vertical applications if you look around Cursor or Codex or all these other AI companies are are doing that. So we'll be talking about how this has happened, why this has happened and how can we utilize this in other verticals which are not coding related, which are very general purpose in the economy.

You probably might be aware of these statistics that ninety five percent of the enterprise Gen AI projects fail. I'm not here to defend these statistics, but I'm I'll be talking about the diagnosis of these statistics. It was mentioned that it is not the models which are actually causing this failure, it is the structure around these models in the organization, the way we consume these models because of which we see all of these failures.

And when we talk about the structure, you might be familiar with this kind of diagram. So an AI agent is actually not a single thing. It is composed of many small little components associated with it and most of the teams in enterprises, so most of the enterprises treat AI agent as a traditional software.

And in principle, it is not a traditional software. It is much more different than a traditional software. So people working in application layer might be familiar with these different components. For example, an AI agentic system is one of the main component of AI agentic system is the model itself.

But the problem is the folks working in the AI vertical do not own that model. They simply rent it from a foundational lab. Then there are tools and APIs. These tools and APIs were usually built for classical software era. They are not built for agentic software era.

They were consumed differently. They were built differently, but now they are consumed by AI agents and there is a mismatch between the way these tools were built and the way the AI genetic systems consume them. This this is one problem. And the other problem is that people who are working in the in the vertical, they do not own these tools and APIs.

These are owned by legacy teams. Then we have the retrieval. Maybe a data team owns the retrieval system. The schemas, the guardrails, the memory, the routing, all these components which make up the agentic system are most of the time owned by other teams, not by the teams who are actually building the agentic system in the application.

Out of all these eight components, there is one which is prompt. We just own that. And then we tune that. So there is one lever in our hand out of all these eight lever to drive this whole system from end to end and deliver the results. And that is the problem. That is the structural problem we have.

The the gap so the as I mentioned, we have only one dimension in our hands. And if we move across this single dimension, the way this whole agentic universe is set up, that we wouldn't be able to test the desired result. So there's just one dimension.

We will move on this line and the actual true desired output lies somewhere in the middle. In order to go there, we have to cross these other dimensions as well. We will have to tune in a joint way to retrieval, maybe memory, the prompt itself, tool calls, all these other components.

And we cannot do that. So the only the problem is that we only have this one dimension in our hand in most of the cases. So the geometry is the problem and that is the obstacle in our way to go to a true desired output in a GenTex system. And that actually makes the system fail most of the time.

The agentic behavior is actually joint. So you might think if I have an agentic system and I change the prompt, then I will be able to drive it into a certain direction. This is, I would say, partially true and partially not true. It feels like when you drive the system through just a prompt, you can give it certain direction and it can move to that direction. But over the time, you will realize that over the vast space of prompts, you will again come across another different set of problems and you will be stuck in this prompt optimization loop. The reason for that is that we don't we do not understand most of the time that this whole behavior is actually a joint. Behavior of the agentic system does not depend on just a prompt individually, it does not depend on the tool calls, it does not depend on the retrieval.

It depends on all these things together in a joint way, and that's the mindset shift. Usually, people who who do not have a training in machine learning struggle to understand this joint distribution and probabilities. Even though we have come to this agentic era, the underlining substrate is still machine learning and and probabilistic modeling.

Most of the organization charts assume that this is how agentic systems work. So the probability of the output of an agentic system is actually the is equal to the probability of the prompt multiplied by probability of the tools and retrieval. So you combine all these things together independently, independent knobs and then you get your desired results.

So you can tune these knobs independently. That's not true. You cannot tune these knobs independently. You have to tune them collectively in a joint manner exactly like we do in any machine learning model. So the way it actually is, the true representation is that the behavior lives in these cross terms.

It does not live in these individual terms. You can see the on the diagram, the off diagonal cells are brightened, which is actually giving you the direction that it is the actual interaction where the off diagonal cells tell you the interaction actually lives in the cross section of these terms, not in the individual terms. You have to tune the whole joint or you don't tune anything. So that's the main thing.

One of the other thing is we have been designing these softwares since seventy years based on contracts. We used to chain these different components of the software and there was a promise at the intersection of each of these component. And the promise was that the output and input of these systems were in coherence with each other, and that's why the whole system worked. So we were we were having contracts, but the contracts are gone now. The contracts does not work now.

We can have a structural contract, but we would not have a semantic contract in terms of agentic era. So on the right side, can see the three components, model, tools and retrieval. And for each of these three components to work together, there should be some sort of a contract and the contract we don't have the contract using classical software era.

So everything actually interacts. And again, you have to tune the whole joint rather than tuning a single box. We used to compose the contracts and now the only thing we have is the joint distribution and we will have to tune the whole joint distribution. But the problem still exists, you can say, and that is even though we understand that we have the joint distribution now, and that's a probabilistic thinking, and we should bring that into the product, but we don't own a lot of these components which compose this joint distribution. The model is actually rented.

The APIs are legacy owned by some other team and retrieval index, let's say, is owned by a data team. So how does that solve the problem? I would argue that we don't need to own these components. We should own the measurement layer over these components. So for example, we don't own the legacy API.

Right? But we can own the eval and adapter around that legacy API. We don't own the foundational model, but we can design our own judges and then we can design our own harnesses. We don't own the retrieval index, but we can create scenario benchmarks or evaluation data sets which we can optimize and then a re ranker as well. We buy our guardrails, let's say, from Avenda.

That's all right. We can own the calibration layer around that guardrail. So the function we we don't own these each components. These all of these components stay where they are. We own the function over them, the shell over them, or in software engineering, you can call a wrapper over them or a logic over them. And that's where the research lab is formed.

Somebody who understands this measurement system around these small components are your research engineers, applied scientists who sits with your software engineers in your product and understand how these systems behave and how to control them and how to apply their joint probability distribution training from their ML era into the product.

You probably know about classical ML. In old charts, we used to have ML teams, and ML team owned the models, and model just meant the weights. And there was a very fixed contract. You apply you give x and you predict y.

And based on different quarters, you will then retrain these models and your product is actually not similar to your training. So the product and the training team sits differently in different labs and they talk over maybe conference pages or through long feedback loops. In agentic era, this theta, which was just the model weights, it has now exploded. It has gone through all those all the surface of the product. Now nobody is the owner.

Right? The contract is gone. We have joint distribution and we can do the loop in days. The product whole surface is the training surface. There's a statistic that a lot of these AI projects fail, and the failure rate of AI projects is double as compared to the failure rate of non AI project.

And this is because of the leadership failure and the structural differences between organization that you have to train a model, then hand it off to another organization, and then they will tune it and give it back to you. This whole loop is broken, and that's something we need to fix rather than the model.

So you have to put these two mindset in the same room to create a research lab inside your application layer to improve your feedback loop. Your application engineer, software engineer, or your applied AI research, when they are in the same loop room, they will be able to drive two very important components. Your agentic product is actually a machine learning system, if you think about it. You have a system which serves real customers, and there is a tunable parameter to the system.

And if you have a machine learning model, you say, then you should have an objective function of that, and you want to optimize that objective function of your machine learning model. In agentic era as well, you have objective function, which is the usability for your users, the usefulness of your product for your users.

You can optimize that using all the parameters which are available to you. The difference is that the parameters which you can tune now are exploded, are so many. And you don't own most of these parameters, but you own a function over them and a wrapper over them.

This is composed of two main parts. One part is owned by the application layer or software engineers sitting in the application application layer. And the other part, which is a very new part, people might not be familiar with that, is the is the measurement part, which is usually was usually owned by researchers in the research lab, but then we have to bring those researchers into the application layer to make this whole thing work and improve the feedback loop while sitting inside the application layer.

So two one theta, one vector, and two disciplines working together, sitting in the application layer and optimizing this whole loop. Before, we used to have these research labs, upstream research labs, and they will feed into the application layer and the application layer will consume the research. The product surface was not the training surface, but currently, application layer is actually the research layer now.

Research lives in the application layer. The theta has been exploded. You can get a lot of insights from the application layer, which means that research is the app. Product surface, the whole product surface is now your training surface. You can extract all these traces, all that insights from your product and retrain your whole system and tune your theta, which is, again, very expanded version.

You might remember the statistics from the previous slide. Eighty five percent failed. The five percent which succeeded are in this category. The five percent are having research labs sitting in the application layer with software engineers, with product engineers. They are not separate. They are single entities sitting in one place.

There is it's not some new concept. We know about cursor. Cursor went from zero to 2,000,000,000 in in in three years' time in annual recurring revenue. And the playbook they used can totally be generalized. One of the good thing for cursor that their domain is coding, which is verifiable uniquely verifiable, but then rest of the economy is not uniquely verifiable. But that does not mean that we cannot apply the same principles to these other domains.

This means that there is a great opportunity for us. We can apply the same principles, the same artifacts like the judges, scenario, flywheel, hardnesses, evaluations, all that feedback loop, we can apply that to to any non verifiable domain. It is challenging, but with the challenge, there is a great opportunity as well and whoever applies that will win in this specific category.

So the product creates traces. Traces goes to then judges, and judges then turn the flywheel. Flywheel updates the theta, which is the whole set of parameters I showed you before, and then that theta produces the product as well. So this whole this loop runs.

Whoever builds this loop is going to win. Whoever loses on this loop is going to lose. We have seen that in many, many cases. If you and your competitor let's say if you switch the model and you have this loop in your application layer and your competitor does not have this loop in your application, in their application layer, they will have to start from zero.

You won't have to start from zero. You will have a flywheel with you to help you further. Thanks so much.