Multi-Model Collaboration with Claude Code: How to Measure What Actually Works

Yeah. Hello, everyone. So I had a joke in the beginning of that conversation that a big screen with the letters, hey, Claude. Help me to make my slides quick in thirty minutes and easy. Unfortunately, that becomes true because I lost my slides. Haven't get it here in Sydney.

And that's why I have recovered. So we are missing a lot of pictures, but it doesn't matter because we have a lot of real evidence to show. So what's happening? Today, I wake up woke up in the morning. And the third thing I had on a X or Twitter, how I still call that, is a lot of people talking about that new feature in, quote, quote, dynamic mode or dynamic agents.

And I was, like, literally bombarded with that. And, like, everyone saying, hey. That's a new thing which makes you better. Like, I already get a couple of messages from my guys from the team who are saying, yeah, that's going to change our lives. That's a very big thing. And it's not the first time when it happened.

Like, last year, I just literally was living in that in that mode when everyone was saying, hey. Yes. New week, new tool, new model, new stuff, everything changing. And it's happening very fast. And even, like, my partner start to suggest me some changes. My kids start to suggest me some new tools.

My cats. And, like, that's getting crazy. And, like, I remember I was skeptical. A lot of us were skeptical in the beginning. But then you see how everyone around started to say, oh my god. Oh my god. That's changing my life. And the small voice in your head starting to think, literally, maybe that's really helpful. Maybe I will find a place and that will do the job. Little spoiler or not, like, one year ago, I was sure that probably in the future, I'm going to be replaced by AI. One year passed, and I have 10 times more jobs than I have one year ago.

So doesn't work yet. And when you think about that new stuff, you usually, what do you do? You get it, install it, run-in your regular environment, Cloud Code or whatever you're using. And it's called the tools to do the stuff, produce the job. And you're happy and say, yeah, that's working.

I see, like, new agents getting new tools calling, like, new output. And you believe that your work improved. Who done it last year, like, during the last year? Who installed a new tool, Trident? After one run, it start it's manual, like, main mainstream.

Yeah. One person. Oh my god. So skeptical. I don't like all of you because this audience is full of AI advocates or enthusiasts, and this is how we came up here. Like, we tried the tools. We love them. And the problem is it's not engineering.

It's not engineering approach to apply tools. And at Metaben and 10x Labs, we deliver the products to our customers. So we literally yeah, this was my joke, we literally brought these AI products to our customer, and they have their own customers.

And we need to convince them that these new AI tools literally making sense. They literally deliver the value. And when we start to talk to the industry, talk to the guys like you, other engineers, other company, interesting stuff happens. Nobody know how to measure the outcome.

And we started to play a lot. We started to play with different models. They produce different results, different harnesses, trying to combine them together, trying to find what's worked, what's not, trying to extract the value, trying to understand the patterns, doing all that engineering stuff we're usually doing. And we found a couple of interesting outcomes.

Now this is literally my screenshot from morning. Like, you see on the screenshot, it's June 3. It's yesterday post, but I saw them today on my feed. It's highlighted. Everyone talking about that harness. And the third thing I did in the morning was actually let me show it to you.

So we are yeah. The third thing which I did in the morning was that one. And all I need to do sorry.

And what I need to do is just write something like models, whatever. What do we have? We have and we have test data. We have predefined SW benchmark.

First, we're going to call without tools and other with these new arguments. So this is done. And then what I do Yeah. I'm fighting focus with agent right now. It's dealing my focus.

Focus. And then what I've done is just do my bench run. Obviously, it's not running now, but that's how my morning started. And oh, not this window. And the result I get was interesting.

And it's not the first time I'm running the harness. Like, we are doing it a lot. And the results we're getting, it's not result of my personally. It's result of a lot of people and the tools we're using, approaches we're using. That's sixteen years, and we have 70 developers. And we're working with clients, providing them the tools.

It's 500 engineers. And for tools, we're already in production, which already mentioned, Clodish, CloudMem, CloudApp, it's 2,000,000 users right now. And we get all that experience, all that knowledge, and start to share that with the community. So you don't have to spend your time to doing that or join our experiments. And we found very interesting patterns.

So the first of all is that models are biased. It's pretty common when people start to get models as humans, as a person. Sometimes it's fun. Sometimes it's not useful. Sometimes it's useful. So because they have characters, they behave like human, and they're very, very smart.

And you sometimes forget that you're talking to the model, but there is carriers. So they are biased. They are biased, and that bias is multilevel. So level number one is species. Like, some almost all models love Python and Shellscript. And this is how they were trained initially by these ML engineers.

And we couldn't do anything. If you remember and you try to ask agent to write something and doesn't provide specific requirements for language, it's 90% is going to use Python. No any specific reason for that. The same thing is all that AWS benchmarking. They show very good results for Python, and people started saying, hey, because models love and know Python better.

No. Because all benchmark we have initially had was Python. And their model providers start to train the models to get a good score on that benchmarks. So that's multilevel human agent bias bring us to that state that everyone believes that models love Python, and models believe they love Pytons as well. The homeland, that's an interesting one.

It's about politics. So it was very much research that shows that Western models are more left winded, and Chinese model right wing it in politics. One exception is rock. The family, it's interesting because if you ask GPD open source, like, 120,000,000,000 parameter model to judge output of GPD 5.5, it's gonna be biased.

If you ask to judge another family of model, it's not. So it's they not only judge to their, like, the same model. They judge to the same manufacturer of the models. It's pretty big bias across. They're individuals. So we yeah.

Wait a sec. Oh, no. We have started to test it and find that Gemini is the only model which can really useful assess your UI, understand there are colors, understand the padding. Still not ideal. You have to use some kind of predefined algorithm to understand the boxes, but that is.

The yeah. The the next model is OPUS. It's just, in general, the best model for coding, understanding orchestration, agentic framework, and writing text documentation. That's actually the revolution nobody expected. One entity can write amazing code and write amazing documentation.

Never happened in our industry before. And GPT, that's pretty interesting model. If we're talking about code, that's or oppose on Sonnet models, that's extremely bad at writing React. Js code. Like, literally, we witnessed this. It's, like, stuck for days trying to fight with useEffect.

And I would not judge it for that because our guys from CloudFire could approve my statement that just a year, a couple of years ago, because of humans' mistake using useEffect, half of Internet was dead. So they DOSed they implemented that mistake in the panel, which DDoSed internal network which had no protection.

Half of internet is dead. So even that brilliant developers couldn't handle useEffect. What are you expecting from the model? And gpd is extremely good. Like, when I mentioned that code stacked, gpd did it in one run. And every time when I'm stuck with front end development, I always refer to gpd model.

And here we are. That's why and, like, it's very, very inconvenient to switch across Gemini, CLI, especially when you have plug in skills supported and ensure that all the plug ins and skill across your team are the same. And agent doesn't fight to each other because they have a different knowledge. That's why we created a team plug in, which run-in, quote, quote, all models you want.

So they roast each other. They judge each other. And the best value you can get is rejection. So when they reject, then you get a best outcome when the model start to get the result. So consensus is cheap, and disagreement bringing the value as then you have the stuff. One of the examples that we have indexing, and that's indexing like, sorry.

That's indexing is actually not just text indexing. It's getting AST tree, get your function, and put it in index. So every time, model not gonna go and do grepping all your source code to understand where is YES or code and how it works. And to make it work, you have to put description of this function on top.

And to make that description, you need to model. And funny stuff, we, like, run 20 models and then run another model to judge how good is that description. And they wrote it like, yeah, that's good, like 80%, 90%. But when we ask them to use that index to retrieve the knowledge, the result was absolutely different.

That's why don't ask the model, have you done the job well? Always try to find practical way to do that. Validate that. That's the statement. And here we come. We are, as an industry, become religion. Become religion, what I mean is that we're not using engineering approaches anymore. If we compare, like, religion and engineering, in engineering, you need you don't believe. You have to prove.

And in religion, you don't need to prove. You just believe. In engineering, you have to trust your evaluation framework and always improve that, like, if your numbers you're getting from checking the stuff are real. In religion, you don't need that. And in engineering, you don't have to trust anyone.

The medicine get a very big what is that? Learn very painful lesson in nineteenth century when there are famous people dictated the way how we should be treated in medicine until we invented double blinded researches. So here, don't trust my favorite guys who build in the industry, even Mila Jovovich. Don't trust her memory model is the best one for agents. Validate that.

So that is the point. Don't fall to the religion. Use engineering approach. Always try to find the way how to measure the outcome and ensure that your way of measure is really based on real evidence. And there is a lot of levels of that. So the first lowest one is just, look.

If it's working, producing result in its website, you can click through. Then you write your own test, just a regular predefined test. Then you just get your harness and do end to end testing, running the quote code with different parameters, and validate the outcome. The third level is when you start to use the tools like PromptoMaster, AWS, to make the bigger approaches bigger.

And the fourth level is real time benchmark when the model is running in the cloud, and you need to validate what is produced. And that's one of example on the left, the thing which, in the morning, validated that new I don't know. Could I improve that? Yeah. It's validated the new approaches that we have done.

Not not new approaches. This new, quote, quote. And I I found the outcome of that that it's just getting twice more tokens. And from my test, from it SWE benchmarking, it haven't solved any any better, like, just a little improvement to one point, which nothing. One thing.

Yeah. So that's technically, that's it. As my message to you as people who are shaping the industry, please put your efforts and intention to grab and build the toolset which we can use and framework we can use to validate the undeterministic outcome from the models.

Thank you.