Your AI Can’t Engineer (Yet)

Boris, thank you so much. Hi, everyone. It's very nice to be here. I'm actually surprised how many of us are here. Very nice. Good way of sticking with it. Being towards the end of the day has the benefits of hindsight. A lot of the things that I will discuss, I think a lot of the threads have been pulled already in other presentations.

I hope to offer a new kind of dimension. And that dimension is that aside of the provocative, slightly provocative title, it's also a bit underspecified. You don't know this. I will explain. So I lead an AI team at the design and engineering consultancy called Orecon. We work across the Asia Pacific. And we have about 4,000 engineers. None of them does software.

So we build bridges, tunnels, harbors, airports, big buildings, data centers, anything around you. Right? We we do. So when I say your AI can't engineer yet, I mean it in my own literal sense. It's something that if any of us here have ever experienced that stepping outside of the distribution of the model, you see it very quickly. It's palpable that the performance changes.

And in fact, in my experience, at least in engineering, it changes quite radically. So most of this talk today is sort of a lost generation of my work around this problem and how we try to adjust for it. But what is engineering? What I'm trying to say, I will show you a couple of examples. This is an easy example.

It's a simple engineering trust. Trust me when I say this is easy. Just the fact that you can see it in numbers, this makes it almost a child's play. Right? The actual problems are very much harder than this. All the information is given. The formula is given, everything. And yet the Grok 4.3, a quite good model, I would say, can really solve it.

Right? Instead of using the formula, it goes back into own pre training knowledge perhaps, takes its own formula, very confidently gives a result. It's the wrong number. Probably some foundation falls down. And so so this is at the core of all of engineering. Right? Like solutions to problems are always contextually related to methods, designs, standards, regions, locations, and infinite amount of things.

Not infinite, but quite large. Another example. In this case, it's an even simpler one. We're trying to sort of like calculate clearance. We have the size of the street and some walking speed. To make it a bit difficult, we adjusted a lever here. Right? We didn't give the walking speed to the model. And the model, of course, did the same thing.

It went in some knowledge, some pre training knowledge, some old knowledge, and it came up with an answer. The answer is wrong. And the problem is not here that the answer is wrong. There is no deception. There is no bad arithmetic. Right? It's rather that in a place of uncertainty, in a space of uncertainty, the model very confidently did the task.

And it did it different at all three times that was tested. Right? And I call this, and it might be a misnomer, a lack of a sort of metacognitive capacity of these models. Right? They don't really know. They they don't really adjust their behavior. And I have actually evidence of that that I can show. The talk is very small.

They can't they don't really adjust their behavior during uncertainty and during complexity in different scenarios. They all attack the problem the same way. And this is, yeah, this is a sign of of the problem I'm I'm going to talk about. Okay. To solve this to work towards a solution, I decided to start building a benchmark. The whole talk is about the benchmark.

It's the AC bench. It's actually available today. And this benchmark is what I I like to call sort of a new generation of AI native benchmarks. This means that it's not a library of solutions. Usually, benchmarks are a library of solutions. You can find them on Hugging Face, download, and run them. It's actually AI by AI native.

I mean that it allows us to create and evaluate data sets, even post trained models with a few lines of code. And as I was building this benchmark and as I was going from between my day job, which is harness engineering for actual real world engineering problems, into building a benchmark, I sort of realized very quickly that we aren't really measuring models anymore.

We are literally building walls around them. Right? And and around every score, you know, in a benchmark, there is actually a whole runnable setup. Right? There are documents. There are tools. There are contracts. People spoke about this. Right? There are verifiers, traces, repair paths. All of this together is important. It kind of shifted how I was thinking.

I suddenly realized something that I will talk in a second, that I was already doing this work somehow by actually doing normal work, everyday work. And yet the first thing I did is we design and run a benchmark. So this is the first sort of pass. It's a very small and very simple first version of the benchmark.

There's no multi modality. There's no complexity. It's incredibly simple. And yet none of these models and yes, some frontier models are missing. It turns out this is quite hard to actually run large scale levels. But still, yet, none of these models saturate this very simple benchmark. Right? And this margin that we see from perfect and by the way, in my domain, perfect is a necessity.

This margin is exactly where you find liability and trust issues. Right? And it's a very difficult thing, especially when it comes to adoption of these technologies in fields that might be a bit more legacy than software engineering. So what we have to do is actually, due to this margin, we actually need to design trust into the task.

Right? We need to and we do this by adding these bindings around the score. Right? We for example, you might have units. You have standards. You have sources. You have contracts. You have reviews. Again, a lot of this was were touched upon in other other discussions. But these bounding bindings and this task world, essentially by building this world around the task, is how we design these links to things that are needed.

Right? And what is a task world, though? So it's actually a knot. And actually, this is the best design that I, not me, 5.5 in this case, could do of the idea in my mind. But actually, it's a knot. It's not a circle. What does a knot mean? In the philosophical sense, I don't know if you dabble in philosophy, a knot is a thing that is when one of these aspects actually embeds all the other aspects in it.

Right? So they're not simply connected. It means that if you are doing harness engineering, you're already doing everything else in the screen. If you aren't, you're not doing it right. Right? So if you're if you're a researcher that builds RL environments, you're already solving problems in the real world. You're just doing it in the lab. Right? And people have talked about this already.

If you're evaluating models, you already connected this to practice. Right? So so that's the idea. And it also means that capability is distributed. Yes. The model is very important, but the actual capability is distributed in this world, right, that we build. And, yeah, that's that's the capability that we observe. We can judge. We trust. We repair. That's how we build things.

So let's open one of these tasks. This is a very simple task, another simple task. It evaluates whether the fire alarm devices we would have on a circuit fit within the power supply, essentially, if your fire alarm is gonna start a fire. So even with this small even this small task has a structure, and you can see it clearly here.

It has some inputs. There is a calculation contract. Right? There is an answer, you know, schema that you need to follow. There is a verifier. Actually, we are kind of like in engineering, at least in this toy level, to have explicit verifiers. So you have a verifier that you can actually evaluate the performance. And you can you can get unexpected results.

And what ACBench does is it sort of like creates these tasks as as units, right, as sort of runnable, checkable engineering solutions, right, situations. That's that's what the task is. And just to give you a quick overview, it's meant to be a much bigger enterprise. I think we've mapped around 2,000 tasks, around for nine disciplines. Right now, there's just five disciplines in the initial release.

And so it's much more modest. Again, again, this will be hopefully crowdsourced through a community of peers. And we also are releasing a small evaluation across all these models that I showed. Again, modest general evaluation. I think 184 templates, so these are the problems, created five fifty two task worlds. So these are instances of these templates.

You can imagine procedural generating worlds from templates. And then running it through 15 different models for about 25,000 planned rollouts and 380,000,000 tokens. It's not it's not a lot, but it's a modest a modest output. And actually, this kind of data doesn't exist to my knowledge yet. At least our industry is not up to date with the evals.

ACBench, ACBench is designed in a way that every run will leave evidence. Right? And again, the same task here on the right. And this is very important because every other people talked about this. But in engineering, it's even more important to have this evidence because it's really important not just to understand the score. Like, the reward really doesn't tell you much, but actually what happens.

Right? How did the mistake happen? And in this example, 5.2 on the right, attempted to solve the same problem. It scored about 50%. You can see the reward there. Rectify just means that if it's called a 100, we don't take it into account. It's kind of like an average between the three runs. It failed into the fields.

You can see some information. And alongside, we have all the output, the conversation, the trajectory, very important. Right? And even behavioral classification, which means how did the actually, the models act while solving this, which is really important. So if we accept that capability is shared between the model and the world, so where does the score come from?

So when the score is low, what do we ask? I think we don't really ask which model is better. And know that in many domains, might. In many domains, we have tasks where we put a model in an environment where we need to run a billion classifications. Right? And all we care about is which one is better?

Let's calculate. In engineering, it's not the right question. The right question is which part of the world actually created the problem? Right? We need to really, in an engineering manner, understand the problem and fix it. So provenance and hidden causes are very important. That's where that's what what is hidden behind the score, behind the performance. And we really need provenance to understand failures.

Right? Because because, again, once the world is part of this task, like, a failure is not really automatically the model's fault. Right? It could be any number of things. You could have retrieval issues. Your answer could drift from what your information should be. For example, this task is different for different areas of the world. You might have completely errors around units or standards.

And if you're like me, you might have bugs in your verifier and output contracts that I found after the fact while building this benchmark. So the same zero can be many things. Right? And we really need to understand the causes to actually be able to repair and even like hill climb, as many said, in this domain that models cannot do perfectly yet.

So yeah, looking back at the same evidence, now we can do that. Right? We can say, okay. Two out of the four parameters fail. We know exactly which parameters are. And if we go in the trajectory, we can exactly find the the point where the model failed, understand why, how, and when, and even understand the action it took before while it while it made a mistake and after and try to rectify this.

And I and I think and I think the point I'm trying to make is obvious, and many made it. Right? Like agent traces, observability, very important. But it's not just that. What I'm trying to make is back to the task world definition is that this is not downstream of building benchmark. This is not downstream of harness engineering.

This is not downstream of anything. This is at the core of actually doing the task. So we need to stop thinking of these things as downstream. I run something. I build something. I'm evaluating it now. That evaluation needs to be embedded in everything that we do, as everything else should be. So it's, yeah, it's not downstream of the harness.

Okay. So now if we zoom out, we can start sort of understanding performance. Right? So if we run it across all these models, we get a picture. And it's hopefully a nice picture that you can see. You can quickly start getting some signals. Okay. This top reward band is pretty good. Right? 4.3 barely wins over 4.2. That's good. It's much more efficient.

The the bottom there is tokens than k 2.6, which is actually a very nice model. I I really like it. And, oh, and deep sig v four there is also very close. That's interesting. Right? But, again, the single reward hides other questions. Right? For example, we could ask a behavioral question of, like, how much work is it actually behind this this reward?

Right? So you can shift that from from the previous view to sort of, like, tokens and tool calls. Right? So, like, what is the actual behavior of the model while solving a problem? How does this change the landscape? Suddenly, we can see that, again, the Croc models are not just very good, but very concise operators. So again, if you're looking for a sort of efficient, high quality solution, which many people would, that's where you would go.

And then you can reshape this again, perhaps by cost, and you get your Pareto frontier. Right? So again, you keep asking different questions. And and one number is not telling the whole story. And in this case, for example, before flash might be very interesting. Right? Like, how come can it go can it do so well there on on the top of value performance and then being eight times cheaper than than the best model.

Right? I would caveat here that in engineering, I think it was discussed in other in other presentations, we we don't we usually have low volume, extremely high value. So we would generally go to the right here. But still, it's an option, and it might be an option at some parts of the solution. So my point here is that reward is not really something magical.

It's an engineering artifact, in this case, an actual engineering artifact. It has assumptions, tolerances, field bindings, blind spots. It has things that we need to uncover. And we need to create and audit these reward contracts. Right? We need to embed them again. They are not downstream of the task. When I build the hardness to solve a problem at work, I'm not thinking I eventually need an evaluator to understand how I do this work. I'm thinking that it's already part of the task that I'm doing. Right? And yeah, otherwise, the eval could be precise, but it's actually wrong, right, if we can really audit this.

Okay. And the next step, as I said, the AC bench, I hope to be an AI native sort of benchmark. So evaluation, when we structure it in this way, also becomes infrastructure. Right? What I mean by this is that that the run, the verified result becomes an audit label and a score and a data set somewhere that you can use downstream to fine tune.

And again, I will make the same point in a while. This fine tuning and the evaluating and training models on your outputs of your harness or evals, it's again part of the same thing. They are not downstream. We are not generating data and fine tuning as follow-up steps. All that should be designed already as you're solving a solution.

And if your solution is not geared in a way that allows fine tuning a model at the end, there is something lacking already, I think. Okay. And ACBench does this very nicely, I think, and quite easily. With five commands here, we create a dataset. We export it to Primelab. I don't know if you know it. It's amazing. And then we can train it and post train a model on the dataset.

That's it in five commands. And, actually, I just wanna I just wanna say here that it blows my mind. My, you know, my twenty twenty self trying to find key models on TPUs and JAKs would be absolutely mind blown by the five Python commands post training a model without touching anything. It's really incredible the times that we live in also at the at the time at the point of infrastructure that is available.

Anyways, the this way, the benchmark becomes more than a benchmark. Right? It's actually a living reusable engineering corpus. And that's the whole idea. It's more than a benchmark because all of these are not. They are one and the same. And it's also how we create, you know, the next training and the next round, and this is how we eventually hill climb.

But hill climbing should not be, again, downstream of what we do. It's part of what we do. That sort of self improvement has to be at the core of every task that we're doing. Okay. And here's an example of what I said. We fine tune very small model for 23.54 b. And, you know, very quickly, if the rollouts verify just an example.

This is not to say that we created a model that we can now put in production. But the point here is that fine tuning this model was not an afterthought. It was a part of the process from the beginning. The ability to fine tune on an eval and a collection of solutions is always part of it. And yeah, if our evals don't connect to real problems, for example, or RL environments or post training, then I think they're not complete.

And their utility is marginalized in some way. Okay. So by building these worlds, the evals, the RL environments, the data generation processes, and fine tuning specialized models. And remember, all this is a not. All on real world, your domain, ideally specific problems. I think that yet can start to move.

Right? And I think we can slowly move to a to a world where models can engineer in my sense of the term. Because while models are smart, they really don't get there on their own. Right? They not yet at least. They are trying now. Capability is something we engineer into these models. And I think it's up to all of us in our own domain expertise to do that.

So, yeah, just the last point. Everything is available. The site is up. This was developed with a lot of support from Moricon and our partners at Nomic AI. There is a hacking phase with all the rollouts if you want to fine tune models. There is a paper in archive. If you, for some reason, want to hear more of what I say, there is a the harness blog for my blog.

And thank you very much.