Slop is a standards problem

go.

Oh, that was not like a long, like, minute or anything. That's that's awesome. How are you all? How's the conference been? It's been pretty awesome, hey? So yeah, my talk is about slop. It's been touched on a couple of times for folks. Really kind of like conscious that you're all like getting to the end of the day, I don't want to get in the way of like getting to the drinks and stuff like that.

But just wanted to, like, say that it's been really inspiring being at the conference in the last couple of days. And I've I've been, like, a big adopter of Claude Code in, like, my private life. Like, I've had my thumb in the laptop, you know, with the way you wedge it open just to keep those agents running.

That's been me. I've seen a lot of you lot doing that as well. I've asked you a few questions when I've seen that going on. So I love it to bits. But that's like that's my alter ego. My day job, I've got concerns, right? I've got like big concerns. Can you guess what the concerns might be that I've got with the whole AI thing?

Deployed slop. Deployed slop. That's right. AI slop, right? I worry about this a lot. I worry about it because when I think of what it means for a software engineer, it's a really evocative term. Right? You probably feel something when you say it and it's like, oh my god, to be accused of writing something that's sloppy is probably the worst kind of like insult that you can possibly have at the moment. I think for software engineers at the moment who are kind of like transitioning through this period and on this roller coaster, the slop represents things like incidents that you're not gonna get know how to resolve.

It represents architectural decisions coming into your code base that you may not have agreed on. Nobody agreed on that AI kind of like decided to do that. So in thinking about SLOP, I kind of wanna define it as well because I think if until I can define it and and get it out of just how I feel about it, I can't really start to think about the things that I can do to take action against it.

Right? So I've been trying to find a good definition for it. I've struggled a little bit. So as we all do now, go to Claude, have a long chat about what is this thing and try and define what slop is. So me and my old mate Claude arrived at this, which I quite like. Right? So it's the idea that slop is code that's cheap to produce but expensive to verify.

What does that all mean? It means that we're at a stage at the moment where things have become imbalanced in terms of the code that we're we're producing. So up until now, 10, a 100 lines of code would take about the same amount of effort as to to review as it would to write. Right? But nowadays, we can write 10 times, a 100 times, a thousand times the code that we could do in the past.

And not it's not just us as well, like we're probably all software engineers And so this can be done by anybody, right? So who's got this concern similar to me? Or is it just me? Is this going be really awkward? There's a few. Yeah, right. Yeah, okay, okay. I'm glad, man. I was gambling when I was going to do that.

Could be like no one was going to say anything, then the whole talk is going get really, really awkward. That's okay. So at nine, starting to see some smells come through in terms of the code that gets kind of like put up by the developers. All well meaning, right? All well meaning. But starting to see things come through where the code that goes up, it just looks out of place, like the way that a service is being accessed might be different to elsewhere in the code base.

You might find that the styling is done a little bit differently. It's dropped in tailwind for you, thank you very much, when you're a SaaS kind of group. So things just look they look fine in the PR itself, but when you take a step back, the the code just doesn't fit in with the rest of the architecture that you've got going on.

The comprehension of the actual change that's going in. Developers putting up code where they actually don't know what's in there anymore. Like, they can't comprehend what the code is that's being contributed, and so there's a big comprehension gap that that builds up. Third smell that we kind of see is verification as well. Again, the PR looks good.

The tests have been written by the model for the code change that's been written by the model, but the tests are shallow. And if you interrogate them a little bit more, you find that they're just a happy path, or they're not quite what you'd expect if you were writing these things yourself. So I worry about that. I feel as though that the start of this stuff as well, it's not kind of kicked in certainly at nine.

It's not kicked in for us yet in terms of of what these kind of consequences mean. So I worry about what happens if we don't put something in place to put this in check. Right? I worry that the code the volume of code goes up, that the review quality goes down as kind of teams start to buckle under the pressure of trying to, like, keep up with these code changes. I worry that the code base is then gonna degrade as as a result as as feel people feel the pressure to put this stuff through.

And then you go again, do another prompt, and you go again, add into the more of the volume of the code that comes through. And it becomes a cycle and things degrade. It could happen quite quickly unless we take actions to be able to to prevent this. So that's how I kind of feel. Bit worried about it.

But you're probably thinking, David, you're just having a bad day. You've not like you've not like like adopted some of the practices that have been kind of like done so far. And that that might be fair to say, right? So it might be it's just how I feel about this stuff at the moment. So that's not really good enough in terms of like talking to you lot today.

So did a bit of digging, have a look at a lot of reports that are going around. There's things like this one from CodeRabbit, but I've centered on one that's from Feros AI. So their report includes telemetry from about 22,000 developers across 4,000 teams. And so I'm gonna take you through through some of their initial findings that they've had right.

So their initial findings is that for developers who are using like AI workflows and really kind of heavily adopting it, that their tasks completed go up 33%. That's pretty good. Right? That's pretty good. You know, Dave's like, yep, that's pretty good. 66 of epics gone up as well. How good is that? Right? That's the big chunky stuff getting over the line.

Brilliant. 210% increase in tasks involving code. How good is that? Right? So maybe I was like a bit delusional, a bit sad and stuff, and I should end the talk there. Is anybody on the leadership track? Anybody? Yeah. You'd probably like these numbers, right? How good are they? Like, it looks like productivity. Awesome. Right? So should we go for drinks now or no.

Alright. So there's more to this report. There's more to this report. The report also talked about what it was to get the changes out that have been produced. So first thing, 157% increase in median time to first review. So that's that's a developer or an AI getting a change up and having it at first review.

To get the code all the way over the line, up by 442% median time for review. And then this next one, this puzzles me. Right? And I've like interrogated this one a bit. 31% increase in PRs going out with no review. And I looked at this. I thought, it's not just like agentic reviews or something like that, But it's not. It's like no reviews.

This is freaking Yolo Town. Do you know what I mean? Like so I don't know what the base number was on that, but I wanted to include it, because I'm just like, what the actual f, right? So okay. So that's code reviews, telling the story that that these teams starting to feel a bit of pressure in terms of like the code that's coming through and the review times are starting to become the bottleneck, which has been covered a couple of in a couple of the other talks. So but what happens next?

Right? So that's about getting the code in for review, getting it approved, all of that kind of stuff. 243 percent increase in incidents per PR. I don't like that number because it represents having to act on stuff and like hidden work coming back through, you know. It's not good. 54% increase in bugs per developer.

That's like the worst KPI that you could possibly offer people. Right? And then the last one. So my favorite number tends to be like lead time. You know what I mean? So lead time is like how long does it take stuff to go from like idea to the the getting it over the line. That number up by 480%.

So those numbers that we first looked at where everything was looking pretty good and you were excited, I could see it on your face, they actually show that teams do look very productive initially, but there's this thing called the whiplash from that report and that's what the report is called, about the fact that lots of hidden work starts to build up and the teams start to struggle a little bit in terms of the influx of code and the fact that these bottlenecks are really kind of like starting to rear their head quite badly. So I still feel like that a little bit.

That report didn't help me, did it? It looked pretty good at the start, but then it got it went really downhill quite fast. So there are options. Right? There's lots of options for us to take. I've picked out three. Now the first one that I'm gonna give you is that you can pretend that imbalance isn't there, that the influx of code and the fact that you need to put more energy in to verify it is it's not gonna be a thing.

Claude five will come in, rewrite your code base, you're sweet. Or you might get a SaaS tool, bring that in. It's making the reviews kinda fly and that kind of stuff. You can bank on all of that stuff. Doesn't feel like it's quite there yet at the moment, so you kind of gambling that those things kind of like can catch up and can like deliver you.

But there is some boring stuff that you can do as well. So you can make verification cheaper. And what I mean by cheaper is you make it kind of like fast and result reliable for people. Like in like years ago, this would have been a manual process. Right? Now there's lots of automation in place. So you're starting to invest more into that to make sure that those steps, those things that developers need and and AIs need now to verify things quickly is like super sharp, super fast to to to run. These are linters.

These are unit tests. These are visual impression tests, all that kind of stuff. Now the other thing that you can do as well is that if you think about the balance getting out, you can actually put more pressure back on the beginning of the process. So this is the idea that you're trying to make writing the code a little bit, like, not slower, but putting a bit more rigor into it earlier.

You're trying to make sure that when the code does get up to that review stage, that it's actually done at the standards that you expect them to be done at. So these are things like making sure that commits are well, like, labeled. You've got y on your PRs, all that kind of stuff to make sure that when it does go through review, it's nice and easy to actually do it.

And teams can kind of like take action quite easily. There's also things around speculative driven development. We saw a great talk on that earlier today and it's touched on a couple of times. It's also making things like ADRs are in the code base, that kind of stuff. So putting pressure back on the process that's earlier in the process. So you've got those.

So yeah. So the second two, really about catching problems like before review. And then the other one, you're really just hoping. Right? And that's okay. I think a lot of us will be we're doing a bit of that. Right? We've got a lot of faith in how this kind of industry is going to shake out. Now, I've got your three options there.

Thinking about standards now. Like, standards are good. Right? I love a good standard. So starting to think about how you break those standards down into like logical kind of groupings and things to to start to carve it up a bit, so you've got action to take. So the first area that you want to be thinking about is like the hard rules, the yes and no things, the things that you would die on a hill for.

So this is things like if your security scanner says that you've introduced a vulnerability, you're like, that is not going anywhere near the code base. It's a hard no. Right? You have measurable thresholds. These are things where the performance performance of your application can be measured in certain ways, and you want to make sure that those standards are high. These are things like the response time of the API maybe, or I'm a client side developer back in the day.

You're talking about like SIT CSS. I'm like a I'm front end developer until I die. Alright? So Lighthouse scores are really important to me. I love a good Lighthouse score, which aggregates a whole bunch of metrics to say how good the performance is is of a client side application. You're also think about rubrics. So Tanya, she was in the other track yesterday.

She took had a great talk on rubrics. These are the things where you define the standard and then you can kind of like mark how close you are to that standard and also include when you don't reach it at all. Right? These are things, again, these are useful for people as well as LLMs nowadays. Right? You've also got the shipping criteria.

I was actually really surprised. We're doing a lot of, like, pilot programs at the moment at nine, and I was actually surprised that teams struggle to define what it was to actually get something out into production tends to be something which is more like tribal knowledge for teams. And no one's actually kind of like written down what the expectation is. So getting that written down and making that as something that's easily available to both people and LLMs is is good too.

Right? So then you got to start you got these standards emerging now. It's all good. Now you can start to think to verify your standards as well. So you got a few options in this space as well. So deterministic checks, really boring, really old school. Right? Just like running the tests as we've we've always done. If you're a node developer, you've probably got npm run test already there. You want your tests here to not really be about the AI, not about the LLM.

You want them to be nice and fast, reliable, and always, like reliable in terms of if I run it this way, one plus one equals two, you know what I mean. And then the other side as well. So the advisory stuff, we're seeing more of this stuff come through. You can use those rubrics that I talked about on the previous slide and you can start to think about how AI might start to run those and flag with a human to say that something is just a little bit off.

If you've got certain tasks which are lower risk, then then you can start to think about just kind of like letting the judgment of the AI kick kick in at that point. So with these ones, with the deterministic checks, the machine checks, those should never be optional, and the AI review ones should should never be blocking. We're starting to see things come through around, like Copilot can do reviews. They're getting pretty good now.

And I think a few folks have talked to me about things like CodeRabbit as well, doing good quality reviews on that front. So we're not just limited to people doing this stuff anymore. We can start to, like, augment the workflow with the AI. So I've got this quote, like, we've been spending a lot of time in this space at nine, and I really like this quote from one of our principals.

Like, early on, when we're starting to look into this, they were very concerned about what's going to happen about this kind of like influx of the code that we might get. So Mitch captured this quite nicely, think, which is that in an age of repeated output, as in things increasing, the volume of code increasing, that we need to balance that with having repeated quality as well. So what happens when you apply this stuff?

I got quite overzealous using Claude at one point. I actually destroyed my own website, like, haven't we all done that? Have we all done that? Is it just me? I'm one of those people, you know. Anyway, I got a bit carried away, destroyed my website. This was on their Mother's Day weekend as well, so that was really good. Like I'm thinking, oh man, I really need to kinda like get this thing back.

Anyway, I thought about restoring from backup and I thought, nah, f it. I'm just gonna rebuild the whole thing with Claude. Like, because I was having some ideas on what it might be to bring in standards and have like a workflow that means that the quality of the final output might be better if I put certain standards in place.

So with that in mind, again, I'm client side developer until I die. Right? So measurable threshold that I picked was Lighthouse. Wanted the standard to be 95 plus. Right? Wrote the standard down. In this case, I just had a a standard markdown file. I think I probably hooked it in through agents markdown.

I just kinda like pointed it to. We saw an example of that in one of the talks earlier. So this is just about me making sure that the LLM knows the standards that I'm trying to reach. I had a deterministic check, which again, just NPM run Lighthouse, runs the Lighthouse tests for me on, like, a sample of the URLs that I've got, and then it came to the results.

So I did it with the standards, and then I rerun the whole exercise of creating this thing without the standards. So with the standards, it hit 100. So it actually exceeded what I expected it to do, which is awesome. Right? Rerun it again without the standards. It was still decent, but the Lighthouse score was 69 on that on that front.

So some things to take home and do at home. Right? Pick one dimension that you feel passionately about or your team feels passionately about. It might be something on an OKR, whatever. Right? Like, pick one dimension to focus on. Pick one document that you can write that clarifies what that standard should be, and then implement one check for that standard as well.

If you bring all these things together, you do it for the first time, you'll feel the benefits, and you'll be able to kind of repeat that process again and again and again until you've got like better coverage. Because if you set the standard, the AI will help you reach it. Thank you.