AGENTS.md is the wrong conversation

So we all wondered at some point and played with what to put into AgentsMD. And just how we heard the the presentation before that, like, putting too much in there can be harmful. Putting too little is harmful as well. And in general, it like, what are the things we should put there? And there is a lot of discussions online talking about, like, put this there, put that, and you could go forever and trying to optimize what you have there.

But I want to make a pitch here and and took for look from a bit different perspective and say, what are you putting there doesn't matter that much, and you shouldn't spend too much effort on trying to perfect what you're putting in there. So but to look on what's the general discourse out there when talking about m agents m d in your repo is, like, treat it as sort of router file or index file that can reveal the context progressively across bunch of files that you may have in your dot agents folder or dot ai folder and so on and sort of allow the agent browse and navigate your markdown files.

And that's a really good advice. And if you're starting, start with that. That's a good one. But if we spent a lot of time fine tuning all the instructions, all the documentation, we will run into a problem. And that everything you put in there is basically capture of single person taste, how they think about the stuff.

It captures the single moment that the user is writing it, so over time it will get worse and worse and worse, and at the end, you doing that for one particular repo at that time, like, how do we work in in organizations where you work across multiple repos, you have multiple teams, like, how do you scale that?

And sure, agents can generate something for you, but the static point of that is is a problem. So instead of doing all that and spending time of that, let's think about, like, how this how the system as a whole learns.

And focusing on the learning loop of the system and instructing it like, hey, every time there's a feedback given, write it into instructions. Every time we use skill, save it in there and commit it with the git. Every time you give it to some spec and do it in the chat, capture it in the markdown as well.

We heard that like two slides, but to two talks before. Right? So all these things are super valuable and yes, because we are on AI memory conference probably, we heard about like how memory solves all these problems, but it's not only the memory. Memory and remembering just don't do a thing is is useful.

It is a good start, but if the memory stays separate from the repo, if it's there separate from the instructions and doesn't update them as such, it will be just another bucket of things that the agent need to navigate, and there will be conflict between your own instructions and your memory, and it will all start falling apart sooner or later. So and if you don't have the learning loop at all, you're just like doomed to repeat the same mistakes over and over again.

Talked about this one. So to talk few other like common issues with memories when you just pick whatever you you found on Hacker News today, many of them focus on user and remembering your own specific preferences. They often solve save that into your user profile, and while that helps you as individual kinda not repeat yourself, it introduces few more problems to that. It introduces cross project pollution.

So when you walk across multiple projects, it thing that is true and instruction that is correct in one project may not be relevant in the other one because they come from different times, different eras, different whatever, not everything is the same. And when those memories are basically instructions what to do and what not to do and they are versioned separately from the code base, when someone else's updates the code base and the memory becomes invalid, how does it get updated?

How does it get synced? It all, falls apart. So the point here is that to the good learning loop makes your harness self improving and self healing. So if you make your harness not only remember things, but update your instructions, even if the instructions are wrong at the start and, they run out of sync or something like that, if the harness can detect that and the instructions can detect that, then it will improve over time. And that's more important than the fact facts that are there right now.

So and it's easier to do than it may sound. So because you already have all the building blocks that are needed for doing a simple learning loop. It won't it ain't gonna be perfect. It ain't gonna solve all your problems, but you already have all the blocks to build a simple one.

So you use agents m d or cloth m d or whatever is your tool doing. You're using markdown files, stored in git. I hope everyone's using it. You're using Git to share the stuff with your teams, and you already have a scope for your projects in shape of repositories.

So we don't need to reinvent any of that, and we can just use as it is. And then we can tell it to and in your specific context, what it should save. You want to save specs. You want to save ADRs, the patterns, glossary, whatever you want to define and how it evolves and you want it to be updated in your setup, you can tailor it and just in in your agent's MD, say, hey.

These informations store here and every time you do that, save it there and so on. You also want to add some parameter or some instructions about how to refine that. So for example, when you in some cases, you will work with more episodic types of things where you think like ADR at this moment is like we made this specific decision at the moment and the old decisions stay there.

In other things like architecture diagrams, you may want to instruct the agent, hey, update those diagrams and update these states. Each of the each of the memories and there were some some descriptions of them work slightly different, but just think about that as an engineer. If you wouldn't do your AI things and you would just do the mechanical work, what are the documentations you would like other people to write for you?

Don't think what you want to write because you don't want to write any, But what do you want other people to write for you? And do that for oh. Yeah. We're back. So another option is that in context pipe, we are trying to solve this in more reasonable fashion that goes beyond the super simple setup.

And, we shipped, this CLI context memory in it that helps you to set up these things and helps you work with that. It's powered by agent memory, dev, a a cool open source tool that manages the user based memory.

We extended that to work with and on top the markdown files and with the git, and you can use that. It's open source. Go go ahead with that. But with this loop, we solved things within the repo. We solved like how we learn from conversations, the agent knows like every time we give it feedback, save it to markdown.

We solve sharing because we have Git, so it's there and the project knowledge kinda starts accumulating. Repository solved scoped for us. We even have review process in the place. So if there are, like, other teams collaborating, oh, yeah. That's true. That's not. We want to agree on things. PR is exists. But it's still in one repo, so not ideal when you want to scale that across the whole company and across many repos. And, also, you don't bring in outside learnings from other knowledge bases like Confluence, like Notion, Jira, and wherever you may have your documents. Those are full of gold and sometimes not, but it's still useful context for the agents.

So if we have a local loop, what if we add outer loop as well? And think about the things that we've committed into Git as one source of information that comes into context. And when you merge it into main, you know it was reviewed. You want you know what is it was clear, and you can ingest that into the overarching large knowledge base.

Into the same overarching knowledge base, you can pull the documents from Confluence and elsewhere and source that and see, like, how that starts, progressing and growing and getting more and more context. With getting more and more context, you will run into problems that it will disagree with each other.

What's Jira and what is Git are two different things. What's in confluence and what was written five years ago is not true, almost never. So the knowledge based system there needs to have some mechanisms how to deal with that. So you need to navigate how information change over time.

So temporality of that, the you need to deal with the confidence of how the things that are coming from confluence, a bit less confidence what's coming from code itself. That's a bit better because it's actually encoded. And some of these things might be might be ambiguous and navigate in in general. So building this knowledge base on top of all your sources is complicated, but when you start doing that and start having a good triggers, you can start using this knowledge base to automatically update your repos back again and your local loops.

So imagine your software architect in your company says like, hey, and now we will start using Postgres for all new databases because MySQL was gone. And then but all your repos say use my my SQL. Right? So having oh, should have disabled that.

Come on. Oh, yeah. Here we go. Back. Back.

So so having a way to go and and distribute again the knowledge into across the reports is super useful. And m c and giving MCPs to the clients, that seems like obvious, and, yes, it can load confluence, but it won't give you the automations across the across the board.

So how you can build this yourself? Like, do you need any complex products for that? No. You don't. And there was a great post by Andrei Karpathy of, like, how he builds his own LLM knowledge bases, and it's actually much simpler than it sounds. You just need some scripts that load documents.

It can be markdown files or you can chuck that into Obsidian or wherever you would like. Then use some, like, context management tools just to be able to look into it. If you're pulling that into Obsidian, that's the thing. You need to select an interface for that so the agents can interact with it and then have some background processes to improve and iterate and actually trigger on the changes and so on.

So if you look on this, it's just a bunch of scripts and document store. And the the background refinement, if it can search your code and navigate your code, that's basically it. And or second option is try to help us a little bit. And we are on the journey of trying to build this as an open as open source and making it easier for everyone to to use and not needing to build a primitives version themselves and do that on that.

You can find us on GitHub. The QR code will be in second. But the main message here is that it's more important to focus on the learning loop and how your agents improve over time than whatever is the perfect instructions right now because the the perfect instructions right now ain't gonna be perfect tomorrow.

Thank you.

Thanks, Jakob. We're actually running a few minutes ahead of time. Are you alright to stick around for some questions, Jakob, if anyone has any? Does anyone have any questions at all? No question? Oh, we got one? Oh, yep.

My question is, how often do you find the AI actually ignores the memories that it's made? Because that's been one of the big problems I've had. You put stuff in agent.md, it ignores it. You create a specialized skill, move it out of the main thing. It decides it's not gonna use the skill and just goes and does it somewhere else.

You then have it create a memory that says, hey, remember, you have to use that skill when you're doing this. And it reads the skill once at the beginning and ignores all the content and

Yeah.

Does its own thing instead. Or it forgets to read the memory and then says, oh, let's make a memory for it. It's like you've already got one.

Yeah. Yeah. It's a it's a it's a real pain. That's that's my life now. The the the like, obviously, in in the oversimplified example I showed, those are exact problems you will run into. There is methods like how various agents have various hooks, how you can force them more forcibly.

It's still not guaranteed, and there is a lot of, like, trial and error to figure that out, how to capture it better and better and recall better and better. The agentmemory. Dev, I've linked up that, is a good example how it can be done. They clearly spend a lot of time tuning all those various agents.

Each of them acts differently. Again, it's very hard to come with universal solution, but in simplicity, adding agent hooks that will force it a

bit more.

Yep. I've I've had it do things like I've had it do things like having a skill that says, this is how you do type checking in this repository. And you've gone and preapproved those tools so it can do it without requesting it. And it's given me a request to do it in a different way.

And I've said, no. Go and use the command from the inside the skill. It's read the skill file and then repeated giving me back the exact command that I just rejected even though it read the skill file. So it it's been really frustrating and about the only hooks I found that actually work is if you can do a rejects or a grep on the code that it's generating and say, no.

Reject it at that point in time because it's violating these rules. But it's it's like it doesn't affect what's actually being generated in the first place.

Yeah. Like, it's a it it it gets to a point of, if if the agent is ignoring the skills, ignoring the instructions, then literally doesn't matter what's what's in there whatsoever. But it's like if we capture that, in some cases, it will listen to them and those we can cover, But it's not like we can solve everything and obviously it's not a solved problem as nothing we talked about here.