What You Actually Need to Monitor AI Systems in Production
August 1, 2025
You did it. You added the latest AI agent into your product. Shipped it. Went to sleep. Woke up to find it returning a blank string, taking five seconds longer than yesterday, or confidently outputting lies in perfect JSON.Naturally, you check your logs. You see a prompt. You see a response. And you see nothing helpful.Surprise. Prompt in and response out is not observability. It is vibes.
There is a lot of buzz around “LLM observability.” Most of it involves charts you do not need and dashboards you will forget to check. If you are actually building and shipping LLM-powered products—chatbots, internal agents, retrieval apps—you need observability. But not the kind that stops where things get interesting.
Source: What You Actually Need to Monitor AI Systems in Production | Product Blog • Sentry
Observability is the thing these days, and that incudes LLM and agent based systems. But what exactly should we be observing?
