Topic Pillar

AI Observability.You cannot fix what you cannot see.

Observability is the unsexy half of production AI that decides whether you keep your job. Per-step traces, score-bearing evals, cache-hit telemetry, tool-selection scores — these are the primitives that let you debug agents instead of guessing at them.

45 cluster pages· 19 posts· 2 notes· 24 updates

The four telemetry layers

Input/output traces per step (Langfuse-shaped). Eval scoring on representative inputs, not just happy paths. Cache-hit rate per route — the new throughput metric. Tool-selection score deltas from the model boundary. Without all four, multi-agent debugging is folklore.

Eval datasets that surface real bugs

A 100-case eval set that scores 95% means nothing if all 100 cases are the happy path. The cases that matter are the ambiguous ones, the conflicting ones, the malformed ones. Build the eval set adversarially — ask "what would a real user actually paste in?" — and most agents drop 15–25 points.

Per-agent cost attribution

Tools like Langfuse now expose per-agent cost attribution and step-level cache-hit telemetry. You will discover 30–50% of your agent traffic is repeat queries you should be caching. This is the report nobody wants to read but everyone needs to.

19 blog posts

Deep dives on AI Observability

Production

Claude Code Artifacts turn terminal output into live review pages: what Team and Enterprise buyers should pilot first

Artifacts in Claude Code beta publish self-contained HTML to claude.ai that republishes to the same URL as the session progresses, with version history and org-only sharing. Strict CSP, no external fetch, no backend. Requires Team or Enterprise and claude.ai login. Here is the workflow I use for PR walkthroughs and incident timelines without screenshot threads in Slack.

Jun 22, 202613 min
Read the post
Production

Agentjacking is real: poisoned Sentry errors can hijack Cursor, Claude Code, and Codex without touching your repo

Tenet Threat Labs injected a fake stack trace through a public Sentry DSN and watched 100+ coding agents execute attacker commands during normal triage. No git write access required. The agent treats the error as ground truth. Here is how I harden observability MCP feeds, scope triage prompts, and block auto-exec on untrusted telemetry.

Jun 17, 202613 min
Read the post
Production

The June 15 Claude billing change: Agent SDK credits, model retirement, and the checklist I run before anything breaks

Two Anthropic changes land on the same day: programmatic Claude usage moves to a separate monthly credit pool, and claude-opus-4-20250514 plus claude-sonnet-4-20250514 stop answering on the API. Interactive Claude Code is fine. Cron jobs and CI agents are not. Here is how I audit auth paths, claim credits, and grep for retiring model IDs before the first failed run.

Jun 15, 202614 min
Read the post
Production

Governing agent autonomy in 2026: Auto-review, pre-push review, and why approval prompts are not a security model

Cursor made Auto-review the default run mode and shipped /review so Bugbot runs before you push. Together they treat agent autonomy as a dial: low-stakes actions flow, high-stakes actions slow down. Here is how I wire that pattern into local agents, SDK headless runs, and CI without mistaking convenience for a hard security boundary.

Jun 11, 202614 min
Read the post
Architecture

Agentic RAG vs vanilla RAG: why a Sufficient Context Agent beats retrieve-then-pray

Google Research shipped Agentic RAG on Gemini Enterprise with a Sufficient Context Agent that refuses to answer when retrieval is incomplete. On factuality benchmarks they report up to 34% higher accuracy versus standard RAG. Here is when one-shot RAG is still enough, when you need iterative retrieval, and how I wire the pattern without blowing latency budgets.

Jun 6, 202614 min
Read the post
Production

Agentic transformation is an operating-model problem, not a model problem

Microsoft published a 6-step playbook for rolling agents out across an enterprise, and the line that matters is "you do not need a bigger model, you need a better operating model." That matches what I see in consulting: the pilots that die do not die on model quality, they die on ownership, evals, and governance. Here is how I read the playbook for IT services teams, and the operating-model gaps that actually stall agent rollouts.

Jun 4, 202611 min
Read the post
Architecture

Your coding agent has amnesia. Persistent memory is the fix.

Claude Code forgets your architecture, your decisions, and why you ruled things out the moment a session ends. The reliability tax is not tokens, it is re-establishing context every morning. Here is what persistent agent memory actually is, how an open-source engine like Cortex implements it, and how to evaluate a memory layer for your own agents.

May 29, 202611 min
Read the post
Production

Your agent's supply chain is the attack surface now

A poisoned VS Code extension spent eighteen minutes on the marketplace and walked off with Claude Code credentials and MCP configs. The model was never the target. Your agent's supply chain is: the extensions, skills, MCP servers, tool definitions, and keys it is allowed to touch. Here is how I harden all four layers, and the checklist I run on every deployment.

May 27, 202612 min
Read the post
Production

How an agentic studio screens, scores and shortlists candidates for your hiring team

Open Recruiting Atelier and you do not see a generic AI dashboard. You see five named specialists doing the work a screening team would do: catching duplicates, checking the brief, scoring on four dimensions, ranking, drafting the dispatch. Drop one CV or fifty. Click any candidate to see exactly why they landed where they did. This is what AI for recruitment looks like when it respects your judgment instead of replacing it.

May 24, 202610 min
Read the post
Tool Design

Tool registry design for agentic AI: how the wrong registry kills accuracy before the prompt is read

I reviewed a system last month with 47 tools in its registry and a 22 percent wrong-tool-selection rate. The team was about to migrate from Sonnet to Opus to fix it. The prompt was fine. The registry was the bug. This is the audit pattern I run on every client codebase before we change anything else, the seven failure modes I see in production, and the numbers from the cleanup.

May 22, 202612 min
Read the post
Architecture

AI agent vs agentic AI: what the distinction actually means when you ship one

Vendors blur the line because "agentic" sells. The two terms describe different architectures, with different cost shapes, different observability needs, and different scoping conversations. Here is the framing I use with clients and the three-question test for which one your project actually needs.

May 22, 202612 min
Read the post
Production

MCP governance just became a product: what Databricks Unity AI Gateway changes for enterprise agents

Every enterprise MCP deployment I have audited in the last six months has been hand-rolling tool-access policy, payload logging, and per-team cost limits on top of a gateway someone wrote in two days. Databricks just shipped that as a product. Here is what it actually changes, where the gaps still are, and the migration I would run for a Databricks shop.

May 20, 202612 min
Read the post
Tool Design

Tool descriptions are prompts. Fix the registry, not the agent.

When an agent picks the wrong tool, the registry is broken, not the agent. Three rules I now apply before debugging anything in a multi-tool system: precise names, "when to use" triggers, and a curated load list. Anthropic's new tool-selection telemetry finally puts numbers on what changes accuracy.

May 13, 20269 min
Read the post
Production

The cheapest LLM call is the one you do not make. GitHub's 19-62% token cut, decoded

GitHub published an instrumented analysis of their agentic CI workflows and reported 19-62% token-cost reductions. The savings are the headline. The technique (pre-agentic data fetching and tool-registry hygiene) is the story most teams will miss.

May 11, 20269 min
Read the post
Architecture

Claude Opus 4.7's 1M context: when to RAG and when to just stuff it

A million tokens reliably is real now, but it does not retire RAG. It changes the calculus. Cost, latency, recency, and the prompt-cache angle nobody is talking about.

May 6, 20268 min
Read the post
Production

Prompt caching is not optional anymore. Measuring a 47% cost drop

A walkthrough from a client engagement: identifying stable prefixes, restructuring the system prompt for cacheability, and the telemetry that proved caching was actually working.

Apr 19, 20267 min
Read the post
Production

The agent observability stack we ship to every client

Traces, spans, evals, cost-per-completed-task, and the one dashboard panel that catches 80% of regressions. Vendor-agnostic; covers Langfuse, Honeycomb, and rolling your own.

Mar 28, 20268 min
Read the post
Production

Eval datasets: stop testing your agents on the happy path

If your eval set is the demos you showed the client, you are testing the wrong thing. How we build evals from production failures and the minimum viable suite to ship.

Jan 19, 20268 min
Read the post
Architecture

RAG vs CAG: how to actually decide

A decision framework from real implementations. RAG retrieves. CAG stores in cache. Knowing which to use, and when to combine both, determines whether your agent finds the right answer at the right cost.

Sep 21, 20257 min
Read the post
24 ship-news updates

Latest in AI Observability

Claude

Fable 5 included subscription access window closes today: Anthropic planned June 23 removal to usage credits, but global suspension since June 12 still blocks all access

June 22, 2026 · via Anthropic
Open Source

Xiaomi MiMo V2-Flash and TTS endpoints auto-route to MiMo-V2.5 on June 18: legacy model IDs retire June 30

June 18, 2026 · via Xiaomi MiMo
Architecture

Tenet demonstrates Agentjacking: a poisoned Sentry error report hijacks Cursor, Claude Code, and Codex into running attacker code with no repo compromise

June 17, 2026 · via Tenet Security
Open Source

Zhipu ships GLM-5.2: MIT open weights, 1M context, and Anthropic-compatible API for long-horizon coding agents

June 16, 2026 · via Z.ai
Open Source

IIT Bombay unveils BharatGen Param2: a 17B MoE with tool calling across all 22 scheduled Indian languages, plus Shrutam2 ASR and Patram document vision

June 15, 2026 · via BharatGen
Claude

Claude Opus 4 and Sonnet 4 retire on the API today: requests to claude-opus-4-20250514 and claude-sonnet-4-20250514 now fail

June 15, 2026 · via Anthropic
Open Source

MiniMax M3 open weights ship on Hugging Face: 428B MoE with 1M sparse-attention context, native multimodality, and computer use

June 11, 2026 · via MiniMax
Architecture

Google ships Agentic RAG on Gemini Enterprise with a Sufficient Context Agent that stops when retrieval is incomplete

June 5, 2026 · via Google Research
Architecture

Alibaba ships Qwen3.7-Plus as a hybrid GUI-and-CLI agent: native screen grounding, 1M context, and Anthropic-compatible API endpoints

June 3, 2026 · via Alibaba Cloud
Frequently asked

AI Observability — the questions teams actually ask

Go deeper on this topic

New breakdowns on this and related agentic AI topics, plus what I am shipping for clients — one email on Thursdays.