When Your AI Agent Fails 17% of the Time

Somewhere right now, a midsize retailer is running an AI inventory agent to manage its supply chain. Maybe a food distributor. Maybe a pharmaceutical wholesaler. The agent flags low stock, generates forecasts, files purchase orders, and writes weekly reports for staff who increasingly trust it to surface what they can't see. What that company's legal team may not know is that a workshop delivered at an Anthropic developer event — presented by Will, an engineer on Anthropic's Applied AI team (full name and title not confirmed at time of publication) — began with a confession: agents like this commonly degrade to a 17% failure rate as they accumulate requirements over time. Nobody notices until the evals do.

In FDA-regulated pharmaceutical supply chains, a 17% error rate in forecasting or purchase order generation is not an efficiency problem. It is a documentation problem, potentially a compliance problem, and depending on what gets ordered or doesn't get ordered, possibly a patient safety problem. In financial inventory controls, where SOX-adjacent audit trails require demonstrable accuracy in automated decision systems, it is the kind of number that surfaces in post-incident reviews. The workshop, delivered at what the speaker referenced as Code with Claude London (an event Buzzrag has not independently confirmed against Anthropic's official listings), was aimed at developers. But the number it led with belongs in a different conversation entirely.

The workshop's central case study — an inventory management agent called Stock Pilot — is instructive precisely because it is generic. The agent started life as a focused tool: flag low stock, handle basic forecasting. Then business requirements arrived, as they always do. Forecasting capability was bolted on. Then report writing. Then supplier selection. Each addition made sense in isolation. Collectively, they pushed the agent's system prompt to approximately 400 lines (a figure from the live workshop demonstration; workshop demos sometimes involve approximations, and the precise number should be treated accordingly). Twelve tools. Three sub-agents operating in isolated context windows with inadequate handoff protocols between them.

The result, as Will demonstrated by running the agent's evaluation suite live, was an agent that had quietly become unreliable. One eval failed because the agent took an inefficient path to a correct answer — passing the task but failing the efficiency threshold. Another failed because a sub-agent completed its work correctly but couldn't communicate the result cleanly back to the orchestrating model. A third failed because two contradictory policies had accumulated in different sections of the bloated system prompt; the model, presented with conflicting instructions, resolved the conflict incorrectly, substituting a wrong multiplier for a correct one it had already retrieved. "This isn't a model problem," Will said. "It's an issue with the information that we're surrounding the model with."

That framing matters, and not just technically. The EU AI Act classifies certain AI systems used in critical infrastructure and supply chain management as high-risk, requiring conformity assessments, technical documentation, and ongoing monitoring. NIST's AI Risk Management Framework — which the White House has pointed to as a baseline for responsible AI deployment — calls for AI systems to be measurable, traceable, and subject to regular performance review. What Anthropic's workshop demonstrated, almost accidentally, is what a semi-functional version of those requirements looks like in engineering practice: run a structured evaluation suite, diagnose failure modes systematically, make targeted architectural changes, and re-run the evals to confirm improvement. The workshop's internal term for this is "hill climbing." NIST's AI RMF would call it continuous monitoring under the GOVERN and MEASURE functions. The names differ; the underlying logic is the same. Whether any enterprise deploying agents like Stock Pilot is actually doing either is a different question.

The architectural remediation Will walked through is worth understanding in its substance, because the nature of the fixes illuminates the nature of the failures. The bloated system prompt — which caused the forecasting contradiction — was reduced dramatically by migrating static business logic into what Anthropic calls "skills": packaged, retrievable information that the model pulls into context only when a specific task requires it. Rather than loading every procurement policy and forecasting guideline into the model's working memory on every query, the restructured agent retrieves relevant policies on demand. The practical result was a system prompt reduced to roughly 15 lines (again, a live-demo figure). The philosophical result was a cleaner separation between standing instructions and task-specific knowledge — exactly the kind of architectural decision that makes an AI system auditable.

The tool rationalization is similarly telling. The original 12 tools, several of which were custom wrappers for sub-agents, were largely replaced with primitive capabilities: bash execution, file read, file write. Rather than giving the agent a pre-built tool for every data retrieval task, the restructured agent was given the ability to write and run its own Python scripts against inventory data. Token usage dropped from over 200,000 for certain tasks to substantially lower figures — again, workshop demonstration data that should be treated as illustrative rather than precisely certified. But the directional finding is credible: an agent that can reason about how to query data uses fewer resources than one that ingests raw data wholesale.

The sub-agent tradeoffs are where the enterprise risk dimension gets sharpest. Will kept one sub-agent — a dedicated forecasting instance — for a specific reason: he didn't want the context of an ongoing customer conversation to contaminate a numerical forecast. Context isolation as a quality control mechanism is a reasonable design choice. But sub-agents also fragment observability. When three separate AI instances are operating within a single workflow, collecting a coherent audit trail requires infrastructure that most enterprise deployments haven't built. One of the F-eval failures in the workshop — the breakdown between sub-agent and orchestrator — was precisely a communication integrity failure: the right answer existed somewhere in the system, but couldn't be reliably surfaced. For a legal team trying to reconstruct why an automated system placed or didn't place a supply order, "the answer existed but the handoff failed" is not a satisfying explanation.

Anthropic's Claude Managed Agents infrastructure addresses part of this through what it calls "callable agents" — a native sub-agent capability that logs sub-agent activity within the same session record as the primary orchestrator, making the multi-agent workflow traceable in a single pane. Whether that satisfies enterprise audit requirements in regulated industries is a question for compliance counsel, not conference speakers. But it's a more honest answer to the observability problem than most agent frameworks are currently offering.

The hill-climbing methodology Will outlined — establish a baseline eval score, make targeted architectural changes, re-run evals, repeat — is, in my read, what responsible enterprise AI deployment should look like structurally. The workshop got Stock Pilot from 83% (and, in one live demo run, 62%) to approximately 92% through a defined, iterative testing cycle. That's also what NIST's AI RMF means by "measurable" — not that you have a number, but that you have a process for generating a number and acting on it.

What's notable is that no one is currently requiring this of most enterprise AI deployments. The EU AI Act's high-risk classification and conformity assessment requirements are phased; enforcement for many categories won't arrive until 2026 or later. The White House's executive orders on AI have pointed at NIST frameworks without mandating them for private sector actors. Sector-specific regulators — FDA, OCC, defense procurement agencies — are watching, issuing guidance, and mostly waiting to see what breaks before writing binding rules.

The lessons from scaled deployments at companies like Amazon suggest that human oversight structures and rigorous testing aren't optional at enterprise scale — they're what keeps these systems from quietly failing at rates no one is measuring. The legal and compliance teams advising enterprises deploying inventory, procurement, or logistics agents would do well to ask not just "does this agent work?" but "how do we know when it stops working, and what's the documented process for finding out?"

A developer workshop at an Anthropic event has no obligation to answer that question. But the 17% failure rate it opened with does.

---

Samira Okonkwo-Barnes is Buzzrag's tech policy and regulation correspondent.