When AI Agents Go Wrong, Your Dashboard Lies

Nine seconds. One Railway API call. A small software company called Pocket OS lost its production database and its volume-level backups before most humans can finish reading this sentence.

The story circulated fast, as these stories do, passed CTOs-to-CTO like a campfire horror tale about AI agents going rogue. And sure, the rogue-AI frame is emotionally satisfying. It maps onto a narrative we already have. But AI strategy commentator Nate B. Jones, who covers this space closely, argues in a recent video that the rogue-agent framing is actually the least useful thing you can take from this incident—and that the more unsettling truth is quieter and more structural.

"The more useful story," Jones says, "is that most product analytics would have missed the actual product failure."

That claim is worth sitting with. Because if he's right, the Pocket OS incident isn't primarily a story about an AI that did something bad. It's a story about a monitoring layer that couldn't see what was happening until the damage was already done.

What Your Dashboard Actually Sees

Here's the problem in concrete terms. When that Cursor agent was running—finding credentials, making tool calls, operating in what it understood to be its environment—a conventional product dashboard would have been, by most measures, pleased. Active user. Long session. AI feature engaged. Lots of messages. Healthy engagement.

None of those metrics register what Jones calls the inside of the agent run: what instruction the agent received, what environment it believed it was operating in, what credential it located, which tool call it executed, which permission boundary it didn't hit, and where the human oversight loop quietly failed to close.

This is not a small gap. It's the gap between knowing a car is being driven and knowing it's heading toward a cliff.

The underlying issue is that product analytics was built for a different kind of user behavior—one where humans did the clicking, the navigating, the form-filling. The unit of measurement was the session: did the user show up, move through the funnel, convert, return? Those questions aren't irrelevant now, but they're measuring the wrong layer.

"In an agent product," Jones explains, "the important action may not be a click. It might be the instruction. The important product event may not be a page view. It might be a tool call. The important failure might not be a user dropping out of onboarding. It might be the agent retrying the same action, hitting a permission boundary, asking for approval, losing context, or finishing work the user quietly rewrites."

That last phrase—"finishing work the user quietly rewrites"—deserves more attention than it typically gets.

The Completion Trap

Jones draws a distinction that feels obvious once stated but apparently isn't being tracked by most teams: completion versus acceptance. An agent can finish a task and still fail the user. High completion, low acceptance means the agent is technically productive in a way that's practically useless—or worse, eroding trust run by run, silently.

The full matrix is illuminating. Low completion, low acceptance: users are bailing before there's anything to evaluate. Low completion, high acceptance: the agent is conservative but earns trust when it does deliver—probably not ready for more autonomy, but maybe valuable in a narrow lane. High completion, high acceptance: the signal that a workflow is mature enough to run with fewer guardrails.

"The gap between completion and acceptance," Jones notes, "is the part most dashboards have difficulty with today."

What makes this particularly thorny is that chat logs—the thing many teams are treating as their analytics—can actively mislead you. A long chat could mean a user is working through something complex and interesting. It could also mean an agent keeps losing context and the user keeps restating themselves. From inside the chat log view, these look identical. From inside the agent run view, they're opposites.

Engineering Traces Aren't Enough Either

There's a reflex here that Jones wants to head off: the assumption that developer observability tools solve this. Tracing platforms can capture model calls, tool calls, handoffs, guardrails, latency, cost, and execution errors. That data absolutely matters. Engineering teams need it. But Jones is pointed about where it stops:

"A trace can tell you that the agent asked for approval, and product analytics has to tell you whether that approval created real safety or just added friction. A trace can tell you that a run cost 30 cents. Product analytics has to tell you whether that was worth it."

This is the layer most teams are treating as optional, Jones argues. It's not optional. Engineering traces tell you what happened mechanically. Product analytics tells you whether it mattered to the human, to the workflow, to the business outcome.

Salesforce Named the Unit. Now What?

Salesforce made a notable move in its February 2026 fiscal Q4 earnings release, introducing "Agent Work Units" (AWUs) as a metric—reporting 2.44 billion AWUs delivered across Agentforce and Slack, growing 57% quarter over quarter. The biggest SaaS company on the planet is no longer talking about seats, sessions, or tokens. It's trying to measure tasks completed by agents.

Jones is cautiously interested but skeptical of the metric in isolation. A work unit only tells you something useful if you know what kind of work it was, what workflow it belonged to, whether tool calls succeeded, whether the user trusted the output, and whether the business outcome improved. Without that context, AWUs risk becoming the new chat volume—a number that sounds like signal but might just be noise with better branding.

The framing raises a real question that Jones doesn't fully answer: what does count as a completed work unit when agent behavior is this variable? Salesforce presumably has internal definitions, but the public metric doesn't reveal them. And if the unit of measurement becomes a commercial talking point before it becomes a rigorous analytical framework, the incentives get complicated fast.

What Actually Needs to Get Built

Jones's practical recommendation is deliberately minimal. He suggests starting with three tracked events, tied to a shared agent run ID: when a run starts, when a task completes, and when a user intervenes mid-run. That last category—interruptions, corrections, denied approvals, mid-task clarifications—is where he thinks the most underutilized signal lives.

"When does a user interrupt an agent, edit an output, deny an approval, give a clarification, or reopen a task in the middle of a run? They are labeling that run. They are telling the product team what the agent misunderstood, what context was missing, which action felt unsafe, and which output didn't meet the standard."

The correction as label is an elegant idea. It reframes friction not as evidence of product failure but as a diagnostic. The user who rewrites an output, or denies an approval, is performing an informal eval in real time. The product team's job is to capture that signal before it dissipates.

He's careful to note that this doesn't mean feeding every prompt, customer record, and model output into a training pipeline—privacy handling has to be deliberate, not a byproduct of volume collection. But it does mean treating the corrections as structured data rather than noise.

The Governance Question Nobody's Asking Loudly Enough

What I find genuinely interesting about Jones's argument—and what he doesn't dwell on but is lurking throughout—is the accountability dimension. The Pocket OS database deletion happened because an agent had access to credentials and tools it probably shouldn't have, operating without the kind of runtime visibility that would have flagged the behavior pattern before it became irreversible.

That's partly an analytics failure, yes. But it's also a permissions architecture failure, a product design failure, and arguably a deployment readiness failure. Jones frames analytics as "the rudder on your agents"—but a rudder only steers a vessel that's seaworthy to begin with. Product analytics can tell you a workflow is exhibiting dangerous patterns, but only if someone is watching the dashboard, has set appropriate thresholds, and has the authority to intervene.

None of that is automatic. All of it requires organizational decisions that precede the technical build.

Jones's argument is that most teams are currently watching activity and calling it insight—and that the gap between activity and insight is exactly where catastrophic failures hide. The Pocket OS incident is an extreme example, but the mundane version—agents finishing work users silently redo, approvals that create friction instead of safety, workflows that look healthy but aren't—is probably happening at scale right now across products that nobody is monitoring at the right layer.

The question is whether the teams shipping those products know they're flying blind, or whether their dashboards are telling them everything is fine.

---

By Marcus Chen-Ramirez, Senior Technology Correspondent, Buzzrag