AI Agent Observability Is Now a Compliance Problem

Here is the liability question that no one at AI Engineer Singapore asked out loud, but that sat underneath every engineering lesson in Arize head of product Salian's Sunday morning talk: when an AI agent deployed at scale forgets what it was supposed to do, takes the wrong action, and there is no trace of what happened — who is accountable, and to whom?

That question used to be theoretical. It is not anymore.

Salian spent roughly thirty minutes walking a sleep-deprived conference room through three years of hard lessons from building Alex, Arize's AI engineering agent. The talk was structured around four problems every agent builder eventually hits: staying on task, managing context, crystallizing good behavior, and debugging. The solutions she described — explicit planning tools, structured state management, production traces as test ground truth, composable debugging infrastructure — are genuinely useful engineering. But the detail that stopped me wasn't the architecture. It was a phrase she returned to twice, almost in passing: "observability before you need it."

That principle reads differently in May 2026 than it would have two years ago.

The Regulatory Clock Is Running

The EU AI Act's requirements for high-risk AI systems — which include systems making or materially influencing decisions in employment, credit, healthcare, and critical infrastructure — mandate logging, traceability, and the ability to audit automated decisions after the fact. Article 12 requires that high-risk AI systems be designed to automatically record events throughout their lifecycle. The UK's AI governance framework, while less prescriptive, is moving in the same direction. In the US, the FTC has signaled through its enforcement actions and guidance on automated decision-making that it expects companies to be able to explain what their systems did and why.

What Arize sells — and what Salian was describing from the inside — is precisely the infrastructure that makes that kind of explanation possible. The fact that she frames it as engineering wisdom rather than compliance posture is, I suspect, intentional. Engineers don't like being told they're building compliance tooling. But the market reality is that for any AI agent operating in a regulated context, "observability before you need it" is no longer a best practice. It is a legal exposure calculation.

What the Engineering Actually Reveals

The technical problems Salian described are worth understanding on their own terms, because they illuminate what auditability of AI agents actually requires in practice.

The task management problem is more fundamental than it sounds. When you ask an agent to complete multiple things, she explained, it's not that it hallucinates or lacks capability — it's what she called "a tension problem." The agent can see the first task clearly, but as context accumulates, subsequent tasks get buried. Alex's solution is explicit planning: a to-do structure with states (pending, in-progress, completed, blocked) that lives outside the conversation history so it can't be truncated away. The plan is injected fresh on every LLM call, after system instructions.

This is not just task management. It is a record of intent. An agent that maintains an explicit, persistent to-do structure — including what it was trying to do, what it completed, and what it was blocked on — is an agent whose decision trail can be reconstructed. That has direct implications for any post-hoc audit of what an agent did and why.

The context management problem surfaces a different dimension of the same issue. Arize's observability data is text-heavy; a single experiment comparison, Salian noted, "can be hundreds of rows, which is like 100,000 tokens." Prompting the agent to limit comparisons did nothing — the data volume alone overwhelmed the context window. The solution Arize built — storing the bulk of tool-return data in external memory and providing the agent with a retrievable ID — means the agent is working from compressed summaries, not full datasets. "We compress the values, not the structure," she said. The shape of the data is preserved; the content is truncated.

For a compliance officer, this creates an interesting question: if an agent made a consequential decision based on a compressed representation of data it never fully saw, is the full dataset part of the audit trail? The engineering answer is yes — it's in serverized memory with a retrievable ID. But whether that satisfies a regulator asking "what information did the system actually use" is an open question that the engineers building these systems are not, by and large, asking.

The Testing Gap

Salian's third lesson — crystallizing good behavior — is where the regulatory stakes are most acute, and also where I find myself most uncertain about the current state of the field.

Her testing approach evolved from "vibe checking" (a spreadsheet and a Google doc) to three tiers: decision-point tests on individual components, trajectory tests that step through production choices row by row with an LLM-as-judge, and CI-integrated prompt validation. The move from manual evaluation to production traces as ground truth is sensible. Real user sessions capture edge cases that synthetic test construction misses.

But the LLM-as-judge model has a known problem: you are using a probabilistic system to evaluate a probabilistic system. Salian acknowledged this directly — outputs are non-deterministic, so evaluation prompts have to "define success for each individual step" rather than expect exact matches. That is the right engineering response. Whether it constitutes sufficient documentation of system behavior for a regulator asking about a specific decision outcome is a different question.

The EU AI Act requires that high-risk AI systems maintain logs that enable "the monitoring of the operation of the AI system." It does not specify that an LLM cannot be part of the evaluation chain. But it also requires that the logs be "appropriate to the intended purpose of the system." For a system operating in healthcare or financial services, "the LLM-judge agreed the response was semantically correct" may not be the audit trail the regulators had in mind.

The Accountability Gap

The teams most exposed by what Salian described are not the ones in that Singapore conference room. They're the teams that didn't spend three years in production discovering these lessons.

Arize's Alex is, by the company's own description, a dogfooding product — they built their observability platform partly by using it on their own agent. That feedback loop is a significant structural advantage. You cannot really have evaluation without observability, as Salian put it, and you cannot have observability without having built it in before you needed it. But most organizations deploying AI agents today are not building their own observability infrastructure from scratch. They're procuring agents, integrating them into workflows, and often doing so without a clear understanding of what happens when the agent encounters a state its designers didn't anticipate.

When something goes wrong — and in any system operating at scale, something eventually goes wrong — the question is whether anyone can reconstruct the decision trail. Under the EU AI Act and analogous frameworks, the obligation to answer that question sits with the deployer of the high-risk system, not the vendor who built the underlying model. That means a hospital, a bank, or a government agency that deploys an AI agent without Arize-equivalent observability infrastructure is taking on legal exposure their compliance teams may not have assessed.

Salian's parting principle — "you can't really fix your agent and make it successful without observability" — is framed as engineering advice. But for the organizations that will face regulators asking why their automated system made a particular decision, it is also a description of the minimum viable defense.

The teams that skipped that infrastructure because it seemed like future-proofing rather than present necessity are going to find out what "before you need it" actually means.

---

Samira Barnes covers technology policy and regulation for Buzzrag.