Why Context Engineering Is AI's Real Bottleneck

The frontier AI models available today are genuinely impressive. Martin Keen, an IBM technologist, spent his week "vibe coding" applications he'd been putting off—including an app that indexes B-roll footage for coffee videos. The models nailed the tasks. But here's what Keen noticed: when these same models fail, it's rarely because they can't reason. It's because they don't know what information matters.

This is the context problem, and it's becoming the defining infrastructure challenge for enterprise AI deployment. The term for solving it is context engineering—the ability of an AI system to discover the right data, understand what it means, and apply it correctly in real time within governance constraints. That definition matters because it clarifies what we're actually building: not smarter models, but smarter systems around models.

Consider Keen's example: an analyst asks an AI assistant for everything needed to prep for tomorrow's client meeting. A model with no context produces a beautifully formatted but utterly generic meeting template. A system with good context engineering knows which client, pulls recent support tickets because there's an ongoing issue, checks deal history and discovers renewal timing, and crucially, does not pull the internal pricing discussion it found because the analyst's role lacks access to that data.

"That's not because the model that made it had superior reasoning," Keen notes. "It's because context engineering has given the model relevant contextual intelligence."

The problem isn't theoretical. Enterprise data doesn't live in one place. Some sits in databases, some in document stores, some behind APIs, some on SaaS platforms. Some is in the cloud, some on-premise. Structure varies wildly. Freshness varies hourly. Access permissions create a maze of who can see what. Getting the right context to an AI model at the right time with the right permissions is fundamentally an infrastructure problem, not a model problem.

The Four Pillars

Keen identifies four requirements for effective context engineering. First: connected access. Rather than copying data to a centralized location—which introduces staleness and breaks existing access controls—zero-copy federation allows AI to query data where it lives. The original permissions stay intact. The data stays fresh.

Second: the knowledge layer. Raw data isn't useful context on its own. The knowledge layer performs entity resolution across systems, maps relationships and hierarchies, and adds decision traces and institutional knowledge. It gives data meaning.

Third: precision retrieval. As Keen's English teacher apparently told him, making essays longer doesn't make them better. More context isn't better context. Precision retrieval means filtering documents by intent, role, time, and policy—delivering only what the model actually needs.

Fourth: runtime governance. This is where most enterprise AI deployments will live or die from a compliance perspective. Governance must be enforced at retrieval time and response time. Can this agent query this data source? Should this result be included given who's asking? These decisions happen live, not in batch processes reviewed quarterly.

What's notable about this framework is that it treats governance as a technical requirement, not a policy afterthought. That represents a maturation in how organizations think about AI deployment—governance as infrastructure, not paperwork.

Beyond Basic RAG

Most organizations' first encounter with external context comes through retrieval-augmented generation (RAG)—chunking documents, embedding them as vectors, and doing similarity searches at query time. RAG works well for simple lookups. But precision retrieval requires more sophistication.

Agentic RAG introduces iteration: the AI agent makes an initial request, evaluates what it received, and decides whether to retrieve more. It's a step beyond one-shot retrieval.

GraphRAG uses graph structures to navigate context. Instead of asking what documents are semantically similar to a query—standard RAG's approach—GraphRAG asks what entities are connected to this client and what documents relate to those entities. The graph provides precision and structure; vector search fills in detail within that scope.

Context compression addresses the practical limit of what models can process at inference time. Even with large context windows, more noise produces worse results. Systems can summarize long documents or rank what's most relevant to the specific task, maximizing signal and minimizing noise.

These aren't competing approaches. They're layers. Agentic RAG decides what context to pursue. GraphRAG navigates relationships to find it. Compression ensures what arrives at the model is lean and useful. That's a system with contextual intelligence.

The Real Constraint

What Keen is describing—and what IBM is clearly positioning to sell—is a shift in where AI deployment complexity lives. The models themselves are increasingly commoditized. GPT-4, Claude, Gemini—they're all remarkably capable. The differentiation isn't in model quality anymore. It's in how effectively organizations can connect those models to their actual operations.

This creates an interesting dynamic for enterprise AI. The vendors who win won't necessarily be the ones with the best models. They'll be the ones who solve the context problem—who can federate access across legacy systems, enforce governance at runtime, and deliver precise retrieval without requiring organizations to rebuild their entire data infrastructure first.

That's a harder technical problem than training better models, and it's much more boring. There's no viral demo for "we successfully enforced role-based access controls during AI retrieval." But it's the problem that actually blocks enterprise deployment at scale.

Keen's framing—"model intelligence and reasoning are not the bottlenecks anymore"—is probably correct for a meaningful subset of enterprise use cases. Not all of them. Not the cutting-edge research applications. But for the analyst prepping for a client meeting, for the support engineer diagnosing a customer issue, for the financial analyst building a forecast, the model is fine. The context is the constraint.

Which raises the question: if context engineering is where the real infrastructure challenge lives, what does that mean for how we should be regulating AI systems? Most proposed AI regulation focuses on model behavior, model training, model outputs. But if the failure mode is bad context engineering—wrong data retrieved, governance not enforced, access controls bypassed—then regulating models won't address the actual risk.

Samira Okonkwo-Barnes