CAG vs Long Context: How LLMs Access External Data

There's a problem baked into every large language model that doesn't get enough attention in the hype cycle: the model only knows what it was trained on. Hand it your company's employee handbook, last quarter's earnings report, or any document that postdates its training cutoff, and it has no idea what you're talking about—unless you tell it, right there in the prompt.

Most people in the AI space have encountered RAG—retrieval augmented generation—as the standard answer to this problem. Build a vector database, embed your documents, retrieve the relevant chunks at query time, inject them into the prompt. It works. But it's also infrastructure: moving parts, retrieval pipelines, embedding models, the occasional misfired retrieval that hands the model the wrong document entirely.

Two alternatives—long context and cache augmented generation (CAG)—take a different approach, and understanding the tradeoffs between them is more practically useful than it might sound.

The "Just Stuff It In" School of Thought

Long context is exactly what it sounds like. Context windows have grown dramatically—GPT-3 in 2020 handled around 4,000 tokens (roughly 10 pages of text). By 2023, GPT-4 Turbo pushed that to 128,000 tokens, about 300 pages. Then Google's Gemini 1.5 Pro arrived in 2024 with a two-million-token window—IBM Technology's Martin Keen puts that in terms of "20 full-length novels." The trend hasn't plateaued.

The logic follows naturally: if the context window is big enough, why maintain a retrieval pipeline at all? You load your documents into the prompt, let the model read everything, and ask your question. No retrieval errors. No missing chunks. The model has access to the full picture.

For one-off tasks—analyzing a single contract, answering a few questions about a specific document—this is genuinely the most sensible approach. There's nothing to set up, nothing to manage. You pay the token cost, get your answer, move on.

The catch is what happens when you do this repeatedly. Keen makes the inefficiency concrete: "10 questions about the same set of documents mean the model reads these documents 10 times." Every query starts from zero. The model re-ingests everything, every time. At scale, or on long documents, that becomes both slow and expensive—most LLM API pricing scales directly with token count, so feeding in 200,000 tokens of context on every single query adds up in ways that tend to surprise people the first time they see the bill.

There's also a less obvious problem. Research has documented what's been called the "lost in the middle" effect: LLMs attend more reliably to information at the beginning and end of a long context than to information buried in the middle. Stuff everything into a giant context window and you may find the model is quietly underweighting the documents that happen to land in the center. It's a limitation that doesn't announce itself—the model still produces a confident, coherent answer, just one potentially informed by an incomplete reading of the material you provided.

What CAG Actually Does

Cache augmented generation addresses the redundancy problem directly. The insight is straightforward: the expensive part of processing a long context is the computation, not the question-answering. If you're going to ask ten questions about the same documents, the model doesn't need to re-read those documents ten times—it needs to read them once and remember what it learned.

The mechanism that makes this possible is the key-value (KV) cache. When a transformer model processes text, each layer computes key and value matrices—essentially the model's working representation of everything it's read. Normally these are discarded after each request. CAG saves them.

In practice, CAG works in three stages: you prepare and format your documents, run them through the model to generate and persist the KV cache, and then at inference time the model loads the precomputed cache instead of re-reading the source documents. The heavy computation happens once, upfront. Subsequent queries are dramatically faster.

How much faster? Research figures cited by Keen are striking: roughly 10x speedup for small datasets in the KV cache, scaling to around 40x for larger ones compared to reprocessing the full context on every query. The first query costs the same as long context; the hundredth query is where the economics shift decisively.

The HR chatbot is Keen's example, and it's a good one: "an HR chatbot answering questions about company policies, because the knowledge doesn't change very often and the cache stays valid." Stable knowledge base, high query volume. Every question after the first one is fast and cheap.

That word stable is doing real work in that description. CAG's Achilles heel is document freshness. When the source documents change, the entire KV cache has to be recomputed from scratch. For a knowledge base that updates frequently—live inventory data, daily news feeds, anything with a high rate of change—the cost of constant recomputation starts eating into the latency savings CAG is supposed to deliver. "If the data changes frequently," Keen notes, "the cost of constantly recomputing starts to eat into any latency savings that you are making."

Prompt Caching: CAG Without the Infrastructure

There's a practical wrinkle worth flagging. Managing KV caches is its own infrastructure challenge—you're persisting and loading internal model representations, which requires knowing what you're doing. For most developers building on top of LLM APIs, that's not a capability they want to build in-house.

This is where prompt caching enters. Major LLM providers now offer it as a managed API feature: you send a long system prompt containing your documents, the provider handles the KV cache management behind the scenes, and subsequent requests that share the same prompt prefix skip the document processing entirely. As Keen puts it, it's "essentially CAG as a service."

The economics are meaningful. Cached token reads can come at around a 90% discount compared to processing those tokens fresh. That's not a rounding error; it's the difference between a feature being economically viable at scale or not.

Prompt caching is, essentially, the productization of a research idea. The underlying technique—precompute, save, reuse—was always elegant. What changed is that you no longer need to build the plumbing yourself.

Picking the Right Tool

Neither long context nor CAG is universally superior. The choice depends on two variables: how often you'll query the same documents, and how often those documents change.

One-time analysis of a document you'll never touch again? Long context. Simple, no overhead, no precomputation wasted on a single query. Querying the same stable knowledge base hundreds of times a day? CAG—or prompt caching if you'd rather not manage the infrastructure. Querying frequently but updating the documents constantly? That's where the calculus gets genuinely complicated, and where RAG's retrieval pipeline may still have advantages neither approach fully replaces.

The deeper question—whether RAG is actually being displaced by any of this—is one the AI community is actively arguing about. Growing context windows reduce one of RAG's traditional advantages (you no longer need to retrieve selectively because you can't fit everything). CAG reduces another (the repeated processing cost). But RAG has its own strengths: dynamic retrieval from truly large corpora, the ability to update source documents without recomputing anything, fine-grained control over what the model sees.

What's actually changing is the assumption that RAG is the only serious answer to the external knowledge problem. It isn't. The toolkit is getting more varied, and that's worth understanding before you reach for the vector database by reflex.

---

Marcus Chen-Ramirez is a senior technology correspondent at Buzzrag covering AI, software development, and the intersection of technology and society.