Prompt Caching: Making AI Actually Cheaper and Faster

Here's a problem you might not know AI has: every time you ask a chatbot about a document, it's rereading that entire document from scratch. Every. Single. Time. Like you're handing someone a 50-page manual, asking "what's the warranty?", getting an answer, then immediately handing them the same manual again to ask "what's the return policy?" They'd look at you like you're ridiculous. But that's basically what happens with large language models.

IBM's Martin Keen just dropped an explainer on prompt caching—a technique that's solving this exact inefficiency. And honestly, it's one of those things that feels obvious in hindsight but requires understanding how these models actually work to appreciate why it matters.

What Prompt Caching Actually Is (And Isn't)

First, let's clear up the confusion. Prompt caching is NOT the same as output caching. Output caching is what you'd expect: someone asks "What's the capital of France?" the AI answers "Paris," and when the next person asks the same question, you just serve up the cached answer without bothering the model. Standard stuff. Database behavior.

Prompt caching is different. It's about caching the input side—specifically, the computational work the model does to understand your prompt in the first place.

When you send a prompt to an LLM, the model computes what are called key-value pairs (KV pairs) at every transformer layer for every token you send in. As Keen explains, "We can think of these KV pairs as the model's internal understanding of your prompt. So how every word relates to every other word, what context matters, what information to focus on."

This is the "prefill phase"—all the work that happens before the model generates even its first word of output. For a simple prompt like "What's the capital of France?" this processing is basically free. But for a complex prompt with a 50-page document embedded in it? That's thousands of tokens across dozens of transformer layers. Millions of operations before the AI can start typing.

Prompt caching saves those precomputed KV pairs. So when you send that same 50-page document with a different question at the end, the system recognizes the document, grabs the cached KV pairs, and only processes the new question. "That's a pretty notable saving in latency and cost," Keen notes.

What Actually Gets Cached

The most common use case is system prompts—those personality-defining instructions every chatbot has. "You're a helpful customer service agent, blah blah blah. That sort of thing can be cached," Keen says. Makes sense. Why reprocess the AI's entire personality description every single time someone asks a question?

But the technique works for any static content:

• Long documents (product manuals, research papers, legal contracts)
• Few-shot examples (when you show the model how to format responses)
• Tool and function definitions
• Conversation history in ongoing chats

Basically, anything you're going to reference multiple times without changing it.

The Structure Problem Nobody Tells You About

Here's where it gets interesting: prompt structure matters way more than you'd think. The cache system uses something called "prefix matching"—it reads your prompt from the beginning, token by token, and stops caching the moment it hits something different from what's already cached.

So if you structure your prompt with the static stuff first (system instructions, then document, then few-shot examples, then user question), the cache matches through all that static content and only processes the new question.

But flip it around—put the dynamic question first—and the cache fails immediately. You'd reprocess everything.

Keen walks through this with a clear example: "Well, this structure puts all of the static content first. So when the next request comes in with just a different question, like now it's going to say, 'What's the return policy?' The cache matches through all of this static content here... and we only need to process the new question at the end."

It's such a simple optimization but it requires knowing this is even a thing. How many developers are accidentally killing their cache efficiency just by putting their question at the top?

The Fine Print

There are some practical constraints worth knowing:

Size matters. You typically need at least 1,024 tokens before caching provides any benefit. Below that threshold, the overhead of managing the cache exceeds the savings. So don't cache "What's your name?" type prompts.

Caches expire. Most clear after 5-10 minutes, though some providers let them hang around for up to 24 hours. This makes sense—you want fresh data, and you don't want stale cached information lingering forever.

Implementation varies. Some providers handle caching automatically. Others require you to explicitly mark which parts of your prompt should be cached in your API calls. Check your provider's documentation because the behavior isn't standardized yet.

Why This Matters for Real Applications

The cost implications are straightforward. If you're running a customer service chatbot that references the same product manual thousands of times a day, you're currently paying for those thousands of reprocessing operations. With prompt caching, you process it once (or once every few minutes as the cache refreshes), then only pay for the lightweight question processing.

The latency impact might be even more important for user experience. Nobody wants to wait while an AI rereads an entire document just to answer "What's on page 37?" Prompt caching turns that into a fast lookup instead of a heavy computation.

For document Q&A, research assistance, chatbots with extensive system instructions, or any application where you're repeatedly working with the same base content, this is kind of a no-brainer optimization.

What's Not Being Said

Keen's explanation is clear and useful, but there's an underlying question: why is this even necessary? The fact that LLMs need this optimization reveals something about their fundamental architecture. They're not actually "reading" documents the way we think of reading. They're computing relationships between every token and every other token, which is computationally expensive and doesn't naturally carry over between requests.

Prompt caching is essentially bolting on a memory system to an architecture that doesn't inherently have one. It works, it's useful, but it's also a bandaid on a deeper architectural reality.

The variance in implementation across providers also suggests this is still early days. When everyone's doing it differently, the pattern hasn't stabilized yet. That could mean better solutions are coming, or it could mean this is just always going to be a thing you need to manually optimize for.

But for now, if you're building anything with LLMs that involves repeated reference to the same content, prompt caching is probably worth your time to understand and implement. Because paying to reprocess the same 50-page manual 10,000 times a day is exactly the kind of inefficiency that makes AI feel expensive—when it doesn't have to be.

—Tyler Nakamura, Consumer Tech & Gadgets Correspondent