How Prompt Caching Cuts AI Agent Costs

Here's a question worth sitting with before you deploy your next AI agent: do you actually know what you're paying for?

Not the monthly bill number. The mechanics of it. Because if you're running multi-turn coding sessions with any of the major frontier models and you haven't thought carefully about prompt caching, there's a good chance you're hemorrhaging money in a way that would be embarrassing in retrospect.

Alejandro AO, writing for the Hugging Face channel, put together a video that gets at this precisely—and the core insight is one of those things that feels obvious the moment someone explains it, but apparently isn't obvious enough, because a lot of people are getting it wrong.

The compounding cost problem nobody talks about

Start here: LLMs don't have persistent memory. Every single time your agent responds to you, it reprocesses the entire conversation history from scratch. That system prompt you wrote. Every tool call. Every result. Every prior message. The whole stack, resent and recomputed.

So if you've accumulated 50,000 tokens in a session and you send a new 1,000-token query, the model isn't processing 1,000 tokens. It's processing 51,000. Then when you follow up, it's 55,000 or more. The cost isn't linear—it compounds. A session you think of as "50K tokens" might actually involve several times that many tokens being processed and billed.

"If you keep doing this," AO explains in the video, "a session of 50K tokens is not really 50K tokens that are spent in your LLM. It's going to be 50K plus 51K plus 54K plus 55K, and that can go exponentially very, very quickly if you are paying full price for these tokens."

Some APIs (like OpenAI's Responses API) paper over this by making it look like the model remembers—but as AO points out, that's a UI abstraction, not a memory mechanism. The reprocessing is still happening; you just can't see it.

What prompt caching actually is (not what you're picturing)

The word "caching" carries baggage from software engineering. If you've dealt with database query caching, your mental model is probably: send query → get result → store result → next time, skip the query and serve the stored result.

That is not what prompt caching is. Caching the outputs of an LLM is not the goal here. The outputs are the whole point—you need fresh ones every time.

What gets cached instead is the input processing. More specifically: the key-value (KV) computations that happen when the model first reads a chunk of tokens. When the same prefix of tokens appears again in a subsequent request, the provider can skip recomputing those KV states and pull them from cache instead. You still get a fresh response. You just don't pay full price to re-read context the model has already digested. IBM's Martin Keen covers the underlying KV mechanics well in his own breakdown of prompt caching—worth reading alongside AO's implementation-focused take.

The savings are substantial. According to AO, cached reads can drop to around 10% of the standard input token price with some providers—meaning the 90% of tokens you're re-sending on each turn cost a fraction of what the fresh tokens do. He demonstrates this with a simulated 200K-token session across multiple providers, showing costs collapsing dramatically once caching is enabled. Without it, the price curve climbs exponentially as context grows. With it, the curve flattens toward something much more manageable.

Where it breaks down

This is where things get interesting, because the gap between "caching enabled" and "caching actually working" is wider than most people realize.

Provider behavior isn't uniform. OpenAI and Hugging Face inference providers cache automatically. Anthropic and Gemini require you to explicitly enable caching in your API calls. If you're building an agent harness and you haven't checked which mode your provider uses, you may think you're caching when you're not.

Cache expiry is a real operational concern. Different providers have different windows before cached states expire. OpenAI's window is longer; Anthropic's default is shorter (the exact duration can vary—check your provider's current documentation, since these numbers update). AO illustrates this with a charming detail from his own session: "Right here is where I went for lunch. So right here, my cache expired." He paid full price when he resumed. Then later, dinner, same thing.

This isn't just inconvenient—it's a reminder that cache optimization isn't a one-time config and forget. Long sessions with breaks will incur cache-miss costs at the resumption points.

Dynamic system prompts are a silent killer. This is probably the most under-appreciated footgun in agent design. If your system prompt includes anything that changes between turns—a timestamp, the current working directory, a dynamically generated tool list—every single request blows the cache for everything that follows that changed element.

Think about that. A 10,000-token system prompt with a timestamp at the end means none of those tokens benefit from caching, ever, because the prefix is never stable. AO is direct about this: "You do not want dynamic things in your system prompt such as timestamps, current working directories... do not add them to your system prompt if you do not want to make your sessions crazy expensive."

The cache requires a stable prefix. Anything that modifies the prefix invalidates everything downstream of the modification point.

Compaction resets the clock. Context compaction—where you summarize a long conversation to free up context window—is a legitimate technique, but it implicitly busts the cache. The new compacted summary is a different token sequence than the history it replaced, so you start fresh. AO notes this isn't a bug so much as an expected tradeoff: "That's exactly how compaction is supposed to work." Just know that your next several turns after compaction will be billed at full input price.

Building for cache-friendliness

The practices AO recommends flow pretty directly from understanding the constraints:

Keep your system prompt static across turns. If your agent needs dynamic context, find a way to inject it as a user message rather than modifying the system prompt. Your conversation history should be append-only—never modify prior messages, only add new ones. Know your provider's caching semantics before you build, not after you get the bill.

And critically: monitor your cache hit rate. AO demonstrates this with Tau, the Hugging Face coding agent (a Python port of the Pi agent), which exposes cache hit metrics both per-request and for the full session. Seeing 10.9 million tokens exchanged in a session for a handful of cents is the kind of number that makes you want to make sure you never lose that cache. Having that visibility is what makes optimization concrete rather than theoretical.

The fact that this monitoring isn't standard across all tooling is its own small critique of the current developer experience. You shouldn't have to build your own instrumentation to know whether you're getting what you're paying for.

The asymmetry worth naming

There's something a little uncomfortable sitting under all of this: the same technique that saves sophisticated users money is invisible to everyone else. If you know about cache-friendly harness design, you pay a fraction of the cost. If you don't, you pay full price—possibly without knowing it—for every token you've ever sent, over and over again.

The providers aren't hiding this. The documentation exists. But the defaults vary, the implementation details differ, and the failure modes are silent—your agent works either way, the bill is just much higher in one scenario.

That asymmetry tends to favor teams with dedicated engineering resources over individual developers and smaller shops, which is worth keeping in mind as AI agent adoption spreads into less infrastructure-savvy contexts.

For now, the knowledge is accessible enough that it doesn't have to stay that way.

---

Yuki Okonkwo covers AI and machine learning for Buzzrag.