Stealing AI Reasoning: How LLM APIs Leak Secret

Here's a fun thought experiment: imagine a bank vault where the door is locked, but you can still reach through a side vent and grab the cash. That's roughly what a new research paper is describing about the inner workings of AI APIs from some of the biggest names in the industry — and it's a more uncomfortable finding than the AI labs would probably like to admit.

The paper, published on arXiv, identifies what researchers call a structural vulnerability in how providers like OpenAI, Anthropic, and Google handle their models' chain-of-thought reasoning — the internal step-by-step thinking that frontier models work through before producing a final answer. These traces have become incredibly valuable. They're the secret sauce in modern reasoning models, and providers have spent considerable effort hiding them.

The architecture of the problem

So how did the providers try to protect this intellectual property? According to the arXiv paper, rather than keeping reasoning traces entirely server-side, providers return them to the client as encrypted blocks. The logic is intuitive: encrypt the output, hide the contents, protect the IP. Problem solved, right?

Not quite. As alphaXiv's summary of the research explains, "researchers uncovered an architectural vulnerability in major proprietary large language model APIs where encrypted internal reasoning traces can be extracted in plaintext by weaker models within" the same API ecosystem. The key move is clever in a way that's borderline elegant if you're not the one being robbed: take a trace produced by a frontier model, replay it into a weaker sibling model in the same family, jailbreak the weaker model, and recover the stronger model's hidden reasoning — in plaintext.

Developer and AI analyst Simon Willison documented this on his site, noting that you can actually observe encrypted reasoning blocks directly by querying OpenAI's API. The attack works precisely because the encrypted blocks travel to the client instead of staying locked up server-side. Once they're in your hands, even in encrypted form, the replay-and-jailbreak pathway becomes viable.

alphamatch.ai's writeup characterizes what an attacker can actually recover: "hidden reasoning traces from proprietary LLM APIs — information that was never meant to be visible." And according to the Hugging Face paper page, the extractable material goes beyond just model reasoning. Encrypted traces "shared across sessions and models can be intercepted and injected into weaker models to extract proprietary reasoning, private data, hidden hazards, and hidden prompts." That last one — hidden prompts — is particularly thorny. System prompts are often where enterprises embed their most sensitive business logic.

This isn't just an IP story

It's tempting to frame this as a corporate espionage story. Company A reverse-engineers Company B's proprietary model. Competitive drama, lots of money at stake, AI arms race intensifies. That framing isn't wrong, but it undersells the broader surface area.

Think about what's actually traveling inside these traces in a real-world deployment. Enterprise customers build elaborate system prompts containing internal procedures, customer data handling rules, and sometimes actual customer data fed into context windows. If those context windows bleed into reasoning traces — and there's no particular reason they wouldn't — then the attack surface includes private user data, not just proprietary model weights.

This connects to a broader catalog of LLM vulnerabilities. OWASP's updated look at LLM attack surfaces flags how easily AI systems can be manipulated into leaking sensitive data through indirect channels — and this research is essentially a worked example of that threat category made concrete.

The audit log problem

There's a second, maybe underappreciated wrinkle here, and it comes from a different direction entirely. A piece from Rye.ai titled "LLM Reasoning Traces Are Not Audit Records" pushes on something that security people in regulated industries are going to care about a lot: reasoning traces are not necessarily faithful records of how a model actually reached its output.

This matters because one instinct, when you hear that reasoning traces can be extracted, is to think: "Well, at least we can audit what the model was thinking." But if the traces aren't reliable representations of actual model reasoning — if they're more like post-hoc rationalizations generated alongside the output rather than a true computational log — then extracting them tells you something, but not necessarily the thing you wanted to know. You might be stealing a performance of reasoning rather than the real thing.

That's not a knock on the security research, which stands on its own merits. But it does complicate what we think we'd gain from access to these traces, which in turn complicates the policy conversation about what "transparency" in AI reasoning actually means. If a model's chain-of-thought is simultaneously secret, stealable, and potentially unfaithful to the actual computation — where does that leave us in terms of accountability?

The structural fix is unclear

What's notable about this class of attack is that it's not a bug in the traditional software sense. It flows from an architectural decision — return encrypted traces to the client rather than store them server-side — that probably seemed reasonable when it was made. Client-side delivery is faster, reduces server-side storage requirements, and lets the model API remain stateless. Perfectly sensible engineering choices that happen to create an attack surface.

Fixing it cleanly isn't obvious. Keeping traces entirely server-side solves the replay problem but reintroduces storage and statefulness costs. Stronger encryption buys time but doesn't change the fundamental dynamic: if you're returning encrypted content to clients, motivated adversaries will eventually find ways to decrypt or circumvent it. The weaker-sibling attack doesn't even require breaking the encryption — it just routes around it.

The providers named in the research — OpenAI, Anthropic, Google — haven't publicly detailed their response to these findings, at least not in the sources available at the time of writing. That's not unusual for active security research; responsible disclosure processes take time. But it does mean that developers building on these APIs right now are operating without clarity on whether or how this is being addressed.

What builders should actually do with this

If you're building on LLM APIs — especially if you're embedding sensitive system prompts or feeding user data into context — the practical takeaway isn't to panic but to audit your threat model. Specifically: what happens if your system prompt leaks? If the answer is "nothing good," you probably want to be thinking about prompt compartmentalization, minimizing sensitive data in model context, and watching for updated guidance from your API provider.

The research also reinforces something that the AI industry has been slow to internalize: security isn't a layer you add to AI systems. It's something that has to be baked into architectural decisions from the start. Encrypted client-side traces sounded secure until someone thought carefully about what you can do with an encrypted blob and a weaker sibling. That's the kind of adversarial thinking that needs to be in the room before the architecture is finalized, not after the paper drops.

The vault analogy from the top of this piece isn't quite right, actually. It's more like a bank that hands you a locked safety deposit box to carry home, then discovers that a locksmith down the street can open it. The lock was real. The threat model wasn't.

Zara Chen covers tech and politics for Buzzrag.