FreeToken vs llama.cpp: A Local AI Engine Reality

One graphics card. One desk. A model with 753 billion parameters running on it — no cluster behind it, no rented node in somebody's data center. Fifteen tokens a second, by the research team's own measurements. Slow enough that you notice, fast enough that you'd sit and watch it work.

That is the opening claim of FreeToken, a new inference engine out of UC Berkeley's Sky Computing Lab (arXiv:2608.16157), carrying names that carry weight: Ion Stoica, who co-wrote the paper behind vLLM — the engine underneath a great deal of the inference you pay for today — alongside Matei Zaharia, Song Han, and Shuo Yang. This is not a weekend project. The people who built the cloud serving stack have turned around and aimed it at your desk.

What they built has to beat llama.cpp: Georgi Gerganov's engine, three and a half years old, the thing most people actually run. FreeToken wins the benchmark, and it wins it clearly. The benchmark is not the interesting part.

The Sparse Trick That Makes It Possible

The "how does 753 billion parameters fit on one card" question has a structural answer. These models are mixture-of-experts architectures. Instead of one enormous feed-forward layer doing all the work, each layer holds many smaller specialist networks, and a router selects a small handful per token. The result — and this is the part that changes everything about the hardware problem — is that only a small fraction of the model's parameters are ever active for any given token. The FreeToken paper puts it plainly: under 5% of the model is awake at once.

Sparse compute is not sparse memory, though. All those parameters still have to live somewhere the card can reach in milliseconds, because you cannot know which specialists the next token will need. They sit in system memory. The card holds what it can. Every time a token routes to a specialist that isn't already on the card, that specialist has to cross the memory bus. That crossing is the entire engineering problem. Every design decision in either engine is a different answer to one question: what do you do in the microsecond after a token asks for an expert you don't have?

Why llama.cpp's Answer Falls Short

llama.cpp has had an answer for a while. Its --n-cpu-moe flag lets you keep specialist weights on the CPU rather than shipping them back and forth across the bus. The documentation for it is one line long, which is unusual — except that one line is also the entire strategy: keep the mixture-of-experts weights of the first N layers on the processor.

The first N layers. That phrasing does all the work, and it's the easiest thing in the sentence to skim past. It's a split by layer number, chosen before the model has seen a single token. Fixed in advance. Static for the entire run. Which would be fine if routing were fixed — but routing is not fixed. Routing changes token by token, decided at runtime, long after you picked your number. A split by layer index cannot see any of that.

It is not a bad heuristic. It is a fast, predictable, sensible heuristic. It is simply blind to the one signal that decides everything.

The FreeToken paper measures what that blindness costs, and it measures it the right way: by replaying identical routing traces through all three engines' placement policies at the same cache size. Same traces, same memory, different policy. FreeToken's cache misses 16% of expert reads. The static split misses 62%. A 46-point gap. Nearly four times the miss rate. And a miss is not a rounding error — a miss is the bus, every single time.

Which raises a question worth holding: if the fix is that legible, why hasn't llama.cpp simply done it?

What the Benchmarks Say — and How to Read Them

According to the FreeToken paper (arXiv:2608.16157), on a 5090 serving a 35B model, FreeToken reaches 77 to 83 tokens per second — between 1.88 and 2.3 times the strongest engine it tested against. On DeepSeek's V4 Flash, 22 to 25 tokens per second, 1.5 to 1.9 times the competition. The one that stays with me is the laptop result: 8 GB of card memory, a half-width bus, 39.3 tokens per second — 92% of what a desktop 4090 manages, per the same paper. That is the machine most people actually own.

The caveat deserves to be said plainly rather than buried: every one of those numbers was produced by the people who built the engine. That is normal for a systems paper. It is still not independent verification. To their credit, the team ran the rigorous version — every engine served bit-identical weights, which is the step most benchmark comparisons skip without saying so, and it is the difference between a benchmark and an advertisement. But as of this writing, no third-party reproduction of FreeToken's results exists anywhere I can find.

The result I'd actually make a decision on is the tail, not the throughput. Not how fast it goes when it goes well — how slow it gets when it goes badly. The paper reports FreeToken's worst single turn staying under 44 seconds. llama.cpp crossing 232 seconds somewhere in the distribution. KTransformers approaching 16 minutes on one turn. The paper's point here is sharper than a speed comparison: OpenClaw ships a two-minute idle watchdog. A turn that takes 232 seconds doesn't run slowly. It gets killed. Tail latency is not a latency statistic in that context. It is whether the software works at all.

The Denominator Problem

The paper draws a dashed line across its headline throughput chart marking what a cloud coding agent actually delivers, derived from real production traces: 33 tokens per second, against FreeToken's bars at 77 and above. That reads like a rout.

Read the trace paper it cites (arXiv:2606.30560) and the picture shifts. The 33.9 figure is a normalized decode speed that, by that paper's own definition, includes time to first token and all the reasoning in between. It is an end-to-end number. The same paper's pure decode median — the figure that actually belongs next to a decode throughput bar, because it measures the same thing — is 61.3 tokens per second. Put those two side by side and the like-for-like ratio is about 1.3, not 2.4. Still a win. A much smaller one than the chart implies.

Both figures sit in the source, correctly cited. This is not deception. But a comparison is only as good as its denominator, and this one reshapes the size of the claim by nearly half. That is the kind of presentation choice that decides whether you trust the rest of the chart.

The Hardware Reality

None of this matters much if you cannot install it. In the 24 hours after the launch post went up, the GitHub repository accumulated eight open issues: support for older hardware, dual-GPU configurations, Docker, GGUF format, a Windows installation failure — and the one with the most comments: a feature request for macOS Apple Silicon support.

Every item on that list is something llama.cpp already ships. The project supports a wide range of hardware backends — AMD, Vulkan, Apple's Metal, even a Snapdragon phone backend — with quantization running from 1.5 bits to 8. Apple Silicon is described in llama.cpp's own readme as a first-class citizen.

FreeToken's own packaging is admirably unambiguous about where it stands. Its published classifiers read: development status beta, environment Nvidia CUDA, operating system Linux. The download page offers Windows, Ubuntu, Arch, and an AppImage. It hands a Mac visitor nothing. Not a slow fallback. Nothing.

That is not a criticism of an eleven-day-old project written by two contributors. It is a description of what you can and cannot run today. The gap between llama.cpp's 446 contributors and FreeToken's current public history is not a flaw in FreeToken — it is the natural condition of something very new up against something very established. But it is the gap you cross before the benchmarks become relevant to you.

The Pricing Argument, Honestly Stated

The launch framing promises you can run a frontier coding agent for zero dollars. A cleaner version of that argument, priced from a real dataset: researchers published 4,200 real coding agent sessions (arXiv:2606.30560). The median step reads roughly 119,000 tokens of context and writes around 214. That step costs about 11 cents on Claude or OpenAI's Codex. The same step on DeepSeek's own published pricing for the exact model FreeToken demonstrates runs somewhere between a tenth of a cent and five cents, depending on caching and time of day.

The model you would buy a graphics card to run at home is also the cheapest model on the commercial menu. A tracked RTX 5090 listing sat above $4,000 all through July against a $2,000 list price. That $4,000 buys somewhere between 500 and tens of thousands of median agent steps depending on which API you were comparing against.

The invoice was never the real argument, and the video says so directly: "Nothing you type leaves the room. Nothing rate limits you at the worst possible moment. Your model does not get deprecated out from under you six months into a project built on it." None of that shows up on an invoice. All of it is real.

Who This Is Actually For

llama.cpp runs local AI better for most people today, and it is not a close call. The first thing an engine has to do is run on the hardware you already own. That is the one benchmark nobody plots.

FreeToken wins outright for a specific person: one who owns a recent Nvidia card, has ample system memory, runs mixture-of-experts models, and points a coding agent at them continuously. For that person, 44 seconds against 232 is not a percentage on a chart. It is the difference between an agent that finishes its task and one that gets killed halfway through.

The paper itself cites prior work that sets up the sharpest question in this space: Fiddler proposed treating a missed expert as compute rather than data transfer. KTransformers made that fast. Neither became a household name. Both of their ideas are now table stakes inside the engines everyone uses.

If the winning move in open source is always to fold the good new idea back into the established engine — if FreeToken's routing-aware cache eventually lands as a merged pull request in llama.cpp rather than as a standalone project with 10,000 stars — what exactly is left for anyone who builds a new one?

By Bob Reynolds, Senior Technology Correspondent, BuzzRAG