Llama.cpp Gets MTP: Local AI Just Got Faster

Every time you type a prompt into ChatGPT, Claude, or Gemini, that text leaves your machine. It travels to a data center, gets processed alongside however many million other requests, and lives—however briefly, however "anonymously"—in logs that are governed by a privacy policy you almost certainly haven't read in full. The company can update that policy. They can get acquired. They can get breached. You are, functionally, trusting a stranger with your curiosity.

That's the actual reason local AI inference matters to me. Not the nerd satisfaction of running a model on your own hardware, not the cost savings, not the offline capability—though all of those are real. It's that when the model runs on your machine, your prompts are yours. No retention schedule. No terms-of-service roulette. No "we may use your inputs to improve our services." The conversation stays between you and your CPU.

The honest friction has always been speed. Cloud models are fast because they're running on server farms with hardware that costs more than most people's cars. Running a capable model locally has meant accepting a performance gap that, for a lot of people, makes the privacy trade-off feel like a downgrade. That's the gap that just got meaningfully smaller.

This week, llama.cpp—the open-source engine that powers most local AI setups, from AnythingLLM to LM Studio—merged support for Multi-Token Prediction (MTP). Tim Carambat, founder of AnythingLLM and one of the more reliable voices covering local AI in practice rather than in press releases, tested it and got roughly a 23-25% speed increase on his M4 Pro with zero measurable accuracy loss on his benchmarks. On better hardware, gains can reportedly reach 2x. The floor is a meaningful bump. The ceiling is substantial.

If you've been on the fence about whether local AI is "good enough," this is the update worth paying attention to.

What MTP Actually Does

You don't need to fully understand the mechanism to use it, but it helps to have a mental model.

Traditional inference is sequential: the model generates one token, then the next, then the next. Speculative decoding—the technique MTP builds on—attempts to predict several tokens ahead using a smaller "draft" model, then lets the main model verify or reject those predictions in bulk. When the draft model guesses right, you get a speed boost. When it guesses wrong, you pay a penalty.

The problem: running two models simultaneously doubles your memory footprint and introduces a second point of failure. It's a real hardware ask, and on consumer machines it often doesn't pencil out.

MTP collapses this into a single model. The prediction capability is baked into the architecture itself—no separate draft model required. As Carambat explains it: "You can just run a single model and enable MTP and then you're fine." The complexity disappears from the user's side.

The configuration knob that matters is --draft-max, which controls how many tokens ahead the model predicts. This is where Carambat's benchmarks get instructive. On his setup, with a Qwen 9B model:

• No MTP: 45 tokens/second
• --draft-max 1: 55.38 tokens/second (~23% gain)
• --draft-max 3: 49 tokens/second (~8% gain)
• --draft-max 6: 28 tokens/second (a loss)

The lesson is immediate and counterintuitive: more aggressive prediction is not better prediction. Set it too high and you're slower than where you started. Start at 1, measure, and only go higher if your specific hardware rewards it.

Which Models Support It (And What's Unresolved)

Not every model has MTP baked in, and the landscape here has some rough edges worth flagging.

DeepSeek V3—released in late December 2024—is widely credited as one of the first models to ship with MTP architecture. Carambat mentions "V4 base and flash models from DeepSeek" as also supporting it, but I'd flag that with a caveat: DeepSeek's publicly confirmed releases as of this writing are V3 and the R-series reasoning models; any V4 claim should be verified against current HuggingFace listings before you go hunting for files.

NVIDIA's Nemotron 3 Super and Ultra models (note: Nemotron, not "Neotron") also support MTP. These are large models—not casual laptop fare—but relevant if you're running on beefier hardware.

The community naming around Qwen models is worth a note: Alibaba's official versioning runs Qwen2.5 and Qwen3, so references to "Qwen 3.5" and "Qwen 3.6" in the local AI community appear to be informal shorthand or model-card naming rather than official Alibaba version designators. If you're pulling files from HuggingFace, check the model card directly; the Unsloth-quantized GGUFs are a reliable starting point.

Google's Gemma 4 models are the interesting outlier: MTP is apparently present in the architecture but was never publicly documented, got discovered accidentally by the community, and as of Carambat's video still hasn't been published in an active form. The capability exists; the tooling to use it doesn't. Google hasn't explained why.

Mixture-of-Experts (MoE) architecture models—like some of the larger Qwen variants—may not see the same gains as dense models. Carambat's guidance: try it, don't expect much, and don't read anything into it either way.

The Setup Tax

Here's the annoying part, and it's genuinely annoying: if you already have a Qwen model downloaded, you can't just flip a flag and unlock MTP. The MTP-enabled GGUF is a different file. New tensors, new ops. Load an MTP model into an older llama.cpp build and it crashes. Load your existing GGUF into the new build and it works fine—backwards compatible—but you don't get MTP.

You need two things: the latest llama.cpp build (the PR that added MTP is #22673, hosted at github.com/ggml-org/llama.cpp—though I'd note the canonical llama.cpp repository has historically lived under the ggerganov org on GitHub; verify the current location before pulling), and an MTP-variant GGUF of your model.

Carambat frames this as a temporary friction: "The next model series that comes out, if it has MTP, they're likely not going to have with and without MTP. It'll just have MTP." Going forward, this should be invisible. Right now, it requires a deliberate download.

The flags you're adding to your llama-server command are --draft-mtp and --draft-max N. That's it. If you're already comfortable launching llama.cpp from the command line, this is a ten-minute upgrade.

What This Means If You've Been Waiting

Cloud AI is fast and frictionless and I understand why most people use it. I also cover what happens when companies get breached, change their terms, or quietly expand how they use input data—and I can tell you the answer is "more often than you'd like." The practical counterargument to local AI has never been about capability; recent models are genuinely impressive. It's been about the experience feeling sluggish enough that the privacy benefit doesn't feel worth the trade-off.

A 23% speed increase doesn't close the entire gap between your laptop and a data center. But it moves the line. And MTP is one improvement among several happening simultaneously—quantization techniques, context cache optimizations, one-bit model research—that are collectively making local inference feel less like a compromise and more like a viable default.

"If I can get that intelligence for free on the computer I already own," Carambat says, "yeah, that's worth it to me."

The privacy case doesn't require you to be paranoid. It just requires you to notice that "free" has a different meaning when the product isn't running on your machine. MTP won't make local AI perfect. But it makes the honest choice a little easier to live with.

---

Rachel "Rach" Kovacs is Buzzrag's cybersecurity and privacy correspondent.