Splitting One LLM Across Two Machines: Does It

Here's a premise that sounds good on paper: Take a GPU that's amazing at crunching through prompts but slow at spitting out tokens. Pair it with a Mac that does the opposite—slow prompt processing, fast token generation. Split one large language model between them. Profit?

Developer Alex Ziskind spent months trying to make this exact setup work, and the results are... honestly more interesting than a simple yes or no.

The Theory That Companies Actually Use

When you run an LLM, there are two distinct phases happening. First is prefill (sometimes called prompt processing)—this is compute-heavy, GPU-intensive work where the model digests your entire prompt. Second is decode, the token-by-token generation phase that's memory bandwidth-heavy. Most of us run both phases on one machine because, well, that's how it works.

But disaggregated prefill and decode—splitting these phases across different machines—isn't just some hobbyist fantasy. Companies like DeepSeek and ByteDance already do this in production. It's one reason inference costs keep dropping. Each phase gets hardware optimized for its specific demands.

The Exo Project has been teasing consumer-grade disaggregated inference for months, complete with blog posts and animations. They just... never actually released it. So Ziskind, sitting there with an MSI Edge Expert (basically Nvidia's GB10 with a Blackwell GPU and 128GB unified memory) and a Mac Mini M4 Pro (64GB unified memory), decided to see if he could make it work himself.

Spoiler: He's a web developer, not a systems programmer. This was going to hurt.

When Theory Meets Rust Networking Code

Ziskind pointed Claude at an experimental pull request adding Blackwell support for disaggregated inference, gave it SSH access to both machines, and said make it happen. What followed was days of compilation, building vLLM from source, compiling CUDA kernels, installing dependencies, and watching dashboards load before everything broke.

The first major hurdle? The two machines couldn't see each other on the network. Exo uses mDNS for peer discovery, and apparently libP2P's mDNS implementation is broken on macOS. Hours of troubleshooting—direct Ethernet, USB adapters, Thunderbolt cables, modifying the networking layer—yielded nothing. The fix turned out to be simple once he found it: have the GB10 dial the Mac Mini directly instead of waiting for discovery. One environment variable later, instant connection.

Then came model loading. Ziskind ran Qwen 3.5 27B in BF-16 (Blackwell-optimized full precision) on the GB10, and the exact same model in 4-bit MLX quantization on the Mac Mini. Different quantizations on purpose—each machine running what it's optimized for. The GB10's 128GB of memory can handle full precision, which is faster for prefill. The Mac's 4-bit version is smaller, meaning faster decode.

When everything finally connected and he sent a long prompt to the Mac Mini, it routed prefill to the GB10. The Blackwell GPU chewed through tokens at 546-937 tokens per second depending on prompt length—up to 14x faster than the Mac Mini running locally. It worked.

Then he looked at end-to-end time and found the plot twist.

The Network Is The Bottleneck (Surprise!)

At 25,000 tokens, the GB10 computed the KV cache in under a second. Transferring it over Ziskind's 2.5GB USB Ethernet adapter? Twenty-five seconds. "In other words, 96% of the total time was just the network. The GPU was still idling, waiting."

This is where the experiment gets real. Ziskind upgraded to a 50Gbps Mellanox ConnectX-4 card (after discovering macOS won't recognize his Intel 100Gbps card but has built-in drivers for Mellanox since 2019—the little things you learn). KV cache transfer improved by 30%.

He also switched from Qwen 3.5 (a thinking model that generates hidden reasoning tokens at decode speed, masking the differences) to Llama 3.1 8B for cleaner benchmarks. With a faster network and a non-thinking model, the picture clarified: disagregated time to first token at 2.4 seconds essentially matched the GB10 alone at 2.3 seconds. Disaggregated decode hit 34 tokens per second—slower than the Mac Mini's 52 running solo, due to KV cache injection overhead.

So... was it worth it? Ziskind's honest answer: "A single RTX Pro 6000, the workstation Blackwell card, would probably demolish the entire two-machine setup on both prefill and decode. This thing has six times the memory bandwidth of the GB10 and 3.5 times the compute."

But there was one variable he hadn't changed yet.

The Mac Studio Makes It Interesting

The Mac Mini M4 Pro has 273GB/s of memory bandwidth. The M3 Ultra Mac Studio sitting nearby? 819GB/s—three times more. Swap the Mini for the Studio, and decode could theoretically jump from 34 to over 100 tokens per second. Combined with the DGX Spark's 1,700 tokens per second prefill, that might actually be worth building.

So Ziskind swapped in the Mac Studio (512GB unified memory) and ran the same tests. The Mac Studio decoded Llama 8B at 106 tokens per second—roughly double the Mini, though not the full 3x the bandwidth specs suggest. Disagregated decode hit 84 tokens per second, down from 106 due to KV cache overhead, but still 6x faster than the Spark alone.

Here's where it gets genuinely interesting: the performance story changes dramatically with model size. At 8B parameters, decode is almost purely bandwidth-bound—the Mac Studio's 3x bandwidth advantage dominates. But at 27B and 32B (Gemma 2 27B and Qwen 2.5 32B), the gap shrinks to 1.25-1.3x.

Why? Two things shift. For Gemma, sliding window attention caps how much KV cache decode needs to read per token—bandwidth stops being the bottleneck. For Qwen, vLLM kernel fusion and torch compilation on the Spark side dramatically cut bandwidth demand per decode step. Different mechanisms, same outcome: the Spark's decode gets relatively better, and the Mac's bandwidth advantage matters less.

"In every case, the disagregated time to first token tracks the Spark," Ziskind notes. The Spark-class prefill is always recovered. And the prefill advantage over the Mac Studio grows with model size—at 8B the Spark is barely ahead (1420 vs 1585 tokens/sec), but at 27-32B it's 2-2.5x faster.

Disaggregation becomes more valuable at larger model sizes, not smaller ones.

Does It Actually Work Though?

Yeah, it works. Two machines doing what they're good at, talking over a 50Gbps link, spitting out tokens faster than either could alone. The pitch delivers.

But should you do it? If you already own both machines—a DGX Spark and a Mac Studio—sure, squeeze that juice. As a proof of concept for heterogeneous inference on consumer hardware, this is genuinely cool. It demonstrates that the production techniques companies use can scale down to desktop setups with the right networking gear.

If you're spending new money though? The DGX Spark costs around $3,000-4,000. A Mac Studio with M3 Ultra runs $4,000-7,000 depending on config. You're looking at $7,000-11,000 minimum, plus high-speed networking hardware. An RTX Pro 6000 with 48GB VRAM costs roughly $6,000 and would likely outperform the entire two-machine setup on both phases.

The real value here isn't the specific hardware combination—it's understanding where the bottlenecks actually live. Theory said disaggregation would combine the best of both worlds. Reality said the network eats your lunch if you're not careful, that model size fundamentally changes the performance dynamics, and that bandwidth advantages don't scale linearly when kernel optimizations enter the picture.

Ziskind spent months making this work so we can see what happens when you actually try the thing everyone talks about but nobody ships. That's worth way more than another benchmark chart showing which GPU has the highest number.

Yuki Okonkwo is Buzzrag's AI & Machine Learning Correspondent