Liquid-Cooled AI Cache Server Built From Off-the-Shelf Parts

The hardware to run serious AI infrastructure is democratizing faster than the software to use it well. That gap — between what you can now bolt together in a 2U chassis and what the tooling can reliably deliver — is exactly what a recent build by YouTube creator Raid Owl ends up documenting, probably more honestly than intended.

The project starts with a straightforward question: what happens when you stop storing an AI model's working memory on the GPU and move it somewhere else entirely? To answer it, Raid Owl built a liquid-cooled flash storage server from off-the-shelf components — 16 NVMe slots, 100-gigabit networking, an eight-core server-class CPU, and a water-cooling loop running through the drives themselves — then connected it to a cluster of Nvidia DGX Spark machines and ran experiments. The hardware story is genuinely impressive. The software story is considerably messier.

First, the machine

The build lives inside a 2U Sliger chassis, which Raid Owl describes as "pretty sure the only 2U case that met my requirements" — four 5¼-inch bays and room for a water-cooling setup. Two of those bays hold NVMe drive cages from Vision3, a Taiwanese manufacturer Raid Owl spotted at Computex. Each cage holds eight drives; one handles M.2 form factor, one handles E1.S. Every drive gets its own dedicated PCIe Gen 5 bandwidth through direct connections to the motherboard, and each bay presses the drives against a water-cooled plate.

The drives themselves are Solidigm D5-P5430 E1.S units, fifteen terabytes each, totaling roughly 120 terabytes of flash storage. Solidigm sponsored the build, and the drives are data-center QLC (quad-level cell) — a type of flash that stores more data per cell than traditional enterprise drives, trading some write performance and longevity for higher capacity and lower cost per terabyte. Raid Owl is clear about their place in the market: "yes, I know these are way too expensive for a home lab, but for data centers, not bad at all."

The cooling system runs dual pumps — redundancy being a lesson learned the hard way, apparently — and the thermal results are striking. Under sustained drive stress testing, the E1.S units barely cleared the mid-40s Celsius, and during actual caching workloads they stayed below 40 degrees. For context, dense NVMe configurations typically require fans drawing significant power and still produce worse results. These are also standard drives, not the purpose-built liquid-cooled variants Solidigm makes for direct cold-plate applications. The entire system idles at around 130 watts and peaks near 170 watts under load — modest numbers for the storage density involved.

What KV cache actually is

Before getting to the experiment, it's worth explaining the thing being tested, because "KV cache" is one of those terms that gets thrown around in AI coverage as though its meaning is self-evident.

When a large language model generates text, it processes your input word by word (technically token by token) and builds up a running understanding of everything said so far. It stores that understanding as a set of numerical values — the keys and values, hence KV — so it doesn't have to re-read the entire conversation from scratch every time it produces the next word. Think of it as the model's short-term working notes on the conversation. Without it, generating each new word would require reprocessing everything from the beginning, which would be both slow and computationally expensive.

The problem is that these notes take up space, and that space comes from the GPU's memory — the same memory the model's weights are loaded into. For long conversations or many simultaneous users, the cache can consume more GPU memory than the model itself. The question Raid Owl wanted to answer: can you move that cache somewhere else and preserve the GPU memory for actual computation?

The tiered experiment

The setup used LM Cache, an open-source tool that runs alongside vLLM (the software most commonly used to serve AI models) and adds the ability to store KV cache on remote hardware. Raid Owl configured four tiers. L0 was local GPU memory on each Spark, kept intentionally small. L1 was peer-to-peer sharing between Spark nodes — one machine pulling cached context directly from another over the network. L2 was the storage server's system RAM, managed through Redis (an in-memory database) and a compatible alternative called Valkey. L3 was the NVMe pool itself, accessed over the network via a protocol called NFS over RDMA — essentially a high-speed remote file system.

The theory was that the system would cascade through these tiers in order, using the fastest available cache first. The practice was more complicated. Initial tests suggested L1 peer-to-peer sharing was performing similarly to the slower remote tiers, which looked suspicious. It turned out to be a silent failure: "it wasn't even using our level one P2P sharing," Raid Owl explains. "It was silently failing and just skipping that step." After fixing that, peer-to-peer performance jumped dramatically, as expected.

A second wrinkle emerged after the P2P fix. The tiering logic had a preference problem: once any cache block was written to the remote storage server — Redis in RAM or the NVMe pool — those tiers took priority over P2P retrieval, even when pulling from a nearby node would be faster. The software was not making optimal routing decisions, and Raid Owl acknowledges uncertainty about whether this is a bug or an intentional architectural choice.

The net result was that L0 (local GPU memory) was fastest, L1 (peer-to-peer) was meaningfully fast once working, and L2 and L3 (remote storage server) were roughly comparable to the time it would take to just recompute the cache from scratch. The remote RAM tier was slowed by TCP communication overhead — the standard networking protocol adds latency even on fast connections. The NVMe tier was slowed by NFS overhead, the file-system layer that manages remote storage access eating into the speed advantage of the fast drives.

Where the value actually is

So far this sounds like a qualified failure. But Raid Owl makes a pivot that reframes the whole exercise, and it's the most interesting part of the project.

Speed is only one dimension of the value proposition. The other is persistence. When cache lives in GPU memory or system RAM, a server reboot erases it entirely. Every update, every maintenance window, every crash means starting over — recomputing potentially gigabytes of context that users had already paid, in time and electricity, to generate. Raid Owl tested this directly: ran a standard set of model calls, shut the entire system down, brought it back up, and hit it with the same calls. The NVMe-stored cache returned a complete hit — no recomputation required, the context was simply retrieved.

"For the average user, this probably doesn't really matter," Raid Owl says, "but what about for companies or data centers that are running tons of servers for hundreds or thousands of users? That's a huge win." By the end of testing, more than a terabyte of cache had accumulated on the NVMe pool.

This is where the 120-terabyte Solidigm array starts to make conceptual sense, even if it's overkill for a home lab. The value isn't raw speed — the network and file-system layers prevent the storage from expressing its full capability at this tier. The value is durability and capacity: a place to park enormous amounts of computed context that survives the operational realities of running production infrastructure.

What it means

Raid Owl concludes that remote KV caching is too complex for casual home lab use — the existing software isn't yet polished enough to configure reliably — but plans to revisit LM Cache as the project matures. That's a fair assessment, and an honest one. The silent P2P failure alone would have been a confidence-destroying bug in any production system.

What the project actually demonstrates, though, is something broader. The gap between what you can assemble and what you can reliably orchestrate is real, and it runs through the entire AI infrastructure stack right now. The hardware — a 120-terabyte, water-cooled, 170-watt flash server built from parts with a parts list in a YouTube description — is no longer the hard part. Getting the software layers to coordinate cleanly, to make sensible routing decisions, to fail loudly instead of silently: that's where the work remains.

Builders like Raid Owl are running into the rough edges so that, eventually, everyone else doesn't have to.

---

Bob Reynolds is Senior Technology Correspondent at BuzzRAG.