High Bandwidth Flash Promises a Lot and Delivers

The pitch sounds reasonable on its face: flash memory is cheap, models are enormous, GPUs are starved for capacity, so why not connect the dots? SanDisk and SK Hynix have formalized exactly that argument into a specification called High Bandwidth Flash, or HBF—a proposed standard that would bring NAND memory directly into the GPU fabric, promising up to 3 terabytes per second of bandwidth via the UCIe interconnect standard. It is a tidy solution to a real problem.

The trouble is that the problem it solves most elegantly is not quite the problem the industry actually has.

The Memory Gap Is Real

To understand why HBF exists, you have to start with the scale of modern AI models. Large language models today routinely reach two terabytes in size. That is a number worth sitting with for a moment: two terabytes, for a model that needs to stream its weights to a GPU core before it can do useful work.

A high-end GPU holds a fixed amount of on-board high-bandwidth memory. The H200, per Nvidia's own specification page, carries 141 GB of HBM3e. That is a lot of memory by most standards, and essentially a rounding error against a two-terabyte model. A standard SSD can hold the whole thing comfortably—but getting data from an SSD to a GPU over a conventional interface is painfully slow. A PCIe 3.0 x4 link, for instance, tops out at roughly 15.75 GB/s according to published interface specifications. Transferring a two-terabyte model across that connection does not take seconds. It takes minutes.

HBF's answer is to eliminate the long trip entirely. Rather than routing data through a conventional storage interface, the flash lives at the chip level, connected at far higher bandwidth. Caleb Eom, who covers semiconductor architecture on the Caleb Writes Code channel, frames the appeal plainly: "Because flash memory is cheap, we can easily support 2 TB of memory directly into the fabric of GPU."

That is the pitch. The bandwidth numbers are real. The capacity numbers are real. And yet Nvidia and AMD have responded with what Eom describes as "rather radio silence." That silence deserves explanation.

The Write Problem That Bandwidth Cannot Fix

Here is where physics reasserts itself. Flash memory—NAND, the kind in every SSD—was engineered for persistent storage. Its read performance can be genuinely impressive, especially when you gang enough chips together in parallel, which is exactly how HBF achieves its headline bandwidth figures. But NAND writes are a different matter entirely. Before you can write new data to a NAND cell, you have to erase it first. That is not a software limitation or a design choice that can be engineered away. It is how the medium works.

DRAM, by contrast, was built for working memory. Reads and writes happen at comparable speeds, and the latency on both is low enough to keep pace with a GPU core running at full tilt. The performance asymmetry between NAND reads and NAND writes is not a minor footnote—it is the central fact that shapes everything downstream.

This matters enormously for inference, which is where the memory capacity problem shows up most acutely. During inference, a model generates new tokens one at a time, and with each token it produces, it adds to something called the KV cache—a running record of intermediate attention computations that the model needs to reference as context grows. KV cache is not static. It expands continuously as a conversation or task grows longer.

Eom identifies this dynamic directly: "Using NAND flash for KV cache means write operation is more dominant than read since KV cache tends to balloon as the model generates more tokens."

In other words, the exact workload that inference generates—constant, growing writes—is the workload NAND handles worst. The 3 TB/s headline figure reflects read performance achieved through massive parallelism. The write story is considerably less impressive. Pointing at aggregate read bandwidth and calling that the inference solution is a bit like advertising a car's top speed and omitting that the brakes are slow.

Training Offers No Rescue

If inference is a bad fit, maybe training offers a better case for HBF? The answer is mostly no, and the reasoning is structurally different.

Training a large language model is not simply a matter of reading weights from memory. The process requires computing gradients—essentially calculating how wrong the model's predictions were and in which direction to adjust—and then applying those gradients through an optimizer to update the weights. This is a cycle of reads, writes, and intermediate computations that traffic heavily in both directions. The memory access pattern is not dominated by reads the way a simpler workload might be.

Beyond that, the established approach to training at scale is parallelism: split the model across many GPUs, each holding a shard, and coordinate across them. Eom's analysis is that training "benefits a lot from scaling the number of nodes and having a faster bandwidth" and consequently "has less room to actually benefit from when we actually make improvements on memory bandwidth and memory capacity." Adding HBF capacity to a single GPU does not change the fundamental geometry of how large-scale training is organized.

So HBF finds itself caught between two workloads. Inference needs fast writes; HBF does not provide them. Training needs scale and interconnect; HBF does not address that either. As Eom puts it: "It really remains as this awkward middle ground that doesn't really stand to gain huge benefits."

Who Owns the Problem

There is a structural dimension to this that the bandwidth debate tends to obscure. SanDisk and SK Hynix make memory. They do not make GPUs, they do not run the AI stack that sits above the GPU, and they do not control the software frameworks that determine how models are loaded, cached, and served. Their tool is the chip-level specification. The proposal they've put forward—attach HBF here, get more capacity there—is the kind of solution available to companies operating at that layer.

Nvidia's position is different. Nvidia sells the GPU, the interconnect fabric, the networking gear, and increasingly the software stack that ties it all together. Its response to the memory pressure problem is not a new chip-level attachment but a system-level architecture: the Context Memory Extension, or CMX, which addresses the KV cache pressure through infrastructure design—dedicated storage and networking bundled into a pod architecture—rather than through a new memory tier on the chip itself.

The contrast is worth noting without overstating it. Nvidia's approach solves the problem on Nvidia's terms, using Nvidia's hardware and software. That is good for organizations running Nvidia-centric infrastructure and less useful for anyone trying to build around a different stack. HBF, as a published specification, is at least theoretically available to any GPU manufacturer willing to implement it. The fact that none have done so publicly suggests the technical friction is real, not just a marketing positioning problem.

The Real Question

None of this means the underlying problem goes away. Models are getting larger. Context windows are growing. The demand for memory—fast memory, cheap memory, lots of memory—is not going to shrink on any timeline that matters to people making hardware decisions today.

HBF's read bandwidth, assembled through parallelism, genuinely approaches what high-bandwidth memory delivers. That is not nothing. For specific workloads where reads dominate and writes are infrequent, there may be a legitimate use case that the current AI training and inference profile simply does not fit. Storage-class workloads, retrieval-heavy applications, long-term KV cache persistence across sessions—these are not obviously stupid places to look.

But the semiconductor industry has a habit of fitting solutions to problems that exist on paper rather than in practice. The question that matters now is less whether HBF could theoretically help some workload, and more whether the workloads that will define the next two years of AI deployment—longer agents, persistent context, massive multi-step reasoning chains—will evolve in a direction where NAND's write penalty becomes acceptable overhead, or whether they will continue demanding exactly the symmetric, low-latency memory access that NAND was never designed to provide.

The silence from GPU manufacturers is data. It is not a verdict, but it is data.

By Bob Reynolds, Senior Technology Correspondent, BuzzRAG