Qwen3.8-27B Fits a 12GB Card, Except When It Can

The headline number on the Hugging Face page is 11.77 GB. A 12 GB graphics card, you would think, has room for that plus a little air. It does not, and the reason is a chain of small print: decimal gigabytes marketed as binary ones, a 931 MB vision projector sold separately from the model it exists to serve, and 150 MB of fixed recurrent state that shows up on no download page anywhere.

Work it through the way The Stack did in its recent breakdown of ISTA-DASLab's Qwen3.8-27B release (youtube.com). A card sold as 12 GB actually holds 12 binary gigabytes. The recommended GSQ-RCO build is listed at 11.77 decimal GB, which is 10.96 GB in the card's own units. That leaves 1.04 GB of headroom, already more generous than the 200 MB the raw headline suggested. Then the mmproj file loads. It is the model's eyes, a 0.87 GB vision projector left at full 16-bit precision, the only artifact in a release devoted to compression that nobody compressed. What remains is 174 MB. The 48 recurrent layers eat a flat 150 MB. At 64 kilobytes per token of key-value cache, the leftover 24 MB buys roughly 386 tokens of context, fewer in practice, before the inference runtime reserves anything for itself.

In other words, the flagship build of Alibaba's 27-billion-parameter vision-language model loads on a 12 GB card and then cannot hold a real conversation once it can see. Alibaba's own model page advertises a native 262,000-token context; on this hardware, this build, you get under four hundred.

The Lab that Quantizes the Field It Founded

ISTA-DASLab is not some third-party repacker. The same group published GPTQ, one of the standard compression methods the field still measures against. For Qwen3.8 they shipped GSQ, a scalar-grid method paired with RCO, an exact budget-constraint solver described in a second paper, "Model Compression with Exact Budget Constraints" (arXiv 2605.00649). The scalar choice was deliberate. In the paper's own words, scalar methods "plateau in accuracy at 3 to 4 bits," while the higher-scoring vector methods like AQLM are notoriously hard to implement and scale. Staying on the scalar grid means the compressed file runs in llama.cpp, Ollama, and LM Studio unmodified.

The RCO solver then spends the memory budget down to the last byte across 851 tensors and 11 formats, from 16-bit down to a 1.75-bit IQ1_M. The lab published the entire assignment as plain text, and it is not intuitive. Block 13's feed-forward gate sits at 1.75 bits inside a block whose neighbors got 2, 3, and 4 bits. More than a tenth of the file, 1.12 GB, is just the embedding table and output head. And the allocation is not monotone: the same starved gate rises to 4 bits at the middle budget and falls below 2 bits at the largest one, because the optimizer protects the total, never any individual tensor.

The Benchmark You Cannot Check

Every performance claim for these builds comes from the lab that performed the compression, and nobody outside that lab has reproduced the comparisons. According to the lab's own evaluation, the recommended build matches the uncompressed 53.8 GB original on AIME25 and LiveCodeBench, trails by half a point on GPQA Diamond, and the 10.09 GB three-bit build still scores a perfect 100 on AIME25. Those may all be accurate. They are also unaudited.

That matters less than you might expect for the practical decision, because the gap between the recommended build and the one a gigabyte and a half smaller is half a point on one benchmark. Picking a file is a memory decision, not a quality decision. The three-bit build leaves 1,624 MB of headroom after weights and projector, enough for a theoretical 26,000-token multimodal context or 40,000 text-only, and even the 2.75-bit build reportedly recovers 101.8 percent of the original zero-shot average.

Why a Hybrid Architecture Makes This Possible at All

The byte math only works because of what is inside Qwen3.8. The base configuration lists 64 layers in a pattern of three linear-attention layers followed by one full-attention layer: 48 recurrent gated-delta layers and 16 full attention layers. Full attention keeps every token of the conversation; a recurrent layer keeps one fixed-size summary, identical at 10 tokens and 10,000. At full precision the attention cache alone would need 16 GB to serve the advertised 262K context, 4.5 GB at 4-bit. The 48 recurrent layers add a flat 150 MB regardless. A developer running the model at full context on a 16 GB card had to drop the cache to 4-bit just to fit.

It is the same architectural bargain behind other attempts to shrink big models onto consumer hardware, from Qwen3.8-Flash-Next putting 180B parameters on laptop hardware to PrismML's Bonsai 27B on 10GB of RAM. Recurrence, lookup tables, and aggressive precision trades are converging on the same target: big-model behavior at small-model memory. Meanwhile, Cerebras lists Qwen 3.8 27B on its inference service at 1500 tokens per second, which is the other answer to "who can run this": not you, very fast, somewhere else.

The Bugs Are in the Least-Tested Corners

Local deployment stories fail in the plumbing, and this one has plumbing failures on all three major runtimes. In llama.cpp, a developer's late-August bug report showed the model occasionally emitting an end-of-text marker as its very first token at long context, reproducing on GPU, CPU, and an independent fork, while the same prompt passed cleanly on vLLM. The reporter eventually withdrew his proposed safe threshold, advising that "you simply have to validate the exact depths you plan to serve." A separate early-September report found llama.cpp's backend tests failing matrix multiplication for three formats on a 16 GB Blackwell card, including the format this build is named after, with errors large enough to produce garbage output; a second user reproduced it, a third could not, and the build-chain-versus-hardware argument is unresolved.

Ollama's failure is a chat template that rejects any system message sent mid-conversation, returning an empty string with a finished status and no error. LM Studio declines to crash and instead spills excess weights into system RAM. None of the three tells you when a model will not fit, which is why counting bytes before you click download is the only reliable method.

What Nobody Shipped

The deepest limitation is not a bug but an absence. Two separate teams have asked the lab on its repository for the pieces needed to run the RCO search themselves, the per-tensor cost model, two budget configurations, and the export code, and have received no reply. Outside developers can read the final map of which format went to which tensor but cannot generate a new one for a different model, a different GPU, or a different budget. One of the requesting teams runs a server fleet limited by memory bandwidth, for whom a smaller file is a direct speed win, and this format is their only real deployment path.

The budget paper frames mixed-precision quantization, non-uniform pruning, and expert selection as the same problem: assign one of K options to each of N groups under a total budget. If that generalizes the way the authors suggest, the release in front of us is a demonstration of a tool rather than the tool itself, one solved instance of a problem everyone with a 12 GB card owns. The finished answer for Qwen3.8-27B is: download the 10.09 GB three-bit build, keep the projector, and expect roughly 26,000 tokens. The method that would produce your answer, for your card, remains in the lab's private directory.

Marcus Chen-Ramirez covers AI infrastructure and the gap between what a spec sheet promises and what a GPU card delivers.