How Spark-X2.5 Fits a Million Tokens Into a 4B

Spark-X2.5's advertised 1,048,576-token context can consume roughly 39 GB of memory for its KV cache alone. Add model weights, processing buffers and the runtime itself, and the compact 4-billion-parameter model starts looking less compact.

That distinction separates a context limit from a usable local configuration. Four billion parameters describe the learned weights loaded when the model starts. The context window describes how much text it can process during a session. As the session grows, the model stores keys and values representing earlier tokens so later tokens can consult them.

Cloud Codes captured the problem neatly in its analysis of Spark-X2.5: "The model is small. The memory it builds while reading your project can be much bigger."

The million-token claim is supported by an unusual architecture and dedicated long-context training. It remains a capacity specification, not a promise of cheap, fast or accurate use across the entire window.

The Memory Invoice Behind the Headline

Cloud Codes calculates that Spark's KV cache requires about 39 GB at the full context length when keys and values use 16-bit precision. The estimate assumes the runtime correctly limits the caches belonging to sliding-window layers. It does not include model weights or temporary buffers, and it is a calculation rather than a measurement from a completed workstation test.

The official GGUF repository lists weight files of roughly 8.2 GB at full precision, 4.4 GB at 8-bit precision and 2.6 GB at 4-bit precision. Combining those figures with the cache estimate produces a more revealing set of minimums:

| Configuration | Weights | Full Context Cache | Combined Before Overhead | |---|---:|---:|---:| | 4-bit | 2.6 GB | 39 GB | 41.6 GB | | 8-bit | 4.4 GB | 39 GB | 43.4 GB | | Full precision | 8.2 GB | 39 GB | 47.2 GB |

Those totals omit runtime overhead and working buffers. A machine with 48 GB of unified memory may see the full-precision combination approach its nominal capacity before the software has found a chair.

Smaller context allocations change the economics. Cloud Codes estimates about 1.3 GB of cache at 32,768 tokens and 4.9 GB at roughly 131,000 tokens. With 8-bit weights, the corresponding minimums become about 5.7 GB and 9.3 GB. Those are far more plausible budgets for local workstations.

Weight quantization and cache precision therefore require separate settings. A 4-bit download shrinks the fixed model file. It leaves a 16-bit KV cache untouched unless the runtime also offers cache compression.

How Hybrid Attention Cuts the Cost

Spark has 36 attention layers, according to the architecture examined by Cloud Codes and described in the Spark-X2.5 model card. Twenty-seven use a sliding window limited to 512 tokens. Nine retain global attention across the growing context.

The local layers keep recent code, conversation and tool results close at hand. Once their 512-token windows fill, a capable runtime can discard older entries from those layers' caches. The nine global layers continue storing long-range history, providing a route between an instruction near the beginning and related material much later.

Cloud Codes puts the limit plainly: "That makes the connection possible without guaranteeing a correct answer."

Spark also uses grouped-query attention. It has 16 query heads but four sets of key and value heads, allowing several query heads to share stored history. Fewer key and value copies mean a smaller cache.

If all 36 layers retained the complete history, the estimated cache would reach about 155 GB. Keeping only nine global caches cuts that hypothetical total to about one quarter. The bounded local caches add comparatively little once their 512-token windows are full.

A three-quarter reduction sounds dramatic because it is. Thirty-nine gigabytes remains thirty-nine gigabytes, an old lesson in computing that percentage savings and affordable totals often live in different brochures.

Capacity Does Not Establish Recall

Spark's developers report a dedicated training stage using sequences reaching one million tokens. That gives the context claim more substance than changing a configuration field after training. The available record still does not establish uniform retrieval accuracy across every position in a million-token prompt.

A model might accept an entire repository while overlooking the one function that governs the requested change. It might identify the correct file yet mishandle an instruction recorded hundreds of thousands of tokens earlier. Hybrid attention preserves a path to old information; benchmarked behavior determines whether the model uses that path reliably.

The token count also needs translation. One million tokens does not mean one million words, nor does it correspond to a fixed number of source files. Tokenization varies with prose, code, punctuation and language. Users must also reserve context for the model's answer and for subsequent tool output.

Speed presents another open question. Nine global layers still process long-range history. Fitting the cache into memory provides no measurement of how long the model takes to ingest a million-token prompt before producing its first answer. A local agent that spends an hour reading a repository has achieved privacy, though perhaps not productivity.

Benchmarks Answer Narrower Questions

The Spark model card reports 44.4 on SWE-bench Pro, 54.6 on MCP-Atlas and 90.7 on AIME 2026. These results point toward coding, tool use and mathematics as intended strengths. They come from the developer's published evaluations, and some competitor figures were taken from other model cards or papers rather than reproduced under one independent test setup.

Other rows complicate any universal ranking. Cloud Codes cites a Spark score of 67.4 on GPQA, behind a listed 9-billion-parameter Qwen model at 77.2. Spark's position also changes between SWE-bench variants. Benchmark names can conceal large differences in task construction, agent scaffolding and grading.

Compressed local downloads introduce another variable. A full or 8-bit checkpoint may preserve behavior that deteriorates at 4-bit precision. The Spark MLX runtime documentation recommends 8-bit or full-precision weights for tool calling and warns that 4-bit weights can reduce the accuracy of structured arguments.

That failure mode matters for agents. A fluent explanation followed by malformed JSON still produces a broken tool call. The language looks competent while the automation stops at a missing brace, a rather efficient recreation of office life.

A Useful Local Test Needs More than One Prompt

Native runtime support is improving. The llama.cpp b10828 release added Spark support, while the model card identifies versions for other local applications. Users should check the installed version because a compatible file format does not ensure that an older runtime understands Spark's architecture.

A fair workstation trial should use the same runtime, quantization and settings planned for actual work. It should record:

Peak system and accelerator memory at 32,000, 128,000 and larger contexts
Time to first token after loading each prompt
Whether an early repository rule affects a later requested patch
Whether the resulting code passes the project's tests
Whether structured tool arguments remain valid
How performance changes between 4-bit, 8-bit and full-precision weights

The comparison should also include a retrieval-based workflow that selects relevant files before prompting. Feeding an entire archive may reduce preparation work, while file selection may save memory and ingestion time. The useful winner depends on whether the model finds and applies the right evidence, not how much text the interface says it accepted.

Spark-X2.5 makes a million-token local context architecturally possible by rationing global attention. Whether it makes that context practical will be decided by memory measurements, waiting time and patches that pass their tests.