Kimi K3 Architecture: KDA, MoE, and Attention

Moonshot AI's Kimi K3 has been making the rounds in benchmark threads and LMArena leaderboards, sitting at or near the top for code and web development tasks — ahead of Claude Opus, GPT-4o, and Gemini depending on which metric you're looking at. The benchmark-to-reality gap is worth keeping in mind, and our earlier look at K3's frontend performance showed the model has genuine rough edges. But benchmarks and practical testing only tell you what a model does. The more durable question — and the one that actually helps you evaluate the next wave of models — is how it does it.

NeuralNine's recent technical breakdown of Kimi K3's architecture is one of the better attempts to answer that, walking through the three mechanical innovations that separate K3 from a generic transformer stack. None of these components were invented from scratch for K3 — the presenter is explicit about that — but their combination and the specific engineering decisions around stability and load balancing are what Moonshot is betting on.

Here's what's actually happening under the hood.

Memory as a First-Class Problem: Kimi Delta Attention

Most people interacting with large language models think about memory in terms of context windows — how many tokens can the model see at once. KDA, or Kimi Delta Attention, is doing something more granular: it's managing what the model keeps and discards inside its internal state as it processes a sequence.

The NeuralNine presenter frames it well: "You can think of this to be a little bit like a recurrent neural network where you have forgetting mechanisms and you want to remember stuff, you want to forget stuff over time, you want to forget specific stuff."

The mechanism is governed by two input-derived parameters. Alpha (α) controls channel-wise decay — think of it as a dial for each information channel that determines how much the previous memory fades before new content is written. Beta (β) is the update strength, a scalar that controls how aggressively a specific piece of information gets overwritten. The elegant part: the same β that erases old knowledge at a given "address" in the embedding space is the same β that writes the new value there. Forgetting and learning happen with equal intensity, which keeps the memory state from drifting.

The output of KDA isn't just the updated memory state — it's the memory state queried by a vector representing the current information request. You write the new value in, you erase what was there before, and then you ask the updated state what it knows. It's a read-write-read cycle that happens continuously as tokens flow through the model.

This is not trivially new — delta networks and linear attention variants have explored similar ideas — but KDA as implemented in K3 (building on the earlier Kimi Linear paper) applies it at scale with input-dependent gating, which is what gives it the flexibility to handle diverse token types without hand-coded forgetting schedules.

Routing at Scale: Stable Latente Mixture of Experts

If KDA is about how K3 manages what it knows, Stable Latente is about how it manages what it computes. The mixture of experts (MoE) pattern is well-established at this point — rather than running every parameter on every token, you route each token through a small subset of specialized sub-networks ("experts") and only those experts do active inference.

K3's numbers here are striking: 896 total routed experts, with 16 active at any given time. That's roughly 1.8% activation density. The compute savings are real; the VRAM requirement is not reduced (you still load all 896 experts into memory), but the inference cost per token is dramatically lower than a dense model of equivalent total parameter count.

Two additional "shared experts" are always consulted regardless of routing decisions, providing a stable baseline that every token passes through. The presenter notes some uncertainty about whether there are exactly two shared experts or more — full architectural details haven't been released — which is worth flagging. We're working from the blog post and architecture diagram, not a full technical report.

The word "stable" in Stable Latente does real work here. Standard MoE implementations have a load balancing problem: the router tends to favor a handful of popular experts, leaving others idle and wasting parameters. K3 addresses this with quantile-based load balancing. Rather than always routing a token to whichever expert scores highest for it, the system asks a different question: for each expert, which tokens fall in the top scoring bracket for that expert specifically? Those are the tokens it gets.

The presenter's analogy is useful: "We're not just asking who's the best expert to handle this workload. We're asking which expert has this as their favorite task."

The result is more even utilization across the expert pool, which matters for training stability and for getting value out of the full parameter count rather than over-relying on a subset. There's also a parallel training mechanism for GPU load balancing, though the implementation details there aren't yet public.

The "latente" part refers to dimensionality reduction: before tokens are fed into the routed experts, they're projected down into a lower-dimensional latent space. Inference happens in that compressed space, then the representations are projected back up and combined with the shared expert outputs before moving to the next layer. Less compute, smoother training — the dimensional reduction is doing work on both fronts.

Selective Memory Across Depth: Attention Residuals

The third component is the one that departs most clearly from standard residual connection design. In a typical deep network, residual connections pass information forward layer-by-layer without modification — they're a gradient highway, not an information filter. K3's attention residuals are something different.

Each layer in K3 maintains a learned pseudo-query vector — a persistent representation of what that particular layer is looking for. When processing, the layer uses this vector to score all previous hidden representations (from earlier blocks and the immediately preceding layer), applies a softmax over those scores, and computes a weighted sum. The result is what actually gets fed into the next processing stage.

Nothing passes directly to the next cell. Everything goes through this selective lookup first.

The presenter describes the flow: "We always use the pseudo vector to specifically look up interesting and relevant information. Then we do a softmax scoring and then we do a weighted sum that we then — the result is then fed into this cell."

What this achieves is a form of learned inter-layer attention — each layer can reach back through the stack and pull information that's relevant to its particular function, rather than receiving whatever the previous layer happened to output. Whether this meaningfully outperforms simpler residual designs at scale is an open empirical question, but the architectural intuition is sound: if different layers specialize in different aspects of processing, giving them a mechanism to retrieve specifically relevant prior representations should help.

What We Don't Know Yet

It's worth being honest about the gaps. The Moonshot AI blog post is a partial disclosure — training details, full parameter counts, and specifics around the GPU load balancing implementation haven't been released. The weights themselves were expected around July 27th; once they're public and researchers start digging, the picture will get sharper. Moonshot has been progressively sharing more with each K-series release, so there's reason to expect a fuller technical report to follow.

What the architecture as disclosed does show is a coherent set of engineering choices aimed at efficiency and stability rather than brute-force scale. KDA handles selective memory without exploding context costs. Stable Latente's MoE layer keeps active compute sparse while its quantile routing prevents expert collapse. Attention residuals give each layer a way to selectively surface relevant prior context rather than blindly inheriting whatever the previous layer produced.

These aren't isolated tricks. They compose into a system where memory management, compute efficiency, and inter-layer information flow are all being actively optimized rather than left to defaults. Whether that composition is what's driving K3's benchmark performance — or whether training data and scale are doing most of the heavy lifting — is exactly the kind of question that open weights will eventually help the community answer.

The architecture is interesting. The proof, as always, is in what the weights actually do.

Dev Kapoor covers open source and developer communities for Buzzrag.