Cross-Entropy and the Math Behind LLM Training

In 2002, a pair of researchers published a paper that Sanderson refers to in this video as "Language Trees and Zipping." The premise was almost absurdly modest: take a pile of text documents in different languages and use gzip — the same compression utility on your laptop — to figure out which languages were related to each other. No linguistic rules. No language models. Just compression.

It worked. The co-compression distance recovered the tree of language lineage. It could also identify authorship.

Grant Sanderson, who runs the 3Blue1Brown YouTube channel, opens his latest video with this paper for a reason. It's a clean demonstration that compression and pattern recognition are, at some level, the same problem. The paper's trick was a rough empirical approximation of something more precise: cross-entropy. And cross-entropy, it turns out, is also the core of how every major language model you've heard of gets trained. That's not a coincidence. The video spends 34 minutes showing you why.

What cross-entropy actually measures

The setup is simple once you see it. Suppose you've built a compression scheme optimized for one distribution of symbols — say, a robot navigation system where "move up" happens half the time. You've encoded "up" as a single bit, "down" as two bits, and left and right as three bits each. Now the mission profile changes: the robot spends most of its time moving right. Your encoding is now wrong for the new reality. The question cross-entropy answers is: how wrong, exactly?

You calculate the average bits per symbol under the new distribution using the old encoding. That weighted average — new probabilities, old code lengths — is the cross-entropy. It's always at least as large as the entropy of the true distribution, and it equals that entropy only when your encoding matches the actual distribution perfectly. The gap between them is the waste.

Sanderson's diagram makes this visual: bars whose widths represent the actual probabilities and whose heights represent what your mismatched code thinks those probabilities are worth. When your code is wrong, the bars don't fit together efficiently. Cross-entropy measures the total area.

This is the core idea. Everything else in the video is an extension of it.

Why the loss function isn't really a choice

Training a language model means defining a loss function — a score that tells you how badly the model is currently predicting the next token, so gradient descent can nudge the parameters in the right direction. The intuitive requirement is obvious: assign high loss when the model puts low probability on the token that actually appeared, low loss when it was confident and correct.

A lot of functions satisfy that basic shape. So why does every serious language model use the negative log of the probability — cross-entropy loss specifically?

Sanderson works through the answer carefully. Imagine the phrase "my name is ___" appearing thousands of times in the training data with different names following it. The model produces a probability distribution Q over all possible next tokens. The training data implicitly contains a distribution P — the actual frequencies of different names appearing in that position across the full dataset. The total loss associated with this input pattern is a weighted average of the per-example losses, weighted by P.

If you want that total loss to be minimized only when the model's distribution Q matches the data distribution P — which is exactly what you want — then the per-example loss function can't be arbitrary. Sanderson shows, via constrained optimization, that the derivative of your loss function with respect to Q must look like a constant divided by Q. Only logarithms have that property. The negative log isn't a stylistic preference. It falls out of the single requirement that the loss function should bottom out when the model is right.

That should give you pause when you evaluate the AI landscape. Every variation companies are currently selling — different architectures, different training recipes, different "novel" loss formulations — is operating on top of a foundation that isn't negotiable. The logarithm is there because the math put it there. Which means the genuine innovation, if any exists, is happening in the layers above: data curation, fine-tuning, architectural choices around the transformer. When a company implies the training objective itself is a proprietary edge, that claim deserves scrutiny.

Distillation: where the formula earns its name

Standard pre-training is blunt. At each position in the training data, the model is measured against a single true next token. All the probability mass is concentrated on that one outcome; every other possibility contributes zero to the loss signal. It works, but it's slow. The model has to see a pattern many times before its loss distribution meaningfully reflects the actual range of possibilities.

Distillation is different. The goal is to train a small model to approximate a large one. Instead of comparing the small model's prediction against the one true next token, you compare it against the full output distribution of the large model — every possible next token, weighted by the big model's probability estimates.

Sanderson compares this to the difference between learning chess by watching games silently versus having a stronger player walk you through all the reasonable moves and explain how heavily they'd weight each one. The signal per training step is substantially richer, because the large model's distribution already encodes a great deal about the range of plausible continuations. This is also where the cross-entropy formula becomes explicit rather than implicit: the loss is defined directly as the cross-entropy between the small model's distribution and the large model's distribution.

Every announcement of a smaller, faster model — the ones that claim near-parity with a much larger predecessor — should prompt one question: what was the teacher? Distillation compresses capability, but it also inherits the large model's distribution of errors and biases. A small model that looks like its teacher on benchmarks may look very much like its teacher on failure cases too.

KL divergence and the asymmetry that matters

Cross-entropy minus entropy gives you a quantity called KL divergence. In compression terms it measures pure waste: how many extra bits per symbol are you spending because your code doesn't match the real distribution? It's zero when the distributions are identical and grows as they diverge.

The detail that matters practically is that KL divergence is asymmetric. The divergence of Q from P is not the same as the divergence of P from Q. This isn't a mathematical curiosity; it has direct consequences for how models behave.

Consider what happens when you evaluate whether a language model understands medical language. If you measure how well the model's distribution fits the actual distribution of medical terminology, you get one answer. If you measure how well a medical terminology distribution fits the model, you get a different one — and those two answers can diverge significantly when one distribution is very concentrated (medical jargon is rare in general text) and the other is spread out. Evaluation tools that treat this as symmetric will systematically understate the model's gaps in low-frequency, high-stakes domains. That matters if you're deciding whether to deploy a model in a specialized context based on benchmark numbers.

What comes next, and what to watch for

Sanderson frames this video as the second part of a larger series on the relationship between compression and intelligence. He signals that the next installment will show how a language model — a prediction machine — can be converted into an actual compressor, and how that equivalence makes precise the claim that training on cross-entropy loss is the same as training for optimal compression.

That argument, when it lands, will be worth watching. The tech industry has been loose with the phrase "compression is intelligence" in exactly the way it was loose with "the cloud" and "big data" — evocative, difficult to falsify, endlessly recyclable in investor decks. A mathematical account of what that phrase actually means, and what it implies about the limits of scaling, is more useful than the slogan.

Next time a company announces a smaller model that supposedly performs like a larger one, ask two things: what teacher model provided the distillation signal, and what does the loss curve actually look like versus the entropy floor? The math is, at this point, well-understood enough that those questions have real answers — and companies that can't provide them are probably selling the slogan.

---

Bob Reynolds is Senior Technology Correspondent at BuzzRAG.