How Deep Networks Learn to Abstract, Not Memorize

There's a question sitting underneath almost every debate about whether AI is "really" intelligent, and it's one that almost never gets asked directly: how does a deep network generalize at all? Not in the hand-wavy "it finds patterns" sense. Mechanically. Mathematically. Why does it work when it obviously shouldn't?

Matthieu Wyart, a statistical physicist at EPFL, thinks he has a framework that actually answers this — and the answer has less to do with scale than most people assume. He laid it out in a wide-ranging conversation with Tim Scarfe (host of Machine Learning Street Talk) that covered everything from sand piles to Chomsky to why you should probably be predicting in latent space instead of token space.

The physics backstory matters here, so bear with me for a minute.

Sand piles, loss landscapes, and a very convenient analogy

Wyart spent years studying jamming transitions — the physics phenomenon where dense systems like sand suddenly shift from stuck to flowing when you tilt them past a threshold. When he started looking at neural networks around nine years ago, he noticed something: the loss landscape (the mathematical surface a model navigates during training, where lower = better fit) behaves almost identically.

Underparameterized models — ones without enough parameters to fit the data — get stuck in rough, bumpy landscapes, like a ball rolling into a valley in the Alps and stopping. Overparameterized models suddenly find flat valleys everywhere, and training flows freely. This is exactly the jamming transition. Other researchers called the same phenomenon double descent; physicists call it jamming. Same thing, different vocabulary.

The analogy isn't decorative. Wyart's argument is that in both cases — sand particles trying not to overlap, model parameters trying to fit data — you have continuous degrees of freedom satisfying a system of constraints, and the math of how those constraints interact belongs to the same universality class. That's a strong claim from physics: it means the behavior is generic, not coincidental.

I find this genuinely compelling as a framing, even if "universality class" is doing a lot of work there. The reason it doesn't feel like hand-waving is that it makes testable predictions — which Wyart keeps returning to as the actual bar for whether a theory is doing anything.

The Chomsky problem

Here's where it gets philosophically interesting in a way that I don't think is resolved by either side, despite what both sides claim.

Chomsky's "poverty of stimulus" argument holds that children encounter far too little linguistic data to induce the full grammar of their language from examples alone — which implies some grammatical structure must be innate. By extension, he's argued that LLMs are doing something categorically different from what humans do: they're bulldozers, not theorists. Great at moving snow, irrelevant to understanding the physics of snow.

Wyart's counter is built on a mathematical model — the Random Hierarchy Model — that encodes the kind of hierarchical, tree-like structure that Chomsky himself argued underlies language. The finding: shallow networks essentially memorize and fail to generalize. Deep networks, given the same synthetic data, do something different. They discover the hidden generative structure — the coarse-grained variables, the abstract regularities — and can generate novel valid sentences having seen only a polynomial (rather than exponential) number of examples.

That's a direct mathematical counterexample to the poverty-of-stimulus argument. And it works, Wyart says, because deep architecture has an implicit bias toward building hierarchical representations. The depth itself is the inductive prior. You don't need innate grammar if you have the right architecture.

Does this move me? Partly. The formal result is elegant and the prediction is specific enough to be falsifiable, which is more than you can say for most things in this debate. But I notice Wyart is careful to say he's arguing against Chomsky's argument, not necessarily against his conclusion. The brain might still do something fundamentally richer than any current network architecture — the sample efficiency gap alone (machines need far more data than humans to achieve comparable linguistic competence) is real and unexplained. Wyart acknowledges this openly, which is more intellectually honest than the usual "scale will fix it" non-answer.

What I'm less sold on: calling this "creativity." Wyart uses the word in a narrow technical sense — generating novel sentences that satisfy syntactic constraints — and he's clear about that. But the conversation around whether that constitutes real creativity keeps slipping between the technical and the intuitive sense, and I think that ambiguity is doing more work than either participant quite notices.

The curse of dimensionality, and why the manifold hypothesis isn't enough

The curse of dimensionality is one of those things that sounds like a minor technical footnote until you sit with it. If your data lives in high-dimensional space — and images, text, audio all do — the volume of that space grows exponentially with dimension. Even a trillion data points are, in high enough dimensions, effectively isolated from each other. No interpolation can save you. Generalization becomes mathematically impossible unless the data has structure you can exploit.

The popular answer has been the manifold hypothesis: real data lives on a much lower-dimensional surface embedded in that high-dimensional space. True enough. But Wyart's objection is sharp: that lower-dimensional manifold is still high-dimensional. And if the manifold hypothesis were sufficient, simple algorithms like kernel methods or shallow networks would work just fine. They don't. Not on real text, not on real images.

The explanation that actually fits the evidence — and explains why you need depth — is that the world is hierarchically structured. Pixels compose into edges, edges into shapes, shapes into objects, objects into scenes. Words compose into phrases, phrases into clauses, clauses into sentences with meaning. Deep networks, Wyart argues, discover this hierarchy from data and use it to compress the effective dimensionality of the problem. Once you've abstracted "street" as a concept from "cars + sidewalks + passersby," you don't need to reason about pixels anymore. The curse is lifted not by ignoring dimensionality but by building the right abstraction ladder.

This is the piece I keep coming back to as genuinely underappreciated in mainstream ML discourse. The architecture isn't just a computational convenience — the depth is load-bearing.

Why predict latents instead of tokens

The most forward-looking part of Wyart's argument is about training objectives. Current LLMs and diffusion models predict at the lowest level: the next token, or the masked pixel. Even this low-level objective, Wyart argues, forces a network to build hierarchical abstractions internally (otherwise it couldn't predict tokens well). But there's a catch: when you're trying to learn highly abstract concepts by predicting low-level features, the signal gets diluted. The correlation between "this configuration encodes the concept of a street" and "the pixels in the surrounding region look like house facades" is real, but noisy. You need a lot of data to measure it reliably.

The alternative — explored in Yann LeCun's Joint-Embedding Predictive Architecture work and now given a theoretical grounding in Wyart's recent paper — is to predict in latent space instead. Once you've built an abstract representation of "house," you can build the representation of "street" by predicting that concept in the neighborhood, not raw pixels. The signal is much stronger. The sample complexity drops dramatically.

In Wyart's theoretical framework, this is quantifiable: predicting in token space requires data exponential in the depth of the concept hierarchy. Predicting in latent space cuts through that exponential. Same abstractions, learned much faster.

He's careful to flag that this paper is about a month old at time of recording and is still being tested empirically. That kind of epistemic humility is, frankly, refreshing. "Being rigorous," he says, "doesn't mean having a theorem. It's us going back and testing those predictions." The open question he's most honest about: even if you build a spectacular encoder of the world through latent prediction, can you then decode it back into something generative — something you can talk to? He doesn't know yet. Neither does anyone else.

Scaling laws, entropy, and the frontier of the theory

Wyart also has a recent theory of scaling laws — the empirical observation that model performance improves predictably as you add data, compute, or parameters, first documented by Kaplan and colleagues in their influential 2020 paper. The observed exponents had been embarrassingly unexplained by theory. Wyart's framework derives them from two measurable quantities: the power-law decay of token correlation with distance, and the decay of text entropy as context grows — a question Shannon was already asking in the 1950s.

The theory makes nontrivial predictions about how loss depends on both context length and dataset size simultaneously, and those predictions check out — at least at the academic scale (roughly 1 billion parameters, 1 billion tokens, contexts of about 50 tokens). Whether it holds at frontier scale is explicitly flagged as unknown.

That bracket matters. The piece of the theory that's been validated is operating in a regime that most people don't find interesting anymore. Wyart's honest that they simply don't have the compute to test it beyond a few sentences of context. The mechanism might be completely different at the scale where GPT-4 operates. Or it might not. That's genuinely open.

The conversation ends on something that stuck with me. Scarfe asks whether Wyart has ever been completely wrong about something. The answer is yes, of course — that's how it works. But the line that landed:

"If you never do mistakes, maybe it's a sign that you're staying a bit on the beaten path in science. Some of us want to explore the jungle. In the jungle, you can be wrong."

The latent-prediction direction, the physics-derived scaling theory, the random hierarchy model — these are all jungle moves. They might not pan out. But the alternative is staying on the path and explaining away the things you can't quite account for.

The question of whether predicting abstractions rather than pixels is the unlock that changes everything — or just a cleaner version of the same fundamental constraints — is one that the field is going to have to answer empirically. That answer isn't here yet.

Yuki Okonkwo is Buzzrag's AI & Machine Learning correspondent.