Rich Sutton Says AI Models Have Stopped Learning

Here's the thing about Rich Sutton: he survived cancer during an AI winter, kept doing his research anyway out of what he describes as pure habit, and has now watched the field he quietly anchored from the University of Alberta become the center of the known tech universe. He does not seem particularly impressed by this.

In a conversation hosted by Sequoia Capital's Sonya Huang and Alfred Lin, Sutton and his former student and Oak Lab co-founder Khurram Javed made the case that the current AI paradigm—pre-train everything, freeze the weights, deploy—is not just incomplete. It's philosophically broken.

"Before there was all this AI craziness," Sutton said, "you wouldn't have to say continual learning because it wouldn't make any sense to talk about learning that wasn't continual. All learning is continual. I'm not weird. The field is weird."

That's a sentence worth sitting with. The term "continual learning" exists because the field invented a new normal—training on static datasets, then stopping—and needed a word for the old normal. From Sutton's vantage point, the field didn't unlock intelligence. It built an elaborate, expensive snapshot of it.

LLMs: A brilliant, partial answer

To be fair to Sutton, he's not dismissing what the past few years produced. He calls large language models "an amazing scientific breakthrough, a breakthrough in the skillful use of language by neural networks, wholly unanticipated." Language was supposed to be the fortress of symbolic AI; LLMs dismantled it completely. That's real.

But then he adds the kicker: LLMs represent "roughly a quarter of intelligence." The frustration isn't with the breakthrough itself—it's with the field's tendency to crown the quarter and declare the race finished.

His take on whether LLMs are consistent with his Bitter Lesson essay is a good encapsulation of the tension. They're both an example and a counter-example. LLMs beautifully demonstrated that scaling computation beats encoding human knowledge—you can "drink in the internet" and get remarkably capable systems. That's the lesson working. But then the internet ran out. The training data has a ceiling. And the response from frontier labs—synthetic data generation—is, in Sutton's estimation, where the wheels come off.

"Just a big mistake"

When asked whether synthetic data is a general method that leverages computation the way the Bitter Lesson recommends, Sutton doesn't hedge: "No. That's just a big mistake."

Javed elaborates with what they call the Big World Hypothesis: the world is massively more complex than any agent, simulator, or dataset that could ever describe it. A simulation is always a "small world"—a simplification built by humans with finite expertise and finite imagination. If you want a drone that navigates via echolocation like a bat, no volume of synthetic data helps until a domain expert designs the right simulation first. You haven't removed humans from the loop; you've hidden them one layer deeper.

The counterargument—the one Huang presses—is that simulation-trained self-driving systems seem to work reasonably well. Javed's response is sharp: how many engineers built and iterated on that simulation? How many cycles of "sim-to-real gap is unacceptable, fix the simulation" happened before deployment? Remove the engineers, and the pipeline stops. The agent isn't generating its own experience and correcting its own model. It's consuming a world that humans spent years constructing.

The more interesting version of this argument isn't that simulation is useless—Javed grants it has value—but that the right kind of simulation is one the agent builds and corrects from its own experience. When the agent's internal model is wrong, it can update it continuously. When engineers' simulation is wrong, it waits for someone to notice and patch it.

The weight-change question

This brings us to the actual technical crux: when you're talking to ChatGPT or Claude, are those systems learning from the conversation?

"Are you serious?" Sutton says, when asked whether LLM-based assistants are continual learners. And then, flatly: "Their weights never change."

What does change is context—the in-context window grows, the system's outputs shift. Cursor's autocomplete model does update its weights, and Sutton acknowledges that as a genuine instance of continual learning. But he notes how it works: they batch data from millions of users and do periodic updates. That means if you want to teach your personal instance of the model something specific, you're competing with the preferences of 100,000 other users. It's continual learning in the aggregate, not for you.

The distinction matters more than it might sound. Sutton's vision of an intelligent assistant is one that actually knows you—not via a memory slot appended to your context window, but because the weights themselves have been shaped by your particular experience. The difference is like the difference between carrying a sticky note that says "she prefers tea" and having internalized, over years of interaction, everything that comes with knowing someone.

Catastrophic forgetting: reportedly curable

The reason continual learning has been so hard to implement is something called catastrophic forgetting: when you update a neural network on new examples, it tends to trash what it already knew. Update too aggressively, and you get amnesia. It's the reason naive weight-updating is worse than useless.

Sutton says this is "totally curable." Javed lays out the approach they published in Nature: an algorithm called continual backprop. Standard backpropagation (the learning algorithm underlying most modern neural networks) starts with random weight initialization and then refines from there—but over time, all that useful randomness gets consumed. The network loses its capacity to grow.

Continual backprop keeps injecting small doses of randomness: new units with fresh random weights are periodically seeded into the network, while backprop acts as the "tester" that determines which seeds are worth keeping. The algorithm also requires per-weight step size optimization—rather than nudging all weights at the same rate, some move quickly while most stay nearly frozen, so new information updates just the right places without corrupting everything else.

The catch: you can't retrofit this onto an existing frozen model. The algorithm needs to metalearn how to learn from the beginning of training. That means starting over—not fine-tuning GPT-5, but training a new foundation model with this architecture baked in from the start.

Why the big labs won't do this

This is where Javed's structural argument gets interesting—and worth scrutinizing independently of whether you buy the rest of it.

"These big labs, they are so locked into a product that it is not possible for them to pursue a path where things get worse first."

His claim is that frontier labs are stuck in a local minimum: the current paradigm produces world-class benchmarks, and any shift to a new paradigm would initially produce worse results. For a company whose product is the benchmark—and whose valuation depends on it—taking that bet is structurally nearly impossible.

This is a real phenomenon in technology. Incumbents don't fail because they're stupid; they fail because the thing that made them successful makes switching costs unbearable. It's not obvious this is unique to AI, and it's also not obvious that it's permanent—several frontier labs are actively experimenting with inference-time learning and on-device adaptation. But the argument has teeth. If a new training paradigm requires you to look worse for 12-18 months before getting better, you're not going to propose it in a quarterly product review.

The actual ambition

What Oak Lab is building toward is something they describe as a trillion-parameter model running on 20 watts, five to ten years out. Javed freely admits this is impossible with current hardware—just storing a trillion parameters uses more than 20 watts today. But he runs the Moore's Law math: if you can run the equivalent on 2,000 watts today with the right algorithms, two orders of magnitude of efficiency improvement over a decade gets you to 20.

The more philosophically interesting piece of the vision isn't the parameter count or the wattage—it's the self-maintenance. What they're after is a mind that stays coherent while it keeps learning. Right now, what keeps an LLM coherent is the freeze: you stop training, you stop drift, you post-train to fix alignment issues, and you lock it down. The alternative—a system that continuously learns without drifting into incoherence—requires the network to have some self-organizing principle that the field hasn't cracked yet. Sutton thinks it's crackable. He's been thinking it was crackable since a winter in 2003 when he was supposed to be dying.

The vision is "a single design, many minds"—not one monolithic system that knows everything, but a unified architecture that different instances learn from different experience. A squirrel-brained version, a spaceship-designer version, your personal assistant version, all running the same underlying algorithm but shaped into different knowledge by different lives.

Whether Sutton and Javed are the ones who build it or just the people who made the core insight legible enough that someone else could—that question is genuinely open. The field has a history of vindicating the patient ones.

---

Yuki Okonkwo is Buzzrag's AI & Machine Learning Correspondent.