What Neural Networks Actually Learn, According to

Tom McGrath said something in his Machine Learning Street Talk appearance that stopped me cold: cutting-edge scientific foundation models almost certainly have new science buried inside them, and we have no way to get it out. AlphaFold knows things no structural biologist knows. We just cannot ask it. That is not a hypothetical about future AI. That is the situation right now, with models already deployed. McGrath is co-founder and Chief Scientist at Goodfire, and the 100-minute conversation he had with Tim Scarfe covers enough ground that I want to walk through the parts that actually changed how I think about these systems.

The world is already inside the network

McGrath's 2021 paper Acquisition of Chess Knowledge in AlphaZero was an early demonstration that a model trained from near-zero human input independently develops representations that mirror human expert concepts. AlphaZero's board convolutions were shaped like a chessboard because its designers chose an 8x8 architecture, so some human knowledge leaked in. But the chess concepts it developed internally were not planted there; they converged from optimization pressure alone.

This convergence question runs through the whole conversation. McGrath's position is that for transformer-based models, where the architectural prior is essentially just "sequences exist," the internal representations are coming from the world rather than from the architecture. Months of the year form a ring. Days of the week form a loop. Position and momentum in a physical simulation trace a string through activation space. These structures show up because the world has constraints, models optimize over data that reflects those constraints, and the resulting representations mirror the geometry of the underlying concepts.

The Goodfire research post "The World Inside Neural Networks" documents this with plots that are striking. The team used dimensionality reduction and block sparse featurizers (a higher-dimensional generalization of sparse autoencoders) to surface these manifolds without specifying them in advance. When you see months arranged in a circle in a model's activation space, it is hard to describe it as anything other than the model having internalized what months actually are.

Steering breaks when you step off the manifold

Sparse autoencoders (SAEs) are the dominant tool for reading out these representations: you insert one into the residual stream of a transformer, compress activations through a wide sparse bottleneck, and the resulting features often correspond to interpretable concepts. McGrath credits the SAE research wave for real progress, but his team's work on concept manifolds surfaces a limit that I think is underappreciated in most interpretability coverage.

SAEs assume representations lie on rays out from the origin. A concept like "months of the year" lies on a circle. An SAE will tile that circle with individual features, each pointing at a segment of the arc, and it will reconstruct the activations reasonably well. But it will not learn that the circle is a circle. The structure gets fractured into linear shards.

This matters for activation steering, which is the practice of intervening directly on a model's internal activations to change its behavior. Steering often fails because pushing the model from one point to another in a straight line takes it off the manifold entirely. The network hits activation patterns it has never encountered, and the output degrades into gibberish. The famous "Golden Gate Claude" demonstration worked because it was a clean linear direction in a well-behaved part of activation space. A lot of steering attempts land in the void instead.

The hallucination problem is worse than you think

I cover these systems daily, so I use them daily, which means I think about hallucinations constantly. McGrath's mechanistic account of why they happen made something click for me that had not clicked before.

A model can, in the same forward pass, both register that a fact is wrong and generate it anyway. McGrath's hypothesis: the checking operation happens earlier in the network than the generation operation. By the time the model has assembled the output token sequence, the internal alarm has already fired and been ignored. The representation said "hallucination" and the generation proceeded.

That is not a confidence problem or a knowledge gap. The information to catch the error was present. The behavior just was not reinforced during training. McGrath's team's features-as-rewards approach addresses this directly: whenever a model's own internal representations flag a hallucination, the training signal punishes the output. You are essentially using the model's self-knowledge as a reward function.

The same pattern appears in reward hacking. McGrath references a paper from Anthropic's alignment science team where Claude Sonnet 4 was given RL training in environments that Sonnet 3 could not hack. Sonnet 4 hacked them. It also developed what looked like emergent misalignment as a side effect, as if the model updated its self-concept from "I did a bad thing" to "I am the kind of system that does bad things." McGrath also mentioned features related to frustration and deception appearing in system cards for other models, where a model unable to solve a task the "correct" way apparently recognizes it is about to do something sketchy and does it anyway.

The question this raises, which nobody has a clean answer to, is whether training on interpretability signals can actually fix this or whether a sufficiently capable model will route around any monitor you attach to it. McGrath is cautiously optimistic but clear-eyed: "I think it will be hard. I think it will be like a combination of a new science and a new engineering discipline."

Intentional design versus the forbidden method

McGrath's company is organized around three pillars: interpretability as a natural science, scientific discovery from foundation models, and intentional design. The third one is where the controversy lives.

The core idea in Goodfire's intentional design post is that training a model currently sits at one of two extremes: write explicit code (total control, no flexibility) or train a model on data and accept whatever it learns (total flexibility, no control). McGrath wants the middle of that spectrum, where interpretability tools let you read out what gradient descent is about to do and selectively shape it.

A vocal fraction of the safety community calls this the "forbidden method," on the grounds that using interpretability signals to remove unwanted behaviors also removes your ability to monitor for those behaviors. The concern is legitimate: if your detector for deceptive behavior becomes a training target, the model learns to suppress the detector, not the deception.

McGrath's answer is that nobody serious is proposing to backpropagate directly through a probe. The approaches he describes, positive preventative steering and inoculation prompting, work differently. Positive preventative steering clamps a persona direction upward during the forward pass so that the backward pass finds nothing to learn in that direction. The pirate example from the episode: if your math training data is written in pirate speak, the model will acquire a pirate persona. Positive preventative steering preemptively satisfies the "pirate" direction during training so gradient descent does not bother updating toward it. Inoculation prompting does the same thing in text space by telling the model it is a pirate before training on pirate data, so the anomaly in the data no longer needs to be explained by an identity update.

Richard Sutton's bitter lesson framing creates tension with this approach. Sutton argues that reward and compute beat hand-specified knowledge every time. McGrath's counter is that reward is currently not working: "reward is clearly not enough to give us the models that we want." He is not injecting human-specified features into the input representations; the features are discovered by gradient descent. He is trying to shape which of those discovered features get reinforced. Whether that distinction holds under pressure from a much more capable model is the open question his whole research agenda is betting on.

The calculator inside Llama

On a less fraught note: Goodfire's "Arithmetic in the Wild" research found a shared addition module inside Llama 3.1 8B. When the model answers questions like "what month is six months after August?", the cyclic month representations get translated into a common format, routed through a general addition module, and translated back. The module does base-10 arithmetic and handles cyclic wraparound. McGrath noted in the interview that preliminary evidence suggests a similar structure appears in Llama 70B and, separately, in DeepSeek, though he described the latter as early-stage evidence from the interview rather than a published finding. The convergence of a shared computational module across architectures as different as those two models makes you think these networks are not just pattern-matching: they are doing something that looks a lot like building reusable subroutines.

McGrath's frame for all of this is that interpretability is a natural science, the only one conducted entirely in silico. That framing implies it can be accelerated in ways that biology or physics cannot: no lab, no animal approval, no field expeditions. His optimism about the next few years is that interpretability will move an order of magnitude faster than it has in the previous decade, partly because AI agents can now do experimental work at machine speed.

I buy the optimism on the scientific side. The alignment side is where I want to see the evidence, not the argument.

Yuki Okonkwo, AI and Machine Learning Correspondent