AlphaGo From Scratch: What Go Teaches Modern AI

Editor's note: The editorial team flagged this article for category reassignment. I'm pushing back on that call. This piece is firmly in AI/Tech—it's about reinforcement learning architectures, search algorithms, and what a 2016 board game engine reveals about the future of language models. If that's "Growth content," then so is every paper DeepMind has ever published. Keeping my byline on it.

---

Here's a question worth sitting with: what does a decade-old Go-playing AI tell us about GPT-5, or whatever comes after it?

More than you'd think, according to Eric Jang. The former VP of AI at 1x Technologies and ex-Google DeepMind researcher spent part of his recent sabbatical doing something that sounds like homework but is actually kind of profound—rebuilding AlphaGo from scratch, using modern tools, to understand what made it tick. He sat down with Dwarkesh Patel for a 2.5-hour deep dive that is, genuinely, one of the more illuminating AI conversations I've come across recently.

The thing that keeps pulling me back to AlphaGo isn't nostalgia. It's that it remains the cleanest demonstration of three primitives that still define what "intelligence" means in ML: search, learning from experience, and self-play. Strip away all the hype around frontier models and you keep bumping into these same ideas, just dressed differently.

Why Go broke every AI approach before deep learning

Go looks deceptively simple—black and white stones on a grid, capture territory, outnumber your opponent. But the search space is cosmically large. On a standard 19×19 board, there are roughly 361 possible moves on the first turn, and games can run 250–300 moves deep. Do the naive math and you get something on the order of 361^300 possible game trees—a number that dwarfs the atoms in the observable universe.

That's not a rhetorical flourish. It's why computer scientists spent decades treating Go as essentially unsolvable through brute-force search. Chess engines could play at grandmaster level by the 1990s partly because the search space, while enormous, was navigable with enough compute. Go wasn't.

Jang explains it clearly: "This is a problem that has long been understood to be kind of intractable for search and yet it was solved through deep learning—and so that was quite mysterious to me."

What AlphaGo did was use neural networks to make the search selective. Instead of expanding every branch of that monstrous game tree, it learned to focus compute on the branches that actually matter. The mechanism for doing that is Monte Carlo Tree Search (MCTS)—and understanding MCTS is basically the key to understanding the whole conversation.

MCTS: the explore/exploit engine

MCTS works by building the game tree incrementally, guided by a scoring function that balances two things: how good a move looks based on past simulations (exploit), and how underexplored that branch still is (explore). AlphaGo uses a variant called PUCT (Predicted Upper Confidence for Trees), which adds a learned prior probability to that equation—essentially letting the neural network say "hey, this branch looks promising, weight it more heavily before you've even simulated it."

The data structure each node carries is elegant: a visit count, a mean action value (Q), a prior probability, and pointers to children. Every time you "roll out" a simulation—play the game forward to completion—you backpropagate the result (win or loss) up the tree, updating the Q values along the way.

What clicked for me watching Jang explain this: the exploration bonus shrinks as you visit a node more often. Initially, underexplored branches get inflated scores to force the algorithm to check them out. But as ln(N) grows slower than N, the Q term gradually takes over. You start exploring wildly, then slowly converge on the moves that actually win. It's a formalization of how a thoughtful person might reason through an unfamiliar problem.

The neural network's job in all this is two-fold: a policy network that predicts which moves are worth exploring (so you don't waste simulations on obviously bad branches), and a value network that estimates who's winning from any given board position (so you don't have to play out every simulation to the very end). Together, they make the search tractable. Without them, you'd need compute that doesn't exist.

The gap that matters: AlphaGo vs. how we train LLMs

Here's where the conversation gets genuinely interesting, and where I think Jang's framing has real stakes beyond Go.

The description from Dwarkesh's episode notes puts it sharply: naive policy gradient RL for LLMs "has to figure out which of the 100k+ tokens in your trajectory actually got you the right answer." This is the credit assignment problem—when your model writes a 10,000-token response and gets a single thumbs-up or thumbs-down at the end, attributing that signal back to the specific tokens that caused it is incredibly hard. Was it the sentence in paragraph three? The chain-of-thought reasoning in the middle? The final answer? You don't really know.

AlphaGo doesn't have this problem in the same way. MCTS gives you a strictly better action at every single step—not just a terminal reward, but a dense, move-by-move signal about what the policy should have done. Your training target is precise. You're not trying to reverse-engineer which of 100,000 decisions led to a win; you know exactly which move was suboptimal because the tree search showed you a better one.

The question Jang raises—and doesn't fully answer, because honestly nobody has—is whether something like MCTS-style process supervision could work for language models. The challenge is that Go has a clear, computable correctness signal at every move. Language doesn't. "Was this sentence the right one to write?" is a question without a clean algorithmic answer. Which is why current RL for LLMs often feels like you're training with oven mitts on.

The democratization angle (and what it costs)

One thing Jang mentions almost in passing that I keep thinking about: what cost DeepMind millions of dollars and an entire research team in 2016 can now be reproduced for a few thousand dollars of rented compute, thanks in large part to open-source work like KataGo (built by David Wu at Jane Street in 2020, which achieved a 40x reduction in training compute for a competitive Go AI).

"What took a whole team of research scientists at DeepMind and millions of dollars of research and compute can now be done for a few thousand dollars of rented compute," Jang says.

That's remarkable, and it's a real data point for anyone thinking about the pace of AI capability diffusion. But it also surfaces a tension: the primitives of AlphaGo are now cheap to implement, but the frontier keeps moving. The thing that's now reproducible for thousands of dollars was frontier research eight years ago. The actual frontier—whatever's happening at the leading labs right now—remains expensive, concentrated, and largely opaque.

Can LLMs automate AI research itself?

The last part of the conversation ventures into territory that feels both exciting and uncertain in equal measure. Jang ran what he calls an "Autoresearch loop"—essentially setting LLM agents loose on his AlphaGo project to see how much of the research process they could handle autonomously.

The results were instructive. LLMs turned out to be genuinely useful for implementing experiments, running them, and tuning hyperparameters. The parts they struggled with: choosing which question to investigate next, and getting unstuck when a line of research hit a dead end.

That division tracks with what I've observed more broadly. Current models are good at executing well-defined tasks with legible success criteria. They're much weaker at the kind of open-ended, direction-setting judgment that defines good research—the ability to look at a set of results and say "this approach is a dead end, let's try something structurally different." Whether that's a fundamental limitation or just a capability gap that more training closes is, genuinely, an open question.

What Jang's AlphaGo rebuild does, at minimum, is give us a controlled environment to think about these questions rigorously. Go is small enough to be tractable, complex enough to be interesting, and well-understood enough that you can tell when your AI is actually good at it. That's a rare combination.

The intelligence explosion debate—when it happens, what it looks like from the inside—probably won't be resolved by any single experiment. But the question of whether AI systems can learn to select the right problems to solve, not just solve the problems they're handed, seems like the crux of it. And AlphaGo, weirdly, is still one of the clearest lenses we have on what that would even require.

---

Yuki Okonkwo is Buzzrag's AI & Machine Learning correspondent. She covers the people and systems building tomorrow's algorithms.