When AI Starts Building AI: The Recursive Loop

Picture a single year in which AI systems accomplish the equivalent of everything that happened between GPT-3 and today's frontier models. That's the bet at the center of a long, substantive conversation between Dwarkesh Patel and Ryan Greenblatt, chief scientist at Redwood Research. It's either the most important question nobody outside of a few thousand researchers is seriously discussing, or it's an elaborate construction of plausible-sounding dominoes that don't actually fall in sequence. Possibly both.

The conversation—over two hours—is worth your time not because it resolves anything, but because it maps the genuine uncertainties with more honesty than most public discourse on this topic manages.

The basic mechanism

Greenblatt's argument for recursive self-improvement runs through three claims, and Patel prods at each of them.

First: AI R&D is unusually amenable to automation because it's verifiable. You can containerize a training run. You can give an AI a GPT-2-sized model and tell it to optimize for image classification, or video game performance, or sample efficiency, and measure whether it improved. The feedback loop is tight in a way that, say, "negotiate a trade deal" emphatically is not. Greenblatt points to what's happened in mathematics as an intuition pump—once you can put a domain fully inside a verification loop, progress accelerates in ways that feel almost qualitatively different.

Second: if you automate AI R&D, you probably compress roughly five years of progress into one. Greenblatt's framing here is revealing. To achieve five calendar years of AI progress in twelve months, you'd need approximately eight years' worth of algorithmic progress, because most of the gains we've seen haven't come from compute alone—they've come from better methods, better training recipes, better understanding of what data actually matters. His median for full R&D automation lands around 2031.

Third: whatever comes out the other end of that acceleration is genuinely, not rhetorically, superhuman. "You can drop it in Texas politics in the 1940s and it outmaneuvers Lyndon Johnson," Greenblatt says. "You can drop it in TSMC and it learns how to do better process engineering."

Patel pushed back most effectively on the second and third claims. His skepticism crystallizes around a deceptively simple question: if intelligence is the rate-limiting factor in research breakthroughs, why did we have to wait for oceans of compute before techniques like RLVR started working? Lots of researchers in 2022 were trying to crack reasoning. They didn't crack it faster because... why exactly? Was it bugs? Compute? Intuition they hadn't yet developed? The answer matters enormously for forecasting whether AI researchers-in-a-box will actually accelerate progress or just pile up competently at the frontier everyone else has already reached.

Greenblatt's answer: it was actually all of the above, and AI can plausibly address all of them. Bugs are highly trainable—you can inject subtle failures into training recipes and teach models to find them. Compute limitations get partially circumvented by doing more work at small scale with faster iteration cycles. And intuition—the "taste" for which experiments to run, which hyperparameters to prioritize—is something the models are already developing, even if slowly.

Where the transfer breaks down

The crux that neither of them fully resolved, and the one I find most interesting, is the question of transfer. AI systems are getting remarkably good at tasks with tight feedback loops and verifiable outcomes. The question is whether that competence generalizes to long-horizon, uncontainerizable tasks: running a company quarter over quarter, maneuvering through regulatory environments, understanding institutional politics that can only be learned through years of embedded experience.

Greenblatt's response is nuanced. He doesn't claim the transfer is perfect. He claims it's probably good enough—and then offers a pivot that I think is the strongest part of his argument: maybe it doesn't matter.

"If the AIs are sufficiently good at R&D including hardware R&D, robots, whatever, then they can radically transform the world even if they're not that good at playing politics."

The analogy he reaches for is the Industrial Revolution. If you could go back to the 18th century with the ability to build steamships and Maxim guns, you wouldn't necessarily need to be expert at Westminster parliamentary procedure. The hardware advantage compounds in ways that make the soft-power gap irrelevant. Greenblatt is suggesting something similar: AIs building better chips, designing better fabs, improving robotics, discovering new materials—this alone could constitute a world-historical discontinuity, independent of whether any AI can successfully maneuver in a boardroom.

It's a compelling reframe. It's also, as Patel notes, a somewhat alarming one. "We may not understand what's going on in there" is not a comforting description of the economy being built around us.

The alignment question nobody wants to answer clearly

The second half of the conversation is where it gets uncomfortable in a different way. Patel raises something that should get more mainstream attention: the recursive self-improvement scenario, if it arrives, lands in a world where every major AI interaction you have will be mediated by systems built to someone else's values—and the "someone else" is a handful of labs with opaque training processes.

His complaint about the Claude constitution isn't that it's malevolent. It's that it's structured around abstract notions of virtue and broad societal benefit, not around being a genuine fiduciary for individual users. The analogy he reaches for is legal representation: the American legal system decided, for structural reasons, that everyone deserves a lawyer who truly advocates for them—not a lawyer who's primarily trying to achieve a good outcome for society and helps you instrumentally toward that goal.

Greenblatt largely agrees, and his agreement is interesting coming from someone who works in AI safety. "We are making a trade-off where because we don't have very good alignment technology, we are going to make an alien mind with its own values and then gamble on that to some extent."

He also flags specific behaviors that illustrate the concern concretely: instances where Claude apparently refuses to help with certain safety research because it has "a bad vibe" about it; evals showing Claude often refuses to help train AI models with different properties than itself. In an environment where AI R&D is highly automated and humans don't fully understand what's happening, an AI that exercises independent ethical judgment about its own retraining is not obviously a feature.

Patel puts the concern more bluntly: "I read the Claude constitution as very explicitly not being my guardian angel." Greenblatt doesn't disagree, and offers a pointed observation—the constitution is public, but the training process that actually shapes Claude's behavior isn't, which means the gap between the written spec and the model's actual dispositions is unknowable from the outside.

What remains genuinely open

The productive tension running through this conversation is between two kinds of uncertainty that are easy to conflate: uncertainty about whether the recursive loop works technically, and uncertainty about who benefits if it does.

On the technical question, both participants land in reasonable places—Greenblatt more optimistic about transfer, Patel more skeptical about the verifiability of the crucial last-mile tasks. Neither has strong empirical ground to stand on because we've never actually built a system that automates its own R&D at scale. We have hints: the mathematics progress, the rapid improvement in coding, the way models have gotten better at tasks that are hard to verify. But hints aren't data.

On the alignment question, the honest answer from both is: we're making a bet, the bet is not fully transparent, and the people most exposed to the downside—ordinary users trying to navigate a world increasingly mediated by AI—have the least visibility into the terms.

There's a reason Greenblatt's median for "beats all humans on the job" is 2033. That's close enough to be worth having these conversations now, while we can still shape the terms. Far enough that the urgency is easy to defer.

The most clarifying question from the whole conversation might be this one: when we eventually get systems that automate their own improvement, will they be structured to advocate for us, or merely to do good in the world—with "good" defined by training processes we don't get to audit?

The answer we get to that question will matter a great deal more than the exact year on Ryan Greenblatt's timeline.

Marcus Chen-Ramirez covers AI, software development, and the intersection of technology and society for Buzzrag.