Inside the Labs, the Doom Estimates Are Getting

A researcher named Jacob Cannell resigned from Anthropic this week with a public charge: neither OpenAI nor Anthropic, in his words, is "acting responsibly in the race toward self-improving superintelligence." He called the competition "gambling with our lives." Within five days, three separate signals had come from inside the frontier labs, all pointing in the same direction.

That sequence, documented on Peter Diamandis's Moonshots podcast with Emad Mostaque and recorded September 10, is the most interesting week in AI safety so far this year. Who is issuing them, how fast they are spreading, and how badly the people issuing them seem to agree with each other.

Three Signals in Five Days

The sequence, as laid out on the Moonshots podcast: first, OpenAI chief scientist Jakub Pachocki called for a voluntary slowdown. Then Sam Altman, reacting to OpenAI's claimed solution of the Navier-Stokes Millennium Prize problem using roughly 10,000 AI agents, wrote: "I did not expect a result of this magnitude to happen so soon. We've been talking a lot about the need to pace progress to ensure safety. For me, this is the strongest evidence yet for that urgency."

Then came Cannell's resignation, which reportedly drew more than 138 million views on X. Evan Hubinger, Anthropic's alignment science lead, followed up with a public confirmation: "Jacob is correct here. We really do earnestly believe AI could kill all humans. I personally think it is greater than 10% within the next decade." As The Verge reported, Hubinger added that Anthropic "does not yet have a plan to solve alignment for superintelligence and is not clearly on track." The Tech Buzz gave that estimate its own headline treatment, and the number is now circulating well beyond the effective altruist forums where it was once confined.

The spread matters more than the substance. A resignation letter from a three-month employee hitting nine-figure view counts, with Elon Musk publicly calling the engagement "very strange," is a new category of event. Peter Diamandis noted his own relatives were messaging him asking whether there was a 10% chance everyone dies. Emad Mostaque predicted the topic would hit "every dinner party, every political agenda" within weeks, ahead of a planned US-China meeting at the United Nations.

The Skeptics Inside the Optimists

Here is where the episode gets useful for anyone trying to actually think about this, rather than merely react. The same panel arguing about slowdowns could not agree on whether the warnings were credible.

Alexander Wissner-Gross, the computer scientist on the panel, dismissed Cannell's resignation as a "rage quit couched as virtue signaling," noting the researcher had joined Anthropic in July, roughly two months before quitting. He also raised the possibility of foreign influence operations behind the viral spread, pointing out that no similar public exits come out of Chinese frontier labs. On the 10% figure itself, he offered a Fermi-style rebuttal: if catastrophic AI outcomes were that likely, the galaxy would already show the wreckage of civilizations that got there first.

Salim Ismail put his personal probability at 0.1%. Diamandis, notably, holds a higher one, around 20%, down from 50%, and treats any nonzero number as unacceptable. Mostaque's concern is different again: the dangerous period is now, before models become reliably smart, while powerful capabilities sit in everyone's hands.

So the terrain runs from 0.1% to 25% (Dario Amodei's own public figure from a year ago), held by people with access to the same information. That spread is not noise. It is the finding.

The Breakthrough Nobody Verified

The other half of the week is the Navier-Stokes claim itself. OpenAI says a swarm of agents produced a solution to one of the seven Clay Millennium Prize problems. Rumors are already circulating that OpenAI and Anthropic hold solutions to the Hodge conjecture and the Birch and Swinnerton-Dyer conjecture as well. None of this has been independently verified, and the panel treated the rumors as speculation.

The key points from the episode are explicit on this: alleged solutions to major mathematical problems require independent validation, reproducibility, and clear benchmarks before counting as established science. A claimed proof from the company selling the compute is a press release until someone else checks the math. History gives us reason to wait. Announced breakthroughs in this industry have a documented habit of shrinking on inspection.

Still, the economics behind the claim deserve attention. Noam Brown of OpenAI addressed the "who can afford that" objection directly: the o3 model cost about $500,000 to score 87.5% on ARC-AGI-1 when announced; today higher scores cost around $20. A 25,000-fold price collapse in under two years. If that curve holds, a Millennium Prize computation by late 2027 could cost about as much as coffee. The bottleneck, as panelists put it, shifts from human genius to the physical world: implementing what the machines derive.

The Steering Problem

The genuine disagreement in the episode, and in the field, is over what to do. Nobody on the panel argued for a hard stop, and Ismail's reasoning is the strongest version of that position: a slowdown mainly burns time while other governments and private actors advance, and no enforcement mechanism exists anyway.

The pro-measurement camp, led by Diamandis on the episode, wants labs to publish alignment benchmarks, optimization functions, and monitoring plans before asking for public trust or public money. That demand is sound and overdue.

But Alexander Wissner-Gross raised a complication that undercuts the easy version of it: instruction tuning, arguably a form of alignment, delivered the equivalent of a 10,000x capability jump. If alignment work and capability work are the same work, public funding for safety is also a public subsidy for power, and the coordination problem gets harder, not easier. That tension connects directly to prior coverage of the antitrust problem with coordinated slowdowns, and to the broader split between safety-first and fast-deployment strategies that has divided the labs for two years.

One more thing the optimists in the room got right, and the doomers rarely concede: the warning system appears to be functioning. Internal critics can resign loudly, get 138 million views, and force their former employers' alignment leads to respond in public. That messy, noisy argument is the common ground between the field's optimists and pessimists, and it is a better sign than silence.

The open question is whether the response to this news cycle will be measurement or messaging. Altman's post reads, at this distance, like both a genuine surprise and a positioning play ahead of US-China talks. The labs say they lack a plan for aligning superintelligence while spending at full speed to build it. Both statements can be true. Only one of them is scheduled to be tested, and the test runs whether or not anyone is ready.

Bob Reynolds is Senior Technology Correspondent at BuzzRAG.