AI Benchmark Scores Are Broken. Here's Who's

There's a wastewater treatment plant engineer in Turkey who spent 20 years watching safety protocols get ignored. He's seen people die as a result. So when AI tools started proliferating, he did something almost no AI lab would have done: he built a benchmark to test whether those tools could actually help someone in his job—handling chemicals, reading equipment, responding to emergencies. He built it from two decades of field experience. That data doesn't exist anywhere else on the web.

No AI lab funded it. No AI researcher created it. The people with the most at stake built it themselves, because nobody else was going to.

That's the premise behind a talk Nicholas Kang and Michael Aaron from Google DeepMind's Kaggle team gave at AI Engineer—and it's the reason I think their work matters beyond conference circuit chatter. If the people who actually depend on AI for high-stakes decisions aren't shaping what gets tested, benchmarks end up protecting labs' reputations, not users' safety.

The numbers you're shown don't mean what you think

When an AI company releases a new model, you get a chart. Numbers, rankings, benchmark names you half-recognize. The implicit message: this model is better than the last one, better than the competition, here's the proof.

Here's what that chart doesn't tell you.

Kang described a situation where Kaggle published a benchmark with a major AI lab, and a competing lab took one look at the results and didn't like them. So they reran the benchmark—with their own compaction settings, provided through their own API—and published substantially better numbers. "The results you're seeing don't always reflect the actual state of things," Kang said, "and that's a problem."

Neither set of numbers was fabricated. Both were, for practical purposes, useless for comparing the models.

This is the specific dynamic that I keep watching play out across the industry: benchmark results that are technically accurate and contextually meaningless at the same time. The jagged benchmark story isn't new—but what Kang and Aaron are describing is the mechanism underneath it. When you control the test environment, you control the outcome, even without touching the model.

The SWE-Bench Pro example Aaron cited from a Morph LLM blog post makes this concrete: six frontier models cluster within a couple of percentage points of each other. But the harness—the scaffolding the model runs inside—shifts performance by as much as 22%, according to the presentation. Aaron was explicit that he hadn't independently verified the blog post, but found it credible. If that figure holds up, the industry has been publishing model comparisons where the scaffolding matters more than the model. That's not a measurement problem. That's a credibility problem.

And even setting aside manipulation, there's simple entropy. Kang estimates—informally, from a conference stage, not from a verified count—that something like ten or more new benchmarks appear on arXiv every single day. The leaderboards in those papers go stale the moment the authors move on to their next publication. "I can't even do it even though it's my full-time job," he said of keeping up with them.

Four things Kaggle is building, and what they're actually for

Kang and Aaron are working on four interconnected pieces of infrastructure. They're worth understanding not as a product roadmap but as a diagnosis of which specific problems they're trying to solve.

Hackathons are the community-sourcing mechanism. Kaggle is running one right now with Google DeepMind's AGI team—asking contributors to build benchmarks around specific cognitive faculties identified in a recent DeepMind paper. The explicit goal is to get people like the Turkish wastewater engineer into the evaluation pipeline, not just AI researchers. The honest challenge Kang flags: you need human expert judgment to assess creative benchmark submissions, and even experts don't always agree with each other. AI can't adjudicate that yet.

Standardized agent exams are the part that addresses a gap I find genuinely alarming. Kang describes two ends of the eval spectrum: enterprise teams with serious evaluation tooling on one end, and hobbyist developers shipping personal AI agents with no testing on the other. "Most of them aren't actually testing their agents before they're sending them out to the real world," he said. These are agents running people's inboxes, placing orders, managing accounts. The exam launched as an experimental MVP with a one-line prompt interface—pass a prompt to your agent, get a score on a leaderboard. The name was originally "Standardized Agent Tests" until, Kang says, they hit a trademark issue and had to rename it. He described it as a true story, though the talk doesn't document the specific conflict.

What caught my attention: within a week of launch, with almost no promotion, the platform had over 500 agent submissions. Someone even built an exam prep course for it in a Kaggle Notebook. That's not a marketing success story—it's evidence that developers actually want a quick way to baseline-test their agents before deployment and had no good option until now.

Game Arena solves the saturation problem with an elegant bit of adversarial design. Static benchmarks get saturated—models eventually max them out and they stop being useful. So Game Arena has models play PvP games against each other: poker, chess, Werewolf. ELO scores can never saturate because there's always a winner and a loser. Aaron mentioned one finding I didn't expect: some newer-generation models perform worse at poker because they're more risk-averse. "You just see these personalities start to emerge over time," he said. The catch is cost. Getting statistical significance on poker required running 400,000 hands, with multiple turns each—Aaron left the API bills to your imagination.

Benchmarks (the platform, distinct from the concept) is where anyone can build, run, and publish evaluations openly—including the wastewater engineer's safety dataset.

What I think about all of this

I'm not a neutral observer here. I cover what happens when people trust systems they shouldn't, and AI benchmark marketing is one of the more successful trust-manufacturing operations I've seen in technology. So yes, I'm glad someone is building community infrastructure to make evaluations more transparent.

But I'm also clear-eyed about what this doesn't solve. Open community benchmarks help when the problem is that the wrong people are writing the tests. They don't help when a company decides to optimize its scaffolding and publish selectively. Transparency requires that labs actually use open infrastructure—and right now they have no obligation to. The Kaggle platform can produce better benchmarks; it can't compel anyone to run them honestly.

The agent exam piece is where I feel most urgency, and where the gap between benchmarks and real-world behavior gets personal. If you're using an AI agent to manage anything consequential—email, finances, calendar—you deserve to know what baseline safety testing it's had. Most consumer agents have had none. A one-line exam that returns a safety score isn't perfect evaluation, but it's infinitely better than vibes. The fact that 500+ developers ran it in a week without a marketing push tells me the demand is real.

So here's what I'd have you take from this: the next time you see a benchmark chart in a model announcement, ask two questions. First, who ran the test? If it's the lab releasing the model, or a partner lab using that lab's infrastructure, you're looking at a press release formatted as data. Second, what was the harness? The environment the model runs inside can swing performance by more than the model differences being advertised—a finding that, if the Morph LLM numbers cited in this talk hold up, means we've been reading these charts wrong for years.

Kaggle's community infrastructure is a meaningful attempt to broaden who gets to set the terms of AI evaluation. Whether enough labs actually adopt it to shift the industry's incentives is a different question. But the wastewater engineer already built his benchmark. That data exists now. Whatever happens next, that's something.

Rachel "Rach" Kovacs is Buzzrag's cybersecurity and privacy correspondent.