AI Leaderboards Are Lying to You About

Someone on your team asks: "what's the best image generation model right now?" You check Design Arena. ChatGPT Image is #1. You ship it. Quick, defensible, done.

Bertrand Charpentier, cofounder and chief scientist at Pruna AI, has a talk that exists specifically to ruin that feeling.

Speaking at AI Engineer, Charpentier walks through why the standard approaches to model selection—public leaderboards, internal benchmarks, vibes-based manual inspection—systematically steer teams toward the wrong answer. Not occasionally. Structurally. The argument is clean enough that I kept nodding even when I knew I should be asking harder questions about who's delivering it.

The leaderboard problem is worse than you think

The obvious critique of public leaderboards is that they're gameable. Charpentier's critique is different and more interesting: they're inconsistent with each other in ways that suggest the rankings themselves are mostly noise.

Take image editing. He compares three major leaderboards—Arena (formerly LM Arena), Design Arena, and Artificial Analysis—and they produce completely different rankings for the same models. One model he cites in the talk jumps from rank 10 on Artificial Analysis to rank 5 on Arena. (These are Charpentier's figures from the talk; leaderboard positions shift constantly and weren't independently verified for this piece, but the directional point holds even if the specific numbers move.)

The deeper issue is what Elo scores actually are. Leaderboards use Elo — a relative ranking system borrowed from chess, where your score reflects how you perform against other competitors in the pool, not some absolute measure of quality. That's why cross-leaderboard comparisons collapse: an Elo of 1,200 on Design Arena and an Elo of 1,200 on Artificial Analysis are measuring performance against different sets of opponents in different battle conditions. Comparing them is like comparing your Elo in a local chess club to a grandmaster tournament — the number looks the same, the context is completely different. This is a big part of why the benchmark reliability question keeps coming up across model families.

Then there's the win-rate problem. "Most of the models they lose at least 40% of their battles," Charpentier says. "And if your use case is in this 40% of the battles, it means that you will just take the wrong model." The model ranked #1 on a general leaderboard is the actively wrong choice for a substantial slice of real tasks. And when you break leaderboards down by specific use case — removing objects, editing text, changing backgrounds — ChatGPT Image, which sits at the top of the aggregate ranking, "is never top one in this ranking." Different models dominate different subtasks, which is exactly what you'd expect from models trained differently on different data mixes.

Manual inspection is also broken, and in a fun way

The internal benchmark section is where the talk gets unexpectedly entertaining. Charpentier runs a live audience experiment: show three images generated by different models (Stable Diffusion, Flux, and a Pruna-optimized model), ask people which they prefer. Audience splits meaningfully. Then he shows three different images with the same model lineup — and some people change their answers.

The point: manual inspection is double-biased. You're biased toward your own aesthetic preferences, and you're biased toward whatever tiny sample you happen to be looking at. "When you do manual inspections, you are two times biased," he says. Good to get a feeling, not enough to make a decision.

Automated metrics have their own version of this. CLIP score — a standard metric for measuring how well an image matches a text prompt — produces wildly inconsistent rankings across datasets, with score differences between models so small they're hard to read as signal rather than noise (that's my read on the variance he's describing, not a claim he makes explicitly). Use-case-specific metrics, like text rendering accuracy, show much larger and more consistent gaps between models. The lesson: generic metrics are probably measuring generic things, and generic things aren't what your product does.

The 400-marathon number

Okay, here's where I need to tell you about the math that made me put my phone down for a second.

Evaluating ChatGPT Image the way Design Arena does it — 26,000 battles, ~62 seconds per generation — takes 20 days of compute, costs $5,000, and burns roughly 556 kWh of energy. Charpentier converted that energy figure using his own Strava data from running marathons. It comes out to approximately 400 marathons.

I don't know if "400 marathons" lands as intuitive for you — I'm not sure it's the most illuminating unit of energy either — but the 20-days-vs-7-hours comparison does land, and it's the one that actually matters for how teams work. Charpentier's own compressed model, he says (per Pruna's own figures, not independently verified), runs the same 26,000-image evaluation in 7 hours at $265. That's not a rounding error. That's the difference between "we'll have results before the sprint ends" and "we'll have results before the quarter ends."

The honest caveat: every one of these figures comes from Charpentier's own talk, presented by a cofounder of the company that makes the faster model. I'm going to get to that.

The Pareto frontier, which is just "good enough at the right price"

The tool Charpentier actually recommends for model selection is the Pareto frontier, and if you haven't encountered this concept it sounds scarier than it is. Here's the version that clicked for me:

Imagine you're buying headphones. You could rank them purely by sound quality — but then you'd always buy the $4,000 audiophile cans, and most people don't need $4,000 headphones for their commute. The real question is: for a given quality level, what's the cheapest option? For a given budget, what's the highest quality? The set of products where you can't improve one dimension without sacrificing the other is the Pareto frontier. Everything else is just a worse deal.

Applied to AI models: plot quality score on one axis, latency (or cost) on the other. The models on the frontier are your actual candidates — the rest you can ignore. Charpentier's finding is that three or four models typically cluster within a narrow quality range (roughly 1,100–1,200 Elo) while varying up to 20x in speed. The "best" model by quality alone might be 20x slower than something with a nearly identical score for your use case.

Evaluated this way, the benchmark story shifts pretty dramatically: smaller, specialized models start appearing on the frontier in ways aggregate leaderboards never surface them.

The conflict of interest is real and you should hold it

Here's the thing I can't not say: Charpentier is cofounder and chief scientist of Pruna AI, a company that sells compressed, efficient AI models. The entire analytical framework he's presenting — efficiency belongs in the state-of-the-art definition, Pareto frontiers surface smaller models, general leaderboards steer you toward large foundation models — happens to lead directly to "and therefore you should consider using something like what we sell."

That doesn't make him wrong. The structural arguments about leaderboard inconsistency and the 40% win-rate floor are solid regardless of who's making them. But the cost comparisons are Pruna's own figures. The compressed model benchmarks are Pruna's own models. When he says evaluating on the Pareto frontier will "usually not find a large foundational model, but more like a lot of small performance models," that is genuinely true and also extremely convenient for his product positioning.

I'm not accusing him of bad faith. I'm saying: the diagnosis here is well-supported, and the prescription happens to be what his company sells, and those two facts can coexist without either canceling the other out. Read the framework. Steal the Pareto plot idea. Just don't let a vendor define your evaluation criteria without sanity-checking that the criteria don't have a thumb on the scale.

The actual question his talk raises — what does state-of-the-art even mean when the answer depends entirely on your use case and your compute budget? — is one the field hasn't fully answered. Benchmarks aren't dead, as Charpentier puts it. But the version of benchmarking where you check one leaderboard, pick the top model, and feel good about it? That one might be.

Yuki Okonkwo is Buzzrag's AI & Machine Learning correspondent. She covers the people building the algorithms and the systems they're releasing on the rest of us.