Why AI Training Data Quality Beats Raw Compute

Every few years, the field discovers that the thing everyone assumed was solved is actually the hard part. In the early days of deep learning, it was compute. Then it was architecture. Right now, if you ask the people actually shipping production AI systems rather than writing about them, a surprising number will tell you the same thing: it's the data.

Mahesh Sathiamoorthy made that case at the AI Engineer conference, and he came with receipts. As co-founder and CEO of Bespoke Labs — and formerly a researcher at Google DeepMind — he has spent the past year building tools and datasets specifically for what the industry calls post-training: the process of taking a capable base model and shaping it into something that reliably does a specific job.

His argument deserves serious attention. Not because it's counterintuitive — it isn't, once you think about it — but because the industry's investment patterns suggest most organizations still haven't internalized it.

Agents Don't Fail at the Algorithm Level

The shift Sathiamoorthy frames first is a useful one. Early AI benchmarks tested what models knew — fill in the blank, answer the question, pass the exam. The benchmarks that matter now test what models can do: complete a multi-step coding task, navigate a web interface, run autonomously for hours without someone supervising every move.

Reliability is what blocks that autonomy. Not raw capability — reliability. An agent that's 95% reliable over a ten-step task has roughly a coin flip's chance of completing it without error. Stretch that to fifty steps and you're hoping for a miracle. "At some point something falls apart," Sathiamoorthy said. "Either they called the wrong tool or they made a mistake."

Post-training is how you close that gap. It's the stage after a model's initial training where you further shape its behavior — either through supervised fine-tuning, where you show it correct examples (think of it as guided practice with worked answers), or through reinforcement learning, where the model tries things and gets scored on outcomes (more like practice with a referee). Both techniques depend entirely on what you feed them.

This is where Sathiamoorthy's central claim lands: "For post-training, be it SFT or reinforcement learning, data is the bottleneck." Compute is well-defined. Infrastructure for running training jobs exists from multiple providers. The base models are good. What's scarce is high-quality, thoughtfully curated training data — and, increasingly, the environments in which agents learn by doing.

I find this persuasive, partly because it rhymes with how quality problems have manifested in every previous software transition. The internet didn't fail because servers were slow; it failed, when it failed, because the content and the incentives around it were a mess. Enterprise software didn't fail because the algorithms were wrong; it failed because the data going into those systems was dirty, incomplete, or structured for the wrong purpose. The pattern keeps repeating: we sort out the plumbing faster than we sort out what flows through it.

What OpenThoughts Actually Taught Them

Bespoke's most public work on this problem is OpenThoughts, a reasoning dataset the team built after DeepSeek's release exposed a gap in publicly available high-quality reasoning data. Sathiamoorthy describes recruiting collaborators from several universities — Stanford, UC Berkeley, and the University of Washington among them, according to his account — to formalize the effort and publish the methodology.

The pipeline they developed is more methodical than the field's typical approach. Start with source questions from existing datasets. Use other models to assess question quality and difficulty. Generate answers using "teacher models" — larger or more capable models whose outputs train the smaller one, a technique sometimes called distillation. Filter those answers. Then test every stage independently — what the field calls ablations — to find out which choices actually move the metrics before committing to the full recipe.

What came out of that process surprised them. Two findings in particular push back against the intuitive approach.

First, sampling multiple answers to the same question outperformed breadth. Rather than covering more ground with one answer per question, generating many different answers to fewer questions produced better-trained models. Sathiamoorthy's explanation is that the variety in how a model reasons through a problem matters as much as the range of problems it sees. The reasoning traces — the step-by-step working that modern models produce before arriving at an answer — carry information about process, not just outcome. Diversity of process, it turns out, is something you can train for.

Second, stronger teacher models don't always produce better students. This one cuts against the obvious assumption that you should always use your best available model to generate training data. In Bespoke's experiments, some smaller or differently-specialized models produced training signal that transferred better to the target model. Sathiamoorthy observed this pattern in both their reasoning work and their subsequent agent-training work, where certain models outperformed ostensibly more capable alternatives as teachers. The mechanism isn't fully understood — this is an empirical finding in search of a clean theoretical explanation — but it recurs consistently enough to be actionable.

Both findings matter because they suggest the intuitive recipe is often wrong. More questions, stronger teachers, filter aggressively — that's the approach most teams would default to. Running the ablations first, stage by stage, to find out what actually works: that discipline is harder to maintain under deadline pressure, but it's what separates a dataset that moves metrics from one that wastes compute.

When the Theory Meets a Compliance Deadline

The most concrete illustration in Sathiamoorthy's talk involves a credit card recommendation feature in the Credit Karma app. Users see an explanation for why a particular card has been recommended. The problem: a raw model, even a good one, kept producing responses that weren't regulatory-compliant. Cramming more compliance rules into the prompt solved the problem but made responses slow — because the model was processing an enormous list of restrictions on every query.

The post-training answer seems obvious. Fine-tune the model on compliant examples. But the dataset was badly imbalanced — certain financial figures, like promotional APR rates, appeared so rarely that a fine-tuned model would hallucinate the numbers when it encountered them. Teaching the model the right pattern without letting it memorize the wrong specifics required a curation trick: adding structured tags to the training data that directed the model's attention toward the form of a compliant response rather than the content of any particular number.

The result was a model that handled compliance correctly, responded faster, and was cheap enough to own outright — meaning Intuit could update it on their own schedule rather than depending on a frontier model provider's release cycle. Sathiamoorthy noted pointedly that as frontier model prices rise, that ownership argument gets stronger.

I'd add: the latency and cost gains here are real and measurable, but the more durable benefit is control. Enterprises that rely entirely on frontier API access are, in effect, renting their capabilities. When pricing changes, when a model is deprecated, when a provider's terms shift — they're exposed. Post-training a smaller model on proprietary data changes that equation. The economics will drive adoption here faster than any benchmark result will.

The Infrastructure Nobody Talks About

The final piece of Sathiamoorthy's talk is about the scaffolding that makes all of this possible at scale. His Curator tool handles the data pipeline side — ingesting prompts from existing datasets or production logs, calling teacher models to generate responses, and outputting clean fine-tuning sets. But for agent training, the infrastructure problem goes deeper.

Reinforcement learning environments for agents — the simulated settings where a model learns by trying things and receiving feedback — are, as Sathiamoorthy puts it, "data in a different shape." Building those environments, measuring whether they're any good, versioning them as you iterate, running the rollouts (the sequences of actions an agent takes through a task), and being able to checkpoint and roll back when something breaks: that's a full engineering problem underneath the research problem.

The field has good algorithms. It has compute. What it doesn't yet have are standardized, verifiable, high-quality environments for training agents on the tasks that actually matter to enterprises. That's the gap Bespoke is building toward.

I don't know whether Bespoke will be the company that closes it. The tooling space for AI development is crowded and moving fast, and the gap between "useful open-source research project" and "enterprise infrastructure layer" is wider than most teams expect. What I do think is that Sathiamoorthy has identified the right problem. The field has been remarkably good at building algorithms and remarkably casual about what those algorithms train on. The next wave of capable, reliable AI agents will be built by whoever gets the data right — and right now, most organizations are still guessing at the recipe.

---

By Bob Reynolds, Senior Technology Correspondent, BuzzRAG