Data Is Now the Hard Part of Building AI

At some point in the last several years, the AI field quietly flipped its priorities. The researchers who used to spend most of their graduate careers thinking about architecture now spend most of their time thinking about data. The venture capitalists who once said data businesses had zero terminal value have since watched the data sector generate over a hundred billion dollars in market capitalization. These aren't subtle shifts.

That reversal was the organizing thesis behind a recent Y Combinator Paper Club session focused entirely on data—training data, evaluation data, and multilingual data. Three speakers made their cases. They weren't selling the same product, but they were diagnosing the same disease.

The Ph.D. Interview That Filtered for the Wrong Instinct

Francois Chaubard, who founded Focal Systems after leaving graduate school in 2016, opened with a story that doubles as a diagnostic. When he ran the deep learning team, he asked every candidate the same question: you've trained your model and you're at 85% F1. What do you do next?

The wrong answers—the ones that got candidates cut—were about architecture. Try ReLU instead of GELU. Add more layers. Read more papers. The candidates giving those answers weren't bad engineers; they'd just been trained in an environment where the benchmark data was clean, representative, and effectively infinite. The real world, Chaubard found, is none of those things.

His example: a computer vision system trying to determine whether grocery items are in stock. When a refrigerator fogs up, or a customer stands in the aisle blocking the camera, no amount of architectural tinkering helps. The model fails because the situation wasn't in its training data—full stop. "Many people think that these are just ZIP files," he said, "but this is not the case at all."

His broader argument is that datasets and reinforcement learning environments should be thought of as products—crafted artifacts requiring the same level of design and iteration as any software application. The analogy he reached for: Apple doesn't build Instacart. The iPhone enables thousands of apps to exist, but the hard domain-specific work—the logistics networks, the supply chains, the edge cases—gets done by companies that specialize. That's roughly where he sees the AI data ecosystem heading: a layer of foundational models sitting above a sprawl of highly specialized data companies building the infrastructure for specific professions. How many data companies can there be? He'd answer that with another question: how many apps are on your phone?

This argument—that data curation rather than raw compute is the real constraint—has been making the rounds with increasing frequency. What makes Chaubard's version interesting is the specificity: he's not talking about data in the abstract, but about the ontological headaches. Should a "product pusher" (a device that shoves items to the front of a shelf) be labeled "out of stock"? Technically, the product is absent. But visually, it looks nothing like an empty shelf, and labeling it that way confuses the model. These are not computer science problems. They're judgment problems. And expert judgment is notoriously hard to scale.

Benchmarks as a Research Instrument

Vincent Chen, a founding team member at Snorkel, picked up where Chaubard left off and moved into the mechanics of how expert knowledge actually gets captured in datasets. His framing: we're moving from "Data 1.0"—basic labeled pairs, thumbs up/thumbs down, 30 seconds of human judgment per example—to something he calls "Data 2.0," where a single evaluation environment can take hundreds of hours to build.

The concrete case is Senior SWE-Bench, developed with the original SWE-Bench team at Princeton. The problem it's trying to solve is one that anybody who has used AI coding tools has bumped into: we're already using these agents to do senior-developer-level work—architectural decisions, codebase refactoring—but we're still evaluating them like junior developers. The existing benchmarks hand agents detailed, step-by-step instructions. Senior SWE-Bench hands them something more like a Slack message: "here's a log dump, figure it out."

Getting evaluation right at this level turns out to require a full spectrum of verification approaches. Hard-coded unit tests are reliable but brittle—they can penalize code that's actually correct just because it takes a different path to the solution. LLM-based judges are flexible but unreliable—they'll sometimes reward plausible-looking code that doesn't actually execute correctly. Snorkel's approach threads between these: a "validation agent" that takes an expert-written specification and generates deterministic tests against the actual code path, checking both correctness and what Chen calls "tasteful pass"—whether the code reflects the practices a senior engineer would recognize as appropriate, not just whether it runs.

That last metric matters more than it might seem. A model that passes every unit test while producing unmaintainable, overengineered code is genuinely useful to no one. The question of whether AI-generated code is any good is still mostly being answered by whether it compiles.

When Synthetic Environments Meet Real Complexity

Volo Kuleshov from Inception Labs came with a different angle: diffusion-based language models, which generate all tokens simultaneously rather than sequentially, hitting speeds that conventional autoregressive models can't match on standard hardware. The application he's most focused on is real-time voice—the kind of low-latency interaction that customer support and educational agents need but haven't been able to deliver because the LLM in the middle of a voice pipeline is too slow.

The speed story is real. What's more interesting is how it loops back to data. Kuleshov's critique of the dominant voice benchmark, Starbench, is that it's been over-tested to the point of unreliability—models that look excellent on the benchmark underperform substantially on real production logs, and the gap between model sizes looks far more dramatic in deployment than the scores would suggest. That gap is the tell: when a benchmark stops discriminating meaningfully between real-world performance levels, it's no longer measuring what it claims to measure.

To address this, Inception built a synthetic environment generator called Dial Forge. Feed it a domain—banking customer support, flight booking, medical scheduling—and it synthesizes realistic reinforcement learning environments populated with simulated users, policy documents, and task objectives calibrated to the target difficulty. The system draws from real production logs where available, and from a scraped "business knowledge graph" where it isn't.

The engineering logic is sound. But there's a specific failure mode worth sitting with. Dial Forge builds banking environments from internet-sourced business knowledge. A real bank's customer support agent faces not just policy documents but exception hierarchies that are never written down anywhere—the unwritten rules about which fee waivers are approved without escalation, which account types trigger manual review, which customer complaints require human handoff regardless of what the policy says. A model trained on a well-constructed synthetic banking environment will perform well on synthetic banking tasks. Whether that transfers to the actual bank is an empirical question, and one that's only answerable after deployment. Kuleshov acknowledged as much in framing the system as iterative, with real production feedback flowing back into the generation procedure. That's the right posture. It doesn't eliminate the gap between the training distribution and the edge cases that accumulate in live systems; it just makes the loop for closing that gap faster.

The Languages AI Forgot

The session closed with Shayne Longpre, who recently finished his Ph.D. at MIT and now works at Anthropic, presenting research on multilingual pre-training conducted during an internship at Google. The framing was blunt: scaling law research is overwhelmingly focused on English. The rest of the world has received, in Longpre's telling, very little attention—which creates real consequences for the billions of people who don't use English as their primary language.

The core research question is about transfer: when you train a multilingual model, which languages help each other, and which hurt each other? The answer is neither intuitive nor linguistically predictable. Longpre's team built a matrix of cross-language transfer effects measured empirically across dozens of languages. For Spanish, Portuguese and French are synergistic; Japanese is among the most harmful. For Thai, Indonesian is surprisingly useful—not because they're in the same family, but because the data coverage overlaps in meaningful ways.

Two things stand out. First, the relationships aren't symmetric: the fact that Portuguese helps Spanish doesn't mean Spanish helps Portuguese by the same amount. Practitioners who run an experiment one way and assume the reciprocal holds are going to make systematic errors in their data mixing decisions. Second, script matters more than language family for determining useful data pairings—probably because of how tokenization works, not because of anything linguistic. A Thai model benefits more from data in scripts that tokenize similarly than from data in typologically related languages that tokenize differently.

This has immediate practical implications for anyone building models for non-English markets. The scaling laws that govern English-centric training don't transfer cleanly to a multilingual setting. Longpre's ATLAS work provides a way to decompose the standard Chinchilla framework into monolingual and cross-language components, making it possible to predict training outcomes before running hundreds of expensive experiments.

What the session kept circling back to, regardless of speaker, is that data problems compound in the same direction as capability claims. Every time AI agents get tasked with something harder—not just "is this item in stock" but "handle this patient's insurance dispute"—the data required to train them gets harder to source, harder to label, and harder to evaluate. The architecture problem was always finite. This one isn't.

---

Marcus Chen-Ramirez covers AI, software development, and the intersection of technology and society for Buzzrag.