How Hugging Face Built FineWeb, a 15T Token LLM

You've noticed it, right? Google Search feels different. Your For You page is full of posts that are weirdly well-structured, suspiciously balanced, never quite wrong but also never quite human. The web has a new texture to it, and it's not just your imagination. Here's the part that gets me: that same texture—AI-generated text flooding the internet—might actually be making the AI models of tomorrow better. Not worse. Better. We'll get to that. First, let me explain how we even know this, because it comes from one of the most transparent looks at LLM sausage-making that exists in the open.

Hugging Face published a project called FineWeb: a 15-trillion-token English dataset derived from Common Crawl, documented step by step in a blog post and accompanying paper, and recently walked through on camera by Alejandro AO in a nearly 30-minute breakdown. Most AI companies publish their models. Almost none publish their training data—there's no competitive incentive to hand that over. FineWeb is the rare exception, and the documentation of how it was built is where the interesting stuff lives.

Raw internet is exactly as gross as you'd expect

Common Crawl is a nonprofit that takes snapshots of the entire web every month or two and has been doing it since 2013. Hugging Face started with 96 of those snapshots. Each one is hundreds of terabytes of raw HTML. Navigation menus, cookie banners, Lorem Ipsum, ads, scraped duplicates of scraped duplicates. Fun stuff.

Common Crawl actually offers a cleaner version of these files—pre-extracted text, no HTML. Cheaper to process, less of a headache. The FineWeb team tried it. Their small test models trained on the pre-processed files performed worse than models trained on the raw HTML that they cleaned themselves using an open-source tool called Trafilatura. As Alejandro AO notes in the walkthrough, that custom extraction step is "probably the most expensive part of the entire process"—and for resource-constrained teams, it's the step most likely to get skipped. But the benchmark gap was real enough that Hugging Face kept the expensive path.

That's already a kind of finding: the quality of your extraction method matters before you've done a single thing that feels like "AI."

After extraction and basic filtering (remove adult content via URL blocklist, remove non-English text, filter obvious repetition), they were sitting on about 36 trillion tokens. Which sounds like a lot until you realize the next step nearly broke everything.

The deduplication trap (this one actually got me)

The web is full of mirrors and copies and syndicated articles that got scraped and re-scraped. Deduplication—removing duplicate text—is supposed to help models generalize better and stop wasting training compute on content they've already seen. More deduplication should mean better data, right?

Wrong. Or at least: not automatically.

The FineWeb team's first approach was aggressive global deduplication: take the most recent snapshot, deduplicate it against itself, then run the previous snapshot against that, then the one before against all of those, all the way back to 2013. The oldest snapshots ended up being deduplicated against a decade's worth of accumulated web text. About 90% of their oldest data got wiped. They went from 36 trillion tokens down to 4 trillion.

Then they ran the models. No improvement. The benchmark scores barely moved.

When they dug into what had been removed from those older snapshots, they found that the actually valuable content was the stuff that got cut—and what remained was navigation templates, ad boilerplate, and site chrome. The aggressive deduplication was functioning as a garbage keeper, not a garbage remover.

The fix was almost embarrassingly simple: deduplicate each snapshot only against itself, then combine all the snapshots. Independent MinHash per dump, not global. That approach got them back to 15 trillion tokens and finally hit benchmark parity with RefinedWeb, the dataset used to train Falcon, which had been the target to beat. "We finally got to the point where we have a dataset that is comparable to the refined web dataset," AO says in the walkthrough. "So pretty good."

Pretty good! After burning through a failed strategy that removed 32 trillion tokens worth of data while achieving nothing. I find the honesty here kind of refreshing, actually.

Custom filters beat off-the-shelf filters (with receipts)

After deduplication, Hugging Face compared FineWeb against C4—a dataset famous for achieving strong results through just four blunt filtering rules. One of those rules: remove any document where sentences don't end in punctuation marks. Simple. Brutal.

Applied to FineWeb, that single rule would have eliminated 30% of their tokens. The team decided that was too aggressive and replaced it with softer custom thresholds: remove documents where more than 12% of lines lack terminal punctuation, remove documents with more than 10% duplicated characters across lines, remove documents where more than 67% of lines are under 30 characters. Result: 22% of tokens removed instead of 30%, and benchmark scores improved over the stricter C4 approach.

The lesson here isn't "rules bad, vibes good." It's that blanket filters need to be tested against the actual data they're applied to. What works for one corpus might obliterate value in another.

FineWeb-Edu: when 1.3 trillion tokens beats 15 trillion

This is the part where the story gets actually wild. Llama 3 and Phi-3 were both reported at the time of FineWeb's development to use educational quality filtering in their training pipelines—the idea being that text with high educational value produces better-reasoning models. Their classifiers weren't public. So Hugging Face built their own.

The process: use Llama 3 70B Instruct (a massive model) to score 500,000 FineWeb documents from 0 to 5 on educational value, with detailed rubrics to reduce the chance of the model just hallucinating scores. Then use those 1.5 million labeled data points to train a smaller classifier. Release the smaller classifier onto the full 15 trillion token dataset. Keep everything that scores 3 or above.

Output: 1.3 trillion tokens. FineWeb-Edu.

On educational benchmarks including MMLU, FineWeb-Edu outperformed the full 15-trillion-token FineWeb dataset. The FineWeb blog post and paper document this comparison, though the specific benchmark deltas are worth checking in the full paper if you want exact numbers rather than the directional finding. The magnitude of the gap was enough that the team describes it as FineWeb-Edu "destroying" the other datasets in aggregate score. That's a 91% smaller dataset winning. Less data, better data, better model.

The LLM-bootstrapping-a-smaller-classifier move is worth sitting with. You're using AI to label data, training a cheaper AI on those labels, and running that cheaper AI at scale. It's turtles all the way down in a way that is either elegant or should concern us, depending on your mood.

Okay but the synthetic data thing 👀

Here's where I need you to actually pay attention because this finding is being talked about too calmly.

When the FineWeb team trained small models on each individual Common Crawl snapshot—one model per snapshot, going back to 2013—they found that the more recent snapshots produced noticeably better-performing models. Recent dumps aren't just bigger. They're better. The quality curve bends upward starting around 2022, based on the comparative model benchmarks documented in the FineWeb blog post.

You know what happened in late 2022.

To test the ChatGPT hypothesis, the team tracked the frequency of phrases that LLMs use at statistically high rates—words like delve and certainly, the kind of language that reads as eerily competent and weirdly hollow at the same time. Before 2023, those phrase frequencies in Common Crawl were basically flat. After 2023: a sharp upward spike. As AO puts it in the walkthrough: "which is of course not a proof that this is LLM-generated data, but it does give us a pretty good hint."

So the working hypothesis—and it is a hypothesis, not a confirmed causal chain—is that AI-generated text flooding the web is improving the quality of training data for future AI models. Right now, synthetic text is still a small fraction of the overall web corpus, and the correlation between more synthetic-seeming snapshots and better model benchmarks holds. What happens when synthetic text is the majority of the web? Does the quality keep improving? Does it plateau? Does the whole thing collapse into a feedback loop of AI learning from AI learning from AI?

The FineWeb team doesn't know. Nobody does yet. What makes this particularly disorienting is that we're not in a position to run the experiment ahead of time—we're already living through it. The web is being written, partially, by the models that the web trained. And those models are getting better at writing things that train better models.

I keep coming back to the phrase detector—delve, certainly, the AI tells—as this accidental fossil record of when the internet changed texture. Someone thought to count those words and graph them over time. The graph looks like a hockey stick. And right now, we're somewhere on the blade, watching the curve and genuinely unsure whether up is good.

Zara Chen covers tech and politics for Buzzrag.