Joint Scaling Laws for Pre-training and RL

The AI field has spent years developing intuitions about how to spend compute during pre-training. Reinforcement learning, the technique behind the reasoning-model boom, has been governed mostly by vibes and expensive iteration. A new paper discussed at Hugging Face's recent journal club session wants to change that—and uses chess, of all things, to make the case.

The paper, "Understanding Reasoning from Pretraining to Post-Training", attempts something genuinely useful: a joint scaling law that covers the full training pipeline, from initial pre-training through supervised fine-tuning (SFT) to reinforcement learning. If it holds, it would give teams a principled basis for deciding how to split their compute budget across all three stages—rather than treating each one as a separate guessing game.

The chess framing isn't arbitrary. It gives the researchers a controlled environment where ground-truth evaluation is possible—you can run Stockfish alongside the model and get an objective read on move quality at every single step. That move-level reward signal is notably different from conventional RL setups, where you typically score only the final outcome. The journal club discussion flagged this as a meaningful design choice: it functions more like process-level supervision, rewarding each decision in a trajectory rather than just whether the game was won.

The three-stage pipeline, briefly

The setup works like this. Models are pre-trained on a dataset of human chess games, learning to predict moves autoregressively. Then comes SFT, where the pre-trained model samples multiple possible trajectories from a given board position, those trajectories are organized into a tree structure based on shared move prefixes, linearized into chain-of-thought data, and used for fine-tuning. The RL stage follows, with Stockfish providing reward signals at each move.

The Hugging Face team noted that the SFT step here doesn't quite replicate how chain-of-thought training works in systems like DeepSeek R1—the mechanics are specific to the chess environment. That's worth keeping in mind when thinking about how far these results travel.

Smaller models, longer training, better outcomes

The headline finding that drew the most animated discussion: when total compute is fixed, smaller models trained for longer tend to outperform larger models trained for shorter periods. The conventional wisdom, loosely anchored to the Chinchilla scaling laws (Hoffmann et al., 2022), which describe an optimal relationship between model size and training tokens, gets complicated here. The paper's results suggest that for downstream RL performance, the Chinchilla-optimal point may not be the right target. You might be better off staying smaller and running longer.

The journal club spent real time unpacking this. One caveat that emerged: the finding might be specific to chess. The game has a compact, constrained action space, and what holds there may not translate cleanly to open-ended language tasks where the token space is orders of magnitude larger. That caveat isn't a dismissal—it's just honest accounting about what a controlled experiment can and can't tell you.

RL does something, just not what the hype says

The more consequential finding for the current moment in AI is what the paper says about what RL actually does.

The researchers tracked two metrics: pass@1, which measures whether a model gets the right answer on its first attempt (a proxy for reliability), and pass@K, which measures whether the model gets it right in at least one of K attempts (a proxy for raw capability). RL consistently improves pass@1—the model becomes more reliably correct. But pass@K? That barely moves, regardless of how much RL compute you throw at it.

The implication is significant. RL is making models more consistent at things they already know how to do. It is not reliably unlocking capabilities that weren't present in the pre-trained model. The journal club framed it carefully: if a model has essentially zero probability of solving something even when given many attempts, RL is unlikely to conjure that capability from nothing. But if the capability exists at low probability—if the model can find the right answer sometimes—RL can amplify that into something reliable.

The paper's own conclusion here is, to the researchers' credit, explicitly mixed. For hard problems, RL occasionally surfaces strong solutions that the base model could barely produce. For easy ones, it mostly locks in what the model already does well. And sometimes, when the model doesn't really know the answer, RL just reinforces whatever it does know—confidently, wrongly. The discussion acknowledged that this isn't a clean result, and that the dynamics might differ significantly in language settings where the action space is far broader and "tail discovery" harder to achieve.

The gap between a clean paper and a messy practice

What the Hugging Face researchers kept returning to is the gap between what this paper proves and what it can actually tell practitioners. Most people working with RL today aren't training models from scratch. They're starting from an open-weight model—a Llama, a Qwen, something else—whose pre-training recipe they don't know and can't recreate. The paper's curves are beautiful precisely because the researchers controlled the pre-training. Strip that control away and you're working with an unknown foundation.

The math-domain extension in the paper is more practically relevant on this front: the researchers explored whether scaling relationships hold when doing continued pre-training on an already-trained model, rather than training from scratch. The journal club flagged this as potentially the most transferable result—that mid-training loss and subsequent RL performance follow a traceable relationship even without full control over the original pre-training. But the details were still being worked through in the session, and it would be a stretch to call that case closed.

There's also the question of cost. Deriving scaling law coefficients requires running many experiments at varying compute budgets to fit the curves. That's not free. The question of whether you can do that fitting at small scale and then extrapolate reliably to larger models is exactly the kind of practical constraint that distinguishes "interesting paper" from "useful paper."

Why this matters right now

RL has been receiving a remarkable amount of credit lately. Models improve, RL gets the spotlight, and the narrative tends to imply that you can train your way to new capabilities if you just do enough of it. This paper is a useful corrective to that story—not a refutation of RL's value, but a structural argument that RL's gains are bounded by what the pre-training put in. The amplifier metaphor from the journal club is apt: you can turn up the volume, but you can't amplify a signal that isn't there.

The Chinchilla paper gave the field a framework for thinking rigorously about pre-training compute allocation. This paper is attempting something analogous for the full pipeline. Whether it succeeds in that ambition—whether the chess-derived relationships generalize to language, math, and the messy real world of AI deployment—is still an open empirical question. But the attempt itself is the right one. The RL era has been running on heuristics and luck. A principled map of the terrain would be worth a lot.

The question worth sitting with: if RL mainly amplifies what pre-training already installed, then the teams with the best pre-training data and compute have a structural advantage that post-training cleverness can't fully close. The hype around RL has sometimes implied otherwise. This paper, quietly, argues it can't.

Marcus Chen-Ramirez covers AI, software development, and the intersection of technology and society for Buzzrag.