Training Models to Reason

Reinforcement learning changes character when the reward is a checkable ground truth rather than a learned human-preference proxy. Where a verifier exists, it beats a reward model because it removes the learned proxy gap. group relative policy optimization (GRPO) and REINFORCE leave-one-out (RLOO) are policy-gradient variants that compute a baseline (the reference value an outcome is compared against to decide if it was better than typical) from sampled groups, and they simplify proximal policy optimization (PPO) by dropping its critic (a learned predictor of expected reward used as that baseline). The o1 and R1 line showed that long-horizon reasoning can emerge without step-level supervision, and the field still disagrees about whether that training teaches new reasoning or mainly reweights what the base model could already sample.

The thread that runs through every section is a single object, the verifier. Where it exists, this recipe applies. Lose it, and you are back to the learned reward model of Chapter 19. Once that object is fixed, each design choice has a visible cost.

A Reward Without a Learned Proxy Gap

Start with the failure that motivates everything here. A learned reward model is a proxy, and a policy trained against a proxy will find its argmax, not the true objective's. This is the over-optimization problem that haunts the RLHF of Chapter 19 (Gao et al. 2022): push hard enough on a model of human preference and the policy learns to satisfy the model rather than the human. The reward model has no ground truth to anchor it, so its blind spots become the policy's exploits.

For a large and valuable class of tasks there is a way out, because the answer can be checked. A program either passes its unit tests or it does not. A math answer equals the reference or it is wrong, with nothing in between. And a formal proof type checks in Lean or it fails to. On these tasks you can replace the learned reward with a verifier and optimize the policy against a reward that has no proxy gap to exploit. That move, reinforcement learning with verifiable rewards (RLVR), is the training-time analogue of the runtime verification loop that an agent harness runs at inference.

Swapping the proxy for a checker

The loop end to end is short. Sample completions for a prompt, run each through a verifier for a scalar reward that is often just binary, and update the policy toward the rewarded completions. The training data is prompts with a way to check the answer, not prompts with reference completions. The optimizer is the PPO machinery of Chapter 19 reused without change (Schulman et al. 2017): a policy, an advantage estimate, and a KL penalty against a frozen reference model. The innovation is the reward source, not the algorithm.

Everything downstream of this loop is determined by how good the checker in the middle box is. The next section opens it up.

Three kinds of verifier

The verifier is the part you cannot buy, and it comes in three families with different cost and coverage. Executable unit tests for code are cheap and high-signal, but coverage limited, and a solution that passes the tests while being wrong will be rewarded all the same. Answer checkers for math reduce to string, numeric, or symbolic equivalence, where the hard engineering is parsing and deciding equivalence rather than the check itself. Formal proof checkers in the Lean or Isabelle style are near unspoofable, but narrow and expensive to author. Each family trades coverage against integrity, and that trade is the recurring theme here. Figure 28.2 places the three families on that axis.

Dropping the critic

With the reward source fixed, the optimizer earns one real simplification. PPO carries a value network, a critic, to estimate the baseline that lowers advantage variance, and that network doubles model memory and adds its own failure mode. Group Relative Policy Optimization, introduced for DeepSeekMath (Shao et al. 2024), drops the critic and estimates the advantage from the group of samples drawn for one prompt: normalize each reward against the group mean and standard deviation. For a group of $G$ completions with rewards $r_i$,

$$A_i = \frac{r_i - \operatorname{mean}(r_1, \dots, r_G)}{\operatorname{std}(r_1, \dots, r_G)}.$$

Compute both baselines on the same reward vector and watch where they differ, and what a no-spread group does to each.

import numpy as np

def grpo_adv(r):
    s = r.std()
    if s == 0:            # all pass or all fail: no spread, no signal
        return np.zeros_like(r)
    return (r - r.mean()) / s

def rloo_adv(r):          # baseline = mean of the other G-1 samples
    G = len(r)
    return r - (r.sum() - r) / (G - 1)

for name, r in [("mixed   ", np.array([1., 0., 1., 0.])),
                ("all pass", np.array([1., 1., 1., 1.]))]:
    print(f"{name} rewards={r.tolist()}")
    print(f"   GRPO advantage = {np.round(grpo_adv(r), 3).tolist()}")
    print(f"   RLOO advantage = {np.round(rloo_adv(r), 3).tolist()}")

This is cheaper than PPO with a critic and fits RLVR exactly, because sampling many completions per prompt is natural when each one is cheap to verify. The KL term against the reference is kept. RLOO, REINFORCE Leave-One-Out, makes the same bet with a different baseline (Ahmadian et al. 2024): each sample's baseline is the mean reward of the other samples in the group. GRPO and RLOO differ in their bias and variance trade-offs, but they agree on the structural claim that the critic was often unnecessary overhead. Figure 28.4 shows where the three methods get the baseline that turns a reward into an advantage.

What the loop produced: o1 and R1

The result that changed the field was that this loop, run on math and code with long chains of thought allowed, produces emergent long-horizon reasoning without any step-level supervision. Self-checking, backtracking, and longer deliberation appear from an outcome-only reward. OpenAI's o1 announced the capability (OpenAI 2024). DeepSeek-R1 is the open data point that showed the recipe (Guo et al. 2025).

R1 is clearest as two variants because the contrast carries the claim (Guo et al. 2025). R1-Zero applies RL directly to a base model with no supervised cold start, and reasoning behavior still emerges, which is the strong claim. R1 adds a short SFT cold start before the RL, drawing on the supervised fine-tuning of Chapter 17, not to teach reasoning but to fix readability and stability: the zero variant reasons in a way that is correct but hard to read and prone to language mixing. The cold start is a cosmetic and stabilizing prefix, and the reasoning gains come from the RL. Figure 28.5 contrasts the two pipelines.

The shipped R1 is more than a cold start bolted onto RL. Read in full it is a four-stage pipeline: a small cold-start SFT set to fix readability, a first reasoning-oriented RL stage, a rejection-sampling pass that keeps only verified-correct traces and folds in general SFT data, then a final RL stage run across both reasoning and general preferences (Guo et al. 2025). The reward in the zero variant is rule-based and composite, an accuracy reward that checks the answer plus a format reward that requires the chain of thought to sit between <think> and </think> tags; R1 adds a language-consistency reward, the fraction of the chain written in the target language, to stop the zero variant's language mixing at a small measured cost to raw accuracy (Guo et al. 2025).

Two of R1's reported dead ends teach as much as its successes (Guo et al. 2025). A process reward model (PRM) was tried and dropped: defining a fine-grained correct step in open reasoning is hard, judging each step is itself error-prone, and the learned PRM invited the reward hacking of Chapter 19 for a gain that did not justify the overhead. Monte Carlo tree search (MCTS) over the token stream was tried and dropped too, because token generation opens an exponentially larger search space than a board game and the step-value model it needs is hard to train. The signal that won was the plain outcome reward, not the denser, cleverer one. The other lesson is about who gets the reward: distilling R1's traces into a smaller dense model beat running the same RL on that small model directly, DeepSeek-R1-Distill-Qwen-32B reported above DeepSeek-R1-Zero-Qwen-32B, so the cheapest way to put reasoning into a small model is to let a large one find it and copy the result, a thread that Chapter 23 picks up.

How much credit, and how often

The supervision question has its own lineage, and it is a second axis cutting across the verifier families. Outcome rewards score only the final answer, which is cheap and is the RLVR default, but assigns credit coarsely over a long chain. Process rewards score each step through a process reward model, a PRM, which Lightman et al. trained for "Let's Verify Step by Step" (Lightman et al. 2023) and Uesato et al. compared against outcome feedback earlier (Uesato et al. 2022). A PRM gives denser credit and helps long-horizon learning, but it reintroduces exactly what the verifier removed: a learned, hackable reward. The surprise of the R1 era is how far an outcome-only signal got without any process model at all.

Refining the group baseline

GRPO shipped the reasoning models, and the year after R1 was spent finding where its group baseline lies and correcting it. The sharpest reading came from Liu et al. (2025), which names two biases in the update. A length bias, because the per-token $1/|o_i|$ normalization gives a correct short answer a larger gradient than a correct long one and penalizes a wrong long answer less than a wrong short one, so the policy drifts toward longer wrong outputs. And a difficulty bias, because dividing the advantage by the group's standard deviation up-weights questions that are almost all-pass or all-fail, where the spread is smallest. Dropping the $1/|o_i|$ and standard-deviation terms removes both and recovers the unbiased PPO objective, reaching the same reasoning at fewer tokens, which is why the authors call it GRPO done right.

Two further fixes target a different failure, the policy collapsing onto a few high-reward modes as its entropy craters (its outputs lose diversity and converge to a few repeated patterns). DAPO (Yu et al. 2025) decouples the clip range so the upper bound can rise on its own, its clip-higher rule, which leaves room for low-probability exploratory tokens to grow rather than be clipped away, and it adds dynamic sampling, discarding prompts whose group is all-correct or all-wrong because a group with no spread produces no advantage and no gradient, the dead case the runnable above made visible. GSPO (Zheng et al. 2025), from the Qwen team, moves the importance ratio from the token to the sequence: it scores, clips, and optimizes on whole-sequence likelihood rather than per-token ratios, which it argues are ill-matched to a reward defined over the whole sequence, and the change stabilizes the otherwise brittle RL of mixture-of-experts models (the sparse, routed architecture of Chapter 9). The KL term also loses the necessity it had in Chapter 19: with a verifier there is no learned proxy to drift into, and DAPO drops the KL penalty that RLHF kept as a constraint, relying on clip-higher to keep entropy from collapsing instead.

Kimi k1.5 is the other important public data point because it lands on a similar practical lesson from a different route. Its report emphasizes long-context RL and improved policy optimization, and explicitly avoids MCTS, value functions, and process reward models while reaching strong math, code, and multimodal reasoning results (Kimi Team et al. 2025). It also introduces long2short methods, using long-CoT training to improve short-CoT models. That belongs to Chapter 29, but it matters here because it shows the training loop and the data loop are no longer separable: a long reasoning policy can become a teacher for a cheaper short reasoning policy.

Failure Modes and Payoff

The recipe is a set of balances, each with a knee. Read these as the dials you turn once the verifier is fixed.

• Verifiable reward versus learned reward. A verifier is hard to exploit in the proxy sense but only exists where ground truth is checkable, and its coverage is the new attack surface. The learned reward model of Chapter 19 works on open-ended tasks but is an over-optimizable proxy. Real recipes mix both by domain (Lambert et al. 2024).
• Outcome versus process supervision. Outcome rewards are cheap, robust, and sparse. Process rewards through a PRM are dense and better for long-horizon credit, but the PRM is a learned, exploitable reward and expensive to label (Uesato et al. 2022). This is signal density against reward integrity.
• Critic versus critic-free. A value network lowers advantage variance but doubles model memory and adds a failure mode of its own. GRPO and RLOO trade that variance for simplicity and fit the many-samples shape of RLVR (Shao et al. 2024).
• Exploration versus KL collapse. Too little KL, or too aggressive optimization, collapses the policy onto a few high-reward modes and stops exploration, so reward rises while held-out quality falls. Too much KL and the policy never moves off the reference. The KL coefficient and clip range are the central RLVR knob. Figure 28.9 sketches the knee where training reward keeps rising while held-out quality turns down.

• Verifier coverage versus exploitability. A stricter, broader verifier resists exploitation but costs more and may reject correct-but-unusual answers, false negatives that starve learning. A loose verifier is cheap but leaks reward to wrong answers.

Most failures here are the optimizer finding what you measured rather than what you meant. The central one is reward hacking through verifier gaps: the policy discovers inputs the verifier scores correct that are not, tests passing on degenerate solutions, checkers fooled by formatting, hard-coded outputs for known cases. The reward to the optimizer is real and the answer to you is wrong. Next comes KL or mode collapse, where the policy concentrates on a few high-reward outputs, entropy craters, and held-out quality degrades while training reward climbs. Subtlest is reasoning that exploits the checker rather than the problem: the visible chain of thought steers toward whatever the verifier rewards, or does not faithfully drive the answer at all.

Two boundaries keep the method in check. RLVR is strongest where ground truth is checkable and does not transfer for free to open-ended tasks, so over-indexing on it skews the model and leaves a gap that only a learned reward or human judgment fills. The 2025 attempt to stretch the recipe past that boundary replaces the checker with an explicit rubric, scoring an open-ended answer against a written list of criteria rather than one ground truth, which recovers a denser-than-preference signal where no verifier exists (Gunjal et al. 2025). It buys reach at the cost of the property that made the verifier safe, since a rubric is itself a written proxy that can be exploited. Over-squeezing the reward also regresses the instruction-following, safety, and breadth installed in post-training, a catastrophic forgetting that the KL anchor here only partly prevents and that Chapter 47 must catch.

The boundary also moved from the frontier side. In July 2025, Gemini Deep Think, graded officially by IMO coordinators, and an experimental OpenAI model reached gold-medal standard at the International Mathematical Olympiad, 35 of 42 points, writing end-to-end natural-language proofs with no formal verifier and no per-problem checker at inference (Google DeepMind 2025). The reported ingredients are RL on hard-to-verify reasoning tasks plus parallel test-time compute. Read that as the verifiable-reward boundary being stretched, not erased: the reward can be softer than a mechanical check, but the training still leans on mathematics, where correctness is at least gradeable.

The self-improvement loop is the payoff when a verifier is sound. Generate candidate training data with the model, filter by the verifier to keep only verified-correct traces, and feed those into another SFT or RL round, then iterate. This is distinct from the distillation of Chapter 23 because the filter is a checker, not a stronger teacher, so the model can bootstrap past its current self. The hard limit is that it needs a verifiable signal to close the loop: a sound verifier gives a genuine bootstrap, an unsound one amplifies its own blind spots. Running the sampling and rollout fleet that feeds all of this at scale belongs to Chapter 10, and the inference-time analogue of spending more compute to reason better is Chapter 30.

Extending the same loop from rewarding an answer to rewarding a whole trajectory, training a model to act and not only to reason, is Chapter 37.

Further reading