Training Models to Reason

A reasoning model does not need a worked solution for every training problem. If a program can check the result, the model can generate its own attempts and learn from the attempts that pass. This is reinforcement learning with verifiable rewards (RLVR), reinforcement learning with verifiable rewards: sample responses, score them with an independently computed rule, and update the model toward responses that earn more reward.

That description is simple, but three details determine whether it works. The checker must represent the task users care about. The base model must already solve enough training problems to produce a useful contrast between passing and failing samples. The policy update must preserve exploration while it concentrates probability on successful behavior. This chapter develops those three requirements in that order.

The reward contract

For one training item, write $x$ for the prompt, $y$ for a sampled response, and $V(x,y)$ for the verifier. The verifier returns a scalar reward $r$. In the simplest case, $r=1$ means that the final answer passed and $r=0$ means that it did not. A training example is therefore not a reference response. It is a prompt paired with an executable acceptance rule.

This distinction changes the data pipeline. Supervised fine-tuning needs a target response to imitate. RLVR needs prompts, a sampler, and a reward contract that states exactly what evidence earns credit. Unit tests can provide that contract for code. A numeric or symbolic equivalence check can provide it for some mathematics. A proof kernel can provide it for a formal proof. The training loop is the same in each case.

flowchart TD
    A[Prompt and reward contract] --> B[Sample G responses]
    B --> C[Run the verifier]
    C --> D[Compute relative advantages]
    D --> E[Update the policy]
    E --> B
    C --> F[Held-out evaluation]

"Verifiable" does not mean "identical to user value." It means that the score can be recomputed from specified evidence without asking the policy being trained to grade itself. Three gaps can still remain:

• A specification gap appears when the rule omits something users care about. Public unit tests may accept a brittle program; final-answer grading may accept a correct answer reached through an invalid argument.
• An implementation gap appears when the checker parses an answer incorrectly, executes untrusted code unsafely, or handles edge cases badly.
• A coverage gap appears when training prompts or tests miss regions that matter after deployment.

A mechanical checker removes the learned-reward-model error studied in RLHF, but only for the criterion it actually implements (Gao et al. 2023). The policy can still optimize a faulty or incomplete contract. Chapter 27 develops the verifier itself; here the verifier is fixed so that we can study the learning loop around it.

How group-relative learning gets a signal

The policy is the model distribution being updated. For a prompt $x$, sample a group of $G$ responses from the policy before the update, $y_i \sim \pi_{\mathrm{old}}(\cdot\mid x)$, and score each response with $r_i=V(x,y_i)$. group relative policy optimization (GRPO), Group Relative Policy Optimization, assigns each response an advantage by comparing its reward with the rewards of the other responses for that prompt (Shao et al. 2024):

$$\bar r = \frac{1}{G}\sum_{j=1}^{G} r_j, \qquad s_r = \sqrt{\frac{1}{G}\sum_{j=1}^{G}(r_j-\bar r)^2}, \qquad A_i = \frac{r_i-\bar r}{s_r+\varepsilon}.$$

Here $\bar r$ is the group mean reward, $s_r$ is its standard deviation, and the small constant $\varepsilon$ prevents division by zero. A positive $A_i$ tells the optimizer to make response $y_i$ more likely; a negative value tells it to make that response less likely. All tokens in an outcome-scored response share this one advantage.

GRPO then compares the updated policy with the policy that generated the response. At token position $t$, the importance ratio is

$$\rho_{i,t}(\theta)= \frac{\pi_\theta(y_{i,t}\mid x,y_{i,<t})} {\pi_{\mathrm{old}}(y_{i,t}\mid x,y_{i,<t})}.$$

The numerator is the new probability of that sampled token and the denominator is its probability during rollout. The original objective averages a clipped term over the tokens in each response:

$$J_{\mathrm{GRPO}}(\theta)= \mathbb{E}\!\left[ \frac{1}{G}\sum_{i=1}^{G}\frac{1}{|y_i|}\sum_{t=1}^{|y_i|} \min\!\left( \rho_{i,t}A_i, \operatorname{clip}(\rho_{i,t},1-\epsilon,1+\epsilon)A_i \right) -\beta D_{\mathrm{KL}}(\pi_\theta\Vert\pi_{\mathrm{ref}}) \right].$$

Here $|y_i|$ is the response length, $\epsilon$ is the clipping range, $\pi_{\mathrm{ref}}$ is a fixed reference policy, $D_{\mathrm{KL}}$ measures divergence from it, and $\beta$ controls that penalty. The clipping term limits one update relative to $\pi_{\mathrm{old}}$; the KL term limits accumulated drift from $\pi_{\mathrm{ref}}$. Implementations differ in how they estimate and aggregate the KL term, so the equation shows the structure rather than one library's exact tensor reduction.

The group provides no relative signal when every response receives the same reward. With a binary verifier, suppose one sample passes with probability $p$. Under independent sampling, the probability that a group contains both a pass and a failure is

$$P(\text{mixed group}) = 1-p^G-(1-p)^G.$$

In words, this formula explains an important data constraint. Problems that are almost always solved or almost never solved waste most group-relative rollouts. Larger groups make mixed outcomes more likely, but generation cost grows with $G$. The useful training set sits near the current policy's capability boundary and moves as the policy improves.

The following small experiment makes both effects visible. It computes GRPO and leave-one-out advantages for one mixed group, then reports how often a binary-reward group should contain a learning signal.

import numpy as np

def grpo_advantages(rewards):
    rewards = np.asarray(rewards, dtype=float)
    spread = rewards.std()
    return np.zeros_like(rewards) if spread == 0 else (rewards - rewards.mean()) / spread

def rloo_advantages(rewards):
    rewards = np.asarray(rewards, dtype=float)
    group_size = len(rewards)
    return rewards - (rewards.sum() - rewards) / (group_size - 1)

def mixed_group_probability(pass_rate, group_size):
    return 1 - pass_rate ** group_size - (1 - pass_rate) ** group_size

rewards = [1, 0, 1, 0]
print("GRPO:", np.round(grpo_advantages(rewards), 2).tolist())
print("RLOO:", np.round(rloo_advantages(rewards), 2).tolist())
print("all equal:", grpo_advantages([1, 1, 1, 1]).tolist())
for pass_rate in (0.01, 0.10, 0.50):
    probability = mixed_group_probability(pass_rate, group_size=8)
    print(f"p={pass_rate:.2f}, mixed group={probability:.3f}")

GRPO is one way to avoid a learned value model, usually called a critic. Common proximal policy optimization (PPO) implementations use a critic to estimate expected future reward and subtract that estimate as a variance-reducing baseline (Schulman et al. 2017). GRPO uses the prompt's sampled group instead. REINFORCE leave-one-out (RLOO), REINFORCE Leave-One-Out, treats each whole response as one sampled action and subtracts the mean reward of the other $G-1$ responses (Ahmadian et al. 2024). It is a REINFORCE estimator, not PPO with a different group normalization, and does not use PPO's clipped token ratios. These methods are not interchangeable names for RLVR. RLVR describes where the reward comes from; PPO, GRPO, and RLOO describe how the policy update is estimated and constrained.

The critic-free methods save the memory and synchronization required by a separate value model, but the group estimate can have high variance. It is also relative: a mediocre response can receive positive advantage if its peers are worse. The verifier reward remains absolute; the advantage used for the update does not.

What the first reasoning models established

OpenAI reported that o1 improved as both reinforcement-learning compute and test-time thinking increased, but did not publish enough of the training recipe to isolate the cause (OpenAI 2024). DeepSeek-R1 supplied the clearer experiment because it described two related models and released their weights (Guo et al. 2025).

DeepSeek-R1-Zero started from DeepSeek-V3 Base and applied GRPO without a supervised fine-tuning stage first. Its rule-based reward combined answer accuracy with an output-format check. During training, benchmark accuracy and response length increased, and sampled traces showed more reflection and backtracking. This demonstrated that final-answer and format rewards, without step-level labels, can strongly change the frequency and shape of long-form reasoning behavior. It did not demonstrate that every displayed reasoning step was correct or causally necessary.

The released DeepSeek-R1 used a larger pipeline:

1. Thousands of cold-start examples established a readable, conversational, human-aligned reasoning pattern before RL.
2. A first RL stage improved reasoning and language consistency.
3. Rejection sampling produced new reasoning data, using rule checks where possible and model judgments for some non-verifiable items. The filtered data was mixed with general data for another supervised fine-tuning stage.
4. A second RL stage combined rule-based reasoning rewards with learned preference rewards for helpfulness and harmlessness.

The cold start was therefore more than cosmetic, and the final model was not the result of pure RLVR. It combined supervised data, verified outcomes, rejection sampling, and preference alignment. R1-Zero is the cleaner evidence that RL can work without a preceding SFT stage or step-level labels; R1 is the stronger product recipe.

DeepSeek also reported that process reward models and Monte Carlo tree search did not justify their complexity in this pipeline (Guo et al. 2025). That is a result about one training system, not a general proof that process feedback or search is useless. Process feedback changes credit assignment and requires step labels or a learned process reward model (Uesato et al. 2022; Lightman et al. 2024). Chapter 27 covers that trade-off in detail. The narrower R1 result is that final-answer and format rewards produced large benchmark and behavioral changes without step-level labels.

What remains contested

Optimization details that change the result

The simple advantage equation hides consequential choices about normalization, clipping, aggregation, and reference control. These choices affect which responses receive the largest update, not merely how fast the same objective is optimized.

The original GRPO loss averaged token losses within each response and divided advantages by the group's reward standard deviation. Dr. GRPO argued that the first choice creates a length bias and the second changes the relative weight of prompts with different reward spreads. Its proposed objective removes the reward-standard-deviation term and replaces response-dependent length averaging with a fixed normalization shared across the batch. In its experiments, this stopped incorrect responses from growing longer while preserving benchmark performance (Liu et al. 2025). The broader lesson is to inspect the effective weight per token, response, and prompt. "Longer reasoning" can be an optimization artifact rather than a capability gain.

DAPO addressed a different set of failures in long-response training (Yu et al. 2025). It raised the upper clipping range separately so low-probability tokens had more room to grow, resampled until batches contained prompts with mixed outcomes, aggregated the policy-gradient loss across tokens rather than giving every response equal weight, and softened penalties near the maximum response length. Together these changes improved sample use and training stability in its reported Qwen2.5-32B run. GSPO later replaced token ratios with a length-normalized sequence ratio, the geometric mean of a response's token ratios, then clipped and optimized the response as one unit. It reported more stable mixture-of-experts training (Zheng et al. 2025). These are empirical recipes with different objectives and systems costs, not a settled progression toward one universal optimizer.

Reference control needs the same precision. DeepSeekMath's GRPO includes a KL penalty against a reference policy. DAPO omits that penalty in its reported setup, controls each update with clipping, and monitors entropy separately. Those mechanisms do different jobs. A verifiable reward makes a learned preference proxy unnecessary for the checkable criterion, but it does not make unconstrained optimization safe. The policy can still overfit training prompts, exploit checker gaps, lose useful general behavior, or collapse to a narrow set of outputs. Whether to use a reference penalty is a measured design choice, not a property implied by the acronym RLVR.

Building a defensible training run

A reliable run begins with an evaluation contract, not an optimizer. Record the prompt source, verifier version, parser, execution environment, model checkpoint, chat template, tokenizer, sampling parameters, and maximum response length. Otherwise a change in formatting or checking can look like a learning gain.

Then separate three datasets. The training prompt pool drives rollouts. A held-out task set chooses checkpoints and stopping time. An adversarial verifier set targets malformed answers, test leakage, hard-coded outputs, parser exploits, and correct but unusual solutions. Hidden tests matter for code because a model can overfit visible examples without learning the requested behavior.

Track the complete distribution, not just mean training reward:

• the fraction of all-pass, all-fail, and mixed groups;
• held-out accuracy, pass@$k$, and response length by difficulty and domain;
• policy entropy, KL from the starting policy, clipping fraction, and token likelihood ratios;
• verifier false accepts and false rejects on audited samples;
• rollout tokens, verifier latency, discarded samples, and useful updates per accelerator-hour;
• regressions in instruction following, safety, writing, and non-reasoning tasks.

Stop and investigate when training reward rises without held-out improvement, when response length grows without accuracy, when mixed groups disappear, or when entropy collapses. DAPO reports that training reward can correlate poorly with validation accuracy, which makes training reward unsuitable as the sole stopping signal (Yu et al. 2025).

Finally, compare against cheaper uses of the same compute. Supervised fine-tuning on verified traces, rejection-sampling fine-tuning, and best-of-$N$ at inference may deliver the desired pass rate without online RL. Tülu 3 is one example of a broader post-training pipeline that combines supervised training, preference optimization, and RLVR rather than treating them as substitutes (Lambert et al. 2024). The comparison should match total generated tokens, verification cost, and inference budget.

Payoff and boundary

RLVR turns a checker into a source of fresh training data. Its capability payoff is strongest on tasks with many hard prompts and reliable acceptance rules. Its efficiency payoff comes from learning without a hand-written target response for every prompt; critic-free variants can also avoid a separate value model. Its trust payoff is conditional: an independently executable rule is auditable, but only within the specification, implementation, and coverage boundaries of that rule.

Open-ended tasks do not suddenly become verifiable because a detailed rubric exists. Rubric rewards can extend online RL beyond exact-answer domains, but they replace a mechanical acceptance rule with a structured judgment and reintroduce judge error and proxy optimization (Gunjal et al. 2025). A practical post-training system therefore mixes signals: exact checks where they are sound, process or preference feedback where they are not, and held-out human evaluation for the behavior that neither captures.

Once a verifier is trustworthy, the same loop can bootstrap data: sample solutions, keep verified successes, train again, and repeat. Chapter 29 continues with the data produced by that loop. Chapter 30 spends extra samples after training instead, and Chapter 37 extends the reward from one response to a trajectory of actions.

Further reading