Verifiable Rewards and Reasoning Transfer

The previous chapters treated reward as a learned proxy for preference. That is the right tool for open-ended behavior: helpfulness, taste, tone, refusal calibration, and many forms of judgment. But some domains contain a different kind of signal. A code solution passes the tests or it does not. A math answer matches the checker or fails it, and a verifier either accepts a proof or rejects it. In those domains, post-training can train against an outcome rather than a person's estimate of quality.

That distinction is the hinge between this part and Part IV. Here the question is what kind of reward source enters the post-training loop. Chapter 28 then follows the algorithms that exploit that reward to train reasoning behavior.

A reward that is not a judge

A learned reward model estimates a preference:

$$r_\phi(x,y) \approx \text{human or AI judgment of } y \text{ for } x.$$

Here, by contrast, a verifiable reward is a predicate:

$$r_{\text{ver}}(x,y) = \mathbf{1}[\operatorname{check}(x,y)=\text{pass}].$$

Here $r_{\text{ver}}$ is one when the task checker passes and zero otherwise. The difference looks small in notation and large in practice. A learned reward can be wrong because it liked the style, missed the factual error, or never saw that region of the response distribution. A verifier can still be incomplete or mis-specified, but where it is sound it does not care whether the answer sounds confident. The model either satisfies the check or it does not.

This is why verifiable domains have become the proving ground for reasoning post-training. The expensive part of reasoning data is not merely the trace. It is knowing whether the trace got to a correct endpoint. Where a checker exists, the model can generate many candidates, the system can keep only those that pass, and training can move probability mass toward behavior the model already occasionally produced.

From selection to training

The first step is selection. Cobbe et al. showed that sampling many reasoning solutions and using a trained verifier to choose among them could substantially improve math word-problem performance (Cobbe et al. 2021). The mechanism is simple: if a model has a nonzero chance of producing a correct solution, then best-of-$n$ (sample $n$ candidates and keep the best) can expose that latent capability, provided the selector can find it. Chapter 24 describes the inference-time version of that idea.

The second step is to train on the selected behavior. Rejection sampling fine-tunes on candidates that pass a scorer or verifier, which Chapter 23 treats as a data flywheel. Reinforcement learning with verifiable rewards, reinforcement learning with verifiable rewards (RLVR), goes further: sample on-policy completions (responses generated by the current model, not a fixed dataset), assign rewards from the checker, and update the policy. DeepSeekMath introduced GRPO, a lighter policy-gradient method for this setting that Chapter 28 takes up in detail (Shao et al. 2024). Tülu 3 coined the name RLVR and demonstrated it as one component of an open post-training recipe alongside SFT and DPO (Lambert et al. 2024), and DeepSeek-R1 then made the recipe visible at frontier scale: a base or cold-started model generates reasoning traces, receives rewards from answer checkers, and acquires longer deliberative behavior through RL (Guo et al. 2025).

The object being optimized can be sparse. A final answer checker gives a single bit at the end of a long trace. If the model never stumbles into a correct answer, no gradient can teach it where to go. If it sometimes gets there, the reward can amplify those paths. This is why cold-start data, curriculum, and prompt distributions matter. They place the policy close enough to successful trajectories that the verifier can begin selecting them.

Outcome rewards and process rewards

Outcome rewards score only the final answer. Process rewards score steps. For a trace $z_1, \ldots, z_T$ ending in answer $y$, an outcome reward is

$$r_{\text{out}}(x,z_{1:T},y) = \mathbf{1}[\operatorname{check}(x,y)=\text{pass}].$$

Here the outcome reward ignores the intermediate path except for its final answer. A process reward is denser:

$$r_{\text{proc}}(x,z_{1:T},y) = \sum_{t=1}^{T} \gamma^{t-1} r_t(x,z_{\le t}).$$

Here $r_t$ scores a prefix step and $\gamma$ discounts later step scores. Lightman et al.'s "Let's Verify Step by Step" showed that process supervision can outperform outcome supervision on math reasoning, and released PRM800K, roughly 800,000 step-level labels (Lightman et al. 2023). The appeal is obvious: the model gets a training signal before the final answer. The cost is just as important. Step labels are harder to define, harder to collect, and more likely to encode a style of reasoning rather than correctness itself. A learned PRM is also a reward model, so it can reintroduce the proxy failures that verifiable outcome rewards tried to avoid. Chapter 27 covers that boundary in more detail.

The practical rule is to separate the source of truth from the density of the signal. An exact final checker is sparse but grounded. A learned process model is dense but fallible. A strong system may use both: a verifier to anchor final correctness and a process model to guide search or credit assignment, with held-out checks to make sure the dense signal has not become the target.

What verifiers buy, and what they do not

Verifiers buy four things.

• Less reward hacking. A unit test or exact answer checker is harder to flatter than a learned judge. It can still be incomplete, but it is not impressed by style.
• Cheaper data. Correct traces can be discovered by sampling and filtering rather than written by humans.
• On-policy improvement. The model trains on its own successful samples, not only on a teacher's distribution.
• A bridge to inference. The same checker can train the policy, select among samples at inference, and evaluate held-out performance.

They do not buy universality. Many important tasks are not checkable. A helpful legal explanation, a nuanced research synthesis, and a product-support answer do not have a single executable truth predicate. Even in code, tests can be weak or leaked; in math, a final answer can be right for the wrong reason. Recent work on CoT-Pass@K, a stricter score that also requires the reasoning path to be sound, makes this point directly: ordinary pass@k (the fraction of problems solved within $k$ attempts) can count a sample as successful when the final answer is right but the reasoning path is not (Wen et al. 2025).

Between the checkable reward and the learned preference reward now sits a third option. Rubric-based rewards decompose an open-ended judgment into per-criterion checks, each graded by a model against a written rubric, and train on the aggregate score (Gunjal et al. 2025). That extends RLVR-style training into domains with no executable checker: medicine, law, research synthesis. It is how frontier labs grade such work, OpenAI's HealthBench among the examples; the price is that the grader is a model again, so each criterion re-imports a measure of the proxy risk the checker had removed.

The result is a mixed post-training stack. Use verifiable rewards where the world supplies a checker. Use learned rewards where it does not. Keep the boundary explicit, because the failure mode changes when the reward changes.