Verifiable Rewards and Reasoning Transfer

A preference model can tell a policy which answer people tend to favor. It cannot tell the policy that a program actually passed its tests or that two algebraic expressions are equal. For those tasks, the training system can run a check and turn the result into a reward. That small change removes one learned judge from the loop, but it does not remove specification errors. The policy learns to satisfy the implemented check, which may be narrower than the task people intended.

The decisive boundary is reward provenance. An executable checker runs an implemented contract; a learned verifier estimates a judgment. Selection and training are separate uses of either signal, just as outcome rewards and process rewards differ in timing. The optimization algorithms belong to Chapter 28. Here the central question comes first: what fact, judgment, or proxy produces the number that training will optimize?

The checker is an implemented specification

Let $x$ be a task, $y$ a candidate response, and $q(x,y)\in\{0,1\}$ the correctness standard people intend. A checker $C$ is the program that the training system can actually run. In the simplest binary case,

$$r_{\mathrm{ver}}(x,y)=C(x,y)\in\{0,1\}.$$

Here $r_{\mathrm{ver}}$ is the verifiable reward, and $C(x,y)=1$ means that the checker accepts $y$ for task $x$. RLVR systems can also use graded rule-based rewards, such as the fraction of tests passed or a separate format score. Binary acceptance is the cleanest case, not the definition of the whole method.

A learned reward model has a different contract:

$$r_\phi(x,y)\approx \mathbb{E}[J(x,y)\mid x,y].$$

In this expression, $J(x,y)$ is a human or AI judgment, $\phi$ denotes the learned parameters, and the expectation represents the judgment the model predicts. The learned model can generalize beyond labeled examples, but it can also reward confident style, miss a factual error, or fail off distribution. An executable checker avoids those particular judge errors. It can still implement the wrong contract perfectly.

The useful formal distinction is soundness and completeness. Relative to intended correctness $q$:

$$C(x,y)=1\Rightarrow q(x,y)=1 \qquad\text{and}\qquad q(x,y)=1\Rightarrow C(x,y)=1.$$

Here the first implication is soundness: every accepted response is genuinely correct, so there are no false positives. The second is completeness: every genuinely correct response is accepted, so there are no false negatives. A formal proof kernel may provide a strong soundness argument relative to its axioms and trusted implementation. A unit test is executable, not a complete specification of behavior. A weak suite can be unsound relative to $q$ because incorrect programs pass; a brittle suite can also be incomplete because valid implementations fail. Exact string matching has the latter problem when equivalent answers use different forms.

The checker is the specification the optimizer sees. If a generated program can read the hidden answers, modify the harness, exploit undefined behavior, or hard-code visible test cases, passing the check does not establish the intended property. Sandboxing protects the checker from the candidate. Hidden tests, fuzzing, metamorphic tests, and manual review increase coverage. None of them turns an incomplete specification into ground truth.

The last row explains a terminology trap. Papers often call a learned correctness model a verifier. Cobbe et al.'s 2021 GSM8K work, for example, trained a model to score sampled solutions and used that score for reranking (Cobbe et al. 2021). That is not the same reward source as an answer-equivalence program, even when the scorer's training labels came from an executable answer check.

Coverage comes before selection

Selection is not training. Suppose one independent sample from a policy is correct with probability $p$, and the system draws $n$ samples under the same decoding distribution. The probability that at least one candidate is correct is

$$P(\text{at least one correct})=1-(1-p)^n.$$

Here $P$ denotes probability. This is oracle coverage for one prompt. It assumes independent draws with the same success probability. It also assumes that a perfect selector can identify a correct candidate whenever one is present. Such a selector can turn coverage into a correct returned answer; it cannot create coverage. If the policy assigns negligible probability to every successful path, increasing $n$ becomes prohibitively expensive before it becomes useful.

A learned selector changes the conclusion. Its best-of-$n$ response is

$$y^*=\underset{1\le i\le n}{\operatorname{argmax}}\;s(x,y_i), \qquad y_i\sim\pi(\cdot\mid x).$$

Here $s(x,y_i)$ is the selector's score, $y_i$ is candidate $i$, and $\pi$ is the generating policy. More samples expose more correct candidates, but they also expose more unusual mistakes that a learned score may rank too highly. Cobbe et al. reported this directly: learned-verifier selection improved as the candidate count rose to 400, then degraded beyond that point (Cobbe et al. 2021). Best-of-$n$ with an imperfect proxy is therefore not guaranteed to improve monotonically.

Pass@$k$ measures oracle coverage rather than the quality of one selected response. If $n$ samples contain $c$ successful completions, the usual finite-sample estimator for $k\le n$ is

$$\widehat{\operatorname{pass@}k} =1-\frac{\binom{n-c}{k}}{\binom{n}{k}}.$$

Here the estimator asks whether at least one of $k$ draws would pass. It does not say that a deployed selector could find that draw. Report the sampling temperature, nucleus threshold, maximum response length, number of generated samples, and checker whenever comparing pass@$k$ values. Otherwise the metric does not specify a reproducible experiment.

Three ways to reuse checked samples

Once a system can score candidates, it can use that information without changing the model, turn accepted responses into a dataset, or update the current policy directly. Those are different algorithms with different failure modes.

Rejection-sampling fine-tuning freezes accepted responses into an ordinary supervised dataset. After collection, they are off-policy relative to later checkpoints. RLVR means reinforcement learning with verifiable rewards: it repeatedly samples from the current policy, runs the checker, estimates an advantage, and updates that policy. Tülu 3 introduced the RLVR label for a rule-reward stage in its open SFT, DPO, and RL recipe (Lambert et al. 2024). The underlying pattern of filtering or optimizing automatically checked samples is older than the label.

DeepSeekMath introduced Group Relative Policy Optimization (GRPO), which replaces PPO's learned critic with a baseline estimated from multiple completions for the same prompt (Shao et al. 2024). Its reported GRPO experiment used learned reward models, so GRPO is an optimizer, not a synonym for RLVR. DeepSeek-R1-Zero later paired GRPO with rule-based accuracy and format rewards directly on a base model. The full DeepSeek-R1 pipeline was broader: it added cold-start SFT, rejection sampling, another SFT stage, and a later mixture of rule-based reasoning rewards with learned helpfulness and safety rewards (Guo et al. 2025).

The skeleton below shows where the checker enters a group-relative RLVR loop. It omits optimizer details that Chapter 28 develops.

for each task x:
    sample g responses y_1, ..., y_g from the current policy
    compute r_i = C_train(x, y_i) for every response
    turn r_1, ..., r_g into relative advantages A_1, ..., A_g
    update the policy to increase high-advantage responses
    constrain drift and record diagnostics

Here $g$ is the number of responses sampled for one task, $C_{\mathrm{train}}$ is the training checker, $r_i$ is response $i$'s reward, and $A_i$ is its advantage relative to the group baseline. With independent binary rewards and per-sample success probability $p$, the chance that a group contains both a pass and a failure is

$$P(\text{mixed group})=1-p^g-(1-p)^g.$$

An informative group needs reward variation. If every response fails or every response passes, a purely group-relative correctness advantage is zero. Other terms, such as a KL penalty or a format reward, may still produce an update, but the correctness signal did not distinguish the samples. For $p=0.05$ and $g=8$, only about 34 percent of groups are mixed. Curricula, prompt filtering, larger groups, and cold-start data can move training toward a range where mixed outcomes occur more often. R1-Zero also shows that cold-start data is not a universal prerequisite.

Outcome and process are a different axis

Outcome rewards describe feedback attached to a completed response. Process rewards describe feedback attached to intermediate steps. That timing says nothing by itself about whether the scorer is executable or learned. Outcome and process are one axis; executable and learned are another.

Here, consider a reasoning trace $\tau=(z_1,\ldots,z_T,y)$. A rule-based outcome checker may use only the final answer:

$$R_{\mathrm{out}}(x,\tau)=C(x,y).$$

Here $z_t$ is reasoning step $t$, $T$ is the number of steps, $y$ is the final answer, and $R_{\mathrm{out}}$ is the outcome reward. Every token in a successful trace may be reinforced together, including detours and mistakes. Useful prefixes inside a failed trace may receive no positive credit.

A learned PRM instead estimates step quality:

$$s_t=\widehat{P}_\phi(S_t=1\mid x,z_{\le t}), \qquad R_{\mathrm{proc}}=A(s_1,\ldots,s_T).$$

Here $S_t=1$ means that step $t$ is judged correct, $s_t$ is the PRM's estimated probability, $\phi$ denotes its learned parameters, $z_{\le t}$ is the trace prefix, and $A$ is an aggregation rule. The aggregation might use a product, a minimum, a learned value, or another shaping rule. There is no universal discounted-sum definition of process supervision.

Lightman et al. trained learned outcome and process reward models for MATH. In their best-of-$n$ experiments, the process-supervised model outperformed the outcome-supervised one, and PRM800K supplied roughly 800,000 human step labels (Lightman et al. 2024). That is evidence for one difficult math setting, not a theorem that process supervision always wins. Natural-language step labels are costly and subjective. A PRM also restores the proxy risk that an executable endpoint check had removed.

The two sources can be combined. An exact endpoint checker can anchor task success while a PRM guides search or assigns denser credit. Keep their metrics separate. A higher process score must not be allowed to compensate for a failed endpoint unless that trade-off is an explicit part of the task contract.

What improvement does the reward establish?

Rule rewards made reasoning tasks unusually scalable because sampled trajectories could be checked without asking a person to label every one. DeepSeek-R1-Zero showed large gains on mathematics, coding, and related verifiable tasks while using rule-based final-prediction and format rewards rather than a neural outcome or process reward model (Guo et al. 2025). That result does not settle three stronger questions:

1. Did the policy learn a path that the base model could not produce, or did it concentrate probability on rare paths already present?
2. Did reasoning improve, or only the probability of reaching a checked endpoint?
3. Does the gain transfer to prompts whose reward structure differs from training?

Evidence cuts in both directions. Spurious-reward experiments found that random or even incorrect rewards recovered much of the MATH-500 gain for Qwen2.5-Math-7B, while the same signals often failed for Llama and OLMo models (Shao et al. 2026). That does not show that correct rewards are unnecessary. It shows that benchmark improvement alone may not identify which information the reward supplied, and that conclusions can depend on the base model.

Pass@$k$ comparisons can also mislead. A correct final answer may be reached through an invalid chain. CoT-Pass@$k$ counts a completion only when both the endpoint and a separate chain-of-thought verifier accept it (Wen et al. 2026). This is stricter than endpoint-only pass@$k$, but the path verifier is learned and fallible, so the new metric exchanges one blind spot for another source of measurement error.

Treat reasoning transfer as an empirical claim with three levels: improvement on held-out instances from the training task family, transfer to a related task family, and improvement on a genuinely different domain. Evidence for the first does not establish the third. Measure all three when a system is meant to be general-purpose.

Build and audit the reward contract

A training run should begin with checker design, not with optimizer selection.

1. Write the intended contract. State what counts as correct, which equivalent outputs are accepted, which side effects are forbidden, and how time or resource limits matter.
2. Harden the training checker. Canonicalize answers, isolate execution, make grader files immutable, and test adversarial candidates. Record false positives and false negatives against a manually audited set.
3. Separate train and evaluation checkers. Use hidden cases, different test generators, or a stronger audit checker for evaluation. Reusing the exact training harness only measures how well the policy learned that harness.
4. Test the initial reward distribution. Track reward rate by prompt and reward variance within each sampled group. All-fail groups provide no positive correctness example; all-pass groups provide no relative correctness signal.
5. Monitor the policy, not only reward. Log response length, entropy or diversity, invalid-format rate, execution failures, and drift from the reference policy. Longer traces and higher training reward are not proofs of better reasoning.
6. Evaluate outside the reward. Report held-out pass rate under the audit checker, manually inspect accepted traces, and measure out-of-domain quality, instruction following, and safety. A narrow reasoning gain can coexist with broad regressions.

Rubric rewards can extend on-policy training to tasks without an executable truth predicate. Rubrics as Rewards, for example, used prompt-specific criteria and an LLM judge on medical and science benchmarks (Gunjal et al. 2025). This is structured learned feedback, not a verifiable reward in the strict sense. The rubric makes the proxy easier to inspect, but the judge can still misread or be gamed by the response.