Verifiers and Process Supervision

Reasoning systems improve when they can tell good work from bad work. The central object is therefore not the chain, the tree, or the reinforcement learning algorithm. It is the verifier. A verifier can be a unit test, an answer checker, a proof kernel, a learned outcome reward model, a process reward model, or a generative judge that writes its own critique. Each version answers a different question, and that question determines what training and test-time compute can buy.

Outcome, Process, and the Unit of Credit

Let a model produce a trace $z_{1:T}$ and a final answer $y$ for prompt $x$. Outcome supervision defines a reward on the terminal object:

$$r_{\text{out}}(x, z_{1:T}, y) = \mathbf{1}\left[\operatorname{check}(x, y)=1\right].$$

Here, by contrast, process supervision defines rewards over prefixes:

$$r_{\text{proc}}(x, z_{1:t}) \in \{-1, 0, 1\}, \qquad 1 \le t \le T.$$

Here $x$ is the prompt, $z_{1:T}$ is the reasoning trace, $y$ is the final answer, and the check function is whatever verifier the task supplies. The first of these is cheap but sparse. The second buys density at a price. Outcome reward only asks whether the answer passed; process reward asks, of each step, whether it was a good move. Uesato et al. compared the two on math word problems and found that outcome feedback could match final-answer error with less label supervision, while process feedback was needed to reduce reasoning-step errors among correct answers (Uesato et al. 2022). Lightman et al. later trained process reward models (PRMs, which score each step of a solution rather than only the final answer) on PRM800K and found process supervision stronger than outcome supervision on challenging MATH problems (Lightman et al. 2023).

This is a credit-assignment trade. If all the system sees is a wrong final answer, it must infer which step ruined the path. A process label points earlier and can prune a branch before it consumes more budget. But the process label is itself a judgment, often learned from humans, so it reintroduces proxy error. The verifier that made RLVR (reinforcement learning from verifiable rewards, training on problems whose answers can be checked automatically) attractive because it avoided a learned reward gap can become a learned reward again.

Where the step labels come from turned out to dominate that trade. Lightman et al. labeled steps with humans; the cheap substitute, scoring a step by Monte Carlo rollouts from it, produces unreliable labels, and the Qwen team's post-mortem of building PRMs found that a consensus filter, keeping only steps where the Monte Carlo estimate and an LLM judge agree, is what made their Qwen2.5-Math-PRM and the ProcessBench evaluation work (Zhang et al. 2025). ThinkPRM merges the generative-verifier idea below with step-level judging: a PRM that reasons about each step in its own verification chain of thought, and matches discriminative PRMs with roughly 1% of the process labels (Khalifa et al. 2025).

The Verifier Ladder

The ladder is ordered by semantics, not prestige.

• Heuristic judges cover open-ended tasks but have weak guarantees. They are useful for triage and rough ranking, yet easy to optimize against.
• Outcome reward models learn to rank finished solutions. Cobbe et al. used trained verifiers to rerank many math completions and showed that verification scaled better with data than direct fine-tuning on GSM8K (Cobbe et al. 2021).
• Executable checks include unit tests, answer equivalence, parsers, and type checks. They are cheap once written, but their coverage is bounded by the specification.
• Process reward models score intermediate steps. They make search and training denser, but the labels are costly and the model can learn labeler artifacts (Lightman et al. 2023).
• Formal proof checkers provide the strongest terminal semantics, because a small kernel checks the proof. They are narrow and demand formalization.

The ladder also explains why different research areas appear to move at different speeds. Math, code, and formal proof offer natural checkers, though the 2025 IMO-gold results discussed in Chapter 28 show even that boundary being stretched. Open-ended advice, synthesis, and policy reasoning do not. The algorithm may be the same, but the reward source is not.

Generative Verifiers

A scalar verifier compresses judgment into one number. Generative verifiers instead use the language model's own next-token machinery to reason about a candidate. GenRM trains verifiers with next-token prediction on verification and solution generation, enabling the verifier to write a chain of thought and spend its own test-time compute through voting (Zhang et al. 2024). This matters because some errors are too subtle for a shallow classifier. A verifier that can reason may catch them.

The cost is that verification becomes another reasoning problem. The selector in Chapter 30 is no longer a fixed component; it has its own latency, sampling budget, and failure modes. A generative verifier can improve best-of-$N$ selection (sample $N$ answers, keep the one the verifier scores highest), but it can also agree with plausible wrong work. Its advantage depends on whether judging is easier than generating for the task at hand.

def best_of_n_with_verifier(prompt, n):
    candidates = [generator.sample(prompt) for _ in range(n)]
    judged = []
    for c in candidates:
        critique = verifier.generate(format_judgment_prompt(prompt, c))
        judged.append((extract_score(critique), c, critique))
    return max(judged, key=lambda x: x[0])

The critique is not only a score. It becomes audit data. If the system later fails, operators can inspect whether the candidate was wrong, the verifier was wrong, or the extraction from critique to score was wrong. That audit trail is one reason to prefer structured verification over an opaque scalar when the task is operationally important.

When the Checker Becomes the Objective

Every verifier is also an attack surface. In training, the policy learns the verifier's boundary. In inference, best-of-$N$ selects the sample that maximizes the verifier's score. If the checker is incomplete, more optimization pressure finds its holes. This is reward hacking in Chapter 19, selector over-optimization in Chapter 30, and unit-test overfitting in code generation. The names differ because the loop differs, but the shape is the same.

There is no universal fix. The practical answer is layered verification. A code answer may need unit tests, hidden tests, static analysis, type checking, and human review for specification fit. For a math answer the relevant layers are different ones: symbolic equivalence, numeric substitution, and a step-level sanity check. An agent trace asks for yet another set, environment feedback, permission checks, and outcome evaluation. The right verifier is a bundle, and the bundle's weakest layer often sets the ceiling.

The verifier is the hinge between this part's chapters. Search calls on it to prune, and program-aided reasoning reaches for external runtimes to strengthen it; RLVR turns it into a training reward, while test-time scaling leans on it to convert coverage into accuracy. The next chapter moves from the verifier as a component to the full training loop that uses it as reward.

Further reading