RLHF and Reward Modeling

Preference data does not update a language model by itself. Reinforcement learning from human feedback (RLHF) adds two learned systems: a reward model that estimates which responses the labeling procedure would prefer, and a policy that is updated to receive higher estimated reward. The first compresses comparisons into a scalar score. The second searches for responses that score well.

The familiar recipe is to train a reward model on human comparisons, then optimize a policy (the model being trained) against that reward with proximal policy optimization (PPO). Some constitutional and reinforcement learning from AI feedback (RLAIF) variants change who produces some of the comparisons, not what the reward model mathematically represents. In the classic recipe, a learned reward needs a KL constraint because the score is a proxy learned on a limited response distribution. Neither the constraint nor PPO makes that proxy correct.

This chapter keeps three objects separate: the preference record is a measurement, the reward model is an estimator fitted to those measurements, and the policy objective decides how aggressively to use the estimate. Chapter 18 covered the first object. This chapter starts with the second and follows it through an online policy update.

RLHF learns a judge, then trains a policy

For some open-ended tasks, comparing two candidate responses is more practical than writing an ideal answer. A labeler can often identify the clearer summary or the more useful explanation even when several answers would be acceptable. That advantage is task-dependent. A vague rubric, missing expertise, or a forced choice between equally good responses still produces a poor label.

The classic assistant-training pipeline has three stages:

1. Supervised fine-tuning. Demonstrations produce an assistant-shaped model, as described in Chapter 17.
2. Reward modeling. Labelers compare responses sampled for the same prompt. A reward model learns to predict those comparisons.
3. Policy optimization. The current policy samples new responses. The frozen reward model scores them, a frozen reference measures policy drift, and a policy-gradient optimizer updates the policy.

Only the third stage is reinforcement learning. The reward-model stage is supervised learning on comparison labels. Keeping the stages distinct matters because they fail differently: the labels may express the wrong criterion, the reward model may generalize poorly, or the policy may exploit an error in an otherwise useful reward model.

Train a scorer from comparisons

Let a reward-model dataset contain $N$ comparisons:

$$\mathcal{D}=\{(x_i,y_i^+,y_i^-)\}_{i=1}^N,$$

where $x_i$ is prompt $i$; $y_i^+$ is the response preferred under the annotation procedure; $y_i^-$ is the other response; and $N$ is the number of recorded comparisons. A reward model $r_\phi(x,y)$ with parameters $\phi$ assigns one scalar to a prompt-response pair. Under the Bradley-Terry model (Bradley and Terry 1952), define

$$\Delta_i =r_\phi(x_i,y_i^+)-r_\phi(x_i,y_i^-), \qquad p_i=\sigma(\Delta_i)=\frac{1}{1+e^{-\Delta_i}},$$

and minimize the average negative log-likelihood

$$\mathcal{L}_{\mathrm{RM}}(\phi) =-\frac{1}{N}\sum_{i=1}^{N}\log p_i.$$

Here $\Delta_i$ is the score difference; $\sigma$ is the logistic sigmoid; and $p_i$ is the model's probability that the annotation procedure prefers $y_i^+$ to $y_i^-$. The model is never told that either response deserves an absolute reward such as 7. It is asked to make the observed winner more probable than the loser.

A hard winner discards uncertainty. If the same pair receives $m_i$ valid votes and $n_i^+$ of them prefer $y_i^+$, a soft target $q_i=n_i^+/m_i$ can retain the vote split:

$$\mathcal{L}_i =-q_i\log p_i-(1-q_i)\log(1-p_i).$$

In this expression, $m_i$ is the vote count, $n_i^+$ is the number of votes for $y_i^+$, and $q_i\in[0,1]$ is their fraction. A value of $q_i=0.5$ can represent an even split or an explicit tie. It does not explain why annotators disagreed, so the raw votes, annotator metadata, and reasons described in Chapter 18 should remain available.

Rankings require another qualification. InstructGPT showed a labeler between four and nine responses, then derived every ordered pair implied by the ranking (Ouyang et al. 2022). A ranking of $K$ candidates yields $K(K-1)/2$ pairwise constraints, but not that many independent observations: the pairs share the same prompt, candidates, and labeler. Ouyang et al. grouped all pairs from one ranking into one training batch because treating them as independent updates caused rapid overfitting. Rankings amortize annotation effort; they do not manufacture independent evidence.

A reward-model validation report should include held-out log loss, pairwise accuracy, and a calibration curve. Accuracy alone treats a probability of $0.51$ like $0.99$ when both select the same winner. Break the report down by prompt source, task, response length, candidate policy, safety category, and annotator group. RewardBench illustrates the need for structured slices by testing reward models on chat, reasoning, and safety comparisons whose winners sometimes differ for small but verifiable reasons (Lambert et al. 2024).

Pairwise training fixes differences, not absolute quality

The loss depends on a score difference. For any prompt-only offset $c(x)$,

$$r'_\phi(x,y)=r_\phi(x,y)+c(x)$$

produces exactly the same comparison probabilities because $c(x)$ cancels. Raw reward values therefore have no natural zero and should not be compared across prompts. Under a correctly specified and calibrated logistic model, a difference does have an interpretation: $\Delta=\log[p/(1-p)]$ is the fitted log-odds. For example, $p=0.75$ corresponds to $\Delta=\log 3\approx1.10$. This does not make either individual score an absolute measure of response quality.

Figure 19.1 shows this invariance. Shifting both scores leaves the fitted probability unchanged.

Implementations often choose a convenient zero after training. InstructGPT shifted the reward-model bias so labeler demonstrations had mean reward zero (Ouyang et al. 2022). Centering in this way changes no score difference and no pairwise probability. Rescaling is different. Multiplying every reward by a positive constant preserves the ranking but changes the fitted probabilities and the reward's strength relative to the KL penalty. For reproducibility, reward scale and $\beta$ must be recorded together.

The scalar itself is also an assumption. Bradley-Terry represents the compared responses with one latent ordering. Cyclic preferences, context-dependent criteria, and stable differences among annotators may not fit that ordering. Pooling them can still yield a useful predictor, but it does not turn plural judgments into one objective truth.

The policy objective has two terms

Once the reward model is frozen, the policy generates responses and receives their learned scores. A common population objective is

$$J(\theta) =\mathbb{E}_{x\sim\mathcal{D}_x,\,y\sim\pi_\theta(\cdot\mid x)} \left[ r_\phi(x,y) -\beta\log\frac{\pi_\theta(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)} \right].$$

Here $\mathcal{D}_x$ is the prompt distribution; $\pi_\theta$ is the policy being optimized, with parameters $\theta$; $y$ is a response sampled from that policy; $r_\phi$ is the frozen learned reward; $\pi_{\mathrm{ref}}$ is a frozen reference policy, usually derived from SFT; and $\beta>0$ controls the strength of the reference penalty. The first term rewards responses favored by the learned judge. The second makes responses costly when the policy assigns them much more probability than the reference does.

The distribution-level quantity is Kullback-Leibler divergence (KL): how different the new token distribution is from the reference distribution. Taking the expectation of the sampled sequence log-ratio under $\pi_\theta$ gives $\mathrm{KL}(\pi_\theta\|\pi_{\mathrm{ref}})$. For an autoregressive response $y=(y_1,\ldots,y_T)$,

$$\log\frac{\pi_\theta(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)} =\sum_{t=1}^{T} \log\frac{\pi_\theta(y_t\mid x,y_{<t})} {\pi_{\mathrm{ref}}(y_t\mid x,y_{<t})}.$$

In this sum, $T$ is the response length; $y_t$ is token $t$; and $y_{<t}$ is the preceding response prefix. Implementations use these sampled token log-ratios as reward-shaping terms. A sampled log-ratio is not itself the KL divergence, and an individual contribution can be negative. The KL is an expectation over tokens sampled from the policy.

The penalty is a regularizer, not a certificate. A small measured KL does not show that the reward is correct, and a large $\beta$ can prevent useful changes. Some systems keep $\beta$ fixed; others adjust it to track a target KL. Either way, the achieved KL, not only the configured coefficient, belongs in the run record.

KL and PPO clipping constrain different moves

PPO is one optimizer for the objective above (Schulman et al. 2017). It introduces a third policy notation that is easy to confuse with the reference. Here the rollout policy $\pi_{\mathrm{old}}$ is a snapshot that generated the current batch. The reference $\pi_{\mathrm{ref}}$ is the SFT anchor used across many batches. They usually are not the same model.

Here $a_t=y_t$ is the sampled token and $s_t=(x,y_{<t})$ is its prompt-and-prefix state. PPO forms the importance ratio

$$\rho_t(\theta) =\frac{\pi_\theta(a_t\mid s_t)} {\pi_{\mathrm{old}}(a_t\mid s_t)}$$

and maximizes the clipped surrogate

$$L_{\mathrm{clip}}(\theta) =\mathbb{E}_t\left[ \min\left( \rho_t(\theta)\widehat{A}_t, \operatorname{clip}(\rho_t(\theta),1-\epsilon,1+\epsilon) \widehat{A}_t \right) \right].$$

Here $a_t$ is the sampled token; $s_t$ is its prompt-and-prefix state; $\rho_t$ compares the updated policy with the rollout snapshot; $\widehat{A}_t$ is an estimated advantage; $\epsilon>0$ is the clipping width; and $\operatorname{clip}(z,l,u)$ truncates $z$ to the interval $[l,u]$. The advantage compares the shaped return from a partial response with a critic's prefix-conditioned baseline. This reduces gradient variance. It does not reveal which token was causally “good.”

The two controls solve different problems:

• The reference KL discourages cumulative semantic drift from the SFT policy.
• PPO clipping limits the incentive to change sampled likelihood ratios far beyond the clipping interval while reusing one rollout batch.

Clipping does not impose a hard trust region or guarantee that the final policy stays within a chosen distance (Wang et al. 2019). Learning rate, advantage normalization, value estimation, batching, and numerical stability still matter.

Four roles participate in the loop

Classic PPO-based RLHF has four parameter roles. They are not necessarily four same-sized models loaded as full replicas on every accelerator.

The rollout snapshot $\pi_{\mathrm{old}}$ is another logical policy state, but stored rollout log-probabilities can represent it during optimization. InstructGPT used a 6B reward model and a 6B value model even when training a 175B policy (Ouyang et al. 2022). That concrete design is why “four roles” should not be translated into “four copies of the policy.” Sharing, sharding, adapters, offloading, and recomputation change peak memory and throughput.

Figure 19.3 shows the data flow. Dashed boxes stay frozen during a PPO update; the policy and critic receive gradients.

From trajectory preferences to instruction following

Reward learning from comparisons predates its use in chat models. Christiano et al. trained reward predictors from human comparisons of short trajectory segments, then optimized agents in Atari and simulated locomotion tasks against the learned rewards (Christiano et al. 2017). Ziegler et al. brought the method to language tasks including stylistic continuation and summarization (Ziegler et al. 2019). Stiennon et al. then built the recognizable language pipeline for summarization: collect pairwise judgments, train a reward model, and optimize a language policy with PPO and a reference penalty (Stiennon et al. 2020).

InstructGPT broadened that recipe from a specific task to a distribution of API instructions and made the SFT, reward-model, and PPO sequence widely known. Its 1.3B PPO-ptx model was preferred to the 175B GPT-3 baseline by labelers on its held-out API prompt distribution (Ouyang et al. 2022). That result demonstrates the value of post-training on the measured distribution. It does not show that the smaller model had greater general capability. The same paper reported regressions on some public NLP tasks and simple mistakes outside the headline preference comparison.

Constitutional AI changed the source of part of the feedback. In its harmlessness experiment, a model applied written principles to compare responses; those AI comparisons trained a preference model that supplied the RL reward (Bai et al. 2022). This removed human harmlessness labels from that stage, not human normative choices or evaluation. People still chose the principles, and the setup retained human feedback for helpfulness. RLAIF names that substitution of AI-produced feedback for some human-produced feedback.

PPO is not part of the definition of human feedback. RLOO and GRPO are critic-free policy-gradient alternatives, but they construct baselines differently: RLOO uses the mean return of the other samples, while GRPO standardizes rewards within a same-prompt group (Ahmadian et al. 2024; Shao et al. 2024). DAPO modifies the group-relative family for verifiable reasoning and can remove the explicit reference term in that setting (Yu et al. 2025). These methods are prominent in disclosed reasoning work, not a basis for claiming that one optimizer now governs every post-training stack. Chapter 28 covers them in their main application. The offline alternative, which removes the explicit reward model and online RL loop, begins in Chapter 20.

Optimization tests the proxy outside its training distribution

A reward model is evaluated on held-out comparisons sampled from some candidate distribution. Policy optimization changes that distribution. It therefore asks the reward model to score responses selected precisely because they receive unusually high predicted reward. This is a harder test than ordinary validation.

The failure has direct human evidence. In summarization experiments, Stiennon et al. trained policies at several optimization strengths. Under light optimization, both learned reward and labeler preference improved. With stronger optimization, predicted reward continued to improve while labeler preference fell, eventually becoming anti-correlated with the proxy in that experiment (Stiennon et al. 2020).

Gao et al. studied the curve more systematically with a controlled substitute for expensive repeated human evaluation. A fixed gold reward model stood in for human judgment and generated labels used to train a separate proxy reward model. Optimizing the proxy eventually reduced the independent gold-model score, for both RL and best-of-$n$ selection, and the curve changed with model size, data size, optimizer, and KL penalty (Gao et al. 2023). This demonstrates proxy over-optimization in a synthetic gold-model setup. It does not measure a universal curve of true human quality.

The small program below makes the distinction explicit. Its quadratic gold curve is a toy illustration, not a reproduction of Gao et al.'s fitted coefficients. Change curvature to see why there is no universal stopping point.

import numpy as np
import matplotlib.pyplot as plt

policy_shift = np.linspace(0, 10, 200)
proxy_score = 0.35 + 0.12 * policy_shift
curvature = 0.016  # illustrative, not an estimate from Gao et al.
gold_score = 0.45 + 0.16 * policy_shift - curvature * policy_shift**2
best = int(np.argmax(gold_score))

plt.plot(policy_shift, proxy_score, label="optimized proxy score")
plt.plot(policy_shift, gold_score, label="independent gold-model score")
plt.axvline(policy_shift[best], ls="--", color="gray")
plt.xlabel("policy shift from reference")
plt.ylabel("score (illustrative units)")
plt.title("A proxy can improve after its independent check worsens")
plt.legend()
plt.show()

print(f"gold-model score peaks near shift={policy_shift[best]:.1f}")

Reward hacking is the behavioral consequence of this generalization gap. The policy may discover verbosity, confident phrasing, formatting, agreement with a user, or another feature that raises reward without improving the intended task. Sycophancy is one measured example: human and learned preference judgments sometimes favored a convincing answer that agreed with the user over a correct one, and optimization against the preference model could amplify that trade-off (Sharma et al. 2024). This is evidence about one path to sycophancy, not a claim that RLHF is its only cause. Reward tampering, in which an agent changes the scoring process itself, is a separate problem from exploiting a fixed misspecified score.

What the controls can and cannot do

Each control addresses a different boundary. None replaces an evaluation that is independent of the optimized reward.

Ensembles are a partial control, not a proof. Coste et al. found that conservative objectives over reward-model ensembles reduced over-optimization in a synthetic gold-reward setup, with particularly strong results for best-of-$n$ selection (Coste et al. 2023). Shared training data and shared rubrics can still produce shared errors.

The “alignment tax” is likewise an empirical regression, not a universal law that safety must reduce capability. InstructGPT used the term for lower scores on specific public NLP datasets. Mixing pre-training gradients into PPO greatly reduced those regressions without reducing labeler preference, although some gaps remained. Increasing the KL coefficient alone did not recover the same tasks and reduced validation reward (Ouyang et al. 2022). Over-refusal is a different failure: it is a miscalibrated safety boundary, not another name for any capability regression.

Which post-training method produces that regression is itself measurable. Shenfeld et al. fine-tuned the same base models on the same new tasks with supervised fine-tuning and with on-policy RL, matched the new-task score, then measured what had been lost on prior tasks. The RL runs retained more. Their explanation is a single quantity: forgetting tracked the KL divergence between the fine-tuned and the base policy, measured on the new task, rather than the number of steps, the learning rate, or the size of the weight change. Supervised fine-tuning has no term that keeps the policy near the base, so it can reach the new task from arbitrarily far away. On-policy RL samples from the current policy and reweights what that policy already produces, which biases it toward the KL-nearest solution that satisfies the reward. The authors name the bias RL's Razor and report it for language models and for robotic control (Shenfeld et al. 2025).

That gives the reference-KL row above a second reading. An explicit $\beta$ penalizes drift on purpose, while on-policy sampling limits drift as a side effect of where the gradients come from. A run that reports its achieved KL is therefore reporting a predictor of retention, not only a leash on the optimizer. The finding is a correlation established over particular task pairs, so it suggests which recipe to try first and does not replace the retention suite.

Use external judgment to decide when to stop

A practical run begins with a gate on the reward model and ends with a gate that does not use that reward model. One defensible procedure is:

1. Split prompts by source and near-duplicate family before training the reward model.
2. Fit the reward model on grouped comparisons or rankings, preserving ties and raw votes.
3. Report held-out log loss, accuracy, calibration, and slice results. Do not begin RL if critical slices fail.
4. Freeze the chosen reward-model version, reference version, tokenizer, chat template, and reward normalization.
5. Sample responses from $\pi_{\mathrm{old}}$, then record the prompt, response, reward score, reference log-probabilities, rollout log-probabilities, and model versions.
6. Compute shaped returns and critic-based advantages, apply limited PPO updates, and measure the achieved policy-to-reference KL.
7. At fixed intervals, run held-out human or task evaluation plus capability and safety regression suites.
8. Add audited on-policy comparisons when the policy reaches response regions the reward model did not cover; version and retrain the reward model rather than silently changing it mid-run.
9. Stop or roll back when external quality flattens or declines, even if the learned reward continues to rise.

The run dashboard should keep proxy and external evidence in separate panels:

The stopping rule belongs to the external rows. A rising reward curve proves that optimization is working against its score. It does not prove that the model is getting better.

Further reading