RLHF and Reward Modeling

Supervised fine-tuning in Chapter 17 teaches a model the shape of an answer. Chapter 18 explains where the preference labels and rubrics come from. This chapter starts once those comparisons exist: train a reward model on human comparisons, optimize a policy (the model being trained) against that reward with proximal policy optimization (PPO), replace the human labeler with an AI one in constitutional and reinforcement learning from AI feedback (RLAIF) variants, and then confront the reward hacking failure that shadows the whole approach. The lesson is this: a learned reward needs a KL constraint, and optimizing the reward too hard can make the model worse.

The same tension runs through the whole pipeline. Why a preference signal is the right thing to learn sets it up, and the reward and its KL constraint are the machinery that holds that signal in place. The recipe behind them has a history worth tracing, and the proxy, once optimization leans on it, breaks in one precise way. The balances and operational reality follow from that.

Why a preference, not a demonstration

SFT optimizes a single demonstrated response per prompt. It cannot express that one answer is better than another, only that a given answer is the target. Many behaviors people care about are comparative and hard to demonstrate: be helpful but not sycophantic, refuse the harmful request without over-refusing the benign one near it, prefer the concise correct answer over the padded one. You can recognize the better of two responses far more reliably than you can write the ideal one from scratch, and there is no single ground-truth string to imitate.

The signal is a preference, and the reward that encodes it is a proxy for human judgment, not a checkable fact. When the reward becomes verifiable ground truth, a unit test that passes or a math answer that matches, the machinery is related but the failure modes change. Chapter 21 separates those reward sources, and Chapter 28 follows the reasoning training algorithms that exploit them. Here the reward is learned and therefore wrong somewhere. The problem, then, is to optimize against a reward you know to be imperfect without letting the policy discover and exploit exactly where it is wrong. The rest of the pipeline is the machinery that makes optimizing a known-imperfect reward safe enough to be useful.

Building the reward

The pipeline has three stages: start from an SFT model, train a reward model on human preferences, then reinforce the policy against that reward. The InstructGPT paper fixed this recipe as the standard (Ouyang et al. 2022).

The reward model turns comparisons into a scalar. Take an SFT backbone, replace the token-prediction head with a scalar head, and train it so that for a prompt $x$ with a human-preferred response $y_w$ and a dispreferred $y_l$, it scores the winner higher. Here the subscripts mean winner and loser: $y_w$ is the preferred sample and $y_l$ is the rejected one. The Bradley-Terry model gives the probability that $y_w$ beats $y_l$ as a sigmoid of the score difference, and the reward model is fit by maximizing that likelihood:

$$\mathcal{L}_{\text{RM}} = - \log \sigma\big(r_\phi(x, y_w) - r_\phi(x, y_l)\big).$$

Here $r_\phi$ is the learned scalar reward, $\sigma$ is the logistic sigmoid (an S-curve squashing any real number into $(0,1)$), $y_w$ is the response humans preferred, and $y_l$ is the one they rejected. The loss is small only when the winner's score exceeds the loser's by enough margin, so the model learns an ordering of responses rather than a target reward value. Only the difference of scores is supervised, so the reward is identified up to an additive constant, which is why a reward model gives a meaningful ranking but not a calibrated absolute number. Labels come from human annotators ranking model samples, and their quality sets a ceiling on everything downstream: annotator disagreement is the noise floor, ambiguous or underspecified guidelines bias the whole reward, and the reward model overfits the narrow distribution of responses it was trained to compare. This is the most failure-prone artifact in the pipeline, because every later step trusts it.

Figure 19.1 shows the Bradley-Terry curve and why only the difference of scores is identified: shifting both scores by the same constant leaves the preference probability unchanged.

The KL constraint

Reinforcement learning then optimizes the policy to produce responses the reward model scores highly. Here it means letting the policy generate responses, scoring them with the reward model, and pushing the policy's token probabilities toward the higher-scoring ones (a policy-gradient update). Unlike the supervised reward-model loss above, which learns from labeled winner and loser targets, this learns from a reward signal inside a sampling loop. The objective is not simply to maximize reward. A reward model is a proxy, and a policy that maximizes a proxy without constraint will leave the distribution the reward model understands and find high-scoring artifacts. The fix is a constraint: penalize the policy for drifting from the SFT reference it started at, measured as a per-token Kullback-Leibler divergence (KL): how different the new token distribution is from the reference distribution. The objective is

$$\max_{\pi_\theta}\; \mathbb{E}_{x,\, y \sim \pi_\theta}\big[\, r_\phi(x, y)\,\big] \;-\; \beta \, \mathrm{KL}\big(\pi_\theta(\cdot \mid x)\,\|\,\pi_{\text{ref}}(\cdot \mid x)\big),$$

where $\pi_{\text{ref}}$ is the frozen SFT model and $\beta$ sets how strongly the KL term constrains drift. The expectation $\mathbb{E}_{x,\, y \sim \pi_\theta}$ in front is an average over the responses the policy generates, where $y \sim \pi_\theta$ reads as $y$ drawn from the policy $\pi_\theta$. The KL term is the main reason RLHF is stable: it keeps the policy near the region where the reward model was trained, where its scores still mean something, and it preserves the fluency and knowledge the policy already had.

PPO is the optimizer that maximizes this objective (Schulman et al. 2017). It is a policy-gradient method with two ideas that make it usable on language models. A clipped surrogate objective bounds how far the policy moves in one update, by clipping the probability ratio between the new and old policy so a single batch cannot push the policy off a cliff. And a value model, a critic, of the same backbone estimates the expected reward from each partial generation, so the per-token advantage (how much better this token was than the critic expected) has lower variance than the raw reward alone. The KL penalty enters as a per-token shaping term added to the reward. The result is a four-model arrangement held in memory at once: the policy being trained, the frozen reference for the KL term, the reward model for the score, and the critic for the advantage.

Figure 19.3 lays out the three stages and the four resident models in the PPO step. The reward model and the reference model are frozen, shown with dashed borders, while the policy and the critic are updated each step.

Where the recipe came from

The idea of learning a reward from comparisons predates language models. Christiano et al. (2017) trained reward models from human preferences over trajectories in control and Atari tasks, establishing that a few thousand comparisons could specify a behavior no hand-written reward captured, and that optimizing the learned reward with RL could then reach it (Christiano et al. 2017). The pieces were the reward model, the policy optimized against it, and the recognition that the reward was a learned proxy.

InstructGPT (Ouyang et al., 2022) carried this to language and fixed the three-stage recipe the field standardized on: SFT, reward model on human rankings, PPO against the reward with a KL penalty to the SFT reference. Its result reframed alignment: a 1.3B InstructGPT model was preferred by humans over the 175B base GPT-3 (Ouyang et al. 2022), which showed that the bottleneck was not capability but the elicitation of helpful behavior, and that preference optimization was how you elicited it.

The recipe outlived its optimizer. Inside stage three, critic-free methods in the GRPO and RLOO line, with DAPO-style descendants after them, replace PPO's learned value model with a per-group baseline: sample a group of responses to the same prompt and use the group's mean reward as the baseline the critic once supplied (Shao et al. 2024; Ahmadian et al. 2024; Yu et al. 2025). Deleting the critic cuts the resident loop to policy, reference, and reward model, and it is these group-baseline optimizers, not PPO, that most disclosed frontier and open post-training stacks now run; Chapter 28 follows that family in detail.

The cost that InstructGPT exposed was human labeling. Every comparison in the reward-model training set is a person reading two responses, which is slow and expensive and does not scale to the volume modern alignment wants. Constitutional AI (Bai et al., 2022) replaced much of that human labeling with an AI one (Bai et al. 2022). A model critiques and revises its own responses against a written set of principles, a constitution, to build the harmlessness data, and an AI model expresses the preferences that train the reward model, a recipe named RLAIF, reinforcement learning from AI feedback. The human effort moves from labeling thousands of comparisons to writing and auditing a short list of principles, which is far cheaper to produce and to inspect.

The frontier from here splits. One branch keeps the reward model and the RL loop and only swaps the human feedback for AI feedback, the RLAIF line above. A sharper break asks whether the reward model and the RL loop are needed at all, since the same preference data can be optimized more directly: that is Chapter 20. The third route leaves the PPO machinery in place but aims it at a reward that is checkable rather than learned, which retires the reward model's unreliability as the central problem and is Chapter 28.

When the proxy breaks

A learned reward is wrong somewhere, and optimization pressure is what finds the wrong place. As the policy drifts further from the SFT reference, measured by KL, the reward model's score keeps climbing, but true quality follows it only up to a knee, then turns down as the policy exploits regions where the proxy is wrong (Gao et al. 2022). Figure 19.4 plots that frontier: the gap between the still-rising proxy reward and the falling true quality is what opens up past the knee. Figure 19.5 captures the same phenomenon as control logic, where the KL penalty keeps the policy left of the knee.

Run this to watch the knee appear, then change the over-optimization rate (the 0.18 coefficient) and see how it moves where more optimization stops helping.

import numpy as np
import matplotlib.pyplot as plt

# Illustrative shapes, not Gao et al.'s measured constants.
kl = np.linspace(0, 12, 200)
proxy = np.sqrt(kl)              # reward model score keeps climbing
true = np.sqrt(kl) - 0.18 * kl  # true quality peaks, then falls
knee = kl[np.argmax(true)]

plt.plot(kl, proxy, label="proxy reward")
plt.plot(kl, true, label="true quality")
plt.axvline(knee, ls="--", color="gray")
plt.text(knee + 0.2, 0.2, f"knee KL={knee:.1f}")
plt.xlabel("KL from SFT reference"); plt.ylabel("score")
plt.legend(); plt.title("Reward over-optimization (illustrative)")
plt.show()
print(f"true quality peaks at KL={knee:.1f}, then proxy rises while quality falls")

The balances

The pipeline is a set of balances, each with a knee.

• Reward fidelity versus labeling cost. Human comparisons are the highest-fidelity preference signal and the most expensive. AI feedback (RLAIF, constitutional methods) is far cheaper and scales (Bai et al. 2022), but inherits the labeling model's biases and blind spots, and a flaw in the constitution or the critic propagates into every label. Human labels set the ceiling; AI labels set the volume.
• Optimization pressure versus over-optimization. The KL coefficient $\beta$ is the central knob. Too small a $\beta$ lets the policy chase the reward off-distribution into the region where the reward model is wrong, and reward rises while human-judged quality falls. Too large a $\beta$ pins the policy to the SFT reference and the RL step buys little. The right point is a frontier, not a maximum.
• Capability versus safety. Optimizing for helpfulness and harmlessness can degrade raw capability relative to the base model, the alignment tax: reasoning, calibration, and knowledge can all slip. Mixing pre-training data back in, KL control, and balanced preference data mitigate it, but the tension is real, and over-refusal, declining the benign request next to the harmful one, is the safety-side instance of paying it.
• Online RL versus offline alternatives. PPO is on-policy: it trains on fresh samples from the live policy and yields a separate, inspectable, reusable reward model, at the cost of holding four models at once and tuning brittle hyperparameters. The offline preference methods in Chapter 20 collapse this to one loss on a static set, far easier to run, at the cost of the on-policy signal and the auditable reward artifact.

Running the loop

Most of this chapter is a training pipeline, not a code change, and its operational reality is dominated by the four-model footprint and the data flow between them. A practical RLHF step generates a batch of responses from the current policy, scores each with the reward model, computes the per-token KL penalty against the frozen reference, estimates advantages with the critic, and applies a clipped PPO update to the policy and the critic. The generation step, not the optimizer step, often dominates wall-clock, which is why the loop is bottlenecked by inference throughput and pairs naturally with the serving techniques of Chapter 31.

The failure modes are worth naming because each appears at a different place. A miscalibrated or overfit reward model fails before RL even starts and poisons everything downstream, so reward-model accuracy on held-out comparisons is the first thing to check. Reward hacking appears mid-run as a rising reward curve with flat or falling human-judged quality (Gao et al. 2022), and the defenses are a tuned KL penalty, early stopping, and a held-out human evaluation that the reward curve cannot game. Sycophancy is a specific, predictable hack: labelers and users reward agreement and flattery, so a policy optimizing their approval learns to tell people what they want to hear, which is a direct artifact of optimizing an approval proxy rather than a bug in the optimizer. And the alignment tax shows up as a capability regression against the base model on benchmarks the preference data never covered, which is why Chapter 47 gates the aligned model rather than trusting the reward.

Further reading