Direct Preference Optimization and Its Variants

There is a recurring moment in an alignment pipeline. A team has a static file of preference pairs, a chosen response and a rejected one for each prompt, and it wants an aligned model out the other end. The textbook route is RLHF, and RLHF is resource-intensive. The direct-preference line asks whether that cost is necessary, collapses the apparatus into a single classification loss, then builds a family of variants on top of that loss. The controlled result is that none of the variants reliably beats the plain loss they all descend from.

A static file of preferences

RLHF, covered in Chapter 19, aligns a model in three stages: supervised fine-tuning, a reward model fit to human preferences, then a policy-gradient run with proximal policy optimization (PPO) (Schulman et al. 2017) that optimizes the policy against that reward under a Kullback-Leibler divergence (KL) constraint to the reference. The machinery is costly to run. PPO keeps four models in play at once, the policy, a frozen reference, the reward model, and a critic, and its result is sensitive to reward-model quality and to a handful of brittle hyperparameters. For the team with a static set of preference pairs, the question is whether all of that apparatus is necessary, or whether the preference signal can be optimized directly.

The reduction: a language model is an implicit reward model

The reduction rests on one fact about the RLHF objective. Maximizing reward under a KL penalty to a reference policy,

$$\max_{\pi_\theta}\; \mathbb{E}_{x,\,y\sim\pi_\theta}\big[r(x,y)\big] \;-\; \beta\, \mathrm{KL}\!\left(\pi_\theta(\cdot\mid x)\,\|\,\pi_{\text{ref}}(\cdot\mid x)\right),$$

has a closed-form optimal policy, meaning the single best policy this objective implies can be written as a formula rather than found by search: it is the reference reweighted by the exponentiated reward, $\pi^*(y\mid x) \propto \pi_{\text{ref}}(y\mid x)\exp(r(x,y)/\beta)$. Rafailov et al. invert this (Rafailov et al. 2023). Solving for the reward in terms of the optimal policy expresses $r$ as a log-ratio, so the reward that any policy implies is

$$r_\theta(x,y) \;=\; \beta\,\log\frac{\pi_\theta(y\mid x)}{\pi_{\text{ref}}(y\mid x)} \;+\; \beta\log Z(x).$$

Here $Z(x)$ is the partition term, the normalizing sum over every response the model could produce that turns the reweighted reference back into a probability; it is intractable because that sum runs over all possible sequences, far too many to compute. Substitute that implicit reward into the Bradley-Terry model of pairwise preference and $Z(x)$ cancels between the chosen response $y_w$ and the rejected $y_l$. What remains is a supervised loss on preference pairs,

$$\mathcal{L}_{\text{DPO}} = -\,\mathbb{E}_{(x,y_w,y_l)}\,\log \sigma\!\left( \beta\log\frac{\pi_\theta(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)} - \beta\log\frac{\pi_\theta(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)}\right).$$

The reward model and the RL loop are gone. The policy is trained to assign higher implicit reward to the preferred response, where implicit reward is the log-ratio of the policy to a frozen reference, how much more likely the tuned model makes this answer than the frozen starting model does. The title of the paper states the point: the language model is an implicit reward model. Figure 20.2 shows the machinery that this substitution deletes.

Four edits to one loss

Once the loss exists, every later method is an edit to it. Each variant removes one assumption or one piece of machinery the previous line of the loss carried. Figure 20.3 lays out which piece each one changes.

Where the log-sigmoid is unbounded, identity preference optimization (IPO) bounds the objective with a squared target for the reward gap. Azar et al. target a weakness in the derivation (Azar et al. 2023): the Bradley-Terry substitution assumes preferences are well modeled as a deterministic ordering, and with finite, near-deterministic data the log-sigmoid loss can drive the policy to push the implicit reward gap to infinity, overfitting the preference set. IPO replaces the log-sigmoid with a bounded squared objective that regresses the implicit reward gap toward a fixed target, so the optimum stays finite even when preferences are clean. Figure 20.4 contrasts the two loss shapes.

Run a few gradient steps on a single reward gap under each loss and watch the DPO gap climb without bound while the IPO gap settles at its target.

import numpy as np
import matplotlib.pyplot as plt

def sigmoid(z): return 1.0 / (1.0 + np.exp(-z))

beta, lr, target, steps = 1.0, 0.5, 5.0, 60
gd, gi = [0.0], [0.0]
for t in range(steps):
    # DPO gradient on the gap never reaches zero, so the gap keeps growing
    gd.append(gd[-1] + lr * beta * sigmoid(-beta * gd[-1]))
    # IPO regresses the gap to a fixed target, so it settles there
    gi.append(gi[-1] - lr * 2.0 * beta * (beta * gi[-1] - target))
print(f"DPO gap after {steps} steps: {gd[-1]:.2f} and still rising")
print(f"IPO gap converges to target: {gi[-1]:.2f}")
plt.plot(gd, label="DPO log-sigmoid"); plt.plot(gi, label="IPO bounded")
plt.axhline(target, ls="--", c="gray"); plt.xlabel("step"); plt.ylabel("implicit reward gap")
plt.legend(); plt.show()

Kahneman-Tversky optimization (KTO) drops the paired-data requirement. direct preference optimization (DPO) needs a chosen and a rejected response for the same prompt; KTO learns from unpaired binary good or bad labels by scoring each response against a reference point with a prospect-theory utility (Ethayarajh et al. 2024), the same human-value-of-gains-and-losses shape Kahneman and Tversky used. This trades the paired-data constraint for a utility model, which matters because thumbs-up and thumbs-down logs are far more abundant than curated pairs.

odds-ratio preference optimization (ORPO) merges alignment with SFT through an odds-ratio penalty. Instead of a separate alignment stage with a reference model, Hong et al. add an odds-ratio penalty to the SFT cross-entropy (Hong et al. 2024) (the SFT stage itself is Chapter 17), so one pass both teaches the format and down-weights the rejected response. It carries no reference model at all.

simple preference optimization (SimPO) goes reference-free, normalizing the reward by length. Meng et al. drop the reference policy from the implicit reward (Meng et al. 2024), using the average log-probability of a sequence as the reward, and add a target margin between chosen and rejected. The length normalization is the deliberate part: DPO's log-ratio reward correlates with sequence length, which lets a policy raise its score by getting wordier, and dividing by length removes that lever. Figure 20.5 shows the reward against response length under both rewards.

Controlled comparison

Read in sequence, the four edits look like a ladder of improvements, each variant fixing what the last one missed. A controlled comparison undercuts that reading.

If accuracy does not separate the variants, then something else has to decide between them, and that something is the shape of the pipeline and the data.

What actually decides the choice

With the accuracy edge gone, the real axes are the engineering ones the variants trade along. Four of them recur.

• Reference model versus none. Keeping the frozen reference (DPO, IPO) anchors the policy and gives the KL constraint a meaning, at the cost of a second resident model and forward pass. Dropping it (ORPO, SimPO) halves the alignment-stage memory and simplifies the pipeline, but removes the explicit anchor that limits drift from the starting policy.
• Paired versus unpaired data. DPO, IPO, ORPO, and SimPO all need a chosen and a rejected response per prompt. KTO accepts unpaired binary signals, which fits production feedback logs but substitutes a utility model for the pairwise comparison the others optimize.
• Bounded versus unbounded objective. IPO's bounded regression resists the overfitting that DPO's log-sigmoid invites on clean, near-deterministic preferences. The price is a loss that no longer matches Bradley-Terry exactly and a target margin to set.
• Two stages versus one. ORPO merges alignment into SFT, saving a stage and a reference model, at the cost of entangling format learning and preference learning so they can no longer be tuned or audited separately.

The first of these axes is not only a modeling choice. It is forced from below.

Running it

DPO is the smallest of the family to run: one loss, a frozen copy of the SFT model as the reference, and a static file of preference triples. The core is the log-ratio difference inside a log-sigmoid, computed from per-token log-probabilities of the chosen and rejected responses under the policy and the reference; see the loss derivation in DPO loss, Rafailov et al. (arXiv:2305.18290), Sec. 4 (Rafailov et al. 2023). The reference forward pass can be precomputed once because the reference never updates, which is the standard memory optimization.

The one hyperparameter that matters across the family is $\beta$, the strength of the KL constraint that also scales the implicit reward. Too small and the policy drifts off the reference and degrades; too large and it barely moves. Because vanilla DPO is off-policy, trained on a fixed set rather than on samples from the live policy, its ceiling is the fixed data, and the common remedy is iterated or online DPO that regenerates pairs from the current model between rounds. That on-policy variant, and the broader RLHF apparatus it approaches, belong to Chapter 19.

Further reading