Direct Preference Optimization and Its Variants

Suppose a team already has prompts with two candidate responses and a judgment about which response is better. Classical RLHF would fit an explicit reward model, generate new responses, and optimize a policy against that reward. Direct Preference Optimization (direct preference optimization (DPO)) takes a shorter route. DPO fits a preference classifier without an explicit reward model. Under a specific model of preferences, its loss trains the language model directly (Rafailov et al. 2023).

This simplification matters. DPO collapses the reward-model and policy-optimization apparatus into a single classification loss over a static dataset. It does not, however, turn alignment into ordinary supervised fine-tuning or make the assumptions behind RLHF disappear. DPO still needs paired judgments, a frozen reference policy, a model of how preferences arise, and evaluation outside the training pairs.

What DPO removes, and what it keeps

Let the preference dataset be

$$\mathcal{D}=\{(x_i,y_{w,i},y_{l,i})\}_{i=1}^{N},$$

where $x_i$ is a prompt, $y_{w,i}$ is the response preferred by the rater, $y_{l,i}$ is the rejected response, and $N$ is the number of comparisons. Starting from an instruction- tuned model, DPO trains one policy $\pi_\theta$ and compares it with a frozen reference $\pi_{\mathrm{ref}}$, usually a copy of that starting model.

Compared with PPO-based RLHF in Chapter 19, DPO removes three components from the training loop:

• no separately trained scalar reward model;
• no rollouts from the changing policy during the preference-training stage;
• no critic or policy-gradient optimizer.

What remains is important. The reference policy defines what counts as movement, the data contains only the candidate responses that were collected, and the pairwise loss specifies how one observed comparison should constrain the policy. The language model is an implicit reward model only in the precise sense derived below. It is not a general-purpose judge that can be queried independently of its reference.

The derivation and its assumptions

The derivation starts from a KL-regularized reward objective. For one prompt $x$, consider all probability distributions $\pi(\cdot\mid x)$ over possible responses:

$$\max_{\pi}\; \mathbb{E}_{y\sim\pi(\cdot\mid x)}[r(x,y)] -\beta\,\mathrm{KL}\!\left( \pi(\cdot\mid x)\,\|\,\pi_{\mathrm{ref}}(\cdot\mid x) \right),$$

where $r(x,y)$ is a scalar reward, $\beta>0$ is the KL coefficient, and $\pi_{\mathrm{ref}}$ is the reference policy. If the reward is finite, its normalizer exists, and $\pi$ puts probability only on the support of the reference policy, the maximizing distribution is

$$\pi^*(y\mid x) =\frac{1}{Z(x)}\,\pi_{\mathrm{ref}}(y\mid x) \exp\!\left(\frac{r(x,y)}{\beta}\right),$$

with

$$Z(x)=\sum_y \pi_{\mathrm{ref}}(y\mid x) \exp\!\left(\frac{r(x,y)}{\beta}\right).$$

The closed form is over distributions, not neural-network parameters. A finite language model $\pi_\theta$ may not represent $\pi^*$ exactly, and gradient descent may not find the global optimum. The formula tells us what the ideal KL-regularized solution looks like; it does not solve for $\theta$.

Rearranging the optimal-policy equation gives

$$r(x,y)=\beta\log\frac{\pi^*(y\mid x)} {\pi_{\mathrm{ref}}(y\mid x)}+\beta\log Z(x).$$

The last term depends on the prompt but not on the response. More generally, reward is identified only up to a prompt-only constant: adding any $c(x)$ to every response reward for the same prompt changes neither pairwise reward differences nor the optimal normalized policy. The policy-to-reference log-ratio is therefore one representative of an equivalent class of rewards, not a uniquely recovered human utility.

DPO next assumes a Bradley-Terry preference model:

$$p(y_w\succ y_l\mid x)= \sigma\!\left(r(x,y_w)-r(x,y_l)\right), \qquad \sigma(z)=\frac{1}{1+e^{-z}}.$$

Here $y_w\succ y_l$ means that $y_w$ is preferred to $y_l$. The Bradley-Terry assumption represents each response with one scalar score and turns the score difference into a logistic choice probability. It cannot directly express ties, cyclic preferences, different rater populations, or judgments that depend on unrecorded context.

Substitute the implicit reward into the Bradley-Terry model. The prompt-only constant cancels between $y_w$ and $y_l$, leaving the DPO loss

$$\mathcal{L}_{\mathrm{DPO}}(\theta)= -\mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}} \log\sigma\!\left(\beta A_\theta(x,y_w,y_l)\right),$$

where

$$A_\theta= \underbrace{\left[\log\pi_\theta(y_w\mid x)-\log\pi_{\mathrm{ref}}(y_w\mid x)\right]}_{\text{chosen movement}} -\underbrace{\left[\log\pi_\theta(y_l\mid x)-\log\pi_{\mathrm{ref}}(y_l\mid x)\right]}_{\text{rejected movement}}.$$

Here $A_\theta$ is the preferred response's movement relative to the reference minus the rejected response's movement relative to the reference. Exact equivalence to the idealized RLHF solution requires the Bradley-Terry model to be well specified, adequate data coverage, compatible reference and behavior-policy support, and successful optimization. DPO removes an explicit reward-model and PPO loop under these assumptions. It is not exactly equivalent for arbitrary neural models, misspecified preferences, or finite off-support data.

Interpreting relative movement

The margin equation is easy to misread. DPO does not merely ask whether the chosen response has a larger probability than the rejected response. It asks whether training has moved the chosen response upward relative to the reference more than it has moved the rejected response. Consequently, a positive DPO margin does not guarantee $\pi_\theta(y_w\mid x)>\pi_\theta(y_l\mid x)$. A response that was already unlikely under the reference can win the relative comparison while remaining unlikely in absolute terms.

The per-example gradient makes the update clearer:

$$\nabla_\theta\mathcal{L}_{\mathrm{DPO}} =-\beta\,\mathbb{E}_{\mathcal{D}} \left[ \sigma(-\beta A_\theta) \left( \nabla_\theta\log\pi_\theta(y_w\mid x) -\nabla_\theta\log\pi_\theta(y_l\mid x) \right) \right].$$

Here pairs with a negative or small margin receive the greatest weight. Once the margin is large, the sigmoid saturates and the pair contributes little. Because the two sequences share model parameters, this contrast does not promise that the chosen-response log-probability will rise monotonically at every checkpoint. Track it instead of inferring it from training loss.

Sequence log-probability is a token sum,

$$\log\pi_\theta(y\mid x)= \sum_{t=1}^{|y|}\log\pi_\theta(y_t\mid x,y_{<t}),$$

where $|y|$ is the number of response tokens and $y_{<t}$ is the prefix before token $t$. The DPO margin therefore sums policy-to-reference changes across all response tokens. Length can confound the margin when the data has systematic length differences, but the sign of each token log-ratio can be positive or negative. DPO reward is not mathematically guaranteed to increase with response length.

The role of $\beta$ also has two layers. In the KL-regularized objective, larger $\beta$ means stronger theoretical regularization toward the reference. In the implemented DPO loss, $\beta$ also sets the logit temperature and multiplies the gradient. Finite-step training need not produce a monotonic relationship between $\beta$ and measured policy drift. Tune it, then report the achieved policy/reference divergence and task quality rather than treating the configured value as the achieved constraint.

The four variants solve different constraints

IPO, KTO, ORPO, and SimPO are often arranged as a sequence of upgrades. That framing hides their actual purpose. Each changes a different part of the problem: the loss shape, the label format, the use of an SFT term, or the response score. They also require different data and different training resources.

IPO: keep pairs and the reference, change the loss shape

Identity Preference Optimization (identity preference optimization (IPO)) starts from a theoretical problem with DPO's Bradley-Terry reduction. If observed preferences are deterministic or nearly deterministic, the negative log-sigmoid loss has its infimum at an ever-larger preference margin. The loss itself is bounded below by zero; the optimizing margin is what tends toward infinity on separable comparisons (Gheshlaghi Azar et al. 2024).

IPO bypasses the pointwise Bradley-Terry reward assumption and regresses the policy/reference log-ratio gap to a finite target. With

$$h_\theta(x,y_w,y_l)= \log\frac{ \pi_\theta(y_w\mid x)\,\pi_{\mathrm{ref}}(y_l\mid x) }{ \pi_\theta(y_l\mid x)\,\pi_{\mathrm{ref}}(y_w\mid x) },$$

Here $h_\theta$ denotes the chosen-versus-rejected gap in policy/reference log-ratios. The sampled IPO objective is

$$\mathcal{L}_{\mathrm{IPO}}(\theta)= \mathbb{E}_{\mathcal{D}} \left[ \left(h_\theta(x,y_w,y_l)-\frac{1}{2\beta}\right)^2 \right].$$

Here this chapter uses $\beta$ for the paper's positive KL-regularization parameter $\tau$. The target is therefore fixed by the theory as $1/(2\beta)$, not chosen independently. IPO still needs paired responses and a frozen reference. Its source paper demonstrates the behavior mainly in illustrative bandit problems, so the derivation should not be read as evidence that IPO wins broadly on language-model tasks.

The runnable below isolates this difference. It optimizes one scalar gap rather than a language model, so it demonstrates loss geometry, not expected model quality.

import numpy as np
import matplotlib.pyplot as plt

def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-z))

beta = 0.2
learning_rate = 0.1
steps = 100
ipo_target = 1.0 / (2.0 * beta)

dpo_gap = [0.0]
ipo_gap = [0.0]
for _ in range(steps):
    # d[-log sigmoid(beta*h)]/dh = -beta*sigmoid(-beta*h)
    dpo_gradient = -beta * sigmoid(-beta * dpo_gap[-1])
    dpo_gap.append(dpo_gap[-1] - learning_rate * dpo_gradient)

    # d[(h - 1/(2*beta))**2]/dh = 2*(h - target)
    ipo_gradient = 2.0 * (ipo_gap[-1] - ipo_target)
    ipo_gap.append(ipo_gap[-1] - learning_rate * ipo_gradient)

print(f"IPO target: {ipo_target:.2f}")
print(f"DPO gap after {steps} steps: {dpo_gap[-1]:.2f}")
print(f"IPO gap after {steps} steps: {ipo_gap[-1]:.2f}")

plt.plot(dpo_gap, label="DPO gap")
plt.plot(ipo_gap, label="IPO gap")
plt.axhline(ipo_target, color="gray", linestyle="--", label="IPO target")
plt.xlabel("gradient step")
plt.ylabel("policy/reference log-ratio gap")
plt.legend()
plt.show()

KTO: replace pairs with desirable and undesirable examples

Kahneman-Tversky optimization (KTO) drops the paired-data requirement by learning from good/bad labels. Kahneman-Tversky Optimization changes the data requirement. Each example is a triple $(x,y,d)$, where $d$ says whether response $y$ is desirable or undesirable. A positive and a negative response for the same prompt are not required (Ethayarajh et al. 2024). This fits logs built from approvals, rejections, or thresholded ratings, although the meaning and calibration of those labels still matter.

KTO retains a reference model. It defines

$$r_\theta(x,y)=\log\frac{\pi_\theta(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)}, \qquad z_0(x)=\mathrm{KL}\!\left( \pi_\theta(\cdot\mid x)\,\|\,\pi_{\mathrm{ref}}(\cdot\mid x) \right),$$

Here $r_\theta$ represents the response's policy/reference log-ratio, while $z_0$ denotes the KL reference point. KTO then scores the example relative to that point:

$$v(x,y)= \begin{cases} \lambda_D\,\sigma\!\left(\beta(r_\theta(x,y)-z_0(x))\right), & d=\text{desirable},\\ \lambda_U\,\sigma\!\left(\beta(z_0(x)-r_\theta(x,y))\right), & d=\text{undesirable}. \end{cases}$$

Its loss is $\mathcal{L}_{\mathrm{KTO}}=\mathbb{E}[\lambda_d-v(x,y)]$, where $\lambda_D$ and $\lambda_U$ weight the desirable and undesirable classes. In practice, exact computation of $z_0$ is expensive. The paper uses a stopped-gradient, biased microbatch estimate constructed from mismatched prompts and responses. Class imbalance therefore affects both the weights and the quality of the baseline estimate.

The prospect-theory connection supplies an inductive bias: gains and losses are evaluated relative to a reference point with a saturating value function. It is not evidence that this formula is a faithful psychological model of how people judge text. KTO's practical advantage is narrower and useful: it can train from independently labeled responses without pretending that those labels form same-prompt pairs.

ORPO: combine chosen-response SFT with a preference penalty

Odds Ratio Preference Optimization (odds-ratio preference optimization (ORPO)) removes the frozen reference and the separate preference stage. It trains on pairs, uses the chosen response as an SFT target, and adds a term that separates chosen and rejected sequence scores (Hong et al. 2024). For a response of $m$ tokens, ORPO first defines the geometric-mean token probability

$$P_\theta(y\mid x)= \exp\!\left( \frac{1}{m}\sum_{t=1}^{m} \log P_\theta(y_t\mid x,y_{<t}) \right),$$

Here $P_\theta(y\mid x)$ represents the geometric mean of the token probabilities. Its odds are $\operatorname{odds}_\theta(y\mid x)=P_\theta(y\mid x)/(1-P_\theta(y\mid x))$. The objective is

$$\mathcal{L}_{\mathrm{ORPO}}= \mathbb{E}_{\mathcal{D}} \left[ \mathcal{L}_{\mathrm{SFT}}(x,y_w) +\lambda\mathcal{L}_{\mathrm{OR}}(x,y_w,y_l) \right],$$

Here $\mathcal{L}_{\mathrm{SFT}}$ denotes chosen-response negative log-likelihood and $\lambda$ sets the weight of the odds-ratio term, defined as

$$\mathcal{L}_{\mathrm{OR}}= -\log\sigma\!\left( \log\frac{ \operatorname{odds}_\theta(y_w\mid x) }{ \operatorname{odds}_\theta(y_l\mid x) } \right).$$

This formula means that minimizing $\mathcal{L}_{\mathrm{OR}}$ increases the chosen response's odds relative to the rejected response. $\lambda>0$ balances demonstration learning against preference separation. ORPO is one stage, not literally one pass over the data. It saves reference inference or cached reference scores, but it also couples SFT and preference learning through $\lambda$. Those two signals cannot be tuned or audited as independently as a staged SFT-then-DPO pipeline.

SimPO: use average log-probability and a target margin

Simple Preference Optimization (simple preference optimization (SimPO)) also removes the reference, but it retains a pure pairwise objective. Its response score is the policy's average token log-probability:

$$r_{\mathrm{SimPO}}(x,y)= \frac{\beta}{|y|}\log\pi_\theta(y\mid x) =\frac{\beta}{|y|}\sum_{t=1}^{|y|} \log\pi_\theta(y_t\mid x,y_{<t}).$$

Here $|y|$ denotes response length and $\beta$ scales the average log-probability. The training loss is

$$\mathcal{L}_{\mathrm{SimPO}}(\theta)= -\mathbb{E}_{\mathcal{D}} \log\sigma\!\left( \frac{\beta}{|y_w|}\log\pi_\theta(y_w\mid x) -\frac{\beta}{|y_l|}\log\pi_\theta(y_l\mid x) -\gamma \right),$$

where $\beta>0$ scales the score difference and $\gamma>0$ is the target margin (Meng et al. 2024). Unlike its role in the original KL objective, $\beta$ is not a KL coefficient here because SimPO has no reference-policy KL term.

The per-token average reduces sensitivity to response length when length and preference are correlated in the dataset. It does not eliminate all length bias, and the SimPO paper notes that DPO's policy/reference ratio can itself counteract some length effects. The original experiments also found that an overly large $\gamma$ could reduce generation quality. Reference-free does not mean regularization-free; initialization, learning rate, data coverage, and early stopping still limit drift.

What the comparison can establish

The original papers report gains in particular settings. KTO reports competitive results with unpaired labels, ORPO reports benefits from its joint objective, and SimPO reports wins over its DPO baselines on chat evaluations. These results use different initial models, datasets, hyperparameter searches, and evaluators. They do not form one common tournament.

The defensible conclusion is not that the losses are interchangeable. It is that there is no established universal winner. Compare them with the same starting checkpoint, data, optimization budget, decoding settings, and external evaluator. Report uncertainty across seeds when the decision matters.

Choose from data and pipeline constraints

The pipeline and data constraints often narrow the choice before benchmark scores do:

Use this table as a filter, not a leaderboard:

• Start with DPO when same-prompt pairs and a suitable reference are available. It is the clearest baseline and keeps SFT separate from preference learning.
• Consider IPO when separable comparisons drive margins upward and reference anchoring is still desirable. Validate the theory-derived target in the actual model regime.
• Consider KTO when feedback is genuinely pointwise. Audit label semantics, class balance, and the microbatch KL estimate before comparing quality with pairwise methods.
• Consider ORPO when a one-stage SFT-plus-preference recipe is operationally useful. Measure how $\lambda$ trades chosen imitation against rejection rather than assuming the two cooperate.
• Consider SimPO when avoiding reference inference and using a per-token score fit the pipeline. Tune both $\beta$ and $\gamma$, and measure drift because there is no explicit reference anchor.

Reference-free methods remove reference inference and storage overhead. They do not necessarily halve peak memory: optimizer states, activations, sharding, offload, and whether reference scores are precomputed determine the realized saving. DPO and IPO can compute frozen-reference log- probabilities before training and store two scalar scores per pair, trading disk and preprocessing for lower training-time memory and compute.

A practical DPO run

A reliable first run is deliberately plain. It establishes whether the preference data carries useful signal before adding a variant.

1. Freeze the starting point. Save the SFT or instruction checkpoint as $\pi_{\mathrm{ref}}$ and initialize $\pi_\theta$ from the same weights. Record tokenizer, chat template, truncation rules, and end-of-sequence handling.
2. Validate every triple. Require one prompt, one chosen response, and one rejected response. Remove exact duplicates, contradictory duplicate labels, empty responses, and pairs made identical by truncation. Split by prompt or source group to prevent near-duplicate leakage.
3. Inspect the preference signal. Measure response lengths, label sources, topic mixture, rater agreement where available, and the starting model's margins. A dataset in which the chosen answer is almost always longer can train style as a shortcut.
4. Compute sequence scores correctly. Mask prompt tokens and padding. Sum response-token log-probabilities under the policy and reference. Use the same tokenization and truncation for both models. Precompute reference scores only after these choices are frozen.
5. Run a small sweep. Vary learning rate, $\beta$, and epochs. Learning rate and stopping time can matter as much as the named objective. Keep the data order and evaluation protocol fixed across methods.
6. Evaluate checkpoints, not only the last step. Training preference accuracy can keep rising after held-out task quality peaks. Select with external evaluation defined before the sweep.

The minimal batch computation is

$$\begin{aligned} \ell_w &= \log\pi_\theta(y_w\mid x)-\log\pi_{\mathrm{ref}}(y_w\mid x),\\ \ell_l &= \log\pi_\theta(y_l\mid x)-\log\pi_{\mathrm{ref}}(y_l\mid x),\\ \mathcal{L}_{\mathrm{batch}} &= -\operatorname{mean}\!\left[ \log\sigma\!\left(\beta(\ell_w-\ell_l)\right) \right]. \end{aligned}$$

$\ell_w$ and $\ell_l$ are sums over response tokens only. Keeping them as named diagnostics makes several failures visible that a single loss curve would hide.

Failure modes and checks

Preference optimization is offline: it only constrains responses represented in the stored comparisons. As the policy changes, it can enter regions where those labels provide little coverage. No loss variant can infer missing behavior requirements from absent data.

Track at least the following on a held-out split and on generated responses:

• Measure held-out preference accuracy and margin. These show whether the loss generalizes to unseen pairs, but they do not by themselves measure open-ended generation quality.
• Chosen-response log-probability and rejected-response log-probability. A widening relative margin can coexist with falling likelihood for both responses. That may be acceptable, but it should be observed rather than assumed away.
• Policy/reference divergence. Estimate token-level KL or policy/reference log-ratios on fresh generations. The configured $\beta$ is not a measurement of achieved drift.
• Response length and format compliance. Compare distributions, not just their means. Length, refusal rate, and formatting can become shortcuts for the preference label.
• Held-out task quality. Use human evaluation, verifiable tasks, or a judge protocol with known limitations. Include safety and capability regressions that are not represented by the training pairs.
• Slice results. Break out prompt source, language, difficulty, response-length gap, and label provenance. Aggregate wins can conceal a severe regression on one group.

If training accuracy approaches one while external quality falls, first shorten training or lower the learning rate. Then inspect data leakage, contradictory labels, length shortcuts, and policy drift. Switching objectives before diagnosing those causes can merely move the same failure to a different loss.

Static preference optimization is valuable precisely because it is simple. Its limit is the same simplicity: the model never asks for a new comparison while it changes. Iterated or online methods refresh candidates from the current policy and collect or generate new judgments, improving coverage at the cost of bringing sampling and feedback infrastructure back into the loop. That tradeoff connects this chapter to the online RLHF system in Chapter 19.

Further reading