Judging and Holistic Evaluation

The benchmarks in Chapter 47 score what has a checkable answer. Open-ended quality, helpfulness, tone, whether one response is better than another, has no answer key, so the field reaches for a strong model as the grader and for human preference as the ground truth. That choice creates three linked problems: a model-as-judge, a model used as the grader, is biased and has to be corrected, arena-style preference ranking turns pairwise comparison votes, head-to-head choices between two responses, into a single number, and a private test set is worth more than any public leaderboard. The human side of that ground truth was designed in Chapter 49; this chapter studies what happens when that design is compressed into a reusable judge.

When the answer key runs out

Most of what a deployed model produces cannot be checked by a test runner. A summary, an explanation, a rewrite, a chat turn: each has many acceptable forms and no reference string to match against. Exact-match and execution graders, the workhorses of Chapter 47, are silent here. The early answer borrowed reference-overlap metrics from machine translation, scoring an output by how much it shared with a human-written reference, but on open-ended text that overlap correlates poorly with judgment, and there is rarely a single reference to overlap with in the first place. That leaves two instruments, both expensive: a human reading every output, or a strong model standing in for that human.

Both are biased, and the bias is not random noise that averages out. It is structured, it correlates with the very properties a training run will optimize, and so it bakes itself into the model if you grade against it without correction. The instrument that grades open-ended output is, when frozen and reused, the instrument that trains the model, so a flaw in measurement becomes a flaw in behavior.

A second problem compounds the first. Any public benchmark, the moment it is published, begins leaking into training corpora and becomes a target to tune toward (Shi et al. 2023). A number that everyone can see is a number everyone optimizes, and Goodhart's law takes it from there: the measure stops measuring. So the question is not only how to grade open-ended output, but how to hold a measurement that stays trustworthy after the field has seen it.

The answer uses three instruments built on top of each other: a model that grades, a ranking that aggregates many grades, and a held-out set that keeps the whole apparatus trustworthy. Holistic evaluation in the sense of Holistic Evaluation of Language Models, scoring models across many scenarios and metrics at once rather than chasing a single headline number, is the frame these three sit inside (Liang et al. 2022).

The grader is the thing it grades

The core idea is to use a strong language model as the grader, the model-as-judge, and to treat its output as a proxy validated against human or executable ground truth, never as ground truth itself. The judge is cheap, fast, and scalable in a way a human panel is not, which is what makes it usable inside a training loop. It became viable once frontier models were strong enough that their preferences tracked human preferences closely. Zheng et al. made the case directly: a strong judge agrees with human raters at roughly the rate two humans agree with each other, which is the bar that makes the judge usable as a stand-in (Zheng et al. 2023).

That agreement rate is the optimistic half of the story. The pessimistic half is that the judge is a biased instrument, and the design is mostly about characterizing and correcting that bias. Worse, the bias does not stay confined to measurement. The constraint arrow below traces why: the same pairwise protocol that scores a leaderboard also gathers the preference data that trains a reward model in Chapter 19, and a reward model is a frozen model-as-judge. A verbosity-biased judge that inflates a leaderboard becomes, distilled into a reward model, a verbosity-biased reward that shapes behavior. The instrument grades and trains, so getting its bias right matters twice.

Four biases and their countermeasures

Four biases recur, and each maps to a property a model can learn to exploit (Zheng et al. 2023):

• Verbosity. Longer answers score higher, independent of whether the extra length adds anything. A judge optimized against naively rewards padding. Figure 50.1 makes the trap concrete: with content quality held fixed, a fair judge would report a flat coin-flip win rate, yet the biased judge's win rate climbs with length, and the whole gap is bias rather than merit. Length-Controlled AlpacaEval made this bias operationally measurable by asking the counterfactual question: what would the preference be if the model and baseline answers had the same length (Dubois et al. 2024)? The arena leaderboard now ships the same correction: style control adds length and markdown terms to the Bradley-Terry fit, and rankings shift when it is turned on (Li et al. 2024).
• Position. When the judge compares two answers labeled A and B, the order changes the verdict. The same pair, swapped, can flip the winner (Wang et al. 2023).
• Self-preference. A judge favors outputs from its own model family, so grading a model with a sibling judge inflates its score.
• Format sensitivity. Markdown, headers, a confident closing line: surface form moves the score apart from substance.

The mitigations attack each bias directly. Position bias yields to symmetry: swap the order and average the two verdicts, or require agreement across both orderings before calling a winner. Wang et al. quantified how far a single swap can move a verdict, which is what makes the swap-and-average protocol standard (Wang et al. 2023). Verbosity calls for length normalization, or a judge told to discount it. Self-preference is handled upstream of the verdict, by drawing the judge from a different family than the model under test, or by using an ensemble. Across all of them, prefer pairwise comparison over an absolute 1-to-10 score: humans and judges both rank two things more reliably than they place one thing on an abstract scale, and a pairwise protocol sidesteps the question of what a "7" means. Anchor the judge with an explicit rubric rather than a vague "which is better," and calibrate the whole apparatus against human labels on a sample. Figure 50.2 lines up each bias with the property a model can exploit and the mitigation that closes it.

The length example is worth keeping separate because it shows what "debias" has to mean. It is not enough to tell the judge "do not prefer long answers." The metric has to estimate the verdict under a controlled condition, or at least report the residual dependence of the verdict on length. That same logic generalizes to the other biases: a corrected judge is one whose remaining dependence on position, family, and surface format has been measured, not one whose prompt says it is impartial.

Two refinements come from practice (Anthropic 2026). Give the judge an escape hatch: let it return "Unknown" when it lacks the evidence to decide, rather than forcing it to manufacture a justification it does not have. And grade each rubric dimension in isolation, with its own judge call, rather than asking one call to score correctness, tone, and completeness at once, since a single call lets a strong impression on one axis bleed into the others.

From one verdict to a ranking

A single pairwise verdict tells you which of two responses won on one prompt. Arena-style preference ranking scales that to a global ordering over many models. Collect many pairwise comparisons, each a vote that response A beat response B on the same prompt, then fit a rating that explains the votes. The fitting model is the one chess uses, an Elo or Bradley-Terry latent strength per model (a hidden skill number inferred from each model's win and loss record) (Elo 1978; Bradley and Terry 1952), so that the probability model $i$ beats model $j$ is

$$P(i \succ j) = \frac{1}{1 + 10^{(r_j - r_i)/400}}$$

with $r_i$ the latent rating. Figure 50.3 plots this curve, which is S-shaped because it swings fastest when two ratings are close and flattens toward a near-certain win as the gap widens: an equal-rating pair is a coin flip, and the 400-point scale fixes the slope so a 400-point gap predicts roughly a 91 percent win. Maximum-likelihood fitting, which picks the ratings that best explain the observed votes, runs over all recorded pairwise outcomes and recovers a single ranking from a sparse, noisy set of head-to-head votes.

The votes can come from humans or from a model-as-judge, and the same machinery serves both. Chatbot Arena took this to open scale: a public platform collecting human pairwise votes in the wild and publishing a live Elo leaderboard, which became the field's default answer to "which model do people actually prefer," because its inputs are fresh and resistant to answer-key contamination (Chiang et al. 2024). MT-Bench, by contrast, replaces the crowd with a model judge, though its standard protocol has the judge grade each answer on an absolute ten-point scale rather than fit a ranking from pairwise votes (Zheng et al. 2023).

The number nobody has seen

The third instrument keeps the other two trustworthy. A private test set is a held-out set you own, that has never appeared in any training mixture, version-pinned, access-controlled, rotated the moment it leaks, and large enough to give a usable signal. A public benchmark you must assume is contaminated over time, and the tooling to detect that contamination after the fact exists precisely because it is so common (Shi et al. 2023). A private held-out set is trustworthy only while its integrity holds, and that integrity is an operational property you maintain, not a one-time guarantee.

The historical arc runs away from the public leaderboard, against the direction the field's incentives push. As each published benchmark saturated and contaminated, practitioners learned that the number that survives is the one nobody outside the team has seen. The lesson generalizes the held-out principle from a single train/test split to a discipline: the trustworthy measurement is the fresh, owned, rotated one.

Choosing where to stand

The three instruments leave several axes on which a team must pick a point, and each choice trades something concrete:

• Judge cost versus reliability. A stronger judge is more reliable per sample and more expensive per sample; a cheaper judge needs more samples and more aggressive bias correction. Pairwise plus position-swap roughly doubles the judge calls but materially improves reliability. Choose the point on this curve by what the verdict gates: a release decision warrants the expensive protocol, a quick iteration loop may not.
• Judge versus human. The judge scales and reproduces; the human is the ground truth that the judge is validated against, but is slow, costly, and itself noisy enough that two annotators disagree (Zheng et al. 2023). The working compromise is to gate continuously on the judge and calibrate it periodically against a human sample, never letting the judge drift unanchored.
• Pairwise versus absolute. Pairwise comparison is more reliable and sidesteps the meaning of an absolute score, but it gives a ranking, not a calibrated number, and the number of comparisons grows with the number of models. Absolute scoring is cheaper to aggregate but more exposed to every bias above.
• Public versus private sets. A public benchmark is cheap, comparable across labs, and reproducible, but saturates and contaminates. A private set resists gaming and stays trustworthy but costs effort to build and maintain, cannot be cited for cross-lab comparison, and decays the instant it leaks.

Building the judge

A judge call is a prompt-engineering problem with a scoring contract on top. The pairwise template presents the prompt and two responses, fixes the rubric, asks for a per-dimension verdict and a winner, and is run twice with the responses swapped; a wrapper averages the two and records both raw verdicts for audit. Keep the rubric explicit and the dimensions separate, and pin the judge model version, since a judge upgrade silently rebases every score and breaks comparability with past runs exactly the way a benchmark version change does.

The ranking fit is small. Given a table of pairwise outcomes, fit a Bradley-Terry model by maximum likelihood, which picks the ratings that best explain the observed votes and is a few lines of logistic regression over the win and loss indicators (Bradley and Terry 1952), and report the rating with a bootstrap confidence interval (resample the recorded votes many times to see how much the rating wobbles). A rating with no interval is folklore: with a sparse vote matrix the uncertainty on a single model's rank can span several places, and two models inside each other's intervals are not separated by the data. The same statistical discipline introduced in Chapter 48 applies here, just with votes rather than exact-match items.

Run this to watch the fit recover four hidden ratings from nothing but simulated head-to-head votes, and confirm the 400-point gap gives the roughly 91 percent win rate the curve promises.

import numpy as np
import matplotlib.pyplot as plt
rng = np.random.default_rng(0)
true_r = np.array([1500., 1400., 1300., 1200.])    # hidden latent ratings
p_win = lambda a, b: 1/(1 + 10**((b - a)/400))     # Bradley-Terry / Elo curve
n, games = len(true_r), 300
wins = np.array([[0 if i == j else rng.binomial(games, p_win(true_r[i], true_r[j]))
                  for j in range(n)] for i in range(n)], float)
r, s = np.zeros(n), 400/np.log(10)                 # MLE: ratings that best explain the votes
for _ in range(500):
    P = 1/(1 + np.exp(np.clip((r[None, :] - r[:, None])/s, -30, 30)))
    r += 0.5*((wins*(1 - P)).sum(1) - (wins.T*P).sum(1)); r -= r.mean()
r += true_r.mean()
print("400-point gap win prob:", round(p_win(400, 0), 3))
print("recovered:", np.round(r).astype(int), " true:", true_r.astype(int))
plt.bar(np.arange(n) - .2, true_r, .4, label="true")
plt.bar(np.arange(n) + .2, r, .4, label="recovered from votes")
plt.ylim(1100, 1560); plt.ylabel("rating"); plt.xlabel("model"); plt.legend(); plt.show()

The private set is an asset with an operational contract. It must never enter a training mixture, so it carries canary strings that let you later detect if it leaked (Shi et al. 2023), it is version-pinned and access-controlled, and it is rotated when a leak is found. Build it early and from real failures, the bug tracker and the support queue, since twenty to fifty tasks mined from things the model actually got wrong are worth more than hundreds of synthetic ones, and an eval is harder to reverse-engineer the longer you wait to write it (Anthropic 2026). Whatever the score, do not take it at face value until someone has read the transcripts: a judge that quietly rejects valid answers, or a contaminated number, ships as real signal only because nobody looked.

Further reading