Benchmarks and Their Discontents

Evaluation is how you know any stage of the stack worked, the cross-cutting gate that data, architecture, pre-training, and post-training all answer to. Its oldest and most contested instrument is the static benchmark. The thread running through static evaluation is a single uncomfortable claim: a published benchmark score is never a fact about a model alone. It is a claim about a model and its measurement apparatus together, and most of the discipline of evaluation is the work of making that claim survive scrutiny. A held-out set set is data deliberately kept out of every training stage; it is a pipeline contract. Benchmark families saturate, train/test contamination silently inflates numbers, and a score counts as evidence only when the harness, data contract, and uncertainty are visible.

The anatomy of a wrong number

A training pipeline produces a new set of weights every time a stage runs, and the only reliable way to know a stage worked is an independent number that moved for a reason that survives scrutiny. A benchmark is meant to be that falsifier. The recurring failure is not the absence of a number but belief in a wrong one: a score contaminated by test data leaking into training, gamed by tuning toward a known format, or simply noise dressed up as signal. Each of these produces a number that looks excellent and does not transfer, and the damage is silent, because nothing in the score itself announces that it is wrong.

No single layer of evaluation catches all three failures. A practical pipeline treats automated benchmarks, production monitoring, A/B tests, user feedback, and human studies as a Swiss-cheese model stack, where each layer catches what the others let through. The first layer here is the static benchmark: the cheapest, the most comparable across labs, and the most prone to quietly lying.

Figure 47.1 shows why no single layer is enough: each of the three failures slips past whatever layer is blind to it, and only the full stack catches all of them.

These three failures, contamination, gaming, and noise, are the spine of the argument. The analysis below builds the static benchmark up from what it measures, exposes where each failure enters, and ends with the discipline that keeps a number trustworthy in spite of them.

One more failure sits outside that spine because it lives in the instrument itself: the answer key can be wrong. FutureHouse's 2025 audit of Humanity's Last Exam found roughly 29% of its text-only chemistry and biology answers contradicted by peer-reviewed literature; the maintainers' own review estimated 18% and moved the benchmark to a rolling expert revision (White et al. 2025; Zhai and others 2026), and MMLU carries an estimated 6.5% label-error rate of its own (Gema et al. 2024). Label error does not flatter a result the way contamination does. It caps the score a correct model can reach, and nothing in the number announces it either, so auditing the instrument belongs to the same discipline as auditing the pipeline.

What a benchmark actually measures

The capability surface is too large to measure with one test, so benchmarks decompose it into families, each probing a different axis and each failing in a characteristic way.

• Knowledge. Broad factual and academic recall, in the style of MMLU (Hendrycks et al. 2020). It saturates and contaminates easily, and frontier high-water marks now sit near the ceiling, which is why harder expert-level successors such as Humanity's Last Exam move toward broad, closed-ended questions at the edge of human academic knowledge (Phan et al. 2025).
• Reasoning. Multi-step inference beyond recall. It is harder to contaminate but harder to score objectively, which pushes the field toward graduate-level, google-proof variants like GPQA (Rein et al. 2023) and abstraction-heavy successors such as ARC-AGI-2 (Chollet et al. 2025); when frontier systems closed in on that too, the family moved to the interactive game environments of ARC-AGI-3, dropping the static answer key altogether (ARC Prize Foundation 2026).
• Math. Grade-school through competition, from GSM8K (Cobbe et al. 2021) to MATH (Hendrycks et al. 2021). The answer is checkable, which is exactly the property that bridges evaluation to verifiable-reward reinforcement learning in Chapter 28.
• Code. Function completion and repository-level repair, from HumanEval (Chen et al. 2021) to SWE-bench-style tasks (Jimenez et al. 2023), now usually reported on the human-validated SWE-bench Verified subset (OpenAI 2024). It is executable, so the judge is a test runner rather than a model.
• Multilingual. The same task across languages. It exposes tokenizer and data-mixture decisions made in Chapter 6 that were invisible while the model was measured only in English.
• Long-context. Retrieval and reasoning over long inputs, from needle-in-a-haystack to harder probes like RULER (Hsieh et al. 2024). It gates the long-context extension discussed in Chapter 35.
• Multimodal. Joint perception and reasoning over diagrams, charts, tables, images, and text. MMMU made this explicit by testing college-level subject knowledge across heterogeneous visual formats rather than treating vision as captioning alone (Yue et al. 2023).
• Agentic. Tool use, multi-step task completion, and web or operating-system interaction, named by benchmarks like GAIA (Mialon et al. 2023), WebArena (Zhou et al. 2023), and τ-bench (Yao et al. 2024). These do not reduce to a single static answer, and their evaluation method is the subject of Chapter 52.

Each family carries its own characteristic failure, but all of them rest on the same precondition, and it is that precondition, not the family taxonomy, that decides whether any of these numbers mean anything.

The held-out contract

Underneath every family sits the same precondition: a held-out set whose answers the model has never seen during training. Held-out is not a property of a file, it is a property of the whole pipeline. A set is held-out only if it appears in no training mixture, which is a contract the data layer must honor, not a checkbox the evaluation layer can tick on its own.

Contamination is the violation of that contract, the first of the three failures and the most consequential, because it is silent and it inflates in exactly the direction that flatters a result. Since it announces nothing, it has to be hunted rather than waited for. Detection works on overlap and membership inference, tests that ask whether a specific example was in the training data. Exact and near-duplicate matching, typically n-gram overlap against the training corpus, catches verbatim and lightly-edited leakage. Embedded canary strings take a different route: unique markers planted in a held-out set, they betray that the set was ingested the moment a model reproduces them. Membership-inference signals go further still, asking whether a specific example was likely trained on at all, for instance Min-K% Prob, which flags an example when its least-probable tokens are suspiciously high-probability under the model (Shi et al. 2023). SWE-bench makes the risk concrete: state-of-the-art models can name the buggy file from the issue text alone, with no repository to look at, up to 76% of the time, and the same models drop sharply on repositories the benchmark never touched, memorization posing as repair capability (Liang et al. 2025). None of these is conclusive alone, which is why contamination is reported as a contract and an audit rather than a single test.

Figure 47.2 traces the held-out contract end to end: the two paths by which a benchmark reaches the weights, and the three detection signals that audit whether it did. The second path, a teacher model that was itself trained on the benchmark, bypasses the corpus entirely and returns in the final section.

Why benchmarks decay

A held-out set is only useful while it stays both held-out and hard. The history of benchmarks is a history of saturation, the slow erosion of the second property. A benchmark discriminates between systems only while models fail some of it; once a family clusters near its ceiling, the remaining headroom is noise, the third failure, and the benchmark stops telling systems apart.

Knowledge benchmarks reached this first. MMLU was a meaningful spread when it appeared (Hendrycks et al. 2020) and is now near ceiling for frontier models, which is why the field moved to deliberately harder, contamination-resistant variants. GPQA was built to be google-proof, with questions that experts outside the field cannot answer even with web access, precisely so that recall and retrieval cannot substitute for reasoning (Rein et al. 2023). That defense held only a few years: by 2026 GPQA Diamond had followed MMLU to the ceiling, with frontier models clustered above 90% (Vals AI 2026), and the discriminating rung moved on again, to Humanity's Last Exam and whatever follows it.

Figure 47.3 shows the shape of saturation. As successive model generations climb toward the ceiling, the spread between systems collapses into noise, which is why a harder successor benchmark has to be introduced to restore discrimination, only to inherit the same fate.

Saturation pushes the design in two directions at once, and the choice between them is the central trade-off of static evaluation. One direction is toward harder static sets, which buys time but inherits the same fate: any fixed set is a target, and a target eventually gets hit, whether by genuine capability or by contamination. The other is toward living evaluation that resists a fixed target altogether. Chatbot Arena replaces a frozen test set with a stream of fresh human-preference comparisons, so there is no static answer key to leak and the measurement renews itself as the model population changes (Chiang et al. 2024). Removing the answer key does not remove the gaming surface, though; Chapter 50 returns to what replaced it. The arc runs from a fixed academic exam, to a harder google-proof exam, to a moving evaluation with no answer key to memorize.

LiveBench occupies the middle ground between these poles. It tries to keep the objectivity of automatic grading while refreshing questions from recent math competitions, papers, news, and data sources, adding new tasks over time so the target is not a single frozen file (White et al. 2024). That design is a useful boundary marker for this chapter: if the task still has an objective answer, a moving benchmark can postpone contamination without handing the whole measurement to a human or model judge. When the target is preference, grounding, or a multi-step environment state, the later chapters of Part VII need different instruments.

Two trade-offs fall out of that arc, and a published number is only as meaningful as the reader's awareness of which side of each it came from.

• Static versus dynamic. Static benchmarks are cheap to run, reproducible to the bit, and comparable across labs and over time, which is why leaderboards are built on them. They also saturate, contaminate, and miss anything that is not a single static answer. Dynamic evaluation, meaning rotating sets and live environments, resists gaming because there is no fixed target to overfit, but costs more to run and is harder to compare across time because the test itself is moving.
• Public versus private held-out. A public benchmark should be assumed contaminated over time, since its answer key is on the internet and flows into the next training run by default. A private held-out set is trustworthy, but only while its integrity holds, and integrity is a wasting asset: every leak, every accidental inclusion, every published example erodes it. The right posture is to trust a private set until it leaks and to assume a public set already has.

Those trade-offs hand the reader to the rest of the part. Chapter 48 asks whether an observed benchmark gap is large enough to believe, Chapter 49 examines who defined the label when the answer key runs out, Chapter 51 separates fluent preference from grounded truth, and Chapter 53 turns held-out sets into release gates rather than leaderboard decorations.

Model, or harness?

Contamination and saturation both attack the test. The third failure attacks the act of measuring itself, and it is where the central claim becomes sharpest: even a clean, unsaturated benchmark does not produce a number until a harness wraps it, and the harness moves the number.

Keeping a number trustworthy

Each of the three failures has a discipline that answers it, and operationalizing held-out integrity is where those disciplines become procedure rather than principle. A set earns the word held-out only if it is never placed in any training mixture, version-pinned so a score names the exact set it was measured against, access-controlled so it does not leak through carelessness, rotated when a leak is detected, and large enough to give a usable signal rather than a single fragile number. This is the asset the evaluation layer owns and defends, and it is the answer to noise as much as to contamination, since a set too small to be stable is itself a source of false signal.

Figure 47.5 makes the size requirement concrete. The confidence interval on an accuracy estimate (a range that likely contains the true score) shrinks only as the square root of the held-out set size, so a small set carries an error bar wide enough to swamp the gap between two systems, which is precisely how noise masquerades as signal.

Try editing p1 and p2 to two accuracies you want to tell apart, and watch how many held-out examples it takes before the error bar fits inside the gap.

import numpy as np
import matplotlib.pyplot as plt

p1, p2 = 0.70, 0.72          # two systems' true accuracies
gap = abs(p2 - p1)
N = np.arange(100, 30001, 100)
p = (p1 + p2) / 2            # pooled accuracy for the standard error
half = 1.96 * np.sqrt(p * (1 - p) / N)   # 95% CI half-width

crossover = N[np.argmax(half < gap)]     # first N where the error bar fits in the gap
print(f"gap between systems: {gap:.3f}")
print(f"held-out size where 95% error bar < gap: {crossover} examples")

plt.plot(N, half, label="95% CI half-width")
plt.axhline(gap, color="r", ls="--", label=f"true gap = {gap:.2f}")
plt.xlabel("held-out set size N"); plt.ylabel("accuracy +/- half-width")
plt.legend(); plt.title("Below the crossover, noise swamps the signal")
plt.show()

The evaluation layer does not, however, scrub the training corpus. The build-time removal of test data, the deduplication and overlap filtering, belongs to the data pipeline. The evaluation layer supplies the held-out sets and the canaries, consumes the data layer's decontamination contract, and audits that the contract held. Auditing is the load-bearing word: the evaluation layer's job is to verify, after the fact, that nothing it defined as held-out made it into a mixture, and to surface it loudly when something did.

A subtler contamination path bypasses the corpus entirely, the indirect path Figure 47.2 drew earlier. Synthetic data generated by a teacher model that was itself trained on a benchmark carries that benchmark's content into the student, even though the student never saw the benchmark directly. This is one reason the synthetic-data practice in Chapter 23 and the held-out contract here are coupled: a clean corpus is not enough if the teacher was dirty.

All of this is why a single benchmark number, reported without context, is folklore rather than measurement. Contamination-aware reporting is the minimum discipline: state the decontamination contract version the score was produced under, and disclose any known leakage alongside the number, so a reader can tell a real result from a contaminated one before building on it.

Further reading