Benchmarks as Measurement Contracts

A benchmark score is not a property of model weights alone. It is the result of running one model revision through one harness, on one version of a test set, with one scorer and aggregation rule. Change any of those inputs and the number may change. A useful report therefore begins with the claim the evaluation is meant to support, then preserves enough evidence for someone else to reconstruct that claim.

This chapter develops that measurement contract. It explains how to choose test cases, keep them separate from development, pin the execution harness, audit the answer key, and retain per-case records. The next chapter, Chapter 48, asks how much uncertainty remains after those choices have been made.

Begin with the decision, not the dataset

Suppose a team asks, “Is the new model better?” No benchmark can answer that question until better is made specific. Better at answering whose questions, under what conditions, at what cost, and for which decision? A multiple-choice exam can measure performance on its questions. It cannot, without further evidence, establish general intelligence, production usefulness, or safety. Raji et al. call out this gap between a benchmark's narrow task and the broad construct it is often made to represent (Raji et al. 2021).

Write the intended claim before choosing the test:

For English-language support requests from the last quarter, using the production prompt and retrieval stack, model B resolves more cases than model A without regressing policy compliance or exceeding the latency budget.

That sentence identifies a population, a system boundary, a comparison, and guardrails. It also reveals that no single public benchmark can settle the decision. Public suites remain useful for external comparison and regression testing, but product claims need samples that resemble the product. The evidence-centered benchmark design framework makes this order explicit: define the capability and intended use, justify the content, document how the model is adapted or prompted, assemble enough representative items, and explain how item-level evidence becomes a test-level conclusion (Liu et al. 2024).

The common benchmark families are implementations of different measurement choices, not interchangeable measures of one capability:

Breadth helps only when the suite's composition matches the claim. The “benchmark lottery” describes how conclusions and rankings can change with the chosen tasks, datasets, and metrics (Dehghani et al. 2021). Adding unrelated tests can make a dashboard larger while making its meaning less clear.

The decision also chooses the metric

Naming the decision settles more than which dataset to run. It settles what to score, and the two decisions a benchmark usually serves want opposite properties. A release gate wants the quantity the product is judged on, which is usually accuracy on parsed final answers. A development decision, such as which data mixture or which of two training runs to keep, is made on small models and partly trained checkpoints, where accuracy is a coarse instrument: a multiple-choice score can sit near chance while the model is in fact improving, and it moves between adjacent checkpoints of the same run for no reason connected to model quality.

Heineman et al. turn that observation into two measurable properties of a benchmark paired with a metric. Signal is how well the pair separates stronger models from weaker ones. Noise is how much it moves between training steps within a single run. Write $s_m$ for the score of model $m$ across a set of models, $\sigma_{\mathrm{models}}$ for the standard deviation of $\{s_m\}$, and $\sigma_{\mathrm{step}}$ for the standard deviation of the score over the final checkpoints of one run. Signal is $\sigma_{\mathrm{models}}$, noise is $\sigma_{\mathrm{step}}$, and their ratio

$$\mathrm{SNR}=\frac{\sigma_{\mathrm{models}}}{\sigma_{\mathrm{step}}}$$

is what decides whether an observed gap carries information. That ratio is the shape of the comparison rather than the paper's own estimator. Across 30 benchmarks and 375 open-weight models from 60M to 32B parameters, they report that pairs with a higher ratio support more reliable comparisons at small scale and produce better scaling-law fits. Three interventions follow from the measurement: prefer a metric with a better ratio, which for many benchmarks means per-token loss over the answer string rather than accuracy on it; drop subtasks that contribute noise to an aggregate; and average adjacent checkpoints to reduce $\sigma_{\mathrm{step}}$ directly (Heineman et al. 2025).

Loss is a development instrument and not a release metric, so this does not collapse the two decisions into one. A checkpoint that assigns higher probability to the reference answer can still emit it in a form the parser rejects, and no product claim rests on per-token loss. The two metrics answer different questions on the same items, and a report that gives only one of them has answered the other question by omission. Chapter 48 supplies the interval machinery for saying when a gap on either metric is large enough to act on.

Specify the evaluation object

An evaluation should be a versioned artifact, not a command remembered from a terminal. At minimum, its specification pins the data, model, harness, and scorer:

EvaluationSpec:
  benchmark_id, benchmark_revision, item_manifest_hash
  model_id, model_revision, tokenizer_revision
  prompt_template_revision, system_policy_revision
  few_shot_policy, tool_environment_revision
  decoding_parameters, seeds, repetitions
  scorer_revision, normalization, exclusion_policy
  contamination_audit_revision, run_environment

EvaluationCase:
  case_id, source, license, created_at, language, domain
  input_hash, reference_hash, rubric_revision
  slice_labels, dependencies, leakage_status

EvaluationRecord:
  spec_hash, case_id, attempt, raw_output_hash
  parsed_output, score, scorer_trace
  latency, token_cost, tool_cost

The separation matters. EvaluationSpec says how the run behaves. EvaluationCase documents what an item represents. EvaluationRecord preserves what happened. An aggregate score can be recomputed from those records; it cannot recreate them after they have been discarded. Dataset documentation has the same purpose at a broader scale: it makes motivation, composition, collection, recommended uses, and limitations inspectable rather than implicit (Gebru et al. 2021).

Let $y_i$ be the score on case $i$, let $w_i \ge 0$ be its predeclared weight, and let $n$ be the number of cases. The reported weighted mean is

$$\widehat{\mu} = \frac{\sum_{i=1}^{n} w_i y_i}{\sum_{i=1}^{n} w_i}.$$

Here, $i$ identifies one evaluation case, $y_i$ may be binary correctness or a numeric rubric score, $w_i$ controls how much that case contributes, and $\widehat{\mu}$ is the estimated mean for the population implied by those weights. Equal weights do not mean “no assumption”; they assume every sampled case should contribute equally.

For a comparison on the same cases, retain $d_i = y_i^{(B)} - y_i^{(A)}$, the case-level difference between systems B and A, and aggregate the $d_i$ values. This paired record shows whether B fixed A's failures or traded them for different ones. If a stochastic code or agent task is attempted repeatedly, average or model repetitions within the task before treating tasks as independent units. Twenty attempts on one repository are not twenty independent repositories. Chapter 48 develops the appropriate intervals and tests.

Figure 47.1 shows the dependency chain. A score is the final projection of several versioned decisions, not the first-class evidence.

Held out from what?

A held-out set is data deliberately kept out of every training stage; it is a pipeline contract, not a property a file can prove by itself. In a closed training pipeline, the contract can cover pre-training, supervised tuning, preference data, synthetic-data generation, prompt selection, and checkpoint selection. For an external model trained on undisclosed web data, an evaluator usually cannot establish that a public test was never seen. The status to record is “exposure unknown,” not “clean.”

Several different failures are often compressed into the word contamination:

1. Training leakage: an input, answer, or close duplicate enters pre-training or later training data.
2. Indirect leakage: synthetic examples inherit benchmark content from a teacher or retrieval source.
3. Development leakage: prompts, parsers, checkpoints, or policies are repeatedly chosen against evaluation results.
4. Reporting leakage: items are filtered or reinterpreted after outputs are visible, without the exclusions appearing in the report.

The third failure is ordinary adaptive overfitting. A test set stops acting like fresh evidence when developers query it, change the system, and query it again. The reusable-holdout literature formalized this problem for adaptive data analysis: repeated access needs controlled disclosure or fresh data if validity is to survive (Dwork et al. 2015). A private final set, accessible only to a release service, can preserve one last independent check. It is not permanently trustworthy; access history and prior decisions determine how much independence remains.

Audits provide evidence about exposure, not proof of absence. Exact hashes and n-gram matching find exact or near duplicates when the training corpus is available. Canaries reveal ingestion when a model reproduces a unique planted string. membership inference, tests that ask whether a specific example was in the training data, can probe weights without access to the corpus. Min-K% Prob, for example, uses unusually high probabilities on a text's least likely tokens as a membership signal (Shi et al. 2024). Its ICLR study evaluates a detection method under particular data and model conditions; a negative result does not certify that an item was unseen.

Record the audit's scope in the status itself: known_overlap, suspected_overlap, no_overlap_found_under_audit_X, or not_auditable. Calling a case clean after one detector returns a negative result discards the most important qualification.

There is no universally safest publication policy. Each choice protects one property by giving up another:

LiveBench is one example of rotating questions with automatic grading (White et al. 2025). Its design reduces some exposure risks; it does not make the questions representative, the scorer correct, or comparisons across changing revisions automatic. “Fresh” and “valid” answer different questions.

Audit the instrument

Even a genuinely unseen test can be wrong. Items may have incorrect keys, ambiguous wording, stale facts, broken dependencies, or a parser that rejects a valid answer. MMLU-Redux manually re-annotated 5,700 MMLU questions and estimated that 6.49% contained errors, with large variation among subjects (Gema et al. 2025). That result is a warning against treating the answer key as ground truth by definition.

Before using a suite as a release gate, audit a sample of cases and every case that changes the decision. Check:

• whether the prompt has one defensible interpretation;
• whether the reference answer and rationale agree;
• whether alternative valid forms survive normalization and parsing;
• whether duplicates or shared sources make cases dependent;
• whether dates, tools, packages, websites, and repositories are still valid;
• whether language, domain, difficulty, and user slices cover the intended population;
• whether licenses and access terms allow the proposed use; and
• whether exclusions were declared before results were examined.

Keep an adjudication log. Correcting an item creates a new benchmark revision; silently changing the key under the old name destroys comparability. Also run simple baselines: random choice where applicable, a trivial lexical heuristic, the previous production system, and an oracle or qualified human estimate when one is meaningful. A sophisticated model losing to a trivial heuristic often reveals a shortcut in the test rather than a subtle model failure.

The harness is part of the measured system

The same weights can produce different scores under different prompt templates, few-shot examples, chat templates, decoding settings, tool versions, and answer parsers. Standardized frameworks such as Holistic Evaluation of Language Models make these choices explicit and evaluate more than accuracy, improving comparison and auditability (Liang et al. 2023). Standardization does not remove the harness; it fixes one so that participants are measured under the same declared conditions.

That creates two legitimate but different experiments:

• A controlled model comparison fixes the harness as far as possible and asks how model revisions differ under the same interface.
• A best-system comparison permits model-specific prompting or tools and asks how complete deployed systems differ under a shared budget and task contract.

Do not label the second as a comparison of weights. Conversely, forcing every model through a prompt format poorly matched to its training may answer a clean but operationally unhelpful question. State which experiment is being run.

The scorer needs the same discipline. Preserve raw output before parsing, record the parser and scorer revision, and distinguish invalid, unknown, timeout, and scorer_error from an ordinary wrong answer. A model-based judge adds its own model revision, prompt, order effects, and stochasticity; Chapter 50 develops that case. A test runner adds container images, dependencies, clocks, network state, and flaky tests. “Automatically graded” does not mean “interpretation-free.”

Benchmarks decay

A fixed benchmark loses decision value through several independent processes:

• Exposure: cases or answers enter training and development loops.
• Adaptive reuse: repeated result-driven changes overfit the test even without direct training leakage.
• Saturation: scores cluster near the ceiling, leaving too little variation to separate systems.
• Population drift: the users, tasks, languages, or threats relevant to the decision change.
• Scorer drift: a formerly valid answer key, package, website, policy, or judge no longer represents the desired behavior.

Figure 47.2 illustrates saturation. The curves are schematic: the point is not that every benchmark follows the same timeline, but that a bounded test loses resolution when most systems succeed on almost every item.

Do not respond by adding harder questions blindly. Difficulty is useful only if the new cases still represent the construct and decision. Maintain a benchmark like a production dependency: record revisions, monitor item statistics and coverage, retire broken cases, add newly relevant slices, and preserve a stable anchor set when longitudinal comparison matters.

Run, retain, and report

A defensible benchmark run can be implemented as a short procedure:

1. State the construct, population, intended use, primary metric, and guardrails.
2. Freeze the model, harness, scorer, item manifest, weights, exclusions, seeds, and repetition policy in an EvaluationSpec.
3. Audit provenance, rights, answer quality, dependencies, and exposure risk.
4. Run baselines and candidate systems under the declared comparison design.
5. Store raw outputs, parsed outputs, scorer traces, costs, and failures for every attempt.
6. Aggregate at the correct sampling unit; report paired changes, slices, and uncertainty.
7. Stress the instrument with format changes, option-order permutations, paraphrases, and environmental reruns where those should preserve meaning.
8. Apply only predeclared release rules, and log every exception or post-hoc analysis.

Four invariants make the procedure auditable: no evaluation case enters a training or tuning path; no item disappears silently; raw output remains immutable; and every reported number resolves to a spec hash and case-level records. These rules do not guarantee a valid benchmark. They make errors visible and repairable.

A report should therefore lead with the qualified claim, not the largest score. Name the benchmark revision, model and system boundary, harness, sample and weights, scorer, known leakage and item defects, uncertainty, relevant slices, and the decision the evidence supports. A leaderboard row omits most of this context. It can point to a result record; it should not replace one.

With the measurement object defined, the next question is how stable its aggregate is. Chapter 48 turns the saved case-level records into intervals, paired comparisons, and release decisions without mistaking precision for validity.

Further reading