Statistical Reliability

A benchmark score is an estimate, not a fact. Once evaluation leaves toy examples, the question is rarely whether one model scored higher than another. The question is whether the observed gap is large enough, stable enough, and measured under a protocol clean enough to support the decision that will be made from it.

This chapter fills the measurement layer between Chapter 47 and Chapter 49. It treats accuracy, win rate, rubric scores, and production rates as random variables; separates uncertainty from bias; and shows why a leaderboard without intervals is an argument with its error bars hidden.

A Score Has a Sampling Distribution

A reported score is one draw from a sampling distribution: the spread of scores you would get by re-running the same evaluation on other comparable samples of items. Suppose an evaluation set has $n$ independent items and a model answers $x$ of them correctly. The reported accuracy is

$$\hat{p} = \frac{x}{n}.$$

Here $x$ is the number of correct items, $n$ is the evaluation-set size, and $\hat{p}$ is the observed accuracy. The hat matters: it marks the number as an estimate of the model's underlying success probability on comparable items, not as that probability itself.

The common normal approximation gives a 95% confidence half-width

$$h \approx 1.96 \sqrt{\frac{\hat{p}(1-\hat{p})}{n}}.$$

This small formula explains a large amount of leaderboard noise. Doubling the held-out set does not halve the error bar; it shrinks it by only $\sqrt{2}$. A 2 percentage point gap can look decisive on a large benchmark and vanish on a small private suite. Chapter 47 already used this argument for held-out size. Here the point is more general: every scalar number in an evaluation table is a statistic, and the statistic needs an uncertainty model before it can become evidence.

The approximation is only a starting point. It assumes independent Bernoulli trials (independent yes/no outcomes, one per test item), a fixed item distribution, and a binary scorer. Many LLM evaluations violate all three. Rubric scores are ordinal or continuous, a dialogue task carries correlated turns, and a production sample arrives clustered by user, tenant, geography, or time of day. The binomial equation should not be forced onto those cases. Better to make the sampling unit explicit, then resample or model at that unit. That is why the bootstrap remains useful: it estimates uncertainty by resampling observed units rather than deriving a new closed-form distribution for every metric (Efron 1979). Miller works the same discipline out for LLM evals directly: report standard errors, switch to clustered standard errors when items arrive in groups (on popular benchmarks they can be more than three times the naive ones), compare models by paired differences, and plan sample sizes in advance (Miller 2024).

Compare on the Same Items

Most model comparisons are paired. Model A and model B answer the same prompts, and the important items are not the ones both get right or both get wrong. The important items are the disagreements. If A and B each score 75%, they are not necessarily tied: they may succeed on the same examples, or on entirely disjoint ones. These cases have different engineering meanings.

For binary outcomes, write the paired disagreement table as

The off-diagonal terms $n_{10}$ and $n_{01}$ are the disagreement counts, and the evidence that A differs from B lives there. McNemar's test was built for exactly this setting (McNemar 1947). Its large-sample statistic is

$$\chi^2 = \frac{(n_{10}-n_{01})^2}{n_{10}+n_{01}}.$$

Here $n_{10}$ is the count of items A answered correctly and B missed, while $n_{01}$ is the reverse. The numerator measures the imbalance between those two disagreement directions; the denominator scales that imbalance by how many disagreements there were at all.

The formula is not the important thing to memorize. The important design rule is that an evaluation runner should save per-item outcomes, not just aggregate scores. Once the per-item table is gone, a team cannot tell whether a new model won by fixing old failures, won by trading one failure class for another, or merely won by noise. That loss matters most when a model is being swapped into a system with known failure modes. A model that improves the mean while regressing the tail may be worse for the product.

The same paired principle applies beyond accuracy. A judge preference table should keep the prompt id, order, judge version, and both raw verdicts. For a coding benchmark, each test failure matters, not just the pass rate. And a RAG evaluation can only be reconstructed later if it retained retrieval ids and claim-level judgments. Statistical reliability begins as data retention.

Many Looks Create False Wins

A modern evaluation suite is rarely one test. It is a matrix: many tasks, languages, judge variants, prompt templates, slices, and cost settings. If a team looks at enough cells, some cell will improve by chance. That is not a philosophical worry. It is the multiple-comparison problem. Benjamini and Hochberg introduced false-discovery-rate control, which caps the fraction of claimed wins that are really noise, as a way to manage this kind of situation without the bluntness of Bonferroni correction, which guards so hard against any single false win that it discards real effects along with the spurious ones (Benjamini and Hochberg 1995).

The practical translation is simple. A broad report should separate three claims:

• Primary metric. The one metric chosen before the run, allowed to gate the release.
• Guardrail metrics. Metrics that block regressions even if the primary metric improves.
• Exploratory slices. Slices used to find hypotheses for the next run, not to declare victory in the current run.

This discipline is easy to state and hard to keep, because evaluation dashboards reward fishing. Once dozens of slices are visible, a small green cell is tempting. The remedy is process: pre-register the release gate, report every tested slice, and mark post-hoc findings as post-hoc. Dror et al. made the same point for NLP experiments: the statistical test must match the metric and task, and significance claims across many datasets need their own replicability analysis (Dror et al. 2018; Dror et al. 2017).

Bias Is Not Noise

Uncertainty intervals answer a narrow question: if the same protocol sampled another comparable set, how much would the estimate move? They do not answer whether the protocol measures the right thing. A contaminated benchmark can have a tiny confidence interval. A biased judge stays consistent precisely because the bias is consistent. And a crowd annotation protocol may return a stable number that tracks worker fatigue rather than answer quality. Statistical reliability is therefore necessary but insufficient.

This distinction is useful because it prevents two common mistakes. The first is overcorrecting: a noisy result is not evidence of bias; it is evidence that the evaluation lacks power, the chance the test detects a real difference when one exists. The second is undercorrecting: a precise result is not evidence of validity; it is only evidence that the measurement is stable under its own assumptions. The varieties of bias are taken up later: Chapter 49 on human protocol bias, Chapter 50 on judge bias, and Chapter 51 on the special case where preference and truth diverge.

Calibration is another place where the distinction matters. A model can be accurate while overstating its confidence. In classification, calibration asks whether items assigned 80% confidence are correct about 80% of the time (Guo et al. 2017). In factual QA, the analogous question is whether the model abstains when it lacks knowledge, the concern that SimpleQA later isolates in Chapter 51. A release gate that only sees correctness will miss overconfident failures, and overconfidence is often the property that turns a wrong answer into a product risk.

What a Result Should Carry

A useful evaluation result is not just a scalar. It is a small record:

EvalResult:
  suite_version       # immutable dataset and task version
  model_version       # model, harness, decoding, tools, judge versions
  sampling_unit       # item, user, conversation, repository, task environment
  primary_metric      # chosen before the run
  estimate            # observed value
  uncertainty         # CI, bootstrap interval, or posterior interval
  paired_delta        # against the baseline on the same items
  slices              # predeclared slices and guardrails
  known_biases        # contamination, judge bias, annotator limits
  decision            # ship, block, investigate, or collect more data

The record forces the team to say what the number is allowed to do. A wide interval can still be useful if its job is to trigger investigation. Narrow intervals carry the opposite trap, precision earned under a biased protocol. A model can win the primary metric and still lose a guardrail. Treating those cases differently is not statistical fussiness; it is how evaluation becomes an infrastructure layer instead of a scoreboard.

Further reading