Statistical Reliability as a Decision Contract

A benchmark score is an estimate only when the evaluation design says what it estimates. On a fixed set of deterministic questions, 73% is simply the fraction answered correctly. It becomes evidence about future tasks only after the team names a target population, a sampling process, and the sources of variation that the analysis is meant to cover.

That distinction changes the release question. “Did B score higher than A?” is usually too weak. A useful question is closer to this:

On the target task population, under the production decoding and scoring protocol, is B better than A by enough to matter, without violating any guardrail?

This chapter turns that question into an analysis contract. It separates item variation from run and scorer variation, preserves pairing, plans for a decision-relevant effect, and distinguishes an uncertain result from a biased instrument. The benchmark contract in Chapter 47 defines what is measured. This chapter defines how strongly the observed data support a decision.

Name the Quantity Before Its Interval

An interval cannot rescue an undefined target. Start with an estimand: the exact quantity the evaluation is intended to estimate. For a system $s$, one general form is

$$\theta_s = \mathbb{E}_{U \sim P,\, R \sim Q,\, J \sim H} \left[m(s, U, R, J)\right].$$

Here $U$ is an independent sampling unit drawn from target population $P$; $R$ is a run configuration drawn from production distribution $Q$; $J$ is a human or automated scorer drawn from scoring protocol $H$; and $m$ is the predeclared metric. The expectation may average over one, two, or all three sources. The evaluation must say which.

This notation prevents several common category errors:

The row count is not automatically the sample size. Twenty attempts on one repository are still one repository for claims about repositories. Fifty turns from one conversation are not fifty independent conversations. Treating correlated rows as independent creates narrow intervals without creating new evidence.

There is also an important fixed-set case. If a deterministic model is evaluated on every item in a frozen manifest and the claim is only about that manifest, the mean is a complete descriptive result; there is no item-sampling error to estimate. An item bootstrap adds a hypothetical population assumption. That can be useful, but the report should state it rather than presenting the interval as a property of the fixed file. Run or scorer uncertainty may still remain.

Match the Estimator to the Design

For $n$ independent binary items with $x$ successes, observed accuracy is

$$\widehat{p} = \frac{x}{n}.$$

Here $x$ is the success count, $n$ is the number of independent items, and $\widehat{p}$ is the observed success proportion.

The familiar normal, or Wald, approximation gives the 95% half-width

$$h_{\mathrm{Wald}} \approx 1.96 \sqrt{\frac{\widehat{p}(1-\widehat{p})}{n}}.$$

Here $h_{\mathrm{Wald}}$ denotes the approximate distance from the point estimate to either endpoint. The square-root factor represents the estimated standard error, and 1.96 is the usual standard-normal multiplier for a two-sided 95% interval.

It is a useful planning sketch: precision improves at roughly the square root of sample size, so halving a half-width requires about four times as many independent items. It is not a universal production interval. Its coverage is poor for small samples and proportions near zero or one. For an independent binomial proportion, use a method with better coverage, such as a Wilson interval, and record the method (Brown et al. 2001).

Most evaluations need something else:

• For a mean rubric score, estimate uncertainty for the mean under the actual sampling design; do not convert the score to fake Bernoulli trials.
• For tasks clustered by user, repository, document, or conversation, resample or model the cluster, not each row inside it.
• For repeated stochastic runs, keep repeats nested within the task. First define how run outcomes combine into the task-level quantity, then aggregate tasks.
• For a weighted metric, apply the same declared weights in every resample and report the effective support behind heavily weighted slices.
• For a ratio, pass rate, quantile, or nonlinear composite, resample the independent units and recompute the whole estimator rather than bootstrapping already-aggregated cells.

The bootstrap is valuable because it can approximate an estimator's sampling distribution without deriving a new closed form (Efron 1979). Recent work applies the same design discipline directly to language-model evaluations (Miller 2024). It does not discover the correct unit or repair a biased sample. A row-wise bootstrap over correlated rows faithfully automates the wrong design.

A 95% confidence interval also has a procedural meaning. Under repeated samples from the assumed design, the interval-producing procedure covers the fixed target at its nominal rate. It is not a statement that this realized frequentist interval has a 95% probability of containing the target. If a Bayesian posterior interval is used instead, name the model and prior; do not silently switch interpretations.

Compare Systems on the Same Units

When A and B receive the same tasks, analyze their paired outcomes. Here $y_{gA}$ and $y_{gB}$ denote the task-level scores for independent unit $g$. Define

$$d_g = y_{gB} - y_{gA}, \qquad \widehat{\Delta} = \frac{1}{G}\sum_{g=1}^{G} d_g.$$

Here $d_g$ is B's change on unit $g$, $G$ is the number of independent units, and $\widehat{\Delta}$ is the estimated mean change. Build the interval from the $d_g$ values, or resample paired clusters. Separate intervals around A and B throw away their covariance. This is why “the error bars overlap” and “the error bars do not overlap” are both unreliable shortcuts for deciding whether the paired difference is supported.

For binary outcomes, retain the disagreement table:

Here only $n_{10}$ and $n_{01}$ distinguish the systems: they represent the two directions in which A and B disagree. McNemar's test is built around that paired evidence (McNemar 1947). The common large-sample statistic is

$$\chi^2 = \frac{(n_{10}-n_{01})^2}{n_{10}+n_{01}}.$$

Here the numerator measures the squared imbalance between the two disagreement directions, while the denominator is their total count.

When disagreement counts are small, use the exact paired-binomial form rather than leaning on this asymptotic approximation. In either case, retain the effect estimate and its interval. A test can indicate incompatibility with a zero-difference model; it does not say whether the difference matters.

Per-unit retention also exposes engineering tradeoffs hidden by an average. The mean can improve while a safety slice regresses, or while B exchanges one failure class for another. Store task ids, both raw outputs, run ids, scorer versions, parsed outcomes, and pairwise differences. Statistical reliability begins with reconstructable records.

Define What Would Change the Decision

A zero-effect threshold is rarely the product threshold. Before collecting the decisive sample, define a smallest effect of interest, $\delta$. Here $\delta$ means the smallest gain or loss large enough to change the decision after cost, latency, and risk are considered.

For a higher-is-better metric, a confidence interval for $\Delta = \theta_B-\theta_A$ can support different claims. Here $\theta_A$ and $\theta_B$ are the two systems' target values, so $\Delta$ represents B's improvement over A:

These are different study designs, not labels to choose after seeing the plot. Equivalence requires evidence that effects outside the predeclared band are implausible under the procedure; failing to reject an exact zero is not evidence of equivalence (Lakens et al. 2018). Guardrails need their own one-sided margins. A safety metric normally asks whether a harmful regression has been excluded, not whether its point estimate happened to improve.

Statistical power belongs here. Power is the probability that a specified procedure will detect a specified effect under stated assumptions. A useful sample-size plan therefore needs the effect or margin, error rate, desired power, outcome variance, pairing, and clustering. “Use 1,000 examples” is not a power analysis. NLP studies are often underpowered for the small improvements they try to claim, which makes both missed effects and exaggerated selected wins more likely (Card et al. 2020).

The plan should also include invalid or missing outcomes. Timeouts, parser failures, refusals, and scorer errors cannot disappear because they complicate the statistic. Predeclare whether each state is a failure, a separate outcome, or grounds for a rerun. Report its rate either way.

Repeated Looks and Many Claims Spend Evidence

Running the same test every hour and stopping on the first favorable interval is not the fixed-sample procedure the interval assumed. Optional stopping increases the chance of a false win. Choose one of two honest designs:

1. Freeze a sample and analysis time, then decide once.
2. Use a sequential design whose error guarantee remains valid under repeated looks, such as a confidence sequence, and record the stopping rule (Howard et al. 2021).

The same problem appears across a dashboard. A suite may contain tasks, languages, prompt templates, judges, slices, and decoding settings. If enough cells are searched, some will look favorable by chance. Define the family of claims before the run and assign each result a role:

• Primary claim: the single comparison allowed to justify the planned release decision.
• Guardrails: predeclared one-sided checks that can block the release.
• Confirmatory secondary claims: a bounded family analyzed with a stated multiplicity procedure.
• Exploratory findings: leads for a new evaluation, not wins established by this one.

Family-wise error-rate control limits the chance of any false rejection in the declared family. False-discovery-rate procedures, including Benjamini--Hochberg, control the expected false-discovery proportion under their assumptions and are often more useful for broad exploratory families (Benjamini and Hochberg 1995). Neither procedure legitimizes selecting the family after seeing the results. Report all tested claims and preserve failed as well as favorable comparisons.

Repeated model and prompt selection on the same benchmark creates a second adaptive loop. Correcting one report's $p$-values does not undo information already leaked through months of leaderboard feedback. Record the selection history, keep a confirmation set inaccessible during development, and evaluate the chosen candidate once on that locked set. Reusable-holdout methods can limit information leakage under defined query mechanisms, but ordinary unrestricted reuse eventually turns the benchmark into development data (Dwork et al. 2015).

A $p$-value, when used, measures how incompatible the data are with a specified statistical model. It is not the probability that the null is true, the probability that the result arose “by chance,” or a measure of effect size or product importance (Wasserstein and Lazar 2016). Report the estimate, interval, assumptions, and decision boundary alongside it.

Precision Cannot Repair Bias

Uncertainty describes variation under the declared protocol. Bias describes a systematic difference between that protocol and the target claim. They require different responses:

A precise estimate can be precisely wrong. Confidence intervals do not cover contamination, label errors, population drift, an unrepresentative prompt mix, or a judge that rewards verbosity. Those threats belong in the benchmark's validity and audit record.

Calibration is a related but separate estimand. A model is calibrated when outcomes assigned confidence $q$ occur at rate $q$ in the target population (Guo et al. 2017). Measuring it requires probability-bearing predictions, a declared binning or proper-scoring method, and uncertainty at the appropriate unit. Accuracy alone cannot establish calibration, and a clean calibration curve on yesterday's traffic does not guarantee calibration after distribution shift.

The Result Is a Reproducible Decision Record

A scalar is not enough. Each release-grade result should resolve to a versioned record:

EvaluationResult:
  decision_id
  evaluation_spec_hash
  target_population
  system_a_revision
  system_b_revision
  item_manifest_hash
  run_protocol_revision
  scorer_revision
  estimand
  estimator
  independent_unit
  cluster_keys
  repeat_policy
  primary_claim
  effect_margin
  interval_method
  confidence_level
  sequential_rule
  multiplicity_family
  selection_history
  confirmation_set_version
  estimate_a
  estimate_b
  paired_delta
  interval
  invalid_outcome_counts
  slice_and_guardrail_results
  known_validity_limits
  decision

The operational procedure is short:

1. Write the decision, target population, estimand, independent unit, and system boundary.
2. Freeze the manifest, model, harness, run policy, scorer, and missing-outcome rules.
3. Declare the primary claim, guardrails, practical margin, analysis family, and stopping rule.
4. Plan sample size from the design and the smallest effect the decision cares about.
5. Run A and B on the same units; preserve raw outputs and nested repeat or scorer records.
6. Compute the declared estimator and paired interval at the independent-unit level.
7. Report every planned claim, invalid outcome, guardrail, and known source of bias.
8. Apply the predeclared decision rule. If the result is unresolved, say so.

Three invariants make the record auditable: no row silently changes the sampling unit, no result silently changes the claim family, and no release decision silently changes the practical margin after seeing the data.

The next chapter, Chapter 49, moves inside one especially important scorer. It asks how an instruction, rubric, annotator pool, and adjudication process create the human label that the statistical analysis otherwise treats as data.

Further reading