Evaluation and Observability

An evaluation does not begin with a leaderboard or a tool. It begins with an evaluation release: a named decision about one version of a complete system. Before running anything, freeze the system fingerprint, decision claim, target population, critical slice definitions, release gate, and rollback target. The result should tell an owner whether to merge, canary, ramp, stop, or continue operating, not merely whether one dashboard number moved.

This chapter turns the measurement principles from Chapter 47, Chapter 48, Chapter 49, Chapter 50, Chapter 51, Chapter 52, and Chapter 53 into an operating protocol. The subject is the running application, including the serving layer (Chapter 82), model gateway (Chapter 88), retrieval, agents (Chapter 85), and any fine-tuned model (Chapter 84).

Freeze the release contract

A good contract answers six questions before results are visible.

The fingerprint matters because a model name is not a system identity. A prompt, index, tool description, safety policy, parser, route, or provider alias can change behavior while the application version appears unchanged. Bind every evaluation result to immutable component digests. Production ML systems otherwise accumulate hidden dependencies and changing-world failure modes (Sculley et al. 2015; Breck et al. 2017).

Three different jobs

Evaluation asks whether declared evidence supports a decision about a fixed system. observability records enough structured events to reconstruct what a running system did. Monitoring compares measurements with thresholds over time and routes exceptions to an owner. These jobs can share storage and instrumentation, but they do not share semantics.

A trace is partial evidence, not a label and not ground truth. Instrumentation can be absent, sampled, dropped, redacted, or wrong. A quality score can also be wrong. Preserve the distinction so a telemetry export failure does not look like an improvement in product quality.

Build an evaluation dataset

An evaluation case is a versioned input plus the environment, expected behavior, allowed actions, label provenance, privacy status, and slice metadata needed to score it. Record whether it came from a designed scenario, a public benchmark, a synthetic generator, expert review, or production.

Production traffic is useful but selected. Consent or another valid authority must cover its use. Before a trace enters any suite, redact it, triage it, adjudicate its expected behavior, deduplicate it, and assign its split. Keep the following roles separate:

• a development suite for prompt, policy, and scorer iteration;
• a diagnostic suite for incidents, adversarial cases, and exploratory slices;
• a locked holdout for confirmation.

Group by the unit that can leak: user, tenant, conversation family, repository, source document, or time block. Do this before split assignment, then search for exact and near duplicates across splits. If a holdout case, answer, label, or close paraphrase influenced the prompt, retriever, judge, parser, or selection of the candidate, mark it as contamination and remove it from confirmatory evidence (Kapoor and Narayanan 2023).

Cases are not trials

A stochastic trial is one run of one system on one evaluation case under a recorded random seed and environment snapshot. A repeated trial estimates run variation; it does not create another independent user, task, or document. Preserve every timeout, refusal, parser failure, tool error, and empty result. Repeated trials are not independent cases, and selecting the best attempt is valid only when production uses that same selection policy.

The independent unit and case weight must follow the deployment claim. For a conversation product, turns and spans are nested within a conversation. For a repository assistant, files and tool calls may be nested within a repository task. Row count is not sample size.

Score behavior without hiding failure

Use the most direct evidence available for each criterion.

1. A deterministic check executes an invariant: schema validity, authorization, exact calculation, tool arguments, citation existence, or a latency ceiling. It cannot establish semantic correctness merely because an output parses.
2. A task or reference score measures an observable outcome: tests passed, evidence retrieved, claims supported, trajectory completed, or user goal achieved.
3. A human judgment applies a criterion-separated rubric when expertise and context are required.
4. A qualified model judge approximates a specific human or executable scoring protocol.
5. An operational metric measures availability, end-to-end latency, resource use, or cost. A separate security criterion measures policy violations and attack success.

Do not collapse these into one “quality” number. Report every predeclared slice, especially low-volume or high-risk slices. Record pass, fail, invalid, error, abstain, and fallback separately. A small slice is unresolved, not automatically safe, and an aggregate gain must not hide its regression. HELM's multi-metric, multi-scenario framing is a useful warning against single-score evaluation (Liang et al. 2023).

Compare complete revisions

Here, compare candidate system $B$ with baseline system $A$ using a higher-is-better score on the same cases and compatible environment. With case $i$, repeat $r$, $R_i$ declared repeats, score $s$, and predeclared case weight $w_i$, estimate

$$\widehat{\Delta} = \frac{1}{\sum_i w_i} \sum_i w_i \left( \frac{1}{R_i} \sum_{r=1}^{R_i} \left(s_{ir}^{B}-s_{ir}^{A}\right) \right).$$

Here $s_{ir}^{B}$ is the candidate score and $s_{ir}^{A}$ is the baseline score for the same case and repeat; $\widehat{\Delta}$ is their deployment-weighted paired change. Share the case, corpus snapshot, tool policy, budgets, and scorer. Do not force a shared random seed when two implementations interpret it differently.

Report a confidence interval for the estimand. Resample cases, or the higher-level user, conversation, repository, or document clusters named by the contract, rather than their nested turns, spans, or repeats. A hierarchical bootstrap can represent both case and run variation; a row-wise bootstrap invents information when rows share a deployment unit (Efron 1979; Bouthillier et al. 2021).

Predeclare the release gate

Here $L_j$ denotes the lower confidence bound for oriented criterion-and-slice change $j$, and $\delta_j$ denotes its allowed non-inferiority margin. A release can require

$$L_j \ge -\delta_j \qquad \text{for every required slice and criterion }j,$$

The rule means that every hard constraint also passes. Examples include zero cross-tenant access, valid tool authorization, an upper risk bound below its ceiling, and a tail-latency budget. There is no universal margin or confidence level: choose them from consequence, desired precision, sample design, and decision cost. Prespecify confirmatory claims and handle multiple comparisons when searching for any favorable result (Dror et al. 2018).

“No observed failures” is not “zero risk.” With sparse critical events, report an upper uncertainty bound; if the sample cannot exclude the risk ceiling, the gate is unresolved. A hard policy failure can remain a stop even when its estimated rate is imprecise.

Treat a model judge as an instrument

A model judge can extend review capacity, but it is a versioned measurement instrument rather than an oracle. Freeze the judge revision, rubric revision, prompt, decoding settings, candidate order, output schema, and parser. Then qualify it on a locked human-labeled set for each criterion and slice where it will operate.

Report the confusion matrix, false-pass and false-fail rates, abstentions, parser errors, uncertainty, and disagreement with humans. Test position bias by swapping candidate order and identifiers; test verbosity bias with meaning-preserving padding; test self-preference with generator-identifying and paraphrased outputs; test repeated-run stability and candidate-borne prompt injection. Research has documented position, verbosity, and self-preference effects, so using another model family is a hypothesis to test, not proof of independence (Zheng et al. 2023; Wang et al. 2024; Panickssery et al. 2024). G-Eval demonstrates that rubric-guided judging can improve task-specific human alignment; it does not establish universal validity (Liu et al. 2023).

Cohen's kappa adjusts observed categorical agreement for agreement expected from the marginal label frequencies (Cohen 1960):

$$p_o=\frac{1}{n}\sum_{a\in\mathcal C}n_{aa}, \qquad p_e=\frac{1}{n^2}\sum_{a\in\mathcal C}n_{a\cdot}n_{\cdot a}, \qquad \kappa=\frac{p_o-p_e}{1-p_e}.$$

Here $\mathcal C$ is the label set, $n_{ab}$ counts human label $a$ and judge label $b$, $n$ is the number of cases, and the dotted subscripts are marginal totals. Kappa measures agreement, not correctness or safety, and it changes with prevalence. Use it beside criterion-specific error rates and confidence intervals, never as a universal trust threshold.

def cohen_kappa(matrix):
    total = sum(sum(row) for row in matrix)
    observed = sum(matrix[i][i] for i in range(len(matrix))) / total
    row_totals = [sum(row) for row in matrix]
    col_totals = [sum(matrix[r][c] for r in range(len(matrix)))
                  for c in range(len(matrix))]
    expected = sum(r * c for r, c in zip(row_totals, col_totals)) / total**2
    return observed, (observed - expected) / (1 - expected)

# Rows are human labels; columns are judge labels: [bad, good].
agreement, kappa = cohen_kappa([[16, 4], [18, 162]])
assert round(agreement, 2) == 0.89
assert round(kappa, 2) == 0.53
print(f"agreement={agreement:.2f}, kappa={kappa:.2f}")

Promote production evidence carefully

The continuous loop should harden the suite without poisoning it. A complaint, incident, judge disagreement, or unusual trace enters a diagnostic queue first. An owner verifies use authority, removes sensitive data, reconstructs the system fingerprint, decides whether the case is reproducible, labels or adjudicates the expected behavior, deduplicates it, and assigns a suite role. Only then can it become a regression case.

Maintain two production samples:

• a probability or stratified sample for estimating population behavior;
• a risk queue for incidents, rare failures, complaints, and uncertain cases.

The risk queue is valuable for discovery, but its average is not a population rate. Record the inclusion probability, assignment unit, nonresponse, and label delay for the inference sample. User feedback, retries, and abandonment are selected observational signals. They do not identify the causal effect of a release; use randomized, sticky assignment and valid exposure logs when the decision requires that claim (Kohavi et al. 2009).

Define the trace contract

A trace groups work that belongs to one distributed request or task. Each span needs a trace ID, span ID, parent span when one exists, operation name, start and end times, status, and attributes. A span event records a timestamped occurrence inside a span; a span link connects causally related work that is not a parent-child edge. This distinction matters for queues, fan-out, retries, and multi-agent work.

For an AI operation, record the immutable system fingerprint, model and route, prompt and policy identifiers, retrieval and tool revisions, input digest, output digest, usage, cache result, retry number, and evidence IDs. Raw content is not required for correlation. OpenTelemetry provides general trace semantics and evolving GenAI conventions, but custom attributes still need a local schema and version (OpenTelemetry 2026; OpenTelemetry 2026).

Use W3C Trace Context to propagate the traceparent identifier across service boundaries (Kanzhelev et al. 2021). Baggage is application metadata, not a secret store: never place credentials, personally identifiable information, tenant content, or authorization grants in baggage. Validate incoming context because external callers can forge it.

Minimize before export

Apply data classification at instrumentation time. Raw payload recording is off by default. Redact before export, tokenize or hash only when the threat model permits it, and keep sensitive-content access separate from ordinary trace access. Enforce tenant isolation, purpose-bound roles, an access log, retention limits, and tested deletion across the collector, backend, cache, archive, and derived evaluation datasets. The telemetry path is a data system and belongs inside the AI risk-management boundary (Tabassi 2023).

Treat prompts, retrieved text, tool results, and model output as untrusted data. They may contain indirect prompt injection or instructions aimed at a later judge. An evaluator should have no production write authority, secrets, network tools, or unbounded budget.

Sampling changes the estimate

Head sampling decides near request start, usually from a fixed probability. Tail sampling waits for outcome information and can retain errors, slow requests, rare routes, or security events. Tail rules improve diagnosis but oversample the very outcomes they select.

Record each stratum's inclusion probability and compute a design-weighted estimate when reporting population rates. Preserve a stable probability sample alongside special retention for errors, rare slices, security signals, and judge disagreement. A dashboard over retained traces is otherwise a biased sample. Monitor missing telemetry, collector queue depth, dropped spans, broken parentage, and export failure as first-class rates; unknown evidence must not be silently treated as success.

Replay has levels

An exact replay requires immutable model artifacts, prompts, index, tool results, environment, and stochastic state. That is uncommon for hosted model aliases and live tools. A structural replay reruns the same operation graph against recorded or stubbed dependencies. A semantic replay asks whether the new system still satisfies the case criterion despite different tokens or steps.

Label the achieved fidelity. Record external state and resolved provider revision; a mutable model alias is not reproducible evidence. Never replay a write, payment, message, or other tool side effect merely to reproduce a trace. Use an idempotent sandbox, recorded response, or explicit dry-run adapter.

Monitor a qualified release

An alert definition is an operational contract. Record its metric revision, numerator, denominator, population and slices, aggregation window, label delay, threshold, required duration, missing-data rule, owner, and runbook. Version the reference window and detector. Alerts without an owner and a tested response are noise generators.

Monitor at least:

• traffic mix, critical slices, and system-fingerprint integrity;
• trace completeness, missing telemetry, and schema validity;
• deterministic, task, security, and delayed human-label rates;
• judge output and judge–human disagreement on a fixed anchor set;
• availability, end-to-end latency, tokens, and cost per accepted task.

End-to-end latency includes queueing, every retry, cache lookup, model call, and tool call until an accepted result or terminal fallback. Cost per accepted task includes failed attempts, judge cost, and allocated human review, not only the last completion.

A drift detector says that a monitored distribution changed. Drift does not identify the cause, prove quality declined, or isolate the changed component. Confirm trace integrity, traffic mix, system revision, label delay, and evaluator stability before attributing the movement. A judge changing on a fixed human anchor set is evaluator drift, not automatically application drift.

Test failures before release

Operate the release lifecycle

1. Freeze the decision, system fingerprint, population, slices, metrics, and privacy boundary.
2. Version the dataset manifest, provenance, consent basis, labels, and split assignment.
3. Declare cases, independent units, repeated trials, weights, invalid outcomes, and uncertainty method.
4. Qualify every human protocol and model judge against appropriate reference evidence.
5. Run baseline and candidate under matched conditions; retain every attempt.
6. Apply hard constraints and confidence-bound gates to every required slice.
7. Use shadow execution where it can reveal integration faults without user effects, then start a small sticky canary.
8. Ramp only while quality, security, latency, cost, and telemetry-completeness stop rules pass.
9. Roll back atomically to the last-known-good revision when a rollback trigger fires.
10. Store estimates, intervals, invalid outcomes, waivers, approvals, canary results, expiry, rollback target, and each requalification trigger.

The output is an evaluation release record. It binds the claim and evidence to the exact system that was released, explains uncertainty and unresolved slices, and makes recovery executable.

Further reading