Operational Evaluation and Governance

An offline evaluation ends with a result. Operational evaluation ends in an operational decision: promote, hold, narrow, roll back, or escalate the complete system revision. It also states what new evidence can change that decision. The object under review is not a model name alone. It is the model together with its prompt, retrieval data, tools, policies, routing, and serving configuration.

This chapter closes Part VII by connecting measurement, deployment, monitoring, and learning. Earlier chapters define valid tasks, statistical comparisons, human rubrics, model judges, factuality checks, and agent outcomes. Here those instruments become one release loop. The implementation tools live in Chapter 87; the concern here is the operating contract they must enforce.

Write the Release Policy Before the Run

A research comparison asks what happened under a published protocol. A release gate asks whether a named candidate may affect a named population. The gate is therefore a versioned policy, not a favorable chart or a meeting-time judgment.

The roots of this practice predate generative AI. Sculley and colleagues described how data dependencies, configuration, monitoring, and other surrounding machinery create hidden technical debt in machine-learning systems in 2015. Breck and colleagues then framed production readiness as tests across data, model, infrastructure, and monitoring in 2017 (Sculley et al. 2015; Breck et al. 2017). Generative systems enlarge the tested surface, but do not remove that lesson: a release decision belongs to the deployed system and its operating environment.

A minimal gate specification makes that surface explicit:

ReleaseGateSpec:
  decision_id
  candidate_system_hash
  baseline_system_hash
  target_population
  evaluation_manifest_hash
  metric_and_scorer_versions
  primary_decision_rule
  guardrail_margins
  invalid_outcome_policy
  stage_plan
  rollback_target
  decision_authority
  override_policy
  evidence_expiry

The two system hashes cover every behavior-bearing component: model revision and route, decoding policy, prompts, tool and API schemas, retriever and corpus, safety policy, sandbox, and application code. The target population names the users, tasks, languages, risk classes, and time period to which the decision applies. The invalid-outcome policy decides in advance how timeouts, grader faults, missing labels, and infrastructure failures enter the analysis. The entire specification is written before the confirmation run. A dashboard is evidence; it is not the policy.

The statistical rule can be stated compactly. Let $v$ be the candidate system revision and $b$ the baseline revision. For higher-is-better quality metric $j$ on predeclared slice $s$, let

$$\Delta_{js}(v,b)=q_{js}(v)-q_{js}(b).$$

Here $q_{js}(v)$ is the target value of metric $j$ for candidate $v$ on slice $s$, $q_{js}(b)$ is the corresponding baseline value, and $\Delta_{js}(v,b)$ is their difference. A useful release rule has the form

$$G(v)=N(v)\land F(v)\land H(v)\land E(v).$$

Here $G(v)$ is the final gate decision. $N(v)$ is true only when the lower confidence bound for every required $\Delta_{js}(v,b)$ exceeds $-\delta_{js}$, where $\delta_{js}\ge 0$ is the predeclared non-inferiority margin. $F(v)$ is true only when the lower confidence bound for every required $q_{js}(v)$ meets its absolute quality floor $f_{js}$. $H(v)$ is the hard operational predicate covering such conditions as safety assertions, valid outputs, latency limits, and error budgets. $E(v)$ is the evidence-validity predicate: the correct manifest ran, required sample sizes were reached, graders passed health checks, and no disallowed exclusions occurred. All four conditions must hold. Lower-is-better measures use the same logic after reversing the sign or defining an upper-bound predicate explicitly.

This rule distinguishes three outcomes that dashboards often blur. A pass meets the declared margins. A fail crosses a blocking boundary. An unresolved result lacks enough valid evidence to exclude a material regression. An unresolved result is not a pass; the policy must say whether to collect more evidence, narrow exposure, or keep the baseline.

Run candidate and baseline on the same independent units whenever possible. Analyze the paired difference rather than two unrelated score intervals. For paired binary outcomes with few disagreements, use the exact McNemar test; for numeric task scores, use a paired randomization or bootstrap procedure that preserves the sampling unit. In every case, retain the effect estimate and interval, not just a thresholded test result. Chapter 48 gives the full analysis contract. Do not search for a favorable slice after the run and promote it to the primary claim.

Collect Different Evidence at Each Stage

No offline suite represents every production condition. Release engineering therefore widens exposure in stages. Google's SRE workbook defines a canary as a partial, time-limited deployment evaluated against a control, and emphasizes representative traffic, attributable metrics, isolation, and rollback (Warner and Davidovič 2018). The same structure applies to an AI system, but each stage must inspect model-specific quality as well as ordinary service health.

Shadow traffic duplicates a request to the candidate but returns only the baseline response. It must not execute user-visible writes, send messages, charge accounts, or mutate a shared environment. A stateful agent needs a sandbox or a dry-run tool layer; otherwise “shadow” is a second production actor.

A canary does affect users. Assign traffic so the canary and control run at the same time, preserve assignment at the user or conversation level, and label every metric by revision. Before-and-after comparisons confound the release with time. Start with cheap, fast blockers such as crashes, invalid tool calls, severe policy violations, and tail latency. Widen only when enough evidence has accumulated for the next rule. Every stage needs an automatic stop or rollback path plus a named human owner for ambiguous results.

No stage substitutes for another. Offline confirmation is strongest on known failures. Shadowing exposes integration faults without user-visible output. A canary measures bounded live impact. Production monitoring catches later changes and rare outcomes. Passing the easiest stage says nothing about the evidence available only in the next one.

Keep the Private Suite Useful

A private suite is valuable because it contains the organization's real task distribution, policies, incidents, and expensive edge cases. Its secrecy alone does not make it valid. Repeated tuning spends its independence, labels age, customer data creates handling obligations, and executable graders break as dependencies change.

Separate cases by purpose:

• The development suite is visible to builders and runs frequently. It supports debugging, so it is not independent confirmation.
• The locked confirmation suite is inaccessible during selection and is opened only by the release process. Its exposure ledger records every person, system, model, and training job that could have seen an item or its answer.
• The diagnostic suite contains exploratory slices, red-team probes, and newly observed failures. It locates problems but does not silently acquire release authority.

An incident is evidence for a candidate case. It does not become a blocking test until the case has passed de-identification, scope review, reproducibility checks, label or outcome validation, and positive and negative grader cases. A one-off support report may be real yet impossible to reproduce; a production trace may include private data; a judge may reward the wording of the original answer rather than the intended behavior. The intake process has to resolve those problems before the next release depends on the case.

Each item needs a source, owner, target failure mode, population tag, severity, immutable input fixture, allowed behavior, grader revision, exposure status, and retirement condition. Run reset checks for stateful tasks. Audit a sample of passing items as well as failures so a permissive grader cannot make the suite look healthy. Retire or relabel an item when policy, facts, APIs, or product behavior change. Preserve its history rather than rewriting old release evidence.

The lifecycle is therefore:

observe -> triage -> de-identify -> reproduce -> label and test grader
        -> assign suite role -> version -> monitor exposure -> retire or relabel

This is evaluation-data maintenance, not clerical cleanup. Hidden dependencies and undeclared consumers are precisely how test infrastructure becomes technical debt (Sculley et al. 2015). Chapter 47 covers contamination and benchmark validity; Chapter 92 operates the intake and curation path from production.

Diagnose Drift Before Changing the Gate

“Drift” is not a root cause. Three different things may have changed:

• System change: the model endpoint, route, prompt, retriever, corpus, tool, policy, or application revision changed.
• Population change: the mix of users, tasks, languages, threats, or external conditions moved away from the release population.
• Measurement change: the rubric, judge, annotator mix, telemetry coverage, parser, or outcome definition changed.

These changes require different responses. A system change calls for comparison with the last known revision. A population change may require a new slice, new weighting, or a narrower product claim. A measurement change requires replaying the old and new instrument on the same stored evidence before attributing a behavior change to the system.

Use three complementary monitors. Fixed sentinel cases detect changes in a supposedly pinned system. Sampled live traffic estimates current behavior on the population being served. Delayed outcome labels connect proxy scores to events such as task completion, correction, appeal, or incident. The first is stable but narrow, the second is current but needs privacy-aware sampling and grading, and the third is closest to user value but often arrives late and selectively. Direct risk-monitoring methods can use sampled or delayed labels while preserving guarantees under repeated looks, but their validity still depends on the declared loss and sampling process (Podkopaev and Ramdas 2022).

Input-distribution alarms can help locate change. Rabanser, Günnemann, and Lipton compared dataset-shift detectors and found representation-based two-sample approaches effective across the image shifts they studied (Rabanser et al. 2019). That result does not turn a distance between prompt embeddings into a quality verdict. A distribution alarm does not prove a quality regression, and a stable input distribution does not prove stable behavior. Route the signal to investigation, then inspect outcomes on the affected slice.

Underspecification adds another warning. Systems with similar held-out scores can behave differently in deployment because the test did not constrain every relevant behavior (D'Amour et al. 2022). Operational monitoring should therefore preserve outcome and slice evidence, not merely confirm that an aggregate resembles the release score.

Every score must be reconstructable. OpenTelemetry's development-status GenAI conventions provide shared names for common model, agent, tool, evaluation, and span data (OpenTelemetry 2026). They cover only part of an auditable evaluation record, and their prompt, output, and tool content can be sensitive. Pin the convention revision; store hashes or controlled artifact references instead of indiscriminately copying raw content. Semantic conventions do not identify the whole experiment. The application still needs its own immutable release record:

ReleaseDecision:
  decision_id
  system_component_hashes
  evaluation_result_ids
  rollout_stage
  traffic_assignment
  stage_results
  monitoring_window
  override_actor_and_reason
  override_expiry
  decision_and_timestamp
  rollback_target_and_trigger
  incident_followup_ids

The evaluation_result_ids point to the immutable statistical records defined in Chapter 48, which already carry the population, suite, scorer, raw outcomes, exclusions, estimates, and intervals. The release record adds what that analysis does not know: rollout stage, authority, monitoring window, and rollback action. Store it and later corrections as append-only evidence. Regrading creates a new result linked to the old one; it does not overwrite the decision that was actually made. Evidence copied from an earlier run must retain its original system identity and age. Otherwise a report can appear reproducible while quietly combining different conditions.

Compare the Operating Frontier Before Choosing a Point

Raw task quality is only one production axis. A candidate also consumes money and time, creates review load, and carries risks that may not be exchangeable. Report the Pareto frontier first: remove options that are no better on any measured axis and worse on at least one. Then choose among the remaining options under the product's constraints.

When the trade-offs are genuinely commensurable, a declared utility function can make the choice inspectable:

$$\begin{aligned} U(m) &= Q(m) - \lambda_c C(m) \\ &\quad - \lambda_l L(m) - \lambda_r R(m). \end{aligned}$$

Here $m$ is a complete system option; $Q(m)$ is its task-quality value; $C(m)$ is metered cost per task; $L(m)$ is a declared latency statistic; and $R(m)$ is an expected review or incident burden measured under a stated policy. The nonnegative weights $\lambda_c$, $\lambda_l$, and $\lambda_r$ convert those burdens into the same decision scale as quality. The weights are policy choices, not facts learned automatically from the evaluation.

Keep the inputs concrete. Report median and tail latency, not only a mean. Join metered cost from the gateway or billing ledger to each run rather than reconstructing an approximate bill from a stale price sheet. Include invalid-run and fallback rates. State whose burden the utility represents.

The scalar comes after hard boundaries. Quality guardrails remain constraints when the product cannot buy its way out of a severe regression. A scalar cannot authorize a safety trade merely because a dangerous option is cheaper or faster. Chapter 76 develops the cost side; Chapter 82 and Chapter 91 determine latency and review burden.

Make the Decision Survive the Meeting

Governance starts when the evidence, authority, and follow-up outlive the people in the release meeting. Model cards provide a structure for intended uses, evaluation procedures, and disaggregated performance (Mitchell et al. 2019). Datasheets for Datasets record the motivation, composition, collection, use, and maintenance of data assets (Gebru et al. 2021). NIST's AI Risk Management Framework organizes risk work around govern, map, measure, and manage, and is explicitly voluntary and use-case agnostic (Tabassi 2023). These artifacts serve different scopes; none replaces the release record.

Every blocking metric, suite, and production monitor needs a named owner. Every manual decision needs the evidence reviewed and the authority used. An override is a new decision, not deletion of a failed gate. Make every override time-bounded, name the accepting owner, record the reason and compensating controls, narrow traffic when appropriate, and set the evidence or date that forces reconsideration.

After an incident, record four separate outputs:

1. the user or system impact;
2. the contributing system and measurement failures;
3. the corrective action and owner;
4. a regression candidate or an explicitly accepted risk.

The regression candidate then follows the private-suite intake process. If it proves reproducible and decision-relevant, it enters a versioned suite. If it cannot become a test, the accepted-risk record must state why and how the risk is monitored. That distinction prevents a slogan such as “every incident becomes a test” from filling the suite with brittle or privacy-unsafe fixtures.

A practical operating review asks:

1. What complete system and target population does this decision cover?
2. Which locked evidence supports the primary claim and each guardrail?
3. Are pairing, uncertainty, invalid outcomes, and slices handled as declared?
4. What distinct evidence did offline, shadow, and canary stages add?
5. Which production signals can stop or roll back the release?
6. Who owns the decision, override, rollback, and evidence expiry?
7. How will confirmed failures close the loop without contaminating confirmation data?

If any answer exists only in someone's memory, the gate is not yet operational.

Further reading