Operational Evaluation and Governance

Offline evaluation tells a team what a model did on a controlled set. Production evaluation tells the team whether a living system should keep running, roll back, ask for review, or collect a new failure for the next regression suite. The two are connected, but they are not the same job.

This chapter closes Part VII by turning measurement into an operating discipline. It does not repeat the tool wiring in Chapter 87. Instead, it defines the release gates, private suites, drift checks, cost-quality frontiers, documentation, and governance contracts that make those tools meaningful.

The Release Gate Is a Policy

An operational evaluation suite has a different job from a research benchmark. A research benchmark asks which system is better under a public protocol. A release gate asks whether this particular system version is allowed to affect users. That makes the gate a policy, not a chart.

The policy needs four fields:

• Scope. Which model, prompt, retrieval index, tool set, guardrail, and routing rule are under test.
• Threshold. Which metrics must improve, which must not regress, and how uncertainty is handled.
• Authority. Who can override the gate, and what evidence must be recorded.
• Fallback. What happens if the gate fails: rollback, shadowing, human review, narrower traffic, or more data.

This is the operational counterpart of Chapter 48. A number with no decision rule produces debate after every run. A decision rule with no uncertainty produces brittle automation. A gate needs both.

The four gates should not all use the same metric. Offline regression suites catch known failures. Where shadow traffic earns its keep is integration problems: missing tools, bad schema, latency surprises, broken retrieval. Canary buys a look at distribution shift while keeping the blast radius small, and production monitoring is what finally surfaces drift and long-tail failures. Treating one of these as a substitute for the rest creates the familiar false comfort: a system passed the test it was easiest to pass.

The Private Suite Is an Asset

The most valuable evaluation set in a production system is often the one that cannot be published. Public benchmarks make work comparable, but they decay through saturation and contamination. A private suite is less useful for public ranking, but more useful for release decisions, because it can contain the organization's own failures, policies, customer edge cases, and business costs.

That suite needs lifecycle management:

PrivateEvalSuite:
  source             # incident, support ticket, red-team run, expert case, synthetic probe
  owner              # team responsible for the failure mode
  label_protocol     # human, executable, judge, state check
  severity           # release-blocking, guardrail, exploratory
  exposure           # who has seen the item or answer
  version            # immutable item version
  rotation_policy    # when to retire, refresh, or hide

This is where operational evaluation inherits the lesson from Chapter 47. A suite is not merely a folder of prompts. It is an asset whose integrity can be spent. Every time answers leak into training data, every time a prompt is pasted into a public issue, every time a model is tuned against the same item, the suite loses some ability to surprise the next model. Rotation is not bureaucracy. It is how the gate stays alive.

Sculley et al.'s hidden technical debt argument applies directly here: the evaluation harness, data dependencies, and undocumented consumers around a model can become the system's largest maintenance burden (Sculley et al. 2015). Breck et al.'s ML Test Score made the same point operationally by treating tests across data, model, infrastructure, and monitoring as production-readiness criteria (Breck et al. 2017).

Drift Is Not One Thing

Drift means the live input or behavior changing over time, after a version that passed its gate is already serving users. Production drift is often named as one phenomenon, but the useful debugging split is finer.

• Input drift. User requests change. A prompt distribution shifts from casual questions to high-stakes workflows.
• Context drift. The retrieval corpus, index, ranking model, or document parser changes.
• Model drift. A hosted model version changes, or a routing policy changes which model sees which tasks.
• Policy drift. The allowed behavior changes because the product, law, or safety policy changes.
• Metric drift. The judge, rubric, or telemetry schema changes.

D'Amour et al. call attention to underspecification: many predictors can have similar held-out performance yet behave differently in deployment (D'Amour et al. 2020). LLM systems inherit that problem and add more moving parts. Two versions can tie offline and diverge online because the prompt template routes users into different behavior, or because the retriever changed the evidence distribution.

Operational evaluation therefore needs provenance for every score. A trace should say not only which model answered, but which prompt template, model gateway route, tool schema, retriever index, judge version, and policy version participated. OpenTelemetry's GenAI conventions are one attempt to make such traces portable across tools and vendors (OpenTelemetry n.d.). Chapter 87 discusses the concrete wiring; here the principle is that a score without provenance cannot be audited.

Quality Is Not the Only Axis

The best model by raw quality may be the wrong production choice. Users feel latency, and the finance team watches cost. Risk is the safety team's axis, while the failure rate is what operators live with. Evaluation must therefore report frontiers rather than single winners.

One practical scalar is a utility score,

$$U(m) = Q(m) - \lambda_c C(m) - \lambda_l L(m) - \lambda_r R(m),$$

where $Q$ is task quality, $C$ is cost per task, $L$ is latency, $R$ is risk or expected review burden, and the $\lambda$ values are weights you set by hand to express the product's operating priorities, not parameters learned from data. The formula is not a universal metric. It is a forcing function: it makes the trade-off explicit. A customer-support bot and a code-repair agent should not use the same weights.

This is the bridge to Chapter 76. A price change can move a model onto or off the frontier with no change in capability at all. Faster serving in Chapter 82 often makes a slightly weaker model the better operating point, and a better human review surface in Chapter 91 lowers $R(m)$ by catching uncertain cases before they cause harm.

Governance Turns Evidence into Memory

Evaluation evidence becomes governance when it is written down in forms that survive the release meeting. Model cards report intended uses, evaluation procedures, and disaggregated performance (Mitchell et al. 2019). Datasheets for Datasets do the same for the data that evaluation and training depend on (Gebru et al. 2021). NIST's AI Risk Management Framework makes "measure" one of the core functions alongside govern, map, and manage (National Institute of Standards and Technology 2023).

These documents are sometimes treated as compliance paperwork. For a production AI system, they are more useful when treated as operational memory:

• which evals blocked the last release;
• which slices are known weak;
• which private suites must not enter training;
• which human review gates are mandatory;
• which benchmark claims are public, private, or deprecated;
• which incidents created which regression tests.

The final bullet is the most important. Every serious production failure should become either a regression test or an explicitly accepted risk. If neither happens, the organization has not learned from the failure. It has only recovered from it.

Further reading