Reliability for Nondeterministic Systems

A reliable model service is not one that repeats the same response bytes. It is one that keeps a stated contract over user-visible outcomes. The contract may require a timely response, a correct answer, a permitted action, fresh evidence, or a side effect that occurs no more than intended. Some of those properties are deterministic invariants; others require statistical evidence. Production reliability needs both.

Nondeterminism changes how semantic outcomes are observed, but it does not replace the foundations of site reliability engineering. Conventional services already have variable latency, partial failures, stale data, and incorrect results. A 200 response has never proved that the result was useful. What model-backed systems add is a large outcome space in which byte equality is a particularly poor proxy for quality. The operational response is therefore not to invent a separate meaning of reliability. It is to write a reliability contract whose event, predicate, evidence, and recovery path match the promise made to users.

This chapter builds that contract. It separates service objectives from their measurement implementations, shows how to estimate semantic quality from a probability sample, states when the familiar $p^n$ task model is valid, and qualifies the retries, fallbacks, degradation, and replay controls placed around generation.

Start from the outcome contract

The SRE event model is a useful starting point. An service level indicator (SLI), a quantitative indicator of one service property, is often implemented as the ratio of good events to eligible events. An service-level objective (SLO) sets a target for that indicator over a window. SRE distinguishes an SLI specification, which states what should be measured, from an SLI implementation, which states how available telemetry estimates it (Thurgood and Ferguson 2018). This distinction matters even more when production outcomes need human or model judgment.

Do not compress the service into one vague "quality" number. Write separate dimensions so that a healthy aggregate cannot hide a broken safety or effect boundary.

The contract must name the eligible event, exclusion rules, population and slice, observation window, system release, measurement implementation, missing-data rule, objective, and recovery action. "Good answers exceed 95%" is not yet a contract: it does not say whose answers, under which release, how they are judged, or what happens when the judge is unavailable.

This structure also prevents a common category error. Reachability, latency, semantic quality, policy compliance, freshness, and effect correctness are related, but not interchangeable. A composite acceptance predicate can be useful for a specific product promise, yet each component still needs its own SLI and slice view. Otherwise an abundant supply of fast, easy requests can mask rare but severe policy failures.

Measure semantic outcomes without mistaking the sample for traffic

Suppose each eligible production event $i$ has a semantic outcome label $Y_i \in \{0,1\}$, where $1$ means that the event passes a frozen rubric. If a probability sample $S$ includes event $i$ with known inclusion probability $\pi_i > 0$, a useful weighted ratio estimator is

$$\widehat{p} = \frac{\sum_{i \in S} Y_i / \pi_i} {\sum_{i \in S} 1 / \pi_i}.$$

This is a Hájek-style weighted ratio: each observed label represents the eligible events with the inverse inclusion weight $1/\pi_i$ (Horvitz and Thompson 1952). It reduces to the ordinary sample pass rate when all inclusion probabilities are equal. The ratio is not magically unbiased in every finite design, but it is a far better description of the target population than an unweighted mixture of oversampled failures and routine traffic.

The sampling policy determines what can be claimed:

• A representative probability sample supports an estimate for its named population because every eligible event has a known, nonzero chance of inclusion.
• A diagnostic sample deliberately enriches incidents, rare slices, or low confidence outputs. It is excellent for finding defects, but its raw pass rate is not the production pass rate. Combine it with the representative stream or use the recorded inclusion probability in the estimator.
• A convenience sample of events that happened to receive labels supports neither claim unless its selection mechanism is justified.

Sampling alone is insufficient. The measurement record must bind the traffic definition, sampling policy, $\pi_i$, rubric version, judge version, system release, and label timestamp. A judge needs a recurring human audit that estimates false positive and false negative behavior on important slices. Recalibrate or replace it when evaluator drift appears; never silently compare two windows scored by materially different judges. The evaluation practice in Chapter 52 and statistical intervals in Chapter 48 provide the offline machinery. Production adds selection, delay, and missingness.

Treat a missing label as unknown, not a pass. Report measurement coverage beside quality. Here $C$ measures the fraction of due sampled events whose labels arrived by the deadline:

$$C = \frac{\text{eligible sampled events labeled by the deadline}} {\text{eligible sampled events due by the deadline}}.$$

The numerator counts labels that arrived on time; the denominator counts every sampled event whose label was due.

If user feedback or downstream effects arrive late, define a label delay and cohort events by the time they became eligible, not merely by the time their label appeared. Outcomes still unresolved at analysis time are censored; show their count and use a method appropriate to delayed outcomes instead of quietly dropping them. When coverage falls below its floor, the quality state is unknown. A broken judge or logging path is a telemetry failure, not evidence that the service improved.

Turn the estimate into an operating policy

An SLO is a target, not a confidence interval. Its document should name the indicator, objective, population, slices, window, and error-budget policy. The uncertainty method then determines whether the available evidence is strong enough to make a decision. For example, a team might set a 30-day semantic success objective and page only when an uncertainty-aware estimate and minimum sample requirement imply rapid budget consumption.

Sampled labels are evidence about eligible served events; they are not the currency of the error budget. Counting one deliberately oversampled incident as one user event would distort both the numerator and the budget. Use the weighted population estimate, retain the design information, and expose effective sample size and coverage beside it.

For high-volume services, multiwindow alerts that compare more than one burn rate can combine a fast signal for severe consumption with a slower signal that resists noise (Thurgood 2018). Low-traffic services need longer windows, synthetic or batch checks, and explicit minimum-evidence states; a ratio based on two labels should not page as though it were precise. Hard safety, authorization, and effect-integrity boundaries need direct alerts regardless of the aggregate error budget. Finally, an SLO decision is not automatically a release gate. The deployment policy in Chapter 89 may require stronger evidence, non-inferiority to a control, and complete guardrails before exposure widens.

Lower-layer constraint: model task reliability

A multi-step agent succeeds only if the required sequence of conditions holds. For step-success events $S_1,\ldots,S_n$, the probability chain rule gives

$$P(S_1,\ldots,S_n) = \prod_{k=1}^{n} P(S_k \mid S_1,\ldots,S_{k-1}).$$

Each factor is the probability that step $k$ succeeds given the successful history before it. The familiar $p^n$ curve is a special case. Here $p$ denotes one identical per-step success probability. The shortcut assumes a fixed number of steps, independent outcomes, and a workflow in which every failed step is fatal and there is no recovery. Real agents often violate all four assumptions: later steps depend on earlier context, task length varies, difficult tasks have correlated failures, and validators can trigger a repair. Use the chain rule or measured end-to-end completion rate for a real claim; use $p^n$ to understand the idealized compounding mechanism.

import math

def iid_task_success(step_success: float, required_steps: int) -> float:
    assert 0.0 <= step_success <= 1.0
    assert required_steps >= 0
    return step_success ** required_steps

def attempts_success(single_attempt: float, attempts: int) -> float:
    """Independent attempts at one recoverable operation."""
    assert 0.0 <= single_attempt <= 1.0
    assert attempts >= 1
    return 1.0 - (1.0 - single_attempt) ** attempts

p = 0.99
for n in (1, 10, 50, 100):
    print(f"p={p:.2f}, n={n:3d}: {iid_task_success(p, n):.4f}")

assert math.isclose(iid_task_success(0.99, 50), 0.6050060671,
                    rel_tol=1e-9)
assert attempts_success(0.8, 2) == 0.96

Reducing unnecessary steps and improving conditional step success can raise task reliability. A checkpoint has a different effect: by itself it does not make the next step more likely to succeed. It reduces lost work and restart cost. It raises completion probability only when paired with a preserved state, a valid retry or repair path, and enough deadline and attempt budget. Measure the whole policy, including recovery exhaustion, rather than substituting the shortest segment length for $n$.

Retry by failure class, not by hope

A retry is another request against the same capacity. It is justified only when the failure class, side-effect semantics, and remaining budget make another attempt useful. Classify before retrying:

Propagate one end-to-end deadline. Assign one retry owner for each call path so nested SDK, gateway, model, and agent loops do not multiply attempts. Give that owner a retry budget, exponential backoff, jitter, and a maximum attempt count. Retries consume capacity during the exact period when a dependency may be overloaded; limiting them is part of availability, not merely cost control (Brooker 2019).

An idempotency key helps only within its declared contract. The caller and service must agree on key scope, retention period, payload identity, concurrent duplicate handling, storage durability, and response replay. The record of the key and the intended mutation must be committed with suitable atomicity (Featonby 2021). Even then, idempotency is not exactly-once execution: effects outside the protected transaction, expired keys, or a downstream system that does not carry the identity can still duplicate work. HTTP's definition likewise concerns the intended effect of repeated requests; it does not promise identical responses or universal automatic retry safety (Fielding et al. 2022).

Make fallback a qualified change of contract

A fallback is not safe merely because it is different. It must satisfy fallback compatibility: the alternate path accepts the same relevant input, preserves authorization and tenant boundaries, produces an outcome the caller can interpret, and stays within its latency, freshness, policy, and effect limits. A cached answer can be stale. A template can omit required context. An abstention can fail to reach a human. Every terminal path can fail, so the design goal is a narrower qualified outcome with an explicit failure mode, not an imaginary infallible last link.

Graceful degradation deliberately narrows the service promise: fewer tools, older but bounded data, a partial answer, or read-only operation. The degraded mode must be prequalified, visible to the user and telemetry, and prohibited when it would change authority or hide unsafe uncertainty. "Answer from the prompt when retrieval is down" is not graceful if the product promises current facts. Load shedding can be safer than accepting work that will miss its deadline, but test the overload path before it is needed (Forero Cuervo 2017).

Abstention is a product outcome, not an evaluation afterthought. A selective system trades accepted-answer risk against coverage (El-Yaniv and Wiener 2010). Its risk--coverage contract should report at least:

• error or harm rate among accepted answers;
• coverage, the share of eligible requests answered under the full contract;
• abstention and handoff rate, latency, and eventual resolution;
• every measure by consequential traffic slice.

Increasing abstention can make accepted answers look better while abandoning more users. Report both sides of the trade.

Redundancy helps only across relevant failure domains

Two outputs are not independent merely because two calls were made. They may have a shared provider, shared model, shared prompt, shared retrieval index, shared tool, and shared judge. Correlated design faults are a longstanding limit of multi-version redundancy (Knight and Leveson 1986). A second sample improves reliability only through conditional diversity: given the request and known state, it has a useful chance of avoiding the first path's failure.

Maintain a failure-domain matrix for each fallback or ensemble member. Record model family, provider and region, prompt lineage, retrieval source, tool dependencies, validator or judge, credentials, and serving infrastructure. Then inject the shared failures. Diversity that exists only in the model name does not protect against a poisoned index, expired credential, broken rubric, or common regional dependency.

Bound dependencies and overload

Model servers, tools, retrieval systems, policy engines, and judges are all distributed dependencies. Any of them may vary, delay, partially commit, or fail. Put each call behind a timeout derived from the end-to-end deadline; isolate capacity with a bulkhead; reject excess work with load shedding; and use a circuit breaker where failing fast is safer than continuing to add load (Nygard 2018).

A production breaker needs more than three state names. Define the measured failure window, minimum sample count, qualifying errors, trip threshold, open duration, probe concurrency, breaker scope, and recovery criterion. One successful probe may prove reachability without proving capacity or semantic recovery. Close only after the stated recovery criterion is met, and preserve enough probes to detect a dependency that remains unhealthy.

Keep latency hedging separate from quality redundancy. A latency hedge sends an equivalent read-only or idempotent request after a delay and accepts the first qualified completion; it should cancel the loser when possible and carry a capacity budget because hedges create load amplification (Dean and Barroso 2013). Quality redundancy obtains multiple outcomes and applies a validator, comparison, or aggregation rule. The fastest response is not necessarily the best response, so "first wins" is not a semantic-quality policy. Never hedge an unprotected side effect.

Engineer reproducibility without confusing it with reliability

Deterministic replay is valuable for debugging, regression tests, caching, and forensics. Treat it as an exact-replay contract over the full system: model revision and weights, request and prompt, sampling configuration and seed, retrieval snapshot, tool responses, policy configuration, serving runtime, kernel implementation, hardware, and batch conditions. Pinning temperature and a seed while allowing the prompt, index, tools, or provider runtime to move is not such a contract.

Even greedy decoding can vary when numerical kernels depend on dynamic batch composition (He 2025). Serving runtimes can offer batch-invariant kernels for supported environments (vLLM 2026), but the exact mechanism and cost are implementation details to benchmark for the pinned release. Exact replay does not prove correctness, safety, or availability. It only proves that the same recorded conditions reproduce the same computation.

This is not a choice between determinism and verification. Strong systems use deterministic invariants at the boundaries: schema, authorization, quotas, effect identity, and provenance. They use statistical evidence for semantic outcomes. They may also preserve a replayable incident fixture. Each mechanism answers a different question. The serving mechanics in Chapter 31 determine which replay inputs and capacity controls must be recorded.

Operate quality failures as incidents

A quality incident begins when user-visible outcomes, a consequential slice, or the measurement system violates its contract. If measurement coverage collapses, mark the semantic state unknown; do not wait for a falsely reassuring pass rate. The runbook should:

1. Confirm and scope. Check the release fingerprint, traffic assignment, slices, judge version, label delay, and telemetry failure modes.
2. Contain harm. Freeze promotion, disable the implicated tool or route, reduce authority, switch to a qualified fallback, shed work, or rollback the complete release bundle.
3. Preserve evidence. Retain request and release identifiers, policy decisions, retrieval provenance, tool receipts, raw outcomes, judge records, and human overrides subject to privacy limits.
4. Repair and validate recovery. Reproduce the failure, test the proposed correction offline, restore limited exposure, and validate recovery against both semantic and deterministic guardrails before widening traffic.
5. Learn. Add the failure to the corpus, update the failure-domain matrix, and decide whether the SLI, policy, or fallback contract missed it.

Exercise the runbook with scenario tests: judge outage and evaluator drift, provider degradation, stale retrieval, a rate-limit storm, a timeout after an unknown commit, duplicate effect delivery, expired idempotency state, correlated fallback failure, missing labels, and overload while the breaker probes. The test should assert the user-visible outcome and recovery action, not merely that an internal component changed state.

The reliability release record

Reliability becomes operational when its evidence travels with the release. The output of this chapter is a reliability release record containing:

• the release fingerprint and owners;
• each SLI specification, implementation, eligible population, slices, window, objective, exclusions, and missing-data rule;
• sampling policy, inclusion probabilities, judge and rubric versions, human audit results, uncertainty method, and measurement coverage;
• retry owners and budgets, deadline propagation, idempotency contracts, fallback compatibility, degraded modes, and abstention policy;
• the dependency and failure-domain matrix, breaker configuration, capacity limits, and common-mode analysis;
• scenario-test results, alert and burn-rate policy, incident runbook, rollback target, and recovery evidence.

That record connects the deployment bundle from Chapter 89 to the operating promise. It also makes disagreements inspectable. Teams may reasonably choose different rubrics, sampling rates, abstention policies, or recovery thresholds. They should not disagree unknowingly because the judge, traffic population, or failure assumptions were left implicit.

Reliability for a nondeterministic system is therefore neither byte equality nor faith in a large sample. It is a traceable contract: define the outcome, observe the right population, quantify uncertainty and missingness, constrain recovery, and preserve deterministic boundaries around effects. Variability is then something the system measures and manages, not an excuse for an unfalsifiable promise.

Further reading