Operating Contracts: SLOs, Cost, Incidents, and Tenancy

A stack becomes infrastructure only when a team can say what it promises, what it costs, how it fails, who shares it, and which record will be trusted after the failure. Everything before this point built pieces of that answer. Chapter 88 named the seams, Chapter 89 made release a statistical promotion problem, Chapter 90 moved reliability from exact responses to judged distributions, Chapter 91 placed human approval in the runtime loop, and Chapter 92 turned production traces into the next round of evidence. The missing object is the operating contract tying those pieces together.

The old production playbook still matters. Service level indicators and service level objectives give a service a measurable promise (Beyer et al. 2016). When the system is under stress, incident command separates decision-making from hands-on repair (Beyer et al. 2016). FinOps treats cost as a shared engineering, finance, and product discipline rather than an accounting report after the fact (FinOps Foundation 2026). Multi-tenancy, for its part, turns fairness, isolation, and chargeback into questions of platform design (Kubernetes 2026). AI changes the payload these disciplines carry. A service that is reachable and fast can still be fluent and wrong. Cost is no longer just CPU hours; it now meters tokens, GPU seconds, judge calls, retrieval, reranks, and human review. An incident need not be an outage at all: it might be a quality regression, a policy failure, a cost runaway, a grounding failure, or a tenant leak. The contract must cover all of them.

The Contract Surface

An AI product has at least five contracts. They should be written, instrumented, and enforceable at the same seams that route traffic.

Figure 93.2 draws the contract surface as a control plane rather than a document. A request enters through the gateway. The gateway does not merely forward it. It attaches tenant identity, budget, routing policy, trace ID, and service target. The agent runtime, retrieval layer, sandbox, eval harness, and serving engine all inherit that envelope. When something goes wrong, the same envelope lets the operator answer the first hard questions: who was affected, what changed, what was spent, what boundary was crossed, and which trace can reproduce it.

SLOs for Semantic Systems

Traditional SRE asks which user-visible behaviors matter and how to measure them (Beyer et al. 2016). In a model-backed system the answer has three families. The first two are familiar: the service must be available and it must be fast enough. The third is semantic quality. A response can arrive quickly and still violate the product's promise.

A quality SLI is a sampled and judged pass rate. For a window of $n$ sampled production events, let $s_i = 1$ when the response passes the task rubric and $s_i = 0$ otherwise:

$$\hat q = \frac{1}{n}\sum_{i=1}^{n}s_i .$$

Here $\hat q$ denotes the observed quality pass rate in the window, $n$ is the number of sampled events, and each $s_i$ is the judged outcome for one event. Because the sample is finite, the operating decision should use a lower confidence bound, not the point estimate. With a target $q^\star$, a release or route is healthy only when the lower bound stays above $q^\star$. That single detail keeps the system from treating a lucky thin sample as evidence. Chapter 48 supplies the measurement discipline; Chapter 90 supplies the nondeterministic service model. The contract connects them to pager rules and release gates.

The interactive curve below shows the operational trade. The x-axis is extra evaluation, fallback, or review spend. The y-axis is residual live risk. The slider sets how quickly the risk falls. A brittle system has a long half-life: each extra dollar removes little risk. A well-instrumented system has a short half-life: targeted evals and fallbacks quickly remove the failures that matter.

import math

def wilson_lower(passes, n, z=1.96):
    if n == 0:
        return 0.0
    phat = passes / n
    denom = 1 + z * z / n
    center = phat + z * z / (2 * n)
    margin = z * math.sqrt(phat * (1 - phat) / n + z * z / (4 * n * n))
    return (center - margin) / denom

target = 0.92
for n, passes in [(50, 48), (200, 188), (1000, 935)]:
    point = passes / n
    lower = wilson_lower(passes, n)
    decision = "ship" if lower >= target else "hold"
    print(f"n={n:4d}, point={point:.3f}, lower95={lower:.3f}, target={target:.2f}, decision={decision}")

This framing also prevents an SLO from becoming a slogan. "Helpful answers" is not an SLO. "At least 92 percent of sampled support answers pass the grounded resolution rubric over a rolling day, with lower 95 percent confidence bound above 90 percent, and p95 latency below four seconds" is an SLO a system can enforce. The exact numbers are product choices; the shape is not.

Cost Governance as a Runtime Control

Cost governance has to sit in the request path. A monthly report arrives too late to stop a runaway agent, a misrouted judge, or a prompt template that quietly tripled output length. FinOps is useful here because it defines cost as a shared operating discipline across engineering, finance, and product, with trade-offs among speed, cost, and quality made explicitly (FinOps Foundation 2026). FOCUS, the FinOps open specification for cost and usage data, matters because normalized cost data lets AI spend sit beside cloud, SaaS, and data-center spend rather than living in a vendor-specific export (FinOps Open Cost and Usage Specification 2026). The token meters below are already in the schema: FOCUS's virtual-currency columns carry token purchases and burn-down, and the 1.4 revision ratified in June 2026 scopes native per-model token consumption for the release after it.

The runtime ledger should record the unit economics of a task, not only the bill:

$$C = \sum_r \left( p_{\text{in}}m_{\text{in},r} + p_{\text{out}}m_{\text{out},r} + c_{\text{gpu}}t_{\text{gpu},r} + c_{\text{judge}}j_r + c_{\text{review}}h_r + c_{\text{store}}g_r \right).$$

In words, each request $r$ contributes input-token, output-token, GPU-time, judge-call, human-review, and storage terms. The coefficients are the prices or internal unit costs attached to those meters. The point is not that every term will be known exactly. The point is that the contract names the meter: input tokens, output tokens, GPU time, judge calls, human review minutes, and storage. Once the meter is named, the gateway can enforce budgets before the bill arrives:

• route easy tasks to cheaper models and hard tasks to stronger ones;
• cap tool loops by task class and tenant;
• reject or defer traffic when a budget is exhausted;
• allocate cached-input discounts and self-hosted capacity to the tenants that create the demand;
• price eval, judge, and review work as part of the product, not as invisible platform overhead.

The difficult part is political rather than mathematical. The model team is pushing for more test-time compute to raise quality. Product would rather have lower latency. Finance wants a hard cost ceiling. The operating contract makes that conflict explicit: it states which target wins under which condition, and which person can change the threshold.

Incident Classes

Incident management exists because humans under stress otherwise freelance, duplicate work, and lose state. Google's SRE practice makes the point in generic terms: separate incident command, operational work, communication, and planning, and keep a live incident state document (Beyer et al. 2016). AI systems need the same roles, but the incident taxonomy is broader than "the service is down."

Figure 93.4 shows the incident loop. Detection is only the first step. The operator has to command, mitigate, communicate, preserve evidence, and turn the failure into either a regression test or a known accepted risk. Sculley et al.'s warning about hidden technical debt in ML systems is relevant here: undeclared consumers, feedback loops, and glue code create failures that are hard to see until the system is already in production (Sculley et al. 2015). The incident loop is how those hidden dependencies become declared.

The last edge is where many organizations fail. They hold the postmortem, but the system remains unchanged. In AI infrastructure a postmortem that does not add an eval case, a budget guard, a route rule, a tenant isolation control, a data quality check, or a human approval gate is only prose. It has not modified the machinery.

Multi-Tenancy and Noisy Neighbors

AI multi-tenancy is more than Kubernetes namespaces. A tenant shares GPUs, KV cache, batch slots, vector indexes, judge capacity, tool servers, sandboxes, and sometimes humans in review queues. Kubernetes documentation names the familiar isolation tools: namespaces, access control, quotas, sandboxing, node isolation, API priority, and QoS (Kubernetes 2026). AI inherits all of them and adds semantic boundaries.

The tenant contract must answer four questions.

1. Capacity: can one tenant's long-context or agentic workload push another tenant over the latency SLO?
2. Data: can one tenant's prompt, retrieval chunk, cached prefix (the key-value cache a serving engine keeps for a shared prompt prefix and replays to skip recomputation), trace, or label enter another tenant's context or training stream?
3. Authority: can one tenant's agent reach a tool, credential, or egress path meant for another tenant?
4. Accounting: can the platform explain who consumed tokens, cached tokens, GPU seconds, judge calls, review minutes, and storage?

Isolation strength should match blast radius. A low-risk internal assistant may share a cluster with namespace and quota boundaries. A customer-facing agent that runs untrusted code should use a sandbox or a tenant-dedicated node pool. A regulated customer may need a separate index, separate encryption keys, separate model route, and separate retention policy. The mistake is treating tenancy as a single platform toggle. It is a bundle of isolation choices, each priced differently.

Governance Without a Separate Book

Governance appears elsewhere in the book as law, safety, provenance, and oversight. The operating contract is where it becomes executable. NIST's AI Risk Management Framework frames AI risk management as a voluntary process for trustworthiness across design, development, use, and evaluation (National Institute of Standards and Technology 2023), with a generative AI profile that calls out risks unique to generative systems (National Institute of Standards and Technology 2024). The operational translation is concrete:

• a policy requirement becomes a gateway rule or approval gate;
• a data requirement becomes a provenance and retention field on traces;
• a transparency requirement becomes a citation, uncertainty, or disclosure surface;
• a safety requirement becomes an eval slice, sandbox boundary, or tool allowlist;
• an accountability requirement becomes an incident role, audit record, and owner.

Microsoft's GenAIOps guidance makes the same practical point from a workload perspective: AI workloads are nondeterministic and require processes for drift, monitoring, governance, deployment, and continuous operation (Microsoft 2024). In this book's terms, governance is not a review board floating above the stack. It is a set of constraints compiled into routing, evaluation, data, incident, and tenancy mechanisms.

Trade-Offs

Operating contracts introduce overhead. They add metadata to every request, state to every incident, meters to every route, and evidence requirements to every release. A team can overdo it: a small prototype can drown under controls meant for a regulated multi-tenant platform. The contract should grow with blast radius. The first contract can be simple: model pin, budget cap, trace ID, rollback rule, and one quality eval. What cannot be deferred is deciding where the contract will live. If it lives in chat messages and spreadsheets, the system cannot enforce it.

Infrastructure Means Being Operated

The title of this book uses infrastructure in a demanding sense. Infrastructure is not only a stack that can be assembled. It is a stack that can be promised, metered, shared, repaired, audited, and improved after it fails. The operating contract is the mechanism that gives those verbs a place in the system.

Capability is what the contract produces: can the system perform the task under the conditions users actually bring. Efficiency is its budget, the test of whether the system spends tokens, GPU seconds, judge calls, and human attention where they change outcomes. Trust is the evidence the contract leaves behind, and it asks the hardest question: can the team explain why it shipped, who it affected, what it cost, where it failed, and what changed after the failure. A stack without those answers may be powerful, but it is not yet infrastructure.

Further reading