Operating Contracts: SLOs, Cost, Incidents, and Tenancy

A production system needs more than a deployment manifest. It needs a versioned operating contract that connects a user promise to measurement, enforcement, decision rights, and evidence. The contract says which traffic is covered, what counts as acceptable service, how much work may cost, which tenant boundaries must hold, what happens during an incident, and who may change those decisions. If a statement cannot change admission, routing, release, incident, or review behavior, it is guidance rather than an operating contract.

Earlier chapters supply the mechanisms. Chapter 88 defines the seams, Chapter 89 governs releases, Chapter 90 defines observable outcomes, Chapter 91 governs human decisions, and Chapter 92 produces qualified evidence. This chapter binds those pieces into one operable policy without pretending that one metric or one control can cover every task.

Site reliability engineering provides the basic language of service level indicators, objectives, error budgets, incident roles, and learning from failure (Beyer et al. 2016). FinOps adds shared accountability for technology value and cost across engineering, finance, product, and business teams (FinOps Foundation 2026). Multi-tenancy adds isolation and fairness. AI systems make the contract wider because a response can arrive on time and still be wrong, a tool can create an external effect, a judge can drift, and one tenant can consume shared model, retrieval, tool, or human-review capacity.

Write the contract as an executable schema

The first artifact is not a dashboard. It is a reviewed schema whose fields have owners and enforcement points.

Unknown values are states, not empty cells. If the service cannot identify the contract version, verify the tenant claim, determine whether traffic is eligible, or locate the required measurement source, the contract defines whether to deny, quarantine, use a qualified safe mode, or escalate. A default assembled during an incident is still a policy choice and needs an owner.

Figure 93.1 shows how the artifact reaches runtime. The control plane validates and publishes an immutable contract version. The request path carries only the minimum claims needed at each boundary. Telemetry and evidence inform a decision, but they do not rewrite live policy directly. A reviewed change creates the next contract release.

The runtime context should contain an opaque tenant identifier, contract version, task class, reservation identifier, service class, policy version, and trace identifier when those fields are required. Do not copy user content, credentials, or broad authorization grants into that context. Each retrieval, tool, model, storage, and review boundary must re-authorize the operation from verified claims. Trace context does not authorize an action; it only helps correlate evidence.

Separate SLI, SLO, and SLA

The terms are related but not interchangeable (Beyer et al. 2016):

• A service level indicator (SLI) is a quantitative measure of delivered service under a precise definition.
• A service level objective (SLO) is an internal target or target range for an SLI over a stated window, plus the policy used when the target is at risk.
• A service level agreement (SLA) is an external commitment that may define a remedy when service misses the agreement. Legal and commercial review belongs around an SLA; an internal dashboard does not create one by itself.

For an event-based SLI, let $W$ be a measurement window and $E_W$ the set of eligible events in that window. Let $G(e)$ be the good-event predicate for event $e$. Then

$$\operatorname{SLI}_W = \frac{\sum_{e \in E_W}\mathbf{1}[G(e)]}{|E_W|}.$$

Here, $W$ is the declared measurement window, $E_W$ is the eligible event set, $e$ is one eligible event, $G(e)$ is true only when that event meets the complete user-visible condition, $\mathbf{1}[G(e)]$ is one for a good event and zero otherwise, and $|E_W|$ is the number of eligible events. The contract must say how cancelled, timed out, duplicated, retried, missing, and unknown outcomes enter the denominator. Counting latency only for successful responses, for example, hides slow failures.

A good-event predicate can combine directly observable conditions, such as "completed before the deadline and committed no unauthorized effect." Do not average distinct task classes merely because they share an endpoint. A fast chat response, a long report, and a tool-using workflow can require different eligibility and good-event definitions. The SRE Workbook recommends user-centric SLIs, explicit ownership, documented calculation, and an error-budget policy (Thurgood and Ferguson 2018).

Semantic quality is a separate measurement contract

Many semantic outcomes are not known at request completion. A grounded answer may need a probability sample, an inclusion probability, a versioned rubric, judge version, human calibration, label delay, confidence interval, and measurement coverage. Those are the sampling and annotation obligations from Chapter 92, not fields that disappear when the estimate reaches a dashboard.

The delayed quality estimate can govern releases, create a ticket, or declare an incident when its decision rule is met. It cannot provide an immediate page for an individual request whose label does not yet exist. Immediate paging should use observable safety, authorization, integrity, availability, latency, or cost signals. A delayed quality process needs an explicit maximum label delay and a policy for judge outage, missing labels, and changed sampling weights.

Spend an error budget deliberately

Suppose the objective is a good-event fraction $S^*$, where $b$ is the allowed bad-event fraction. If $q_W$ is the observed bad-event fraction in an alert window $W$, its burn rate is $\beta_W$:

$$b = 1 - S^*, \qquad \beta_W = \frac{q_W}{b}.$$

Here, $S^*$ is the objective, $b$ is its error-budget fraction, $q_W$ is the bad-event fraction observed in window $W$, and $\beta_W$ is the rate at which that window consumes budget relative to the objective. A sustained burn rate of one uses the budget at exactly the planned rate. A larger value exhausts it sooner.

Alerting on a confidence bound from a small rolling sample is not a general pager policy. For directly observed event SLIs, multiwindow, multi-burn-rate rules can distinguish a fast page from a slower ticket and confirm that budget is still burning when the alert fires (Thurgood et al. 2018). Thresholds and windows are service decisions, not universal constants. Low-traffic services need special treatment because one event can consume a large fraction of their budget. Use synthetic probes, related task groupings, lower-impact service design, or manual case review only when those choices match user impact.

An error-budget policy must say what changes when budget is healthy, threatened, or exhausted. Possible actions include slowing rollout, raising review, disabling an optional feature, prioritizing reliability work, or requiring an exception. The action should be reversible and should not silently route work to an unqualified model or unsafe fallback.

Control cost before execution

A billing report can explain past spend, but it arrives too late to stop a loop already running. Runtime cost governance follows five steps: estimate, reserve, admit, meter, and reconcile. The budget hierarchy may include a task, request, tenant, product, environment, and billing period. A reservation must count at every applicable level so concurrent requests cannot each spend the same remaining budget.

Cost belongs to the accepted task, including all attempts. Let $A_t$ be the set of attempts for task $t$. Its total cost is

$$\begin{aligned} C_t &= \sum_{a \in A_t} \bigl( C^{\mathrm{model}}_a + C^{\mathrm{retrieval}}_a \\ &\qquad + C^{\mathrm{tool}}_a + C^{\mathrm{judge}}_a \bigr) \\ &\quad + C^{\mathrm{human}}_t + C^{\mathrm{storage}}_t. \end{aligned}$$

Here, $t$ is one user task, $A_t$ contains its original attempt and every retry or fallback, $a$ is one attempt, the four $C_a$ terms are model, retrieval, tool, and judge cost for that attempt, and the two $C_t$ terms are task-level human review and storage cost. Every term uses the price version and currency recorded when the meter was reconciled. A late charge updates the ledger rather than rewriting the historical price context.

For a task set $T$, cost per accepted task is

$$U_T = \frac{\sum_{t \in T} C_t} {\sum_{t \in T}\mathbf{1}[\operatorname{accepted}(t)]}.$$

Here, $T$ is the measured task set, $C_t$ is total task cost, $\operatorname{accepted}(t)$ is the versioned acceptance predicate, and $U_T$ is unit cost per accepted task. Report the denominator and reasons for rejection; otherwise a cheaper system can appear efficient by failing more work.

Before admission, reserve an estimate under the active contract. After execution, reconcile observed usage, release unused reservation, record any overrun, and route unknown usage or an absent price version to an exception queue. A cheaper route is a qualified fallback only if it still meets the task's security, quality, latency, and tenancy requirements. Cost alone cannot qualify it.

This dependency-free ledger illustrates idempotent reservation and reconciliation with integer cost units. Production ledgers also need durable transactions and a defined recovery rule for unknown commit.

class BudgetLedger:
    def __init__(self, contract_version, tenant_id, budget):
        self.contract_version = contract_version
        self.tenant_id = tenant_id
        self.available = budget
        self.spent = 0
        self.reservations = {}

    def reserve(self, reservation_id, estimate):
        prior = self.reservations.get(reservation_id)
        if prior is not None:
            if prior["estimate"] != estimate:
                raise ValueError("reservation payload changed")
            return prior["state"] == "reserved"
        if estimate < 0 or estimate > self.available:
            return False
        self.available -= estimate
        self.reservations[reservation_id] = {
            "estimate": estimate,
            "actual": None,
            "state": "reserved",
        }
        return True

    def reconcile(self, reservation_id, actual):
        item = self.reservations.get(reservation_id)
        if item is None:
            raise KeyError("unknown reservation")
        if item["state"] == "settled":
            if item["actual"] != actual:
                raise ValueError("settlement payload changed")
            return
        if actual < 0:
            raise ValueError("negative actual cost")
        self.available += item["estimate"] - actual
        self.spent += actual
        item.update(actual=actual, state="settled")


ledger = BudgetLedger(contract_version="contract-v7", tenant_id="tenant-a", budget=100)
assert ledger.reserve(reservation_id="task-42", estimate=30)
assert ledger.reserve(reservation_id="task-42", estimate=30)  # no double reservation
assert ledger.available == 70
ledger.reconcile(reservation_id="task-42", actual=25)
ledger.reconcile(reservation_id="task-42", actual=25)  # idempotent settlement
assert not ledger.reserve(reservation_id="task-43", estimate=80)
print(f"spent={ledger.spent} available={ledger.available}")

FinOps provides a cross-functional operating framework for technology value and financial accountability (FinOps Foundation 2026). FOCUS provides a vendor-neutral schema for billing data, not a live admission controller. FOCUS 1.4 was ratified on 4 June 2026; earlier FOCUS support for virtual currencies can represent token purchase and consumption in billing records (FinOps Open Cost and Usage Specification 2026). Provider adoption and delivery latency still vary. The runtime ledger therefore reconciles with FOCUS or another billing source without waiting for it to protect the current request.

Enforce tenancy one boundary at a time

A tenant is a scoped security, capacity, data, and accounting principal, not a string copied from a request header. Kubernetes documents control-plane and data plane isolation, quotas, network policy, sandboxing, node isolation, and fairness controls, while warning that a namespace alone requires other controls to provide isolation (Kubernetes 2025). An AI service adds model and semantic state to those boundaries.

Fairness needs more than a rate limit. Test quota, reservation, concurrency, priority, queue discipline, backpressure, and load shedding together under contention. A noisy-neighbor test should replay representative workloads for Tenant A and Tenant B, then compare admitted volume, deadline misses, and quality with and without contention. The contract states which work may wait, degrade, or be rejected. Silent priority inversion is a failure even if aggregate throughput rises.

Isolation is a vector. Dedicated nodes may reduce compute sharing while leaving the control plane, model endpoint, index, keys, evidence system, or humans shared. Sandboxing an untrusted tool does not partition retrieval. A separate index does not constrain egress. Choose and test each boundary according to the threat model and required blast radius.

Run incidents as controlled state transitions

An AI incident is any event that requires coordinated action to limit material service, safety, security, privacy, cost, or tenant harm. NIST SP 800-61 Revision 3 integrates preparation, detection, response, and recovery into ongoing risk management rather than treating response as an isolated emergency activity (Nelson et al. 2025). SRE practice adds explicit command, operations, communications, planning, and live state (Beyer et al. 2016).

The minimum roles are an incident commander who owns priorities, an operations lead who coordinates changes, a communications lead who owns stakeholder and tenant updates, and an evidence owner who preserves the decision record. One person may hold several roles in a small incident, but the responsibilities stay explicit. Track detection time, acknowledgement time, containment time, restoration time, and final verification separately. A single "resolution time" hides where the process stalled.

Figure 93.2 makes the state transition explicit. Communication and evidence preservation begin at declaration and continue through recovery. Automation may execute a prequalified containment action, but it must retain the before state and execution acknowledgement. An unknown commit remains a distinct state until reconciliation.

The review distinguishes root cause from contributing conditions. Every corrective action has an owner, deadline, and verification method. It can add an evaluation case, budget guard, route rule, isolation control, data check, or approval boundary. It may also record an accepted risk or a justified no-change decision when evidence shows that a modification would not improve the outcome. Either decision is versioned and reviewable.

Release changes to the contract

An executable policy can fail like code. Its lifecycle is draft, validate, shadow, canary, activate, monitor, rollback, and deprecate. Validation checks schema, ownership, indicator semantics, dependency compatibility, authorization, budget arithmetic, tenancy boundaries, runbooks, and evidence sinks. Shadow mode compares proposed decisions without enforcing them. Canary limits the blast radius. Activation records the effective version, and rollback names a tested prior version rather than reconstructing one during an incident.

An emergency exception names its authority, scope, reason, start time, maximum impact, evidence requirement, and expiry. It expires automatically. Keeping it requires the normal review path. An exception must not erase the underlying event, bypass tenant isolation, or prevent cost reconciliation.

Hidden consumers and configuration dependencies make ML systems expensive to change safely (Sculley et al. 2015). Record every system that reads the contract: gateway, scheduler, model router, retriever, tool policy, sandbox, evaluation, incident automation, billing, and reporting. A consumer that cannot report its active version is not ready for coordinated rollout.

Compile governance into bounded controls

The NIST AI Risk Management Framework is voluntary and is not a certification (Tabassi 2023). Its Generative AI Profile identifies generative-AI risks and candidate actions (National Institute of Standards and Technology 2024). These sources can help teams map a risk to an owner, control, measurement, and evidence requirement. An operating contract does not replace legal analysis, security engineering, privacy governance, employment duties, or a negotiated SLA.

Microsoft's GenAIOps guidance is a vendor-specific implementation reference for monitoring, drift, deployment, governance, and lifecycle operations (Microsoft 2024). It can supply examples, not universal requirements. The contract should point from each external obligation or internal policy to a specific enforcement point and test, while retaining the source and reviewer who made that translation.

Decision rights close the organizational loop. The service owner proposes the promise; measurement owners define reproducible indicators; product and business approvers accept objectives and budget policy; security and privacy owners approve their boundaries; on-call roles operate incidents; and a named authority accepts residual risk. Separation of duties is proportionate to impact, but an emergency operator should not silently become the permanent policy approver.

Test the negative paths before release

The end-to-end suite should exercise more than a happy request:

• a missing tenant claim, spoofed claim, and unknown contract version;
• an ineligible event, duplicated event, metric pipeline outage, delayed label, changed judge version, and missing measurement source;
• a duplicate reservation, changed reservation payload, concurrent overspend, retry bypass, late cost, unknown usage, and changed price version;
• an exhausted budget, qualified fallback unavailable, and an emergency exception that reaches expiry;
• a cross-tenant cache hit, cross-tenant retrieval result, wrong encryption key, tool credential leak, evidence export, and review-queue mix-up;
• a noisy-neighbor load test across model, retrieval, tool, judge, and human capacity;
• an incident with failed automation, unknown commit, evidence-store failure, stale contact, and contract rollback;
• shadow and canary decisions that differ from the active contract.

The operating contract release record is the handoff artifact. It contains:

• contract identity, version, state, effective time, expiry, owner, approver, scope, dependencies, and rollback target;
• each user promise, eligible population, indicator implementation, objective, error-budget and alert policy, and missing-data treatment;
• cost meters, price version, budget hierarchy, reservation and reconciliation evidence, allocation policy, and qualified fallbacks;
• tenant identity and isolation profile across capacity, cache, data, keys, tools, egress, evidence, review, and billing;
• incident severity, roles, runbooks, communication duties, recovery criteria, immutable evidence location, and open corrective actions;
• validation, shadow, canary, negative-path, load, rollback, and browser-visible evidence, plus every active exception and its expiry;
• the release decision, decision rights exercised, residual risks, and next review date.

Useful operational reporting follows the same contract. Show SLO compliance and burn rate by declared service class, cost per accepted task, reservation error, late or unattributed cost, incident clocks, recurring mechanisms, corrective action verification, isolation test results, noisy-neighbor impact, rollout status, and exception age. Protect low-volume tenant privacy and do not present a slice with inadequate measurement coverage as a precise comparison.

Infrastructure is a maintained promise

An operating contract does not guarantee that a system is reliable, affordable, safe, or fair. It makes those claims specific enough to test and those conflicts specific enough to decide. It also leaves a record when the decision was wrong.

That is the final step from a collection of AI components to infrastructure. The system has a declared service, a measured population, bounded authority, owned cost, tenant-specific isolation, controlled incident response, and a release record connecting each promise to running machinery. The contract will change as users, models, prices, policies, and risks change. Operating the system means changing it through the same visible, testable process used to publish it.

Further reading