Wiring the Stack

A collection of compatible products is not yet a system. It becomes one only when every boundary has a contract, every behavior-bearing component has an identity, and a failed release has a tested way back. The deliverable of this chapter is therefore an integration release: a versioned statement of what is connected, what each connection means, who owns it, and what evidence makes the assembled behavior safe to release.

Start with the release decision, not a shopping list. Name the capability being released, its users and tenants, its permitted data classes and effects, its quality and operational thresholds, and its rollback trigger. Then freeze a system fingerprint containing the model revision, prompt revision, retrieval snapshot, tool schema, policy revision, router configuration, telemetry schema, and executable artifact digests. A model name alone cannot identify the system that produced an answer.

This contract-first view has deep roots. Fielding described networked systems in terms of constrained interfaces rather than particular products (Fielding 2000). Sculley and colleagues later showed how glue code, configuration, and undeclared consumers accumulate technical debt around machine-learning systems (Sculley et al. 2015). Breck and colleagues turned that lesson into production-readiness tests (Breck et al. 2017). The practical chapters in this part supplied the components; this chapter makes their composition testable (Chapter 1).

Freeze the integration release

An integration release owns two records. The system manifest identifies the assembled system. The boundary contract defines each edge between a producer and a consumer. Neither record may contain mutable aliases such as latest.

The manifest itself should be signed or otherwise integrity-protected. Record a software bill of materials, build provenance, container and binary digest values, schema revisions, policy bundles, and the exact route table. in-toto supplies a model for verifiable supply-chain steps (Torres-Arias et al. 2019), while SLSA defines progressively stronger provenance requirements (SLSA Community 2026). These controls do not make the release correct; they make the thing that was tested distinguishable from a later substitution.

Separate the planes

The stack has three planes with different permissions and failure modes.

• The data plane carries model requests, typed stream events, retrieval evidence, tool proposals, and tool results.
• The control plane admits requests and applies routing, budgets, capability constraints, authorization policy, and overload rules.
• The management plane changes route tables, credentials, policy bundles, schemas, and integration releases.

A model gateway may enforce useful data- and control-plane policy, but it is not a universal hub. Retrieval authorization belongs at the retrieval service. A tool executor must authorize its own resources. Management-plane changes require a narrower administrative path than ordinary inference. A single proxy process may implement several roles, but the contracts and privileges must remain separate.

Make compatibility explicit

Similar JSON shapes do not imply the same semantics. HTTP defines transport semantics (Fielding et al. 2022); server-sent events define stream framing (WHATWG n.d.). Neither defines a provider's tool-call lifecycle, refusal state, structured-output subset, usage counters, retention behavior, or safety policy. An adapter must therefore publish a versioned capability profile.

For every requested capability, record one of four states:

• native: the provider implements the declared meaning;
• adapter-emulated: the adapter preserves the declared postcondition;
• lossy: the adapter changes meaning, and the caller has explicitly accepted that downgrade;
• unsupported: admission rejects the request before upstream work begins.

The profile should declare modalities, endpoint and state model, tool and parallel-call semantics, the supported JSON Schema dialect and subset, typed stream events, context and output limits, usage fields, region, retention, and policy restrictions. JSON Schema is a language family with identifiable dialects, not a promise that every provider implements every keyword (Wright et al. 2022). Translation has only three defensible outcomes: preserve, perform an explicitly accepted downgrade, or reject. Silently dropping a tool, truncating an identifier, or coercing an unknown field is not compatibility.

Routing follows the same discipline. First filter by hard constraint: capability, tenant policy, data residency, retention, region, and authority. Only then rank eligible candidates by measured quality, latency, availability, or cost. Resolve an immutable provider, model revision, adapter revision, capability-manifest revision, and route-policy revision for every accepted operation. Use sticky assignment for experiments and canaries so repeated work does not wander between systems.

A reference architecture

Figure 88.2 shows the smallest useful architecture. The application is the caller. Policy admits work. An adapter converts the caller contract into a provider contract. Tool effects take a separate path through authorization and an executor. Retrieval returns identified evidence. Telemetry observes every decision but grants no permission.

This picture deliberately avoids a universal product center. The model boundary, tool boundary, retrieval boundary, and evidence boundary solve different problems covered in Chapter 82, Chapter 85, Chapter 86, and Chapter 87.

Define the model-operation state machine

Treat a request as a logical operation containing a verified tenant, operation_id, payload digest, remaining deadline, requirements, and policy revision. Admission either rejects it or records the immutable route decision. Each upstream call is a physical attempt beneath that logical operation.

The state machine is explicit:

1. Receive and authenticate the operation.
2. Validate its schema and capability requirements.
3. Admit it under tenant, budget, region, and policy constraints.
4. Resolve and record an immutable route.
5. Open an upstream attempt and normalize its frames into a versioned typed event stream with operation, attempt, and sequence identifiers.
6. Accumulate streamed tool fragments without executing them.
7. On a terminal tool block, parse the complete arguments, validate them against the pinned schema, authorize the exact effect, and execute once.
8. Preserve refusal, incomplete output, cancellation, partial output, error, and normal completion as distinct terminal outcomes.

Stream chunks are not a complete document. Unknown event types are retained or surfaced for forward compatibility; they are not reinterpreted as text. Partial tool arguments must never execute. A structured result receives final validation against the declared JSON Schema profile only after assembly. Refusal, truncation, transport failure, and schema violation each produce a separate invalid outcome rather than a plausible-looking object.

After caller-visible output, an automatic retry can duplicate or contradict the stream. Do not replay transparently after visible output. Propagate cancellation and the remaining deadline downstream; stop reading and cancel upstream when a caller disconnects. Every admitted operation must reach a terminal state through finite attempts, a finite deadline, and bounded resources.

Normalize errors without hiding causes

Use a stable machine-readable envelope based on RFC 9457 (Nottingham et al. 2023). Its media type is application/problem+json. Define a stable problem type and include the safe upstream category, HTTP status, provider code, upstream request ID, operation ID, attempt number, whether output was partial, whether the operation is retryable, and any Retry-After value. Preserve the original cause in protected diagnostics.

The human-readable detail field is for people. Never parse human-readable detail to decide whether to retry. Authentication, authorization, invalid input, unsupported capability, policy refusal, and exhausted quota are normally not retryable. Overload and transient transport failures may be retryable only under the operation's budget and deadline.

Bound retries and overload

Nested client, gateway, adapter, and provider retries create retry amplification. Here $\ell$ denotes a retrying layer and $r_\ell$ its maximum retry count. The maximum number of physical attempts generated by one logical operation is

$$A_{\max}=\prod_{\ell=1}^{L}(1+r_\ell),$$

where $L$ is the number of independently retrying layers. Two retries at each of three layers can create $3^3=27$ attempts. Assign one retry owner. That owner obeys a total attempt limit and remaining deadline, honors Retry-After, and uses bounded exponential backoff with jitter. Record all physical attempts even when the logical operation succeeds. Tail latency makes these controls especially important (Dean and Barroso 2013).

An idempotency key is an application contract, not magic HTTP exactly-once delivery. Bind it to tenant, route class, operation key, canonical payload digest, and time to live. Reusing a key with a different payload is an error. Where the provider or target cannot deduplicate, state that exactly-once is not guaranteed. A side-effecting tool needs its own effect-level key and receipt; a model-request key cannot make the tool idempotent.

Overload controls must also be bounded: per-tenant concurrency, queue length, stream buffer, input and output tokens, and tool fan-out. A bounded queue exposes backpressure instead of turning latency into hidden memory growth. Shed work with an appropriate 429 or 503 and Retry-After; use circuit breakers and retry budgets to prevent recovery traffic from becoming a retry storm.

Fallback changes the system identity. A fallback candidate must be prequalified inside a declared compatibility class and must satisfy the same hard policy and data constraints. Record the new route decision. Never silently switch after visible output or after a side effect whose result is ambiguous.

Keep identity, credentials, and effects separate

A gateway credential, a workload identity, a delegated user token, and a downstream provider secret answer different questions. Prefer short-lived workload identity and OAuth token exchange (Jones et al. 2020). Bind a token to an audience, scope, verified tenant, subject or actor, and time to live. Every resource server validates its own audience. Never pass a token issued for one resource through to another.

A virtual key is one implementation of model-budget and routing policy; it is not universal tool authorization. Where a provider supports only a static secret, keep that static secret in a broker and substitute it only at an allowed egress boundary. The application and sandbox should not receive the raw provider secret. Zero-trust guidance similarly treats network location as insufficient authority (Rose et al. 2020).

Tools and MCP

Model Context Protocol (MCP) standardizes host, client, and server messages over JSON-RPC and supports capability negotiation, discovery, and tool schemas (Model Context Protocol Contributors 2026). Discovery does not grant authority. A tool annotation is untrusted metadata, not proof that a call is safe. Pin the MCP protocol revision, transport, and negotiated capabilities, then perform authorization per invocation under the controls in Chapter 56.

The trusted runtime should perform this sequence:

1. Authenticate user and workload; derive the tenant from verified identity.
2. Validate the selected tool name against an allowlist and canonicalize the complete arguments against pinned input and output schemas.
3. Classify the effect as read, reversible write, or irreversible/external.
4. Authorize the principal, workload, tenant, tool revision, resource, audience, canonical arguments, effect class, and policy revision; bind expiring consent or approval where required.
5. Append an intent record, broker a least-privilege credential, and execute with an effect-level idempotency key.
6. Persist an effect receipt before advancing the workflow checkpoint; validate and redact the result before returning it.

On an ambiguous timeout, query the receipt or target state. Do not blindly retry a non-idempotent effect. Cancellation is not rollback. Parallel calls require independent call IDs and receipts. A tool result must retain its originating call ID so the runtime cannot attach it to the wrong proposal.

An sandbox reduces blast radius; a sandbox is not authorization. Its threat model should bound filesystem access, syscalls, processes, CPU, memory, disk, wall-clock time, workspace lifetime, and network access. Enforce both an egress allowlist and a resource allowlist. Resolve and validate DNS and every redirect hop, blocking private, link-local, and metadata endpoints. Isolation can contain an inappropriate action; it cannot make the action appropriate.

Retrieval and evidence

Authorize before retrieval, not after candidate documents have crossed the boundary. Every result should carry an evidence ID, source and corpus revision, index revision, authorization decision, and tenant boundary. Cache, checkpoint, session, task, and deduplication keys must include tenant and system-policy identity. Chapter 86 develops the underlying evidence contract.

Treat telemetry as evidence, not authority

W3C Trace Context supplies correlation identifiers across services (Kanzhelev et al. 2021). It is not authority. Never infer principal, tenant, or permission from traceparent or baggage. Baggage has no built-in integrity and must contain no secret, personal data, or authorization decision; allowlist and strip it at trust boundaries.

Record the operation ID, physical attempt ID, route and adapter revisions, capability decision, error category, tool intent and receipt, evidence IDs, and terminal outcome. Still report missing telemetry, exporter loss, sampling, and redaction failure. A trace is partial evidence, not automatic proof that an effect did or did not occur (Chapter 87).

Account for the accepted task

Optimize the whole accepted task, not a convenient model-call subtotal. Cost includes every physical attempt, model usage, retrieval, tool execution, judge, infrastructure, and human review. Latency is the root elapsed time; do not sum parallel child durations. Use the exact response model, region, billing time, currency, and price-catalog revision rather than a mutable alias (Chapter 76, Chapter 31).

For task $q$, a useful accounting identity is

$$C_q=\sum_{a\in\mathcal{A}_q}\sum_{m\in\mathcal{M}_a}u_{a,m}p_m(v_a,\rho_a,t_a,k_a) +C_{\mathrm{infra}}+C_{\mathrm{human}},$$

where $\mathcal{A}_q$ is the set of physical attempts, $\mathcal{M}_a$ contains disjoint billing meters for attempt $a$, $u_{a,m}$ is usage on meter $m$, and $p_m$ is its price for response-model revision $v_a$, region $\rho_a$, billing time $t_a$, and price-catalog revision $k_a$. The remaining terms cover shared infrastructure and human review. Disjoint meters prevent cached or reasoning subtotals from being counted twice.

Build-versus-rent is likewise a measured crossover, not a universal percentage. Change the two prices below to current, contract-specific values. The chart's defaults are an editable example, not a recommendation.

def crossover_utilization(serverless_rate, reserved_monthly, hours=720):
    """Return the utilization fraction where pay-per-use equals the flat bill."""
    if serverless_rate <= 0 or reserved_monthly < 0 or hours <= 0:
        raise ValueError("rates and hours must define a non-negative comparison")
    return reserved_monthly / (serverless_rate * hours)

example = crossover_utilization(4.0, 1440.0)
assert abs(example - 0.5) < 1e-12
print(f"illustrative crossover = {example:.0%}")

Chapter 5 and Chapter 62 explain why workload shape, memory, batching, and hardware efficiency matter beyond the simple curve.

Test the seams, not just the happy path

Every adapter and boundary should pass the same conformance fixtures before it joins a release.

Add negative fixtures for unknown stream events, duplicate delivery, a worker crash after commit but before receipt, malicious tool output, schema drift, DNS rebinding, redirect to a metadata endpoint, filesystem escape, and runaway process creation. These tests verify behavior that a successful demo cannot.

Cut over as a protocol

Migration is itself an integration. Begin with adapter contract tests. Run new read-only behavior in shadow mode with no side effects. Then use a sticky canary so one task or tenant stays on one fingerprint. Ramp only while quality, security, latency, error, and cost gates hold. Publish route and policy changes atomically. Keep the last-known-good manifest deployable and test rollback before the first canary receives traffic.

The failure matrix turns vague caution into ownership.

The release lifecycle

The complete workflow is short enough to rehearse:

1. State the capability, population, constraints, and rollback trigger.
2. Inventory producers, consumers, owners, trust boundaries, and data classes.
3. Freeze the system fingerprint and signed manifest.
4. Publish capability, identity, failure, effect, and evidence contracts.
5. Implement adapters that preserve, explicitly downgrade, or reject.
6. Assign one retry owner and bound deadlines, queues, and concurrency.
7. Bind credentials and tool effects to verified identity and tenant policy.
8. Run protocol, policy, isolation, failure, and supply-chain conformance tests.
9. Measure accepted-task quality, latency, cost, and missing evidence.
10. Shadow without side effects, then run a sticky canary and staged ramp.
11. Publish the atomic manifest and retain the tested rollback revision.
12. Record owners, known gaps, expiry, and requalification triggers.

The output is an integration release record: the exact manifest, boundary contracts, test evidence, accepted degradations, rollout state, known gaps, owners, expiry, and rollback procedure.

Further reading