Serving, Gateways, and Compute

A model is not served merely because an endpoint returns text. It is served when a declared class of requests receives acceptable results within a latency, availability, cost, and policy envelope. The unit to operate is therefore a versioned serving contract: the model artifact, runtime, adapter, request semantics, route policy, and compute placement that jointly produced the result. This extends the versioned served system from Chapter 81 into production.

The theory behind the runtime is developed in Chapter 31 and Chapter 32. This chapter turns it into an operating method. First freeze the contract. Then measure the complete path under representative load. Only after that should a team choose an engine, insert a gateway, or buy capacity. Those components are means, not the architecture's starting point.

Freeze the serving contract

An HTTP request shape is useful plumbing, but it is not a behavioral standard. Providers differ in message roles, tool and JSON-schema dialects, tokenization, stream events, stop conditions, usage fields, errors, safety behavior, data handling, and model-specific defaults. Official documentation shows the gap: Anthropic lists transformed, ignored, and unsupported fields, while Google recommends its native API when an application is not already tied to the common client (Anthropic n.d.; Google n.d.). Treat an OpenAI-shaped API, or any other shared envelope, as a common transport subset. Put provider-specific behavior behind adapters and retain native escape hatches where the common subset is insufficient.

The internal request contract should state at least the following.

For each provider adapter, run an adapter conformance corpus. It should cover ordinary messages, every tool/schema feature the application uses, partial and cancelled streams, context-limit errors, usage accounting, timeouts, rate limits, refusals, and malformed responses. A field that is transformed, emulated, or dropped must be rejected or reported; silently ignoring it is not conformance. Record that outcome as a translation loss report. The same corpus must run against every candidate route and during every adapter upgrade.

The deployment identity must include every layer that can change behavior. One useful fingerprint is

$$c = (a, e, m, z, p, k, d, h, r),$$

where $a$ is the provider adapter and version; $e$ the engine image and configuration; $m$ the model, tokenizer, and artifact digests; $z$ the prompt template, tool parser, and structured-output settings; $p$ the numerical precision and parallel placement; $k$ the attention and KV-cache policy; $d$ the decoding, batching, and admission policy; $h$ the accelerator and network topology; and $r$ the route, retry, and fallback policy. Every symbol denotes a versioned record, not a product name. Changing any term creates a new candidate that must pass the relevant evaluation again.

Figure 82.2 shows the boundary. A mediation service is optional, and the self-managed branch is only one possible deployment mode. The contract and evidence cross every branch.

Measure what the user waits for

The engine processes a prompt during prefill, then produces further tokens through autoregressive decode. Prefill often has enough parallel work to be compute-limited; decode often has low arithmetic intensity and becomes limited by memory traffic. These are common regimes, not laws. Prompt length, batch shape, model architecture, kernels, cache state, and hardware can move either phase to another bottleneck. Measure the exact fingerprint $c$ rather than assigning it a bottleneck by name.

For a simple request with $O$ output tokens, a useful critical-path decomposition is

$$T_{\mathrm{e2e}} = T_{\mathrm{queue}} + T_{\mathrm{prefill}} + \sum_{k=2}^{O} T_{\mathrm{decode},k} + T_{\mathrm{tools}}.$$

Here $T_{\mathrm{queue}}$ is admission and scheduling delay; $T_{\mathrm{prefill}}$ includes the work needed to produce the first token; $T_{\mathrm{decode},k}$ is the interval for output token $k$; and $T_{\mathrm{tools}}$ is serial tool or external-service time on the critical path. For parallel branches, add the maximum branch duration, not their sum. Network and client buffering should be recorded separately when they are material.

Time to first token (TTFT) measures the interval from accepted request to the first visible token. For $O>1$, time per output token (TPOT) is commonly reported as the elapsed time from first to last visible token divided by $O-1$. End-to-end latency still matters: an attractive TTFT can hide a slow decode, tool call, or retry. Report p50, p95, and p99 by request class, together with the rejected and timed-out fraction. A system can otherwise manufacture good latency by refusing its hardest traffic.

Engine mechanisms are conditional levers

Modern serving runtimes combine several techniques. Their history clarifies what each one actually solves. Orca introduced iteration-level scheduling: the active batch can change between decoding iterations instead of waiting for a static batch to finish (Yu et al. 2022). vLLM's PagedAttention applied block-based allocation to the growing KV cache, reducing over-reservation and fragmentation that limited resident sequences (Kwon et al. 2023). Later systems explored chunked prefill and separate prefill/decode placement (Agrawal et al. 2024; Zhong et al. 2024). None of these results establishes a universal production default.

Prefix reuse is not semantic caching: similar meaning does not imply reusable KV state. It also saves prefill work only. When decode dominates or prefixes do not repeat, retaining cached state can be a net capacity cost. Likewise, a lower-bit artifact can be slower if the runtime lacks an efficient kernel and must repeatedly convert it. Validate the exact artifact on the exact hardware, then rerun the task-quality gate from Chapter 87.

Structured decoding deserves the same restraint. After generation, check the finish reason, parse and schema result, domain invariants, caller authorization, and any effect-specific preconditions. A syntactically valid request to transfer money is still an authorization decision. That boundary connects serving to Chapter 56.

Benchmark the complete fingerprint

A useful comparison fixes a production-shaped, open-loop arrival trace. It preserves the observed request-class mix, input and output length distributions, prefix-reuse distribution, tool and schema mix, cancellations, and bursts. Open-loop means arrivals follow the recorded schedule rather than waiting for the previous response; otherwise a slow system reduces its own offered load and looks deceptively stable.

For each candidate:

1. Pin the full fingerprint $c$ and verify adapter conformance and task quality.
2. Run both cold and representative warm-cache conditions.
3. Sweep offered load from idle through saturation without changing the trace.
4. Record admitted and rejected requests, TTFT, TPOT, end-to-end tails, output rate, memory, power where available, and cost per accepted task.
5. Change one mechanism at a time. A one-feature ablation makes the cause of a gain inspectable.
6. Repeat enough trials to expose variance, then test worker loss, network delay, cancellation, and recovery.

Do not compare two engine names while also changing model revision, template, precision, hardware, or request mix and attribute the result to the engine. If those changes are intentional, compare them as two complete served-system candidates.

A configuration is eligible only when it passes every hard constraint and its joint service objective. For request class $s$, define all three gates together:

$$\begin{aligned} H_j(c) &= 1 && \text{for every hard constraint } j, \\ Q_s(c) &\ge q_s, \\ \Pr\!\left( \mathrm{TTFT}\le F_s, \mathrm{TPOT}\le D_s, T_{\mathrm{e2e}}\le L_s \mid \mathrm{admitted},s \right) &\ge \alpha_s. \end{aligned}$$

Here $H_j$ covers such constraints as license, region, interface, and capacity; $Q_s$ is measured quality with threshold $q_s$; $F_s$, $D_s$, and $L_s$ are the TTFT, TPOT, and end-to-end limits; and $\alpha_s$ is the required joint attainment rate. Unknown hard-constraint evidence does not pass. Optimize cost or throughput only over this feasible set, and always publish the admission rate beside the conditional latency result.

Use mediation where it earns its boundary

A gateway or mediation service is a design option, not a mandatory layer. It can centralize identity, credential references, quotas, routing, budgets, and telemetry when many callers or providers would otherwise duplicate them. A direct, well-tested adapter can be simpler for one application and one provider. The question is whether centralization creates a boundary the team can actually enforce and operate.

A mediation boundary is valuable only if its invariants are explicit:

• Authenticate the caller and tenant, then authorize the requested model, tools, data class, and effect. Possession of a proxy key is not sufficient.
• Route only within the allowed candidate set that has passed the same contract and policy-compatible fallback evaluation.
• Check and reserve the budget atomically before dispatch; reconcile the reservation against measured usage afterward.
• Propagate the original remaining deadline and cancellation. A retry does not receive a fresh timeout.
• Retry only when the failure class and operation are eligible, an idempotency key or read-only guarantee makes repetition safe, and budget and deadline remain. An ambiguous outcome is not proof that no work occurred.
• Correlate every attempt, selected deployment, policy decision, and result in one trace. W3C Trace Context defines interoperable propagation fields for that purpose (Kanzhelev et al. 2021).
• Fail closed when identity, authorization, route qualification, or budget evidence is missing. An observability outage should not silently disable policy.

Separate the administrative control plane---where routes, policies, and credentials change---from the request data plane that evaluates those records. Protect and audit both. Prefer workload identity or short-lived, audience-restricted credentials over long-lived bearer keys, and resolve provider secrets through a credential reference rather than copying them into route configuration.

HTTP distinguishes idempotent methods, but inference is commonly invoked with POST, and the generated result may cause an external action. Transport retry safety therefore cannot establish application-effect safety by itself (Fielding et al. 2022). If a timeout occurs after a tool may have executed, query the effect ledger by idempotency key or surface an ambiguous outcome for reconciliation. Do not blindly submit the action to a different model.

Assign one retry owner and set a bound on total attempts. Nested retries in the client, mediator, and provider SDK multiply rather than add. Distinguish a transport retry to the same deployment, provider failover to another qualified deployment, and model fallback to a different served system. A fallback is eligible only before the first byte has reached the caller and only within the same contract; once streaming has started, return a visible partial-stream failure. Never treat authentication, authorization, invalid input, or an unreconstructable stateful session as a reason to try another provider.

The following policy object is illustrative. It is an internal design record, not a promise that a particular gateway accepts this schema.

route_contract: support-summary-v3
request_class: read_only_summary
candidates:
  - deployment_ref: hosted-eu@sha256:...
    adapter_ref: adapter-a@sha256:...
    data_classes: [internal]
    regions: [eu]
  - deployment_ref: managed-eu@sha256:...
    adapter_ref: adapter-b@sha256:...
    data_classes: [internal]
    regions: [eu]
credential_ref: workload-identity://inference/support
deadline_ms: 2500
budget_reservation: required
retry:
  attempts: 1
  eligible_failures: [connect_before_send, rate_limited]
  requires: read_only_or_idempotency_key
fallback:
  requires_same_contract: true
telemetry:
  record_content: false
  propagate_trace_context: true

This route has two candidates only because both have passed the conformance, quality, region, and load gates. A different model behind the same alias is not automatically a fallback. It can change tool behavior, safety policy, context limits, data handling, latency, and price.

Record low-cardinality operational facts by default: request class, logical route, deployment version, token counts, finish reason, latency, attempts, policy outcome, and cost ledger references. OpenTelemetry provides evolving generative-AI semantic conventions, but input and output content can contain PII and other sensitive data (OpenTelemetry n.d.). Keep prompts, completions, tool arguments, and retrieved documents out of ordinary telemetry unless a specific retention, access, and redaction policy authorizes them.

Keep an unsampled logical request ledger separate from sampled diagnostic traces. The ledger links every attempt to the route revision, actual provider and deployment, upstream request ID, translation loss, usage, price version, estimated charge, policy decision, and stream termination. Reconcile that ledger with provider invoices; a local estimate is not invoice reconciliation and cannot by itself prove a hard budget ceiling.

Procure capacity from measurements

Deployment and procurement are separate choices. A team can use a first-party API, a provider-operated managed endpoint for a selected artifact, or a self-managed engine on rented or owned accelerators. The right comparison applies the same hard constraints, quality gate, workload, and accounting horizon to all three.

There is no universal utilization percentage at which one mode becomes cheaper. Capacity is purchased in steps, demand is bursty, warm replicas and redundancy are billed, and a cheaper route may produce fewer acceptable results. Model low, base, and high demand, including capacity steps, commitments, interruption, egress, operations labor, and incident cost. Then compare the end-to-end cost per accepted task from Chapter 81 and Chapter 76.

Establish a planning bound, then load-test it

Here $\lambda_{\mathrm{peak}}$ is the declared peak arrival rate, $\mathbb{E}[S]$ is measured accelerator-seconds of service per request under the production mix, and $\rho_{\max}<1$ is a chosen planning utilization cap. A rough lower bound on accelerator count is

$$g_{\min} = \left\lceil \frac{\lambda_{\mathrm{peak}}\,\mathbb{E}[S]} {\rho_{\max}} \right\rceil.$$

Here the expression is a planning approximation, not an autoscaling formula. Dynamic batching makes service demand non-linear; prompt and output tails matter; tensor or pipeline placement can require several accelerators as an indivisible unit; and startup delay prevents instantaneous scale-out. For measured service rate $\mu_g$ at placement size $g$, queue stability requires

$$\lambda < \mu_g,$$

where $\lambda$ is offered load and $\mu_g$ is measured service rate for the placement. That necessary condition does not guarantee p95 or p99 latency. Validate $g_{\min}$ with the open-loop sweep, then add explicit headroom and redundancy for failures, deploys, maintenance, and forecast error. Provisioned capacity is the feasible placement size plus those reserves, rounded to the supplier's allocation unit.

For each demand scenario, report accepted tasks, rejected tasks, quality-pass rate, SLO attainment, billed idle time, and total cost. Divide the full ledger by accepted tasks, not raw requests or generated tokens. That prevents a cheap but unreliable or low-quality path from appearing economical.

Choose orchestration by workload semantics

Once the team controls a pool, it needs placement and lifecycle controls. Do not choose the control plane from an industry popularity claim. Write down the workload first.

• A long-lived service needs health and readiness checks, rolling deployment, bounded disruption, stable addressing, traffic draining, autoscaling, and fast rollback.
• A finite distributed job may need gang scheduling, topology-aware placement, checkpoint and restart, preemption, priorities, and fair quota across teams.
• Both need accelerator identity, health, isolation, and accounting. Multi-node jobs also need network-topology constraints and all-or-nothing admission.
• A Python or other application execution framework can manage tasks and actors, but it does not replace cluster admission, device allocation, or service lifecycle controls.

Slurm is designed as a cluster workload manager with allocation, queueing, and job-step execution (SchedMD n.d.). Kubernetes supplies service controllers and pod placement; extensions such as Kueue add quota and workload admission without replacing the scheduler or job controller (Kubernetes SIG Scheduling n.d.). Either can be extended, and some organizations combine them. The durable distinction is the required lifecycle and scheduling semantics, not a slogan such as “training uses Slurm, inference uses Kubernetes.”

Autoscaling is bounded by physical reality. If a replica takes minutes to allocate hardware, load weights, compile kernels, and warm caches, a seconds-old queue signal cannot save the current request burst. Maintain warm capacity or admit less traffic, forecast ahead of known peaks, and test the full scale-up path. Device plugins and dynamic allocation likewise do not prove that a model fits, that the requested topology exists, or that degraded hardware is safe to serve.

Operate the decision

The end-to-end workflow in Figure 82.3 keeps architecture and evidence together.

1. Freeze the contract and workload. Declare request classes, arrival traces, quality gates, SLOs, admission policy, data rules, and cost horizon.
2. Pin one baseline. Record the complete fingerprint and establish a simple, colocated configuration before adding cache reuse, speculation, or phase separation.
3. Verify behavior. Run adapter conformance, task evaluation, safety checks, and structured-output invariants. Reject unknown hard-constraint evidence.
4. Load test and size. Sweep offered load through saturation, including cold and warm states. Size capacity with headroom and redundancy, then price low, base, and high demand.
5. Run failure injection. Exercise connection failure, provider 429 and 5xx responses, slow streams, cancellation, worker loss, scale-up delay, corrupted output, budget exhaustion, and an ambiguous tool action. Verify that deadline, retry, authorization, and accounting invariants hold.
6. Roll out reversibly. Shadow production traffic where policy permits, canary by request class and tenant, and maintain tested rollback for the adapter, route, engine, model, and scheduler configuration.
7. Keep an evidence record. Define a re-evaluation trigger for artifact or adapter change, provider behavior drift, workload shift, SLO regression, price or quota change, policy change, and scheduled expiry.

Prefill/decode disaggregation, learned routing, semantic caches, and multi-cloud failover enter this workflow as hypotheses. Add one only when the simpler baseline misses a declared objective and the new design wins under the same trace after transfer, duplicate capacity, policy, and failure costs are counted.

The worked stack in Chapter 88 is one implementation of these boundaries. The method here is the acceptance test for it: an endpoint, gateway, or GPU pool earns its place only when the complete served system produces better accepted results without violating the contract.

Further reading