Serving, Gateways, and Compute

The theory of why serving is hard lives in Chapter 31 and Chapter 32; the practice is a request path with three decisions. A self-hosted endpoint only pays when the workload justifies it, a gateway sits in front for keys and cost control, and the whole thing has to land on compute the team can actually pay for. The path runs top down through three layers: the gateway you talk to, the serving engine that decodes tokens, and the GPUs underneath.

The three layers

A self-hosted inference stack is three layers with one interface running through all of them. Figure 82.2 shows the request path.

The interface is the key to the whole design. By 2026 almost every engine and gateway speaks the OpenAI HTTP shape (/v1/chat/completions, /v1/completions, /v1/embeddings). That single convention is what lets you swap a self-hosted model for a hosted one, put a gateway in front of many engines uniformly, and point an eval harness (Chapter 87) at the same URL as production. A second shape is emerging beside it: OpenAI's own primary interface is now the stateful Responses API, and vLLM serves /v1/responses alongside the classic endpoints. Chat Completions remains the least common denominator that engines and gateways interoperate on. Build against it and keep every layer replaceable.

Layer 1: the serving engine

A serving engine turns a set of weights into a service: it loads the model onto accelerators, batches requests, runs the forward passes, manages the key-value cache, and streams tokens back. This is where most controllable cost and most user-visible latency live. The same weights under two engines, or one engine with two configs, can differ by an order of magnitude in tokens per second and tail latency.

Two phases drive every design decision (Chapter 31). Prefill processes the whole prompt in parallel and is compute-bound. Decode emits one token at a time, each step re-reading the cache, and is memory-bandwidth-bound. The features that matter are answers to the tension between them, and most are developed in Chapter 32 and Chapter 33; the short version follows. Continuous batching adds and retires requests every step, table stakes by 2026. PagedAttention stores the cache in non-contiguous pages so memory is not fragmented. Prefix caching / KV reuse computes a shared leading prefix once, the dominant win for agents and RAG, generalized to a radix tree (a trie keyed by token prefixes) by SGLang's RadixAttention. Quantization (Chapter 34) shrinks weights to fewer bits: FP8, FP4, INT4 via AWQ/GPTQ, GGUF K-quants. Speculative decoding (Chapter 33) drafts cheap tokens and verifies them in one pass. Structured output is grammar-constrained decoding via XGrammar, so tool calls and extracted data parse. Prefill/decode disaggregation puts the two phases on separate worker pools, the headline datacenter pattern of 2026.

The players

The table below is a mid-2026 snapshot. Licenses and "best for" are durable; the benchmark notes are third-party and workload-specific.

As of mid-2026, secondary benchmarks report SGLang ahead of vLLM on some H100 single-model tests (one cites roughly 29 percent) with larger gains on prefix-heavy shapes, the gap narrowing at 70B scale; LMDeploy near SGLang on throughput with best-in-class TTFT for INT4 70B; MLX 30 to 50 percent faster than llama.cpp on some Apple Silicon models. These come from vendor and blog benchmarks that vary by model, GPU, batch shape, and tuning, and they age fast. Verify on your own workload before treating any of them as a tiebreaker.

How to choose an engine

The decision is mostly a function of where you run, what your traffic looks like, and how much performance engineering you can afford.

• Pick vLLM when you want one safe, broadly-supported default on GPUs, need a wide range of models and features now, and value ecosystem maturity over the last 20 percent of performance. This is the right answer for most teams.
• Pick SGLang when your traffic is prefix-heavy (agents, RAG, multi-turn) or you serve large MoE models, and you want top throughput on those shapes. Verify the win on your own traffic first.
• Pick TensorRT-LLM when you are all-NVIDIA, latency and throughput are first-order costs, and you can invest in the toolchain (the default PyTorch backend lowers that cost versus the old build flow).
• Pick LMDeploy when quantized w4a16 serving and low TTFT on a NVIDIA fleet matter most.
• Pick llama.cpp when you need portability (CPU, AMD via Vulkan/ROCm, Metal), edge or local deploys, and GGUF quant, and you do not need datacenter-grade concurrency.
• Pick Ollama or MLX when the target is a laptop or a few internal users; pick MLX specifically on Apple Silicon for the fastest single stream.
• Reach for Dynamo or llm-d when you have outgrown a single node and need KV-aware routing, disaggregation, and SLO-driven autoscaling across a fleet.
• Do not start on TGI in 2026; it is archived and maintenance-only.

Sensible default (mid-2026): vLLM behind an OpenAI-compatible endpoint, prefix caching on, an appropriate quantization (FP8 on Hopper/Blackwell, or AWQ/GPTQ INT4 to fit a bigger model on smaller cards), and XGrammar-backed structured output for any tool-calling or extraction path. It is the broadest, lowest-regret choice. For local and dev, the default flips to Ollama or llama.cpp on Linux/Windows and MLX on Macs. Those are single-node, low-concurrency tools; do not promote them into a high-traffic path.

A minimal single-node vLLM launch:

vllm serve <model-id> \
  --enable-prefix-caching \
  --quantization fp8 \
  --guided-decoding-backend xgrammar
# exposes an OpenAI-compatible server on :8000

Flag names and structured-output parameters drift between vLLM versions; check the docs for your release.

Layer 2: the gateway

A model gateway is the control plane between your application code and the many model providers it talks to. In the thinnest framing it is a reverse proxy that speaks one API and fans requests out to OpenAI, Anthropic, Google, Bedrock, Azure, your own vLLM, and aggregators like OpenRouter. That undersells it. The gateway is where five concerns nobody wants scattered across application code get centralized:

1. Unified API. One request schema, so swapping gpt-5 for claude-opus-4 or a self-hosted variant is a config change, not a code change.
2. Key custody. Provider secrets live on the server. Apps and agents hold short-lived virtual keys scoped to a model allowlist and a budget, never the raw provider credential.
3. Cost tracking. Per-key, per-team, per-tag spend, attributed and exportable. This is the single most cited reason teams adopt a gateway.
4. Fallback and routing. Retries, provider failover, load balancing, context-window fallback, cost- or latency-aware model selection.
5. Budgets and policy. Hard spend caps, rate limits, guardrails (PII redaction, jailbreak filtering), and an audit trail.

The gateway is the seam between the serving layer and everything above it. The newest gateways extend the idea past plain LLM calls into agent-native protocols (MCP for tools, A2A for agent-to-agent), so the gateway becomes the policy point for tool and agent traffic too, which connects directly to the harness work in Chapter 85 and the authorization model in Chapter 56.

The players

Several vendors publish dramatic overhead numbers (Bifrost "50x faster than LiteLLM" and "~11µs"; Portkey "<1ms"; LiteLLM "~8ms P95 at 1k RPS"). These are vendor benchmarks under inconsistent conditions and are not independently verified. The durable point: compiled-language gateways (Go/Rust: Bifrost, agentgateway, Helicone, Envoy) generally claim lower proxy overhead than the Python LiteLLM proxy, but for most workloads provider and model latency dominate gateway overhead by orders of magnitude. Benchmark on your own traffic before treating overhead as a tiebreaker. A controlled benchmark in Chapter 88 sharpens where the gap actually shows: not per-request overhead but throughput, with the Python proxy saturating at a few hundred RPS while the compiled gateways stay flat through 1000.

How to choose a gateway

Work two axes in order: managed versus self-hosted, then plain-LLM versus agent-native.

• Pick OpenRouter when you are a solo dev or small team and want the largest catalog with no infra and no contracts.
• Pick LiteLLM when you want a self-hosted, broadly compatible gateway and have the DevOps to run it, especially on a Python stack, at low-to-moderate traffic where its provider coverage matters more than its throughput ceiling.
• Pick Portkey (or Kong if you already run Kong) when governance, guardrails, and compliance are requirements, not nice-to-haves.
• Pick agentgateway when your roadmap is agentic and you need LLM, MCP, and A2A traffic under one governed policy point.
• Pick Envoy AI Gateway when your platform team already lives in Envoy/Istio/Gateway API and wants the AI control plane in the same CRDs.
• Use a managed gateway (OpenRouter, Cloudflare, Portkey managed) when you want zero infrastructure and accept that keys and request metadata transit a third party. Move to self-hosted when key custody, audit, or compliance forces it.

Sensible default (mid-2026): self-host a performant compiled gateway, agentgateway (Rust, Apache-2.0, neutral) or Lux (Go, this book's worked example), paired with an observability layer for spend and request analytics. It gives you virtual keys with budgets, spend tracking, and fallbacks in your own environment, and stays flat under agent load where a Python proxy saturates (Chapter 88). LiteLLM stays a reasonable pick for the broadest provider coverage at low-to-moderate traffic, where its ecosystem reach outweighs its throughput ceiling. agentgateway also puts LLM, tool, and agent traffic under one policy point, which matters once the roadmap is agent-heavy.

A minimal gateway config that unifies a hosted and a self-hosted model behind one name and adds a fallback. The shape is illustrative, not one product's schema:

# Each gateway has its own schema; this shows the shape, not a product's exact keys.
models:
  - alias: smart                 # the name your app requests
    target: anthropic/claude-opus-4
    api_key: ${ANTHROPIC_API_KEY}
  - alias: smart                 # same alias, load-balanced / fallback target
    target: openai/gpt-5
    api_key: ${OPENAI_API_KEY}
  - alias: local
    target: vllm/<model-id>      # your vLLM, OpenAI-compatible
    base_url: http://vllm-svc:8000/v1

routing:
  smart:
    strategy: fallback           # if the hosted target fails, use the next
    fallback: [local]
    retries: 2
# Virtual keys, team budgets, and spend caps are minted out of band,
# through an admin API or dashboard, not in this file.

The application then points a standard OpenAI client at the proxy and requests model "smart". The gateway resolves the alias, load-balances or fails over, records spend against the virtual key, and emits traces. Upstream, "local" resolves to your Layer-1 engine, so self-hosted and hosted models become interchangeable behind one name:

from openai import OpenAI
client = OpenAI(base_url="https://gateway.internal/v1", api_key="<virtual-key>")
resp = client.chat.completions.create(
    model="smart",
    messages=[{"role": "user", "content": "..."}],
)

Each gateway has its own schema and it drifts between versions; check its docs for the exact keys.

Layer 3: the compute under it

Everything above eventually lands on a GPU someone has to provision, schedule, and pay for. Two questions sit at the center, and they are mostly orthogonal: where the GPUs come from (the build versus rent axis) and how they are scheduled. The hardware itself is Chapter 62; the orchestration substrate is Chapter 65; the cost arithmetic is Chapter 76. Here we choose.

Self-host versus call an API

Before sizing GPUs, settle the prior question: should you self-host at all? Most teams should start on a hosted API and self-host only when a concrete pressure forces it. The decision rule is blunt:

• Call a hosted API (through your gateway) when traffic is bursty or low volume, when you want the frontier model without operating it, when time-to-ship matters more than per-token cost, and when your data can leave your environment. This is the right default for most application teams.
• Self-host when at least one of these binds: sustained high utilization makes per-token economics dominate (you keep the hardware busy), data residency or compliance requires weights and prompts to stay in your environment, you need an open-weight model or a fine-tune no API offers, or you need latency and tail-behavior control the API will not give you.

The gateway makes this reversible. Because both paths speak the same OpenAI-shaped interface, you can start hosted, add a self-hosted route as one more alias, and shift traffic with a config change. Choosing a model for either path is Chapter 81; the wiring of the whole stack is Chapter 88.

Where the GPUs come from

A spectrum runs from serverless GPU platforms (push a function or a model, they handle the machine), through rented GPU clouds and neoclouds, to the hyperscalers, to owning hardware in a colo. The deciding variable is sustained utilization.

Pricing is volatile and aggregators disagree, so anchor on the shape, not the dollar. As of mid-2026, list-price anchors put serverless H100 around 3 to 4 dollars per hour (Modal ~3.95/hr, RunPod ~2.89/hr), with roughly a 5x spread in H100 on-demand rates across tiers. Re-check before committing; these move quarterly.

The rule of thumb: below roughly 40 to 50 percent sustained utilization of a dedicated GPU, serverless is cheaper because you pay nothing when idle. Above it, reserved GPU-hour rates on a neocloud or hyperscaler undercut both serverless and on-demand. Own hardware only at sustained very-high utilization where amortized capex beats years of rental. Use a marketplace or spot only for fault-tolerant batch. Use SkyPilot across all of these when you want to stay portable, chase the cheapest available capacity across clouds, and avoid lock-in.

Edit the two hourly rates and watch where the serverless and reserved cost lines cross; with the chapter's price anchors the break-even lands near the stated 40 to 50 percent band.

import numpy as np, matplotlib.pyplot as plt
serverless_rate = 3.95   # $/GPU-hour, pay only when busy (scale-to-zero)
reserved_rate   = 2.00   # $/GPU-hour, billed 24/7 whether idle or not
hours = 730              # one month
u = np.linspace(0, 1, 101)
serverless = u * hours * serverless_rate      # cost scales with utilization
reserved   = np.full_like(u, hours * reserved_rate)  # flat cost
breakeven = reserved_rate / serverless_rate   # crossover utilization
print(f"break-even utilization: {breakeven*100:.1f}%")
plt.plot(u*100, serverless, label="serverless (pay-per-use)")
plt.plot(u*100, reserved, label="reserved (flat)")
plt.axvline(breakeven*100, ls="--", color="gray")
plt.xlabel("sustained utilization (%)"); plt.ylabel("monthly cost ($)")
plt.legend(); plt.title("GPU cost: serverless vs reserved"); plt.show()

How the GPUs are scheduled

Once you control a pool, a control plane decides placement. The three dominant ones, and how they split:

• Slurm when the dominant workload is large multi-node training and batch: jobs that start, run for hours, checkpoint, and exit. Gang scheduling (all of a job's workers scheduled together or not at all) is native and simple. The cost is no native services, ingress, or autoscaling.
• Kubernetes when you run inference serving, mixed workloads, or microservices alongside AI, and want the autoscaling, ingress, and service-mesh ecosystem. Add Volcano or NVIDIA KAI Scheduler for gang scheduling, Kueue for multi-team quota and admission, and DRA for fine-grained GPU/MIG allocation.
• Ray (on KubeRay) when your team is Python-native and wants distributed training, RL, data, and serving in one framework, riding on K8s.

The common large-team split is Slurm for training, Kubernetes for serving. The 2026 convergence is Slurm-on-K8s projects (SchedMD Slinky, CoreWeave SUNK, Nebius Soperator) that keep Slurm's gang semantics on shared K8s infra, narrowing the split. The momentum is toward standardizing on Kubernetes even for training, via the add-ons above, because inference, not training, is now the majority of AI compute spend and serving wants exactly the elastic, scale-to-zero, latency-aware scheduling that Kubernetes provides.

Sensible default (mid-2026): for a team shipping AI features with mixed bursty inference and occasional fine-tuning and no dedicated infra team, start on a serverless GPU or managed-inference platform (Modal for general serverless Python ergonomics, Baseten for production serving with SLAs, Fireworks or Together for pure open-model token inference). Promote to reserved neocloud clusters scheduled with Slurm-for-training and Kubernetes-for-serving as utilization climbs and the rental premium starts to dominate the bill.

Wiring it together

The three layers compose into one request path, shown in Figure 82.3, and the OpenAI-compatible interface is what keeps each replaceable.

Three facts make this hold together in practice. Applications never hold provider keys: they hold a virtual key minted by the gateway, so rotating a credential or capping a runaway agent is a server-side operation with no client redeploy. One base URL, OpenAI-shaped, serves app code, eval harnesses, and agent runtimes alike, so switching models or adding a fallback is configuration. And the engine, not the prompt, enforces structured output: guided decoding via XGrammar is the reliable path when downstream code parses tool calls or extracted data.

The gateway as the policy point

The layers tie back to the book's spine through the gateway. Capability is its reach: one interface that makes every model, hosted or self-hosted, addressable, so the best model for each call is a routing rule. Efficiency comes from the engine and the compute tier working in concert, where continuous batching, prefix caching, and quantization raise tokens per second while the build-versus-rent and utilization decision sets the dollar floor under every token. Trust brings the gateway back as the policy point, the place for key custody, spend caps, audit, PII redaction, and the agent-native protocols that bring tool and agent traffic under one governed seam, which is where this chapter hands off to the safety and authorization work in Chapter 56.

Further reading