Wiring a 2026 Stack

The book opened by promising that a capability has a lifecycle, from compute up to a governed behavior, and that the layers connect in a specific order (Chapter 1). The capstone is a concrete, end-to-end reference architecture that wires the pieces from the practical chapters into one running system. Its test is whether the seams can be named and the glue can be written: model calls, tools, telemetry, budgets, and virtual-key custody all have to meet at explicit contracts.

The trap to avoid is treating this as a seventh landscape survey. The six chapters before it already surveyed models (Chapter 81), serving and compute (Chapter 82), training and fine-tuning (Chapter 84), agents and sandboxes (Chapter 85), retrieval (Chapter 86), and evaluation (Chapter 87). This chapter does not re-list their players. It answers a different question: given those choices, how do the parts compose into something you can run, observe, and pay for?

The thesis: three contracts and one discipline

A 2026 stack composes not because the tools agree on features, they do not, but because they agree on three wire-format contracts. Each contract is a narrow, stable interface that lets you swap what is behind it without rewriting what is in front of it. The whole architecture is legible once you see that everything meets at these three seams.

1. Model calls speak the OpenAI-compatible HTTP shape. /v1/chat/completions, /v1/responses, /v1/embeddings. By 2026 every serving engine exposes it (Chapter 82), every hosted frontier API offers it or a near-twin, and every gateway normalizes to it. This is what makes a self-hosted vLLM model and a hosted Claude endpoint interchangeable behind one name.
2. Tool and agent traffic speaks MCP. The Model Context Protocol is to tool integration what the OpenAI shape is to model calls: the common interface that lets one tool server be reused by every agent framework (Chapter 85). By 2026 it is the default tool layer under almost every framework, and the stable spec is 2025-11-25 with a stateless revision in its validation window.
3. Telemetry speaks OpenTelemetry GenAI spans. The gen_ai.* semantic conventions are the vendor-neutral format for LLM, retrieval, and agent spans (Chapter 87). Instrument once and you can point at Langfuse, Phoenix, Datadog, or Grafana without re-instrumenting, though the conventions are still marked in development as of mid-2026, so pin the schema version you emit.

The one discipline that ties the contracts together is virtual-key custody. No application, agent, eval harness, or sandbox ever holds a raw provider credential. They hold a short-lived virtual key minted by the gateway, scoped to a model allowlist, a budget, and a rate limit. Rotating a provider secret or capping a runaway agent becomes a server-side operation with no client redeploy. The same idea extends to compute: secrets live outside the sandbox and reach the agent as short-lived scoped tokens where a provider mints them per call, or as an opaque placeholder a TLS-terminating egress proxy swaps for a static secret it will not, never baked into the box (Chapter 56).

The component that all three contracts converge on is the model gateway, and this chapter is its home. There is no separate gateway chapter in this book, the gateway is where the layers join, so its landscape and the one comparison table this chapter owns live here.

The gateway: the one table this chapter owns

A model gateway (also "LLM gateway" or "AI gateway") is the control plane between your application code and the many providers it talks to. The thin description is a reverse proxy that speaks one API and fans out to many. The real job is to centralize five concerns nobody wants scattered through application code: a unified API, key custody, cost tracking, fallback and routing, and budgets and policy. The newest gateways extend that role from plain token traffic to agent-native protocols (MCP for tools, A2A for agent-to-agent), making the gateway the single policy point for the whole stack's traffic.

All claims below are as of mid-2026 and attributed in Further reading. Vendor performance numbers are not independently verified here.

How to choose splits on two axes that matter more than feature checklists: managed vs self-hosted, then plain-LLM vs agent-native.

• Pick a managed gateway (OpenRouter, Cloudflare, Portkey managed) for zero infrastructure and fast access to a broad catalog, when you are comfortable that provider keys and request metadata transit a third party. Best for prototypes and small teams.
• Pick a self-hosted gateway (LiteLLM, agentgateway, Envoy, Bifrost) when provider secrets must stay in your environment, or you need data residency and per-request audit you own. Best for regulated industries and platform teams.
• Reach for an agent-native gateway (agentgateway) when your roadmap needs MCP tool calls and agent-to-agent traffic under the same policy point as token traffic, instead of bolting agent governance onto a chat-only proxy later.

On performance, vendors publish dramatic latency numbers ("50x faster", "sub-millisecond overhead"), measured under inconsistent conditions and not verified here. For almost all workloads, provider and model latency dominate gateway overhead by orders of magnitude, so per-request overhead rarely breaks a tie. What breaks a tie under agent load is the throughput ceiling, and there the implementation language shows. A controlled benchmark by the author, comparing the gateways in a stateless passthrough configuration at an equal CPU budget, found the Python proxy (LiteLLM) saturating between roughly 100 and 500 requests per second and falling over near 1000, while the compiled gateways (agentgateway in Rust, Lux in Go) stayed flat through 1000 RPS, with added first-byte latency at low load in the low single-digit milliseconds. The two compiled gateways were statistically indistinguishable from each other: the gap that mattered was Python versus compiled, not one product against the field. That is the bare-forwarding tier, not a feature-parity test, since a production gateway also authenticates, meters, and applies policy, so benchmark your own configuration before you let it decide.

The sensible default for a team that wants control of its keys and costs is therefore a performant compiled gateway paired with an observability layer. Two lead the benchmark above: agentgateway (Rust, Apache-2.0, Linux Foundation), the neutral choice, and Lux (Go), the author's own gateway and the stack the rest of this book wires (Chapter 75). LiteLLM keeps a real place for the broadest provider coverage and for low-traffic or prototype work, where its ecosystem reach outweighs its throughput ceiling. (Lux appears throughout as a worked example because the author built it, not as a neutral pick over agentgateway, with which the benchmark ties it.)

The reference architecture

Figure 88.2 merges the per-layer wiring diagrams from the practical chapters into one picture. Read it top to bottom as a request flow and side to side as the contracts. The gateway is the hub; everything that needs a model, an embedding, a reranker, or a judge call goes through it. The sandbox is the agent's compute seam, the counterpart to the gateway's model seam. Retrieval sits beside the agent. Eval and observability tap the gateway and the spans. The compute and orchestration layer is the floor everything stands on.

Three things make this diagram hold up in production, and each is one of the contracts from the thesis:

• One base URL, OpenAI-shaped. The app, the retriever (for embed and rerank), the eval harness, and the judge all point at the same gateway address. Swapping a model or adding a fallback is configuration, not code. A self-hosted bge-m3 embedder and a hosted Cohere reranker are addressable by name through the same seam (Chapter 86).
• One tool protocol, MCP. Agents do not dial arbitrary tool servers. MCP traffic flows through a gateway that owns the credentials and the allowlist, the same way the model gateway owns token traffic (Chapter 85).
• One telemetry format, OTel. Every component emits gen_ai.* spans pinned to one schema version, so the eval loop can read any trace (Chapter 87).

The three arcs

The book built the stack along three arcs. The reference architecture is where they meet. Walking each arc is the fastest way to see how the whole thing flows.

From chatbot to agent

A chatbot is a single model call: prompt in, completion out. An agent is a loop around that call: the model emits a tool request, the runtime executes it, feeds the result back, and the model decides what to do next, possibly for many turns (Chapter 38, Chapter 41). Three additions turn the first into the second, and all three appear on the right and top of Figure 88.2.

• Grounding via retrieval. Instead of trusting the model's weights, the agent retrieves current, private, or authoritative context and injects it (Chapter 44, Chapter 86). The 2026 consensus pipeline is hybrid search (BM25 plus dense vectors, fused with reciprocal rank fusion) then a reranker, with the embedding and reranker models called through the gateway so they are keyed and budgeted like any other model call. Prefer retrieval over stuffing a million-token context window: long context is expensive, and the price cliffs above 200K tokens (Chapter 76) usually make retrieval both cheaper and more faithful.
• Tools via MCP and a sandbox. Reading a result is the easy half. Acting, running code, hitting an API, writing a file, is where safety stops being about the model and becomes about the runtime (Chapter 85). A jailbreak filter on the gateway does nothing against rm -rf the model was induced into emitting. The enforcement layer is the sandbox: a microVM (Firecracker, used by E2B, Fly, Vercel) or userspace kernel (gVisor, used by Modal) that bounds the blast radius, with default-deny egress, hard timeouts, and human-in-the-loop gates for irreversible actions (Chapter 56).
• Memory via a checkpointer. A long-running agent must survive a restart, so its state lives in a durable, checkpointed store (LangGraph's checkpointer, Pydantic AI's durable execution), not in process memory (Chapter 39). This is what lets a human pause, inspect, and resume a task mid-flight.

The constraint arrow runs downward through this arc: the agent cannot be more reliable than the gateway, serving, and sandbox layers beneath it, which is exactly why "durable execution" and "checkpointing" became the headline 2026 framework features. They are the framework's attempt to survive the failures of the layers below.

From inference to training

Most of a 2026 stack is inference: serving a fixed model behind the OpenAI seam. But the same gateway address that fronts a hosted Claude endpoint can front a model you serve yourself, and the same address can front a model you trained yourself (Chapter 82, Chapter 84). The arc from inference to training is the arc from renting capability to owning it, and the gateway is what makes the two interchangeable.

The economics decide where on the arc you sit. With a closed API you do not run the inference server: the lab owns batching, KV-cache, speculative decoding, and the accelerators (Chapter 31), and you consume them as knobs (cached-input pricing, priority tiers). When volume and control justify it, you cross to self-hosted serving (vLLM by default, SGLang for prefix-heavy agent and RAG traffic), and the per-token cost moves from a vendor invoice to a GPU bill you schedule yourself. Further along the arc, you fine-tune (Chapter 17) or post-train a model and serve the result behind the same seam. The training run itself lands on the compute floor, gang-scheduled by Slurm or by Kubernetes with a gang-scheduling add-on (Chapter 65, Chapter 62).

The deciding variable is sustained utilization. Below roughly half a GPU's worth of steady load, a serverless or managed-inference platform (Modal, Baseten, Fireworks, Together) is cheaper than paying for idle reserved hardware. Above it, reserved clusters on a neocloud or hyperscaler win. The constraint arrow here is the one from Chapter 5: the serving cost of a token, paid forever, is what justifies over-training a smaller model once, which is why a model's size is set by how many tokens it will decode over its lifetime, not by its training budget alone.

Change the two price knobs below and watch where the build-vs-rent crossover lands: serverless wins below it, a flat reserved bill wins above it.

import numpy as np
# Illustrative figures only (mid-2026 indicative, not real quotes). Tune freely.
serverless_per_gpu_hour = 4.0   # $/GPU-hour, pay only while running
reserved_per_gpu_hour   = 2.0   # $/GPU-hour, billed 24/7 whether idle or not
u = np.linspace(0.01, 1.0, 100)                      # sustained utilization
serverless = serverless_per_gpu_hour * u * 24 * 30   # pay per busy hour
reserved   = reserved_per_gpu_hour * 24 * 30          # flat monthly bill
crossover = reserved_per_gpu_hour / serverless_per_gpu_hour
print(f"crossover utilization = {crossover:.0%} of one GPU")
print("below it serverless wins, above it reserved wins")
import matplotlib.pyplot as plt
plt.plot(u, serverless, label="serverless / managed (pay per use)")
plt.axhline(reserved, color="C1", label="reserved (flat 24/7)")
plt.axvline(crossover, color="gray", ls="--")
plt.xlabel("sustained utilization (fraction of one GPU)")
plt.ylabel("monthly cost ($)")
plt.legend(); plt.title("Build vs rent crossover"); plt.show()

From usage to eval

The third arc closes the loop. Every model and tool call the app makes is usage, and every unit of usage is something you can observe and score (Chapter 87). A model that tops a public leaderboard can still misbehave in your product, because the benchmark measured a generic capability (Chapter 47) while your app runs a specific prompt over your retrieval context inside an agent loop.

The loop has two halves that meet at the OTel seam. Observability captures the trace: the full tree of a request, every LLM call, retrieval, and tool invocation, with inputs, outputs, timing, tokens, and cost on each span. Evaluation scores those traces with deterministic assertions, reference-based metrics, and LLM-as-judge (Chapter 50), where a second model grades the first against a calibrated rubric. The two connect because you evaluate the traces you observe: a production trace that fails an online judge gets promoted into the offline regression suite, which gates the next release (Chapter 52). Because the app, the evals, and the judge all call the gateway, an A/B between two models is a routing rule and the judge runs on a scoped, budget-capped virtual key.

This arc feeds the other two. Eval datasets become preference data for the training arc, and failing agent traces become the fixtures that harden the agent arc. The three arcs are not parallel tracks. They are one cycle.

Pick a reference stack

There is no single right stack, but there are three archetypes that cover most teams. The rows are the layers; pick the column that matches your constraints, then read down it. These are starting points to adapt, dated mid-2026, not rankings.

The durable advice underneath the table: minimize new moving parts, put your quality budget where it pays (retrieval and reranking over a fancier vector DB, calibrated judges over more benchmarks), keep every layer addressable through the three contracts so each is swappable, and never hard-code a single vendor. The archetype you start in is not the one you end in; the contracts are what keep the migration cheap.

Wiring: the seams in code

The glue that matters is not a tool tutorial. It is the code at the seams, the handful of lines that make the layers interchangeable. Two snippets show the load- bearing pattern: a gateway config that unifies a hosted and a self-hosted model with a fallback, and an agent that points at that one address with a virtual key.

First, the gateway. The config below exposes one alias, smart, that load-balances or fails over between a hosted and a self-hosted model, so the application never learns which one served the request. The shape is illustrative, not any one product's schema: LiteLLM expresses it as a static YAML model_list, while agentgateway and Lux bind the routing strategy (weighted, least-latency, round-robin, or fallback) to a virtual key through an API or dashboard rather than a file (Chapter 75). The seam the application sees is the same either way.

# Illustrative gateway config: one alias, hosted primary with a self-hosted fallback.
# Each gateway has its own schema; this shows the shape, not a product's exact keys.
models:
  - alias: smart                  # the name the app requests
    target: anthropic/claude-opus-4-8
    api_key: ${ANTHROPIC_API_KEY} # provider secret stays server-side
  - alias: smart                  # same alias, fallback target
    target: openai/gpt-5.5
    api_key: ${OPENAI_API_KEY}
  - alias: local                  # self-hosted, behind the same seam
    target: vllm/your-open-weight-model
    base_url: http://vllm-svc:8000/v1

routing:
  smart:
    strategy: fallback            # if the hosted target fails, use the next
    fallback: [local]
    retries: 2
# Virtual keys, team budgets, and spend caps are minted out of band,
# through an admin API or dashboard, not in this file.

Second, the agent. It holds a gateway virtual key, not a provider secret. It retrieves context, runs a tool, and never learns which model answered. The MCP tool is in-process here for brevity; in production it sits behind the MCP gateway over HTTP, and the agent code barely changes, which is the whole point of MCP being a wire protocol rather than a library API (Claude Agent SDK shape; see the SDK docs for the current API).

from claude_agent_sdk import tool, create_sdk_mcp_server, query, ClaudeAgentOptions

@tool("search_docs", "Retrieve grounding context for a query", {"q": str})
async def search_docs(args):
    # hybrid retrieve + rerank; embed/rerank calls go through the gateway too
    hits = await retriever.hybrid_search(args["q"], top_k=8)
    return {"content": [{"type": "text", "text": "\n\n".join(h.text for h in hits)}]}

server = create_sdk_mcp_server(name="rag", version="1.0.0", tools=[search_docs])

async for msg in query(
    prompt="Using the docs, summarize our refund policy and cite the source.",
    options=ClaudeAgentOptions(
        mcp_servers={"rag": server},
        allowed_tools=["search_docs"],
        # model access resolves through the gateway base_url + a scoped virtual
        # key, never a raw provider credential. Swapping the model behind "smart"
        # or capping this agent's spend is a server-side change with no redeploy.
    ),
):
    print(msg)

The eval harness closes the loop with the same discipline. A Promptfoo config points its provider apiBaseUrl at the gateway and uses a different model family as the judge, on its own budget-capped virtual key (Chapter 87). Every seam in the system, model calls, tool calls, retrieval, and judging, meets at the same two addresses (the model gateway and the MCP gateway) and emits the same span format. That is what makes the architecture one system rather than seven separate tools that merely sit side by side.

Contracts that make the stack operable

The three contracts matter because they line up the stack's operating axes. On capability, they let you assemble a system more capable than any single layer: a grounded, tool-using, durable agent backed by the best model for each step, with each layer best-in-class for its job and still composable. Efficiency holds because per-task routing, fallback, and cost caps live in gateway configuration, so an expensive model is used only where a step needs it and load shedding does not touch application code. Trust is the third axis, and it rests on all three at once: virtual-key custody, the portable OTel substrate, and the eval loop that turns "it feels better" into a release gate. The stack is worth wiring this way only if all three hold. By mid-2026 the contracts have matured enough that they do, and the lifecycle the book opened with, from compute to a governed behavior, finally runs end to end through one architecture.

Further reading

Gateways:

• LiteLLM proxy (cost tracking, load balancing, fallbacks): https://docs.litellm.ai/docs/proxy/cost_tracking, https://docs.litellm.ai/docs/proxy/reliability
• OpenRouter, "What is an LLM gateway": https://openrouter.ai/blog/insights/llm-gateway/
• Portkey AI gateway: https://portkey.ai/buyers-guide/ai-gateway-solutions
• agentgateway: https://agentgateway.dev/ and https://github.com/agentgateway/agentgateway; Linux Foundation announcement: https://www.linuxfoundation.org/press/linux-foundation-welcomes-agentgateway-project-to-accelerate-ai-agent-adoption-while-maintaining-security-observability-and-governance
• Envoy AI Gateway: https://aigateway.envoyproxy.io/docs/capabilities/
• Cloudflare AI Gateway and Bifrost (Maxim AI): https://github.com/maximhq/bifrost
• Lux (author's own gateway): https://lux.latere.ai

The layers this chapter wires together:

• Inference serving (vLLM disaggregation, structured output): https://docs.vllm.ai/en/latest/ and https://github.com/sgl-project/sglang
• Agent frameworks and MCP: OpenAI Agents SDK https://openai.github.io/openai-agents-python/; Claude Agent SDK https://github.com/anthropics/claude-agent-sdk-python; LangGraph https://github.com/langchain-ai/langgraph; MCP spec https://modelcontextprotocol.io/ and 2026 roadmap https://blog.modelcontextprotocol.io/posts/2026-mcp-roadmap/
• Sandboxes (isolation models, safe-usage discipline): E2B https://e2b.dev/; Modal https://modal.com/resources/best-code-execution-sandboxes-ai-agents; Firecracker https://firecracker-microvm.github.io/; Firecrawl security model https://www.firecrawl.dev/blog/ai-agent-sandbox; Cella (author's own) https://cella.latere.ai/
• Retrieval (hybrid + rerank, pgvector): https://github.com/pgvector/pgvector; hybrid search and RRF overview https://www.digitalapplied.com/blog/hybrid-search-bm25-vector-reranking-reference-2026
• Eval and observability (OTel GenAI): Promptfoo https://www.promptfoo.dev/docs/intro/; Langfuse https://langfuse.com/docs/observability/overview; OpenTelemetry GenAI conventions https://github.com/open-telemetry/semantic-conventions-genai
• Compute and orchestration (build vs rent, schedulers): Modal pricing https://modal.com/pricing; SkyPilot, "Slurm vs K8s" https://blog.skypilot.co/slurm-vs-k8s/; KubeRay https://github.com/ray-project/kuberay
• Frontier model access layers (Bedrock, Vertex, Azure): Anthropic models overview https://platform.claude.com/docs/en/about-claude/models/overview; OpenAI on Bedrock https://aws.amazon.com/bedrock/openai/; Gemini API pricing https://ai.google.dev/gemini-api/docs/pricing