The Tooling Ecosystem

A tooling stack is a set of versioned contracts, not a framework list. Chapter 74 ended with a verified model bundle. Making that bundle useful may involve a trainer, conversion tools, a serving runtime, a model gateway, an agent host, capability servers, sandboxes, policy engines, and telemetry. A product name does not say whether those components agree on file formats, schemas, defaults, authority, cancellation, or failure recovery. The broad categories remain training frameworks, serving engines, agent frameworks, and the standards that let them interoperate, but those category names do not define the contracts between them.

The useful question is therefore not “Which framework is best?” It is “What does each boundary promise, under which version and workload, and how will we know when that promise stops holding?” A component earns its place through explicit compatibility, failure containment, replaceability, and observed evidence. Those properties belong to the composed system, not to a logo.

Start with planes, not products

Three concerns recur at every layer (Figure 75.1):

• The execution plane executes the call. It loads a checkpoint, schedules tokens, invokes a tool, runs code, or delegates a task.
• The control plane decides whether that execution may happen and which version, identity, budget, route, and policy apply.
• The evidence plane records what happened: the resolved inputs, decision, timing, output status, resource use, and effect.

A single process may implement all three, but they still answer different questions. A successful tool response does not prove that the call was authorized. An authorization decision does not prove that the tool performed the intended operation. A trace does not enforce either property; it preserves evidence about what the system observed.

Each control point must therefore be explicit and testable, while execution, policy, and evidence remain separate contracts.

The layer boundaries are also different:

This table is not an ownership chart for vendors. It is a review checklist for interfaces. If two boxes are supplied by one platform, test the boundary anyway. If they are supplied by different projects, record the adapter that joins them.

Make compatibility a reviewable claim

Represent component (i) by the contract

$$C_i = (V_i, I_i, O_i, A_i, S_i, F_i, E_i).$$

For a toolchain (T=(C_1,ldots,C_n)) and workload (w), acceptance requires both component readiness and edge compatibility:

$$\operatorname{accept}(T,w) = \bigwedge_{i=1}^{n}\operatorname{ready}(C_i,w) \;\land\; \bigwedge_{(i,j)\in E_T} \operatorname{compatible}(C_i,C_j,w).$$

Every term has an operational meaning:

where:
  C_i            : the versioned component contract for component i
  i, j           : component indexes from 1 through n
  V_i            : supported version and extension set
  I_i            : accepted inputs, including formats and schemas
  O_i            : produced outputs and their semantics
  A_i            : authority requirements for identity, consent, and budget
  S_i            : state semantics for persistence, retry, and cancellation
  F_i            : failure semantics, timeouts, and recovery behavior
  E_i            : emitted evidence, including logs, traces, metrics, and receipts
  T              : the ordered set of n component contracts in the toolchain
  n              : the number of component contracts in T
  w              : the declared workload and operating environment
  E_T            : the set of directed integration edges in T
  (i, j)         : an edge from the output of component i to component j
  ready(C_i, w)  : true when component i meets its own contract under workload w
  compatible     : true when one edge preserves versions, schemas, authority,
                   state, failures, and evidence under workload w
  bigwedge       : logical AND across every listed component or edge
  land           : logical AND between the two required groups of checks

An API schema is only part of edge compatibility. Two runtimes can accept the same request while applying different chat templates, sampling defaults, stop rules, tool-call encodings, retry policies, or cancellation semantics. Those differences are semantic, not syntactic. They belong in contract tests rather than prose claims about compatibility.

The release record should pin at least the model-bundle digest, runtime image digest, exact configuration, dependency and compiler versions, hardware and driver class, protocol revisions and extensions, policy revision, and workload definition. A container digest is useful but incomplete: it does not identify the host driver, accelerator, mounted configuration, or remote services.

The ecosystem grew around different boundaries

The major tool families did not arrive as one coherent stack. Megatron-LM in 2019 made tensor-parallel training practical within its evaluated PyTorch setup, while ZeRO, published at SC in 2020 after its 2019 preprint, partitioned the states that ordinary data parallelism replicated (Shoeybi et al. 2019; Rajbhandari et al. 2020). Their mechanisms belong to Chapter 10. The tooling lesson is that distributed layout and checkpoint state became explicit interfaces between a training program and a cluster.

Serving developed a different interface. The vLLM system presented at SOSP 2023 used PagedAttention to manage key-value cache blocks without requiring contiguous physical memory (Kwon et al. 2023). Its reported throughput gains belong to the complete evaluated system, models, workloads, and baselines, not to a universal property of one algorithm. Chapter 31 and Chapter 32 cover that mechanism; here it illustrates why a runtime contract must include scheduler policy and workload.

Agent tooling exposed a third boundary. ReAct interleaved model reasoning with environment actions in 2023 experiments on question answering and interactive tasks (Yao et al. 2023). It did not define a production framework or security model, but it made the repeated model-action-observation loop concrete. MCP, introduced by Anthropic in November 2024, later standardized an application-to-capability-server interface (Anthropic 2024). A2A, introduced by Google in April 2025, standardized messages and task lifecycles between independently operated agents (Google Cloud 2025). These histories overlap, but they do not form a maturity ladder. Training, serving, host-to-tool integration, and peer delegation solve different problems.

Once a shared boundary is explicit, implementation choices that do not cross it move back into the layer itself, where the relevant chapters examine them.

Training and serving expose portability limits

A training checkpoint is useful only if the intended loader can reconstruct the required state. PyTorch Distributed Checkpoint supports load-time resharding, including changes in trainer count or parallel layout. The same documentation gives no backward compatibility guarantee for state dictionaries across PyTorch versions (PyTorch Contributors 2026). “Can reshard” therefore does not mean “can load in any future framework.” A training release should record:

• canonical model and optimizer state and the code that maps it into the running program;
• scheduler, sampler, random-state, data-position, and progress state needed for the promised form of resume;
• distributed layout and resharding assumptions;
• framework, extension, kernel, compiler, and checkpoint-schema versions;
• a tested restore procedure that resumes on the target topology and compares the next steps with the declared tolerance.

Serving has a similar gap between model format recognition and behavior. NVIDIA Triton's model-repository documentation, for example, says TensorRT plans are tied to a CUDA compute capability, ONNX support depends on the bundled ONNX Runtime and operator set, and TorchScript compatibility can change with the PyTorch version (NVIDIA 2026). A recognized filename is not a serving contract.

KServe's V2 inference protocol standardizes health, metadata, and inference operations over HTTP or gRPC. Optional behavior remains outside its minimum conformance surface (KServe Contributors 2026). Passing that wire test does not establish equivalence for tokenization, chat templates, streaming chunks, sampling defaults, structured output, tool calls, adapters, or error mapping. Test those semantics against a locked corpus before replacing a serving runtime.

Performance claims need the same discipline. Measure the production-shaped arrival process, prompt and output length distributions, concurrency, accelerator, runtime flags, failures, and tail latency. Report useful throughput only for requests that also satisfy the quality and latency gates. A benchmark that sends every request at time zero answers a different question from a bursty interactive service. Every result is workload-specific.

An agent host is not a protocol

An agent host chooses what enters context, when the model runs, how tool calls are dispatched, which state persists, and when a person must approve an effect. That is the harness studied in Chapter 41 and the model loop developed in Chapter 38. A protocol standardizes messages across one of the host's edges. A runtime or capability server decides which real effects are possible. Conflating these roles makes protocol support look like a safety or portability guarantee.

The risk also changes when the caller stops being an application and becomes an agent: a model-generated request can depend on untrusted context and can be repeated after a partial failure. The host must still mediate that request.

MCP and A2A occupy different edges (Figure 75.2):

• A tool server exposes operations or context to a host. Its schema may describe a bounded operation, but the host must independently bound real side effects.
• A peer agent has independent task state and may manage its own model, tools, policy, and lifecycle. Delegating to it crosses an accountability boundary that an ordinary function call does not.
• The agent host remains responsible for application state, policy, context, approvals, and the interpretation of results received through either protocol.

MCP: application host to capability server

Model Context Protocol (MCP) (Anthropic 2024), a common protocol for connecting models to tool servers, is more precisely a host-client-server protocol. In the current 2026-07-28 specification, the base protocol uses JSON-RPC, is stateless, and performs per-request capability negotiation. The host owns application state; an open connection is not a protocol-level conversation or session. Servers can expose tools, resources, and prompts, while optional extensions add other facilities (Model Context Protocol Contributors 2026).

MCP uses JSON Schema 2020-12 by default. Schema validation checks structure, not meaning, side effects, publisher identity, or truthfulness. The current specification also makes client and server implementation names self-reported metadata that must not drive security decisions. A production registry should therefore bind an authenticated server origin or package digest, tool name, canonical descriptor digest, protocol revision, and approved capability set. If discovery reports a changed descriptor, quarantine it and repeat review.

MCP does not grant permission merely because a tool was discovered. Its HTTP authorization framework defines how protected resource servers and clients use OAuth, while local stdio deployments obtain credentials through their runtime. The surrounding application must still provide least privilege, explicit consent, effect-specific approval, and isolation. The specification treats tool descriptions and annotations as untrusted unless they came from a trusted server and notes that the protocol itself cannot enforce the required safety principles (Model Context Protocol Contributors 2026; Model Context Protocol Contributors 2026).

A2A: client to independent agent

A2A v1.0 describes a longer-lived application interaction. An A2A server publishes an Agent Card that declares interfaces, skills, media types, capabilities, and authentication requirements. An A2A client sends a Message; the server may create a stateful Task; progress changes the task state; and results should be returned as Artifacts. Each request identifies wire version 1.0 with A2A-Version, and a tracked task eventually reaches a terminal state such as completed, failed, canceled, or rejected (A2A Protocol Working Group 2026).

Those structures make delegation observable, but they do not prove that the remote agent is correct, safe, honest, or authorized for a particular resource. An Agent Card describes claimed skills. An optional card signature can authenticate its bytes under a chosen trust policy; it does not verify competence. Authentication and authorization are applied by the deployment on every operation. A2A also permits at-least-once push delivery, so consumers must deduplicate effects rather than assume that one notification means one action.

Use a tool call when the other side performs a defined operation under the host's task. Use A2A when the other side owns independent task state, may ask for more input, and returns reviewable artifacts. Do not turn every function into an agent, and do not hide an independent agent behind a tool label. The design choices for composing several such peers continue in Chapter 43.

Security belongs to the composed path

Discovery is not authorization. A catalog entry, MCP descriptor, or Agent Card helps a caller form a request; the enforcement point must authenticate the caller and subject, check the exact resource and action, and bind the decision to current policy. The controls in Chapter 56 apply at every hop, including model gateway, agent host, tool server, peer agent, sandbox, and downstream API.

At minimum:

• issue audience-bound, short-lived credentials with least privilege; token passthrough is forbidden by MCP's authorization guidance because it bypasses resource-server validation and weakens attribution;
• keep each provider or downstream secret outside model-visible context, tool arguments, workspace files, and logs;
• sandbox local capability servers and generated code, restrict filesystem and network access, and put outbound destinations behind an allowlist;
• bind human approval to the exact tool identity and descriptor digest, normalized arguments, target resource or recipient, cost or effect bound, and expiry;
• recheck policy at execution time, since an approval does not authorize a changed request or a later retry;
• validate results and downstream effects, not only inputs, and redact secrets and sensitive content from retained evidence.

Protocol metadata can itself influence a model. The archival MCPTox study constructed 1,348 malicious descriptor cases over 353 authentic tools drawn from 45 real-world MCP server toolsets and tested 20 agent settings (Wang et al. 2026). It is controlled evidence about descriptor poisoning, not evidence that 45 deployed servers were compromised. AgentDojo separately evaluated indirect prompt injection through untrusted environment and tool content while measuring benign task utility (Debenedetti et al. 2024). These tests support a bounded conclusion: syntactically valid, interoperable content can still be adversarial, so both utility and attack resistance must be tested on the exact composed system.

Build an evidence plane across the edges

A request can cross a gateway, model runtime, host, sandbox, capability server, and downstream API. Give the logical operation one trace id and preserve child span relationships across those hops using a documented propagation format; W3C Trace Context standardizes the traceparent and tracestate headers for that purpose (Kanzhelev et al. 2021). A trace id correlates records. It is not an identity credential and must never authorize a call.

For each privileged operation, retain enough evidence to reconstruct:

• actor and subject identity, tenant, delegated authority, and authorization decision;
• resolved model bundle and serving runtime, agent-host version, protocol version, server origin, tool or Agent Card schema digest, and policy revision;
• normalized argument digest and target, with sensitive content redacted or stored under separate access control;
• approval identifier when required, plus budget reservation and reconciliation;
• start, deadline, retry, idempotency key, cancellation request and acknowledgement, terminal status, returned artifact digest, and observed downstream effect;
• latency, token and resource usage, cost, and failure classification.

Do not equate “logging enabled” with a complete audit trail. Completeness requires mediation of every effect, durable writes, stable identifiers, clock handling, retention, and access controls. Tamper evidence is another property again. Likewise, a telemetry convention is a vocabulary, not proof that every component emitted the required record. Chapter 87 develops the evaluation and observability loop that consumes this evidence.

Test portability before depending on it

Protocol adoption can reduce adapter work, but portability has layers:

1. Wire compatibility: messages parse, required methods exist, versions and capabilities negotiate, and errors have the expected shape.
2. Semantic compatibility: defaults, tool effects, streaming, artifacts, cancellation, retries, and state mean the same thing to the application.
3. Operational compatibility: the replacement meets workload-specific latency, throughput, availability, isolation, and cost requirements.
4. Governance compatibility: identity, authorization, approval, evidence, retention, and deletion obligations remain enforceable.

A conformance test usually proves only part of the first layer. Add contract tests for application semantics, negative authorization tests, malformed and adversarial content, failure injection, retry and idempotency, cancellation, restart, rollback, and workload-shaped performance. Reject silent fallback to an older protocol or a smaller capability set when the missing behavior matters.

An exit test is the clearest measure of replaceability. In a staging environment, swap the component using only the documented contract and adapter, restore state, replay the locked test corpus, and recover the original version. Record every undocumented dependency discovered. The resulting migration work, data conversion, retesting, retraining, and operational learning are switching cost even when the wire protocol stays the same.

Run this small compatibility matrix before treating a candidate as a drop-in replacement. It intentionally checks exact values: a newer schema and a missing lifecycle guarantee are incompatible until reviewed.

requirements = {
    "artifact.bundle": "sha256:release-a",
    "serving.api": "chat.v2",
    "mcp.protocol": "2026-07-28",
    "tool.schema": "calendar.v3",
    "auth.audience": "calendar-service",
    "task.cancel": "required",
}

def evaluate(label, offered):
    for key, expected in requirements.items():
        actual = offered.get(key, "missing")
        if actual != expected:
            return f"{label}: rejected ({key}: expected {expected}, got {actual})"
    return f"{label}: compatible ({len(requirements)} contracts)"

candidate_a = dict(requirements)

candidate_schema_drift = dict(requirements)
candidate_schema_drift["tool.schema"] = "calendar.v4"

candidate_no_cancel = dict(requirements)
del candidate_no_cancel["task.cancel"]

print(evaluate("candidate-a", candidate_a))
print(evaluate("candidate-schema-drift", candidate_schema_drift))
print(evaluate("candidate-no-cancel", candidate_no_cancel))

Adopt through quarantine and promotion

Treat a new framework, runtime, server, extension, or upgrade as an untrusted change to the system tuple:

1. Inventory the affected edges. Name every input, output, state store, credential, policy decision, side effect, and evidence consumer.
2. Pin the candidate. Record source or package revision, image digest, configuration, dependencies, protocol versions, extensions, and platform.
3. Build the compatibility matrix. Compare versions, formats, schemas, defaults, authority, lifecycle, failures, and telemetry with each adjacent component.
4. Update the threat model. Classify discovery metadata, tool output, artifacts, callbacks, remote prompts, and peer messages as untrusted where appropriate; narrow the sandbox and network allowlist.
5. Run conformance and contract tests. Include malformed input, negative authorization, changed descriptors, timeouts, duplicate delivery, idempotency, cancel and resume, and an unavailable dependency.
6. Run workload and adversarial evaluation. Hold the bundle and corpus fixed, measure quality, service behavior, cost, benign utility, and attack success, and compare with the approved component.
7. Exercise exit and rollback. Export state, switch away, restore the prior version, and confirm that in-flight and terminal work remain interpretable.
8. Promote the exact tuple. Sign or approve the release record, deploy progressively, watch the gates, and revoke or roll back on divergence.

From tools to economics

The toolchain record identifies more than software. It identifies engineering labor, hosted services, accelerators, storage, network paths, observability, and the operational work needed to keep every contract valid. Those costs and the switching cost exposed by the exit test feed directly into Chapter 76. Later, Chapter 88 turns the same contracts into a deployable reference stack, while Chapter 89 governs promotion and rollback over time.

A protocol makes an edge legible; only tests and policy make the composed system acceptable.

Further reading