Confidential Inference: Trusted Execution and Private Serving

Confidential inference is an end-to-end property of a serving session. It means that declared assets remain inside declared boundaries while a request is processed. Three mechanisms contribute different properties. Transport encryption protects data while it moves. Isolated execution protects plaintext while approved code processes it. Remote attestation supplies evidence about a target environment. None of these properties proves that the approved application is safe, that the model will not disclose its input, or that every component has been included. A TEE is one component, not the whole design.

This distinction turns a vague privacy claim into a systems question. Which assets are protected? From which actors? Where can plaintext exist? What does the evidence cover? Who evaluates it? Which decision releases the keys? The answers must describe one concrete deployment rather than trusted execution in the abstract. Contracts, access controls, retention rules, and audits can complement this design; they support different claims and should not be treated as either useless or equivalent to a client-enforced confidential path.

Start with assets and adversaries

The protected assets often belong to different principals. A user may need to protect prompts, retrieved context, outputs, and account identifiers from the provider's infrastructure. A model owner may need to protect weights, adapters, and system instructions from a cloud operator or customer-controlled host. During execution, both may also care about activations, the key-value cache, tokenizer state, and cryptographic keys. These goals can share machinery, but they are not symmetric by default: their owners, release policies, and accepted recipients differ (Anthropic and Pattern Labs 2025).

A useful threat model names actors and capabilities rather than saying only "the operator." The list may include a network observer; a cloud administrator who controls the host; a hostile hypervisor or host operating system; a co-tenant able to exercise shared hardware; the serving application operator and its developers; the hardware vendor and verifier whose roots and policy are trusted; and the client endpoint that creates and eventually displays the request. Authorization inside the workload remains the concern of Chapter 56, while disclosure caused by model behavior or tool use remains part of Chapter 57.

Write explicit exclusions beside the protected claims. Common exclusions are a compromised client endpoint, malicious approved application code, model output disclosure, side channels outside the vendor threat model, denial of service and other availability failures, and physical access beyond the product's declared assumptions. Rollback of sealed state is also excluded unless the design adds a freshness or monotonic-state mechanism. An exclusion is not a dismissal. It is a boundary that another control or an accepted risk must own.

Follow every plaintext copy

End-to-end review begins at the actual TLS termination, not at the first TEE in an architecture diagram. Trace the request through any load balancer or relay, tokenizer and other preprocessing, scheduler, CPU buffers, accelerator transfer, device memory, activations, key-value cache, GPU-to-GPU links, post-processing, safety filters, and response encryption. Then trace secondary copies through tool calls, telemetry, logs, crash dumps, caches, storage, and backups. If a stage sees plaintext outside the stated boundary, the stronger claim ends at that stage.

This inventory frequently changes the design. TLS may need to terminate inside the attested workload. A safety filter may need to join the measured image, or receive only an explicitly released projection. A scheduler may operate on length and resource metadata while remaining unable to read tokens. Debugging may require bounded counters rather than prompt-bearing traces. Tool calls may need independent end-to-end encryption and authorization. The data plane from Chapter 65 and the artifact lineage from Chapter 74 are therefore part of the confidentiality argument, not implementation details below it.

Verification is a release decision

The IETF Remote ATtestation procedureS architecture provides precise roles (Birkholz et al. 2023). An Attester collects claims about a Target Environment and produces Evidence. A Verifier appraises that evidence using Endorsements from manufacturers or supply-chain actors, approved Reference Values, and an Appraisal Policy for Evidence. It returns an Attestation Result. A Relying Party uses that result in its own policy, for example to admit a node or instruct a key broker to release a secret.

The separation matters. The attester does not decide whether its own state is acceptable. The verifier does not have to own the secret. The relying party does not need to parse every vendor token. In some systems one service performs several roles, but the logical responsibilities and audit records should remain distinct.

At an abstract level, evidence can bind a fresh nonce, a measurement, security claims, and an endpoint key:

$$E = \operatorname{Sign}_{sk_A} \left(n \parallel m \parallel c \parallel pk_E\right).$$

Here $E$ is evidence, $sk_A$ is an attestation signing key or a key in an endorsed signing chain, $n$ is a fresh nonce, $m$ is a launch or runtime measurement, $c$ is the set of covered security claims, $pk_E$ is an ephemeral public key for this endpoint, and $\parallel$ denotes unambiguous concatenation. Real evidence formats differ: an AMD report, an Intel quote, an Arm token, and an NVIDIA token should not all be called the same thing.

A measurement is a digest commitment to covered bytes, metadata, or events. It does not prove source identity unless a reviewed source-to-binary process binds that identity to the accepted reference value. It does not prove semantic behavior, absence of vulnerabilities, correct inference, or complete path coverage. Runtime-loaded models, drivers, configuration, or plugins are not covered merely because the VM launch measurement was accepted; the measured boot or runtime event chain must include them, or policy must bind their digests through another protected mechanism.

The verifier should validate the certificate chain and signature, freshness, endorsements, reference values, TCB status and security-version floor, revocation state, debug and migration attributes, required measurements and event logs, and deployment policy. It should also confirm that the CPU, every required accelerator, and every protected switch describe the same session and topology. Only then should the relying party communicate through the channel bound to $pk_E$ and release a session key, weight key, or data key.

Freshness and binding address different attacks. A fresh nonce rejects quote replay. Binding the ephemeral public key rejects an unbound session key and helps prevent mix-and-match evidence, cuckoo attacks, and relay attacks in which valid evidence from one machine is presented for another endpoint. Context such as tenant, service, protocol transcript, and purpose should also be bound when the protocol could otherwise reuse evidence across domains.

CPU TEEs draw different boundaries

"TEE" is a category, not a common guarantee. Intel SGX isolates enclave pages within an application address space; its initial measurement covers the enclave construction, not arbitrary host code or a GPU. AMD SEV-SNP protects private pages and CPU state of a confidential VM against a hostile host within AMD's threat model. Intel TDX places a guest in a trust domain, with an initial MRTD and runtime measurement registers whose event log gives the extensions meaning. Arm CCA creates a Realm under trusted monitor firmware and combines platform and Realm evidence. Whole-VM approaches usually reduce application porting, but guest kernels, drivers, shared-memory I/O, and attestation integration still need confidential-computing support (Costan and Devadas 2016; AMD 2020; Cheng et al. 2024; Li et al. 2022).

These boundaries have failed and evolved. Foreshadow extracted secrets from then-current SGX implementations and undermined their attestation assumptions (Van Bulck et al. 2018). The durable lesson is narrower than "hardware always leaks": the product threat model, firmware status, microcode, advisories, revocation, and side-channel posture belong in every acceptance decision.

A GPU creates a composite boundary

A CPU confidential VM does not automatically cover an accelerator. A confidential GPU has its own identity, firmware, mode, claims, and endorsement chain. CPU and GPU reports are therefore parallel evidence roots. Composite attestation is the relying party's policy for accepting those roots together, not a single linear chain in which the CPU measures every GPU.

For NVIDIA Hopper systems, evidence can cover GPU identity, confidential-mode configuration, VBIOS and firmware measurements, and driver microcode claims. Application, container, serving-engine, model, and data revisions still need to be covered by CPU measured boot, runtime measurements, or separately bound digests. The verifier must also appraise IOMMU policy, exclusive device assignment, GPU firmware and driver compatibility, the CPU-to-GPU link, GPU-to-GPU links, any NVSwitch, and the declared topology (Dhanuskodi et al. 2023; NVIDIA 2025).

The data path is product- and topology-specific. On Hopper, CPU-to-GPU PCIe traffic in confidential mode is encrypted and authenticated. Bounce buffers are shared, unprotected host pages that carry protected payloads; they are not an extension of private VM memory. On-package HBM holds plaintext and is not bulk encrypted. NVIDIA's stated physical threat assumptions for that memory are narrower than protection against a sophisticated package-level attacker. Hopper GPU-to-GPU NVLink protection depends on the platform arrangement; a multi-GPU design may instead place the GPUs and switches inside one trusted assignment. Blackwell adds protected NVLink in supported configurations, but that fact does not establish protection for every rack, switch, or topology (NVIDIA 2025).

The checklist is consequently concrete:

• verify the CPU measurement, guest attributes, event log, and endpoint key;
• verify every GPU's identity, mode, firmware, driver claims, TCB status, and freshness;
• verify each required switch and the intended GPU and switch topology;
• require exclusive device assignment and the expected IOMMU configuration;
• record whether each CPU-to-GPU link and GPU-to-GPU link provides confidentiality and integrity in this configuration;
• place the tokenizer, scheduler, model, activations, key-value cache, post-processing, and encryption endpoint inside the claimed boundary, or narrow the claim.

These checks connect directly to the memory and fabric hierarchy described in Chapter 62. A topology change is a security-relevant change, not merely a performance adjustment.

Key release closes the loop

Attestation has operational effect only when a decision controls access to a secret or service. A key broker, KMS, or client should release a weight, data, or session key only after fresh composite evidence satisfies policy. A useful acceptance predicate is:

$$\begin{aligned} &\operatorname{Fresh}(E,n) \\ {}\land{} &\operatorname{Verify}(E) \\ {}\land{} &\operatorname{Bind}( \\ &\quad E_{\mathrm{CPU}}, E_{\mathrm{GPU}*}, \\ &\quad E_{\mathrm{switch}*}, pk_E) \\ {}\land{} &\operatorname{Policy}( \\ &\quad \operatorname{claims}(E)) \\ &= \mathrm{allow}. \end{aligned}$$

Here, $E$ denotes the composite evidence set from the CPU, required GPUs, and required switches; $n$ is the verifier's nonce; and $pk_E$ is the bound endpoint key. The rule should also bind the tenant, purpose, workload measurement, model and data revision, minimum security version, expiry, and endpoint key. A model-owner key and a user session key can use different release policies even when they enter the same workload. Fail closed when required evidence is missing, stale, in debug mode, revoked, or inconsistent.

Lifecycle changes must trigger new decisions. At launch, configure and reset devices, establish the CPU confidential VM, boot and scrub GPUs, collect fresh CPU, GPU, and switch evidence, and establish an attestation-bound channel. On patch, model swap, scaling, migration, topology change, or verifier-policy change, re-attest before further release. On removal or failure, rotate or revoke keys and reset and scrub device memory. Rollback policy must distinguish an approved older image from a downgrade below the TCB or application floor.

Published systems are case studies

Apple Private Cloud Compute, Meta Private Processing for WhatsApp, and Google Private AI Compute are useful because they expose decisions above the TEE. Each is a vendor-stated design, not an independent certification. Read the source for what the vendor claims, then look for external review, deployable verification, and evidence that the described controls match production.

Apple's 2024 design states five goals: stateless processing, enforceable guarantees, no privileged access to user data, non-targetability of individual users, and verifiable transparency through published software measurements and client checks (Apple Security Engineering and Architecture (SEAR), User Privacy, Core Operating Systems (Core OS), Services Engineering (ASE), and Machine Learning and AI (AIML) 2024). Relays help separate user identity from compute routing. Apple described an expanded 2026 deployment that composes Intel TDX, NVIDIA confidential computing, and Google infrastructure with its existing verification design, illustrating that these properties are system choices rather than features of one chip (Apple Security Engineering and Architecture (SEAR), User Privacy, Core Operating Systems (Core OS), Services Engineering (ASE), and Machine Learning and AI (AIML) 2026).

Meta's 2026 Private Processing design states an AMD SEV-SNP and NVIDIA H100 boundary, RA-TLS, an oblivious HTTP relay, transparency records, artifact expiry and revocation, and constrained metrics and logs (Meta 2026). Those are separable controls: attestation covers the admitted workload, the relay reduces targetability, and telemetry rules limit secondary copies.

Google's Private AI Compute brief describes SEV-SNP frontends, hardened TPU infrastructure, mutual internal attestation, encrypted peer links, IP-blinding, and published binary digests (Google Platforms and Devices et al. 2025). Its brief also distinguishes current controls from future external inspection of remote-attestation evidence. It should therefore not be described as offering the same client-verification path as Apple's design.

AWS Nitro Enclaves illustrate a different boundary. They isolate assigned memory and vCPUs from the parent instance and provide attestation rooted in the Nitro platform. Verification still trusts AWS's attestation PKI and the Nitro system's stated operator-access properties. The useful review question is not whether this label qualifies as a TEE, but whether the platform boundary, evidence, key release, build reproducibility, and operator threat model match the application (Amazon Web Services 2025; Trail of Bits 2024).

Measure cost without inventing a universal number

Confidential mode adds work to some paths and not others. Here is a transparent latency decomposition:

$$\begin{aligned} T_{\mathrm{base}} &= T_{\mathrm{compute}} + T_{\mathrm{io}}, \\ T_{\mathrm{CC}} &= T_{\mathrm{compute}} + T_{\mathrm{io}} \\ &\quad + \Delta_{\mathrm{crypto}} \\ &\quad + \Delta_{\mathrm{copy}} \\ &\quad + \Delta_{\mathrm{driver}}, \\ \mathrm{overhead} &= \frac{T_{\mathrm{CC}}-T_{\mathrm{base}}}{T_{\mathrm{base}}}. \end{aligned}$$

$T_{\mathrm{base}}$ is baseline request time, $T_{\mathrm{CC}}$ is time in the confidential configuration, $T_{\mathrm{compute}}$ is accelerator computation, $T_{\mathrm{io}}$ is baseline input and output movement, and the three deltas represent added cryptography, copies, and driver or protocol work. A compute-heavy request can amortize fixed transfer work, but the equation does not promise that it will.

Benchmark the same hardware, same model, precision, batch, context length, output length, accelerator topology, serving engine, concurrency, padding, and warm-up state. Report throughput, time to first token, inter-token latency, p99, startup and attestation time, protected-memory capacity, and failure or retry behavior. Separate CPU-only, single-GPU, and multi-GPU results.

One H100 study using two systems, three model families, and a particular vLLM version reported less than 7% overhead for a majority of its tested typical queries, with lower throughput overhead for several larger-model cases but larger time-to-first-token effects in some configurations (Zhu et al. 2024). That is useful evidence for those workloads, not a constant for all models or deployments. Quote the hardware, model, software, metric, and workload whenever using the result.

Alternatives change the trust and cost

A TEE relocates trust to hardware, firmware, endorsements, reference values, and verifier policy. Multi-party computation distributes trust across parties; fully homomorphic encryption can let a server compute without receiving the input plaintext. Their costs depend on the model and protocol, security parameters, hardware, network conditions, approximation, and workload. Results for one transformer and sequence shape should not be turned into a universal overhead ratio.

PUMA demonstrated secure three-party inference for LLaMA-7B and made the remaining latency visible in its title and measurements (Dong et al. 2025). Newer work continues to change the boundary, including FHE designs with KV-cache support (Yu et al. 2026). Such systems may fit batch, specialized, hybrid, or high-assurance workflows even when they do not meet a particular interactive latency target.

Zero-knowledge proof systems can establish claims about correct inference and can hide a witness from a verifier, but zero-knowledge alone leaves the prompt visible to the prover that performs ordinary inference: integrity, not prompt confidentiality, for that threat (Chen et al. 2024). Differential privacy addresses training privacy and the influence of training records, not who reads a live prompt; Chapter 59 develops that distinction. On-premises serving shortens the provider path, but it still needs a local plaintext boundary, operator policy, endpoint security, and supply-chain controls.

State the residual risks

Confidential execution does not close every disclosure path:

• Traffic analysis. Encrypted streams reveal length and timing. Under the conditions of one study, an observer reconstructed 27% of tested responses exactly and inferred topics for 53% (Weiss et al. 2024). Padding, grouping tokens, or batching complete responses trades bandwidth or streaming latency for less traffic-length leakage.
• Side channels and physical scope. Cache, contention, power, fault, and package attacks are covered only where the selected product and deployment say they are. Co-tenancy and physical access policy must match that scope.
• Malicious approved code. Valid evidence for an approved debug image, unsafe parser, malicious model, or over-permissive service still admits that behavior. Measurement is not code review.
• Output exfiltration and tool exfiltration. The workload can disclose protected context through generated output, a tool call, or an allowed network destination. Authorization and egress policy remain necessary.
• Rollback and sealed state. A valid old image or restored state can violate current policy unless versions and freshness are checked outside the rolled back state.
• Supply chain and verification. Firmware signers, build systems, endorsement roots, reference-value publishers, verifier operators, and revocation channels remain trusted dependencies. Their ownership and jurisdiction are deployment-specific.
• Endpoint compromise and availability. The client sees input and output in plaintext, while the host can still delay, interrupt, or deny service. A TEE neither hardens the client nor guarantees service.

Keep an operating record

The deployment record should be machine-readable, versioned, and tied to the policy that actually released keys:

protected_assets_and_data_classes:
threat_model_and_explicit_exclusions:
plaintext_path_and_boundary_inventory:
workload_measurement_and_source_revision:
attestation_format_and_verifier_policy:
endorsements_reference_values_and_tcb_status:
freshness_and_session_key_binding:
cpu_gpu_switch_topology_and_link_protection:
image_model_data_and_key_revisions:
key_release_rotation_and_revocation:
ingress_egress_logging_and_storage_paths:
rollout_rollback_reset_and_scrub_policy:
side_channel_and_traffic_analysis_controls:
benchmark_workload_and_overhead_results:
verification_and_exception_owner:

Record rejected evidence and policy exceptions as well as successful releases. The record itself may expose software versions, infrastructure layout, or data classes, so access to it also needs a policy.

Regression scenarios

Exercise failures across evidence, topology, application, and operations:

• quote replay, stale TCB, revoked firmware, a mismatched measurement, and an unbound session key;
• TLS termination outside the boundary, an unattested GPU, mixed CPU and GPU evidence from different sessions, and a plaintext GPU link;
• an approved debug image, a crash dump or log copy, a model swap, and a targeted relay;
• a traffic-length leak, rollback of sealed state, tool exfiltration, and a verifier outage.

For every case, assert whether verification rejects, key release fails closed, an alert reaches an owner, and already released keys are rotated or revoked. Also test recovery: a verifier outage must not silently become "allow," and a patched node must not inherit the authorization of the node it replaced.

Confidentiality is a path, not a product

A defensible deployment starts with assets and adversaries, inventories every plaintext copy, composes fresh CPU and accelerator evidence, binds an endpoint key, and lets a relying party control key release. It then keeps working after launch: version floors change, firmware is revoked, topologies move, models are updated, and debugging creates new copies. A TEE can make a strong path possible. Only the complete path, verifier policy, release protocol, and operating discipline determine what the system actually protects.

Further reading