The Deployment Lifecycle

Deployment is a controlled state transition from a known baseline to a candidate. The deployment release contract names the compatibility conditions, the evidence required to widen exposure, and the recovery action for each failure. Without that contract, a successful build says only that some files exist. It does not say that the candidate can coexist with the live system, that the comparison is trustworthy, or that service can be recovered.

This chapter consumes the integration release from Chapter 88 and turns it into an operated change. The unit of control is a release manifest: a signed record that composes the revisions used by a behavior, even when those parts are separately deployed. The lifecycle ends only when desired and observed state agree, the evidence is attached, and a release owner accepts the remaining risk. The output is a deployment release record.

Production ML exposed this problem before generative AI. Learning code is a small part of a deployed ML system; data dependencies, configuration, serving, and monitoring create most of its operational surface (Sculley et al. 2015). The ML Test Score consequently treats data, model, infrastructure, and monitoring tests as production-readiness evidence (Breck et al. 2017). Generative AI adds prompts, retrieval, tools, policies, stochastic evaluation, and often a hosted model whose implementation the operator cannot inspect. It enlarges the release state; it does not remove the earlier engineering obligations.

Name the release state

A model checkpoint is an artifact. It is not a release. A release manifest binds the behavior-bearing components to one reviewable identity. A deployment event maps that manifest to an environment and cohort. Observed runtime state then reports what is actually serving. Keeping those records separate matters: an approved manifest does not prove that every cell converged to it.

The manifest should fingerprint at least the following surfaces.

Controllable artifacts use immutable content digests rather than mutable tags such as latest. Secret values never belong in the manifest; it records only a scoped secret-version reference. A hosted API, live tool, or changing corpus may not offer byte identity. Record that limitation as a bounded mutable dependency with an owner, contract expiry, change-detection probe, requalification rule, and fallback. An opaque provider identifier is useful evidence, but it does not prove byte reproducibility.

Identity alone is not trust. A digest proves byte identity, not who published the bytes. A signature authenticates a statement under a trust policy, not that the release is safe. A software bill of materials inventories declared composition, not correctness. Provenance records how a build was produced, not whether its source was benign. Use these controls together: verify the artifact digest, signed manifest, attestation subject, allowed builder and source, and software bill of materials at admission. Supply-chain provenance such as SLSA and in-toto makes that chain inspectable (SLSA Community 2025; Torres-Arias et al. 2019). Update metadata should also resist a rollback attack, freeze attack, or mix-and-match of revisions that were never approved together, the threats formalized by The Update Framework (Cappos et al. 2026).

The manifest records the system fingerprint, but the release record also needs scope, owner, authority, acceptance evidence, known gaps, expiry, desired state, observed state, and recovery target. Every request, conversation, or long-running job records its assigned release so an incident can be traced to the behavior that handled it.

Separate the lifecycle verbs

The following verbs are not synonyms.

• Deploy makes a candidate available in an environment without necessarily exposing users.
• Release activates the candidate for a declared cohort.
• Promote widens exposure after the required evidence passes.
• Abort stops a transition before more exposure or irreversible work occurs.
• Rollback restores a previous compatible serving path.
• Roll forward deploys a repair when reversal is unsafe or impossible.

A feature flag can separate deployment from release. A rolling update replaces capacity while old and new versions coexist. Blue-green maintains two environments and switches traffic; it becomes a canary only if both receive concurrent traffic. A canary is a partial, time-limited release evaluated against a control (Warner and Davidovič 2018). The mechanism follows from the state and risk being changed; none is universally safest.

Prove coexistence before exposure

During a rollout, old and new code run at the same time. The compatibility envelope therefore covers more than request and response types. It includes the API and event schema, data schema and serializer, old reader and new reader, old writer and new writer, retrieval index generations, cache namespaces, session state, checkpoint state, queued jobs, and tool-effect records. The old version must safely interpret what the candidate can write if traffic rollback is meant to remain available.

Test both directions before production:

AWS's rollback-safety guidance describes the same reader-before-writer and writer-before-reader discipline for distributed deployments (Pokkunuri 2019). The conformance test matrix should run in CI and in a production-like environment with the real serializers, queues, schemas, and adapters. A passing unit test against one version cannot prove mixed-version compatibility.

When a schema cannot change in one compatible step, use expand, migrate, contract (Sato 2014). First expand readers and schemas to accept both forms while writers preserve the old form. Then dual-read or dual-write where necessary, backfill with resumable checkpoints, and verify counts, referential integrity, and semantic equivalence. Activate the new form only after mixed versions pass. Contract in a later release, after old readers, old writers, sessions, and jobs are gone and the rollback window has closed. Removing the old form is a declared recovery boundary, not routine cleanup.

Promote by widening evidence and exposure

Promotion begins with build and verify, not with user traffic. Verify digests, signatures, provenance, configuration, schemas, capacity, credentials, and the compatibility fixtures. Then run the offline gate from Chapter 87, including deterministic invariants and quality estimates over representative slices. A dark launch starts the candidate and its dependencies without routing requests, revealing startup, capacity, and control-plane faults.

Shadow execution can expose gaps between an offline set and live inputs, but it is not harmless. Duplicating a request may disclose data, consume provider quota, pollute a shared cache, mutate shared state, or execute a tool. A safe shadow path must apply current authorization before duplication, suppress side effects, use sandboxed tools, restrict egress, use an isolated cache and isolated state, cap load, quarantine its traces, and discard the candidate output. Some workloads cannot be shadowed safely.

Real exposure starts with a limited cell, then a sticky canary. Declare the assignment unit: tenant, user, session, conversation, or task. Use the unit that prevents one workflow from crossing candidate and baseline when state or experience can carry over. Keep a concurrent control, record both the assignment record and actual exposure, and label telemetry by observed release. Before interpreting outcomes, test for sample ratio mismatch; a mismatch between planned and observed allocation can reveal assignment or instrumentation faults (Fabijan et al. 2019).

Regional waves bound correlated failure. Start with a small but representative cell, then a contrasting or higher-volume region. Increase parallelism only after a bake period. Preserve failover capacity and capacity headroom; honor residency and local dependency constraints; never update every replica in one failure domain together. Where possible, keep one active change per comparison so its effect remains attributable. Publish routing and configuration atomically from a last-known-good record, and give every wave the same global stop condition. This staged shape follows established hands-off deployment practice (Liguori 2020).

Make the gate answer the release question

Deterministic checks and statistical checks answer different questions. Schema validation, authorization, policy, idempotency, tool permission, migration integrity, and route identity are deterministic invariants: one violation blocks promotion. Statistical comparison does not replace them. It estimates uncertain quality or performance differences over a population.

For required metric or slice $j$, define

$$\Delta_j = \mathbb{E}[Y_j(1)-Y_j(0)],$$

where $Y_j(1)$ is the candidate outcome and $Y_j(0)$ the concurrent-control outcome for the same declared population. Let $L_j$ be a valid lower confidence bound for $\Delta_j$, and let $\delta_j \geq 0$ be the predeclared practical loss, or non-inferiority margin, the service can tolerate. A necessary quality condition is

$$L_j \geq -\delta_j \qquad \text{for all required } j.$$

This is not the whole gate. Declare one primary metric, guardrail metrics, minimum detectable effects, slices, minimum traffic, dwell time, and delayed outcome window before the release. Require evidence completeness and fail closed when assignment, exposure, evaluator, or telemetry data is missing. Apply an absolute SLO and harm boundary even if the candidate looks better than a failing control. Shared failure, interference between cohorts, carryover from earlier assignments, novelty effects, and a delayed outcome can all invalidate a naive comparison. Online controlled-experiment practice provides the experimental foundation for these checks (Kohavi et al. 2009).

Repeatedly checking an ordinary fixed-horizon interval until it passes changes its error rate. Choose a fixed horizon or an always-valid method designed for continuous monitoring and optional stopping (Johari et al. 2022). “No significant difference” is not evidence of safety: the experiment may simply be underpowered. Promotion requires the predeclared bounds and operational guardrails to pass, not the absence of an alert.

Detect changes after promotion

A release can move without a new model. Configuration drift, policy drift, retrieval-corpus or index drift, schema drift, tool drift, evaluator drift, telemetry drift, runtime drift, and infrastructure drift can all change observed behavior. Self-hosted systems are not exempt: mutable tags, kernels, hardware, caches, and live data can move under them.

For a hosted dependency, prefer an immutable revision and record the response model or provider revision actually observed. If the provider exposes only a mutable alias, shorten the evidence expiry, run a change-detection probe, watch deprecation notices, and require requalification before or immediately after an identity change. Fixed probes cannot establish complete equivalence, but they can make an otherwise silent change visible. Production failures should feed a curated regression set through the data loop in Chapter 92; raw trace replay still requires current authorization and side-effect isolation.

Continuously reconcile desired and observed state in every cell. A healthy control plane is not proof that every replica loaded the right prompt, index, or policy. Mixed-version detection, release-labeled telemetry, and periodic black-box probes close that gap.

Recover the system, not just traffic

Recovery has several actions. An abort stops further exposure. A traffic rollback routes new work to a compatible earlier release. A state restore recovers persisted data to a known point. Feature disable or a kill switch contains one capability. Credential revoke limits a compromised dependency. Compensate reconciles an external effect that cannot be erased. Roll forward repairs a state the previous release cannot safely read. These actions are not interchangeable.

Before calling rollback safe, prove that the previous artifact is retained and loadable, its credentials and dependencies remain valid, sufficient capacity headroom exists, its readers are backward-compatible with current persisted and queued state, and no irreversible effect crosses the declared recovery window. Record the recovery time objective (RTO) and recovery point objective (RPO) (Swanson et al. 2010). Backup restoration is a recovery procedure with measurable data loss and time; it is not an ordinary routing operation.

Classify each change before rollout:

• Reversible: traffic rollback to the previous release restores service.
• Recoverable: state restore or an explicit compensate action is also required.
• Roll-forward-only: rollback becomes unsafe after a named migration or external effect.

Stopping traffic is only the start. Decide whether to drain or cancel in-flight requests, how to handle a queue and backlog, which long sessions retain their assignment, and how to prevent mixed-version writers during recovery. Reconcile the tool-effect ledger and external side effects. Then perform recovery verification: compare desired and observed manifest hashes per cell, run black-box tasks and deterministic invariants, inspect SLO and policy metrics, validate data and serializer integrity, inspect queues and caches, check dependencies, and verify the telemetry path itself. Measure RTO and RPO, observe through a bake period, and close only when explicit termination criteria pass. NIST recovery guidance likewise requires plans, testing, restoration validation, and continued monitoring (Bartock et al. 2016).

If impact or uncertainty is material, declare an incident early. Assign an incident commander, operations lead, communications lead, release owner, and recorder; freeze unrelated releases; maintain a timeline of evidence, decisions, and observed impact. Mitigation restores safe service. Root-cause analysis can continue after the emergency. Clear roles and a working record keep those jobs from competing during response (Mace et al. 2018).

Exercise failure, not only success

The release procedure is software and needs its own conformance test and failure matrix.

Drill mixed-fleet upgrade and downgrade, rollback after candidate writes, backup restore, regional evacuation, control-plane failure, telemetry failure, and effect compensation. A plan that has never restored a representative state is a hypothesis, not recovery evidence.

The complete lifecycle

For each candidate:

1. Freeze the signed release manifest, trust evidence, owner, scope, expiry, compatibility envelope, rollout policy, and recovery class.
2. Build and verify artifact identity, provenance, schemas, configuration, capacity, credentials, and mixed-version conformance fixtures.
3. Pass the offline gate: deterministic invariants, representative quality estimates, safety policy, migration checks, and recovery prerequisites.
4. Dark launch, then shadow only where data handling and effect isolation have been proved.
5. Release to a limited cell and sticky canary with a concurrent control, recorded assignment and exposure, minimum dwell, and complete telemetry.
6. Promote through regional waves only while deterministic blockers, absolute guardrails, and predeclared non-inferiority bounds pass.
7. Complete the migration only after the rollback window closes; reconcile desired and observed state and retain the last-known-good recovery path.
8. Record evidence, incidents, deviations, RTO/RPO results, known gaps, expiry, and the owner who accepted the final state.

The output is a deployment release record. It says what was intended, what actually ran, why exposure widened, what changed during the transition, and which recovery path was proved. The next chapter asks how to define reliability for the nondeterministic behavior now serving users (Chapter 90).

Further reading