The Deployment Lifecycle

A trained model that has been made to serve tokens (Chapter 31) is still not an operated system. Something has to decide which version is live, move traffic onto a new one without betting the whole userbase on it, notice when the behavior drifts, and undo the change when it goes wrong. Operating AI in production treats that work as its own discipline: the deployable artifact is a bundle, promotion is a statistical pipeline rather than an exact diff, and rollback has to restore the model, prompts, retrieval indexes, tools, and guardrails together.

The recurring lesson of a decade of running machine learning in production is that the model is the small part. The 2015 paper that named the problem put it bluntly: only a tiny fraction of a real machine-learning system is the learning code, and the rest is configuration, data plumbing, serving infrastructure, and the monitoring that keeps it trustworthy (Sculley et al. 2015). The deployment lifecycle is where that surrounding mass is operated, and it is the run-time half of what this book has so far built only at training and serving time.

The artifact is not the model

The first mistake is to think the thing being deployed is a set of weights. In a modern stack the unit that produces a behavior is a bundle, a container image plus its environment but for a model behavior: the model version, the system prompt, the decoding parameters, the tool schemas the model is allowed to call, the retrieval index it reads (Chapter 44), and the guardrail configuration around it. Change any one of them and the observable behavior can change as much as swapping the weights would. A new system prompt shipped against the same checkpoint is a new deployment, and treating it as a mere text edit is how a quiet prompt change becomes an unattributable regression a week later.

So the registry has to version the bundle, not the checkpoint. The operational rule is that every element that can move the output is pinned and recorded together, and a deployment refers to one immutable bundle by a single identifier. This is the discipline behind the ML Test Score rubric, which scores a system not on its accuracy but on whether its data, model, and infrastructure are tested and reproducible enough to be run by someone who did not build them (Breck et al. 2017). An interview study of practitioners found the same thing from the field: the teams that stayed sane were the ones who treated versioning of data and configuration as seriously as versioning of code, and the ones who did not spent their time chasing changes no one could attribute (Shankar et al. 2022).

The promotion pipeline

With the artifact defined, the question becomes how a candidate bundle replaces the live one. The answer is a pipeline of widening exposure, and the reason it has stages is that each stage catches a failure the previous one could not.

The first gate is offline. A candidate runs against the benchmark suite (Chapter 47), with the uncertainty discipline of Chapter 48, and, for an agent, against the trajectory evaluations of Chapter 52, the judged rubrics of Chapter 50, and the operating policy of Chapter 53. This gate is cheap and fully under your control, and it catches gross regressions before any user sees them. Its weakness is that it measures the world you already knew to test, and production is the world you did not.

So the candidate next sees real traffic without affecting anyone. In shadow or mirror mode the live request is duplicated to the candidate, the candidate's output is logged and scored, and only the incumbent's output is returned to the user. Shadowing surfaces the distribution gap between your eval set and live input at zero user risk, and its only cost is the compute to run two models for a while.

When the candidate survives shadowing, exposure begins for real. A canary routes a small percentage of live traffic to the candidate and watches online metrics, latency, error rate, refusal rate, and whatever task-success proxy the product can measure, then ramps the percentage up only while those metrics hold. The discipline of ramping on a small blast radius and watching real signals before widening is the oldest idea in the Google production playbook, predating machine learning entirely (Beyer et al. 2016). Blue-green is the same idea with two full environments and an instant cutover, traded against running two fleets at once. The pipeline is drawn in Figure 89.3.

When the ground moves

A deployment that was good on the day it shipped does not stay good, because the world it serves keeps moving. Two kinds of drift matter, and they fail in opposite directions.

Input drift is the familiar one: the distribution of what users send shifts week to week, a new use case arrives, a product launch changes who is asking, and the candidate that aced last month's eval is now answering questions it never saw. The defense is to grow the regression suite from production itself. A trace that the live system handled badly, caught by a user report or by an online judge, is promoted into the offline suite so the next candidate is graded on it. The suite is therefore never finished; it is a sediment of every way the system has already failed, and the chapter on the production data engine (Chapter 92) is where that loop is operated as infrastructure.

Behavior drift is the one peculiar to building on models you do not host. When the product calls a third-party endpoint (Chapter 73), the provider can upgrade the model under a stable name, and the bundle you thought was pinned moves without a deployment on your side. The output you validated last quarter is produced by a different model this quarter, and nothing in your own change log records it. The only defenses are to pin an explicitly dated model version when the provider offers one, and to run a continuous canary of fixed inputs against the live endpoint so a silent upgrade shows up as a metric shift you can see rather than a support ticket you cannot explain.

Incident response and the rollback you must have

Everything above assumes the change can be undone, and for a model system the undo is less obvious than for code. Rolling back weights is easy if the previous bundle is still addressable in the registry, which is the entire payoff of versioning the bundle rather than the checkpoint. A behavior that emerged from a prompt-and-retrieval interaction is harder: it reverts cleanly only when the prompt and the index version were pinned in the same bundle, which is why the registry discipline is not bookkeeping but the precondition for recovery. The operational minimum is that the previous good bundle is always one routing change away, and that the canary's online metrics are wired to an alert so a regression triggers a rollback before it reaches everyone, not after.

A minimal promotion-and-rollback control loop makes the shape concrete. The gate is a statistical comparison, and the rollback is a routing change, not a rebuild.

def promote(candidate, incumbent, traffic, alpha=0.01):
    # Offline gate: candidate must not regress the held-out suite.
    if score(candidate, OFFLINE_SUITE) < score(incumbent, OFFLINE_SUITE):
        return hold(candidate, reason="offline regression")

    # Widen exposure in steps, comparing distributions, not single outputs.
    for pct in (0, 1, 5, 25, 100):          # pct == 0 is shadow: log, do not serve
        route(candidate, fraction=pct)
        cand, base = sample_scored_metrics(candidate, incumbent, traffic)
        if worse_with_significance(cand, base, alpha):
            route(incumbent, fraction=100)   # rollback is a routing change
            return hold(candidate, reason=f"regression at {pct}%")
    return live(candidate)

What deployment really buys

The deployment lifecycle buys almost no new capability; a model is exactly as capable after it is promoted as before. What it buys is the other two. Efficiency is the cost of the machinery: shadowing and canarying mean running more than one bundle at once and paying for the eval compute that gates every promotion, and the operational question is how much of that insurance a given risk level justifies. Trust is the whole point. The pipeline exists so that a change to a production behavior is attributable, reversible, and caught small, and a team that cannot say which bundle is live, cannot promote without diffing exact outputs, or cannot roll back in one routing change has a capable model and an untrustworthy system. The next chapter takes the trust question further into the reliability of a system whose outputs are nondeterministic by construction (Chapter 90).

Further reading