The Machine That Breaks at Scale

The previous chapters assembled accelerators, connected them, packaged the silicon, and supplied the site. Operation introduces a different question: does the system keep producing accepted work when parts of it fail? The answer cannot be read from component reliability alone. A stalled training step, a wrong tensor, a late serving request, and an agent that changes the wrong file are different outcomes. They are not interchangeable.

Reliability therefore starts with a failure contract. It names the workload boundary, the event that counts as failure, the observation window, and the test for recovery. For training, the objective may be useful progress without corrupting model state. For serving, it may be a correct result within a latency objective. For an agent, it may be a verified state transition rather than a plausible final message. Scale makes all three harder: more components create more opportunities for interruption, wider synchronization turns one fault into a larger stall, and longer trajectories give undetected errors more time to propagate. At enough scale, some component is failing somewhere almost all the time. Reliability engineering determines whether that local event becomes lost progress, a missed objective, or a wrong result. The operating objective shifts from preventing failures to amortizing them within an explicit failure budget.

Count failure before predicting it

A useful event taxonomy separates causes from symptoms:

• a planned interruption is an operator, software, data, or maintenance event known in advance;
• a fail-stop event ends a process or collective with an explicit signal;
• a fail-slow event leaves the process alive but reduces or stops progress;
• silent data corruption returns an incorrect value without an error; and
• a correlated failure affects a shared fault domain such as a host, rack, switch, storage service, power feed, or software release.

For every class, record event count, exposure, affected ranks, detection latency, diagnosis time, recovery time, blast radius, and lost work. A machine can have few crashes yet poor useful time because hangs are detected slowly. It can have excellent availability yet produce a wrong checkpoint. One aggregate “failure rate” hides both cases.

Meta's Llama 3 paper provides a rare public ledger. During a 54-day snapshot of 405B-model pretraining, which used up to 16,384 H100 GPUs, the authors reported 466 total interruptions: 47 were planned and 419 were unexpected. They classified 58.7 percent of the unexpected events as GPU issues and reported more than 90 percent effective training time (Grattafiori and others 2024). The arithmetic average between unexpected interruptions is about 3.1 hours, but the paper does not establish a fixed per-GPU law or say that every moment of the window used exactly 16,384 devices.

The familiar shorthand is mean time between failures (MTBF), the expected interval between failures for the whole job. For repairable equipment, between failures can be appropriate. For a training job, mean time between job-stopping interruptions is often the more literal name. Under a deliberately simple model, the relationship is

$$\Lambda_{\mathrm{job}} = \sum_{j=1}^{m}\lambda_j, \qquad R_{\mathrm{job}}(t) = e^{-\Lambda_{\mathrm{job}}t}, \qquad M_{\mathrm{job}} = \frac{1}{\Lambda_{\mathrm{job}}}.$$

where:

• $m$ is the number of modeled failure domains whose failure stops the job;
• $j$ indexes one such failure domain;
• $\lambda_j$ is that domain's job-stopping hazard rate, in events per unit time;
• $\Lambda_{\mathrm{job}}$ is the aggregate job-stopping hazard rate;
• $\sum$ adds the rates of the modeled failure domains, and $e$ is the base of the natural logarithm;
• $t$ is elapsed operating time;
• $R_{\mathrm{job}}(t)$ is the probability of no modeled interruption during $[0,t]$; and
• $M_{\mathrm{job}}$ is the modeled mean interval between interruptions.

This result assumes stationary, independent Poisson events. Real fleets have burn-in, aging, common software, shared networks, maintenance bursts, and different fault-containment policies. Those effects violate the assumptions. Multiplying the Llama event rate by a device-count ratio can be shown as a conditional projection: it gives about 30 minutes at 100,000 devices if every interrupting hazard scales linearly, but it is not a universal scaling law or a forecast.

A failure budget connects events to useful time

Once interruptions are measured, checkpointing becomes an economic choice. Checkpoint too often and saving state consumes the run. Checkpoint too rarely and each interruption discards more computation. Young introduced the classic first-order approximation in 1974; Daly later refined it for a more complete restart model (Young 1974; Daly 2006). A transparent first-order waste budget is

$$W(\tau) \approx \frac{C}{\tau} + \frac{\tau}{2M} + \frac{D+R}{M}, \qquad E_{\mathrm{approx}} = 1-W(\tau), \qquad \tau^*=\sqrt{2CM}.$$

where:

• $W(\tau)$ is the approximate fraction of wall time lost;
• $\tau$ is useful computation between completed checkpoints;
• $C$ is the blocking checkpoint cost;
• $M$ is mean time between detected job-stopping interruptions;
• $D$ is detection and diagnosis time after an interruption;
• $R$ is recovery time after diagnosis;
• $E_{\mathrm{approx}}$ is the remaining useful-time fraction; and
• $\tau^*$ is the Young interval that minimizes the two terms depending on checkpoint frequency.

The $\tau/(2M)$ term assumes an interruption loses half an interval on average. The whole expression is a small-waste approximation with stationary Poisson interruptions, blocking periodic checkpoints, and no failures during recovery. It does not model asynchronous snapshots, correlated faults, checkpoint failures, or silent corruption discovered long after it occurred. Use it to expose assumptions, not to replace measurement; a production system should measure ETTR directly over a declared window.

from math import sqrt

hours_observed = 54 * 24
unexpected_interruptions = 419
mean_minutes = hours_observed * 60 / unexpected_interruptions

checkpoint_minutes = 1.0
detection_minutes = 1.0
recovery_minutes = 1.0
young_minutes = sqrt(2 * checkpoint_minutes * mean_minutes)
waste = (
    checkpoint_minutes / young_minutes
    + young_minutes / (2 * mean_minutes)
    + (detection_minutes + recovery_minutes) / mean_minutes
)

conditional_100k = mean_minutes * 16_384 / 100_000

print(f"Observed interruption interval: {mean_minutes / 60:.2f} hours")
print(f"Young interval: {young_minutes:.1f} minutes")
print(f"Approximate useful fraction: {1 - waste:.1%}")
print(f"Conditional 100k interval: {conditional_100k:.1f} minutes")

The one-minute checkpoint, detection, and recovery values are illustrative; they were not reported for Llama 3. The runnable deliberately separates observed input from assumed operating costs.

Recovery is a verified control loop

A checkpoint file is only one part of recovery. The operational loop must detect loss of progress, isolate the smallest credible fault domain, restore state on known-good resources, and verify that progress and numerical behavior have resumed. Each transition has a timeout, an owner, and an acceptance criterion. Fast restart without isolation can recreate the same failure; restart without verification can continue from a corrupt state.

ByteRobust is a useful production example, not a universal prescription. The authors report up to 97 percent ETTR for one three-month training job on 9,600 Hopper GPUs. Their system combines live checks, stop-time diagnosis, over-eviction of suspected failure domains, warm standby machines, and in-memory checkpoints with a peer backup outside the evicted group. They also report every-step checkpointing with less than 0.9 percent overhead (Wan and others 2025). These numbers are a reported production result on that platform. The paper reports speedups for recovery mechanisms, but it does not justify a generic claim that every full restart takes only seconds.

The design lesson is the separation of concerns. Liveness signals answer “are ranks moving?” Performance signals answer “are they moving at the expected rate?” Correctness checks answer “are they producing an acceptable state?” Quarantine policy answers “which resources must leave the next attempt?” The control plane needs all four.

Silent corruption needs a correctness path

silent data corruption (SDC), silent data corruption, is an incorrect computation that produces no explicit hardware or software error. It matters because ordinary availability telemetry can remain green while model state diverges: the computation may raise no alarm. Its rate and effect must be kept separate: a study of what corruptions look like does not establish how often they occur in a fleet, and a study of already-faulty nodes does not estimate the prevalence among healthy nodes.

Ma and colleagues paired fifteen unhealthy nodes flagged by production fleet management with fifteen healthy nodes. Under deterministic execution they found cases where weights drifted while pretraining loss remained nearly unchanged; in separate fine-tuning experiments, some affected nodes produced visible loss spikes, including one run with zero final accuracy. The study used single-node tensor parallelism and explicitly notes its limited sample and scale (Ma et al. 2025). It demonstrates possible consequences, not the probability that an arbitrary frontier run is corrupted.

A second study asked a different question. Tung and colleagues ran a gate-level stuck-at fault-injection simulation on a reduced two-SM model of one production-class GPU architecture using 63 CUDA micro-benchmarks. Within that simulated campaign, NaN and infinity values were 1.01 percent of observed corruptions, and single-bit events were under 40 percent of non-special bit-flip corruptions (Tung et al. 2026). Those figures are a fault-model outcome distribution, not a fleet incidence rate. They show why a detector that only watches for NaN or injects uniform single-bit faults has weak coverage; they do not show that error-correcting memory is ineffective.

A practical correctness path uses several layers: admission stress tests against a golden output; online numerical invariants and progress canaries; comparison or replay when a rank becomes suspect; quarantine of the implicated failure domain; rollback to a checkpoint older than the first divergent step; and a clean replay test before return to service. Coverage, overhead, detection latency, and false-positive rate must all be measured. “We watch the loss” is not a coverage statement.

Locality sets the synchronization boundary

Failures are amplified by synchronization. In bulk-synchronous training, a straggler or failed participant can delay every rank at the next collective. The cost depends on message size, collective algorithm, placement, congestion, and topology, not distance alone. Chapter 62 introduced these network domains; Chapter 66 connected them to parallelism placement.

NCCLX reports one concrete hierarchy. In Meta's topology connecting more than 100,000 GPUs, latency across racks within one AI Zone, across AI Zones, and across data-center buildings was about 7, 15, and 30 times same-rack traffic, respectively (Si and others 2025). These are topology-specific ratios for that RoCE fabric, not constants of nature. They motivate placing exposed, latency-sensitive communication in the smallest practical domain while measuring whether larger collectives can overlap their traffic.

Relaxing global lockstep changes both the failure domain and the optimization algorithm. Decoupled DiLoCo provides two distinct demonstrations that should not be merged. In one experiment, a synthetic event tape represented 1.2 million chips with a one-year per-chip interruption interval. Eight learners retained 88 percent goodput, compared with 58 percent for elastic data parallelism, with similar reported evaluation results. Separately, the authors trained a 12B model using eight learners placed across U.S. regions and reported step speed close to a collocated DiLoCo run (Douillard and others 2026). The first result is simulated failure exposure; the second is a geographically distributed research experiment. The latter demonstrates feasibility at that scale; it is not frontier-scale production.

This is the connection to Chapter 68: distributing power may create a reason to distribute training, but it does not make asynchronous optimization free. A proposal needs an algorithmic quality comparison at matched tokens and compute, a network and failure trace, a stale-update policy, a recovery test, and a cost for duplicated model state. Goodput without final quality is not useful progress.

Serving: a split adds a handoff

Serving has a different transaction boundary. A request passes only when its output is accepted and its latency meets the service objective. The two common latency measures are time to first token (TTFT), dominated by queueing and prompt processing, and time per output token (TPOT), experienced during generation. SLO goodput is the sustainable request rate whose TTFT and TPOT remain within declared thresholds; raw tokens per second can rise while SLO goodput falls.

Prefill and decode often have different resource profiles, but the labels are conditional. Prefill exposes parallel work across prompt tokens and commonly reaches higher arithmetic utilization. Autoregressive decode at modest batch size repeatedly streams weights; with conventional attention and a long context it also reads a growing key-value cache. Batch size, prompt length, output length, model architecture, precision, parallelism, and network latency can move either phase to a different bottleneck (Erdil 2025). “Decode is bandwidth-bound” is a workload statement, not an identity. This interaction is often summarized as the latency hierarchy and the decode bandwidth wall, but a deployment must measure both under its own request mix.

DistServe showed one response: place prefill and decode on separate GPU pools, provision each independently, and account for the KV-cache transfer between them. Its OSDI 2024 evaluation reported up to 7.4 times more requests or 12.6 times tighter latency objectives than its tested baselines, not a universal speedup (Zhong et al. 2024). Mooncake provides production evidence for a related design. The Kimi serving platform uses separated pools, a distributed KV cache, and a transfer engine; its FAST 2025 paper reports results from real traces and deployment across thousands of nodes (Qin et al. 2025).

The split also creates another failure boundary. The scheduler must know whether a failed KV-cache transfer can be retried, recomputed, or routed to a co-located replica. It must protect decode from head-of-line blocking and keep duplicate model capacity available in both pools. A co-located design with continuous batching or chunked prefill avoids the handoff and may win for another request mix. The decision needs arrival traces, TTFT and TPOT targets, KV size, transfer bandwidth, queueing, failure behavior, and cost.

The serving mechanics connect to Chapter 31 and Chapter 32. Long-context cache pressure is developed further in Chapter 35.

Agents need transactions too

An agent trajectory is not a single model call. An agent that runs for hours performs a sequence of proposed state changes, tool responses, validations, and retries. The probability that all required steps succeed follows the chain rule:

$$\Pr(S_{1:n}) = \prod_{i=1}^{n} \Pr(S_i \mid S_{1:i-1}).$$

where:

• $n$ is the number of required steps in the declared task decomposition;
• $i$ indexes the current step;
• $S_i$ is the event that step $i$ satisfies its acceptance condition;
• $S_{1:i-1}$ is the history of acceptance events before step $i$; and
• $S_{1:n}$ is the event that all steps from 1 through $n$ are accepted;
• $\Pr$ denotes probability; and
• $\prod$ multiplies the conditional probabilities across the indexed steps.

If every step is independent, has the same success probability $p$, and no earlier error can be corrected, the expression reduces to $p^n$. That is a useful null model, not a forecast for arbitrary agents. Tool failures, self-correction, retries, shared hidden state, and a growing error-filled context make the conditional probabilities history-dependent.

Sinha and colleagues isolated execution in a controlled multi-turn task. They found that errors inserted into prior turns reduced later-step accuracy, an effect they call self-conditioning. Scaling the tested non-thinking models did not eliminate the effect, while the thinking variants tested in the study did not show it (Sinha et al. 2026). That result is evidence for one controlled task, not proof that every agent decays at the same rate.

Two other metrics answer different questions. METR's time horizon is the human expert task duration at which a fitted agent has a stated probability of success; it is not a count of model steps and remains benchmark- and scaffold-dependent (METR 2026). The $\tau$-bench reliability metric asks whether the same task succeeds across repeated trials:

$$\operatorname{pass}^{k} = \Pr(Y_1=1,\ldots,Y_k=1).$$

where:

• $k$ is the number of repeated trials of one task;
• $Y_r$ is 1 when trial $r$ reaches the verified goal state and 0 otherwise;
• $r$ indexes a trial; and
• $\operatorname{pass}^{k}$ is the probability that all $k$ trials pass (Yao et al. 2025).

The distinction matters: $p^n$ is a within-trajectory baseline, whereas $\operatorname{pass}^{k}$ measures consistency across repeated trajectories. A deployment should report task distribution, scaffold and tool versions, retry policy, partial-credit rule, and confidence intervals rather than a single pass rate.

The systems response is to give each consequential action a transaction boundary: declare preconditions, execute an idempotent or compensatable step, check postconditions against external state, persist a compact checkpoint, and either continue, retry, roll back, or escalate. This is the role of the harness in Chapter 41 and the evaluators in Chapter 52; it does not turn a weak model into a capable one. Chapter 71 tracks the changing model frontier, while this chapter asks what must surround any given model before it can be trusted to run for longer.

Make the evidence replayable

A reliability review should be reconstructible from records, not remembered as an outage story. Freeze an event taxonomy and attach a source timestamp, software and firmware versions, topology, affected ranks, and failure-domain classification to every event. Record detection time, isolation time, recovery time, checkpoint age, lost accelerator time, first divergent step when known, and the verification that closed the incident.

Training reports should separate planned and unexpected stops, fail-stop and fail-slow exposure, checkpoint overhead, ETTR, and numerical-integrity alerts. Serving reports should include the arrival and length distribution, TTFT and TPOT percentiles, SLO miss count, retry amplification, and accepted-output rate. Agent reports should retain the task version, complete action log, external-state diff, verifier output, and intervention. For every layer, keep a replay test, a fault injection that exercises the recovery path, an acceptance criterion, and an owner for failed evidence.

This evidence contract prevents three common mistakes: projecting one operator's event rate as a law of hardware, quoting a simulated failure model as field incidence, and reporting throughput without correctness or latency. It also makes improvements falsifiable: a new detector should reduce detection latency at stated coverage; a new checkpoint path should reduce lost work; a new serving split should improve SLO goodput; and a new agent harness should improve verified task completion on repeated trials.

A machine that fails within bounds

A reliable machine is not one that never fails. It is one whose failures are defined, detected, contained, recovered, and checked before more state is trusted. Training reliability protects expensive progress. Serving reliability protects a latency and correctness contract. Agent reliability protects the world the agent is allowed to change. The mechanisms differ, but the operating discipline is consistent: count at the right boundary, make assumptions visible, and close every recovery loop with evidence.

The infrastructure part of the book ends here. The next chapter moves from limits of the machine to limits of learning itself.

Further reading