Part IX · Chapter 69

The Machine That Breaks at Scale

AuthorChangkun Ou

Reading time~12 min

The previous three chapters built and powered a machine of a hundred thousand accelerators. Running it exposes the same constraint at several layers. A frontier training job is failing somewhere almost all the time, so the engineering discipline has pivoted from preventing failures to amortizing them. The most dangerous failures are the ones that raise no alarm; the network's latency hierarchy and the decode bandwidth wall set hard ceilings on training and serving; and an agent that runs for hours hits the same compounding-failure logic one layer up. Fault tolerance, silent corruption, the networking ceiling, inference economics, and agent reliability are not five problems but one phenomenon seen at five layers: each is a place where a lower physical limit dictates an upper architectural choice, and where the field has not yet agreed where the tolerance should live.

Figure 69.1. Schematic of fleet reliability as systems scale. The chance that every component stays healthy falls quickly, while recovery mechanisms turn total failure into degraded progress. Idealized curves, not measured data.

A statistical certainty of failure

Synchronous training is a single distributed transaction. Every accelerator must finish its share of a step before the next step begins, so if any one of $N$ devices fails, all $N$ stall. The per-device failure rate is roughly fixed by the silicon and its memory, which makes the math of large $N$ unforgiving: the mean time between failures (MTBF), the expected interval between failures for the whole job, falls in proportion to the device count.

The public ledger that made this concrete is Meta's Llama 3 405B run. On 16,384 H100s over 54 days it logged 419 unexpected interruptions, about one every three hours, with roughly 59 percent traced to GPU or high-bandwidth- memory faults (Grattafiori and others 2024). Hold the per-device rate fixed and scale the job past a hundred thousand accelerators, and the expected interval between failures drops under half an hour. A job that takes tens of minutes to recover from each failure would then spend more time recovering than computing. The frontier is past the point where failure is an exception to plan around; it is the steady state.

Figure 69.2. Why the steady state is failure. A fixed per-device fault rate times a synchronous job of N devices drives the whole-job mean-time-between-failures below the recovery time, so the discipline shifts from preventing failures to amortizing them. The Llama 3 405B run is the public anchor.

The response is not to prevent failures, which is impossible at this rate, but to make recovery cheap enough that frequent failures barely dent throughput. The metric that captures this is the effective training time ratio (ETTR), the fraction of wall-clock time the job spends making real progress rather than recovering. A production system reporting 97 percent effective training time across a three-month run on nearly ten thousand GPUs, atop a platform of hundreds of thousands, is the state of the art (ByteDance Seed 2025), and it gets there by replacing periodic disk checkpoints with in-memory and peer-replicated checkpoints so a restart costs seconds rather than the tens of minutes a multi-terabyte reload from storage would take.

Constraint arrow

The fixed per-device failure rate dictates the training-software architecture. At a hundred thousand accelerators it forces sub-minute recovery, in-memory checkpointing, and redundant nodes, because the alternative, periodic disk checkpoints, would let recovery overwhelm computation. The parallelism of Chapter 10 is the thing this layer must keep alive through constant interruption.

The failures that raise no alarm

The failures counted in that ledger announce themselves: a GPU falls off the bus, a process crashes, a collective times out. The harder problem is the failure that does not. silent data corruption (SDC) is a fault that produces a wrong number with no crash and no error, a bit that flips in an arithmetic unit or a memory cell and quietly poisons a result. At the scale of a frontier run, with that many devices computing for that many weeks, such an event is not a possibility but a near certainty, and the danger is specific: corruption during training can steer the model toward a different optimum, degrading the result without ever tripping a flag (Ma et al. 2025).

What makes this unsolved rather than merely hard is that the cheap detectors miss most of it. A gate-level study of data-center GPUs found that not-a-number and infinity values, the signals a naive guard watches for, account for only about one percent of silent-corruption outcomes, and that single-bit flips, the model most error-correcting codes assume, are under forty percent of bit-flip events (Silent Data Corruption Study 2026). The corruption that matters is the plausible wrong answer, a finite number in the right range that is simply incorrect, and catching it requires algorithm-level checks on the matrix multiplies or deterministic replay to localize the offending node, both of which cost compute and neither of which has an agreed coverage standard. The open question is not how to detect a crash but how to detect a plausible wrong answer.

The latency hierarchy is a wall

Chapter 62 established the bandwidth tiers; here they impose a ceiling. A measurement study of collective communication across more than a hundred thousand GPUs put hard numbers on the hierarchy: relative to the latency inside a rack, crossing to another rack costs about seven times, crossing to another zone about fifteen times, and crossing to another building about thirty times (Meta 2025). Those multipliers are physics, not tuning, and they decide the parallelism map. The most communication-hungry axis must stay inside the high-bandwidth scale-up domain, which, as Chapter 66 described, has grown from a node to a rack but stops hard at the domain's edge. Past that edge, only axes that communicate sparingly can afford to cross, and the further they cross the more they pay.

This ceiling is why one of the field's live debates is shifting under it. A job that outgrows a single building's power, the constraint Chapter 68 described, must span buildings or sites, and across that distance the thirty-times latency penalty makes strict global synchronization uneconomical. The orthodox answer is to keep synchronous training and engineer around the latency; the challenger is to relax synchrony, taking local steps and reconciling occasionally. What changed recently is that the challenger acquired evidence: a cross-datacenter training method reported keeping the useful-work fraction near 88 percent while training across regions over ordinary wide-area links, under high failure rates (Google DeepMind 2026). Asynchronous cross-site training moved from a theoretical option to a demonstrated one, driven less by a wish to relax synchrony than by the simple fact that the power does not fit in one place.

The decode wall and the serving split

Serving inherits the same memory wall. Chapter 66 established the asymmetry and proposed a hardware split for it: decode, the token-by-token generation, is bandwidth-bound, its throughput set by how fast the weights and the accumulated key-value cache stream out of memory rather than by arithmetic (Erdil 2025), while prefill, the processing of the prompt, is compute-bound and tolerates cheaper memory. What that chapter left open, and this one takes up, is what serving deployment does about the asymmetry.

Figure 69.3. Why serving splits in two. Prefill is compute-bound and tolerates cheap memory; decode is bandwidth-bound and wants scarce HBM. Disaggregating them onto differently-provisioned pools lets each run on cost-optimal silicon, at the cost of streaming the KV cache between them. Whether that trade pays is contested.

A request comes in with its full prompt. Nothing is generated yet, and the work ahead splits into two phases with opposite resource profiles.

The whole prompt is processed in one parallel pass. This phase saturates the arithmetic units and tolerates cheaper, slower memory.

Prefill leaves behind the key-value cache for every prompt token, a multi-gigabyte artifact that decode reads on every step.

Tokens come one at a time. Each step reads the weights and the growing cache out of memory, so throughput is set by memory bandwidth, not arithmetic. Adding FLOPs buys nothing.

Because the phases want different hardware, run them on separate pools and stream the cache between them. The split pays only when the workload and context length repay moving a multi-gigabyte cache.

Figure 69.4. Prefill and decode are one pipeline with two resource profiles. Step through to see why a single pool fits neither well, and what disaggregation trades to fix it.

The structural response is to disaggregate: run prefill and decode on separate, differently-provisioned pools and stream the key-value cache between them, a pattern a production serving system shipped in 2026 with direct memory-to-memory cache transfer and large reported throughput gains on reasoning workloads. The cross-references for the mechanics are Chapter 31 and Chapter 32; the long-context pressure that makes the cache so large belongs to Chapter 35. The frontier question is whether the split always pays, because moving a multi-gigabyte cache between pools is itself a cost that only the right workload mix and context length repay.

The agent breaks the same way

One layer up, the same compounding logic governs how long an agent can run. If a model succeeds at each step of a task with probability $p$ , then a task of $n$ independent steps succeeds with probability $p^n$ , which decays fast even for high $p$ . Long-horizon reliability is therefore not a smooth extension of single-step accuracy; it is a product that collapses. Recent work formalized this and named a sharper effect: self-conditioning, where a model's own earlier mistakes, sitting in its context, raise the probability of later mistakes, so the decay is worse than independent steps would predict, and scaling parameters does not remove it (Study 2025). Measuring agents accordingly has its own emerging vocabulary, reliability-decay curves and degradation scores rather than a single pass rate (Study 2026), because a number that describes one attempt says little about an agent asked to work for hours.

Whether this is a model problem or a systems problem is the debate, and the chapter's framing comes down on the systems side: if raw reliability decays super-linearly with task length, the fix is the harness, checkpointing the agent's state, verifying intermediate results, and scaffolding recovery, rather than waiting for a model that never errs (cross-ref Chapter 41, Chapter 43, Chapter 52). The capability trend in the underlying models is real and is the subject of Chapter 71; the point here is that the machine fails the same way at every layer, and the answer is the same at every layer, which is to amortize failure rather than to assume it away.

What's contested

Is silent corruption a tail risk or a routine hazard? A classical view holds it is rare per device and absorbed by gradient noise. The frontier-lab view holds it is near-certain at scale and can alter convergence, so it demands always-on detection. The real disagreement is coverage versus overhead: nobody agrees what fraction must be caught or what cost is acceptable.
Synchronous or asynchronous at extreme scale? The orthodox camp keeps bulk-synchronous training and fixes it with fast recovery and faster collectives. The challenger camp accepts stale gradients and local steps to survive failures and cross-site latency, and now has demonstrated goodput, the share of throughput doing useful work rather than stalling or recomputing, to point to. Whether async matches frontier loss curves is still open.
Where should fault tolerance live? Silicon vendors push better error-correction and telemetry; the network camp wants self-healing collectives that treat a flapping link as first-class; the framework camp argues only the application knows the checkpoint and parallelism structure, so recovery belongs there. The layers are not mutually exclusive, and the field has not settled the division.
Does prefill/decode disaggregation win? Its proponents say each phase runs on cost-optimal silicon and both latency metrics improve. Skeptics say moving the cache only pays at high context lengths and specific mixes, and that a unified deployment is cheaper for short prompts.

A machine that must fail well

Here the three terms collapse into one theme: survival. Capability is bounded by survival, because a run that cannot recover faster than it fails never finishes; at the frontier, the hardest capability is staying alive rather than going fast. Efficiency, measured as the effective training time ratio and the tokens per second per accelerator, is won by attacking failure and the bandwidth wall, not by adding arithmetic. And trust here is literal: the numbers have to be right, which silent corruption quietly undermines, while an agent left to run must not compound itself into nonsense. The machine breaks at every layer in the same way, and the discipline of the frontier is keeping a machine that is always partly broken running and producing correct results. Having followed the physical and engineering stack to its limits, the next chapter turns to a different kind of limit, the one inside the learning itself.