Orchestration and Data Infrastructure

A frontier run holds thousands of accelerators in lockstep for weeks, and the hardware underneath it fails on a schedule of its own. The compute work, the matmuls and the collectives, is the part that looks hard, but it is not the part that decides whether the run finishes. At the scale of thousands of nodes for weeks, failure is the steady state, not the exception: a GPU falls off the bus, a NIC flaps, a host OOMs, and because the collectives are gang-scheduled (every node in the job is scheduled together or not at all), a single dead participant stalls the entire job. The operational plane is what keeps such a run alive and correct while the hardware under it breaks: checkpointing, the data plane, and observability.

Those three mechanisms draw the boundary with the neighboring chapters. The accelerators and interconnect this plane runs on are Chapter 62; the parallelism layout it checkpoints and feeds is Chapter 10. Here the question is what happens between steps and across failures: how often to save, what state must be saved, how data order survives a restart, and how operators notice faults before the loss curve misleads them.

Three obligations fall out of treating failure as the steady state, and they organize the rest of the chapter. The run must be recoverable, resuming from a recent saved state rather than from scratch when a node dies. The data-loading plane has to keep the accelerators fed, delivering tokens at the rate they consume them and doing so reproducibly. And the run needs to be observable enough that a fault surfaces from telemetry rather than from a quietly wrong loss curve days later. None of this is glamorous work, yet all of it is operational, and whether the expensive compute work pays off depends on every piece. The scheduler and cost knobs reappear once those obligations are in view.

Surviving failure: the checkpoint and the restart

The core mechanism against failure is the checkpoint: periodically serialize model parameters, optimizer state (the running statistics the optimizer keeps for every weight), the learning-rate schedule position, and the data sampler state, so the run can be reconstructed from that snapshot. The optimizer state dominates the write, since AdamW carries two moment estimates per parameter, which is the same state Chapter 10 shards across the data-parallel group (Rajbhandari et al. 2019).

A naive checkpoint stops every rank, writes to storage, and resumes, putting the full write latency on the critical path. This was the obvious first design: pause, write everything to a shared filesystem, resume. It was tolerable when models fit comfortably and clusters were small, but it scaled badly, because both the state to write and the failure rate grow with the run, so the synchronous write went from a rounding error to a visible tax. Two ideas remove the write from the critical path. Asynchronous checkpointing copies the tensors to host memory or a staging buffer and lets training proceed while a background writer drains them to storage; CheckFreq is the clean statement of this fix, pipelining the snapshot and the write so the overhead is low enough that a short cadence becomes affordable, turning checkpointing from a coarse, costly event into a routine one (Mohan et al. 2021). Sharded or distributed checkpointing has each rank write only its own shard of the state in parallel, so write time scales with per-rank state rather than total state; this followed the parallelism layout of fully sharded data parallel, where the state is already partitioned across the cluster (Zhao et al. 2023).

Cadence is an economic decision

The cadence itself is an optimization, not a constant. Checkpoint too rarely and a failure rolls back a lot of work; checkpoint too often and the writes themselves tax storage bandwidth and steal time. The optimum balances the expected work lost per failure against the cost of a write, which is why a cluster with a lower mean time between failures wants a shorter cadence. A useful first cut: if failures arrive at rate $\lambda$ and a checkpoint costs $C$ to write, the interval $T$ that minimizes total wasted time scales like $T \approx \sqrt{2C/\lambda}$, the familiar square-root tradeoff between save cost and replay cost. Frequent checkpoints shrink the work lost per failure but tax storage bandwidth and can stall the run; rare checkpoints are cheap to write but expensive when a failure hits. Asynchronous and sharded checkpointing are what make a short cadence affordable in the first place, and the optimum depends on cluster MTBF and write cost, not on a fixed number of steps.

Pick a write cost and a failure rate, then watch the numeric optimum fall out of the square-root formula and land at the bottom of the U-shaped waste curve.

import numpy as np
import matplotlib.pyplot as plt

C = 30.0          # checkpoint write cost, seconds
mtbf = 6 * 3600   # mean time between failures, seconds
lam = 1.0 / mtbf  # failure rate

T = np.linspace(60, 36000, 2000)        # checkpoint interval, seconds
waste = C / T + lam * T / 2             # write overhead + expected replay, per unit time
T_star_numeric = T[np.argmin(waste)]
T_star_formula = np.sqrt(2 * C / lam)
print("numeric optimum T*:", round(T_star_numeric), "s")
print("formula sqrt(2C/lambda):", round(T_star_formula), "s")

plt.plot(T / 60, waste, label="total waste rate")
plt.axvline(T_star_formula / 60, color="C1", ls="--", label="T* = sqrt(2C/lambda)")
plt.xlabel("checkpoint interval T (minutes)")
plt.ylabel("wasted time fraction")
plt.title("U-shaped waste versus cadence")
plt.legend(); plt.grid(alpha=0.3); plt.show()

Recovery, and the faults that do not crash

Recovery is the other half. Elastic restart detects the dead node, fences it out of the collective, and resumes the survivors from the last checkpoint, ideally onto pre-warmed spare capacity so the job does not wait for a human or for the failed host to come back. The harder faults are the ones that do not crash cleanly: a straggler node that runs slow drags every collective down to its pace, and silent data corruption (SDC) lets the run continue while producing subtly wrong numbers without an explicit crash, which no checkpoint will save you from because the checkpoint faithfully records the corruption.

The split in Figure 65.2 is the design point: a clean crash travels the fast path of detect, fence, and resume on a pre-warmed spare, while the faults that never trip the crash detector are the ones that have to be caught by observability or are beyond recovery entirely.

Where the checkpoint lives, and what is still contested

Writing checkpoints to the storage fabric is durable and survives a cluster-wide event but is slow; staging in host memory or replicating across peer nodes is fast to write and fast to restore but is lost if enough of the cluster goes down at once. The choice trades durability against checkpoint cost, and most runs combine a frequent cheap tier with an occasional durable one. Holding warm spare nodes lets an elastic restart resume in seconds instead of minutes, but those nodes are paid for and idle until a failure uses them: the reservation is insurance whose premium is utilization.

Feeding the accelerators: the data plane

The second pillar keeps the accelerators fed. The corpus is tokenized into many shards on a high-throughput storage fabric, a parallel filesystem or NVMe-over-Fabrics tier sized to feed the cluster at multiple TB/s, usually with a local NVMe cache in front of it. Each rank reads its assigned shards, prefetches and pipelines the next batch while the current one trains so the accelerators never wait on I/O, and a global shuffle keeps any single shard's local correlations from biasing a step. The load order obeys the mixture weights from Chapter 6, which set how often each source is drawn; holding to those proportions over the run is the mixture contract.

This plane evolved from map-style datasets that index a corpus held in memory to streaming, sharded pipelines that never materialize the whole corpus, because a multi-trillion-token corpus does not fit and cannot be randomly indexed cheaply.

Reproducibility ties the data plane back to the checkpoint

The property that ties the data plane back to checkpointing is reproducibility. A resume must continue the exact data order the run would have followed without the failure, which means the sampler and shuffle state, the position in the permutation, the epoch, the per-source draw counters, are part of the checkpoint, not derived fresh on restart, as Figure 65.4 shows. Without that, a resume re-feeds or skips data and silently breaks the mixture contract.

The reproducibility requirement arrived with scale. At small scale a non-deterministic resume is a nuisance, but at frontier scale, where resumes are frequent and the mixture contract is load-bearing, checkpointing the sampler state became non-negotiable. It is not free: exact data-order reproducibility costs determinism overhead, since the sampler state must be checkpointed and the shuffle must be replayable, which constrains how aggressively you can reorder for I/O efficiency. Giving it up buys some throughput but forfeits the ability to resume cleanly and to attribute a loss discontinuity to its cause.

Knowing the run is healthy: observability

The third pillar's design follows from one fact: the loss curve alone cannot tell you whether the systems work is correct. A run can be slow, comms-bound, or quietly degraded while the loss still falls. So the plane instruments the step itself: per-step wall time and its breakdown, exposed versus overlapped communication, and the grad-norm (the overall magnitude of the gradient, whose sudden jumps can destabilize training) and loss telemetry that catch a divergence or a spike early enough to act. The diagnostic unit is a per-step timeline or profile, not a glance at the loss.

This is where the faults that escaped the crash detector get caught. A straggler shows up as a slow rank in the per-step breakdown; a divergence shows up as a grad-norm spike before the loss visibly turns. Silent data corruption is the one class that even good observability struggles with, because the numbers are wrong but not anomalous, which is why it sits at the unrecoverable end of Figure 65.2.

What ties the three together: the scheduler and the cost knobs

Cost and scheduling sit underneath all three pillars. Because the collective is gang-scheduled, the job needs all its nodes simultaneously or none of them make progress, so the scheduler reserves the gang and keeps spare capacity warm to absorb a failure without a full restart. Checkpoint cadence, spare-node provisioning, and the storage-fabric bandwidth are the operational cost knobs, and they trade directly against the throughput of the run. Each pillar has already spent one of these knobs. Cadence is what buys recoverability, a fast restart comes from holding spare capacity, and storage bandwidth pays for two things at once: a durable checkpoint tier and a fed data plane. The scheduler is what lets a single reservation hold all of it at once.

How the plane is assembled

In practice the operational plane is assembled rather than bought whole. Model and optimizer sharding come from the parallelism framework, but the checkpoint, data, and observability planes are frequently custom, because their cadence, sharding, and resumability must match the exact layout of a given run.

A checkpoint at this scale is not one file. Each rank serializes its shard of parameters and optimizer state, plus the small but load-bearing scalars: the step count, the learning-rate schedule position, and the data sampler state. A sketch of the contract:

state = {
    "model_shard":     shard_of(model.parameters()),
    "optim_shard":     shard_of(optimizer.state),   # dominant term
    "step":            step,
    "lr_schedule_pos": scheduler.position(),
    "sampler_state":   loader.sampler.state_dict(),  # makes resume reproducible
}
async_write(state, path_for(rank, step))             # off the critical path

The two failure modes worth naming both surface late and quietly. A node failure with no recent checkpoint and no fast detection rolls the run back to its last save and burns the gap as pure waste, which is why detection, fencing, and a recent checkpoint are one system, not three. A non-reproducible data order on restart, from a sampler state that was not checkpointed, re-feeds or skips data and breaks the mixture contract from Chapter 6; it shows up as an unexplained loss discontinuity at every resume and is easy to misread as a training-recipe problem when it is an operational bug. Both are caught by the observability the plane installs, and both are prevented by treating the sampler state and the recovery path as first-class parts of the checkpoint.

Further reading