Accelerators and Networking

A training run lives or dies on bandwidth, not peak FLOPs. The useful view is a bandwidth hierarchy: the high-bandwidth memory (HBM) sitting beside the accelerator die feeds a chip; NVLink and NVSwitch connect accelerators inside a node; and InfiniBand or RoCE carry bytes between nodes. That hierarchy explains why a tensor-parallel group almost never spills past the NVLink boundary, why data and pipeline parallelism can cross the network, and how model FLOPs utilization measures whether the hardware floor is being used.

A run that the spec sheet cannot explain

Picture a frontier run spread across thousands of accelerators for weeks. The spec sheet promises a peak arithmetic rate, and the run delivers a fraction of it. The missing factor is rarely the matrix-multiply units sitting idle for want of work. It is the bytes those units wait on: how fast a chip reaches its own memory, the rate at which two chips in a chassis can talk, and what a message costs to travel between two chassis across the hall. Those three numbers differ by more than an order of magnitude each step out, and a parallelism axis that needs more talking than its tier can afford stalls the whole collective, the coordinated cross-device communication operation it depends on.

So the spec sheet's peak FLOPs is the wrong place to start. The right place is the hierarchy of bandwidth tiers underneath it, because every layout decision is a question of which tier a given exchange can afford. Here the hardware floor means the accelerators, the interconnect hierarchy, and the single number, MFU, that says whether the floor is being used. The parallelism algorithms built on top belong to Chapter 10; cluster orchestration, checkpointing, and the data plane belong to Chapter 65.

Three tiers of bandwidth

The machine is a hierarchy of bandwidth tiers, and tracing a byte from the die outward shows why each tier exists and what it can host.

The innermost tier is a chip and its memory. An accelerator pairs a large matrix-multiply throughput with HBM sitting next to the die. The forward and backward matmuls are fed from HBM, so HBM bandwidth, not raw FLOPs, often sets the achievable rate for a kernel.

Drag the ceiling to see why a kernel that does little work per byte fetched stays bandwidth-bound and never reaches the peak FLOPs the spec sheet promises.

The next tier is intra-node: the few accelerators inside one chassis. On NVIDIA hardware these are wired with NVLink, a dedicated chip-to-chip fabric, and NVSwitch, which lets every accelerator in the node reach every other at full NVLink bandwidth rather than over the slower PCIe bus. This tier is where the most communication-hungry axis can live, because NVLink bandwidth is high enough to hide a collective behind compute. Whether a collective hides at all is a ratio, as Figure 62.2 sketches: as long as a step does more compute than communication the collective vanishes entirely under it, and the inter-node network falls short precisely because its bytes are dear enough to leave some communication exposed.

The outermost tier is inter-node: chassis talking to chassis across a datacenter network. Here the fabric is InfiniBand or RoCE (RDMA over Converged Ethernet), wired in a topology such as a fat-tree or a rail-optimized layout, with a network interface card (NIC) per accelerator setting the per-device injection bandwidth. This tier is an order of magnitude slower per byte than NVLink, so only axes that communicate sparingly can afford to span it.

Mapping parallelism onto the tiers

Once the tiers are fixed, the placement of each parallelism axis follows directly, and it follows from communication volume, not from taste. Tensor parallelism (TP) shards a single matmul across devices and exchanges activations, the intermediate values passed between devices, with an all-reduce, in which every device sums its partial result with all others, twice per layer, so it is the most bandwidth-hungry axis and is pinned to the NVLink tier inside a node (Shoeybi et al. 2019). Data and pipeline parallelism communicate far less per step, a gradient reduction once per step for DP and a single activation hand-off per stage boundary for PP, so they are the axes allowed to cross the inter-node network (Narayanan et al. 2021). Expert parallelism for MoE rides its own all-to-all group, treated in Chapter 9. The rule of thumb that follows, TP inside the node and DP/PP across nodes, is not a convention but a consequence of the bandwidth gap, as Figure 62.5 lays out.

The TPU detour: a torus instead of a switch

The tensor processing unit (TPU) pod is Google's accelerator cluster counterpart with a different JAX/XLA cost model, and it makes the point that the tier hierarchy above is one design choice rather than a law of nature. The chips connect through the inter-chip interconnect (ICI), the pod's internal inter-chip interconnect, and an optical circuit switch (OCS) into a torus topology rather than a switched fat-tree, so neighbor-to-neighbor bandwidth is plentiful and the sharding is co-designed with the torus shape through GSPMD-style annotations rather than mapped onto a fat-tree after the fact (Xu et al. 2021). The same byte that would cross a switch on a graphics processing unit (GPU) cluster, a cluster of graphics processors used as accelerators, travels to a physical neighbor on a TPU pod, which is why bandwidth there scales with locality.

MFU: what fraction of the hardware you actually used

The single number that tells you whether all of this is working is model FLOPs utilization. MFU is the FLOPs the model actually needs, divided by the peak FLOPs the hardware could deliver over the same wall-clock:

$$\mathrm{MFU} = \frac{\text{model FLOPs per step}}{\text{peak hardware FLOPs} \times \text{step time}}$$

The numerator is the useful arithmetic the model's forward and backward pass requires for one step. The denominator is the machine's theoretical arithmetic capacity over the wall-clock time that step actually took. MFU therefore counts only the useful arithmetic of the model. Its sibling, hardware FLOPs utilization (HFU), also counts work the hardware genuinely did but the model did not strictly need, most notably the extra forward passes of activation recomputation, which is why HFU exceeds MFU whenever recomputation is on. MFU answers the question the budget cares about: of the arithmetic I paid for, how much moved the model.

Why the tiers look the way they do

The interconnect tiers are the product of the bandwidth gap widening faster than any single fabric could close it. Early multi-GPU training leaned on PCIe, which was already too slow for sharding a matmul, so NVLink and then NVSwitch were introduced precisely to make the intra-node tier fast enough to host tensor parallelism (Shoeybi et al. 2019). The accelerator families also moved the precision floor: each generation has widened the set of formats the matmul units execute natively, with Hopper-class and newer hardware adding fp8, which lets the same silicon deliver more matmul throughput per byte moved (Micikevicius et al. 2022) (the kernel-level consequences are in Chapter 34). On the network side, InfiniBand and RoCE converged on remote direct memory access (RDMA), which lets one machine move bytes into another's memory without routing through the CPU, and on rail-optimized topologies that give each accelerator a dedicated path to its peer on other nodes, so the inter-node tier could grow without every flow contending for the same switch ports. TPU pods evolved along a parallel track entirely, choosing a torus over a switched fabric so that bandwidth scales with locality.

What each axis costs

Every parallelism axis buys a different resource and pays in a different currency, and the interconnect tier sets the exchange rate.

• Tensor parallelism against bandwidth. TP cuts per-device memory and latency but spends intra-node bandwidth on two all-reduces per layer. That spend is affordable only on NVLink, so the TP degree caps at the NVLink domain. Push it past the node and the all-reduce falls onto the inter-node network, where it cannot hide under compute and shows up as lost MFU. Figure 62.6 makes the cliff visible: MFU holds a plateau while TP fits inside the NVLink domain, then falls away the moment the degree spills onto the inter-node network.

Change nvlink_size and slowdown and watch where the plateau breaks and how steep the cliff becomes.

import numpy as np
import matplotlib.pyplot as plt
nvlink_size = 8       # accelerators reachable at NVLink bandwidth
peak = 0.55           # MFU plateau when communication hides under compute
slowdown = 10.0       # inter-node bytes are this many times dearer than NVLink
tp = np.arange(1, 33)
# fraction of the step lost to exposed (un-hidden) all-reduce once TP spills
comm = np.where(tp <= nvlink_size, 0.0, (tp - nvlink_size) / tp * (1 - 1 / slowdown))
mfu = peak * (1 - comm)
plt.plot(tp, mfu, marker='o')
plt.axvline(nvlink_size, ls='--', color='gray', label='NVLink boundary')
plt.xlabel('tensor-parallel degree'); plt.ylabel('MFU')
plt.ylim(0, 0.6); plt.legend(); plt.title('TP past the node falls off a cliff')
print('MFU at TP=8:', round(mfu[7], 3), ' at TP=16:', round(mfu[15], 3))
plt.show()

• Pipeline parallelism against the bubble. PP is bandwidth-cheap, a single activation hand-off per stage boundary, so it crosses nodes comfortably, but it pays a pipeline bubble at the fill and drain of each step.
• MFU against memory. Recomputation lets a bigger model or longer sequence fit by redoing forward work in the backward pass, which raises HFU but lowers MFU. The number you optimize depends on whether memory or throughput is the binding constraint.

Where MFU becomes visible

The collectives that move bytes across these tiers are provided by NCCL on NVIDIA hardware (RCCL on AMD), which maps each collective onto the topology, choosing a ring or a tree and routing it over NVLink or the network depending on where the ranks sit (NVIDIA 2024). A practitioner rarely calls it directly; the parallelism framework does. What the practitioner reads is the per-step profile, because MFU is diagnosed from a timeline, not from the loss curve. Exposed communication that did not hide under compute, an oversized pipeline bubble, too much recomputation, and unfused kernels each take a slice of the gap between achieved and peak FLOPs, and the timeline is where each slice is visible.

Figure 62.7 reads the timeline as a descent from peak FLOPs down to MFU. Idle time, where a device waited on exposed communication or sat in a pipeline bubble, is pure loss. Of the FLOPs the hardware then actually ran, recomputation did work the model did not strictly need: that work lifts HFU above MFU, but only the useful arithmetic counts toward the true figure.

Further reading