Accelerators and Networking

An accelerator specification does not predict application performance. Peak arithmetic, memory capacity, and link rates are ceilings for particular operations under particular conventions. A useful performance claim must also name the workload, tensor shape, precision, software stack, topology, and measurement method. Change any of them and the bottleneck may move.

This chapter builds the hardware model needed to reason about that movement. It starts inside one accelerator, follows data through memory and interconnects, then shows how to measure the complete system. Parallel algorithms are covered in Chapter 10; cluster control, checkpointing, and the data plane belong to Chapter 65.

What an accelerator actually accelerates

An accelerator is not one large arithmetic unit. A host CPU starts the program, manages operating-system work, and launches device kernels. The device combines general-purpose parallel lanes, memory controllers, caches, and specialized matrix engines. On NVIDIA GPUs, groups of threads execute through the single-instruction, multiple-thread (SIMT) programming model. Other vendors use different names and execution details, so SIMT should not be treated as a universal accelerator instruction set (NVIDIA 2026).

A matrix engine performs tiled matrix multiply-accumulate operations. The tile shape, accepted input formats, accumulation precision, and instruction semantics vary by architecture. A systolic array is one possible dataflow design in which values move rhythmically through a grid of processing elements. Google's first TPU used a documented systolic matrix unit, but that fact does not establish the physical implementation of every GPU matrix engine or later accelerator (Kung 1982; Jouppi et al. 2017).

The kernel is the executable unit that turns this hardware into useful work. Performance depends on whether its tensor dimensions fit the available tile shapes, memory accesses coalesce, branches diverge, and enough independent work exists to occupy the device. Lower-precision inputs can reduce storage and traffic, and native low-precision matrix instructions can raise peak arithmetic. Those gains require compatible kernels and shapes. Casts, scaling, non-matrix operations, and numerical safeguards remain real costs (Micikevicius et al. 2022). Likewise, a structured sparsity peak applies only when the required sparsity pattern and supported instruction path are actually used. Dense and sparse peaks are not interchangeable.

The roofline is a bound, not a diagnosis

The roofline model relates arithmetic to data movement at one named memory boundary (Williams et al. 2009), where $F$ denotes the number of floating-point operations performed by a kernel and $Q$ denotes the bytes transferred across that boundary. Their ratio is

$$I=\frac{F}{Q}$$

where $I$ represents arithmetic intensity in FLOPs per byte. Here, $P_{\mathrm{peak}}$ denotes peak arithmetic throughput for the chosen instruction and precision, while $B_{\mathrm{mem}}$ denotes sustainable bandwidth at the same boundary. The resulting bound is

$$P_{\mathrm{achieved}} \le \min\!\left(P_{\mathrm{peak}},\;I B_{\mathrm{mem}}\right).$$

Here, the two ceilings meet at the ridge point

$$I^*=\frac{P_{\mathrm{peak}}}{B_{\mathrm{mem}}}.$$

Below $I^*$, data movement bounds throughput; above it, arithmetic can become the tighter bound. This does not make runtime equal to the bound. Launch cost, dependencies, occupancy, cache misses, imbalance, power limits, and unsupported tile shapes can all lower achieved throughput.

The byte count also needs a boundary. A kernel may have high intensity when traffic is counted at high-bandwidth memory (HBM), but low intensity at a cache level. Report whether $Q$ means HBM traffic, host-device traffic, or network traffic, and use a measured sustainable bandwidth rather than silently substituting the vendor peak.

Capacity, bandwidth, latency, and traffic

Inside a typical GPU, data can move among registers, software-managed shared memory, cache, and device memory. The high-bandwidth memory (HBM), high-bandwidth memory packaged beside the accelerator, provides the large device-memory pool. These levels do different jobs:

• Capacity decides whether weights, gradients, optimizer states, activations, temporary workspaces, collective buffers, and allocator headroom fit at all. Serving adds the KV cache described in Chapter 31.
• Bandwidth limits how many bytes can cross a boundary per second under a given access pattern.
• Latency is the delay before a dependent access can be used. Parallelism can hide some latency, but not an unmet dependency.
• Traffic is the number of bytes the program actually causes to move. Kernel fusion, recomputation, sharding, caching, and layout can change it without changing a link's rated bandwidth.

Capacity is not bandwidth, and neither is utilization. A workload can fit in HBM yet run slowly because it repeatedly streams data. Another can be compute-bound even on a machine with the same HBM. State explicitly where traffic is counted before comparing results.

The same caution applies as data moves outward. The interactive below shows the conceptual hierarchy, not measured ratios. Actual systems differ in domain size, directionality, endpoint injection, and sustained throughput.

An interconnect is a stack

“The network” hides three separate design layers:

1. Physical topology. Links and switches determine hop count, path diversity, failure domains, endpoint injection, bisection bandwidth, and oversubscription. A fast endpoint link does not make an oversubscribed fabric non-blocking.
2. Transport. PCIe and NVLink move data within some systems; InfiniBand or Ethernet with RoCE move it across a scale-out fabric. The physical domain of each transport is product-specific. Some NVSwitch systems connect a small server, while newer designs can extend an NVLink domain across a rack (NVIDIA 2026).
3. Collective library. Software such as NCCL maps logical operations onto available paths and algorithms. It is topology-aware, but it cannot create bisection bandwidth or repair a poor rank placement (NVIDIA 2026).

remote direct memory access (RDMA), remote direct memory access, lets a network adapter transfer data between registered memory regions without putting the host CPU on the data path. RoCE supplies RDMA semantics over Ethernet; InfiniBand supplies them in its own network stack. GPU-direct variants can expose GPU memory to a peer device, but memory registration, PCIe placement, drivers, and synchronization still matter (NVIDIA 2026). “Bypasses the CPU” describes the data path, not a system with no CPU setup or control work.

Scale-up and scale-out are therefore useful descriptions, not fixed synonyms for “inside a node” and “outside a node.” Record the actual endpoints and switches. Do not compare a bidirectional aggregate NVLink figure with a one-direction NIC rate, or an aggregate switch number with per-device injection.

Collectives have semantics and algorithms

A collective describes what a group of ranks receives:

• broadcast: copy one root's buffer to every rank;
• all-reduce: combine values across ranks and return the result to every rank;
• all-gather: concatenate every rank's shard at every rank;
• reduce-scatter: combine values and leave each rank with one result shard;
• all-to-all: send a distinct shard from every rank to every other rank.

These semantics do not choose the algorithm. An all-reduce may use a ring, a tree, a hierarchical composition, or another schedule. The library can also split one semantic operation into phases; reduce-scatter followed by all-gather is equivalent to all-reduce for matching reductions and layouts (NVIDIA 2026).

A simple latency-bandwidth model makes the trade-off visible. For a message of $n$ bytes,

$$T_{\mathrm{msg}}(n)=\alpha+n\beta,$$

where $\alpha$ is the per-message startup latency and $\beta$ is seconds per byte, the reciprocal of effective bandwidth. Under an idealized ring with $p$ ranks, equal links, no contention, and no overlap, a large all-reduce is approximately

$$T_{\mathrm{ring}}(n,p)\approx 2(p-1)\alpha+\frac{2(p-1)}{p}n\beta.$$

The ring is bandwidth-efficient for large messages because each rank transfers about $2(p-1)n/p$ bytes, but its $2(p-1)$ phases make startup expensive for small messages (Patarasuk and Yuan 2009). A tree changes the latency term and traffic pattern. Real results also include protocol choice, chunking, reduction work, routing, congestion, and unequal links.

Communication overlaps compute only when the dependency graph exposes independent work and the runtime schedules both successfully. Bucket size, kernel duration, stream priority, memory pressure, and concurrent traffic alter the overlap window. Here, $T_{\mathrm{comp}}$ denotes isolated compute time, $T_{\mathrm{comm}}$ denotes isolated communication time, and $T_{\mathrm{overlap}}$ denotes their overlap. The accounting identity is

$$T_{\mathrm{step}}=T_{\mathrm{comp}}+T_{\mathrm{comm}}-T_{\mathrm{overlap}}, \qquad 0\le T_{\mathrm{overlap}}\le \min(T_{\mathrm{comp}},T_{\mathrm{comm}})$$

makes the condition explicit. A profile may show a collective fully hidden in one model shape and exposed after a batch-size, placement, or software change.

Place a communication graph, not an acronym

Each parallelism method creates a different communication graph. The table gives common patterns, not fixed operation counts.

The original Megatron tensor-parallel layout is an important example, not a universal law: its transformer formulation uses two all-reduces in the forward pass and corresponding reductions in backward for each layer (Shoeybi et al. 2019). Later compositions change the operations and overlap schedule. Data-parallel systems may use all-reduce, or reduce-scatter and all-gather under ZeRO and fully sharded designs (Rajbhandari et al. 2020). Pipeline traffic repeats with microbatches rather than appearing as one hand-off per step (Narayanan et al. 2021). Expert parallelism and its all-to-all behavior are developed in Chapter 9.

A practical mapper measures or estimates bytes, frequency, latency sensitivity, memory saved, and available overlap for every edge. It then maps the heaviest or most latency-sensitive mesh dimensions onto the fastest adequate physical dimensions. “TP inside, DP outside” is often a useful starting heuristic. It is not a proof, especially on rack-scale fabrics, asymmetric networks, or workloads whose context and expert traffic dominate.

TPU v4 as one bounded case study

The tensor processing unit (TPU) pod, a cluster of Google tensor processing units, illustrates why topology claims need a generation. TPU v4 used an inter-chip interconnect (ICI), the inter-chip interconnect joining TPU chips, plus optical circuit switches to configure three-dimensional torus slices. A twisted torus was one supported choice; the switches also helped reconfigure around unavailable components (Jouppi et al. 2023). This is a TPU v4 statement, not a claim that every TPU pod uses the same torus.

GSPMD is a compiler partitioning system. It propagates user sharding annotations through a computation graph and targets a logical device mesh (Xu et al. 2021). It does not by itself make a physical topology fast or remove multi-hop communication. Similarly, a graphics processing unit (GPU) cluster, a cluster that uses graphics processors as accelerators, may expose several logical meshes over PCIe, NVLink, InfiniBand, or Ethernet. In both cases, the compiler and runtime need an accurate physical map.

The lesson is generation-specific: accelerator architecture, logical sharding, and physical topology must be read together.

Precision and portability are separate contracts

Precision changes more than one number. Input storage, multiplication, accumulation precision, output format, scaling policy, and reduction precision may all differ. FP8, for example, defines E4M3 and E5M2 interchange formats, but an accurate training recipe also chooses which tensors use them and where wider accumulation or scaling is required (Micikevicius et al. 2022). Compare peak rates only for the same dense or structured sparsity convention and an instruction path the workload actually uses. Chapter 34 covers the numerical and kernel consequences.

Portability also has layers: source compatibility, numerical correctness, feature availability, and performance portability. A framework backend may run the same model on two accelerator families while custom kernels, collective algorithms, supported formats, and best tensor shapes still differ. Keep model and sharding intent above vendor adapters, isolate custom kernels, and repeat the same correctness and performance matrix on every target.

MFU is an accounting ratio

Model FLOPs utilization (MFU) compares an analytical estimate of useful model work with a selected hardware peak (Chowdhery et al. 2023), where $F_{\mathrm{model}}$ denotes estimated useful model FLOPs per global step, $N_{\mathrm{device}}$ denotes the number of accelerators, $P_{\mathrm{peak}}$ denotes the per-device theoretical peak FLOP/s for the declared precision and dense or sparse path, and $t_{\mathrm{step}}$ denotes measured seconds per step. The ratio is

$$\mathrm{MFU}= \frac{F_{\mathrm{model}}} {N_{\mathrm{device}}P_{\mathrm{peak}}t_{\mathrm{step}}}.$$

Publish the FLOP-count convention, global batch and sequence shape, advertised precision used for the peak, accelerator count, and timing boundary. An MFU computed with a dense peak is not directly comparable with one computed using a structured-sparse peak. Sparse and mixture-of-experts models also need an explicit active-parameter and routed-token convention.

Hardware FLOPs utilization (HFU) uses an estimate of executed arithmetic in the numerator, which can include rematerialization, also called activation recomputation. With consistent accounting, rematerialization can make HFU higher than MFU because the hardware repeats forward work. Neither ratio is a hardware counter, and MFU is not a diagnosis. Low MFU can come from memory traffic, exposed collectives, pipeline bubbles, small kernels, input stalls, thermal throttling, failures, or an inconsistent FLOP estimate. Pair it with tokens per second, step-time distributions, achieved memory and collective bandwidth, kernel timelines, power, clocks, and error telemetry.

Measure from components to the run

Use a ladder of evidence. A full-model benchmark alone tells you that something changed; component measurements locate the changed boundary.

For every benchmark, record hardware SKU and revision; driver and firmware; compiler, framework, and collective-library version; precision and sparsity mode; tensor dimensions; batch and sequence distribution; message size; physical topology and oversubscription; rank placement; algorithm and protocol; warm-up and measured iterations; one-way or bidirectional accounting; p50, p95, and p99; concurrent flows; and power and temperature. A single-node microbenchmark does not establish cluster scaling.

The operating record should preserve enough context to reproduce and compare a result:

accelerator_system:
  hardware: {device: ..., count: ..., memory: ..., power_limit: ...}
  software: {driver: ..., firmware: ..., compiler: ..., framework: ..., collective_library: ...}
  workload: {model_revision: ..., batch: ..., sequence_distribution: ..., precision: ..., flop_convention: ...}
  topology: {scale_up_domain: ..., scale_out_fabric: ..., oversubscription: ..., rank_placement: ...}
  measurements: {window: ..., warmup: ..., repetitions: ..., throughput: ..., mfu: ..., latency_percentiles: ...}
  health: {link_errors: ..., ecc: ..., device_errors: ..., clocks: ..., power: ..., temperature: ...}

Correlate those records with synchronized job, rank, node, accelerator, NIC, switch, rack-power, and cooling-domain identifiers. A collective timeout may be a software ordering bug, a failed rank, a degraded link, or congestion. One utilization counter cannot distinguish them. Reliability and silent failures are treated in Chapter 69; facility power and cooling constraints are treated in Chapter 68.

Regression scenarios

• Small-message latency: reduce message sizes until startup dominates and verify that the selected collective algorithm and p99 remain acceptable.
• Large-message bandwidth: sweep beyond cache and confirm sustained, not peak, bandwidth without assuming linear cluster scaling.
• Oversubscribed uplink: place communicating ranks across the constrained cut and verify admission, routing, and performance alarms.
• Degraded link: lower or remove one path and confirm topology discovery, rerouting behavior, and the resulting tail latency.
• Rank-placement change: permute ranks over the same hardware and detect a performance regression caused by a larger communication cut.
• Collective timeout: stop one rank or violate collective ordering in a test job and verify bounded failure, diagnosis, and cleanup.
• Silent data corruption: inject a detectable fault and verify end-to-end checks, quarantine, and recovery rather than relying only on a device alarm.
• Thermal throttling: run a soak or controlled power cap and verify that clock, power, temperature, throughput, and scheduler evidence stay linked.
• Software change: alter one driver, compiler, kernel, or collective-library version and rerun the same shapes, placements, and statistical comparison.
• Capacity edge: approach the memory budget, including temporary buffers and allocator headroom, and verify deterministic admission or failure. This connects to Chapter 32.

The durable method is simple: name the boundary, define the accounting, measure the real shape, and preserve the context. Peak FLOPs, link rate, and MFU are useful only after those four steps make their meaning explicit.

Further reading