The Compute Frontier: Bandwidth, Not FLOPs

Chapter 62 introduced arithmetic intensity and the bandwidth tiers inside an accelerator cluster. This chapter turns those ideas into a method for locating a system's limit. Its working thesis is that the frontier's scarce resource is bytes rather than arithmetic. But that claim is not universal: it is meaningful only after naming the workload, the memory boundary, and the measured traffic that crosses it. A matrix multiplication with enough reuse can exhaust the arithmetic units; token-by-token inference can instead wait on weights, the KV cache, or a collective. “Bandwidth-bound” is therefore a diagnosis, not a property of AI hardware in general.

That distinction matters because accelerators have become chiplet packages and the NVLink boundary has moved from a chassis to a multi-node domain in one prominent system. Neither change abolishes data movement. Each moves a boundary, changes its cost, and creates a new place to measure.

The frontier is a vector of bounds

The original Roofline model compares a kernel's operational intensity with peak compute and memory bandwidth (Williams et al. 2009). A frontier system needs more than one roofline because bytes can cross several boundaries. One optimistic lower bound for a step is

$$T_{\text{step}} \ge \max\left( \frac{F}{P_{\text{eff}}}, \frac{Q_{\text{HBM}}}{B_{\text{HBM}}}, \frac{Q_{\text{up}}}{B_{\text{up}}}, \frac{Q_{\text{out}}}{B_{\text{out}}} \right).$$

where:

• $F$ is the useful arithmetic work and $P_{\text{eff}}$ is the effective, workload-specific arithmetic rate;
• $Q_{\text{HBM}}$ is traffic volume between the processor and HBM, while $B_{\text{HBM}}$ is sustainable bandwidth at that boundary;
• $Q_{\text{up}}$ and $B_{\text{up}}$ describe traffic and sustainable bandwidth inside the scale-up domain; and
• $Q_{\text{out}}$ and $B_{\text{out}}$ describe traffic and sustainable bandwidth across the scale-out fabric.

The largest term identifies the first optimization target. The equation is not a latency predictor. It assumes perfect overlap among independent transfers and computation. Dependencies, queuing, synchronization, protocol overhead, and contention can make observed time larger; phases that cannot overlap may need to be added rather than maximized. Nor should a datasheet peak substitute for $P_{\text{eff}}$ or any sustainable bandwidth.

# Illustrative units: work in operations, traffic in bytes, rates per second.
terms = {
    "arithmetic": 2.0e15 / 1.0e15,
    "HBM traffic": 8.0e12 / 3.0e12,
    "scale-up traffic": 1.0e12 / 9.0e11,
    "scale-out traffic": 5.0e11 / 2.0e11,
}

for name, seconds in terms.items():
    print(f"{name:18s} {seconds:.2f} s")

binding = max(terms, key=terms.get)
print(f"Binding term: {binding}")

This model also explains why lowering precision is not automatically a speedup. It helps when it reduces the binding traffic or increases the binding compute rate. Extra conversion, padding, metadata, or an unchanged KV-cache path can leave the largest term where it was.

The package moves the memory boundary

A reticle field limits the area that lithography exposes in one image. A large monolithic die near that limit is expensive to grow and vulnerable to defects. Splitting logic into chiplets can improve yield and reuse, but it adds die-to-die links, packaging work, and new failure surfaces. The interposer provides dense wiring among compute dies and HBM stacks; the package substrate connects that assembly to power and board-level signals. These are distinct layers, not synonyms.

This is the chip-on-wafer-on-substrate package: an advanced package that places compute chiplets and HBM on one silicon interposer. TSMC documents CoWoS as a family of integration technologies and describes interposers larger than a single reticle field (TSMC 2026). That is evidence for a packaging direction, not permission to extrapolate a vendor roadmap into exact future system capacity.

HBM adds several separate quantities to the design:

• Memory capacity determines how much parameter, activation, optimizer, and KV-cache state can reside near the processors.
• Sustainable bandwidth is the rate a real access pattern reaches, after accounting for controllers, locality, concurrency, and protocol effects.
• Traffic volume is created by the algorithm, tensor layout, precision, recomputation policy, and cache behavior.
• Energy per bit makes unnecessary movement a power and cooling cost even when latency is hidden.

HBM4 documentation provides a useful, product-scoped example: SK hynix and Micron describe a 2,048-bit interface, twice the previous generation's width (SK hynix 2025; Micron Technology 2026). HBM4 also introduces a more capable base die beneath the stacked memory dies. The interface width alone does not reveal delivered bandwidth, capacity, yield, power, or application throughput. Those depend on the complete package and workload.

The familiar memory wall can now be stated precisely. If effective arithmetic rate grows faster than sustainable HBM bandwidth, the arithmetic intensity required to become compute-bound rises. That rising arithmetic-intensity floor affects only work below it; kernels with sufficient reuse can remain compute-bound.

The scale-up boundary can span multiple nodes

A scale-up domain is the set of accelerators connected by the system's lowest-latency, highest-bandwidth device fabric. A scale-out fabric connects those domains. The distinction is operational: it changes collective latency, bisection bandwidth, routing, congestion, and the cost of a failure. It does not imply one universal topology.

NVIDIA's DGX GB200 NVL72 is a current, product-scoped example. Its official documentation describes a 72-GPU NVLink domain in one rack, with 18 compute trays, nine NVLink switch trays, and a passive copper cable backplane (NVIDIA 2026). The accompanying tuning guide describes the system as an expansion beyond earlier eight-GPU scale-up systems (NVIDIA 2026). The protocol boundary is the NVLink domain, not the cabinet: other reference layouts can distribute that domain differently. And the dense wiring does not make 72 GPUs one device. Software still manages separate memory, processes, address spaces, logical topology, synchronization, and a multi-node failure domain.

Placement must follow traffic, not acronyms. Tensor parallel groups often run frequent all-reduces; expert parallel groups exchange routed token data; pipeline parallel stages send activations; data parallel replicas synchronize gradients or parameters. Yet message size, frequency, overlap, collective algorithm, model shape, and contention determine which group benefits most from the fast domain. The sound procedure is to measure each edge, map the heaviest latency-sensitive communication onto the logical topology, and then verify end-to-end time.

An open specification is also not the same as an available implementation. UALink 1.0 specifies a 200 Gb/s-per-lane scale-up interface and a switched domain of up to 1,024 accelerators (UALink Consortium 2025). It gives builders a common target. It does not establish that every switch, cable, software stack, or large deployment described by the specification is available.

At rack-to-rack distance, scale-out networks already use optics. Co-packaged optics changes where electrical-to-optical conversion happens: placing optical engines near switch silicon shortens the lossy electrical channel and can reduce faceplate-density pressure. It also couples optics to a hot, expensive package. Reliability, laser placement, test, repair, and serviceability then become architectural questions (Minkenberg et al. 2021). This does not extend the NVLink domain.

Optical circuit switching is a different idea: it reconfigures physical paths. TPU v4 is a documented deployed example of optical circuit switches changing a large system's topology (Jouppi et al. 2023). A proposal such as Opus instead reconfigures optical rails at parallelism-phase boundaries and remains a research prototype (Ding et al. 2026). Neither technology makes communication free.

Architecture choices move terms rather than erase them

The useful comparison among accelerator designs is which term in the lower bound above they change and which cost they expose elsewhere.

A conventional HBM accelerator keeps a large software surface and targets many tensor shapes. Its package offers high external-memory capacity and bandwidth, but weights and activations still cross the HBM boundary. A custom accelerator can co-design dataflow, compiler, memory, and network for a narrower workload; the gain depends on whether that workload resembles the design target.

Wafer-scale integration chooses a different locality point. Cerebras' architecture distributes SRAM and compute across a wafer-scale device and uses dedicated on- and off-wafer interconnects (Lie 2023). That can increase locality and on-device bandwidth, but SRAM capacity, defect management, compilation, system I/O, and multi-system scaling remain constraints. “No HBM” removes one named boundary; it does not remove state or movement.

Technology claims should therefore carry an evidence status. As of 7 August 2026, this chapter uses these labels:

For example, the NVL72 rack above is an available system documented by its operator guide. Opus is a research prototype (Ding et al. 2026). UALink 1.0 is an open specification. A vendor roadmap counts only as an announced product until dated production or deployment evidence supports a different label. These labels can change; the cited evidence should change with them.

Prefill and decode are phases, not fixed hardware classes

Prefill processes a prompt in parallel across its sequence length. Decode usually advances one or a few positions at a time while reading model state and the KV cache. This often gives prefill more reuse and decode lower arithmetic intensity, but the result is conditional. Batch size, sequence length, model architecture, quantization, tensor shape, kernel fusion, cache layout, and parallel placement can move either phase between bounds.

Splitting the phases across different device pools can specialize capacity, bandwidth, and scheduling. It also creates a phase boundary: KV-cache state must be transferred or made remotely accessible, requests must be routed, and the two pools must be balanced under a changing workload. A design is better only if saved compute or memory time exceeds transfer, queueing, and under-utilization costs. Chapter 32 develops those serving tradeoffs; Chapter 69 follows their operational failure modes.

Escape routes change a named term

There is no single maturity ladder for escaping a bandwidth limit. The mechanisms are composable, and each trades one resource for another.

Chapter 34 explains when FP8 and FP4 reduce bytes without unacceptable numerical loss. Chapter 68 follows the energy and cooling cost when the chosen escape route adds packages, links, or conversion.

How to compare frontier systems

A useful hardware comparison starts with an operating contract, not two peak numbers. Record at least:

1. the model and operation being measured;
2. tensor shape, batch size, and sequence length;
3. precision for inputs, weights, accumulation, communication, and stored state;
4. software version, compiler flags, kernels, and communication libraries;
5. device count, placement, physical and logical topology, and competing traffic;
6. traffic volume and sustained bandwidth at each relevant boundary;
7. latency and throughput distributions after warm-up, not only a best run;
8. the power boundary: chip, board, rack, cooling overhead, or facility;
9. availability, failure behavior, repair assumptions, and recovery time; and
10. cost scope, currency, utilization, and time horizon.

Measure in three passes. First, microbenchmark each boundary with the same message sizes and access patterns the workload uses. Second, profile the important kernels and collectives to estimate the terms in the lower bound above. Third, run the end-to-end workload and explain the gap between the lower bound and observed time. A claimed improvement is credible when the named binding term falls, the end-to-end result follows, and quality and reliability remain inside the stated acceptance limits.

The compute frontier is therefore not a race between one FLOPs number and one bandwidth number. It is a set of boundaries whose traffic changes with the program. The durable skill is to locate the active boundary, distinguish sustainable rates from peaks, and follow the cost of moving that boundary into the next layer.

Further reading