Part IX · Chapter 66

The Compute Frontier: Bandwidth, Not FLOPs

AuthorChangkun Ou

Reading time~16 min

Chapter 62 framed a cluster as three bandwidth tiers and model FLOPs utilization as the number for diagnosing a run. That chapter described how the tiers work. This one asks where each tier breaks. The thesis is that the frontier's scarce resource is bytes rather than arithmetic: accelerators have become chiplet packages, the NVLink boundary has moved from chassis to rack, accelerator entrants are differentiated by their bandwidth bets, and the viable escape routes are the ones already shipping rather than promised.

Figure 66.1. Schematic of the compute frontier's imbalance. Peak FLOPs can rise faster than memory bandwidth, making bandwidth and data movement the binding constraint. Idealized relative growth, not measured data.

The book's closing part follows the same constraint arrow as the rest of the book, the one that points from the bottom of the stack to the top, but now the question at each layer changes: not how the layer works, but where it is breaking and where it is heading. The path starts at the silicon, because the bottleneck that reorganized everything above it in the last two years started here.

The headline number is the wrong constraint

A frontier accelerator's datasheet leads with a peak arithmetic rate, and the rate climbs fast across generations, faster still once the precision drops from FP16 to FP8 to FP4, the 16-, 8-, and 4-bit number formats a chip computes in, each step down trading numeric accuracy for throughput. Read only that line and the machine looks like it is getting better at the thing training and inference need most. The line is true and almost beside the point. The work a transformer does is not bound by how fast the multiply-accumulate units can run; it is bound by how fast operands reach them. Attention decode (generating one token at a time, which rereads the whole prior context from memory at every step), mixture-of-experts routing (sending each token to a few of many expert sub-networks chosen per token), and any kernel that touches more memory than it reuses spend their time waiting on high-bandwidth memory, not on the matrix units. The roofline of Chapter 62 already named this: a kernel's arithmetic intensity is the number of FLOPs it performs per byte it fetches from memory, and below the intensity the hardware needs the achievable rate is set by HBM bandwidth, with the peak FLOPs ceiling unreachable.

The frontier problem is that the two numbers on the roofline diverge, and they diverge in the same direction every generation. Across recent accelerator families, low-precision matrix throughput has climbed much faster than HBM bandwidth per chip. The accurate way to state the trend is as a direction rather than a single headline multiple, because the multiple depends on which precision and which two generations you pick: from the V100 era to Blackwell, peak compute grew by roughly an order of magnitude more than memory bandwidth did, and the recent FP4 and FP8 jumps widen the gap further because they multiply arithmetic throughput without moving a single byte faster. The arithmetic intensity a kernel must reach to escape the memory-bound region therefore rises every generation. This is the memory wall, and it is not a temporary manufacturing lag. It is the structural fact that organizes the whole chapter: every tier of the machine, the chip, the package, the rack, and the power behind them, is hitting a ceiling that bandwidth, not arithmetic, sets.

Figure 66.2. The frontier as a stack of ceilings, each binding at a different tier. The spec sheet's FLOPs sits at the bottom and is the least scarce; bytes, package area, fabric reach, and megawatts bind above it. Each later chapter of this part follows one of these ceilings.

The chip: a die too small for its own memory

The innermost ceiling is geometric. A photolithography scanner can expose at most one reticle field per shot, capped near 858 mm² (Shilov 2026). A frontier compute die already presses against that limit, so the only way to add transistors is to stop building one die. Modern accelerators are several compute chiplets and a row of HBM stacks tiled side by side on a silicon interposer, the chip-on-wafer-on-substrate package: an advanced package that places compute chiplets and HBM on one silicon interposer. The interposer, not the logic die, is now the object whose area sets how much compute and how many memory stacks one accelerator can carry. TSMC's interposer roadmap is the supply-side clock on the whole frontier: roughly 5.5 reticles of area with five to a dozen HBM stacks today, a nine-reticle "Super Carrier" near 7,722 mm² in qualification for 2027, and a path past fourteen reticles with HBM5-class memory toward the end of the decade (Shilov 2026). Chapter 67 follows whether that interposer can be fabricated and bought; here it is enough to see that the reticle limit pushed packaging up into the architecture, making interposer area a first-class design parameter above the microarchitecture.

What gets tiled onto the interposer is mostly memory, and the memory is the point. NVIDIA's Rubin generation reaches roughly 288 GB of HBM4 at about 22 TB/s per accelerator. Google's Ironwood, the TPU v7, carries 192 GB of HBM3e at 7.37 TB/s and is the first TPU to support FP8 natively rather than through emulation (Google 2025). The HBM stacks themselves are a moving frontier: HBM4 doubles the interface to a 2,048-bit bus, the first sixteen-high stacks were demonstrated in early 2026, and the base die at the bottom of the stack is migrating onto a logic process so that memory now competes for the same leading-edge wafers as compute, a coupling Chapter 67 develops. The numbers move every quarter, and any specific figure in this chapter is a date stamp rather than a constant. The durable observation is the ratio: capacity and bandwidth grow, but they grow more slowly than the arithmetic that consumes them, so the fraction of real workloads that are memory-bound rises over time.

Constraint arrow

HBM bandwidth growing slower than low-precision matrix throughput sets a rising arithmetic-intensity floor. That floor dictates choices two layers up: an ever-larger share of inference kernels land in the memory-bound region, which is why the serving layer leans on FP8 and FP4 weights and on aggressive key-value cache management, where the cache holds each token's saved attention keys and values so later steps reuse them instead of recomputing, rather than on more FLOPs (cross-ref Chapter 32 and Chapter 34). Adding arithmetic to a bandwidth-bound kernel buys nothing.

The rack: the scale-up boundary that moved

Chapter 62 drew a hard line at the node: tensor parallelism, the most communication-hungry axis, had to stay inside the NVLink domain, and that domain was the eight or so accelerators in one chassis. That line has moved, and watching it move is the clearest way to see the frontier as evolution rather than a list of products. The NVLink fabric began as a chip-to-chip link inside a board, grew an NVSwitch crossbar so every accelerator in a chassis could reach every other at full bandwidth, and has now been deliberately stretched across an entire liquid-cooled rack. NVIDIA's GB200 and GB300 NVL72 wire seventy-two GPUs into a single NVLink-coherent domain spanning the rack, and the Rubin NVL72 keeps the form factor while roughly doubling the fabric bandwidth again. The rule from Chapter 62 still holds, but its boundary is now the rack rather than the chassis: tensor parallelism stays inside the NVLink domain, and the domain got an order of magnitude larger.

Figure 66.3. The scale-up domain that Chapter 62 pinned to a node has expanded to a rack. The most bandwidth-hungry parallelism axis can now span 72 accelerators inside one coherent fabric, where before it spanned roughly 8; beyond the domain edge, only sparse communication can afford the slower fabric.

Why expand the domain at all? Because the memory wall and the parallelism math interact. A larger coherent domain lets a model's tensor-parallel and expert-parallel groups span more silicon at full fabric bandwidth, so the per-device memory pressure of a trillion-parameter model is spread without spilling the most bandwidth-demanding collective onto the slow inter-rack network. The rack became the unit of frontier compute because that is the largest domain where the fabric can still hide a collective.

The expansion is not free, and copper is running out of reach. Past a rack, the per-interface power of pluggable optical transceivers, on the order of tens of watts each, becomes a budget the rack cannot spend. Co-packaged optics move the fiber onto the switch package and cut the per-interface power roughly from 30 W to 9 W, which is the case NVIDIA makes for Quantum-X and Spectrum-X co-packaged switches shipping through 2026 (Seyedi and NVIDIA 2025). Beyond a fixed topology, optical circuit switching lets the fabric itself be reconfigured per training phase; recent work on parallelism-driven optical rails reports large network-power and cost reductions at a few percent training overhead (Ding et al. 2026). The through-line is that the fabric's energy budget, not only its bandwidth, now bounds how large a coherent domain can be built and cooled, which is the thread Chapter 68 picks up.

One more move complicates the old picture. NVIDIA has begun opening the NVLink domain to third-party silicon, so a rack can mix a custom accelerator or CPU into an otherwise NVIDIA fabric. The scale-up domain is therefore no longer strictly the property of one vendor's chips, which matters for the debate over whether the future fabric is proprietary or open.

The accelerator field: who is betting on bandwidth

For most of the deep-learning era the merchant GPU was the only serious frontier accelerator, and its software ecosystem was the moat. The bandwidth wall has opened room for a wider accelerator field, and reading the entrants by what they optimize is more instructive than ranking them by peak FLOPs.

The custom-application-specific integrated circuit (ASIC) bet is vertical integration: co-design the chip with the network and skip the merchant margin. Google's Ironwood scales to a 9,216-chip superpod wired as a torus with optical circuit switches rather than a switched fat-tree, and targets the inference-dominated workload mix directly (Google 2025). Amazon's Trainium makes the same wager for its own fleet. The advantage is co-design: the chip, the interconnect topology, and the compiler are tuned together for one company's models.

The merchant counter is AMD's memory-density play. Its rack-scale Helios system fills seventy-two accelerators, each carrying about 432 GB of HBM4, into a domain that totals on the order of 31 TB of HBM at roughly 1.4 PB/s of aggregate bandwidth, which exceeds the comparable NVIDIA rack's 21 TB or so of memory capacity while trailing it on aggregate bandwidth and on peak FP4 throughput. That is the chapter's thesis stated as a product decision: if the wall is bytes, build the rack with more bytes and concede some arithmetic. Under export controls, Huawei's CloudMatrix 384 makes the same move with watts instead of process node: it knits 384 domestic NPUs across sixteen racks into one optically-meshed all-to-all scale-up domain that reaches frontier aggregate compute and memory by brute force, at several times the power of the Western rack it answers. The announced follow-on, the Atlas 950 SuperPoD, scales the same watts-for-nodes bet by an order of magnitude, 8,192 Ascend chips in one optically interconnected scale-up domain with self-developed memory stacks, slated for late 2026 (Huawei 2025), which makes CloudMatrix a trajectory rather than a one-off. The pair is a reminder that the scale-up-domain idea is general, that the memory wall can be paid down with energy when the best silicon is unavailable, and that the compute frontier has a second front defined by who is allowed to buy what, which Chapter 67 takes up.

At the contrarian edge sits wafer-scale. Cerebras keeps an entire model's weights in on-die SRAM across a single wafer of 900,000 cores, with no HBM to stream from, and reports multiples of GPU per-user token rate on latency-bound decode (Cerebras Systems 2024). It is the purest bet against the memory wall: if you never leave the die, there is no HBM bandwidth to be bound by. Its ceiling is capacity, since on-wafer SRAM cannot hold a trillion-parameter model, so it wins latency-bound serving rather than the largest training runs. In-memory and analog accelerators push the same idea further and sit further from production.

Figure 66.4. The accelerator field organized by what each entrant optimizes against the bandwidth wall, not by peak FLOPs. The merchant GPU and custom ASIC fight over the same workload from opposite integration strategies; AMD and Huawei answer the wall with bytes and watts; wafer-scale answers it by never leaving the die.

Escape routes, in order of maturity

The chapter's payoff is to sort the ways out of the bandwidth wall by how real they are today, because the field's discourse mixes shipping hardware with research slides, and a reader deserves to know which is which.

Shipping now is rack-scale copper: the NVL72-class domain is in production and is the reason tensor parallelism can span seventy-two devices. Co-packaged optics follows through 2026, the next step once copper runs out of reach past the rack (Seyedi and NVIDIA 2025).

A numeric escape route is already in training pipelines. Four-bit pretraining formats such as NVFP4 relax the arithmetic-intensity floor by halving operand bytes while preserving accuracy: a multi-billion-parameter model trained over many trillions of tokens in NVFP4 has matched an FP8 baseline, and the four-bit matrix multiply runs at a multiple of the FP8 rate on recent hardware. This is the cleanest way to make a memory-bound kernel less memory-bound without new silicon, because it attacks the bytes directly; the format mechanics belong to Chapter 34, but the reason the frontier cares is bandwidth, not storage.

A structural route is to split the workload across chips specialized for each phase. Prefill, the processing of a long prompt, is compute-bound and tolerates cheaper high-capacity memory; decode, the token-by-token generation, is bandwidth-bound and wants scarce HBM. A prefill-optimized accelerator built around abundant GDDR memory can therefore pair with HBM parts that handle decode, putting expensive bandwidth only where the workload is actually bound by it. NVIDIA announced exactly such a prefill-specialized part in 2025, then pulled it from the roadmap at its March 2026 conference, with reports that the idea may return in a later generation (Tom's Hardware 2026); a part withdrawn within six months of its announcement is itself a frontier-uncertainty signal worth recording rather than smoothing over, because it shows the field still disagreeing about whether the split pays once the cost of moving the key-value cache between the two chips is counted, a systems question Chapter 69 returns to.

Further out sit reconfigurable optical fabrics (Ding et al. 2026) and wafer-scale integration (Cerebras Systems 2024), real research with real demonstrations but not yet the default.

Figure 66.5. Escape routes from the bandwidth wall, ordered by maturity. The left column is in production; the right column is demonstrated but not default. The middle two attack the wall through numerics and through phase specialization rather than through new fabric.

What's contested

Four debates are genuinely live, and the camps disagree on which non-FLOPs wall binds first rather than on whether FLOPs stopped being the limit.

One coherent fabric, an open standard, or cheap Ethernet? NVIDIA bets ever-larger proprietary NVLink domains are worth their cost. The UALink consortium wants an open switched standard to commoditize the scale-up tier. A pragmatist camp reuses Ethernet switch silicon for scale-up. Through 2026 the pragmatists shipped first, because rack vendors fell back to Ethernet-substrate fabrics when native open-standard switch silicon was not yet available. NVIDIA has meanwhile partially opened its own fabric to third-party silicon, so the proprietary-versus-open line is itself blurring.
Is co-packaged optics mandatory or a reliability risk? Proponents say copper is out of runway past a rack and optics is the only way to scale the domain. Skeptics point to unproven laser reliability and repairability at fleet scale and defend copper inside the rack. The dividing line is distance: optics for inter-rack, copper still defended intra-rack.
Do custom ASICs displace the merchant GPU? The vertical-integration camp says co-designed silicon wins perf-per-watt on the inference-heavy mix. The incumbent camp says the software ecosystem and flexibility keep the GPU the default, and ASICs win only captive first-party workloads.
Is wafer-scale the answer or a niche? It clearly wins latency-bound single-user decode and clearly cannot hold the largest models in SRAM. The unresolved question is whether model sizes plateau enough for on-die capacity to catch up.

The scarce resources above FLOPs

At this layer, capability, efficiency, and trust all point above raw FLOPs. Capability is no longer set by how many FLOPs a chip can retire but by how many bytes it can move and how much coherent silicon one fabric can span, which is why the rack replaced the chip as the unit of frontier compute. Efficiency moved into the numerics and the network: a four-bit operand and a nine-watt optical interface buy more frontier than a faster matrix unit, because both attack bytes rather than arithmetic. Trust, at the silicon layer, is supply and concentration: a frontier run now depends on whether a handful of buyers can secure interposer area and HBM stacks at all, and on which side of an export-control line the buyer sits, which is the allocation gate Chapter 67 takes up next. The spec-sheet FLOPs was never the scarce resource. The scarce resources are bytes, package area, fabric reach, and the megawatts behind them, and the rest of this part follows that scarcity up the stack.