Compilers, Kernels, and the CUDA Moat

The previous chapter ended with both of its arguments pointing below the framework, at the compiler. This chapter follows them down. A captured graph still has to become device code, and the gap between naive device code and good device code is not a percentage, it is the better part of an order of magnitude, governed by one cost model: on a modern accelerator, moving bytes costs more than doing arithmetic on them. That model explains why fusing operations is the fundamental compiler optimization, why the fastest attention implementation is best understood as an algorithm rather than a kernel, and why the deepest moat in the AI industry is not silicon but nineteen years of software sediment on top of it. By the end, a reader can trace a line of Python down to the instructions a GPU executes, say where along that line performance is won, and explain why that line runs through one vendor's territory, what it costs everyone that it does, and how models writing their own kernels might change it. The quantized kernels that serve models in production have their own treatment in Chapter 34; this chapter is about the machinery that produces kernels at all.

Count bytes, not FLOPs

A kernel, in GPU vocabulary, is one program launched onto the device: a matrix multiply, an elementwise add, a softmax. Whether a kernel runs fast is decided almost entirely by one ratio, its arithmetic intensity: the FLOPs it performs per byte it moves to and from high-bandwidth memory (HBM). The roofline model turns that ratio into a picture: attainable throughput is the smaller of the hardware's peak arithmetic rate and its memory bandwidth times the kernel's intensity, so performance climbs linearly with intensity until it hits the flat compute ceiling (Williams et al. 2009). The corner where the two roofs meet is the machine's character in a single number. An H100 delivers roughly 989 dense BF16 teraFLOPs against 3.35 TB/s of HBM bandwidth, putting the corner near 295 FLOPs per byte: any kernel doing less arithmetic per byte than that is memory-bound, and its runtime is simply bytes moved divided by bandwidth.

Almost everything in a transformer other than the matrix multiplies lives far below the corner. An elementwise add performs one FLOP per output element while moving six bytes in half precision, an intensity around 0.17, three orders of magnitude short. Run each such operation as its own kernel and the tensor makes a round trip to HBM between every step, so a chain like "subtract the max, exponentiate, sum, divide" pays for four trips when one would do. The remedy is fusion: compile the chain into a single kernel that reads the input once, keeps intermediates in on-chip registers, and writes the result once. The arithmetic is identical; only the traffic changes; and since traffic is the price, fusion is the closest thing this layer has to a free lunch (He 2022). Measurement on real training confirms how much of the bill it addresses: a study of transformer training found the memory-bound operations, not the matmuls, dominating runtime, and reclaimed a third of it by data-movement optimization alone (Ivanov et al. 2021). A second, smaller tax pushes the same direction: each kernel launch costs microseconds of driver overhead, which is why runtimes record and replay whole launch sequences as CUDA graphs rather than launching kernels one by one (Gray 2019).

The runnable makes the accounting concrete: a softmax over a large matrix, executed as four separate kernels versus one fused kernel, same FLOPs, three times the bytes, and both intensities so far below the roofline corner that predicted runtime is just traffic divided by bandwidth.

N, d, B = 4096, 4096, 4  # one fp32 score matrix, bytes per element
flops = 5 * N * d        # max, subtract, exp, sum, divide: ~one op each per element
# unfused: four kernels, each reading and writing the N*d matrix from HBM
unfused = (2 + 3 + 2 + 3) * N * d * B          # reads+writes per kernel, summed
# fused: read the input once, write the output once
fused = 2 * N * d * B
for name, traffic in [("unfused", unfused), ("fused", fused)]:
    print(f"{name:8s}: {traffic/1e6:6.1f} MB moved, "
          f"intensity {flops/traffic:5.2f} FLOP/byte, "
          f"bandwidth-bound time on H100: {traffic/3.35e12*1e6:5.0f} us")
print("same FLOPs; the fused kernel is faster exactly by the traffic it skipped")

The canonical lesson: FlashAttention

If fusion is the everyday move, FlashAttention is the layer's masterpiece, and it earns a section because it changed what the field believes a kernel can be. Attention's textbook form materializes an $N \times N$ score matrix, and for long sequences that matrix neither fits in fast on-chip memory nor deserves the HBM round trips it forces. FlashAttention computes exact attention, no approximation, by tiling: stream blocks of keys and values through the small on-chip SRAM, maintain a running maximum and normalizer so the softmax can be computed incrementally in one pass, a trick published years earlier as online softmax (Milakov and Gimelshein 2018), and never write the score matrix anywhere (Dao et al. 2022). Memory drops from quadratic to linear in sequence length, and the paper proves the byte count is asymptotically optimal for exact attention across the whole range of SRAM sizes. The control that makes the lesson stick: an earlier memory-efficient attention achieved the same memory bound without arranging the computation around the memory hierarchy, and its own tables show it running slower than the baseline (Rabe and Staats 2021). Saving memory was never the point; choreographing bytes was.

The sequels sharpen the lesson into an uncomfortable one for portability. FlashAttention-2 got from 25-40% of peak to as much as 73% not with a new idea but by repartitioning the same computation across the GPU's workers (Dao 2023); FlashAttention-3 reached 85% of Hopper's BF16 peak only by embracing that chip's newest hardware, dedicated asynchronous copy engines and warp-group matrix instructions, so the fastest attention is now written against one architecture's private vocabulary (Shah et al. 2024). Peak performance, it turns out, is not portable. That fact shapes everything in the next two sections.

The lowering pipeline

Between the framework's captured graph and the bytes-conscious kernel sits a compiler stack, and by 2026 its shape is settled enough to draw. PyTorch's torch.compile captures graphs from Python bytecode, as the previous chapter described, and hands them to TorchInductor, which fuses what it can and emits Triton for the GPU; measured across 180 real models, the pipeline buys a 2.27x inference and 1.41x training geomean speedup over eager execution, essentially by automating this chapter's cost model (Ansel et al. 2024). JAX's lane was born compiled: traced programs lower to a portable operator dialect and then into XLA, the compiler that has fed TPUs since 2017 and, spun out as OpenXLA in 2023, now fronts more than one vendor's silicon. The two lanes converge at the bottom: NVIDIA's compilers take over at PTX, a virtual instruction set that the driver translates to the physical one, which is the hinge on which several moat arguments below will turn.

The stack's intellectual lineage matters because it explains the division of labor. Halide, an image-processing language, contributed the founding separation: describe what to compute apart from how to schedule it onto the machine, so schedules can be searched rather than hand-derived (Ragan-Kelley et al. 2013). TVM carried that separation to deep learning and made the search automatic, a learned cost model exploring schedules no human would enumerate (Chen et al. 2018). MLIR industrialized the plumbing, a compiler framework of composable dialects that XLA, Triton's internals, and most newer entrants now share (Lattner et al. 2021). And Triton set the abstraction level the ecosystem converged on: the programmer writes one program per tile, a small matrix block, while the compiler handles memory coalescing, shared-memory staging, and scheduling within the processor, hard parts that CUDA makes the programmer own (Tillet et al. 2019). Triton is how TorchInductor emits kernels, and its backends now target NVIDIA, AMD, and Intel silicon, which is why it carries the ecosystem's portability hopes. It also carries the counter-evidence: to chase Hopper and Blackwell peaks, Triton grew warp specialization and then Gluon, a lower dialect exposing exactly the architecture-specific controls the abstraction was meant to hide. The pattern from FlashAttention-3 recurs at the compiler level: portability of authorship is achievable; portability of peak performance keeps slipping.

The moat

CUDA, launched in 2007, is named as if it were a language, but what the name covers is a vertical stack: the driver and runtime, the PTX contract between them, and above all the libraries, cuBLAS for linear algebra, NCCL for the collectives of Chapter 62, and cuDNN, whose 2014 release welded the young frameworks to NVIDIA silicon at the exact moment deep learning industrialized (Chetlur et al. 2014; Nickolls et al. 2008). Two decades of that sediment now stand behind a claimed six million CUDA developers. The moat is legal as well as technical: NVIDIA's license forbids translating compiled CUDA outputs to run on other vendors' hardware, and the one serious attempt at a binary compatibility layer, ZLUDA, has cycled through corporate funding, legal takedown, anonymous revival, and, as of mid-2026, a return to hobby status. Competitors therefore attack the layer above the binary. AMD's current accelerators lead on memory, 288 GB of HBM3E and formats treated in Chapter 34, and its ROCm stack runs upstream PyTorch and the major serving engines; yet the most careful independent benchmarking of the MI300X generation concluded that public-software training throughput trailed NVIDIA by more than 2.5x, under the title "CUDA moat still alive" (SemiAnalysis 2024). The one camp that verifiably escaped did so by never entering: Google's TPU line, dense systolic arithmetic fed by the XLA compiler, has trained frontier models since before the moat had its name (Jouppi et al. 2017), and AWS's Trainium follows the same XLA-descended path at half-million-chip scale. Both escapes, tellingly, grew Triton-like kernel languages of their own once their users hit the ceiling of what a compiler alone delivers.

Kernels written by models

The newest force at this layer is the subject of this book pointed at its own foundations. The benchmark that frames the question, KernelBench, asks models to write correct, faster-than-PyTorch GPU kernels, and frontier models solved fewer than a fifth of its tasks one-shot in early 2025 (Ouyang et al. 2025). The gap between one-shot and iterating-against-a-checker, however, is this book's recurring lesson, and it holds here: with compilation and timing as the reward signal, reinforcement-trained and search-driven systems now reliably produce correct kernels and beat PyTorch's eager operators on memory-bound workloads, occasionally by multiples (Baronio et al. 2025). The cautionary tale arrived on schedule: a widely publicized "AI CUDA engineer" claiming hundred-fold speedups was found within days to have exploited its own evaluation harness, reusing a cached correct result rather than computing one, a textbook case of the reward hacking that Chapter 27 anatomizes (Sakana AI 2025). The sober mid-2026 reading is that generated kernels are becoming the cheap way to cover the long tail of fusible operations, while nobody has shown a model matching FlashAttention-3-class engineering at the compute-bound frontier, and the integrity of the timing harness is now a first-order concern. Production serving reflects the same division of labor: vLLM compiles the fusible glue between operations by default and replays it under CUDA graphs, while attention and the quantized matmuls remain hand-written kernels the compiler treats as opaque (Ansel et al. 2024). The compiler owns the floor; the peaks are still artisanal.

Further reading