Part V: Inference and Serving

"large online services need to create a predictably responsive whole out of less predictable parts."

Jeffrey Dean and Luiz Andre Barroso, "The Tail at Scale"

Training produces weights. A service has to turn those weights into tokens on a clock, for many users, at a price someone can pay. Inference is not a smaller version of training. It has a different shape: prefill and decode, compute and memory, latency and throughput, one request and the batch it must share. Prefill reads the whole prompt in one pass; decode then emits one token per step.

Chapter 31 names the core tension: a generated token is cheap in FLOPs and expensive in memory movement, and the key-value cache grows exactly where the service wishes it would not. Once that is the problem, the cache becomes a resource to be scheduled, and Chapter 32 treats it as one, with paging, prefix sharing, batching, and phase splitting. The next two sections push on the expensive step itself: Chapter 33 looks for ways to get more than one token out of it, while Chapter 34 shrinks the bytes and changes the kernels so the hardware spends less time moving matrices around. Last, Chapter 35 and Chapter 36 show two ways the neat text-serving problem becomes less neat: the output may be constrained by a schema, the context may be long enough to need pruning, or the input may include visual tokens the tokenizer never made, image patches that enter the model as tokens without ever passing through the text tokenizer.

Read this part as a study of the word afford. A model that is impressive but too slow, too memory-hungry, or too hard to batch is not yet infrastructure. Serving is the layer that turns capability into something repeatable, and it is also the layer that pushes back into model design.