Summary

Serving begins after training has already spent the largest visible budget, but it introduces a different shape of cost. Prefill and decode stress the machine in different ways. KV caches grow with context and concurrency, batching buys throughput at the cost of latency, and quantization and kernels change how many bytes have to move. Structured output, long context, and multimodal input then make the neat text-serving problem less neat.

The binding constraint is often not raw FLOPs. It is memory movement, tail latency, cache residency, batch shape, or the cost of supporting a special request path without breaking the common one. Serving is where capability becomes a product constraint, and the product constraint pushes back into model design.

The part leaves a practical standard: a model is not infrastructure merely because it is impressive. It must be repeatable at a latency and price someone can operate. The open question is which model and runtime designs will make long-context, multimodal, and structured generation affordable without turning the serving layer into a maze of exceptions. Part VI begins when a served model is asked to do work beyond completion, where tools, memory, retrieval, and recovery loops become part of the system rather than decorations around it.