The Serving Problem

A trained model is a file; a served model is a system measured in seconds and dollars. Serving begins with a split inside one forward pass: prefill reads the whole prompt and is compute-bound, while decode emits one token at a time and is usually bandwidth-bound. That split turns latency, throughput, and memory into the same scheduling problem. Goodput, not raw throughput or raw latency, is the objective, and the key-value cache is the resource that sets the ceiling.

The chapter follows the resource that makes those claims concrete. The cache is fixed by architecture, paid for in accelerator memory, and stretched by every in-flight request. Once it becomes the constraint, the history of serving engines is easier to read. Continuous batching reclaims the slots that finished requests held idle, and PagedAttention stops reserving cache a request never touches. Prefix caching skips recomputing shared prompts, while chunked or disaggregated prefill keeps one phase from blocking the other.

A file that has to keep time

Pre-training spends compute once to make a weights file. Serving spends compute forever, every time a user sends a request, and it spends it under a clock the user can feel. The problem is no longer minimizing a loss. It is running a fixed set of weights against a stream of requests so that each request returns fast enough and the cluster stays full enough to pay for itself.

Two latencies define "fast enough," and they are not the same number. Time to first token (TTFT) is how long a user waits before the first piece of the answer appears. Time per output token (TPOT), sometimes called inter-token latency, is how long each subsequent token takes once the stream has started. A chat interface needs a short TTFT so the response feels prompt; a long generation needs a short TPOT so the text does not stutter. These two clocks are not arbitrary product targets. They fall out of the way a single forward pass divides into two phases with different hardware bottlenecks.

Two Phases, Two Bottlenecks

When a request arrives, the model first reads the entire prompt. This is the prefill phase: all prompt tokens are processed together in one pass, every position attending to every earlier position, which is a sequence of large matrix-matrix multiplications. Prefill is compute-bound. It saturates the accelerator's arithmetic units, and its cost grows with prompt length. TTFT is essentially the time prefill takes.

Then the model emits the answer one token at a time. This is the decode phase: each step feeds the single most recent token through the network to produce the next one, a sequence of thin matrix-vector multiplications. Decode is memory-bandwidth-bound, not compute-bound. The arithmetic per step is tiny, but every step must stream the full weight matrices and the growing cache of past keys and values out of memory, and that streaming, not the math, dominates the time. This is the roofline split: the roofline model bounds a kernel by either compute or memory bandwidth, and here prefill lives against the compute ceiling while decode lives against the bandwidth ceiling, the organizing fact of inference performance (Yuan et al. 2024; Pope et al. 2022). TPOT is essentially the per-step decode latency, and an idle decode step leaves most of the accelerator's compute unused.

Figure 31.1 traces one request through both phases and lines up, for each, the shape of its arithmetic, the resource it saturates, and the latency metric it sets.

The split matters because the two phases put pressure on the same hardware in different ways. Prefill needs enough work to keep the arithmetic units busy. Decode needs enough requests in the batch for one expensive weight-streaming pass to serve many users at once. Batching is the lever that turns a bandwidth-bound decode into a compute-efficient one, because the weights are read once and amortized across every request in the batch (Pope et al. 2022). So throughput, tokens per second across all users, rises with batch size, while the latency any single user sees can degrade as the batch fills. The serving problem is this tension made concrete: win throughput from batching without spending a user's latency budget.

Why One Number Misleads

It is tempting to chase one headline metric, and both obvious candidates lie. Throughput counts every token the cluster produces. Latency counts how long one user waits. Neither is the business objective. A server can post enormous throughput by batching so aggressively that every individual request misses its deadline, and it can post tiny latency by running one request at a time on idle hardware. The objective that survives contact with reality is goodput: the rate of requests completed within their latency targets. Tokens produced after a request has already blown its TTFT or TPOT budget are not useful work. Goodput is throughput counted only over requests that met their service-level objectives (SLOs).

Goodput reframes every later decision. The scheduler's job is to pack as many requests as possible onto the accelerators without pushing any of them past their SLO. That is a packing problem with a hard constraint, and the constraint is set by memory because decode is forced to keep state around.

The cache is the constraint

Attention at step $t$ needs the keys and values of all earlier positions, so rather than recompute them each step, the server caches them. This is the KV cache. Its size is fixed by the architecture, not by the serving layer:

$$\text{KV bytes} = 2 \cdot L \cdot n_{\text{kv}} \cdot d_{\text{head}} \cdot b_{\text{dtype}} \cdot T$$

where $L$ is the number of layers, $n_{\text{kv}}$ the number of key-value heads, $d_{\text{head}}$ the head dimension, $b_{\text{dtype}}$ the bytes per element, and $T$ the number of tokens cached. The factor of two is for keys and values, the two vectors every attended-to token contributes: the key it is matched against and the value summed into the output. The cache grows linearly with sequence length and linearly with the number of concurrent requests, and it lives in the same scarce accelerator memory that holds the weights. A reader can now see the squeeze: the weights take a fixed slice of memory, and whatever is left over is divided among the KV caches of the requests currently in flight. The number of requests you can batch, and therefore your throughput and goodput ceiling, is set by how much KV cache memory remains and how efficiently you use it. Figure 31.4 shows the partition, and Figure 31.3 makes the linear growth and its consequence concrete: each request's cache rises in a straight line with context length, and the leftover pool fixes how many such requests fit at once.

This is why the architecture chapter's attention variants matter to a serving engineer. Multi-query and grouped-query attention shrink $n_{\text{kv}}$, and multi-head latent attention compresses the cached state, each cutting the KV bytes per token directly. They were designed, in part, to relax exactly this serving constraint, and Chapter 35 returns to what happens when $T$ grows into the hundreds of thousands.

Four Ways to Waste Less Cache

With the cache identified as the binding constraint, the history of serving engines reads as a sequence of ways to waste less of it. The serving stack got its present shape by removing one source of waste at a time, and each removal was a scheduling idea before it was a kernel.

Stop holding a finished request's slot. The first systems served requests the way a web server serves HTTP: collect a batch, run it to completion, return the batch. This is request-level batching, and it wastes the accelerator badly, because a short generation in the batch finishes early and then idles while the longest generation runs on, holding its slot. Orca replaced request-level batching with iteration-level scheduling: the scheduler decides what to run at the granularity of a single decode step, so a finished request leaves the batch immediately and a newly arrived one can join without waiting for the rest (Yu et al. 2022). This idea, widely called continuous batching, is the foundation every modern engine builds on.

Stop reserving cache you will not use. Iteration-level scheduling exposed the next bottleneck, which was memory. Continuous batching can admit a request the instant a slot opens, but only if there is KV cache memory to hold it, and early engines allocated that memory in one contiguous block per request sized for the maximum possible length. Most requests are far shorter than the maximum, so most of that reserved memory sat empty, fragmenting the pool and capping the batch size far below what the hardware could run. PagedAttention and the vLLM engine borrowed virtual memory's paging from operating systems: store the KV cache in fixed-size blocks that need not be contiguous, allocate blocks on demand as a sequence grows, and let a block table map logical positions to physical blocks (Kwon et al. 2023). Fragmentation collapses, the batch grows, and throughput rises by a reported two to four times at the same latency (Kwon et al. 2023). The mechanism of that paging, and how it enables prefix sharing across requests, is the subject of Chapter 32; here it is enough to see that the memory pressure named above forced the design.

Stop letting a prefill stall the decodes. The most recent turn confronts the prefill-decode tension head-on. Admitting a new request runs an expensive prefill that competes with the decodes already in flight: favor prefill and TTFT improves while running requests stutter; favor decode and TTFT climbs while ongoing streams stay smooth. Because a prefill is a large compute burst and a decode step is a small one, mixing them in a batch lets a prefill stall the decodes waiting behind it, spiking their TPOT. Two families of answer emerged. Sarathi-Serve keeps the phases on the same hardware but splits each prefill into chunks and interleaves them with ongoing decodes, so no decode is stalled by a long prompt (Agrawal et al. 2024). DistServe goes the other way and disaggregates the phases onto separate pools of accelerators, so prefill interference cannot touch decode latency at all, at the cost of shipping the KV cache between pools (Zhong et al. 2024). Both are explicitly optimizing goodput rather than raw throughput, which is why chunked prefill and disaggregation are the two principled ways to stop having to choose between TTFT and TPOT.

Where the remaining knobs trade

The two prefill-decode answers above resolve one of serving's trade-offs. Three more remain, and each one trades latency, throughput, or memory against another, with the KV pool sitting underneath all of them.

• Batch size: throughput versus latency. A larger decode batch amortizes the weight-streaming cost across more requests and raises throughput, but it consumes more KV cache memory and lengthens each step, raising TPOT. The right batch is the largest one that still meets the TPOT SLO, and it moves as request lengths change. Figure 31.5 draws the two curves against batch size: throughput climbs with diminishing returns toward a bandwidth-bound ceiling while TPOT rises, and the chosen batch sits at the point where TPOT meets its budget.

• KV cache memory versus admitted requests. Memory spent on weights cannot hold KV cache, and memory spent caching a long-context request cannot admit a new one. When the pool fills, an engine must either reject or queue new requests, or evict and later recompute the cache of a paused one, trading memory pressure for recompute cost.

• Latency floor versus utilization. Holding accelerators idle in reserve protects TTFT against bursts but wastes the very compute serving exists to monetize. Goodput is the function that prices this trade, because it counts utilization only when it produces in-SLO completions.

The loop, and what it reports

A serving engine is a loop that on each iteration chooses a set of requests to run, runs one model step on them, and frees the resources of any that finished. The shape below is the continuous-batching loop that Orca introduced and every modern engine refines (Yu et al. 2022).

# One iteration of a continuous-batching scheduler (sketch).
while running or waiting:
    # Admit new requests while KV-cache blocks are free and SLOs allow.
    while waiting and kv_pool.can_allocate(waiting.peek()):
        running.add(waiting.pop())          # its prefill joins this step
    batch = scheduler.select(running)       # may chunk prefills, cap by memory
    outputs = model.step(batch)             # one prefill/decode pass
    for req in batch:
        req.append(outputs[req])
        if req.is_finished():
            kv_pool.free(req.blocks)        # return blocks to the pool
            running.remove(req)

The load-bearing decisions are not in the model step but in the two lines that admit requests and select the batch. kv_pool.can_allocate is where the memory constraint becomes a scheduling decision, and scheduler.select is where the latency-throughput trade is struck, by capping batch size for TPOT, by chunking prefills so they do not stall decodes, and by ordering against SLOs. The corresponding logic in a real engine lives in the engine step, for example EngineCore.step, vllm/v1/engine/core.py in vLLM's V1 engine, which retired the older LLMEngine loop, with the batch formed by the scheduler in vllm/v1/core/sched/scheduler.py and the block accounting in its KV-cache manager.

The operational metrics that matter follow directly from the problem. TTFT and TPOT are measured per request and tracked as percentiles, not means, because a tail miss is an SLO miss. Goodput is the headline number, computed as in-SLO completions per second. KV cache utilization and the preemption rate, how often the engine has to pause and later restore a request because memory ran out, tell you whether the memory constraint or the compute constraint is currently binding. When preemptions climb, the cache is the bottleneck and the fixes are architectural and in Chapter 32; when the accelerators are full but latency is fine, throughput is the frontier and the fixes are in Chapter 33 and Chapter 34.

Through the lens this book carries: serving is where capability meets its price, because a model only delivers a behavior if it can be served inside a latency budget; efficiency is the entire game, since goodput per accelerator is what makes a capability affordable; and trust enters as the SLO itself, the promise that a request will return on time, which the scheduler must keep under load it did not choose.

Further reading