Part V · Chapter 36

Serving Multimodal Models

AuthorChangkun Ou

Reading time~18 min

This part built a serving stack on one assumption: tokens arrive cheaply. A tokenizer turns a prompt into a few hundred integers, and from there the cost model of Chapter 31 holds: prefill is compute-bound, decode is memory-bound, and the key-value cache grows one entry per token. Multimodal serving starts before the language model sees a token. A vision encoder converts raw pixels into visual tokens, often thousands of them, and it commits to that count before the scheduler can react. That count sets prefill FLOPs, key-value cache bytes, and batch size. A resolution setting is therefore a serving-cost decision, and where the encoder runs weighs on that cost just as directly. The fusion architectures these pipelines serve, the cross-attention and projection connectors that turn pixels into tokens a language model can read, are Chapter 15; here the concern is running them.

Figure 36.1. Schematic of why multimodal serving can be token-expensive. Fixed image patching grows quickly with resolution, while adaptive crops or routing reduce the slope. Idealized token counts, not measured data.

The token the tokenizer never made

In a text request, the expensive object is the prompt, and its length is the user's to set. In a multimodal request, the expensive object is the image, and its length is set by a model the user never sees. A vision encoder, almost always a Vision Transformer (the image counterpart of the text Transformer, running self-attention over patches the way a language model runs it over words; Chapter 15), cuts the image into a grid of fixed-size patches and produces one embedding per patch, a fixed-length vector standing in for that patch (Dosovitskiy et al. 2021). The grid is the whole story of cost. A CLIP-style encoder at 336 pixels with a patch size of 14 emits a 24 by 24 grid, 576 tokens, for a single image. Those 576 tokens are not metaphorical. They are concatenated into the backbone's input sequence and from that point are indistinguishable from text tokens: they consume positions, they attend and are attended to, and each one takes a full row in the key-value cache at every layer.

So a one-sentence question about a screenshot does not enter the model as a short sequence. It enters as 576 image tokens plus a handful of text tokens, and if the product allows high-resolution images the count climbs into the thousands. The serving consequences follow directly. Prefill, already compute-bound, now has far more tokens to process, and the encoder itself is an extra compute stage in front of it. The key-value cache, already the binding constraint on batch size in Chapter 32, fills with image tokens that were generated in a single forward pass rather than decoded one at a time. The image is paid for entirely at prefill and then resides in the cache for the life of the request, like an enormous prefix nobody typed.

Figure 36.2. The multimodal serving pipeline. The vision encoder and projector run once per image at prefill, producing visual tokens that are concatenated with text tokens and then live in the key-value cache exactly like text. The encoder is an extra compute stage the text stack never had.

Three stages, one sequence

The pipeline has three stages, and keeping them separate is what makes the cost legible. The vision encoder turns pixels into visual features. A projector, usually a small multilayer perceptron or a resampler, maps those features into the backbone's embedding space, reshaping them into vectors of the same width and format the language model's own token vectors use, so they can sit in the same sequence as text. The backbone, an ordinary decoder, then processes text and visual tokens together. The early bridged designs froze a language model and attached vision through cross-attention, with a Perceiver Resampler compressing variable visual features into a fixed small budget, on the order of 64 tokens, before they reached the backbone (Alayrac et al. 2022). The design that became standard instead projects visual features into the input sequence directly and trains the projector on instruction data, the recipe that made open multimodal models tractable (Liu et al. 2023). The two choices set the cost ceiling: a resampler fixes the visual token budget regardless of image size, while direct projection lets the budget grow with the grid, trading a fixed cost for fidelity.

The reason direct projection won despite its cost is resolution. A fixed 576-token grid at 336 pixels is too coarse to read a dense document, a chart, or a phone screenshot, the tasks that make a multimodal model useful for work. The field's response was to raise the effective resolution, and every method for doing so multiplies the token count.

The levers that set the count

Three levers govern how many tokens an image becomes, and a serving engineer should be able to read all three off a model card.

The first is the base grid: input resolution divided by patch size, squared. Doubling the resolution quadruples the tokens. The second is tiling. To reach high resolution without retraining the encoder, AnyRes-style schemes cut a large image into tiles, encode each tile at the native resolution, and concatenate the results, so a 1344 by 1344 image becomes a grid of sub-images and its token count is the per-tile count times the number of tiles, often several thousand. The third is dynamic resolution: rather than forcing every image to a fixed size, an encoder can process the image at its native aspect ratio and emit a variable number of tokens, with positional information carried by a multi-axis rotary embedding so the backbone knows the two-dimensional layout (Wang et al. 2024). Dynamic resolution makes the token count a function of the input, which is faithful but turns capacity planning into a distribution problem: the same endpoint might serve a 256-token thumbnail and a 16000-token full-page scan in adjacent requests.

The runnable below makes the explosion concrete. Token count scales with image area, and the key-value bytes scale with it linearly, so a product decision to support larger images is a decision about cache pressure and batch size. The estimate applies the formula of Chapter 31 with the grouped-query attention of a Llama-3-class backbone, eight key-value heads against thirty-two query heads.

# Visual token count and KV-cache footprint as resolution grows.
# Tokens = (W/patch) * (H/patch); KV bytes ~ tokens * layers * 2 * (hidden/gqa) * dtype.
patch, layers, hidden, dtype = 14, 32, 4096, 2  # ViT patch 14; a 7B-class backbone, fp16
gqa = 4  # n_head / n_kv: GQA divides KV bytes by this (8 KV heads vs 32 query heads)
for w, h in [(336, 336), (672, 672), (1008, 1008), (1344, 1344)]:
    tokens = (w // patch) * (h // patch)
    kv_mb = tokens * layers * 2 * (hidden // gqa) * dtype / 1e6
    print(f"{w}x{h}: {tokens:5d} visual tokens, {kv_mb:6.1f} MB of KV cache")
# One high-res image can cost more cache than a long text conversation.

Constraint arrow

The visual token count is set at the modality layer, by the encoder's resolution and tiling policy, and it dictates the two quantities Chapter 31 and Chapter 32 treat as given. Prefill FLOPs scale with the token count, and key-value bytes scale with it linearly, so the product decision "support 4K screenshots" reaches down and sets the prefill budget and the maximum batch size. A resolution choice is a serving-cost choice. The encoder, sitting one layer below the serving stack, decides how expensive every image is before the scheduler in Chapter 32 ever sees it.

Cutting the count back down

Because the count drives the cost, much of multimodal serving is the problem of spending fewer tokens per image without losing the pixels that matter. The methods divide by where they reduce the count.

Some reduce it before the backbone. Pixel-shuffle and similar spatial-to-channel folds merge a small neighborhood of patches into one token, dividing the visual token count by four at the cost of fine spatial detail, a trade many production encoders take by default (Chen et al. 2024). A resampler, the Flamingo move, goes further and fixes the budget outright, attending a large set of patches down to a constant small set (Alayrac et al. 2022). Both decide the count before a single backbone layer runs, which makes them cheap and predictable but forces the loss to happen blind to the prompt.

Others reduce it inside the backbone, after the model has begun to read. The observation that drives them is that visual token contribution falls early: after the first couple of layers, attention to image tokens collapses, so most of them can be pruned with little effect on the answer. FastV prunes visual tokens after an early layer and reports that an image is worth roughly half its tokens past layer two (Chen et al. 2024). Pruning inside the backbone can be prompt-aware, keeping the tokens the question actually looks at, but it complicates the serving path, because a pruned sequence changes length partway through prefill and the key-value cache must be compacted rather than simply appended to.

What's contested

Continuous embeddings or discrete tokens? Projecting continuous visual features into the sequence preserves fidelity but keeps vision outside the clean next-token abstraction; quantizing images into discrete tokens unifies the model at the cost of a quantization cliff (cross-ref Chapter 16). The flagships have not converged.
How many tokens does an image deserve? A fixed small budget is cheap and predictable; native dynamic resolution is faithful but makes capacity planning a distribution problem. There is no agreed answer, and the same lab ships different budgets for documents and for natural images.
Is aggressive pruning free? FastV-style pruning reports large savings at little accuracy cost on average, but the average hides the dense-text and small-object cases where the pruned tokens were the answer. Whether pruning is safe is task-dependent, not a global setting.

Where the encoder runs

The encoder is a compute stage the text stack never had, and encoder placement is a serving decision in its own right. Colocating it with the backbone on the same accelerator is simplest and keeps the visual features local, but it lets a burst of image-heavy requests starve the decode loop of compute, because the encoder and prefill now contend for the same matrix units. Disaggregating it, running a pool of encoder workers that ship visual embeddings to the backbone over the network, mirrors the prefill/decode disaggregation of Chapter 32 and for the same reason: it isolates a bursty, compute-bound stage from a latency-sensitive one, at the cost of moving embeddings across the wire and managing a second fleet. That mirror is no longer hypothetical: encoder disaggregation shipped as a native mode in vLLM (encode-prefill-decode, EPD, since v0.11.1), with the encoder pool scaled by data parallelism independently of the backbone's tensor parallelism and goodput roughly doubling on multi-image workloads (vLLM Multimodal Workstream 2025).

Image prefix caching helps more here than in text, because images repeat. A conversation that refers back to the same uploaded document re-sends it every turn, and a system that decorates every request with the same brand image pays for it every time, unless the encoder output or the resulting key-value entries are cached and reused. Prefix caching, already standard for shared text prefixes, extends naturally to a leading image: hash the pixels, and on a hit skip both the encoder and the image's prefill entirely (cross-ref Chapter 32). The economics are better than for text caching, since an image is a far larger block of tokens to reuse.

Video is the same problem with a larger multiplier. A clip is a sequence of frames, each frame is an image, and each image is hundreds to thousands of tokens, so a few seconds of video at a naive frame rate exceeds the context window before the question is even appended. Every video model therefore pools in time, sampling frames sparsely and merging tokens across adjacent frames, and the pooling rate is the same kind of lever as image resolution: it trades temporal fidelity for a token budget the serving stack can afford.

Serving generation: when the model makes the pixels

Everything so far has treated media as input, consumed and turned into tokens the decode loop of Chapter 31 understands. Turn the arrow around, and the serving problem inverts with it. A model that generates an image or a video is almost always a diffusion model, and diffusion breaks every assumption the autoregressive serving stack was built on. There is no key-value cache dribbling out one token at a time, because the sample does not exist until the final step: a diffusion model starts from noise and denoises it over many passes, each pass a full dense forward over every latent element of the image, and only the last pass yields something a user can see. So there is nothing to stream. The user gets a progress bar, not a token stream, and the cost is not bytes-per-token but FLOPs-per-step times the number of steps, with the per-step FLOPs scaling with resolution and, for video, with frame count.

That single structural fact reshapes the serving tier. Autoregressive decode is memory-bandwidth-bound and packs many cheap concurrent sequences into one batch; a diffusion request is compute-bound and a single high-resolution sample can saturate an accelerator by itself, so the serving shape is a job queue with latencies in seconds to minutes, not an interactive stream (Li et al. 2024). It also opens optimizations that have no autoregressive analogue. Because the same network runs many times over activations that change only slowly between steps, work can be reused: feature maps cached across adjacent steps and only the fast-changing parts recomputed (Ma et al. 2024), or a single sample split across GPUs by tolerating one-step-stale patch boundaries (Li et al. 2024). What gets cached here is the denoiser's own intermediate features across timesteps, not context across requests, and the caching is what a text stack would call recomputation avoidance rather than a KV cache.

Since cost is steps times per-step FLOPs, the dominant lever is collapsing the step count, and it is a training-time intervention with a direct line to the invoice. Progressive distillation halves a sampler's steps per round, taking thousands of steps down to single digits (Salimans and Ho 2022); consistency models map any point on the denoising trajectory straight to the result, enabling one-step generation (Song et al. 2023); adversarial distillation reaches one-to-four-step synthesis good enough for real-time use (Sauer et al. 2024). The pricing follows the physics closely: at one image API, the same flow-matching architecture in its full-sampler form and its few-step distilled form are billed per megapixel at an eight-fold price gap that tracks the step-count gap (Black Forest Labs 2024). Nothing about the bill is per-token; it is resolution times a quality tier, which is the cost model passed straight through. The runnable reconstructs the gap from the FLOPs alone: cut the steps, and the cost per image drops in proportion.

# one image = steps x a dense forward over every latent token; no KV cache, no stream
P, TOK = 12e9, 4096                        # 12B diffusion transformer, ~4096 latent tokens/megapixel
tflops_per_step = 2 * P * TOK / 1e12       # ~98 TFLOPs per step per megapixel
gpu_tflops, mfu, price_hr = 989, 0.40, 2.0 # H100 bf16 dense, utilization, illustrative $/GPU-hr
def per_image(steps, mp=1.0):
    secs = steps * tflops_per_step * mp / (gpu_tflops * mfu)
    return secs, secs / 3600 * price_hr
for name, steps in [("base sampler", 50), ("distilled", 4), ("one-step", 1)]:
    s, usd = per_image(steps)
    print(f"{name:>12}: {s:5.2f} GPU-s  ${usd:.4f}/image")
print("50 -> 4 steps is ~12x cheaper: step count is the pricing knob")

Video multiplies every term. A diffusion transformer over spatiotemporal patches (Brooks et al. 2024) must attend over a sequence whose length scales with frames as well as pixels: one production video model carries a 73,000-token context for sixteen seconds of high-definition footage, against roughly four thousand latent tokens for a single megapixel image (Polyak and others 2024), and attention cost grows with the square of that length. A single video sample is therefore a minutes-long, multi-GPU dense job, which is exactly why the product shape is submit-poll-download priced in credits or per output-second rather than a chat stream. The frontier is now bending this batch shape back toward streaming: few-step distillation combined with causal generation, generating frames left to right conditioned on past frames, streams video at interactive frame rates, and to do it the field reinvented the KV cache it had discarded (Yin et al. 2024). Real-time interactive world models push further still, turning a request from a job into a stateful per-user session with a frame deadline measured in tens of milliseconds, which is game-server economics rather than throughput serving (Google DeepMind 2025).

What's contested

Whether generative-media serving converges with the LLM serving stack is unsettled and still moving. The convergence case is already in production: unified engines now fold diffusion transformers into the same runtime as language models, sharing sequence-parallel and classifier-free-guidance parallelism and reusing the same kernels, and the causal-video trick reimports the KV cache and continuous batching wholesale. The divergence case is that the two workloads share almost nothing operationally: dense compute-bound multi-second kernels against bandwidth-bound millisecond iterations, queue and session scheduling against continuous batching, per-sample multi-GPU parallelism against per-token weight amortization, and no prefix-cache economics at all. The likely resolution is that the kernels and engines converge while the tier above them, the scheduler, the SLO, the pricing unit, stays split until interactive generation makes video look enough like decode to merge.

Resolution is a serving decision

Multimodality buys capability mostly by spending efficiency. The capability is real: reading a chart, a screenshot, or a scanned form is what makes a model useful beyond chat. But every increment of visual capability is an increment of tokens, and tokens are prefill FLOPs and key-value bytes, the two quantities the whole serving stack is built to conserve. Efficiency is therefore the binding term, and the engineering, dynamic resolution, pixel-shuffle, resampling, prompt-aware pruning, encoder disaggregation, image prefix caching, is all in service of spending the fewest tokens that still let the model see. Trust enters as the quiet risk under pruning: a token-reduction setting that looks free on a natural image benchmark can silently drop the one region a dense document or a small defect lived in, so the safe deployment couples any aggressive reduction to an evaluation on the dense-detail tasks the product actually serves (cross-ref Chapter 52). The constraint arrow is the lesson to carry up the stack: the cheapest place to control multimodal serving cost is the resolution policy at the modality layer, long before the scheduler sees the request.