Serving Multimodal Models

A text server receives token IDs. A multimodal server may first have to fetch, decode, resize, sample, encode, and project an image, a video, or an audio clip. Those stages add their own queues and failure modes before language-model prefill begins. They may also produce hundreds or thousands of features whose count depends on the media and its preprocessing policy. Prompt length alone is therefore not enough for admission control.

This chapter covers two workloads that share media formats but little else. In media-input serving, an encoder supplies evidence to a language model, which then generates text. In media-generation serving, a model produces an image or video through an iterative or autoregressive process. The first extends the serving path developed in this part; the second needs a different cost model and scheduler.

Start with the fusion contract

The architecture determines where media features live and which cache pays for them. Two common contracts illustrate why there is no universal “visual token” cost (Qiu et al. 2025).

Discrete image or audio codes add another representation choice, but they do not remove this architectural question: after lookup, code IDs also become continuous vectors, and the model must still specify whether they join self-attention or a separate memory. Chapter 15 develops these model designs. Here, the important point is operational: capacity accounting is architecture-dependent.

Account for the input after preprocessing

For decoder-only fusion, let the final input length be

$$N_{\mathrm{in}} = n_{\mathrm{text}} + n_{\mathrm{special}} + \sum_{i=1}^{q} n_i, \qquad n_i = g(m_i; P,E,C).$$

Here, $m_i$ is media item $i$; $q$ is the number of media items; $P$ is the decode, resize, crop, frame-sampling, and padding policy; $E$ identifies the media encoder revision; $C$ identifies the connector, merger, or resampler; $g$ returns the number of features inserted for one item; and $n_{\mathrm{text}}$ and $n_{\mathrm{special}}$ count text and model-specific separator positions. Every symbol is a deployment value. Counting raw image patches without $P$, $E$, and $C$ is not enough to predict the sequence seen by the language model.

If all $N_{\mathrm{in}}$ positions remain in self-attention, their KV allocation is

$$M_{\mathrm{KV}} = 2 L N_{\mathrm{in}} n_{\mathrm{kv}} d_h b_{\mathrm{kv}}.$$

In this expression, $L$ is the number of language-model layers that store self-attention KV state, $n_{\mathrm{kv}}$ is the number of key-value heads, $d_h$ is one head's width, and $b_{\mathrm{kv}}$ is the bytes used for each cached scalar. The factor 2 represents keys and values. This formula applies to the projected-prefix contract. A cross-attention model needs a separate term for media keys and values at the layers that actually contain cross-attention; it should not be charged as though every feature were an ordinary text prefix.

The time to first token has more stages as well:

$$T_{\mathrm{TTFT}} = T_{\mathrm{ingress}} + T_{\mathrm{preprocess}} + T_{\mathrm{encoder\ queue}} + T_{\mathrm{encode}} + T_{\mathrm{connector/transfer}} + T_{\mathrm{LM\ queue}} + T_{\mathrm{prefill}}.$$

Each $T$ is elapsed time in the named stage. Some terms can overlap, so the sum is an upper-level accounting model rather than a promise that every system runs them serially. It is still more useful than a single “prefill latency” number because it reveals whether a slow request waited for an encoder, crossed a network boundary, or occupied the language backbone.

Language prefill itself is not simply linear in media length. Dense layers add work roughly proportional to $N_{\mathrm{in}}$, while dense self-attention has a quadratic attention term in the combined sequence length. Optimized kernels change constants and memory traffic, not this distinction. During decode, each new text query also reads the retained media state: from the self-attention KV cache in decoder-only fusion or through the model's cross-attention path.

Derive visual length from the actual policy

For a fixed grid with preprocessed height $H'$, width $W'$, and patch dimensions $P_h \times P_w$, the raw patch count is

$$n_{\mathrm{patch}} = \left\lceil \frac{H'}{P_h} \right\rceil \left\lceil \frac{W'}{P_w} \right\rceil.$$

Here, $H'$ and $W'$ are dimensions after resizing, cropping, or padding; $P_h$ and $P_w$ are the patch height and width; and the ceiling accounts for a policy that pads incomplete edge patches. A policy that rejects or center-crops non-divisible dimensions has different behavior. The language-model count may then remove a class feature, add separators or a global view, concatenate several tiles, merge neighboring features, or replace the whole grid with a fixed number of resampler queries.

For example, LLaVA-1.5 uses a 336-pixel CLIP ViT-L/14 encoder. After the class feature is removed, its $24\times24$ patch grid contributes 576 projected image positions (Liu et al. 2024). That is a model-specific result, not a generic property of CLIP or of 336-pixel images.

The runnable below isolates that one fixed-grid contract. It assumes divisible dimensions, no class or separator positions, no tiling or merge, and insertion of every patch into all 32 language-model layers. It uses explicit GQA head dimensions and reports binary MiB.

def visual_tokens(width, height, patch=14):
    if width <= 0 or height <= 0 or patch <= 0:
        raise ValueError("dimensions and patch must be positive")
    if width % patch or height % patch:
        raise ValueError("this fixed-grid example requires divisible dimensions")
    return (width // patch) * (height // patch)

def kv_mib(tokens, layers=32, n_kv_heads=8, head_dim=128, bytes_per_value=2):
    if min(tokens, layers, n_kv_heads, head_dim, bytes_per_value) <= 0:
        raise ValueError("KV dimensions must be positive")
    kv_bytes = 2 * layers * tokens * n_kv_heads * head_dim * bytes_per_value
    return kv_bytes / 2**20

for width, height in [(336, 336), (672, 672)]:
    tokens = visual_tokens(width, height)
    print(f"{width}x{height}: {tokens:4d} visual tokens, {kv_mib(tokens):5.1f} MiB KV")

Real systems use several mechanisms to change this count:

• A fixed grid makes cost predictable but may resize away small text or small objects.
• Tiling preserves more spatial detail by encoding multiple supported crops. It is a trained model feature, not an arbitrary server-side operation; global views, overlap, separators, and padding make the final count model-specific.
• Dynamic resolution lets count vary with aspect ratio and input size. Qwen2-VL combines this policy with a $2\times2$ visual merger and multimodal rotary positions, so its raw patch count and language-backbone count are not the same (Wang et al. 2024).
• A spatial merge or pixel shuffle folds neighboring spatial features into channels before projection. This shortens the sequence, but it does not by itself prove that fine detail was discarded; the trained connector decides what survives. InternVL 1.5 combines such a connector with dynamic tiling (Chen et al. 2024).
• A fixed-budget resampler maps a variable encoder grid to a chosen number of features. Language-model cost becomes predictable per media item, although the encoder and resampler still process the larger input (Alayrac et al. 2022).
• Token pruning removes features after some backbone layers. FastV, for example, applies a layer-2 attention-based policy to evaluated vision-language models and reports lower FLOPs (Chen et al. 2024). Earlier layers still pay for the full sequence; only later layers become cheaper. Attention weight is not a proof of causal importance, so aggressive pruning must be tested on OCR, charts, small text, and small objects rather than accepted from an average VQA score.

Video and audio add time as another length axis. A video policy may sample frames, form spatiotemporal patches, or merge across time. An audio policy may chunk a waveform or spectrogram into encoder frames. Duration, sample rate, frame rate, temporal stride, and per-request media limits therefore belong in the same capacity model as image resolution. None of these policies can be inferred from the user's text-token count.

Schedule a resource vector, not a prompt length

A useful admission record describes the work before it reaches any queue. At a minimum it contains a resource vector with media bytes, decoded pixels, image count, video frames or audio duration, estimated encoder work, projected token count, text-token count, output-token limit, required model and adapter, and the applicable latency class. Limits should be checked before allocating large decoded buffers or accelerator memory.

The request then passes through two different batching problems:

1. The media side benefits from shape buckets. Grouping compatible resolutions, frame counts, or audio lengths reduces padding waste and makes encoder kernels efficient. A single large item should not force every item in the batch to pad to its shape.
2. The language side batches by total prefill and KV demand. It must reserve enough cache for the projected token count and output-token limit, and may chunk a long multimodal prefill so that it does not monopolize the engine.

Separate encoder and language-model queues make their backlogs visible. They also allow admission control and fairness policies to prevent a burst of high-resolution requests from causing head-of-line blocking for short text or small-image traffic. Production traces are commonly heavy-tailed and bursty, so averages hide the requests that set tail latency (Qiu et al. 2025).

Colocation keeps media features on the same accelerator and avoids network transfer, but encoder work can interfere with prefill and decode. Disaggregation lets encoder replicas scale independently and protects the language tier from a modality burst. Its cost is explicit: feature tensors must be serialized, transferred, routed, and invalidated across another service boundary. The transfer grows with feature count and width, and an undersized encoder pool or network simply becomes the new queue.

EPD systems separate encode, prefill, and decode for this reason. An ICML 2025 study reports that the stages have different resource profiles and benefits from independent placement (Singh et al. 2025). vLLM later exposed native encoder disaggregation and reported roughly 2--2.5 times goodput in specific Qwen3-VL benchmarks on four A100 GPUs (vLLM Multimodal Workstream 2025). Those figures demonstrate the mechanism; they are not a capacity promise for another model, workload, or network.

Cache the stage you intend to skip

“Image caching” can refer to three different objects. They have different keys and save different work.

vLLM, for example, manages encoder outputs by multimodal-item hash and reference count, separately from its language-model prefix cache (vLLM Project 2026). This separation matters. An encoder-output hit does not imply a KV prefix-cache hit, and a changed system prompt, adapter, preprocessing rule, or projector can invalidate one cache while leaving another usable.

Cache keys also need a tenant scope, authorization decision, collision-safe comparison, retention policy, and revision-based invalidation. Reuse across a conversation can be valuable for a document or repeated video, but global reuse must not reveal that another tenant supplied the same private media. Measure hit rate and saved stage time under the real workload before reserving a large cache.

Verify media-input serving end to end

Profile the complete request path under a replay that preserves modality mix, item counts, resolution, duration, text length, output length, and arrival bursts. Report p50, p95, and p99 rather than a single average.

For language-producing requests, record at least:

• ingress/decode time, preprocessing time, encoder queue and execution time, connector or transfer time, language queue time, time to first token, and inter-token latency;
• decoded pixels or frames, projected token count, peak encoder memory, peak KV memory, rejected or downsampled requests, and accelerator utilization by stage;
• processor, encoder cache hit rate, and KV prefix-cache hit rate, including bytes retained and time actually saved;
• goodput and tail latency at matched load, both for the total workload and for each modality and size bucket;
• answer quality at each shipped resolution, frame, merge, resampler, or pruning policy on the product's OCR, chart, small-object, multi-image, video, and audio tasks, compared with a full-detail baseline.

This measurement prevents two common mistakes: calling an optimization useful because one isolated kernel is faster, and calling compression safe because an aggregate benchmark hides the detail-sensitive cases.

Media generation is a different serving path

Media output should not be forced into the input-serving model. A common latent image pipeline has a text encoder, an iterative denoiser or flow model, and a latent decoder (Rombach et al. 2022). Other systems generate discrete visual codes autoregressively or predict coarse-to-fine blocks. The scheduler must know which contract it is running.

For an iterative latent model, a useful latency decomposition is

$$T_{\mathrm{output}} = T_{\mathrm{condition}} + \sum_{s=1}^{S} e_s T_{\mathrm{denoise}}(z_s) + T_{\mathrm{decode}} + T_{\mathrm{encode/store}}.$$

Here, $S$ is the number of solver steps or neural-function evaluations; $s$ identifies one step; $z_s$ is the latent state processed at that step; $e_s$ is the number of denoiser evaluations required at that step, including any guidance branches that are not fused; $T_{\mathrm{condition}}$ covers text or reference-media conditioning; $T_{\mathrm{denoise}}$ is measured denoiser time for the chosen shape and batch; $T_{\mathrm{decode}}$ converts the final latent to media; and $T_{\mathrm{encode/store}}$ covers output encoding, storage, or delivery. The sum makes clear why fewer steps help without claiming that steps are the only cost.

Resolution and duration change the latent length. For a two-dimensional latent with image size $H\times W$, autoencoder compression $f_h\times f_w$, and transformer patch size $p_h\times p_w$, the latent token count is

$$N_z = \left\lceil \frac{H}{f_h p_h} \right\rceil \left\lceil \frac{W}{f_w p_w} \right\rceil.$$

In this formula, $H$ and $W$ are output pixels; $f_h$ and $f_w$ are the autoencoder's spatial compression factors; $p_h$ and $p_w$ are the denoiser's latent-patch dimensions; and $N_z$ is the number of latent tokens per denoiser evaluation. Video adds a temporal factor after temporal compression and patching. Dense global attention can add a quadratic term in $N_z$; local, factorized, or sparse attention changes that term.

The runnable computes a transparent token-pass proxy for a 1024-pixel square, 8-fold latent compression on each spatial axis, $2\times2$ latent patches, and two guidance branches. It is not a FLOPs estimate: it omits attention shape, model width, fixed stages, communication, sparsity, batching, and utilization.

def latent_tokens(width, height, compression=8, patch=2):
    if min(width, height, compression, patch) <= 0:
        raise ValueError("all dimensions must be positive")
    stride = compression * patch
    return ((width + stride - 1) // stride) * ((height + stride - 1) // stride)

def token_pass_proxy(tokens, steps, guidance_branches=2):
    if min(tokens, steps, guidance_branches) <= 0:
        raise ValueError("tokens, steps, and branches must be positive")
    return tokens * steps * guidance_branches

tokens = latent_tokens(1024, 1024)
base = token_pass_proxy(tokens, 50)
fast = token_pass_proxy(tokens, 4)
print(f"latent tokens: {tokens}")
print(f"50-step proxy: {base} token-passes")
print(f" 4-step proxy: {fast} token-passes")
print(f"denoiser-only ratio: {base / fast:.1f}x")

Progressive distillation, consistency models, and adversarial diffusion distillation show that a trained model can reduce neural-function evaluations from many steps to a few (Salimans and Ho 2022; Song et al. 2023; Sauer et al. 2024). The result is a new quality--latency point, not a free runtime switch: sampler, guidance, and step count can all change output quality and must be evaluated together.

Repeated denoising creates other serving opportunities. Compatible requests can batch at the same model, shape, and iteration, while mismatched step counts can cause head-of-line blocking or padding-like waste. A scheduler should support cancellation between iterations so abandoned jobs stop consuming accelerator time. Within a request, DeepCache reuses selected U-Net features across nearby steps (Ma et al. 2024), and DistriFusion partitions a high-resolution sample across devices using stale context from the previous step (Li et al. 2024). Both are method-specific approximations with quality and hardware assumptions, not generic KV-cache behavior.

A conventional bidirectional denoiser updates the whole latent repeatedly, so it does not provide independent finalized tokens like autoregressive text. Services can still decode and stream a preview, but preview cadence adds decoder work and a noisy intermediate image is not a completed result. Causal video is different again: CausVid generates frame blocks left to right and uses a KV cache, demonstrating that causal media generation can expose a streaming contract (Yin et al. 2025).

For media output, measure queue time, first-preview latency, final latency, neural-function evaluations, guidance branches, accelerator-seconds per output, cancellation waste, output encoding and egress, and quality at each step count, resolution, duration, sampler, and batch policy. Compare quality with matched prompts, seeds, and settings. A cheap output that no longer meets the product's quality target is not a serving improvement.

The operating rule

Treat media policy as part of the service contract. Publish accepted formats, byte and dimension limits, image and frame counts, duration, resize or sampling behavior, token or feature ceilings, downsampling and rejection behavior, and output quality tiers. Then make the scheduler enforce the same contract used by capacity tests.

The point is not that every extra pixel is bad. It is that media capability has an explicit path through preprocessing, encoder work, persistent state, queueing, and quality. Once those terms are visible, encoder placement, image prefix caching, resolution limits, pruning, and step reduction become testable engineering choices instead of slogans.

Further reading