Multimodal Models: Fusion and Generation

The previous chapters separated the engines: diffusion for pixels, token models for non-left-to-right generation, and realtime systems for speech. Multimodal models have to join those engines back together. A system that sees, speaks, and generates images or video is not defined by one architecture so much as by where its representations meet: a vision encoder projected into a language model, a text encoder steering a diffusion model, a spacetime autoencoder making video small enough to model, or one unified sequence that tries to carry every modality at once. The chapter follows those junctions, because most of the trade-offs in multimodality are set at the boundary between representations. Those boundaries surface as concrete costs and controls: image token count at serving time, classifier-free guidance for prompt adherence, rectified flow and spacetime autoencoders for image and video generation, and the still-open split between modular systems and unified models, including the discrete-versus- continuous representation fight.

Fusing vision into language

The shared space comes first. contrastive language-image pretraining (CLIP) is the paired image-text training recipe that contrastively aligns an image encoder and a text encoder, training so that a matching image and its caption land near each other in one shared vector space while mismatched pairs land far apart. It does this over 400 million pairs with a softmax InfoNCE loss over an in-batch similarity matrix, a loss that rewards the one correct pairing in each batch against all the wrong ones, so an image can be classified zero-shot by matching it against text prompts (Radford et al. 2021), and that aligned space is the conditioning signal much of image generation later reuses. SigLIP replaces the softmax with a pairwise sigmoid loss, turning every image-text pair into an independent binary decision; because a sigmoid needs no global normalization across the batch, it drops the all-gather of the full similarity matrix, decouples the loss from batch size, and trains well at both small and very large batches (Zhai et al. 2023). A separate and often-confused fact is the modality gap: in CLIP-style spaces the image embeddings and the text embeddings do not intermix but sit in two narrow cones held apart, a geometry that forms at initialization and that contrastive training preserves (Liang et al. 2022). The gap is geometric, the sigmoid loss is computational, and they are unrelated results worth keeping apart.

To make a language model itself see, three connector patterns compete, and the design axis is how much of each tower stays frozen, how many trainable parameters the bridge adds, and where the modalities meet.

Cross-attention injection keeps both towers frozen and inserts trainable layers between them; cross-attention is the mechanism that lets one stream of tokens read from another, here letting the language model attend to the visual features. Flamingo bridges a frozen vision encoder and a frozen Chinchilla language model with gated cross-attention dense layers interleaved through the LM, each opened by a tanh gate initialized at zero so the layers are no-ops at the start and do not perturb the pretrained weights; a Perceiver Resampler in front maps a variable number of visual features to a fixed 64 visual tokens, which is what lets the model ingest interleaved images and do few-shot multimodal tasks (Alayrac et al. 2022). BLIP-2 keeps both towers frozen too but bridges them with a lightweight Querying Transformer: a small set of 32 learnable query vectors (32 by 768) cross-attends to the frozen image features and passes that fixed-length summary to the frozen LLM, the lightest connector of the three and a hard information bottleneck, trained in two stages (Li et al. 2023). Projection instead meets the modalities at the input: LLaVA maps the vision encoder's patch features straight into the language model's token space, as a single linear layer in the original and a two-layer MLP in LLaVA-1.5, and fine-tunes on instruction data (Liu et al. 2023; Liu et al. 2024). The clean contrast is Flamingo deep, frozen-both, heavy, fixed at 64 tokens; BLIP-2 a frozen bottleneck at 32; LLaVA shallow and cheap, with the token count set by whatever patch sequence the vision transformer (ViT) emits.

That last clause is where the cost lives. A vision-language model spends most of its inference budget on the image's tokens, because each visual token is a sequence position the language model attends over, and attention is quadratic in sequence length while the key-value cache grows linearly. CLIP ViT-L/14 at 336 pixels emits a 24-by-24 grid, so LLaVA-1.5 spends 576 tokens per image (Liu et al. 2024), and the push to read dense documents and small text makes that number grow. LLaVA-NeXT tiles a high-resolution image into 336-pixel crops plus a downscaled global view, so a many-tile image can reach thousands of visual tokens (Liu et al. 2024). Qwen2-VL processes images at native resolution into a variable token count and shares one positional scheme across text, image, and video with a multimodal rotary embedding (a scheme that encodes a token's position by rotating its query and key vectors, here extended across the image and video axes) (Wang et al. 2024), and InternVL scales the vision tower itself to a six-billion-parameter encoder (Chen et al. 2024). Higher resolution buys accuracy on dense visual content and is paid for, per image, on every forward pass, the same token-count pressure Chapter 36 feels at serving time.

The training recipe that made open vision-language models practical is visual instruction tuning, and its critical move was to manufacture the data. With no large image-instruction dataset, LLaVA bootstrapped one from a language-only GPT-4: it fed the model an image's symbolic description, its captions and bounding boxes, which are text, and prompted it to write conversations and reasoning grounded in that scene, yielding about 158 thousand samples from a model that never saw a pixel (Liu et al. 2023). Training runs in two stages, first aligning only the projection layer so visual features land in the right space, then fine-tuning the projection and the language model together on the instruction data. Whether to also unfreeze the vision encoder is unsettled: freezing preserves CLIP's general representation and avoids forgetting on limited data, while unfreezing lets the encoder adapt to fine-grained tasks like OCR and grounding that frozen features under-serve, at the cost of more data and compute.

Generating images

Built on the diffusion engine of Chapter 12, the systems that reach text-to-image share one ingredient before they split into routes. Classifier-free guidance is the main control for prompt adherence. A single network is trained both conditional and unconditional by randomly dropping the text condition for a null token on a fraction of steps, and at sampling the two predictions are extrapolated apart,

$$\tilde{\boldsymbol{\epsilon}}_\theta(z_t, c) = (1 + w)\,\boldsymbol{\epsilon}_\theta(z_t, c) - w\,\boldsymbol{\epsilon}_\theta(z_t),$$

Here $z_t$ is the current latent, $c$ is the text condition, $\boldsymbol{\epsilon}_\theta(z_t,c)$ is the conditional noise prediction, and $\boldsymbol{\epsilon}_\theta(z_t)$ is the unconditional prediction made with the null condition. The weight $w$ is the control dial: $w = 0$ recovers the unguided model, while larger values extrapolate away from the unconditional direction (later papers rename the dial as a guidance scale $s = w + 1$) (Ho and Salimans 2022). The extrapolation pushes a sample toward what the conditional model prefers and away from the unconditional prior, sharpening alignment without a separately trained classifier of the kind Dhariwal and Nichol (2021) had needed. It is also a control with a cost: small guidance gives the best distributional fidelity, while strong guidance gives the sharpest, most on-prompt single samples at the expense of diversity, which is the fidelity-versus- variety trade-off every production system exposes.

The diffusion route then conditions a diffusion model on text in one of two shapes. The cascaded shape generates a small image and upsamples it with further diffusion stages: Imagen renders at 64 pixels and super-resolves to 256 and 1024, conditioned on a frozen T5-XXL text encoder, and reports the finding that scaling that frozen language model lifts sample fidelity and image-text alignment far more than scaling the diffusion model itself (Saharia et al. 2022), a clean case of a language model dictating a visual generator's ceiling. The latent shape, the Stable Diffusion line of Chapter 12, runs diffusion in a pretrained autoencoder's latent space instead of on pixels (Rombach et al. 2022), and SDXL scales that line with a roughly threefold larger UNet, a second text encoder, and a separate refinement model applied as a post-hoc step (Podell et al. 2024). DALL-E 2 splits the problem differently, mapping the caption to a CLIP image embedding with a prior and rendering from it with a diffusion decoder (Ramesh et al. 2022). The sharpest lesson of the route, though, is about data, not architecture: DALL-E 3 retrains on a corpus relabeled by a bespoke captioner, a blend that is 95 percent synthetic captions, and finds image-text alignment improves monotonically with the synthetic share (Betker et al. 2023).

The training objective itself moved. The modern text-to-image systems train with rectified flow rather than noise-prediction, learning a straight velocity field from noise to data so an ODE sampler needs far fewer steps, the flow-matching idea of Chapter 12. With the linear interpolant $x_t = (1-t)\,x_0 + t\,\boldsymbol{ \epsilon}$ the target is simply $\boldsymbol{\epsilon} - x_0$ (here $x_0$ is the data and $\boldsymbol{\epsilon}$ the noise endpoint, Stable Diffusion 3's data-to-noise convention, the reverse of Chapter 12's noise-to-data labeling where the data sits at $x_1$), and Stable Diffusion 3 is the canonical instance: it adds a logit-normal weighting that concentrates training on the perceptually important middle noise scales (it draws each step's training noise level from a logit-normal distribution), and an MMDiT (multimodal diffusion transformer) that keeps separate weights for the text and image token streams while letting information flow between them, improving typography and prompt comprehension (Esser et al. 2024). The recipe outlived its lab: the same authors, reassembled as Black Forest Labs, shipped the FLUX line in August 2024, a twelve-billion-parameter rectified-flow MMDiT that carried the design into the dominant open-weights image models (Black Forest Labs 2024).

The autoregressive route is the diffusion-versus-tokens tension of Chapter 13 applied to pixels: tokenize the image into a learned discrete codebook with VQGAN (Esser et al. 2021), improved by a ViT-based quantizer (Yu et al. 2022), then model those tokens with a transformer, which Parti scales to twenty billion parameters as text-to-image sequence-to-sequence over image tokens (Yu et al. 2022). The route was widely written off until a 2024 revival recast image autoregression as coarse-to-fine next-scale prediction: VAR predicts each higher-resolution token map from all coarser ones rather than scanning left to right, recovering state-of-the-art class-conditional ImageNet generation at an order of magnitude less inference and showing language-model-like scaling (Tian et al. 2024). That result is class-conditional rather than text-to-image, so at the time it reopened the route as an architecture rather than settling it as a system; the system-level settling came in March 2025, when OpenAI shipped GPT-4o's native image generation, disclosed as an autoregressive model, in production (OpenAI 2025).

Generating video

Video adds time, and time adds cost. The first step extended image diffusion with a space-time factorized 3D U-Net to keep frames temporally coherent (Ho et al. 2022). The 2024 shift puts a diffusion transformer over latent patches, the diffusion transformer (DiT) backbone whose sample quality scales with its compute (Peebles and Xie 2023), and trains it across variable durations, resolutions, and aspect ratios; OpenAI's Sora is the prominent example, compressing video with a causal autoencoder into a spacetime latent grid and cutting that grid into patches that act as transformer tokens, so packing rather than a fixed frame count is what lets one model span clip lengths and shapes, described in a non-peer-reviewed report (Brooks et al. 2024). The open systems make the compression concrete. CogVideoX pairs a 3D causal autoencoder that compresses by four in time and eight on each spatial axis with an expert transformer for text-video fusion (Yang et al. 2024), and Meta's Movie Gen trains a thirty-billion-parameter transformer with the flow-matching objective over a temporal autoencoder that compresses by eight on every axis, a 512-fold reduction before the transformer runs (Polyak and others 2024).

The 2025 systems moved the fusion boundary inside the generator itself. Veo 3 (May 2025) and Sora 2 (October 2025) generate video and synchronized audio, dialogue, ambience, and sound effects, jointly in one sampling pass rather than dubbing a soundtrack onto finished frames (Google DeepMind 2025; OpenAI 2025). Lip sync and sound-event timing then come out of the sample itself rather than a stitching step, the same modular-to-unified motion the next section examines for images.

What makes any of this tractable is the autoencoder compression, and the reason it bites on the time axis is specific. Spatial compression removes more redundancy per axis, eight by eight against four, but temporal compression is the lever because duration scales the token count linearly and without bound while spatial resolution is fixed per model. Compressing time is what keeps a longer clip inside the quadratic-cost transformer's reach.

# Why video generation needs a latent autoencoder: count the tokens a diffusion
# transformer must attend over, in pixel space vs a compressed latent space.
fps, seconds = 24, 5
H = W = 512
patch = 2                          # transformer patch size in the working space
for name, f in [("pixel space", 1), ("latent (x8 VAE)", 8), ("latent (x16 VAE)", 16)]:
    h, w = H // f, W // f
    tokens = fps * seconds * (h // patch) * (w // patch)
    print(f"{name:18s} -> {tokens:>10,d} spacetime tokens")
print("attention is quadratic in tokens; latent compression is what makes video tractable")

Unified or modular

The deepest open question is whether one model should both understand and generate every modality, or whether a system should compose specialized pieces. Inside the unified camp there is a sharper fight, and it is being run inside one lab: whether to quantize images into discrete tokens or keep them continuous.

Chameleon takes the discrete, early-fusion bet, quantizing images into codebook tokens, interleaving them with text into one sequence, and training a single transformer with one next-token objective over the mixed vocabulary (Chameleon Team 2024). Its load-bearing contribution is a stability story that says something general about mixed-modal training. Because softmax is invariant to adding a constant to all its logits, two modalities sharing one set of weights can compete by slowly growing the norm of their activations, harmless at first but eventually pushing values outside the range of bf16 and diverging, what the paper calls the logit drift problem. The fix is layer-norm on the query and key vectors before the attention dot-product to bound the norm feeding the inner softmaxes, without which their seven-billion model diverges after about a fifth of training, plus a z-loss to control the final output softmax and a norm reordering for the SwiGLU path. Transfusion takes the opposite bet, keeping images continuous: one transformer carries a next-token loss on discrete text and a diffusion loss on continuous image patches in the same sequence, and reports that not quantizing beats Chameleon at less than a third of the compute on image generation (Zhou et al. 2025). The reason is the tokenizer: a VQ codebook is lossy compression, so its reconstruction error sets a ceiling that no model over those codes can exceed, which is the deep argument for staying continuous. Variants sit between the poles, Show-o unifying autoregressive text with masked-token image diffusion (Xie et al. 2025) and Emu3 pushing the pure-discrete bet to image, text, and video under one next-token loss (Wang and others 2024).

The commercial instance of the unified bet is GPT-4o, which OpenAI describes as a single model trained end-to-end across text, vision, and audio so that one network processes every input and output, replacing the older three-model voice pipeline and cutting audio latency to a few hundred milliseconds (OpenAI 2024). The deeper details, tokenizer, training mix, layer structure, remain undisclosed, but which family the image generator belongs to no longer is: in March 2025 OpenAI disclosed that 4o's native image generation is an autoregressive model embedded in ChatGPT, shipped to developers as the gpt-image-1 API (OpenAI 2025). The modular bet keeps a strong vision-language model for understanding and separate specialized generators for images and video, composed through tool calls; OpenAI ran that design in production too, GPT-4 invoking the diffusion-based DALL-E 3 as a tool, until the native autoregressive generator replaced the tool call in ChatGPT and GPT-4 itself was retired in April 2025, which makes the modular-to-unified transition a concrete production data point rather than an abstraction. The two evaluation axes that keep these systems comparable are understanding and text-to-image alignment. On the understanding side, MMMU was hard enough at release that the first frontier models scored in the high fifties (Yue et al. 2024); by 2026 frontier models score in the low-to-mid eighties, the benchmark is near saturation, and the discriminating work has moved to the harder MMMU-Pro (Yue et al. 2024) and to video and audio understanding. On the alignment side sit the object-focused GenEval (Ghosh et al. 2023) and the dense-prompt DPG-Bench (Hu et al. 2024).

Further reading