Multimodal Models: Fusion and Generation

Multimodality creates interfaces, not one architecture. Images arrive as grids, text as sequences, audio as timed frames, and video as all three dimensions at once. A system must decide how to encode each input, where representations can interact, how each output is generated, and what the serving path can afford. Those decisions are related, but they are not interchangeable. A model can be unified while using different losses for text and images, and a modular system can still train its components in a shared representation space.

This chapter follows the interfaces. It starts with paired image-text training, then compares connectors that let a language model consume visual features. It next separates image conditioning from image generation, extends the accounting to video, and ends with the independent choices hidden by the phrase “unified multimodal model.”

Separate the design decisions

Four questions keep the architecture legible:

A continuous image latent is not the same thing as a discrete image token. Early fusion is not the same thing as end-to-end product integration. Keeping these axes separate prevents an improvement on one boundary from being credited to the whole system.

Learn paired image-text geometry

contrastive language-image pretraining (CLIP) is the paired image-text training recipe that made open-vocabulary visual recognition practical (Radford et al. 2021). Given a batch of (B) matched image-text pairs, let (v_i) and (t_i) be unit-normalized embeddings of image (i) and text (i). Their scaled similarity is

$$s_{ij}=\frac{v_i^\top t_j}{\tau},$$

where (\tau>0) is a learned temperature. CLIP applies cross-entropy in both directions:

$$\mathcal{L}_{\mathrm{CLIP}} = -\frac{1}{2B}\sum_{i=1}^{B} \left[ \log\frac{\exp(s_{ii})}{\sum_{j=1}^{B}\exp(s_{ij})} + \log\frac{\exp(s_{ii})}{\sum_{j=1}^{B}\exp(s_{ji})} \right].$$

The first term asks image (i) to retrieve its paired text; the second asks text (i) to retrieve its paired image. Every other item in the batch acts as a negative, although an off-diagonal pair can still be a valid semantic match. CLIP trained this objective on 400 million web image-text pairs. At inference, class names written as prompts become text embeddings, and an image is classified by similarity to those prompts. Prompt wording and the pretraining distribution therefore remain part of the classifier.

SigLIP changes the batch objective, not the meaning of the embeddings. It treats each image-text pairing as an independent positive or negative logistic example, so it does not require a batch-wide softmax normalization (Zhai et al. 2023). That changes distributed training and small-batch behavior. It does not establish that image and text embeddings have identical distributions. Studies of CLIP-style spaces find a modality gap, with the two modalities occupying separated regions even when matched pairs have useful cosine similarity (Liang et al. 2022). CLIP aligns paired examples; it does not force the two modalities to occupy identical distributions.

That geometric gap is different from the connector's interface mismatch. CLIP's retrieval head produces one global embedding, while a visual language model often consumes a grid of spatial features from an earlier encoder layer. The connector must match that grid's width, length, and semantics to the language model. It need not make the two global embedding distributions identical.

Connect vision to a language model

An aligned encoder supplies visual features, but a language model still needs a bridge into its hidden space. Three influential systems place that bridge at different depths:

Flamingo freezes a pretrained vision encoder and language model, resamples a variable visual sequence to 64 outputs, and inserts newly trained gated cross-attention layers through the language model (Alayrac et al. 2022). The gates begin at zero, so the added path initially leaves the language model unchanged. Because the language stream can repeatedly attend to visual features, Flamingo can process interleaved images, video, and text.

BLIP-2 also starts from frozen towers, but its Querying Transformer, or Q-Former, uses 32 learned queries to extract a fixed-length summary from the vision encoder (Li et al. 2023). It is pretrained first for vision-language representation learning, then for vision-to-language generation. The fixed query count makes the interface cost predictable, while also creating an explicit information bottleneck.

LLaVA takes a shallower route. Its original connector was one learned linear projection from CLIP patch features to the language model's embedding width (Liu et al. 2023). LLaVA-1.5 replaced that layer with a two-layer MLP and used a 336-pixel CLIP encoder (Liu et al. 2024). In the alignment stage, the projection learns while the pretrained towers stay frozen. During instruction tuning, the language model and projection are updated while the vision encoder remains frozen. A connector name alone therefore does not say which weights train.

Count the visual sequence

For an image padded to height (H) and width (W), divided into square patches of edge (P), a single-crop vision transformer (ViT) emits approximately

$$N_{\mathrm{vis}} = \left\lceil\frac{H}{P}\right\rceil \left\lceil\frac{W}{P}\right\rceil$$

patch positions, before any class token, pooling, or token dropping. Here (N_{\mathrm{vis}}) is the visual sequence length. A 336 by 336 image with 14-pixel patches produces (24\times24=576) patch positions, which is the LLaVA-1.5 case (Liu et al. 2024).

Tiling changes the formula from one grid to a sum of grids. LLaVA-NeXT combines several 336-pixel tiles with a global view (Liu et al. 2024). Qwen2-VL instead maps native-resolution images and video to variable-length visual sequences, merges each two-by-two group of spatial features, and extends rotary positions across temporal, height, and width axes (Wang et al. 2024). These designs help with documents, charts, and small objects, but they also enlarge prefill work and the key-value cache. In standard dense attention, prefill attention over the combined text and visual prefix has a quadratic term in sequence length; each generated text token then reads a cache whose size grows linearly with that prefix. Visual token count is a serving decision, not merely an encoder detail.

Visual instruction tuning supplies the behavioral layer. LLaVA first trained its projector on about 595,000 image-caption pairs from CC3M, with both pretrained towers frozen. It then used a language-only GPT-4 to generate about 158,000 instruction examples from textual descriptions of images, including captions and bounding boxes (Liu et al. 2023). GPT-4 did not inspect the pixels, although the student later trained against the corresponding images. The resulting data taught conversation formats and task behavior after the projection had been aligned. This distinction matters: paired pretraining learns correspondence, connector training learns an interface, and instruction tuning teaches how to answer. Changing one stage does not replace the others.

Guide an image generator

Understanding an image and generating one use different outputs. Text-to-image diffusion first encodes the prompt, then denoises pixels or continuous latents conditioned on that representation. Classifier-free guidance controls how strongly sampling follows the condition (Ho and Salimans 2022). During training, the model sometimes receives an empty condition. At sampling time, one common convention is

$$\widehat{\epsilon}_\theta(z_t,c) = \epsilon_\theta(z_t,\varnothing) + s\left[ \epsilon_\theta(z_t,c)-\epsilon_\theta(z_t,\varnothing) \right].$$

Here (z_t) is the noisy state at step (t), (c) is the text condition, (\varnothing) is the empty condition, and (\epsilon_\theta) is the model's noise prediction. Under this convention, (s=0) gives the unconditional prediction, (s=1) gives the ordinary conditional prediction, and (s>1) extrapolates past it. Papers and libraries sometimes reparameterize the same line, so a reported scale is meaningful only with its equation.

Larger (s) often improves prompt adherence or the apparent quality of selected samples while reducing diversity and sometimes exaggerating artifacts. Guidance scale is not a confidence score. It also commonly requires conditional and unconditional model evaluations, even if an implementation batches them together.

Choose an image representation and objective

Image generators differ first in the space they model:

Imagen illustrates pixel-space cascades: it generates 64-pixel images, then uses separate diffusion super-resolution stages for 256 and 1024 pixels (Saharia et al. 2022). Its experiments found more benefit from scaling the frozen T5 text encoder than from scaling the image diffusion model within the studied range. Latent Diffusion instead trains an autoencoder and runs the denoising process in its continuous latent grid (Rombach et al. 2022). SDXL enlarged that route with a 2.6-billion-parameter UNet, two text encoders, size and crop conditioning, and an optional refinement stage (Podell et al. 2024). DALL-E 2 uses another factorization: a prior maps text to a CLIP image embedding, and a diffusion decoder renders an image conditioned on that embedding (Ramesh et al. 2022).

Data quality can dominate these architectural differences. DALL-E 3 trained a captioner to replace terse web text with detailed synthetic descriptions, then mixed those recaptions into generator training (Betker et al. 2023). The paper reports improved prompt following as the share of synthetic captions increased in its ablations. That is evidence for better supervision in that setup, not a claim that synthetic labels are always better.

Rectified flow changes the training target. With the data-to-noise convention

$$x_t=(1-t)x_0+t\epsilon, \qquad u=\epsilon-x_0, \qquad \mathcal{L}_{\mathrm{RF}} = \mathbb{E}\left[\left\lVert v_\theta(x_t,t,c)-u\right\rVert_2^2\right].$$

Here, (x_0) is sampled from the data distribution, (\epsilon) is Gaussian noise, (t\in[0,1]) locates a point on the straight conditional path, (c) is the text condition, (u) is the target velocity for that endpoint pair, and (v_\theta) is the learned velocity field. Generation numerically integrates that learned field from noise toward data. Averaging velocities from many endpoint pairs can still produce a curved marginal flow, so the straight training paths do not guarantee one-step sampling. Stable Diffusion 3 combined rectified flow, sampling weights biased toward intermediate noise levels, and MMDiT, a transformer with separate text and image parameters that allows bidirectional interaction between their token streams (Esser et al. 2024). FLUX.1 later used a 12-billion-parameter hybrid of multimodal and parallel diffusion-transformer blocks trained with flow matching, according to its vendor announcement (Black Forest Labs 2024). Flow matching changes the objective, but the sampler, number of function evaluations, autoencoder, and hardware still determine speed.

The discrete route converts an image grid to codebook indices. VQGAN couples a learned quantizer to an autoregressive transformer (Esser et al. 2021), while ViT-VQGAN improves the quantizer and Parti scales text-conditioned prediction of those indices to 20 billion parameters (Yu et al. 2022; Yu et al. 2022). VAR predicts progressively finer token maps rather than scanning one flattened image left to right (Tian et al. 2024). Its reported speed and quality comparisons are class-conditional ImageNet results, so they establish an alternative schedule, not a general text-to-image winner. OpenAI later classified GPT-4o's native image generation as autoregressive, but did not publish its tokenizer, factorization, or decoder design (OpenAI 2025). A latent is not automatically a token: continuous grids support diffusion or flow, while discrete indices support categorical prediction.

Compress space and time for video

A video contains (T) frames of height (H) and width (W). Suppose an autoencoder compresses the temporal and spatial axes by ((d_t,d_h,d_w)), and the transformer groups the latent grid into patches of ((p_t,p_h,p_w)). Its sequence length is approximately

$$N_{\mathrm{video}} = \left\lceil\frac{T}{d_t p_t}\right\rceil \left\lceil\frac{H}{d_h p_h}\right\rceil \left\lceil\frac{W}{d_w p_w}\right\rceil.$$

Here (N_{\mathrm{video}}) counts transformer positions; (d_t,d_h,d_w) are autoencoder compression factors; and (p_t,p_h,p_w) are transformer patch sizes measured in latent cells. Channels affect the width of each position, not this sequence count. The formula is accounting, not a quality estimate.

The runnable reconstructs Movie Gen's published maximum video context: 16 seconds at 16 frames per second and 768-pixel square frames. Its temporal autoencoder compresses each axis by eight, and its transformer groups two-by-two spatial latent cells into one position.

frames, height, width = 16 * 16, 768, 768
compression = (8, 8, 8)
patch = (1, 2, 2)

latent = tuple(
    size // stride
    for size, stride in zip((frames, height, width), compression)
)
tokens = (
    latent[0] // patch[0]
    * latent[1] // patch[1]
    * latent[2] // patch[2]
)
raw_positions = frames * height * width
latent_positions = latent[0] * latent[1] * latent[2]

print(f"raw spacetime positions: {raw_positions:,}")
print(f"latent positions:        {latent_positions:,}")
print(f"transformer tokens:      {tokens:,}")
print(f"position reduction:      {raw_positions // tokens:,}x")

An early video-diffusion design extended image models with space-time-factorized 3D UNets (Ho et al. 2022). Diffusion Transformers, or diffusion transformer (DiT) models, instead apply transformer blocks to latent patches, with the original DiT study showing quality improvements as model compute increased in its ImageNet experiments (Peebles and Xie 2023). Sora's 2024 technical report described a diffusion transformer over spacetime latent patches and variable video shapes, but disclosed too little to reproduce the system (Brooks et al. 2024).

Open papers make the compression boundary more concrete. CogVideoX reports a causal 3D autoencoder with fourfold temporal and eightfold compression on each spatial axis, followed by a text-video diffusion transformer (Yang et al. 2024). Movie Gen reports a 30-billion-parameter flow-matching video transformer over a temporal autoencoder and uses a separate audio model to produce sound conditioned on video (Polyak and others 2024). Synchronized output does not therefore imply that frames and audio came from one joint sampler. Compression changes representation cost, not semantic understanding; physical consistency is a separate evaluation problem taken up in Chapter 16.

Keep integration and representation separate

Modularity and representation are separate decisions. A product can call a specialized image generator from a language model, or one trained backbone can alternate text and images. Either system may use discrete codes, continuous latents, or both.

Chameleon is an early-fusion discrete model. It interleaves text tokens and image codebook indices and trains one transformer with next-token prediction (Chameleon Team 2024). Its 7B and 34B training recipes used stability controls that included query-key normalization and an output z-loss; the paper's ablations found query-key normalization essential in that setup. Those results do not prove that every unified model requires the same recipe.

Transfusion keeps image patches continuous. One transformer applies next-token loss to text positions and a diffusion loss to image positions (Zhou et al. 2025). In the paper's controlled comparisons, continuous representations reached better image-generation results than quantized ones with less training compute. The result isolates a representation choice under that experimental setup, not a universal comparison with every tokenizer. Show-o combines autoregressive text with discrete masked diffusion for images (Xie et al. 2025), while Emu3 applies next-token prediction to discrete text, image, and video sequences (Wang and others 2024). A unified model can still use different losses, and a single loss can still operate over modality-specific tokenizers.

At the system level, modular components remain attractive when they need separate release cycles, safety policies, latency budgets, or provenance controls. End-to-end training can preserve signals that a narrow interface discards, but it also makes failures harder to localize. OpenAI described GPT-4o as one network trained across text, vision, and audio (OpenAI 2024). Public information does not reveal its internal representations, so that announcement cannot settle the discrete-versus-continuous question.

Benchmark the boundary you intend to ship

“Multimodal quality” is too broad to be a useful metric. Test each interface:

MMMU contains 11,500 college-level questions across 30 subjects and tests both visual perception and expert knowledge (Yue et al. 2024). MMMU-Pro removes some questions answerable without the image, expands four answer choices to ten, and adds a vision-only condition to reduce shortcutting (Yue et al. 2024). A score from either benchmark must name the model version, prompting protocol, and use of tools.

For generation, GenEval uses detectors and classifiers to score object presence, count, color, position, and attribute binding (Ghosh et al. 2023). DPG-Bench uses dense prompts decomposed into many semantic propositions (Hu et al. 2024). These automated judges are useful diagnostics, not complete measures of aesthetics, diversity, typography, memorization, or harmful output. Benchmark the boundary you intend to ship, then add end-to-end task and human evaluation.

Further reading