Diffusion and Flow Matching

Part II begins where next-token prediction stops being the natural generative order. Image, audio, and video models mostly do not produce one item after another from left to right; they start from noise and remove it. diffusion, the train-by-adding-noise and generate-by-denoising recipe, and its flow matching generalization are the generative principle behind almost all non-text media, and they are now reaching back into text. The central shift is simple but deep: generation becomes a path from disorder to structure, with the same network readable as denoiser, score, or velocity. EDM and flow matching are two ways of making that path better conditioned, and the frontier pressure is to collapse a long denoising chain, sometimes a thousand evaluations, toward one.

Why not autoregression

Autoregressive generation is sequential by construction: one token after another, left to right, a serial chain whose length is the output. For text that ordering is natural. For an image it is not: pixels have no canonical order, and forcing one wastes the structure. Diffusion takes a different route. Corrupt data into noise by a fixed, known process, then learn to reverse it, denoising the whole signal in parallel at each step. There is no left-to-right, the output size is fixed up front, and the model can attend to the entire partially-denoised signal at once. The cost moves from sequence length to step count, and most of this chapter is about driving that step count down.

The forward and reverse processes

The forward process gradually adds Gaussian noise to a datum over $T$ steps until little but noise remains, a fixed Markov chain (each step depends only on the one before it) with no parameters. Gaussian composition gives its marginal, the distribution of $x_t$ on its own with the intervening steps summed out, in closed form, so any noise level is reachable in one shot,

$$q(x_t \mid x_0) = \mathcal{N}\!\left(x_t;\, \sqrt{\bar{\alpha}_t}\,x_0,\; (1-\bar{\alpha}_t)\,\mathbf{I}\right), \qquad x_t = \sqrt{\bar{\alpha}_t}\,x_0 + \sqrt{1-\bar{\alpha}_t}\,\boldsymbol{\epsilon},$$

where $\bar{\alpha}_t$ is the surviving signal fraction. Generation runs the chain backwards, learning to undo one noising step at a time. The surprise of denoising diffusion probabilistic models (DDPM), the original denoising-diffusion objective, is that the variational bound on this reverse process collapses, once the time-dependent weights are dropped, to an ordinary regression: predict the noise $\boldsymbol{\epsilon}$ that was added at a randomly chosen step (Ho et al. 2020),

$$\mathcal{L}_{\text{simple}} = \mathbb{E}_{\,t,\,\mathbf{x}_0,\,\boldsymbol{\epsilon}} \left\lVert \boldsymbol{\epsilon} - \boldsymbol{\epsilon}_\theta\!\left( \sqrt{\bar{\alpha}_t}\,\mathbf{x}_0 + \sqrt{1-\bar{\alpha}_t}\,\boldsymbol{\epsilon},\; t \right) \right\rVert^2.$$

Two design choices inside this loop turned out to matter more than they look. The first is the noise schedule. The original linear $\beta_t$ destroys information too fast at high resolution, and a cosine schedule that keeps more signal late in the chain, $\bar{\alpha}_t = \cos^2\!\big(\frac{t/T + s}{1+s}\cdot \frac{\pi}{2}\big)$ with a small offset $s$, improves likelihood and sample quality, alongside learning the reverse variances rather than fixing them (Nichol and Dhariwal 2021). The second is what the network predicts. The same target can be expressed three ways, and they are not numerically equal across the schedule: predicting $\boldsymbol{\epsilon}$ degenerates at high noise where the target carries little signal, predicting $x_0$ degenerates at low noise, and the $v$-prediction $v = \sqrt{\bar{\alpha}_t}\,\boldsymbol{\epsilon} - \sqrt{1-\bar{\alpha}_t}\,x_0$ stays balanced across the whole range, which is why it became the parameterization of choice once distillation made each step span a wide band of signal-to-noise (Salimans and Ho 2022). The toy below makes the loop concrete: for one-dimensional Gaussian data the optimal noise-predictor is analytic, so we can sample from noise and watch the distribution come back.

import numpy as np
rng = np.random.default_rng(0)

mu, v0 = 3.0, 1.0                          # data ~ N(mu, v0)
T = 1000
beta = np.linspace(1e-4, 0.02, T)         # noise schedule
alpha = 1 - beta
abar = np.cumprod(alpha)                   # surviving signal fraction

def eps_star(x, t):                        # optimal epsilon-prediction (analytic)
    var = abar[t] * v0 + (1 - abar[t])
    return np.sqrt(1 - abar[t]) * (x - np.sqrt(abar[t]) * mu) / var

x = rng.normal(size=5000)                  # start from pure noise
for t in range(T - 1, -1, -1):
    mean = (x - beta[t] / np.sqrt(1 - abar[t]) * eps_star(x, t)) / np.sqrt(alpha[t])
    x = mean + (np.sqrt(beta[t]) * rng.normal(size=x.shape) if t > 0 else 0.0)

print(f"data N(mu={mu}, var={v0});  samples: mean={x.mean():.2f}, var={x.var():.2f}")

One framework: scores and SDEs

This section reaches the same model from the mathematics of random processes, where the denoiser becomes an arrow at every noise level pointing toward where data is dense and generation becomes a differential equation that carries a sample from noise back to data; it is the most mathematical stretch of the chapter and can be skimmed. A second lineage arrived at the same place from a different direction. Score-based models learn the score, the gradient of the log-density $\nabla_x \log p(x)$, at many noise levels and sample by Langevin dynamics, repeatedly nudging a sample along the score and adding a little random noise each step until it settles into the distribution (Song and Ermon 2019). The link to the denoiser above is exact and worth seeing: by Tweedie's formula the optimal noise predictor is the score up to a scale, $\nabla_{x_t} \log q_t(x_t) = -\boldsymbol{\epsilon}_\theta(x_t, t) / \sqrt{1-\bar{\alpha}_t}$, and the theorem that licenses training a score model by denoising rather than by the intractable marginal is older than diffusion itself (Vincent 2011). Song et al. then showed the two lineages are one object: the forward process is a stochastic differential equation (SDE) carrying data to noise, generation is the reverse-time SDE, whose drift contains exactly that score (Anderson 1982), and both DDPM (a variance-preserving SDE) and score matching (a variance-exploding one) are discretizations of it (Song et al. 2021). The same work derives an equivalent deterministic probability-flow ordinary differential equation (ODE),

$$\frac{dx}{dt} = f(x, t) - \tfrac{1}{2}\,g(t)^2\,\nabla_x \log p_t(x),$$

with the same marginals as the SDE. That ODE is the bridge to everything fast: the deterministic samplers below, and flow matching at the end of the chapter.

The recipe, put in order

For a few years the design space was a thicket of schedules, parameterizations, and scalings, each paper choosing differently. EDM cut through it by writing every diffusion model as one denoiser $D_\theta(x; \sigma)$ at a continuous noise level $\sigma$ and separating three choices that had been tangled together (Karras et al. 2022). Preconditioning wraps the raw network so its input and output stay unit-variance at every noise level, $D_\theta(x;\sigma) = c_{\text{skip}}( \sigma)\,x + c_{\text{out}}(\sigma)\,F_\theta(c_{\text{in}}(\sigma)\,x;\, c_{\text{noise}}(\sigma))$, which makes the loss weight uniform across $\sigma$ and removes a class of tuning. The noise levels are sampled from a log-normal during training and laid on a schedule $\sigma_i = (\sigma_{\max}^{1/\rho} + \frac{i}{N-1}(\sigma_{\min}^{1/\rho} - \sigma_{\max}^{1/\rho}))^{\rho}$ with $\rho = 7$ at sampling, and the probability-flow ODE is integrated with a second-order Heun step rather than Euler, reaching strong quality in about thirty-five evaluations. The lasting lesson is the disentangling: schedule, preconditioning, and sampler are orthogonal axes, and naming them separately is what let the rest of the field stop re-deriving the same choices.

Making it cheap and steerable

Three more moves turned a slow research method into a deployed one. Fewer steps came first: DDIM is a non-Markovian sampler that reuses the trained model but, at its deterministic setting, integrates the probability-flow ODE directly, reaching comparable quality 10 to 50 times faster (Song et al. 2021), and dedicated high-order solvers go further by analytically handling the linear part of that ODE in log-signal-to-noise coordinates, cutting good samples to roughly ten to twenty network evaluations (Lu et al. 2022). Cost fell next. An autoencoder is a network that compresses an image to a small vector and reconstructs it; latent diffusion runs the whole process in that compressed latent space of a pretrained autoencoder rather than on pixels, cutting compute by a large factor, and injects text or layout through cross-attention, the recipe behind Stable Diffusion (Rombach et al. 2022). Control, the third move, took two forms. Classifier guidance steers a sample with the gradient of a separately trained noised classifier (Dhariwal and Nichol 2021), and classifier-free guidance (CFG) removes that classifier by training one model both with and without the conditioning signal and extrapolating between them at sampling,

$$\tilde{\boldsymbol{\epsilon}}_\theta(x_t, c) = (1 + w)\,\boldsymbol{\epsilon}_\theta(x_t, c) - w\,\boldsymbol{\epsilon}_\theta(x_t),$$

trading diversity for fidelity with a single weight $w$ (Ho and Salimans 2022). Progressive distillation then halves the step count repeatedly by training a student so one of its steps matches two of a deterministic teacher's, which is exactly where $v$-prediction earns its keep (Salimans and Ho 2022). The backbone caught up with the rest of the book last: diffusion transformer (DiT) replaces the U-Net, the convolutional network shape diffusion models originally used, with a transformer over latent patches and finds sample quality improving as compute rises, putting diffusion on the same scaling curve as everything else (Peebles and Xie 2023).

Flow matching: learn a velocity, not a denoiser

Flow matching reframes the problem in the cleanest terms the field has found. Think of generation as an ODE $dx/dt = v_t(x)$ that transports a noise distribution into the data distribution, and recall that a velocity field generates a given path of densities exactly when it satisfies the continuity equation $\partial_t p_t + \nabla \cdot (p_t v_t) = 0$, mass transported, never created. The catch is that the marginal velocity is an intractable average over all data points. The resolution is the conditional flow matching objective: regress the network onto the conditional field that transports noise to one fixed data point $x_1$,

$$\mathcal{L}_{\text{CFM}} = \mathbb{E}_{t,\,x_1,\,x_t} \left\lVert v_\theta(x_t, t) - u_t(x_t \mid x_1) \right\rVert^2,$$

Here $t$ chooses a point along the path, $x_1$ is a data point, $x_t$ is the intermediate sample on the path to it, $v_\theta$ is the network's velocity, and $u_t(\cdot \mid x_1)$ is the analytic velocity of that conditional path. The loss asks for the same thing a denoising loss asks for in diffusion, but in the language of motion: at every noisy point, point in the direction that would carry this sample toward data. Lipman et al. prove it has the same gradient as regressing the intractable marginal field, because the marginal target is itself the conditional expectation of the per-sample targets (Lipman et al. 2023). Training becomes simulation-free: sample a data point, sample a point on the path to it, regress onto a closed-form velocity, never integrate the ODE. Diffusion paths fall out as one choice of conditional path; straight, constant-speed optimal-transport paths fall out as a more efficient other.

That straight-path choice is the lever. rectified flow learns the velocity of the linear interpolant $x_t = (1-t)\,x_0 + t\,x_1$, whose target is the constant $x_1 - x_0$, and adds a "reflow" step that re-couples noise to the data the model itself generates and retrains, straightening the trajectories further; a straight trajectory is integrated exactly by a single Euler step, which is the path to one-step sampling (Liu et al. 2023). Stochastic interpolants are the umbrella over all of this, showing that one fixed interpolation between noise and data yields both a deterministic transport and a noisy diffusion as a free choice, so flow matching, rectified flow, and diffusion are reframings of a single construction (Albergo and Vanden-Eijnden 2023; Albergo et al. 2023).

The last step is to collapse the path entirely. Consistency models train a network with a self-consistency property, that every point on a probability-flow ODE trajectory maps to the same origin, so one forward pass jumps from noise to data; it can be distilled from a diffusion teacher or trained standalone (Song et al. 2023). The same one-step target is reached by other routes: distilling in latent space (Luo et al. 2023), matching the output distribution of a multi-step teacher rather than its per-trajectory points (Yin et al. 2024), or adding an adversarial loss to reach few-step photorealism (Sauer et al. 2024). A successor line drops the diffusion teacher altogether: shortcut models, inductive moment matching, and MeanFlow train few- and one-step generators from scratch, the last by exploiting an identity between the average velocity over an interval and the instantaneous velocity that flow matching learns (Geng et al. 2025). The arc across this section is a single quantity coming down, the number of sequential network evaluations a sample costs, from a thousand for DDPM toward one.

The loop back to language

Diffusion is now returning to text. The discrete analogue replaces Gaussian noise with a categorical corruption: D3PM diffuses over a transition matrix, and its absorbing-state kernel, which sends each token to a [MASK] with growing probability, is the bridge that makes a masked language model a diffusion model (Austin et al. 2021). SEDD carries the score idea across by learning ratios of the data distribution rather than a Gaussian score, reaching autoregressive quality at GPT-2 scale (Lou et al. 2024). LLaDA scales the masked-denoising objective to train a language model from scratch, predicting masked tokens by a reverse process rather than left to right, and reports it competitive with an autoregressive LLaMA3-8B at in-context learning (Nie et al. 2025); commercial diffusion language models have since appeared, trading some quality for parallel decoding speed. Chapter 13 takes up this thread in full; here it is enough that the denoising principle has come back to the modality this book began with.

Further reading