Diffusion and Flow Matching

Part I followed models that assign probability by predicting the next token. Part II begins with another choice: define a simple corruption or transport path between data and a known reference distribution, then learn how to travel back to data. diffusion, the train-by-adding-noise and generate-by-denoising recipe, is a major model family for image, audio, and video generation. flow matching trains a continuous transport velocity instead. The two families overlap mathematically, but neither is a universal replacement for the other or for autoregression.

The important change is where sequential dependence lives. An autoregressive model waits for the previous output position. A diffusion or flow sampler can update all positions in its current state together, but it repeats that full network evaluation over several noise levels or solver steps. Each evaluation still pays for the representation's size and the model's architecture. Fewer sampling steps help only when quality and per-step cost are held in view.

Choose a factorization, not a camp

For an ordered representation $x=(x_1,\ldots,x_n)$, where $x$ denotes the complete output, autoregression writes

$$p(x)=\prod_{i=1}^{n}p(x_i\mid x_{<i}).$$

Here, $x_i$ is output position $i$, $x_{<i}$ is every earlier position, and $p$ is the modeled probability distribution. This factorization gives text a natural streaming interface and lets an end-of-sequence symbol choose the output length. Images and audio can also be serialized into tokens, so the issue is not that they lack any order. It is that no single serialization is always the best inductive bias for their spatial or temporal structure.

Diffusion and flow models choose a state shape first, then revise that state as a whole. That makes editing, infilling, and global coordination natural, but usually requires a fixed or separately predicted shape. The contrast is operational:

These are tendencies, not boundaries. Blockwise and speculative methods mix them, while latent representations change what one “position” costs.

Build the discrete diffusion process

Diffusion probabilistic models were introduced by Sohl-Dickstein et al. in 2015 as a learned reversal of a gradual noising process inspired by nonequilibrium thermodynamics (Sohl-Dickstein et al. 2015). The widely used denoising diffusion probabilistic models (DDPM), Ho et al.'s 2020 denoising-diffusion formulation, chooses a prescribed Gaussian Markov chain (Ho et al. 2020):

$$q(x_t\mid x_{t-1}) =\mathcal N\!\left(x_t;\sqrt{\alpha_t}\,x_{t-1},\beta_t\mathbf I\right), \qquad \alpha_t=1-\beta_t, \qquad \bar\alpha_t=\prod_{s=1}^{t}\alpha_s.$$

Here, $x_0$ is a data sample; $x_t$ is its corrupted state at step $t\in\{1,\ldots,T\}$; $q$ is the fixed forward process; $\beta_t\in(0,1)$ is the variance added at step $t$; $\alpha_t$ is the retained fraction for that step; $\bar\alpha_t$ is the cumulative retained fraction; $\mathbf I$ is the identity covariance; and $\mathcal N(x;m,C)$ is a Gaussian density over $x$ with mean $m$ and covariance $C$. The schedule has hyperparameters, but no learned parameters.

Gaussian composition makes the marginal at any step available without running the intervening chain:

$$q(x_t\mid x_0) =\mathcal N\!\left(x_t;\sqrt{\bar\alpha_t}\,x_0, (1-\bar\alpha_t)\mathbf I\right), \qquad x_t=\sqrt{\bar\alpha_t}\,x_0 +\sqrt{1-\bar\alpha_t}\,\epsilon, \quad \epsilon\sim\mathcal N(0,\mathbf I).$$

Here, $\epsilon$ is fresh standard Gaussian noise. If $\bar\alpha_T$ is close to zero, the terminal marginal $q(x_T)$ is approximately standard normal. It is not exactly standard normal for an arbitrary finite schedule.

Generation needs a learned reverse transition,

$$p_\theta(x_{t-1}\mid x_t) =\mathcal N\!\left(x_{t-1};\mu_\theta(x_t,t), \Sigma_\theta(x_t,t)\right),$$

where $p_\theta$ is the model with parameters $\theta$, $\mu_\theta$ is its reverse mean, and $\Sigma_\theta$ is a fixed or learned reverse covariance. In the common noise-prediction parameterization, the mean is

$$\mu_\theta(x_t,t) =\frac{1}{\sqrt{\alpha_t}} \left(x_t-\frac{\beta_t}{\sqrt{1-\bar\alpha_t}} \epsilon_\theta(x_t,t)\right).$$

Here, $\epsilon_\theta(x_t,t)$ estimates the noise component in $x_t$. Ho et al. derive a variational objective and then use a practical reweighted version that drops its timestep-dependent coefficients:

$$\mathcal L_{\mathrm{simple}} =\mathbb E_{t,x_0,\epsilon} \left[ \left\|\epsilon- \epsilon_\theta\!\left( \sqrt{\bar\alpha_t}x_0+\sqrt{1-\bar\alpha_t}\epsilon,t \right)\right\|_2^2 \right].$$

In this expectation, $x_0$ comes from the data, $t$ is sampled from the training steps, and $\epsilon$ is standard Gaussian noise. The loss teaches the mean needed for the reverse transition; the reverse variance remains a separate design choice.

The runnable example removes the neural approximation so the reversal itself can be checked. For one-dimensional Gaussian data, every forward marginal and reverse conditional is analytic. Starting from the exact terminal marginal and applying those reverse conditionals recovers the original distribution.

import numpy as np

rng = np.random.default_rng(0)
data_mean, data_variance = 3.0, 1.0
steps = 1000
beta = np.linspace(1e-4, 0.02, steps)
alpha = 1.0 - beta
alpha_bar = np.cumprod(alpha)


def marginal(alpha_bar_t):
    """Mean and variance of q(x_t) for Gaussian data."""
    mean = np.sqrt(alpha_bar_t) * data_mean
    variance = alpha_bar_t * data_variance + (1.0 - alpha_bar_t)
    return mean, variance


# Start from q(x_T), which is nearly N(0, 1) for this schedule.
terminal_mean, terminal_variance = marginal(alpha_bar[-1])
x = rng.normal(terminal_mean, np.sqrt(terminal_variance), size=10_000)

for t in range(steps - 1, -1, -1):
    previous_alpha_bar = alpha_bar[t - 1] if t > 0 else 1.0
    previous_mean, previous_variance = marginal(previous_alpha_bar)
    current_mean, current_variance = marginal(alpha_bar[t])

    # Exact Gaussian conditional q(x_{t-1} | x_t).
    gain = np.sqrt(alpha[t]) * previous_variance / current_variance
    reverse_mean = previous_mean + gain * (x - current_mean)
    reverse_variance = previous_variance - gain**2 * current_variance
    x = reverse_mean + np.sqrt(max(reverse_variance, 0.0)) * rng.normal(size=x.shape)

print(f"terminal signal fraction: {alpha_bar[-1]:.6f}")
print(f"target mean={data_mean:.2f}, variance={data_variance:.2f}")
print(f"recovered mean={x.mean():.2f}, variance={x.var():.2f}")

Keep schedules and prediction targets distinct

The noise schedule decides which signal-to-noise ratios receive training and sampling effort. Nichol and Dhariwal found the original linear schedule suboptimal for their $32\times32$ and $64\times64$ experiments because its late steps were already almost pure noise. Their cosine schedule, where cumulative signal follows a normalized squared cosine, is (Nichol and Dhariwal 2021)

$$f(t)=\cos^2\!\left( \frac{t/T+s}{1+s}\frac{\pi}{2} \right), \qquad \bar\alpha_t=\frac{f(t)}{f(0)}, \qquad s=0.008.$$

Here, $T$ is the number of noising steps, $t$ is the step index, $s$ is the paper's small offset, and $f(0)$ normalizes the cumulative signal fraction to one. This is one documented schedule, not a universal best setting.

The network target is another choice. Let

$$a_t=\sqrt{\bar\alpha_t}, \qquad \sigma_t=\sqrt{1-\bar\alpha_t}, \qquad x_t=a_t x_0+\sigma_t\epsilon.$$

Here, $a_t$ is the signal coefficient and $\sigma_t$ is the noise coefficient. The model can predict $\epsilon$, $x_0$, or the variance-preserving target

$$v_t=a_t\epsilon-\sigma_t x_0, \qquad x_0=a_t x_t-\sigma_t v_t, \qquad \epsilon=\sigma_t x_t+a_t v_t.$$

Here, $v_t$ contains the same denoising information for this specified path. Recovering $x_0$ from an $\epsilon$ prediction divides by $a_t$ and becomes ill-conditioned at high noise. Recovering $\epsilon$ from an $x_0$ prediction divides by $\sigma_t$ and becomes ill-conditioned at low noise. The conversions from $v_t$ keep bounded coefficients, which made it useful in progressive distillation (Salimans and Ho 2022). This $v_t$ is tied to the angular parameterization of a variance-preserving diffusion path. It is not, without a time conversion, the flow-matching velocity introduced later.

Connect denoising to scores, SDEs, and ODEs

A score is a spatial derivative of log density. At noise level $t$, define

$$s_t(x)=\nabla_x\log p_t(x).$$

Here, $p_t$ is the marginal density of the noisy data, $x$ is a point in its state space, $\log$ is the natural logarithm, and $\nabla_x$ differentiates with respect to $x$. Noise prediction and score prediction coincide only at the population optimum:

$$\epsilon^*(x_t,t)=\mathbb E[\epsilon\mid x_t], \qquad s_t(x_t)=-\frac{\epsilon^*(x_t,t)}{\sigma_t}.$$

Here, $\epsilon^*$ is the conditional mean of the injected noise and $\sigma_t=\sqrt{1-\bar\alpha_t}$. A trained network approximates this relation; an arbitrary $\epsilon_\theta$ is not the exact score. This denoising-score connection predates DDPM (Vincent 2011) and underlies noise-conditional score networks sampled with annealed Langevin dynamics (Song and Ermon 2019).

Continuous time puts the common constructions in one stochastic framework. A forward stochastic differential equation (SDE) can be written as

$$dx=f(x,t)\,dt+g(t)\,dW_t.$$

Here, $f(x,t)$ is the drift, $g(t)$ is a scalar diffusion coefficient, $dt$ is an infinitesimal time increment, and $W_t$ is standard Brownian motion. Under the conditions used by Song et al., the reverse-time process is (Anderson 1982; Song et al. 2021)

$$dx=\left[f(x,t)-g(t)^2\nabla_x\log p_t(x)\right]dt +g(t)\,d\bar W_t, \qquad dt<0.$$

Here, $\bar W_t$ is Brownian motion when time is integrated from noise back to data. The DDPM perturbation converges to a variance-preserving SDE, while the perturbation used by score matching with Langevin dynamics converges to a variance-exploding SDE. Generic score matching is a training principle, not itself one of those discretizations.

The same exact score defines a deterministic probability-flow ordinary differential equation (ODE):

$$\frac{dx}{dt}=f(x,t)-\frac{1}{2}g(t)^2\nabla_x\log p_t(x).$$

Here, the symbols retain their SDE meanings. The ODE and SDE share the same one-time marginal densities under the exact score and regularity assumptions; they do not share individual stochastic paths or transition laws. A learned score makes the equivalence approximate.

Separate the model from the sampler

A deployed diffusion system is not identified by “uses diffusion.” At least six choices must be recorded separately:

Latent diffusion reduces the denoiser's state from pixels to a lower-resolution spatial tensor produced by a pretrained autoencoder; the encoder and final decoder sit outside the denoising chain. Cross-attention can inject text, boxes, or other conditions (Rombach et al. 2022). A latent bottleneck saves compute but can also discard detail, so the autoencoder reconstruction error belongs in the evaluation contract.

The backbone is independent of that representation choice. A U-Net uses multi-resolution convolutional blocks. diffusion transformer (DiT) replaces it with a transformer over latent patches. In the paper's class-conditional ImageNet experiments, increasing forward-pass compute through depth, width, or more patches correlated with lower FID (Peebles and Xie 2023). That is controlled evidence within one model family, not a universal scaling law.

Conditioning also changes inference cost. classifier-free guidance (CFG) trains a single model with the condition sometimes replaced by a null input, then combines two predictions at sampling (Ho and Salimans 2022):

$$\hat\epsilon_{\mathrm{cfg}}(x_t,c) =\epsilon_\theta(x_t,\varnothing) +w\left[ \epsilon_\theta(x_t,c)-\epsilon_\theta(x_t,\varnothing) \right].$$

Here, $c$ is the condition; $\varnothing$ is the null condition used during dropout training; $\epsilon_\theta(x_t,c)$ and $\epsilon_\theta(x_t,\varnothing)$ are the conditional and unconditional noise predictions; and $w$ is guidance strength. With this convention, $w=1$ returns the ordinary conditional prediction and larger values extrapolate away from the unconditional result. The method removes a separate classifier, but normally requires both network predictions per sampling step, whether run separately or batched together.

Sampler step count and network-function evaluations (NFE) are not synonyms. A first-order step may use one denoiser call; a second-order method may use two. DDIM defines non-Markovian forward processes with the same DDPM training objective, and its deterministic setting produced 10 to 50 times lower wall-clock sampling time in the paper's experiments (Song et al. 2021). In the continuous limit and with an optimal predictor, its ODE matches a probability-flow ODE after reparameterization. A coarse finite DDIM update is not simply Euler's method on the usual time coordinate. DPM-Solver instead constructs high-order updates in a log signal-to-noise coordinate and reports strong samples in roughly 10 to 20 NFE for its tested models (Lu et al. 2022). Neither number transfers automatically to another model, resolution, or guidance setting.

EDM made this separation explicit for a broad class of Gaussian diffusion models (Karras et al. 2022). It combines input, output, skip, and noise preconditioning with a chosen training distribution over continuous noise levels. Its power-law sampling grid is

$$\sigma_i=\left( \sigma_{\max}^{1/\rho} +\frac{i}{N-1} \left[\sigma_{\min}^{1/\rho}-\sigma_{\max}^{1/\rho}\right] \right)^\rho, \qquad i=0,\ldots,N-1.$$

Here, $\sigma_i$ is noise level $i$; $\sigma_{\max}$ and $\sigma_{\min}$ are the finite endpoints; $N$ is the number of nonzero levels; and $\rho$ controls where the grid is dense. The paper used $\rho=7$ in reported settings and then added a final $\sigma_N=0$ step. These are paper-specific settings. EDM's reported 35-NFE result also belongs to its stated benchmark configuration, not to the definition of EDM.

Flow matching learns transport directly

Flow matching starts from a continuous normalizing flow rather than a reverse Markov chain. Let an ODE move samples as

$$\frac{dx}{dt}=v_t(x),$$

where $t\in[0,1]$ is path time, $x$ is the current state, and $v_t(x)$ is a time-dependent velocity field. A sufficiently regular density path $p_t(x)$ is transported by this field when it satisfies the continuity equation

$$\partial_t p_t(x)+\nabla_x\!\cdot\!\left(p_t(x)v_t(x)\right)=0.$$

Here, $\partial_t$ is the time derivative and $\nabla_x\cdot$ is divergence in state space. The equation conserves probability mass, but it does not select a unique velocity field by itself.

Marginal flow matching would regress against the velocity of $p_t$, which is usually unavailable. Conditional flow matching (CFM) chooses tractable conditional paths instead (Lipman et al. 2023):

$$\mathcal L_{\mathrm{CFM}}(\theta) =\mathbb E_{t,z,x} \left[ \left\|v_\theta(x,t)-u_t(x\mid z)\right\|_2^2 \right],$$

where $t\sim\mathcal U[0,1]$; $z$ identifies the sampled conditioning data for one path; $x\sim p_t(\cdot\mid z)$ is a point on that conditional path; $u_t(x\mid z)$ is its analytic conditional velocity; and $v_\theta$ is the learned marginal velocity. Under the paper's regularity and integrability conditions, the CFM and inaccessible marginal objectives have the same gradient with respect to $\theta$; their scalar losses need not be identical.

A simple paired linear path makes the training target concrete:

$$x_0\sim p_0, \qquad x_1\sim p_{\mathrm{data}}, \qquad x_t=(1-t)x_0+t x_1, \qquad u_t=x_1-x_0.$$

Here, $p_0$ is the reference distribution, usually Gaussian; $p_{\mathrm{data}}$ is the data distribution; and $(x_0,x_1)$ comes from a declared coupling, often independent sampling unless another coupling is constructed. The target $x_1-x_0$ is constant for one pair. The population regressor is instead

$$v^*(x,t)=\mathbb E[x_1-x_0\mid x_t=x].$$

Here, $v^*$ averages all pairwise targets that can pass through state $x$ at time $t$. Its marginal trajectories can therefore curve even when every conditional interpolation is straight. Flow-matching training is simulation-free because these training points and targets require no ODE solve; sampling still integrates the learned ODE.

The “optimal transport” phrase also needs scope. Lipman et al. study conditional Gaussian optimal-transport paths. That does not mean independent noise-data pairs form the globally optimal coupling between the two marginal distributions. Coupling choice, path choice, and solver choice are separate.

Straighten or collapse the sampling path

rectified flow uses the linear paired target above and can apply reflow: integrate the learned model, pair each reference sample with its generated endpoint, and train again on those new pairs (Liu et al. 2023). Reflow tends to straighten the model-induced coupling and the paper proves non-increasing convex transport costs. It does not guarantee the globally optimal data coupling. One Euler step is exact only for an exactly constant learned trajectory; empirical straightness makes coarse integration useful, not automatically exact.

Stochastic interpolants expose a broader shared construction (Albergo and Vanden-Eijnden 2023; Albergo et al. 2025):

$$X_t=I(t,X_0,X_1)+\gamma(t)Z.$$

Here, $I$ is a chosen interpolation satisfying the endpoint conditions; $X_0$ and $X_1$ are endpoint random variables; $Z$ is independent standard Gaussian noise; and $\gamma(t)$ controls extra stochasticity and vanishes at the endpoints. Under suitable regularity, one interpolant supports a deterministic ODE and families of SDEs. The SDE construction additionally needs a score and a chosen diffusion coefficient, so “unified” does not mean that every algorithm is interchangeable.

There are three distinct routes to fewer evaluations:

Progressive distillation repeatedly trains one student step to reproduce two deterministic teacher steps (Salimans and Ho 2022). Latent Consistency Models apply consistency distillation to a classifier-free-guided latent diffusion probability-flow ODE and target two to four steps (Luo et al. 2023). Distribution Matching Distillation uses approximate distribution matching plus a regression term (Yin et al. 2024), while Adversarial Diffusion Distillation combines teacher score distillation with an adversarial loss for one to four steps (Sauer et al. 2024). Their objectives and failure modes are not interchangeable.

Consistency models learn a function that maps any point on one probability-flow ODE trajectory to the same data-side boundary, with an identity condition at that boundary. They support one-step generation by design and can also use several steps to trade compute for quality; training may distill a pretrained diffusion model or use standalone consistency training (Song et al. 2023). MeanFlow is a later teacher-free route that learns an interval-average velocity from a relation to the instantaneous flow field (Geng et al. 2025). It is not the claim that average and instantaneous velocities are equal.

Evaluate the whole generation contract

Training loss alone cannot compare two deployed generators. A useful evaluation holds the prompt or class distribution, output resolution, sample count, and random-seed policy fixed, then reports:

• Distribution quality and coverage: FID or another declared feature-space statistic, together with a coverage or diversity measure. The feature encoder and sample count are part of the result.
• Conditional correctness: prompt alignment, class accuracy, or task-specific constraints, plus human evaluation when the automatic proxy is incomplete.
• Representation loss: reconstruction quality of the autoencoder separately from denoising quality in latent space.
• Efficiency: solver steps, NFE, wall-clock latency, throughput, peak memory, batch size, hardware, precision, and whether conditional and unconditional guidance passes were batched.
• Stability: variation across seeds and guidance scales, plus failure rates for malformed outputs, saturation, or mode loss.

The checkpoint is only one part of the reproducibility record. Also preserve the autoencoder and its scaling constant, prediction parameterization, training noise distribution, loss weighting, conditioning dropout, sampler equations, time or noise grid, solver tolerances, guidance convention, output shape, and decoding post-processing.

Common symptoms point to different layers:

Hand continuous states to discrete text

Text changes the state space. D3PM replaces Gaussian noise with categorical transition matrices; one option sends tokens to an absorbing [MASK] state (Austin et al. 2021). That connects diffusion to mask-based generation, but an ordinary fixed-mask-rate language model is not automatically a complete diffusion generator. SEDD learns ratios between discrete-state probabilities and reports competitive results against GPT-2-scale autoregressive models in its evaluated settings (Lou et al. 2024). LLaDA trains an 8B masked-diffusion model from scratch and reports broadly competitive results on selected zero- and few-shot tasks, with important differences in data and evaluation from external autoregressive models (Nie et al. 2025).

Length control, likelihood bounds, cache reuse, and iterative unmasking now become central. Chapter 13 develops those discrete objectives and their serving tradeoffs rather than compressing a fast-moving language-model literature into this continuous-state chapter.

Further reading