Non-Autoregressive and Diffusion Language Models

Chapter 12 described paths through continuous states. Text poses a different problem: its state is categorical, and its length is usually not known in advance. non-autoregressive generation (NAR) generation attacks the serial dependency of left-to-right decoding by predicting several positions together. diffusion adds a forward corruption process and a learned reverse process. The two ideas overlap, but they are not synonyms. A one-pass translation model can be non-autoregressive without being a diffusion model, while a block diffusion model can remain autoregressive across blocks.

The attraction is shorter dependency depth. For an $L$-token output, autoregression has $L$ causally ordered decode steps; a masked model may revise all $L$ positions in $K$ rounds. That does not make its work $L/K$ times cheaper. Each round can score a full sequence, and changing bidirectional states prevents the usual causal key-value cache from being reused unchanged. The useful comparison is therefore not “serial versus parallel,” but dependency depth, total model work, output-length handling, and quality under one serving contract.

Separate the factorization from the execution plan

Here, $c$ is a source or prompt and $y=(y_1,\ldots,y_L)$ is its output. An autoregressive model writes

$$p_\theta(y\mid c) =\prod_{i=1}^{L}p_\theta(y_i\mid y_{<i},c).$$

Here, $\theta$ denotes model parameters; $L$ is output length; $y_i$ is token $i$; and $y_{<i}$ contains the earlier output tokens. The factorization gives a natural stopping rule because the model can emit an end-of-sequence token. With a causal key-value cache, each decode invocation processes one new query position while reusing keys and values from the prefix.

The original one-shot non-autoregressive translation model instead used

$$\begin{gathered} p_\theta(y\mid c,L,z) =\\[-0.4em] \prod_{i=1}^{L}p_\theta(y_i\mid c,L,z). \end{gathered}$$

where $z$ is an optional latent alignment or fertility variable. Output positions share the encoded source and latent variables, but their sampled tokens do not condition on one another. All positions can therefore be scored in one model evaluation once $L$ and $z$ are chosen. Across operating points, Gu et al. reported substantial latency reductions; the smallest reported gap to their autoregressive teacher was 2 BLEU, while higher-quality decoding reduced the speed advantage (Gu et al. 2018). BLEU measures n-gram overlap with reference translations; its difference is specific to the dataset, tokenization, and decoding setup, not a universal quality unit.

Iterative models introduce states $y^{(K)},\ldots,y^{(0)}$ and factorize one refinement step across positions:

$$\begin{gathered} p_\theta\!\left(y^{(k-1)}\mid y^{(k)},c\right) =\\[-0.4em] \prod_{i=1}^{L} p_\theta\!\left(y_i^{(k-1)}\mid y^{(k)},c\right). \end{gathered}$$

Here, $k$ is the current refinement round, $y^{(k)}$ is the whole current sequence, and $y^{(0)}$ is the final output. Positions are parallel within a round, but round $k-1$ waits for round $k$. The model may revise masks, tokens, or sequence length depending on the algorithm.

In the last row, $B$ is block size and $K$ is the number of denoising rounds per block. This table counts dependency, not FLOPs or wall-clock time.

Why an independent draft can mix valid translations

Translation is multimodal: one source sentence can have several valid targets. If one mode says “I do not know” and another says “I don't know,” independent position predictions can combine pieces of both. The resulting token marginals may each look plausible while the joint sentence does not. Repetition and omission are visible symptoms, but the underlying issue is the conditional independence assumption, not a rule that every non-autoregressive model must fail.

Early systems reduced this uncertainty in two ways. Gu et al. used a fertility latent, an integer count describing how many target positions each source token produces. It supplies an alignment and helps determine output length. They also trained on decoded outputs from an autoregressive teacher. Sequence-level knowledge distillation replaces each set of human references with a narrower teacher-generated target (Kim and Rush 2016). Zhou et al. measured this effect as lower alignment-based word-translation entropy on WMT14 English-German. In their experiments, stronger non-autoregressive students tended to benefit from more complex distilled data (Zhou et al. 2020). This is an empirical proxy and capacity trend, not a general theorem about the true sequence entropy.

Distillation was central to classic non-autoregressive translation, but it is not a mathematical requirement for all parallel generation. It changes the training distribution, transfers the teacher's errors and preferences, and can make a student's apparent speed-quality result depend on an unreported teacher. A fair record names the teacher, decoding method, and whether evaluation uses human references or teacher outputs.

Refinement predates diffusion language models

Several pre-diffusion systems added structure without returning to strict left-to-right decoding:

Mask-Predict improved over one-shot decoding by spending a predetermined number of confidence-based refinement rounds (Ghazvininejad et al. 2019). The partially autoregressive Levenshtein Transformer made insertion and deletion explicit (Gu et al. 2019). CTC systems marginalized monotonic alignments containing blanks and repeats (Libovický and Helcl 2018; Chan et al. 2020). Glancing training exposed a decreasing number of target tokens as training progressed (Qian et al. 2021), while SUNDAE trained a denoiser through more than one of its own predicted states (Savinov et al. 2022).

These methods established useful operations: draft, mask, insert, delete, collapse, and refine. Mask-Predict resembles absorbing-state diffusion at the algorithmic level, but it was not derived as the reverse chain of D3PM and does not inherit a diffusion likelihood bound merely because it uses masks. That distinction matters when comparing objectives.

Put diffusion on categorical states

D3PM replaces Gaussian corruption with a Markov chain over a finite vocabulary (Austin et al. 2021). Represent one token as a one-hot vector $x_t\in\{0,1\}^{V}$, where $V$ is vocabulary size. With a column-stochastic transition matrix $Q_t\in[0,1]^{V\times V}$, one forward step is

$$\begin{aligned} q(x_t\mid x_{t-1}) &=\operatorname{Cat}(x_t;Q_t x_{t-1}),\\ \bar Q_t&=Q_tQ_{t-1}\cdots Q_1. \end{aligned}$$

and the direct marginal is

$$q(x_t\mid x_0) =\operatorname{Cat}(x_t;\bar Q_t x_0).$$

Here, $q$ is the fixed forward process; $t$ is a corruption step; $\operatorname{Cat}(x;\pi)$ is a categorical distribution with probability vector $\pi$; and $\bar Q_t$ is the cumulative transition. For a sequence, the usual construction corrupts token positions independently even though the reverse model reads the whole corrupted sequence.

An absorbing process adds a special mask state $m$. In continuous time, the single-position marginal can be written

$$\begin{gathered} q(z_t\mid x)=\operatorname{Cat}(z_t;\pi_t(x)),\\ \pi_t(x)=\alpha_t x+(1-\alpha_t)m,\\ t\in[0,1]. \end{gathered}$$

Here, $x$ is the clean one-hot token, $z_t$ is its corrupted state, $m$ is the one-hot mask token, $\pi_t(x)$ is the displayed categorical probability vector, and $\alpha_t$ is a decreasing keep probability with ideal endpoints $\alpha_0=1$ and $\alpha_1=0$. Once masked in the forward process, a token remains masked. The network therefore receives sequences made from clean vocabulary items and a known mask symbol, rather than arbitrary Gaussian vectors that later need rounding.

Continuous embedding diffusion is a different branch. Diffusion-LM corrupts word embeddings and uses gradients through the continuous state for controllable generation (Li et al. 2022). Plaid develops a likelihood-based continuous language model at larger scale (Gulrajani and Hashimoto 2023). Those models must connect continuous outputs back to discrete tokens; absorbing diffusion stays categorical throughout. Results from the two branches should not be merged into one quality claim.

Derive the masked-diffusion training loss

For the absorbing process, a substitution-based reverse parameterization copies unmasked tokens and predicts a clean-token distribution only at masked positions. Under that parameterization, the continuous-time negative evidence lower bound reduces to (Sahoo et al. 2024; Shi et al. 2024)

$$\begin{gathered} \ell_\theta(x,z_t) =\sum_{i:\,z_t^{(i)}=m}\\[-0.3em] \quad\times\left[-\log p_\theta\!\left(x^{(i)}\mid z_t,t\right)\right],\\ \mathcal L_{\mathrm{MDLM}} =\int_0^1 \frac{-\alpha_t'}{1-\alpha_t} \\[-0.3em] \quad\times\mathbb E_{x,z_t}\!\left[\ell_\theta(x,z_t)\right]dt. \end{gathered}$$

Here, $x=(x^{(1)},\ldots,x^{(L)})$ is a clean sequence; $z_t$ is produced by independently masking its positions with keep probability $\alpha_t$; $\alpha_t'=d\alpha_t/dt$; $i$ indexes positions currently equal to mask $m$; and $\ell_\theta$ sums negative log-probabilities over those positions, using $p_\theta(x^{(i)}\mid z_t,t)$ as the model's clean-token probability at position $i$. Because $\alpha_t$ decreases, the weight $-\alpha_t'/(1-\alpha_t)$ is nonnegative. In practice, training samples times and masking patterns rather than evaluating the integral exactly. The apparent singularity at the clean endpoint is interpreted as a limit; the probability of observing a masked position vanishes there, and implementations use numerically stable finite sampling.

The inner term is masked-token cross-entropy. The diffusion objective is therefore a weighted family of masked-language-model losses over many corruption levels. It is not identical to ordinary BERT training: BERT uses a particular masking recipe and was not defined with this reverse process, carry-over rule, time weighting, or likelihood bound. An existing masked language model does not become a complete generator until its training and sampling contracts supply those missing pieces.

Generation starts from an output canvas of $L$ masks. At each reverse step, the model predicts clean tokens from the entire current state, the sampler reveals a schedule-dependent subset, and the absorbing reverse parameterization carries already revealed tokens forward. Unlike confidence-based Mask-Predict variants, this ancestral sampler need not re-mask a revealed token. The final step must remove every remaining mask. Length $L$ still has to be chosen, predicted, or handled by a blockwise extension.

Distinguish concrete-score caching from a KV cache

SEDD describes a continuous-time discrete diffusion through ratios of noisy marginal probabilities. For neighboring sequences $x$ and $y$ that differ at one token position, its concrete score is

$$s_t(x)_y=\frac{p_t(y)}{p_t(x)}.$$

Here, $p_t$ is the noisy sequence distribution at time $t$, and $s_t(x)_y$ is the ratio associated with the transition from $x$ to $y$. A neural network approximates the ratios required by the sparse token transition graph rather than representing an exponentially large sequence transition matrix. SEDD's score-entropy objective improved perplexity over earlier diffusion language models and was competitive with GPT-2-scale autoregressive baselines in its experiments (Lou et al. 2024). Its reported network-evaluation tradeoff is internal to those tested samplers; one full-sequence diffusion evaluation is not cost equivalent to one cached autoregressive token step.

RADD specializes the score for absorbing diffusion. It factorizes the required ratio into a clean-data conditional distribution and a known time-dependent scalar (Ou et al. 2025). If $x_t^{i\leftarrow y}$ replaces mask position $i$ in $x_t$ with clean token $y$, then the population relation is

$$\begin{gathered} \frac{p_t\!\left(x_t^{i\leftarrow y}\right)}{p_t(x_t)} =\frac{\alpha_t}{1-\alpha_t}\\[-0.2em] \quad\times p_0\!\left(X_i=y\mid X_U=x_{t,U}\right). \end{gathered}$$

Here, $U$ is the set of unmasked positions; $X_U=x_{t,U}$ means that the clean random sequence $X$ agrees with the visible tokens in $x_t$; $p_0$ is the clean data distribution; and $\alpha_t/(1-\alpha_t)$ is the known schedule factor. The learned conditional on the right need not take $t$ as input even though the analytic scalar remains time-dependent. If a reverse transition leaves the corrupted sequence unchanged, that denoiser's output can be reused at the next time point. This is model-output caching across an unchanged state. It is different from a causal KV cache, which reuses prefix keys and values while appending a new token.

RADD also connects absorbing diffusion to any-order autoregressive factorizations. The probability model may reveal variables in many orders rather than one fixed left-to-right order. That representational equivalence does not make the execution plans equivalent: a bidirectional denoiser can still score a whole canvas repeatedly.

Recover length and prefix caching with blocks

Full-sequence masked diffusion normally begins with a fixed canvas. Full bidirectional attention also changes hidden states throughout that canvas after each reveal, so the standard causal prefix KV cache cannot simply be carried between rounds. Block diffusion changes the factorization: it generates blocks autoregressively, then denoises positions within the current block in parallel (Arriola et al. 2025). Completed blocks form a stable prefix, so their KV state can be cached, and generation can continue for an arbitrary number of blocks.

The following runnable makes only this dependency accounting concrete. It assumes that a masked pass predicts every position in its current block, ignores prompt prefill, and does not pretend that one position prediction has constant hardware cost.

from math import ceil

length = 128
rounds = 8
block_size = 16

autoregressive = {
    "dependent_evaluations": length,
    "position_predictions": length,
}
full_masked = {
    "dependent_evaluations": rounds,
    "position_predictions": length * rounds,
}
block_diffusion = {
    "dependent_evaluations": ceil(length / block_size) * rounds,
    "position_predictions": length * rounds,
}

for name, cost in [
    ("autoregressive", autoregressive),
    ("full masked", full_masked),
    ("block diffusion", block_diffusion),
]:
    print(
        f"{name:16} dependent={cost['dependent_evaluations']:3d}, "
        f"positions={cost['position_predictions']:4d}"
    )

With these assumptions, full masked diffusion has the shortest dependency chain but predicts eight times as many token positions as cached autoregression. Block diffusion lies between them in dependency depth while doing the same illustrative number of within-block position predictions as full masked diffusion. Actual latency depends on attention shape, kernels, batch, memory traffic, and how much parallel hardware each evaluation can use.

Read scale demonstrations as scoped evidence

Large diffusion language models established that the objective can train or be adapted beyond the earlier GPT-2-scale experiments. They did not create one controlled comparison in which only the factorization changed.

LLaDA's paper reports strong results against its own autoregressive baselines and selected external models, including an 8B model trained from scratch (Nie et al. 2025). Dream instead initializes from Qwen2.5 autoregressive weights (Ye et al. 2025). LLaDA-MoE reports 7B total parameters, 1.4B active parameters, and approximately 20T training tokens (Zhu et al. 2025). LLaDA 2.0 converts pretrained autoregressive models through staged block and full-sequence diffusion training, reaching 100B total parameters in its sparse variant (Bie et al. 2025). These are distinct claims about training route, active compute, and total model capacity.

Mercury and Seed Diffusion also report high code-generation throughput (Inception Labs et al. 2025; Song et al. 2025). Those results show that engineered diffusion systems can occupy useful speed-quality operating points. They are not a portable multiplier for the model class: the reports use different models, tokenizers, accelerators, output lengths, batches, and quality suites.

Benchmark the deployed generator, not the label

A reproducible comparison fixes the task and reports the complete decode contract:

Perplexity is not directly comparable when tokenizers differ because a “token” then represents different amounts of text. Tokens per second has the same problem. Character or byte throughput can supplement it, but quality and latency must still be measured on the same requests. For interactive use, a system that finishes quickly but exposes no stable prefix may also feel different from a streaming autoregressive system with a low time to first token.

Method choice follows the workload:

Further reading