Non-Autoregressive and Diffusion Language Models

Chapter 12 ended with denoising returning to text. This chapter follows that return. Autoregressive language models generate left to right, one token conditioned on all the tokens before it; non-autoregressive and diffusion language models ask whether text can instead be drafted, masked, and refined in parallel. The appeal is latency and bidirectional context. The cost is the quality, length, and serving machinery that autoregression had already solved. That cost explains the lineage. Early systems learned from an autoregressive teacher rather than from raw data. Discrete diffusion revived the route, and masked diffusion language models turned out to be masked language models trained across masking ratios. Yet through the whole 2025 wave of fast systems, none displaced the frontier autoregressive default.

The appeal and its cost

autoregression is serial: generating $n$ tokens takes $n$ sequential forward passes, each waiting on the last. non-autoregressive generation (NAR) produces the tokens in parallel instead. Gu et al. first did this for translation, reporting an order-of-magnitude lower latency at a cost of around two BLEU against the autoregressive teacher it distilled from (Gu et al. 2018). BLEU is an automatic translation-quality score from 0 to 100 that measures n-gram overlap with reference translations, higher being better, so two BLEU is a small but real drop. The cost has a precise cause, the multimodality problem: a parallel decoder factorizes the output as $\prod_t p(y_t \mid x)$, so each position hedges across the several valid outputs independently, and the joint sample mixes them, committing to no single coherent sentence and producing repeated or dropped words. Gu's fix was a fertility latent, an integer per source token saying how many target tokens it spawns, which both picks a single mode and answers the length question that autoregression gets for free by emitting an end-of-sequence token. Length is the structural problem a parallel decoder cannot avoid, and the designs below are in part four different answers to it.

The deeper reason a non-autoregressive model works at all is that it is almost never trained on real data. It is trained on the output of an autoregressive teacher, through sequence-level knowledge distillation (Kim and Rush 2016), and the reason this is essential, not merely helpful, is mechanical. Zhou et al. measured the complexity of a dataset as the conditional entropy of the target given the source and showed that a teacher's decoded outputs have far lower complexity than the gold data: distillation removes modes, making the target more nearly deterministic and so learnable by a one-shot decoder (Zhou et al. 2020). They also found a capacity match, that a weak non-autoregressive student does best on data distilled from a small teacher while a stronger student can exploit the higher-complexity data a larger teacher leaves, so the teacher is to be matched to the student, not maximized.

Predict in parallel, then refine

Pure one-shot decoding leaves quality on the table, and the productive line buys it back by refining a parallel draft over a few rounds. Mask-Predict trains a conditional masked language model and decodes in a constant number of rounds, each round re-masking the lowest-confidence tokens and re-predicting them with full bidirectional context, reaching within about one BLEU of an autoregressive transformer (Ghazvininejad et al. 2019). Other designs answer the length problem differently: the Levenshtein Transformer makes insertion and deletion its atomic operations, so the sequence grows and shrinks dynamically rather than having its length predicted up front (Gu et al. 2019); CTC-based decoders emit an over-length sequence with blank tokens and collapse repeats under a monotonic alignment, marginalizing length and alignment jointly (Libovický and Helcl 2018; Chan et al. 2020); and the Glancing Transformer keeps a single pass but trains with a curriculum that reveals some ground-truth tokens early and fewer as the model improves, teaching word interdependence without iterative decoding, within a BLEU of the transformer at eight to fifteen times the speed (Qian et al. 2021). Step- unrolled denoising starts from random tokens and applies a denoiser to convergence, a further step toward the diffusion framing (Savinov et al. 2022).

That framing is the bridge. "Predict in parallel, then re-mask and re-predict the uncertain tokens" is, read carefully, a fixed-step absorbing-state diffusion: the masking is the forward corruption and the re-prediction is the learned reverse step. Mask-Predict is a discrete diffusion sampler that did not yet call itself one, and the rest of this chapter is what happened when the field made the connection explicit.

Diffusion for discrete text

D3PM generalized diffusion to discrete data, replacing Gaussian noise with a categorical corruption matrix $Q_t$ acting on one-hot tokens, and its absorbing-state variant is the one that matters here (Austin et al. 2021). In cumulative form each token survives unchanged with a keep-probability $\alpha_t$ that falls monotonically from one to zero, and is otherwise sent to an absorbing [MASK],

$$q(x_t \mid x_0) = \alpha_t\,\delta(x_t, x_0) + (1 - \alpha_t)\,\delta(x_t, [\text{MASK}]).$$

This is BERT-style masking with a continuum of mask rates rather than a fixed fraction, and crucially the denoiser only ever sees clean or masked tokens, never an off-distribution noisy one. Two flavors then diverge. Continuous diffusion runs in embedding space: Diffusion-LM denoises Gaussian vectors into word vectors, which makes gradient-based controllable generation easy but trails autoregression in raw perplexity (how surprised the model is by held-out text, lower being better) (Li et al. 2022), and the likelihood-based Plaid line scales the same idea but still lags (Gulrajani and Hashimoto 2023). The discrete line proved more competitive, because it avoids the embedding-rounding bottleneck and keeps the denoiser on in-distribution inputs. SEDD is its sharpest form, learning the concrete score, the ratio $p_t(y)/p_t(x)$ between marginals of states that differ by one token, through a score-entropy loss that extends score matching to discrete spaces; it cuts perplexity 25 to 75 percent over prior diffusion language models, matches or beats GPT-2 at that scale, and trades quality for as much as 32 times fewer network evaluations (Lou et al. 2024). These numbers are at small scale and author-reported.

A masked diffusion language model is a masked language model

The deepest result in this area is that the absorbing-state construction is not a new objective at all. Two independent derivations showed that the continuous-time training bound of a masked diffusion language model collapses to a weighted average of ordinary masked-language-modeling cross-entropies (Sahoo et al. 2024; Shi et al. 2024),

$$\mathcal{L} = \int_0^1 \frac{-\alpha_t'}{1 - \alpha_t}\; \mathbb{E}_{q(x_t \mid x_0)} \Big[ \sum_{n:\,x_t^{(n)} = [\text{MASK}]} -\log p_\theta\big(x_0^{(n)} \mid x_t\big) \Big]\, dt.$$

Read the pieces: $x_0$ is the clean sequence, $x_t$ is the corrupted sequence at time $t$, and $q(x_t \mid x_0)$ is the forward masking process. The inner sum is the cross-entropy of predicting the masked tokens from context, exactly the BERT objective, taken only over the positions currently masked. Because $\alpha_t$ is the keep-probability sweeping from one to zero, the integration variable $t$ is in one-to-one correspondence with the mask ratio $1 - \alpha_t$, so integrating over $t$ is averaging the masked-LM loss over every masking ratio, with the weight $-\alpha_t'/(1-\alpha_t)$ saying how much training mass each mask ratio receives. A masked diffusion language model is a masked language model trained over the whole continuum of mask rates, and generation is any-order iterative unmasking. That two derivations land on the same weight $-\alpha_t'/(1-\alpha_t)$ is strong evidence this is the canonical form. RADD sharpens it from the score side, showing the absorbing-diffusion score factorizes into the conditional distribution of clean data times an analytic scalar in $t$, so the denoiser can drop its time input entirely and its output can be cached across a sampling interval, and proving absorbing diffusion equivalent to an any-order autoregressive model (Ou et al. 2025). The non-autoregressive model is, in the end, autoregression set free of a fixed order, which is both why it can infill and reverse and why it inherits a serving cost autoregression had solved.

Scaling up, and meeting autoregression halfway

LLaDA is the scale data point, an 8B masked-diffusion model trained from scratch, competitive with LLaMA3-8B at in-context learning (handling a new task from examples in the prompt, with no weight updates) and ahead of GPT-4o on a narrow reversal-completion task where bidirectional context helps (Nie et al. 2025). Dream reaches similar 7B-scale quality by initializing from autoregressive Qwen2.5 weights rather than training from scratch, a sign that the masked-diffusion and autoregressive objectives are close enough to transfer (Ye et al. 2025). But the decoding reality is the catch. Parallel generation is real, yet it needs several denoising steps, and full bidirectional attention forfeits the key-value cache of Chapter 32, the saved per-token attention state that autoregressive decoding reuses each step, recomputing the whole sequence instead, so the speedup partly erodes. The most practical line meets autoregression halfway: block diffusion is autoregressive across blocks of tokens, which restores arbitrary- length generation and KV-caching, while denoising in parallel within each block (Arriola et al. 2025). The commercial edge is already shipping. Inception Labs' Mercury reports throughput above a thousand tokens per second on an H100 and up to ten times the speed of speed-optimized autoregressive models at comparable quality, and Google's experimental Gemini Diffusion reports similar speeds, though as unaudited vendor numbers both should be read with care (Inception Labs et al. 2025; Google DeepMind 2025). The scale objection then fell. Seed Diffusion, a ByteDance code model, reports 2,146 tokens per second on H20 GPUs at competitive code quality (Song et al. 2025); LLaDA-MoE trains a sparse mixture-of-experts diffusion model from scratch on roughly 20T tokens (Zhu et al. 2025); and LLaDA 2.0 pushes the recipe to a hundred billion total parameters by converting pretrained autoregressive weights into a mixture-of-experts diffusion model (Bie et al. 2025).

Further reading