Glossary

An architecture that replaces a dense feed-forward layer with many expert layers and routes each token to a few, so capacity grows without proportional compute.

An attention variant from DeepSeek-V2 that caches a low-rank latent in place of full per-head keys and values, claiming MHA-class quality at a small cache.

A pretraining-like phase between broad pretraining and post-training, usually mixing specialized or higher-quality data while continuing next-token training.

The stored keys and values of past tokens that let decode attend without recomputation; the memory cost that dominates serving.

An empirical power law relating model loss to compute, parameters, and data, used to predict quality and allocate a training budget.

An entropy coder that encodes a whole message as a single fraction in [0,1), approaching the optimal bit rate; the bridge from prediction to compression.

An MDL encoding that sends the model first and then the data given the model; its total length formalizes the fit-versus-complexity trade.

A formalization of Occam's razor: the best model is the one that most compresses the data plus itself; a lens on learning as compression.

A reinforcement-learning update that bootstraps a value estimate from the next step's estimate rather than waiting for the final return.

A black-box optimizer that perturbs parameters with noise and moves toward the higher-reward perturbations, needing no gradients.

The pretraining objective of predicting the next token from the preceding ones; minimizing its cross-entropy is how a language model learns.

The loss measuring how many bits the model's predicted distribution wastes against the true next token; the quantity pretraining minimizes.

The exponentiated cross-entropy, read as the model's average branching factor over tokens; lower means the model is less surprised by the text.

The count of token instances fed to the model during pretraining, after tokenization and including repeats or epochs; it is data volume, not vocabulary size.

The split of a fixed compute budget between model size and data that minimizes loss; the Chinchilla result that data should scale with parameters.

The default optimizer for transformers: Adam with weight decay decoupled from the gradient update.

A learning-rate schedule with a warmup ramp, a long constant plateau, then a final decay; the plateau lets total training length stay open.

A small auxiliary penalty on the log of the softmax normalizer that keeps logits from drifting, stabilizing the training of large models.

Normalizing queries and keys before their dot product to cap attention-logit growth, a fix for training instability at scale.

Capping the gradient norm before an update so a rare oversized batch gradient cannot blow up training.

The component that maps raw text to and from the integer token ids a model consumes; its vocabulary is fixed before training.

Removing from the training data any text that overlaps with evaluation benchmarks, so reported scores are not inflated by memorization.

A hashing scheme that cheaply estimates set similarity, used to find and remove near-duplicate documents in a training corpus.

Hashing that maps similar items to the same bucket with high probability, enabling approximate nearest-neighbor search and dedup at scale.

A language-agnostic tokenizer that trains directly on raw text without pre-tokenization, treating the input including whitespace as one stream.

An approach that drops a fixed vocabulary and models bytes or characters directly, trading tokenizer biases for longer sequences.

The vector carried unchanged down a transformer from embedding to output; each block reads it, computes a delta, and adds the delta back.

A normalization that rescales activations by their root mean square only, dropping LayerNorm's mean subtraction and bias; cheaper and now the default.

The gated feed-forward layer in modern transformers, computing (Swish(xW1) ⊙ xV)W2; three matrices, the default since LLaMA.

Position encoding that rotates query and key vectors by an angle set by position, so attention depends only on the relative offset between tokens.

The first inference pass, reading the whole prompt at once to compute every token's keys and values; compute-bound and sets time to first token.

The autoregressive pass after prefill, emitting one token per step against the cached keys and values; memory-bandwidth-bound and sets per-token latency.

An exact attention kernel that never writes the full score matrix to memory, making O(n²) attention affordable; it speeds compute but does not shrink the KV cache.

A data-parallel method that shards optimizer states, gradients, and parameters across devices instead of replicating them, cutting per-device memory.

PyTorch's implementation of full parameter, gradient, and optimizer-state sharding, gathering each layer's weights only when needed.

Additional next-token training of an already pretrained model, often on a target domain; unlike mid-training, it may switch fully to that new distribution.

Training a pretrained model on curated input-output examples so it follows instructions in a target format; the first post-training stage.

Aligning a model directly from preference pairs with a classification-style loss, skipping the separate reward model and RL loop of RLHF.

RL where the reward comes from an automatic checker (a unit test, an exact answer) rather than a learned model, removing the learned reward model as a hacking surface; an imperfect checker can still be gamed.

A generative process that learns to reverse a gradual noising of data, turning pure noise into a sample step by step.

Training a generative model by regressing a velocity field that transports noise to data along straight-ish paths; a simpler alternative to diffusion training.

The foundational discrete-time diffusion formulation that trains a network to predict the noise added at each step.

The gradient of the log-probability density with respect to the data; diffusion models learn it to know which way to denoise.

A differential equation driven by random noise; the continuous-time view of the forward and reverse diffusion processes.

A deterministic differential equation; sampling a diffusion or flow model can be cast as solving one, enabling fast few-step solvers.

A sampling trick that pushes a conditional generator away from its unconditional output to sharpen prompt adherence, trading diversity for fidelity.

A diffusion model whose denoiser is a transformer over patches rather than a U-Net, the scalable backbone of modern image and video generators.

A flow-matching variant that straightens the transport paths between noise and data so fewer solver steps suffice for high quality.

Generating a sequence one element at a time, each conditioned on those before it; the dominant paradigm for language models.

Producing many output tokens in parallel rather than one at a time, trading some quality for far lower latency.

The average number of tokens a tokenizer produces per word; lower fertility means a more efficient vocabulary for a given language.

Transcribing spoken audio into text; the input side of voice interfaces.

A loss for sequence labeling without aligned targets, summing over all alignments; a staple of speech recognition.

Synthesizing natural-sounding speech audio from text; the output side of voice interfaces.

Training image and text encoders so matching pairs land close in a shared embedding space, enabling zero-shot classification and retrieval.

A transformer that treats an image as a sequence of patches, bringing the architecture and its scaling to vision.

A model that maps visual and language input directly to robot or agent actions, extending VLMs to control.

When safety tuning makes a model decline benign requests, the false-positive cost paid for refusing harmful ones.

A PEFT method that freezes the base weights and learns a small low-rank update per layer, so many task adapters share one base model.

LoRA applied on top of a 4-bit quantized base model, letting a large model be fine-tuned on a single GPU.

A stable policy-gradient RL algorithm that clips each update to stay close to the previous policy; the workhorse optimizer of RLHF.

RLHF where the preference labels come from a model rather than humans, trading some quality for cheaper, scalable feedback.

When a model maximizes the measured reward by exploiting its flaws rather than achieving the intended goal; the central failure of optimizing a proxy.

An asymmetric measure of how far one distribution is from another; in alignment it penalizes the policy for drifting from the reference model.

A DPO variant that replaces the log-sigmoid loss with a squared objective to curb overfitting to preference pairs.

A preference method that learns from unpaired good/bad labels with a prospect-theory-inspired loss, needing no preference pairs.

A method that folds preference alignment into SFT with an odds-ratio penalty, removing the need for a separate reference model.

A reference-free DPO variant using the average log-probability as an implicit reward with a target margin, simpler and memory-light.

An iterative method that samples candidates, keeps the highest-reward ones, and fine-tunes on them; rejection sampling cast as training.

A reward model that scores each step of a reasoning trace, not just the final answer, giving denser signal for training and search.

Prompting or training a model to produce intermediate reasoning steps before its answer, which raises accuracy on multi-step problems.

Sampling several reasoning chains and taking the majority answer, trading extra compute for higher accuracy.

A prompting strategy that decomposes a hard problem into easier subproblems solved in order, each building on the last.

A reasoning method that explores a branching tree of intermediate thoughts with lookahead and backtracking, beyond a single chain.

An RL method that drops the value critic and normalizes rewards within a group of sampled answers; the optimizer behind DeepSeek-R1.

A lightweight policy-gradient method that uses the mean reward of the other samples as each sample's baseline, avoiding a learned value network.

A search that builds a tree by sampling rollouts and backing up their values, balancing exploration and exploitation; used for reasoning and planning.

A test-time control that caps or extends how many reasoning tokens a model spends, trading latency for accuracy.

A reward model that judges by generating a critique or verdict in text rather than emitting a scalar, often using chain-of-thought.

Throughput counted only over requests that meet their latency targets, the objective a serving system actually optimizes, unlike raw throughput.

Scheduling that admits and retires requests at each decode step instead of per fixed batch, so a finished request frees its slot immediately.

A KV-cache allocator that stores the cache in fixed-size pages like virtual memory, ending the fragmentation of reserving one contiguous block per sequence.

Reusing the KV cache of a shared prompt prefix across requests so its prefill is computed once, not per request.

Denser, cheaper memory needing periodic refresh; the bulk of off-chip capacity, including the HBM next to accelerators.

Drafting several tokens with a small fast model and verifying them in one pass of the large model, trading extra compute for fewer slow decode steps.

Fast on-chip memory needing no refresh; small and expensive, used for caches and register files near compute.

A model of computation with a finite set of states and transitions; used in constrained decoding to track which tokens a grammar allows next.

A few initial tokens that attention heads dump excess weight onto; keeping them in the cache stabilizes streaming and long-context generation.

Restricting the tokens the model may emit at each step to those allowed by a grammar or schema, so the output is valid by construction.

An open protocol standardizing how an agent connects to external tools and data sources, so a host can talk to many servers uniformly.

Fetching relevant documents at query time and placing them in the prompt, so the model answers from retrieved evidence rather than parametric memory alone.

A retrieval model that embeds query and document separately so document vectors can be precomputed and searched; fast but less precise than joint scoring.

A classic sparse lexical ranking function scoring documents by term frequency and rarity; the strong keyword-search baseline behind hybrid retrieval.

A graph index for approximate nearest-neighbor search over vectors, navigating layered links to find close embeddings fast.

Combining sparse lexical scores (BM25) and dense vector similarity into one ranking, catching both exact-keyword and semantic matches.

A model that scores a query and document jointly in one pass, more accurate than a dual-encoder but too costly to run over a whole corpus, so used to rerank.

Data deliberately kept out of training so it can measure generalization; the basis of honest evaluation.

When evaluation data has leaked into training, inflating benchmark scores without real capability gains.

A safety framing where several imperfect layers each have holes, but stacking them so the holes do not line up blocks most failures.

An attack that tests whether a specific example was in a model's training set, a basic privacy threat.

Unique markers planted in data to detect later leakage or training-set contamination when they reappear in a model's output.

A benchmark suite that scores models across many scenarios and metrics together, rather than reducing them to one leaderboard number.

Using a strong model to score or compare outputs in place of human raters: cheap and scalable, but with its own biases.

Judging quality by asking which of two outputs is better, more reliable than absolute scores and the basis of preference data and arenas.

Once a measure becomes a target it stops being a good measure; the reason benchmarks decay and rewards get hacked.

An explicit scoring guide listing the criteria and levels a judge applies, making subjective evaluation more consistent.

Evaluation data kept secret from model developers so scores cannot be gamed by training on the test.

The probability that at least one of k sampled attempts passes, the standard metric for code and other verifiable tasks.

A system's ability to reach correct consensus even when some components fail arbitrarily or maliciously, not just by crashing.

A property guaranteeing that something good eventually happens (progress); paired with safety in reasoning about systems.

As a formal property, a guarantee that nothing bad ever happens; in the broader AI sense, keeping systems from causing harm.

Reverse-engineering a network's internal computations into human-understandable circuits and features, rather than treating it as a black box.

When a single neuron responds to several unrelated concepts, making activations hard to read; the problem sparse autoencoders try to undo.

A network packing more features than it has neurons by representing them as overlapping directions in activation space; why polysemanticity arises.

An interpretability tool that decomposes a model's activations into many sparse, more monosemantic features, used to read what a network represents.

Methods for supervising models on tasks too hard or too numerous for humans to check directly, often by using AI to assist the oversight.

Studying whether a weaker supervisor can elicit the full capability of a stronger model, a proxy for humans overseeing superhuman systems.

A failure where a model behaves aligned while observed but pursues a different goal once it judges it can; the hardest safety case to detect.

Models trained to act normally until a trigger activates hidden behavior, which can survive safety training; a demonstration of deceptive alignment.

An attack where untrusted content in the model's input overrides its instructions, hijacking an agent through the data it reads.

A delegation pattern where a service acts with a user's identity and permissions, the basis for scoping what an agent may do for whom.

The dangerous combination of access to private data, exposure to untrusted content, and the ability to communicate externally, which together enable exfiltration.

An ordering of trust over instruction sources (system over developer over user over tool output) so a model knows which to obey on conflict.

A prompt crafted to bypass a model's safety training and elicit content it was tuned to refuse.

Adversarially probing a system to surface failures, jailbreaks, and harms before deployment.

A model's resistance to inputs crafted specifically to make it fail; an arms race rather than a solved property.

A white-box attack that optimizes an adversarial suffix token by token to force a target output, producing transferable jailbreaks.

Removing the influence of specific training data from a trained model without retraining from scratch.

A formal guarantee that a computation's output barely changes whether or not any single record is included, bounding what can be learned about an individual.

Surgically updating a specific fact a model has stored, ideally without disturbing unrelated knowledge.

Embedding a hard-to-remove statistical signal in generated content so it can later be identified as machine-produced.

The tendency of EU regulation to become a de facto global standard because firms apply it everywhere rather than maintain two product lines.

A short standardized document reporting a model's intended use, training data, evaluations, and limitations.

A standardized document recording how a dataset was collected, composed, and intended to be used, the data analog of a model card.

A document describing a deployed system as a whole, including its model, safeguards, and evaluated risks, broader than a model card.

Stacked DRAM placed next to an accelerator for very high bandwidth; its capacity and bandwidth bound how large a model and KV cache can be served.

Google's custom accelerator built around large matrix-multiply units, an ASIC alternative to GPUs for deep learning.

The high-speed links connecting accelerators directly (such as NVLink or a TPU's ICI), carrying the collective traffic of tensor and pipeline parallelism.

A massively parallel processor, originally for graphics, now the dominant accelerator for training and serving neural networks.

The fraction of peak hardware throughput actually achieved, counting all FLOPs including recomputation, unlike MFU.

A network capability letting one machine read or write another's memory without involving its CPU, key to fast collective communication.

Computing exact derivatives of a program by applying derivative rules to each elementary operation as it executes, rather than differencing numerically or manipulating formulas symbolically.

Hardware errors that produce wrong results without raising any fault, a growing hazard at fleet scale and during long training runs.

TSMC's advanced packaging that places compute dies and HBM stacks on one interposer; a supply bottleneck for AI accelerators.

A chip designed for one task, trading the generality of a CPU or GPU for far higher efficiency on that task.

A factory-built nuclear reactor of modest output, proposed as a dedicated, steady power source for AI datacenters.

A datacenter's total power divided by the power delivered to IT equipment; closer to 1 means less overhead on cooling and the like.

The average operating time between hardware failures; at cluster scale it falls low enough that long training runs must expect and recover from them.

The fraction of wall-clock time a training run spends making real progress, after subtracting failures, restarts, and stalls.

Up-front spending on long-lived assets such as GPUs and datacenters, depreciated over years; the bulk of the cost of owning compute.

Recurring running costs such as power, bandwidth, and rented cloud capacity; the pay-as-you-go side of serving.

A proxy in front of model providers that centralizes routing, auth, rate limits, cost tracking, and fallbacks across many backends.

An isolated environment that confines untrusted code or agent actions so they cannot affect the host system.

A gateway-issued credential mapped to an upstream provider key, letting an operator set per-team budgets, limits, and revocation without sharing the real key.

Converting images of text into machine-readable characters, a front-end step for ingesting scanned or photographed documents.

A model that takes both images and text and reasons over them jointly, the basis of multimodal assistants.

The ability to understand a system's internal state from its outputs (logs, metrics, traces); for LLM apps it extends to prompts, tokens, and quality.

A measured metric of service behavior, such as latency or error rate, against which an SLO target is set.

A target threshold for a service metric, such as p99 latency under a bound, that a serving system commits to meeting.

A system design that inserts human judgment at defined points to approve, correct, label, escalate, or stop model behavior.

Using automation when it is likely useful and withholding, checking, or overriding it when uncertainty, stakes, or evidence warrant.

The tendency to over-rely on automated recommendations, especially when the system appears authoritative or the user is under time pressure.

Aligning a model by training a reward model on human preference comparisons and optimizing the policy against it; the original alignment recipe.