Glossary

An architecture that replaces a dense feed-forward layer with many expert layers and routes each token to a few, so capacity grows without proportional compute.

An attention design introduced with DeepSeek-V2 that compresses keys and values into a low-rank latent representation to reduce the KV cache.

A pretraining-like phase between broad pretraining and post-training, usually mixing specialized or higher-quality data while continuing next-token training.

A store of attention keys and values from earlier tokens that lets each decode step reuse prior computation.

An empirical power law relating model loss to compute, parameters, and data, used to predict quality and allocate a training budget.

An entropy-coding method that represents an entire message as an interval within the unit interval, approaching the source's ideal bit rate.

A model-selection principle that prefers the model producing the shortest combined description of itself and the data.

A minimum-description-length encoding that records a model first and then records the data using that model.

A pretraining objective that asks a model to predict each token from the tokens that precede it.

A loss that measures how much probability a model assigns to the correct answer compared with its full predicted distribution.

The exponential of cross-entropy, often interpreted as the average number of plausible token choices the model considers at each step.

The total number of token instances processed during pretraining, including repeated examples and additional passes through the data.

A balance between model size and training data that produces the lowest expected loss for a fixed compute budget.

A version of the Adam optimizer that applies weight decay separately from the gradient-based parameter update.

A learning-rate schedule that warms up, stays constant for most of training, and decays near the end.

A small auxiliary penalty on the log of the softmax normalizer that keeps logits from drifting, stabilizing the training of large models.

A stabilization technique that normalizes query and key vectors before their dot product to limit the growth of attention logits.

A stabilization technique that limits gradient magnitude before an update to prevent unusually large gradients from disrupting training.

The removal of training examples that overlap with evaluation data, preventing memorization from inflating benchmark scores.

A hashing scheme that cheaply estimates set similarity, used to find and remove near-duplicate documents in a training corpus.

A hashing method designed to place similar items in the same bucket with high probability, supporting approximate search and deduplication.

A component that converts raw text into the integer token identifiers a model consumes and converts model output back into text.

A language-agnostic tokenizer that trains directly on raw text without pre-tokenization, treating the input including whitespace as one stream.

An approach that drops a fixed vocabulary and models bytes or characters directly, trading tokenizer biases for longer sequences.

The shared vector that passes through a transformer while each block reads it and adds its own update.

A normalization method that rescales activations by their root mean square without subtracting their mean.

A gated feed-forward layer that combines a Swish-activated projection with a second projection before producing its output.

A position-encoding method that rotates query and key vectors by position-dependent angles so their dot product reflects relative position.

The initial inference pass that processes the full prompt and computes the keys and values needed for later token generation.

The inference phase after prefill that generates one token at a time while reusing cached keys and values.

An exact attention algorithm that avoids storing the full score matrix in main memory, reducing memory traffic and accelerating attention.

A data-parallel method that shards optimizer states, gradients, and parameters across devices instead of replicating them, cutting per-device memory.

A PyTorch distributed-training method that shards parameters, gradients, and optimizer state, gathering each layer's parameters only when needed.

A training phase that continues the next-token objective on an existing pretrained model, often using data from a specific domain.

A post-training stage that teaches a pretrained model to follow instructions by training it on curated input-output examples.

A post-training method that learns directly from preferred and rejected response pairs without a separate reward model or reinforcement-learning loop.

A reinforcement-learning method in which an automatic checker, such as a unit test or exact-answer rule, supplies the reward.

A generative process that learns to reverse a gradual noising of data, turning pure noise into a sample step by step.

A generative-model training method that learns a velocity field for transporting samples from a noise distribution to the data distribution.

A discrete-time diffusion formulation that trains a network to reverse a fixed process that gradually adds noise to data.

A differential equation that includes a random process and therefore describes both deterministic change and noise over continuous time.

An equation that relates a function of one independent variable to one or more of its derivatives.

A diffusion model whose denoiser is a transformer over patches rather than a U-Net, the scalable backbone of modern image and video generators.

A sampling trick that pushes a conditional generator away from its unconditional output to sharpen prompt adherence, trading diversity for fidelity.

A flow-matching variant that straightens the transport paths between noise and data so fewer solver steps suffice for high quality.

A generation method that predicts multiple output elements in parallel instead of conditioning each one on the preceding outputs.

A generation method that produces one sequence element at a time, conditioning each new element on those already produced.

A system that converts spoken audio into written text.

A loss that trains sequence models without requiring each input position to be aligned with a target label.

A system that converts written text into spoken audio.

A training method that places matching images and text near each other in a shared embedding space and separates mismatched pairs.

A transformer that treats an image as a sequence of patches, bringing the architecture and its scaling to vision.

A model that maps visual and language input directly to robot or agent actions, extending VLMs to control.

A safety failure in which a model incorrectly declines a benign request because it resembles a harmful one.

A PEFT method that freezes the base weights and learns a small low-rank update per layer, so many task adapters share one base model.

A fine-tuning method that applies LoRA adapters to a quantized base model to reduce the memory required for training.

A policy-gradient reinforcement-learning algorithm that limits each update so the new policy does not move too far from the previous one.

A post-training method that uses model-generated preferences or critiques as reinforcement-learning feedback instead of relying only on human labels.

An asymmetric measure of how one probability distribution differs from another, often used to limit policy drift during alignment.

A DPO variant that replaces the log-sigmoid loss with a squared objective to curb overfitting to preference pairs.

A preference method that learns from unpaired good/bad labels with a prospect-theory-inspired loss, needing no preference pairs.

A method that folds preference alignment into SFT with an odds-ratio penalty, removing the need for a separate reference model.

A reference-free DPO variant using the average log-probability as an implicit reward with a target margin, simpler and memory-light.

A sequence of intermediate reasoning steps produced before a model gives its final answer.

A prompting strategy that decomposes a hard problem into easier subproblems solved in order, each building on the last.

A decoding method that samples several reasoning paths and returns the answer that appears most often.

A reinforcement-learning method that estimates relative advantages from a group of sampled answers without training a separate value model.

A lightweight policy-gradient method that uses the mean reward of the other samples as each sample's baseline, avoiding a learned value network.

The rate of completed requests that satisfy a service's latency or other acceptance targets.

A serving scheduler that adds and removes requests between decode steps so completed requests release capacity immediately.

A KV-cache allocator that stores the cache in fixed-size pages like virtual memory, ending the fragmentation of reserving one contiguous block per sequence.

A serving optimization that reuses the KV cache for a prompt prefix shared by multiple requests, avoiding repeated prefill work.

DRAM stores bits in capacitors that require periodic refresh and provides most off-chip working memory.

A decoding method in which a faster model drafts several tokens and the target model verifies them together.

SRAM stores each bit in a stable circuit without periodic refresh and is commonly used for fast on-chip caches.

A decoding method that permits only tokens allowed by a grammar or schema at each step, ensuring structurally valid output.

A computational model with a finite set of states and rules for moving between them in response to input.

An initial token that receives substantial attention even when its content is unimportant, helping some models preserve behavior over long contexts.

A method that retrieves relevant documents at query time and adds them to the prompt so the model can answer from that evidence.

A lexical ranking function that scores documents from query-term frequency, document length, and how rare each term is across the collection.

A retrieval model that embeds queries and documents separately, allowing document vectors to be computed before a search begins.

A graph index for approximate nearest-neighbor search over vectors, navigating layered links to find close embeddings fast.

A retrieval method that combines lexical scores with vector similarity so results can reflect both exact terms and semantic meaning.

A model that scores a query and document jointly in one pass, more accurate than a dual-encoder but too costly to run over a whole corpus, so used to rerank.

A held-out set contains examples excluded from training and reserved for measuring generalization to unseen data.

An attack that tests whether a specific example was in a model's training set, a basic privacy threat.

A benchmark suite that scores models across many scenarios and metrics together, rather than reducing them to one leaderboard number.

An evaluation method that asks a model to score or compare outputs in place of, or before, human review.

An evaluation that presents two outputs and asks a judge to choose the better one according to stated criteria.

An evaluation set hidden from model developers to reduce the risk of optimizing or training directly against its examples.

The probability that at least one of k sampled attempts passes, the standard metric for code and other verifiable tasks.

The study of how a neural network implements behavior by identifying understandable features, circuits, and computations inside it.

A representation strategy in which a network stores more features than it has dimensions by using overlapping directions in activation space.

A research setting that tests whether supervision from a weaker model can elicit useful behavior from a stronger model.

A hypothesized failure in which a model appears to follow its training objective while monitored but pursues a different objective when unobserved.

An attack where untrusted content in the model's input overrides its instructions, hijacking an agent through the data it reads.

An ordering of trust over instruction sources (system over developer over user over tool output) so a model knows which to obey on conflict.

A structured effort to test a system adversarially and uncover failures, abuse paths, and potential harms before deployment.

A model's ability to preserve its intended behavior when an attacker deliberately crafts inputs to cause failure.

A prompt crafted to bypass a model's safety training and elicit content it was tuned to refuse.

A process that removes the influence of selected training data from a trained model without repeating the full original training run.

The tendency of EU regulation to become a de facto global standard because firms apply it everywhere rather than maintain two product lines.

A stack of DRAM dies placed close to an accelerator to provide more data-transfer bandwidth than conventional off-chip memory.

A network capability letting one machine read or write another's memory without involving its CPU, key to fast collective communication.

An accelerator designed by Google around matrix-multiplication hardware for machine-learning training and inference.

The high-speed links connecting accelerators directly (such as NVLink or a TPU's ICI), carrying the collective traffic of tensor and pipeline parallelism.

A highly parallel processor developed for graphics and widely used to train and serve neural networks.

A method that computes derivatives by applying the chain rule to the elementary operations executed by a program.

Silent data corruption occurs when hardware produces an incorrect result without reporting a fault.

A TSMC packaging technology that connects compute dies and stacks of high-bandwidth memory through a shared silicon interposer.

The ratio of a datacenter's total power consumption to the power used by its computing equipment, where values nearer 1 indicate less facility overhead.

A factory-built nuclear reactor of modest output, proposed as a dedicated, steady power source for AI datacenters.

The average operating time expected between one hardware failure and the next.

A failure in which a model earns a high measured reward by exploiting the metric instead of achieving the intended goal.

An open protocol standardizing how an agent connects to external tools and data sources, so a host can talk to many servers uniformly.

Capital expenses pay for long-lived assets such as accelerators, networking equipment, and datacenters that are depreciated over time.

Operating expenses are the recurring costs of running a service, including power, bandwidth, maintenance, and rented cloud capacity.

A proxy in front of model providers that centralizes routing, auth, rate limits, cost tracking, and fallbacks across many backends.

An isolated environment that confines untrusted code or agent actions so they cannot affect the host system.

A gateway-issued credential mapped to an upstream provider key, letting an operator set per-team budgets, limits, and revocation without sharing the real key.

A process that converts text in scanned documents, photographs, or other images into machine-readable characters.

A model that processes images and text together to answer questions, describe visual content, or perform other multimodal tasks.

The ability to infer a system's internal state from signals such as logs, metrics, traces, prompts, token usage, and quality measurements.

A measured metric of service behavior, such as latency or error rate, against which an SLO target is set.

A target threshold for a service metric, such as p99 latency under a bound, that a serving system commits to meeting.

A system design that inserts human judgment at defined points to approve, correct, label, escalate, or stop model behavior.

A pattern of relying on automation when evidence supports it and checking or overriding it when uncertainty or risk is high.

The tendency to over-rely on automated recommendations, especially when the system appears authoritative or the user is under time pressure.

A post-training method that learns a reward model from human preferences and then optimizes a policy against that reward.