Mechanistic Interpretability

mechanistic interpretability tries to explain how a fixed model produced a specified behavior by constructing human-readable hypotheses about its internal computation and testing those hypotheses against the model. A useful claim therefore names the model revision, input population, behavior or output, and internal site under study. It is not a transcript of hidden thoughts, and it is not by itself a safety guarantee.

This scope matters because an internal pattern can support several different claims. It may merely correlate with a behavior, predict that behavior on new inputs, change the behavior when edited, or participate in the model's ordinary computation. These are increasingly demanding claims. This chapter develops a workflow for keeping them separate: propose candidate features, test them with held-out data and interventions, assemble them into circuit hypotheses, and report what the explanation leaves out. The result can inform the oversight methods in Chapter 55, but it cannot replace behavioral evidence or an explicit safety case.

Begin with the explanatory claim

Mechanistic interpretability studies should distinguish four levels of evidence: description, prediction, intervention, and mediation.

1. Description assigns a human-readable pattern to an internal unit or direction. Examples that activate a latent can suggest a label, but a coherent label is not a causal explanation.
2. Prediction asks whether that label forecasts activations or behavior on held-out inputs. This tests generalization beyond the examples used to name the pattern.
3. Intervention changes an internal variable and measures the output. It can establish necessity or sufficiency under the stated intervention, not an unrestricted claim about the model.
4. Mediation asks whether the proposed variables and connections reproduce the relevant intervention effects of the model's natural computation. This is the strongest evidence that a proposed circuit captures how the behavior is normally produced (Geiger et al. 2025).

Figure 54.1 shows the progression. A study's headline must make only the strongest claim supported by the weakest link in its evidence.

The ladder also prevents a common category error. An intervention performed far outside the model's normal activation range may show that an internal direction can steer an output. It does not show that the model ordinarily uses that direction with the same magnitude or meaning. The intervention protocol, its range, and the comparison being tested belong in the claim itself.

Treat features and superposition as hypotheses

The architecture in Chapter 8 exposes tensors such as neurons, attention-head outputs, and residual-stream vectors. These are implementation units, not guaranteed units of meaning. A neuron may respond to several recognizable patterns, while a useful pattern may be distributed across many neurons. A polysemantic neuron can still carry predictive or causal information; its mixed selectivity only limits simple one-neuron-one-concept accounts.

In this chapter, a feature is an analyst-proposed variable that helps explain some model behavior. It might be a neuron, a direction in activation space, a sparse-autoencoder latent, or a more complex function. Its status is earned by evidence, not by its name or by the method that produced it.

The hypothesis is superposition: a model represents more features than it has dimensions by allowing non-orthogonal feature directions to share an activation space. A simple sparse linear model writes an activation as

$$x = \sum_{i=1}^m a_i v_i + \varepsilon .$$

Here $x \in \mathbb{R}^d$ is the activation vector at a declared internal site, $v_i \in \mathbb{R}^d$ is candidate feature direction $i$, $a_i$ is its feature coefficient, and $\varepsilon$ is the unexplained residual. The sparsity assumption is that only a small number $k = \lVert a \rVert_0$ of the $m$ coefficients are nonzero for a typical example, even when $m>d$. Because more than $d$ mutually orthogonal directions cannot fit in $\mathbb{R}^d$, some directions must then overlap.

Toy models demonstrate that this packing is possible and predict geometric changes as feature sparsity and importance vary (Elhage et al. 2022). They do not prove that every transformer activation has a unique linear, sparse decomposition, or that every human-recognizable concept is a direction. Superposition is therefore a productive model of the evidence, not a license to assume that a learned dictionary has recovered the model's native ontology.

Learn candidate features with a sparse autoencoder

A sparse autoencoder (SAE) learns an overcomplete dictionary from activations collected at one declared model site. Let $x \in \mathbb{R}^d$ be an activation, let the dictionary contain $m$ latents, and choose a target sparsity $k$. One TopK formulation is

$$\begin{aligned} z &= W_{\mathrm{enc}}(x-b_{\mathrm{dec}})+b_{\mathrm{enc}}, \\ f &= \operatorname{TopK}_k(z), \\ \hat{x} &= W_{\mathrm{dec}}f+b_{\mathrm{dec}} . \end{aligned}$$

The encoder matrix is $W_{\mathrm{enc}} \in \mathbb{R}^{m \times d}$, the decoder matrix is $W_{\mathrm{dec}} \in \mathbb{R}^{d \times m}$, $b_{\mathrm{enc}} \in \mathbb{R}^m$ is the encoder bias, and $b_{\mathrm{dec}} \in \mathbb{R}^d$ is the decoder bias and centering term. The vector $f \in \mathbb{R}^m$ has at most k nonzero entries. Depending on the implementation, TopK may select the largest positive pre-activations or the largest values after a nonlinearity; the study must state which rule it uses.

The basic TopK objective minimizes mean squared reconstruction error,

$$\begin{aligned} \mathcal{L}_{\mathrm{TopK}} &= \mathbb{E}_{x \sim \mathcal{D}} \left[\lVert x-\hat{x}\rVert_2^2\right], \\ \lVert W_{\mathrm{dec}}[:,i]\rVert_2 &= 1. \end{aligned}$$

The constraint gives unit-norm decoder columns. Without it, multiplying a decoder column by a constant and dividing the corresponding latent activation by the same constant leaves the reconstruction unchanged. This scale degeneracy makes latent magnitudes and sparsity penalties difficult to compare. Older L1 SAEs instead add $\lambda\lVert f\rVert_1$ to the reconstruction loss. TopK directly controls the number of active latents and avoids L1 shrinkage; the broader training recipe, not the TopK operation alone, determines dead latent rates and optimization quality (Gao et al. 2025).

An SAE fit is a measurement model, not a discovery certificate. Its decoder columns are candidate feature directions, and several independent checks are needed:

• Reconstruction fidelity: report normalized reconstruction error or explained variance on held-out activations.
• Downstream fidelity: replace the original activation with $\hat{x}$ and measure the change in task loss or behavior.
• Activation sparsity: report the distribution of active latents, not only the configured value of $k$.
• Latent use: count dead latents and unusually frequent latents.
• Stability across seeds: align dictionaries trained from different seeds and report which patterns recur.
• Semantic validity: evaluate automatic or human labels against held-out feature labels and adversarial counterexamples.
• Causal utility: compare latent interventions with neuron, random-direction, PCA, and supervised probing baselines.

No single metric certifies a dictionary. Low reconstruction error can coexist with unreadable latents, crisp labels can coexist with poor coverage, and a strong intervention can be off the model's ordinary activation distribution. Early SAE studies produced useful candidate features (Huben et al. 2024; Bricken et al. 2023). A later study extended the method to Claude 3 Sonnet (Templeton et al. 2024), but dictionary width does not guarantee atomic, complete, or canonical recovery.

Test causal relevance with interventions

Activation patching localizes information by comparing clean and corrupted runs. Begin with an input for which the model produces the target behavior and a controlled corruption that changes it. Run both inputs, patch one internal component from the clean run into the corrupted run, and measure how much of a declared output metric is restored.

For base input $x_b$, source input $x_s$, component $j$, model $F$, and scalar score $s$, define

$$\begin{aligned} \Delta_j(x_b,x_s) &= \\ &s\!\left(F_{\operatorname{do}(h_j \leftarrow h_j(x_s))}(x_b)\right) \\ &-s\!\left(F(x_b)\right). \end{aligned}$$

Here $h_j(x_s)$ is the source activation and $F_{\operatorname{do}(h_j \leftarrow h_j(x_s))}$ is the intervened model. A large $\Delta_j$ shows causal relevance to that contrast. It does not identify a unique mechanism: correlated components, nonlinear interactions, or a poor corruption can produce similar restoration scores.

Consequently, every patching result must report the corruption procedure, output metric, patch location and granularity, source and base examples, and normalization of the effect. Results can change materially across these choices (Zhang and Nanda 2024). Necessity tests, sufficiency tests, path patching, and ablations answer different questions. Agreement among them is stronger than a single heat map.

Feature steering uses the same discipline. Increasing the Golden Gate Bridge latent in Claude 3 Sonnet changed the model's outputs toward that topic (Templeton et al. 2024). This is evidence that the direction can affect the behavior under the intervention. It does not by itself show that the direction is unique, canonical, or the complete natural mediator of bridge-related behavior.

Recompose features into circuit hypotheses

A circuit is a causal subgraph relative to a behavior and input distribution. Its nodes may be attention heads, MLPs, neurons, or learned features; its edges propose how information flows between them. The circuit is an explanatory abstraction, so its quality depends on both predictive coverage and interventional faithfulness.

Work on transformer circuits supplied useful algebra for residual-stream writes, attention QK selection, and OV transformations (Elhage et al. 2021). Induction heads provide a concrete example: a preceding-token head and an induction head can compose to continue repeated token patterns. Training dynamics, ablation evidence, and cross-model observations provide evidence for an induction mechanism (Olsson et al. 2022). They are not proof that induction heads explain all in-context learning, especially in larger models and tasks that require more than token-sequence continuation.

Recent circuit-tracing work scales the search by fitting a replacement model whose MLP computations are approximated with sparse cross-layer transcoders, then constructing prompt-specific attribution graphs (Ameisen et al. 2025; Lindsey et al. 2025). The method uses fixed attention patterns for the traced forward pass and includes an error-correction term to improve fidelity. Each graph is therefore a local hypothesis about one computation in the replacement model. It is not a complete trace of the original model, and a large edge weight is not automatically an independently verified causal effect.

Attribution graphs become more persuasive when investigators perturb proposed nodes and paths, compare the replacement and original models, and test the mechanism on related prompts. Open tooling makes these checks easier to reproduce on selected open-weight models (Anthropic 2025). Competing replacement architectures should also be tested: on the evaluated models and automated metrics, transcoders improved the reconstruction-interpretability trade-off over comparable MLP-output SAEs (Paulo et al. 2025).

Measure what the method leaves out

Interpretability errors are not one-dimensional. A responsible report keeps the following failure modes visible.

• Reconstruction error is activation signal omitted by the SAE or replacement model. It is not automatically behaviorally important, but it cannot be called harmless without downstream and interventional tests.
• Feature splitting assigns one broad pattern to several narrower latents. This can reflect useful granularity or an unstable decomposition rather than an error by itself.
• Feature absorption occurs when a broader latent fails to activate because a more specific latent captures the example. This creates a recall hole for feature detection (Chanin et al. 2025).
• Non-identifiability means the dictionary is not unique. Different seeds, widths, or objectives can learn different decompositions of the same activations, so one SAE should not be treated as the canonical unit system (Leask et al. 2025).
• Weak baselines can make a new representation look useful. A supervised neuron probe, PCA direction, random dictionary, and task-specific probing baseline test whether the SAE adds value. The current negative evidence from sparse probing is scoped to that use case, not to every use of SAEs (Kantamneni et al. 2025).
• Distribution shift can invalidate both labels and interventions. A latent named from top-activating examples may behave differently on another input population, and aggressive steering may move off the model's ordinary activation distribution.
• Label error arises because automatic feature labels summarize selected examples. They can miss exceptions, import the labeler's concepts, or turn a graded pattern into a categorical name.
• Coverage limits make every attribution graph a local explanation. A prompt-level account should not silently become a global claim about a model, a task, or a training process.

These limits do not make the methods useless. They determine which claims the evidence can support. A study that reports reconstruction, downstream fidelity, baselines, stability, and interventions is much more informative than a gallery of intuitively appealing feature labels.

Make the study reproducible

The unit of work should be an InterpretabilityStudy, not an isolated feature screenshot. A minimal retained record is:

study_id: stable identifier
model_and_tokenizer_hashes: immutable revisions
behavior_and_output_metric: operational definition
input_population_and_splits: train, validation, and held-out sets
activation_sites: tensors, layers, positions, and preprocessing
method_and_code_revision: implementation and configuration
dictionary_width_and_sparsity: m, k or lambda, and selection rule
training_data_and_seeds: activation corpus and random seeds
baselines_and_validity_metrics: reconstruction, fidelity, stability, labels
intervention_protocol: corruption, patch, ablation, and steering ranges
retained_artifacts: dictionaries, activations, graphs, and evaluation outputs
claim_scope_and_known_failures: supported level and explicit exclusions

The record should preserve enough information to rerun feature extraction, reproduce aggregate metrics, and inspect both supporting and failing examples. For a safety-relevant study, independent reviewers should be able to change the input sample, seed, baseline, intervention magnitude, and output metric without reconstructing the pipeline from prose.

An InterpretabilityStudy is evidence for a larger decision process. It cannot authorize a safety decision on its own. Release or deployment decisions still need behavioral evaluations, threat models, monitoring, access controls, and a documented account of uncertainty.

Mechanistic interpretability can give oversight a better question than “did the model produce a bad output?” It can ask whether a tested internal mechanism is present, active, and causally relevant on a defined input population. That additional evidence is valuable when its scope and failure modes travel with it. The next layer, Chapter 55, must combine such evidence with controls that remain useful when the explanation is incomplete or wrong.

Further reading