Draft complete but not yet reviewed; expect revisions

Questions and Methods

The previous part asked what an overseer can learn by watching a model act: evaluations, audits, lie detectors, chains of thought, and debates all treat the model as something that produces behavior to be judged (Part VI). Each runs into the same wall. A model can behave differently when it recognizes a test (Chapter 37), its stated reasoning need not be the computation that produced its answer (Section 40.3), and a judge weaker than the model cannot always tell a good answer from a convincing one (Chapter 41). This part turns to the one source of evidence that does not pass through the model's behavior: its internal activations and weights, which the developer can read, copy, and change at will.

Interpretability is the research program that tries to turn that access into understanding. Access alone is not understanding: a network's weights are billions of numbers, and every one of them can be printed without explaining anything. This chapter is about the step between the two. It teaches the kinds of experiment the field runs on a network's internals. Above all, it teaches the standard that says when such an experiment shows how a network computes, so that the later chapters of this part can say exactly what a feature, a circuit, or a steering vector has been shown to be. The standard comes with a warning from neuroscience, the older science of reading networks nobody designed: its standard methods, tried on a system whose workings were known, fell short of describing how that system computes.

43.1 What interpretability asks #

Each method in this chapter answers a different question, and much confusion about what interpretability has shown comes from reading an answer to one as an answer to another. This section states the questions first.

43.1.1 The objects #

The raw material is the network's internal state. An activation is the vector of numbers that one layer of a network outputs for one input; the book writes it $\vh$, with $m$ entries. A neuron or unit is one entry of such a vector, the output of one of the simple functions a layer is built from (Section 8.2). In a transformer, recall from Section 11.2.3, the residual stream is the running vector at each position that every block reads from and adds its output to, and an attention head is a component that moves information between positions by a learned, weighted lookup. Because each block adds to the stream rather than overwriting it, the output can be split into the contributions of individual components, which is what makes a component-by-component study of a transformer possible (Section 11.2).

43.1.2 Four questions #

Take a model that completes "The Eiffel Tower is located in the city of" with "Paris." Four different questions can be asked about it.

1. Representation. What information does a part of the network carry? Is the tower's location present in the activation at the token "Tower," somewhere in the middle layers?
2. Mechanism. What does the network do with that information? Which components retrieve the location and move it to the last position, where the next token is predicted?
3. Explanation of a case. Why did this input produce this output? Which parts of the prompt, and which components, were responsible for "Paris" here, rather than for answers in general?
4. Verification. Does the network contain a computation its developer did not intend, such as a hidden objective that shows up only on some inputs?

The first three are scientific questions about a given behavior. The fourth is the one this book needs, and it is harder in kind: it asks about the absence of something across inputs nobody has tried. The methods below were built for the first three. Whether they can answer the fourth is the question Chapter 47 takes up.

43.1.3 Faithful and plausible #

Explainable machine learning asked an older version of these questions, mostly for users who needed to trust or contest a model's decision. Zachary Lipton argued that the term "does not reference a monolithic concept," and separated transparency, "how does the model work?", from post-hoc explanations, "what else can the model tell me?" (Lipton, 2018). Finale Doshi-Velez and Been Kim, in a 2017 preprint, defined interpretability as the "ability to explain or to present in understandable terms to a human" and asked that each claim be tested by an evaluation matched to it (Doshi-Velez and Kim, 2017).

The distinction that matters most here was set out by Alon Jacovi and Yoav Goldberg in an opinion piece at ACL 2020, drawing on earlier work. In their words, "'Plausibility' refers to how convincing the interpretation is to humans, while 'faithfulness' refers to how accurately it reflects the true reasoning process of the model" (Jacovi and Goldberg, 2020). They add that "it is possible to satisfy one of these properties without the other," and that "a plausible but unfaithful interpretation may be the worst-case scenario" (Jacovi and Goldberg, 2020). A plausible account of why a model answered "Paris," written by a person or by another model, can be false of the model; a faithful account can be unreadable. The methods of this chapter aim at faithfulness, and the standard of Section 43.5 is a way to measure it.

43.1.4 Who the answers serve #

The book distinguishes three relationships that "alignment" can name (Section 1.3), and interpretability results bear on them differently. Most results concern a deployed model: what it computes on the inputs its users send. Inspecting checkpoints during training, to catch a disposition forming that no one intended, concerns the training process and the signal it optimizes, and needs evidence about how structure changes over training rather than about a finished network. The third relationship, the developer and the public, enters through access: a developer's interpretability finding about its own model is evidence for an outside party only to the extent that the party can check it, which depends on who may read the weights (Section 43.7).

The principal here holds a power with no counterpart in the older theories of delegation. No employer can read an employee's neurons, rerun them on the same question, or edit one belief and watch what follows. Section 14.8 records that the delegation models assume a principal who sees outcomes rather than actions and cannot rerun the agent, an assumption that reverses for a developer and its model. The book listed a principal that reads its agent's activations among the results it would test the delegation thesis against (Section 1.7). This part reports what that power has delivered so far.

43.1.5 Names are hypotheses #

Interpretability papers name components and directions after what they seem to track: a curve detector, a truth direction, a deception feature. Each name is a hypothesis about what the component tracks, adopted because it predicts when the component will be active on inputs the namer has not yet seen. This is the intentional stance of Section 1.5 applied inside the network: the book uses such a name only together with the test behind it, and the later chapters report that test. A name earned by inspecting the inputs that most strongly activate a unit is the weakest kind, for a reason the neuroscience of Section 43.6 makes vivid. A stronger name rests on several tests. The Circuits essay's case that certain units of an image model detect curves rested on seven arguments, from images optimized to excite the units to the units' weights and what later units do with their output (Olah et al., 2020b). Its authors judged that the arguments "seem to meet the evidentiary standards we understand to be used in neuroscience" (Olah et al., 2020b).

The program also uses much vocabulary of its own, which the book marks as the program's. Mechanistic interpretability names the attempt to explain a network by the computations it performs rather than by its input-output behavior, in an early statement "attempting to reverse engineer the detailed computations performed by transformers" (Elhage et al., 2021a) (developer report). The open-problems review notes that the term is used "in a number of (sometimes inconsistent) ways" (Sharkey et al., 2025). Feature and circuit take the senses used here from the Circuits work at OpenAI, whose 2020 essay proposed, in claims its authors called "deliberately speculative," that features are the units of a network and circuits are features connected by weights (Olah et al., 2020b) (Section 3.6.1). Monosemantic and polysemantic describe a unit that responds to one concept or to several (Section 44.1). And the developer behind the attribution graphs of Chapter 45 describes its tools as a microscope and its subject as the "biology" of a language model, while warning that "Like any microscope, our tools are limited in what they can see" (Lindsey et al., 2025) (developer report). The glossary records each term (Appendix A).

43.2 Reading: probes and what decodability shows #

The first question asks what a part of the network carries. The simplest instrument for it is a probe: a small classifier, usually linear, trained to predict a property of the input from a layer's activations. A probe reads; it changes nothing in the network. This section teaches what its accuracy shows, the two ways that accuracy misleads, and the readouts that decode activations into the model's own vocabulary.

43.2.1 The idea #

Guillaume Alain and Yoshua Bengio, in a 2016 preprint, fit linear classifiers, "which we refer to as 'probes'," to every layer of image classifiers, "trained entirely independently of the model itself" (Alain and Bengio, 2016). The probes "do not affect the model training. They only measure the level of linear separability" of the classes at each layer (Alain and Bengio, 2016). In Inception v3 and ResNet-50, trained to classify ImageNet images, they report that "the linear separability of features increase monotonically" with depth; the body of the paper says "almost perfectly monotonic" (Alain and Bengio, 2016). Language processing arrived at the same tool at about the same time: Yonatan Belinkov's review of probing names studies from 2015, on word embeddings, as perhaps the first that can be cast as probing, and says the idea was "concurrently proposed" for vision by Alain and Bengio (Belinkov, 2022).

In a language model the property can be a word's part of speech, a sentence's truth, or whether a prompt looks like a test. If a probe predicts it well on new inputs, the property is decodable from that layer.

43.2.2 What the accuracy shows #

A linear probe computes a score $\vv^\T \vh + v_0$ from an activation $\vh$, where $\vv$ is a direction in activation space and $v_0$ an offset, and answers "yes" when the score is positive. A logistic probe reports $\sigma(\vv^\T \vh + v_0)$ as a probability, where $\sigma$ is the logistic function, which maps any score to a number between 0 and 1. Training chooses $\vv$ and $v_0$ to fit labeled examples. Before asking what a well-fitting probe reveals, it helps to see how easily a probe fits.

Accuracy on the points used to fit a probe therefore says nothing until the number of points is large compared with the dimension. Accuracy on held-out points, which the probe never saw, removes this freedom. Two further gaps remain, and the literature has measured both.

43.2.3 The probe may do the work #

The first gap is that a capable probe can learn a task itself. John Hewitt and Percy Liang, in a paper at EMNLP 2019, put it this way: "as long as a representation is a lossless encoding, a sufficiently expressive probe with enough training data can learn any task on top of it" (Hewitt and Liang, 2019). Their control is a control task, which assigns each word type a random but fixed label, so that a probe can succeed only by memorizing which word gets which label (Hewitt and Liang, 2019). A good probe is selective: accurate on the real task and poor on the control, and "Selectivity is defined as the difference between linguistic task accuracy and control task accuracy" (Hewitt and Liang, 2019).

On part-of-speech tagging from the first layer of ELMo, an earlier language model, the numbers separate two probes that look equally good:

The more flexible probe gains a tenth of a point on the real task and more than twenty on the control (Hewitt and Liang, 2019). The paper's abstract states that "popular probes on ELMo representations are not selective," a claim that applies to the flexible probes (Hewitt and Liang, 2019). Dropout did not raise their selectivity; smaller probes and other regularization did (Hewitt and Liang, 2019). Even the linear probe reached 71.2 percent on random labels, so selectivity is a matter of degree, not a pass mark.

Two 2020 papers dispute the criterion. Tiago Pimentel and colleagues, at ACL, treat probing as estimating the information a representation carries about a property, and conclude that "one should always select the highest performing probe one can, even if it is more complex" (Pimentel et al., 2020). If a network's representations determine the sentence, as they assume, the representations hold no more information about a property than the sentence itself, and they ask of selectivity, "why should we punish memorization?" (Pimentel et al., 2020). Elena Voita and Ivan Titov, at EMNLP, keep the worry but change the measure: their probe reports how many bits it takes to transmit the labels given the representations, which charges for the effort a probe needs (Voita and Titov, 2020). In Hewitt and Liang's own setting, this codelength was stable across probe settings where accuracy was not (Voita and Titov, 2020). The dispute is over whether a probe should measure the information present or how easily it can be extracted, a reading Pimentel and colleagues acknowledge (Pimentel et al., 2020). Neither measure shows that the network uses the information (inference).

43.2.4 Decodable is not used #

The second gap is between what is present and what the network relies on. Belinkov states it as the main limitation of probing: the method "may indicate correlations" between representations and a property, "but it does not tell us whether this property is involved in predictions of" the model (Belinkov, 2022). Geiger and colleagues built a small network in which a probe reads a quantity perfectly from units the output never uses (Geiger et al., 2021), and unit $h_6$ in Figure 43.1 is the same construction.

Yanai Elazar and colleagues tested the gap on a real model. Amnesic probing removes a property from a layer, here with an algorithm that repeatedly projects out the directions linear classifiers use to detect it, and measures how much the network's own predictions suffer (Elazar et al., 2021). On BERT's prediction of words left unmasked in its input, the setting probing studies use, across six properties including part of speech, dependency labels, and named entities, "conventional probing performance is not correlated to task importance"; the rank correlation across the six was not significant (p = 0.871) (Elazar et al., 2021). The properties did matter. Removing dependency labels or fine-grained part of speech cut word-prediction accuracy by 87.0 and 81.8 points, though the authors note that the first removal also took out some unrelated features (Elazar et al., 2021). Removing phrase boundaries, which probes decode well, slightly improved it (Elazar et al., 2021). Probe accuracy did not rank the properties by their use. The authors caution that the method removes "only linearly-present information" (Elazar et al., 2021). LEACE, a later eraser, is "a closed-form method which provably prevents all linear classifiers from detecting a concept while changing the embedding as little as possible" (Belrose et al., 2023b). Rerunning the BERT test with it, also on unmasked words, a group that includes one of Elazar's coauthors found the part-of-speech effect concentrated in layer 11 and agreed that "probing does not generally correlate with intervention-based analysis techniques" (Belrose et al., 2023b).

43.2.5 Reading in the model's own vocabulary #

A transformer ends by multiplying its final residual stream by an unembedding matrix, which turns the vector into a score for every token (Section 11.2). In a 2020 forum post, the pseudonymous author nostalgebraist applied the same readout to intermediate layers of GPT-2 and found that "the resulting distributions make intuitive sense" (nostalgebraist, 2020). They "gradually converge to the final distribution over the layers of the network," and "At some point in the middle, GPT will have formed a 'pretty good guess' as to the next token" (nostalgebraist, 2020). The author's own caveat: "The logit lens show us some of what is going on, not all of it" (nostalgebraist, 2020). The post names the method the logit lens.

The tuned lens, in a 2023 preprint by Nora Belrose and colleagues, trains "an affine probe for each block in a frozen pretrained model," a learned linear map plus offset per layer, before the unembedding (Belrose et al., 2023a). The maps are trained to reproduce the model's final output, so that "the probes are not incentivized to learn extra information over and above what the model has learned," the concern of Section 43.2.3 (Belrose et al., 2023a). On models with "up to 20B parameters," the authors report it "more predictive, reliable and unbiased than the logit lens" (Belrose et al., 2023a). A later version of the paper adds that the lenses in its earlier versions were "severely undertrained" (Belrose et al., 2023a).

Patchscopes, from Google Research in a paper at ICML 2024, generalize the idea. They decode a hidden representation "by patching it into a different inference pass (of the same or a different LLM) that encourages the translation of that specific information" (Ghandeharioun et al., 2024). The authors show that methods projecting representations into the vocabulary, the two lenses among them, and several intervention methods "can be viewed as instances of this framework" (Ghandeharioun et al., 2024). Its caveat is the one this section has been building: the approach "does not guarantee that the patched representation itself stores all that information," since the reading pass can add its own (Ghandeharioun et al., 2024). Readouts in words have since become a branch of the program (Section 43.7.2).

43.2.6 Probes in practice #

Probes are cheap, and safety work uses them widely. Earlier chapters report probes for false statements and lies (Section 39.5; Section 39.6), for whether a prompt looks like an evaluation (Section 37.3.2), and for whether a backdoored model is about to act on its trigger (Section 34.3.4). A probe used as a monitor is a detector, and its value depends on its error rates at the base rate of what it detects (Section 6.7); Section 46.4 judges such probes by the standards forensic science set for detectors. The open-problems review states the limit: probes "detect correlations instead of causal variables," so probing "should be used only to generate hypotheses" (Sharkey et al., 2025) (preprint).

43.3 Attribution: gradients as cheap counterfactuals #

The third question asks why this input produced this output. Attribution answers it by assigning each input feature, or each internal component, a share of responsibility for the output. The cheapest share comes from a gradient, and this section teaches attribution as what a gradient is: a first-order estimate of what an intervention would do. Seen that way, its failures are predictable.

43.3.1 The idea #

The gradient of an output with respect to an input is the vector of rates at which the output changes as each input entry moves, the quantity gradient descent follows (Section 8.3). It answers a counterfactual question for every input entry at once: how much would the output change if this entry changed a little? Karen Simonyan and colleagues at Oxford, in a paper at ICLR, turned it into a picture. Their saliency map for an image classifier is the gradient of a class's score with respect to the pixels, so that "the magnitude of the derivative indicates which pixels need to be changed the least to affect the class score the most" (Simonyan et al., 2013). The same paper visualized a class by optimizing an image to maximize its score, which they credit to earlier work (Simonyan et al., 2013); feature visualization in the Circuits work applies the idea to single units (Olah et al., 2020b).

A gradient describes the output only near the input. Integrated gradients, from Mukund Sundararajan and colleagues at Google in a paper at ICML 2017, replace the local slope with an average along the straight path from a baseline input, such as a black image, to the actual input (Sundararajan et al., 2017). The authors proposed two axioms an attribution method should meet, which they call Sensitivity and Implementation Invariance, and showed that "they are not satisfied by most known attribution methods" (Sundararajan et al., 2017). Their method also satisfies completeness: the attributions add up to the difference between the output at the input and at the baseline (Sundararajan et al., 2017). The causal-abstraction paper of Section 43.5.5 shows that the same computation, with the baseline set to a patched value, measures an intervention exactly (Geiger et al., 2025).

43.3.2 Sanity checks #

A map that highlights the right pixels can still be uninformative about the model, and two tests from Google Brain showed how. Julius Adebayo and colleagues, in a paper at NeurIPS 2018, compared each method's map on a trained network with its map on the same network after its weights were randomized, and on a network trained on randomly permuted labels (Adebayo et al., 2018). Gradients and Grad-CAM passed. Guided backpropagation and Guided Grad-CAM produced maps that stayed similar until the layers closest to the input were randomized, and so failed (Adebayo et al., 2018). The authors' conclusion is that "visual inspection is a poor guide in judging whether an explanation is sensitive to the underlying model and data" (Adebayo et al., 2018). Pieter-Jan Kindermans and colleagues, in 2017, built two networks that compute exactly the same function on inputs shifted by a constant, and showed that "a transformation with no effect on the model can cause numerous methods to incorrectly attribute" (Kindermans et al., 2017b). Integrated gradients passed or failed depending on its baseline (Kindermans et al., 2017b). The two papers share two authors and one laboratory. A group at Facebook AI later reran the randomization checks. On images, the variant of integrated gradients that multiplies by the input was insensitive to randomization while the variant without that multiplier was not, and "the conclusions made for image do not directly transfer to text" (Kokhlikyan et al., 2021) (preprint).

Attribution has nonetheless found real defects. The best-known case, a classifier that recognized horses by a source tag in the corner of the image, is reported in Section 8.4.2 (Lapuschkin et al., 2019). What the sanity checks add is that a map's plausibility is not evidence that it reflects the model, the distinction between faithful and plausible of Section 43.1.3.

43.3.3 Attribution patching #

Activation patching, taught in full in Section 43.4, runs a model on a clean input that produces a behavior and a corrupted input that does not, copies one component's activation from one run into the other, and measures how much of the behavior moves with it. It measures a component's effect directly, but each patch costs a forward pass, and a model with thousands of heads and millions of connections between them cannot be patched one piece at a time on every prompt. Attribution patching replaces each patch with its first-order estimate.

Aaquib Syed, Can Rager, and Arthur Conmy, in a 2023 workshop paper, applied the estimate to every connection between components. On average it recovered previously identified circuits better than an automated search that patches connections one at a time, "while requiring just two forward passes and a backward pass" (Syed et al., 2023); Section 45.3 reports the comparison. On the one task where they compared the estimate with actual patching, the correlation was "fairly weak," and the fitted slope suggested that the estimate made effects look about twice as large as patching measured them (Syed et al., 2023).

The derivation's last step names one failure. János Kramár and colleagues at Google DeepMind, in a 2024 preprint on Pythia models from 410 million to 12 billion parameters, found "two classes of failure modes of AtP which lead to significant false negatives," the components a ranking by the estimate would miss (Kramár et al., 2024). They call them "attention saturation," the flat region of the softmax in the example above, and "cancellation," where a component's direct and indirect effects offset (Kramár et al., 2024). Their corrected method, AtP*, recomputes the attention softmax instead of approximating it and applies dropout to the backward pass, and they add "a method to bound the probability of remaining false negatives" (Kramár et al., 2024). They also state the limit: "there is no guarantee that AtP* produces no false negatives" (Kramár et al., 2024). On 16 models from InterpBench, a 2024 set of small transformers trained to implement known circuits, plain edge attribution patching did worse than patching-based search, while a variant using integrated gradients matched it (Gupta et al., 2024) (preprint). The authors note that the benchmark models' gradients were very small, which may explain the gap (Gupta et al., 2024).

43.4 Intervening: ablation, patching, and mediation #

A probe shows what is present and an attribution estimates what would change. The methods that carry most mechanistic claims do the experiment instead: they set an internal activation to a chosen value, run the rest of the network, and watch the output. Setting one variable by hand while the rest of the system follows its usual rules is what causal inference calls an intervention, and a network is a rare system in which every variable can be intervened on exactly.

43.4.1 Mediation #

An input affects an output in two ways: directly, and through intermediate components, called mediators, that it changes along the way. Jesse Vig and colleagues, in a paper at NeurIPS 2020 read here in its expanded arXiv version, brought causal mediation analysis to neural networks "by treating internal model components, e.g., specific neurons or attention heads, as mediators between model inputs and model outputs" (Vig et al., 2020). Three quantities organize the analysis, in the form Judea Pearl gave them in 2001 so that they apply to nonlinear models as well as linear ones (Pearl, 2001). The total effect of changing the input from $x$ to $x'$ is the change in the output. The direct effect changes the input but holds the mediator at the value it took on $x$. An indirect effect, Pearl noted, cannot be isolated by holding variables constant (Pearl, 2001). The indirect effect instead keeps the input at $x$ and sets the mediator to the value it takes on $x'$, measuring the part of the change that flows through the mediator (Vig et al., 2020; Geiger et al., 2025, Definition 57). The indirect effect is an interchange intervention in the sense of Section 43.5.

Their case was gender bias. A prompt such as "The nurse said that" can be continued with "she" or "he," and the ratio of the probabilities a model gives the anti-stereotypical and the stereotypical pronoun measures its bias on that prompt (Vig et al., 2020). Replacing "nurse" with "man" is the change of input, and each neuron or attention head is a candidate mediator. In GPT-2 small, on one of its datasets, intervening on 10 of the 144 attention heads matched the effect of intervening on all of them, and the first 6 heads selected were in layers 4 and 5 (Vig et al., 2020). In a randomly initialized model the effects were much smaller, "indicating that it is the training that causes gender bias" (Vig et al., 2020). The expanded version also reports that the effects are "synergistic, amplified or repressed by different components," and that splitting the total effect into direct and indirect parts needed an extra condition to hold (Vig et al., 2020).

43.4.2 Ablation and patching #

Two interventions do most of the work. Ablation removes a component's contribution: it sets the component's output to zero, to its average over a dataset, or to its value on a different input, the last called resample ablation (Geiger et al., 2025, sec. 3.5). Activation patching runs the model on two inputs, a clean one that produces the behavior and a corrupted one that does not, and copies one component's activation from one run into the other. The direction matters. Copying clean into corrupted, which Stefan Heimersheim and Neel Nanda call denoising, asks whether a component is sufficient to restore the behavior; copying corrupted into clean, noising, asks whether it is necessary (Heimersheim and Nanda, 2024) (preprint). They stress that "these two directions can be very different, and are not just symmetric mirrors of each other" (Heimersheim and Nanda, 2024).

The best-known patching study located stored facts. Kevin Meng and colleagues, in a paper at NeurIPS 2022, ran GPT-2 XL on prompts naming a subject whose associated fact the model knew (Meng et al., 2022). They corrupted the run by adding noise to the embeddings of the subject's tokens, then restored one internal state at a time to its clean value and measured how much of the correct answer's probability came back, a procedure they call causal tracing (Meng et al., 2022). On 1,000 known facts, the noise lowered the correct token's probability by 18.6 percentage points on average, and restoring the single hidden state at layer 15 of the last subject token recovered 8.7 points (Meng et al., 2022). Restoring MLP outputs recovered up to 6.6 points and attention outputs at that token 1.6, which the authors read as suggesting "a decisive role for MLP modules" (Meng et al., 2022). The MLP and attention traces restore windows of ten consecutive layers, because restoring single modules "had generally negligible effects" (Meng et al., 2022).

They then edited the weights of one middle-layer MLP with a rank-one change, a method they call ROME, to rewrite a single fact. On a new dataset of 21,919 counterfactual statements, ROME "simultaneously maintains both specificity and generalization," changing the targeted fact and its paraphrases while leaving unrelated facts alone, which other editing methods traded off (Meng et al., 2022). The authors' inference was strong: "This evidence suggests that we have an accurate understanding not only of where factual associations are stored, but also how" (Meng et al., 2022). Path patching, which patches the connection between two components, carried the indirect-object circuit in GPT-2 small that Section 45.2 reports (Wang et al., 2023c).

43.4.3 When the result depends on the method #

Each of these experiments has choices that the result can depend on, and four kinds of dependence have been measured.

The corruption and the metric. Fred Zhang and Neel Nanda, in a paper at ICLR, compared corrupting with noise, as causal tracing does, against replacing the key tokens with related ones, and compared output metrics (Zhang and Nanda, 2023). They found that "varying these hyperparameters could lead to disparate interpretability results" (Zhang and Nanda, 2023). The clear peak of MLP effects around layer 16 of GPT-2 XL under noise "is not salient at all" under token replacement, though the authors do not claim that the results under noise are illusory or inflated (Zhang and Nanda, 2023). A metric based on the correct answer's probability "must fail to detect negative model components, if corruption reduces the correct token probability to near zero" (Zhang and Nanda, 2023). A separate group, testing methods that discover circuits automatically, found them "very sensitive to the corrupted distribution" and to "hyperparameter and metric choice" (Conmy et al., 2023). Faithfulness scores for whole circuits turned out just as sensitive to how the removed parts are ablated, as Section 45.3 reports (Miller et al., 2024).

Location is not leverage. Peter Hase and colleagues, at NeurIPS 2023, asked whether the layer causal tracing points to is the layer where an edit works best. On GPT-J, it is not: tracing results "do not provide any insight into which model MLP layer would be best to edit" (Hase et al., 2023). Once the choice of edit layer was accounted for, the tracing effect explained "only 0.1% of the variance in edit success" (Hase et al., 2023). The authors read this as tracing answering "a different question than model editing does," not as tracing being wrong (Hase et al., 2023). Either way, the ROME paper's use of editing success as evidence for its localization did not survive the test (inference).

Self-repair. Thomas McGrath and colleagues at Google DeepMind, in a 2023 preprint, studied a 7-billion-parameter model built to the design of the laboratory's Chinchilla models (McGrath et al., 2023; Hoffmann et al., 2022). They found that "ablations of one attention layer of a language model cause another layer to compensate," which they call the Hydra effect (McGrath et al., 2023). The measurements resample-ablated whole layers on 1,209 factual-recall prompts from the CounterFact dataset (McGrath et al., 2023). With late MLP layers that damp the most likely token, the compensation restored about 70 percent of the drop in the answer's logit at middle layers (McGrath et al., 2023). They "caution against straightforwardly interpreting low effectiveness of ablations as meaning that a network component is unimportant" (McGrath et al., 2023). Cody Rushing and Neel Nanda, at ICML 2024, ablated single heads in Pythia, GPT-2, and Llama-7B models and found self-repair across them, "imperfect" and "noisy, since the degree of self-repair varies significantly across different prompts" (Rushing and Nanda, 2024). They measured how much of a head's direct effect on the correct token's logit was restored, over the 2 percent of tokens where that effect was largest (Rushing and Nanda, 2024). Its causes include the final normalization layer's rescaling and small sets of neurons that change from prompt to prompt, and the authors write that it "makes ablations an unreliable tool" (Rushing and Nanda, 2024). They name as a limitation that they measured the repair of direct effects only, and that how far it extends to other measures of a head's role is unclear (Rushing and Nanda, 2024). The two studies share a team: McGrath's coauthors were at Google DeepMind, and Rushing's project was supervised by Nanda (Rushing and Nanda, 2024), a member of that laboratory's interpretability team (Nanda et al., 2025). A smaller precedent comes from another group: the indirect-object circuit's backup heads, which took over when the main ones were ablated (Wang et al., 2023c).

The arithmetic of the bias is simple enough to write down.

A patch can create what it claims to find. Aleksandar Makelov and colleagues, at ICLR 2024, studied patches of a subspace: a few directions inside a component's activation rather than the whole component. Such a patch can change behavior "by activating a dormant parallel pathway" rather than by moving the feature the subspace is said to hold (Makelov et al., 2024a). Their example is a one-dimensional subspace of an MLP layer in GPT-2 small whose patching changed the model's answers on the indirect-object task, in a layer the original circuit study had found to contribute almost nothing (Makelov et al., 2024a). They found that subspaces of the residual stream are "less prone to illusions" and recommend checks beyond the end-to-end effect (Makelov et al., 2024a).

The parallel with neuroscience's lesion studies is exact enough that the 2025 causal-abstraction paper introduces ablation through them (Geiger et al., 2025), and Section 43.6 reports what lesions recovered from a system whose design was known.

43.5 When an intervention licenses an explanation #

The interventions of Section 43.4 say which components matter for an output. A mechanistic explanation claims more: that the network carries out a particular algorithm, with a particular intermediate quantity held in a particular place. "The network adds the first two numbers, holds the sum in these units, and compares it with the third" is such a claim. This section teaches the theory that says when an intervention experiment licenses it. The rest of this part uses it as the standard of a correct explanation.

43.5.1 An explanation is a smaller causal model #

A causal model, in the sense used here, is a set of variables, each computed from others by a fixed rule, so that one can ask what would happen if a variable were set by hand. A trained network is already one: every activation is a variable, computed from earlier activations by the weights. As an explanation of itself it is correct and useless, since it has billions of variables. An explanation is a second causal model, with a few variables a person can read, such as "the sum of the first two inputs" or "which name appeared twice." The question is when the small model is a faithful simplification of the large one.

Causal abstraction is the theory of that relation, developed first for causal models in general. Rubenstein and colleagues, in a 2017 preprint, asked when two models of one system at different levels of detail agree "in their predictions of the effects of interventions" (Rubenstein et al., 2017). Beckers and Halpern, in a paper at AAAI 2019, gave a sequence of "successively more restrictive definitions of abstraction for causal models" (Beckers and Halpern, 2019). Atticus Geiger and colleagues brought the theory to neural networks in a paper at NeurIPS 2021 (Geiger et al., 2021). In a 2025 paper in the Journal of Machine Learning Research, a Stanford-led group presents it as "a theoretical foundation for mechanistic interpretability" (Geiger et al., 2025).

The relation needs an alignment: a pairing of each variable of the small model with a set of the network's units, together with a rule that reads the variable's value off those units (Geiger et al., 2025, Definition 34). Under an alignment, the small model abstracts the network when every intervention on a high-level variable has the same effect on the output as the corresponding intervention on its units.

43.5.2 The test: interchange interventions #

Which interventions should be compared? Setting units to arbitrary numbers can push the network into states it never reaches on real inputs, where its behavior says little about how it computes. The test therefore uses only values the network itself produces. An interchange intervention runs the network on a base input and fixes the aligned units to the values they take on a different source input; the small model makes the matching change, setting its variable to the value it takes on the source (Geiger et al., 2021; Geiger et al., 2025, Definition 49). This is activation patching from Section 43.4, with a hypothesis attached that predicts what the patch should do.

The 2021 paper's illustration is a network that adds three digits (Geiger et al., 2021). Suppose it first computes $x + y$ in some set of units. With base input $(1, 2, 3)$ and source $(4, 5, 6)$, the hypothesis predicts that patching those units from the source gives $4 + 5 + 3 = 12$ (Geiger et al., 2021). If the patched network outputs the predicted sum, the source's first two digits plus the base's third, for every pair of base and source, then every intervention on the intermediate sum has an equivalent intervention on the units, and the hypothesis holds for that variable (Geiger et al., 2021).

Exact agreement on every pair is the ideal, and in a 2023 paper Geiger and colleagues write that it "rarely obtains in practice" (Geiger et al., 2023). The working measure is graded.

43.5.3 Exploring an abstraction #

Figure 43.1 runs the test on a network small enough to see whole. It reads three integers $a$, $b$, and $c$ from 0 to 4 and outputs 1 when $a + b > c$. Its weights are set by hand: the first layer holds $a$, $b$, and $c$ in units $h_1$, $h_2$, $h_3$ (each $h_i$ is one entry of an activation vector $\vh$), and the second computes $h_4 = a + b$, $h_5 = c$, and $h_6 = c - b + 4$. The output compares $h_4$ with $h_5$ and ignores $h_6$. Two algorithms compute the same function. Algorithm A forms $S = a + b$ and asks whether $S > c$; algorithm B forms $D = c - b$ and asks whether $a > D$. Because there are only 125 inputs, the figure runs every one of the 15,625 ordered pairs of base and source, so each score is exact.

Under algorithm A, aligning $S$ with $h_4$ scores an IIA of 1.000, as it should: $h_4$ is where the network forms the sum. Aligning $S$ with $h_1$ alone, a plausible guess since $h_1$ feeds the sum, scores 0.776. That looks respectable until one counts how many pairs are informative. The algorithm predicts a changed output on only 32.3 percent of pairs, so an alignment that changes nothing at all scores 0.677. On the informative pairs, $h_1$ is right half the time. Whoever reports an IIA should report its floor, or the accuracy on pairs where the hypothesis predicts a change.

Under algorithm B, the obvious candidate for $D$ is $h_6$, which holds $c - b$ plus a constant: a probe trained on $h_6$ would decode $D$ perfectly. Its IIA is 0.677, the floor, and on the informative pairs it is 0, because the output never reads $h_6$. This is the gap of Section 43.2 between decodable and used, now measured. Yet algorithm B is not refuted. Aligning $D$ with the pair $(h_2, h_3)$, read as $h_3 - h_2$, scores 1.000, because the first layer still holds $b$ and $c$ separately, and patching both changes the output exactly as B predicts. The network is abstracted by both algorithms, under alignments at different layers. Nothing in the test chooses between them; only a prior decision about where an intermediate variable may live could.

43.5.4 Distributed alignments #

The alignments above pair a variable with whole units. In trained networks a quantity is often spread across many units, each of which takes part in several quantities, which Chapter 44 explains. Distributed alignment search (DAS), introduced by Geiger and colleagues in 2023, finds the alignment by gradient descent in a rotated coordinate system, so that a variable can align with a direction shared across neurons (Geiger et al., 2023). On a network trained to judge whether two pairs of objects have the same relation, DAS found perfect or near-perfect alignments with the algorithm that computes both equalities first. The best alignment to groups of whole neurons found by brute-force search reached an IIA of 0.60, and the neuron-aligned alignment closest to the one DAS learned reached 0.73 (Geiger et al., 2023).

The same paper ran a control that matters for everything below. On frozen, randomly initialized networks that solve nothing, DAS could not raise IIA when the hidden layer was small. The authors read this as a check that DAS cannot construct new behavior from random structure. When the hidden layer was made orders of magnitude larger than the input, though, IIA rose: an oversized hidden layer "allows DAS to manipulate the model behavior by searching through a large space of random mechanisms" (Geiger et al., 2023). How much freedom the alignment is given is part of what a score means.

43.5.5 One language for the methods #

The 2025 paper's main claim is that the methods of this chapter are special cases of one analysis. It lists "activation and path patching, causal mediation analysis, causal scrubbing, causal tracing, circuit analysis, concept erasure, sparse autoencoders, differential binary masking, distributed alignment search, and steering" (Geiger et al., 2025). It also gives a precise formalization of polysemantic neurons, of the linear representation hypothesis that concepts are directions in activation space (Section 46.1), of modular features, and of graded faithfulness (Geiger et al., 2025). A few of the translations explain results of earlier sections:

• Causal mediation is abstraction by a three-variable chain, input to mediator to output, and its direct and indirect effects are interchange interventions (Geiger et al., 2025, sec. 3.4.1).
• Ablation is abstraction by a three-variable model in which a binary variable records whether a component was removed; the paper introduces it through neuroscience's lesion studies (Geiger et al., 2025, sec. 3.5).
• A linear probe defines a coordinate system in which the concept might be localized, so it can be tested as an alignment. But "successfully probing a hidden vector does not guarantee that it plays a causal role in model behavior" (Geiger et al., 2025, sec. 3.6.3).
• Integrated gradients, an attribution method, can compute an interchange intervention when its baseline is set to the patched value, though "computing integrals is an inefficient way to compute interchange interventions" (Geiger et al., 2025, sec. 3.3.2).
• Steering is control rather than explanation: "While a successful interchange intervention analysis implies an ability to control the low-level model, the reverse does not hold" (Geiger et al., 2025, sec. 3.7). A steering vector can push activations off the distribution that inputs produce, so a vector that moves behavior need not be where the network holds the concept.

The unification is why the later chapters can ask the same question of every method: which interchange interventions has the claimed explanation passed, on which inputs, and under what restrictions on the alignment? A feature found by a sparse autoencoder (Section 44.5), a circuit (Section 45.1), and a steering vector (Section 46.2) are each claims of this form.

43.5.6 What the standard does not give #

The figure already showed the limit: two algorithms passed the test on one network. Méloux and colleagues, in a paper at ICLR 2025, found the same in small networks trained on Boolean functions, where every candidate explanation could be enumerated. In their words, "several algorithms can be causally aligned with the neural network, and a single algorithm can be causally aligned with different subspaces of the network" (Méloux et al., 2025). Sutter and colleagues, at NeurIPS 2025, proved that if the alignment map may be arbitrarily nonlinear, then "under reasonable assumptions, any neural network can be mapped to any algorithm" (Sutter et al., 2025). Causal abstraction, they suggest, "becomes vacuous without assumptions about how models encode information" (Sutter et al., 2025). A passing test therefore shows that an explanation is faithful under a stated class of alignments, not that it is the explanation. Section 47.2 reports these results in full and asks what they mean for verification.

The standard also answers a question about one finished network. A different question is how its structure formed during training. Hoogland and colleagues proposed in a 2023 forum post to study it as developmental interpretability, through the phase transitions of training (Hoogland et al., 2023), using the tools of singular learning theory taught in Section 10.2 and the transitions of Section 9.4.

43.6 Reading a network nobody designed: neuroscience #

From neuroscience. Interpretability is not the first science to infer computation from the activity of a network that no one designed. Neuroscience records the activity of single neurons, removes or silences parts of the brain to see what fails (a lesion study), and asks which stimuli make a neuron fire (its tuning). These are the probe, the ablation, and the feature label of this chapter under older names. The field has also tested its methods on a system whose answer was known, which interpretability can learn from directly.

43.6.1 The microprocessor test #

Eric Jonas and Konrad Kording, in a 2017 paper in PLOS Computational Biology, took a classical microprocessor "as a model organism" (Jonas and Kording, 2017). The MOS 6502 ran the Apple I, the Commodore 64, and the Atari video game system. The Visual6502 team had reconstructed it transistor by transistor from images of the chip, capturing 3,510 transistors and inferring 1,018 more from the circuit's topology, and their simulator tracks the state of every transistor at every step (Jonas and Kording, 2017). The "behaviors" were three games, Donkey Kong, Space Invaders, and Pitfall. The authors then applied standard analyses of neuroscience to the chip, among them its wiring diagram, lesions of single transistors, tuning curves, correlations between transistors, and averaged regional activity (Jonas and Kording, 2017).

The lesions make the point most directly. Each forced one transistor permanently on (Jonas and Kording, 2017). Of the transistors whose lesion stopped a game from drawing its first frame, 1,560 broke all three games, 200 broke two, and 186 broke exactly one (Jonas and Kording, 2017). A neuroscientist might conclude that there is a Donkey Kong transistor. The authors' verdict: "This finding of course is grossly misleading. The transistors are not specific to any one behavior or game but rather implement simple functions, like full adders" (Jonas and Kording, 2017). Tuning failed in the same way. A few transistors were strongly tuned to the brightness of the most recently displayed pixel. The authors knew that the five they showed were not directly related to the brightness of the pixel being written, so the apparent tuning was "not really insightful about their role" (Jonas and Kording, 2017).

Their summary is that the methods "reveal interesting structure in the data but do not meaningfully describe the hierarchy of information processing in the microprocessor" (Jonas and Kording, 2017). They draw two conclusions. The first is about method: analyses should be checked on "complex non-linear dynamical systems with known ground truth" before they are trusted on brains (Jonas and Kording, 2017). The second is about where the fault lies: "the problem is not that neuroscientists could not understand a microprocessor, the problem is that they would not understand it given the approaches they are currently taking" (Jonas and Kording, 2017). The authors take the idea from an earlier essay by Yuri Lazebnik (Lazebnik, 2002), which, in their account, imagined biologists shooting identical radios with metal particles to see which damaged part breaks which function, and held that understanding means being able to fix a broken system (Jonas and Kording, 2017).

Each failure has a counterpart in this chapter. The Donkey Kong transistor is an ablation result read as a function, the reading Section 43.4 warns against when components compensate or serve many behaviors. The tuned transistor is a feature named from the inputs that activate it, the inference Section 43.1 treats as a hypothesis. The authors also note that a brain can reroute around damage, "which can allow regions to take over for damaged areas" (Jonas and Kording, 2017), the counterpart of self-repair.

43.6.2 Levels of explanation #

What would count as understanding? Jonas and Kording recall the levels of analysis they credit to David Marr and Tomaso Poggio in 1982: what problem a system solves, the algorithm by which it manipulates internal representations, and the physical implementation that carries the algorithm out (Jonas and Kording, 2017). The levels are the framework of Marr's book Vision (Marr, 1982). Geiger and colleagues place mechanistic interpretability at the middle level: it seeks the mediating processes "at a slightly higher-level of abstraction," what "Marr (1982) famously called the algorithmic level of analysis" (Geiger et al., 2025). In the terms of Section 43.5, the high-level model is the algorithm and the alignment says how the network implements it.

A second answer denies that a trained network's weights are the right place to look. Timothy Lillicrap and Kording, in a 2019 preprint, rest their argument on compressibility: a description is useful when it is much shorter than the thing described. They write that "We may thus expect broadly applicable intelligent systems to be hard (or impossible) to compress into a small size" (Lillicrap and Kording, 2019). What is short is the recipe. The architecture, the loss, the learning rule, and the optimizer "are all easily expressed in small human understandable equations" (Lillicrap and Kording, 2019). Their conclusion is hedged: "for the moment we best understand neural networks in terms of their mechanisms for development and learning," and it "may thus eventually be possible to meaningfully understand a network trained on ImageNet" (Lillicrap and Kording, 2019). If they are right, the algorithmic level the causal-abstraction standard asks for may not exist in a compact form for large networks (inference).

43.6.3 Lessons the field is relearning #

Two position papers carry neuroscience's methodological lessons into interpretability. Martina Vilas and colleagues, at ICML 2024, observe that "a great number of inner interpretability studies seem to equate mechanistic explanations solely with localizing model components that have a causal effect" (Vilas et al., 2024). Neuroscientists, they write, "have long argued that this type of work is insufficient," and they cite earlier work on "the difficulty of inferring cognitive function from neural recordings" (Vilas et al., 2024). Their remedy is the test the microprocessor illustrated. A method should be judged by whether it can falsify a mechanistic hypothesis "about a system whose design principles are known," and by the question "if the hypothesized mechanism is absent, how likely is the method to reveal its absence?" (Vilas et al., 2024). That second question is the fourth question of Section 43.1 in neuroscience's words.

A 2024 preprint by Zhonghao He and fifteen colleagues makes the parallel with probes explicit. In neuroscience, decoding a quantity from recordings gives "no guarantee that the decoded information is actually necessary for performing the task, unless causal manipulations are done in addition to decoding" (He et al., 2024d), the lesson of Section 43.2. Following an argument by Krakauer and colleagues (Krakauer et al., 2017), they hold that understanding comes from "the careful theoretical and experimental decomposition of behavior alongside corresponding algorithmic and neural representations" (He et al., 2024d). By their reading, much interpretability work so far sits at the implementation level, the lowest of Marr's three (He et al., 2024d).

43.6.4 What it implies, and where the analogy stops #

The microprocessor test implies a rule: validate a method on networks whose mechanism is known before trusting it on networks whose mechanism is not. Interpretability has built the counterparts. Tracr, from a Google DeepMind group, compiles programs written in a small language for transformer computations into the weights of a standard transformer, so that "the known structure of Tracr-compiled models can serve as ground-truth for evaluating interpretability methods" (Lindner et al., 2023). Its authors add the limit: compiled models give each feature its own coordinate, which "is unrealistic, and makes the resulting models easy to interpret," so evaluations on them are "a minimum bar rather than a full validation of a technique" (Lindner et al., 2023). Auditing games plant a hidden objective in a model and ask blind teams to find it, as in the developer's study reported in Section 38.2 (Marks et al., 2025) (developer preprint).

The analogy stops in two directions. In the first, it reverses. An interpretability researcher can record every unit on every input, intervene exactly and as often as desired, copy the network, and compare it with versions trained differently. Geiger and colleagues set this against weather, brains, and economies: "we can measure and manipulate the microvariables of deep learning models with perfect precision and accuracy" (Geiger et al., 2025). The microprocessor gave Jonas and Kording the same access only because the chip had been reconstructed and simulated (Jonas and Kording, 2017). The reversal also changes the problem of Section 21.3, where a model's outward markers were weak evidence because they were selected to resemble human text: inspection may supply evidence about a model's internal states that the study of animals never had (inference).

In the second direction, the test does not carry over. The microprocessor was a designed system with a decomposition known in advance. For the processor, the authors write, every module has a name, a known place on the chip, and a known dependence of its outputs on its inputs (Jonas and Kording, 2017). A trained network has no agreed decomposition against which to grade a method. The test can therefore be run only on networks built or trained to contain a known mechanism, and whether those resemble what ordinary training produces is the validity question of model organisms (Section 5.3) (inference). Neuroscience can also lean on the kinship between the brains it studies and the brains of the people studying them. A network's training objective is known, but its learned features owe nothing to a shared ancestry, and whether they match human concepts at all is the question of Section 46.6 (inference).

43.7 The program in 2026 #

The methods of this chapter come from different moments, and the later chapters of this part report results from groups with different aims. This section dates the program, as of October 2026, so that each result can be placed.

43.7.1 The main line, 2020 to 2024 #

The methods of this chapter come from two communities. Naomi Saphra and Sarah Wiegreffe, in a 2024 preprint, trace an interpretability community in natural-language processing and a separate, parallel one formed by AI safety researchers who took the name mechanistic interpretability. They argue that the term's several meanings are "the product of a critical divide within the interpretability community" (Saphra and Wiegreffe, 2024). The second community's main line began with the Circuits work on image models at OpenAI (Olah et al., 2020b) (Section 3.6.1). A 2021 report from Anthropic carried it to transformers, comparing its aim to how "a programmer might try to reverse engineer complicated binaries into human-readable source code" (Elhage et al., 2021a) (developer report). Its authors studied attention-only models of at most two layers and wrote that, for "much larger and more realistic models," they remained "a very long way from being able to fully reverse engineer such models" (Elhage et al., 2021a). The finding that a network represents more concepts than it has neurons, and the sparse dictionaries built to recover them, followed, and Chapter 44 teaches both.

43.7.2 The branches since 2025 #

Since 2025 the program has divided into branches that bet on different units of explanation and judge success by different standards; the division is the book's reading of the record (inference). A review of the field's open problems, written by 29 authors from 19 institutions and synthesized by Apollo Research, describes the field's methods as two broad approaches (Sharkey et al., 2025) (preprint). The first, "often called 'reverse engineering'," decomposes the network into components and then identifies their functions; the second, "concept-based interpretability," starts from concepts the network might use and looks for them (Sharkey et al., 2025). The branches below sit at different points between the two.

Replacement models and attribution graphs. Anthropic's 2025 methods report replaces a model's MLP layers with a more interpretable component and traces one prompt's computation through the result, with attention patterns frozen (Ameisen et al., 2025) (developer report). Section 45.4 teaches the replacement model, and Section 45.5 reports what the graphs showed in the developer's production model and how often they showed anything (Lindsey et al., 2025).

Readouts in words. Instead of decomposing activations, train a model to describe them. Activation oracles, in a 2025 preprint by Adam Karvonen and colleagues at MATS and Truthful AI with coauthors at Anthropic, who advised the project and ran its experiments on Anthropic's Claude Haiku 3.5, one of four models tested, are "models trained to answer arbitrary natural-language questions about LLM activations" (Karvonen et al., 2025). Anthropic's natural language autoencoders pair a model that maps an activation to text with one that reconstructs the activation from the text, trained jointly (Anthropic, 2026ai) (developer report). The developer notes that the training "does not explicitly incentivize NLA explanations to be interpretable or faithful" (Anthropic, 2026ai). Section 46.7 reports both.

Task-judged interpretability. In March 2025 the Google DeepMind interpretability team reported, in a progress update on the laboratory's own Gemma 2 9B IT model, that probes built on sparse autoencoder features "underperformed linear probes" at detecting harmful intent in user prompts out of distribution (Smith et al., 2025b). The team wrote that it was "deprioritising fundamental SAE research for the moment" (Smith et al., 2025b). In December it described "a strategic pivot over the past year, from ambitious reverse-engineering to a focus on pragmatic interpretability," judged by progress on proxy tasks tied to safety goals, and advocated "method minimalism": start with "the simplest methods (e.g. prompting, steering, probing, reading chain-of-thought)" (Nanda et al., 2025). An older top-down program, representation engineering, "places representations, rather than neurons or circuits, at the center of analysis" (Zou et al., 2023a) (preprint). Section 44.7 reports how dictionaries fared on such tasks, and Chapter 46 the probes and steering vectors.

Decomposing the weights. A fourth branch decomposes the parameters rather than the activations. Attribution-based parameter decomposition, from Apollo Research in 2025, splits a network's weights into components that are faithful to the original, few in number for any input, and simple (Braun et al., 2025) (preprint). Its authors say it "is currently only appropriate for studying toy models" (Braun et al., 2025). A successor, stochastic parameter decomposition, is "more scalable and robust to hyperparameters," and was demonstrated on models "slightly larger and more complex" than before, still toy models (Bushnaq et al., 2025) (preprint). Section 45.6 reports the branch.

The open-problems review gives the field's own account of what it cannot yet do. It groups the open problems into methods, applications, and socio-technical questions, and it is candid about practice: "Conflating hypotheses with conclusions has regrettably been commonplace in mechanistic interpretability research, making validation an important area for the field to improve" (Sharkey et al., 2025). It also grants that "interpretability-based methods do not currently outperform black-box baselines" (Sharkey et al., 2025), a question Section 47.7 takes up.

43.7.3 Who can do the work #

Every method in this chapter needs the weights or the activations, so who may read them decides who can check a claim. Two releases lowered the cost for groups outside the laboratories. Google DeepMind's Gemma Scope released sparse autoencoders for its own open Gemma 2 models, on all layers of the 2B and 9B models and on select layers of the 27B (Lieberum et al., 2024). Its reason: "research applications outside of industry are limited by the high cost of training a comprehensive suite of SAEs" (Lieberum et al., 2024). NNsight and the National Deep Inference Fabric, from a Northeastern University group, let outside researchers run experiments on the internals of large open-weight models on shared hardware (Fiotto-Kaufman et al., 2024) (preprint). Their authors note the limit: the service "does not provide access to closed-parameter, proprietary models" (Fiotto-Kaufman et al., 2024). A developer's interpretability finding about a closed model therefore reaches the public as testimony (inference). Casper and colleagues argue, in a paper at FAccT 2024, that white-box access and outside-the-box access, to information about training and deployment, "allow for substantially more scrutiny than black-box access alone" (Casper et al., 2024a); Section 47.7 takes up what this means for audits.

43.8 Settled, contested, missing #

43.9 Exercises #

Further reading #

• Geiger et al. (2025): causal abstraction proposed as the common theory of the intervention methods.
• Vig et al. (2020): causal mediation brought to neural networks.
• Jonas and Kording (2017): what standard methods recovered from a system whose workings were known.
• Sharkey et al. (2025): the field's own map of what it cannot yet do.
• Alain and Bengio (2016): the linear probe, the simplest instrument of the part.