Draft complete but not yet reviewed; expect revisions

Superposition and Features

The methods of the previous chapter need a unit to work on: a direction to probe, a component to patch, a variable to align with an algorithm (Section 43.4; Section 43.5). The natural unit is the neuron, one coordinate of a layer's output after its nonlinearity, and in trained networks it is often the wrong one. A single neuron in a vision model or a language model responds to several unrelated things, and a concept the network clearly uses is spread across many neurons. Explanations built on neurons are tangled before they start.

This chapter explains why, and what has been built in response. The explanation is geometric. A space of a few thousand dimensions holds exponentially more almost-perpendicular directions than it has dimensions. When the things a network needs to represent are rarely active at the same time, it can give each its own direction and tolerate the small overlaps. That geometry is the chapter's formal object, and the chapters on circuits, steering, and the limits of inspection recall it. The tool built to undo superposition, sparse dictionary learning, recovers directions that people find far easier to read than neurons. Several groups have also shown that the dictionary it returns is one of many that fit about equally well, so the "features" of a model are partly a choice the analyst makes.

Superposition and the non-uniqueness of learned features are on the list of results the book fixed, before drafting, as having no counterpart in the older fields (Section 1.7); this chapter tests that entry against neuroscience's theory of sparse codes. Most of its claims concern the second of the book's three relationships, the deployed system and the people it serves, here in the person of an overseer who must check it, since they are about what an inspector can see inside a model. Some concern the third, the developer and the public: the largest dictionaries are developers' reports on their own proprietary models, which outsiders cannot repeat, while open dictionaries on open models can be checked by anyone.

44.1 Polysemantic neurons and the superposition hypothesis #

Interpretability would be simpler if each neuron detected one thing. In some networks many neurons do, and that fact made the first circuit analyses in vision models possible. This section starts from the observation that many neurons do not, and from the hypothesis offered to explain it.

44.1.1 The observation #

In a 2020 article in the journal Distill, Chris Olah and colleagues, then at OpenAI, introduced the program of reading circuits in the weights of image classifiers (Olah et al., 2020b). Many neurons responded to one recognizable thing, such as a curve, and some did not. InceptionV1, an image classifier, "contains one neuron that responds to cat faces, fronts of cars, and cat legs" (Olah et al., 2020b). Their visualizations showed that the neuron was not detecting some shared property: it looked for the eyes and whiskers of a cat, for furry legs, and for shiny car fronts (Olah et al., 2020b). They called such neurons polysemantic, and explained why they hurt: "If one neuron with five different meanings connects to another neuron with five different meanings, that's effectively 25 connections that can't be considered individually" (Olah et al., 2020b).

A neuron that responds to one thing is monosemantic. Both words belong to the vocabulary of this interpretability program, and the glossary records them as such (Appendix A). Both presuppose a notion of the "thing" a neuron responds to, which the program calls a feature: a property of the input that the network computes and uses, such as "this token is in Arabic script". The term has no agreed definition. Nelson Elhage and colleagues, at Anthropic and Harvard, weighed three candidates in 2022 and wrote with the last in mind: features are "properties of the input which a sufficiently large neural network will reliably dedicate a neuron to representing" (Elhage et al., 2022b). They call the definition "slightly circular" (Elhage et al., 2022b), and the circularity returns at the end of this chapter.

Language models show the same mixture. In a one-layer transformer studied by Trenton Bricken and colleagues at Anthropic, the developer that trained it, a single neuron "responds to a mixture of academic citations, English dialogue, HTTP requests, and Korean text" (Bricken et al., 2023). Researchers at Meta studied the developer's own family of language models, from 125 million to 66 billion parameters, and found the early layers sparse (Voita et al., 2023). In some layers of the largest model, more than 70 percent of the neurons in the MLP layers, the small two-layer networks inside each transformer block, never activated on a large and diverse collection of text, and many of the rest detected single tokens or short token sequences (Voita et al., 2023). Wes Gurnee and colleagues trained sparse probes, classifiers allowed to read only a few neurons, on seven models from 70 million to 6.9 billion parameters. They found that early layers use sparse combinations of neurons "to represent many features in superposition", while middle layers have "seemingly dedicated neurons" for some higher-level features (Gurnee et al., 2023). In a later study of GPT-2 models trained from five random initializations, only 1 to 5 percent of neurons were universal, in the sense of activating on the same inputs across initializations, and those usually had clear interpretations (Gurnee et al., 2024).

44.1.2 The hypothesis #

The explanation Elhage and colleagues offered starts by moving the unit of analysis from neurons to directions. Suppose each feature is represented by a fixed direction in the space of a layer's activations, and the activation vector is the sum of those directions, each scaled by how strongly its feature is present. The authors call this a linear representation, and they hold the view to be supported by both evidence and theory (Elhage et al., 2022b). It does not require directions to line up with neurons. If the network has more features to represent than it has dimensions, the directions cannot all be perpendicular, and every neuron will carry a share of several features. Elhage and colleagues named this superposition and stated the hypothesis in their own words: networks "want to represent more features than they have neurons", so they exploit a property of high-dimensional spaces "to simulate a model with many more neurons" (Elhage et al., 2022b). The verb "want" is their shorthand for what training favors, not a claim about the network's aims. In the authors' formal version, a linear representation is in superposition when the matrix of inner products of its feature directions cannot be inverted, which happens when there are more features than dimensions (Elhage et al., 2022b).

The idea was older than the name. Sanjeev Arora and colleagues showed in 2016 that the several senses of an ambiguous word "reside in linear superposition" within its word embedding, and that sparse coding can recover vectors approximating each sense (Arora et al., 2016). In 2021, Zeyu Yun, Yubei Chen, Bruno Olshausen, and Yann LeCun used dictionary learning to describe a transformer's internal vectors as "linear superpositions of transformer factors" (Yun et al., 2021). Olshausen is the neuroscientist whose sparse coding model of the visual cortex this chapter returns to in Section 44.5.7.

A software engineer can read the hypothesis through an analogy Olah drew in an informal 2022 note at Anthropic. Reverse engineering a compiled program requires splitting memory into variables, and a reverse engineer without variable names "must assign meaning to positions in memory" (Anthropic, 2022). Neuron activations play the role of memory and features the role of variables. A monosemantic network is a program whose every byte holds one variable. A network in superposition stores its variables as overlapping patterns across many bytes, readable only by a decoder that knows the patterns.

44.1.3 Why neurons are sometimes aligned with features #

If features are directions, why would any neuron be monosemantic? Elhage and colleagues answer with the idea of a privileged basis. In a word embedding, applying any invertible linear map to the vectors, and its inverse to the next layer's weights, gives an identical model, so no coordinate is special. An elementwise nonlinearity breaks that symmetry and can encourage features to align with the neurons, though it does not guarantee that they will (Elhage et al., 2022b).

The residual stream of a transformer, the running vector that every layer reads from and adds to (Section 11.2.3), was expected to have no privileged basis, since every read and write passes through an arbitrary linear map. A 2023 note from Anthropic's interpretability team reported that this expectation "is false in practice": the stream's coordinates had far heavier tails than random directions would give (Elhage et al., 2023). The team ruled out layer normalization, the rescaling of each activation vector, and floating-point precision as causes and "provisionally" concluded that "the per-dimension normalizers in the Adam optimizer are to blame" (Elhage et al., 2023). Adam, the usual training method, scales each parameter's update separately. Whether a network's coordinates mean anything is therefore a fact about how it was trained, which an inspector has to check, not assume (inference).

One more distinction prepares the geometry. In a 2023 note, Olah separated composition, which gives independent features their own directions so they combine freely, from superposition, which packs more items than dimensions and needs a nonlinearity to read them back. With $m$ binary neurons one can store $m$ freely composable features, or exponentially many items that never occur together (Anthropic, 2023d). Whether superposition pays thus depends on how often features occur together, which is the quantity the next section makes exact.

44.2 Why many features fit in few dimensions #

A space of $m$ dimensions has room for only $m$ mutually perpendicular directions. How can it hold many more features without hopelessly confusing them? The answer has three parts. High-dimensional spaces contain enormous numbers of nearly perpendicular directions. Reading a feature off such a direction picks up a little of every other active feature, which is harmless when few are active at once. And a nonlinearity with a threshold can remove what remains.

44.2.1 The idea: two features in one dimension #

Start with the smallest case, which Elhage and colleagues solved in closed form (Elhage et al., 2022b). Two features must share one dimension. Store the first at $+1$ and the second at $-1$, so that the stored value is $f_1 - f_2$, where $f_1, f_2 \ge 0$ are the features' strengths. Read the first feature as $\mathrm{ReLU}(f_1 - f_2)$ and the second as $\mathrm{ReLU}(f_2 - f_1)$, where $\mathrm{ReLU}$, the rectified linear unit of Section 8.2.2, returns its input when positive and zero otherwise. If only the first feature is active, the first read-out returns $f_1$ and the second returns $\mathrm{ReLU}(-f_1) = 0$: exact. The same holds when only the second is active. The two read-outs fail only when both features are active at once, and then both underestimate their features by the smaller of the two strengths, so the weaker feature reads zero.

This arrangement, which the authors call an antipodal pair, turns each feature's interference with the other into a negative number, and the ReLU discards negative numbers for free (Elhage et al., 2022b). Its cost is paid only on inputs where both features occur together. If each is active on one input in a hundred, independently, both are active on one input in ten thousand. A linear read-out has no such escape: whatever leaks from one feature into the other's read-out passes straight through.

44.2.2 Nearly orthogonal directions #

The general case replaces the two opposite directions with many directions that are nearly perpendicular. Write the features present on an input as a vector $\vf \in \R^{n_{\text{f}}}$, one entry per feature, with $n_{\text{f}}$ features in all, and give feature $i$ a unit direction $\vw_i \in \R^m$. The activation vector is the sum $\vh = \sum_i f_i \vw_i = \mW \vf$, where $\mW$ has the $\vw_i$ as its columns. Reading feature $i$ by projecting onto its own direction gives

The first term is the signal. The sum is the interference: every other active feature leaks in, weighted by the inner product of the two unit directions, which is the cosine of the angle between them. Perpendicular directions have cosine zero and do not interfere. How many directions can be almost perpendicular at once?

The exponential count is what Elhage and colleagues invoke: only $m$ directions can be exactly orthogonal, but exponentially many can have cosine similarity below a small $\varepsilon$ (Elhage et al., 2022b). They point for it to a lemma of William Johnson and Joram Lindenstrauss (Elhage et al., 2022b; Johnson and Lindenstrauss, 1984), which Micah Adler and Nir Shavit also cite for the observation that merely representing $n_{\text{f}}$ features can take as few as $O(\log n_{\text{f}})$ neurons (Adler and Shavit, 2024).

44.2.3 The price of interference #

Plentiful directions are not enough, because the interference in Equation (44.1) adds up over every active feature. Sparsity is what keeps the sum small. Let the sparsity $S$ be the probability that a feature is zero on a given input, the parameter Elhage and colleagues use (Elhage et al., 2022b), so that $1 - S$ is the probability that it is active.

The derivation makes the section's claim concrete: sparsity, more than the dimension alone, sets how many features a layer can hold. The same space that holds a hundred thousand rare features with modest noise would be swamped by a hundred thousand common ones.

44.2.4 Why a nonlinearity is needed #

The noise still has to be removed. A linear read-out cannot do it: whatever interference enters the projection reaches the output. A read-out with a negative bias $b_i$ and a ReLU, $\mathrm{ReLU}(\vw_i^\T \vh + b_i)$, outputs zero whenever the projection stays below $-b_i$. With the bias set at a few standard deviations of the interference, an inactive feature reads as zero on almost every input, and an active one is reported with its strength reduced by $|b_i|$. This is the role Elhage and colleagues found for the negative bias in their toy model: it "allows models to discard small amounts of noise" (Elhage et al., 2022b).

Without the nonlinearity, superposition does not pay. A linear model of the same shape learns to keep only the $m$ most important features, along perpendicular directions, as principal component analysis (which finds the directions of greatest variance) would, and superposition is never optimal for it (Elhage et al., 2022b). With the ReLU, the authors summarize, sparse features allow "compression beyond what a linear model would do, at the cost of 'interference' that requires nonlinear filtering" (Elhage et al., 2022b). The toy model of the next section shows the trade being made.

44.2.5 The ancestry, and where it stops #

The mathematics of storing sparse vectors in few dimensions is older than neural networks. Compressed sensing (Donoho, 2006) studies when a vector projected into fewer dimensions can still be recovered, which becomes possible when the vector is known to be sparse (Elhage et al., 2022b). Elhage and colleagues set their toy model beside it. In various compressed-sensing settings, a sparse vector can be recovered from $m$ measurements if and only if $m$ is at least of the order of its number of nonzero entries times a logarithmic factor (Elhage et al., 2022b). With sparsity $S$ there are about $(1 - S)\,n_{\text{f}}$ nonzero entries, so the number of features that can be recovered grows linearly in $m$, by a factor that rises with sparsity (Elhage et al., 2022b). The two problems also differ. Compressed sensing "recovers vectors by solving an optimization problem using general techniques", while the network must recover them with a single layer, so the authors judge compressed-sensing algorithms "in principle, much more powerful" (Elhage et al., 2022b). A network therefore tolerates less interference than the theory of recovery allows, and the recovery bounds are upper bounds on how much superposition a network can use (inference, following the authors' reading).

Two cautions follow. A recovery algorithm far stronger than the network could find structure the network itself cannot read, a worry that returns with dictionary learning (Section 44.5). And these derivations assume independent features of similar density in random directions, while real features co-occur, vary in density, and are placed by training.

44.3 Toy models: the phase diagram and the polytopes #

The geometry says what is possible. Whether training finds it, and which arrangement it finds, needs a model small enough to watch. Elhage and colleagues built one, and the work reported below builds on it.

44.3.1 The toy model #

The toy model of superposition compresses a vector of synthetic features through a narrow layer and asks for it back (Elhage et al., 2022b). Each input is a feature vector $\vf \in \R^{n_{\text{f}}}$ whose entries are zero with probability $S$ and otherwise uniform on $[0, 1]$. The model stores it in $m < n_{\text{f}}$ dimensions as $\vh = \mW \vf$ and reconstructs it as

with a bias vector $\vb$, training $\mW$ and $\vb$ to minimize the squared error weighted by each feature's importance $\nu_i$, a scalar multiplier on that feature's error: $\E\big[\sum_i \nu_i (f_i - \hat f_i)^2\big]$ (Elhage et al., 2022b). The same matrix writes and reads, so its column $\vw_i$ is the direction in which feature $i$ is stored, and its length says how strongly. Removing the ReLU gives the linear model it is compared against. The features stand in for those of an imagined larger network with no superposition; the toy asks how a smaller one would imitate it.

The authors' first experiment used 20 features in five dimensions, with importance falling by 30 percent from one feature to the next (Elhage et al., 2022b). The linear model "always learns the top-m most important features", the analog of keeping the top principal components. The ReLU model did the same on dense inputs, but as sparsity grew it began to store more features than dimensions, starting with the less important ones. It first arranged them "in antipodal pairs", then moved to other geometric structures as it stored more (Elhage et al., 2022b).

44.3.2 The phase diagram #

Whether a feature is stored at all, stored in superposition, or given a dimension of its own turned out to change abruptly. Elhage and colleagues trained models with two features and one dimension, varying the second feature's relative importance and the sparsity of both, ten models at each point (Elhage et al., 2022b). Sparsity was necessary for superposition and interacted with relative importance, and closed-form losses for the three natural solutions (drop either feature, or store both as an antipodal pair) showed "a first-order phase change": the best solution switched discontinuously where two losses crossed (Elhage et al., 2022b). The authors mean a discontinuous change, not the physicist's limit of infinite system size (Elhage et al., 2022b).

With many identical features, the stored directions formed regular geometric figures, polytopes in the authors' term, the higher-dimensional relatives of polygons and polyhedra. In models with 400 features and 30 dimensions, each stored feature's share of a dimension clustered at a few exact fractions: 3/4, 2/3, 1/2, 2/5, and 3/8 (Elhage et al., 2022b). The authors matched them to a tetrahedron, a triangle, an antipodal pair, a pentagon, and a square antiprism (Elhage et al., 2022b). These are among the arrangements that solve the Thomson problem, the old puzzle of placing charged points on a sphere so that they repel least, to which the authors liken the toy model's loss when inputs are very sparse (Elhage et al., 2022b). They were candid about how far this goes. The structure "seems 'too elegant to be true'" and may be partly idiosyncratic to the toy, and whether the geometry carries over to real networks is "much more uncertain" than whether superposition does (Elhage et al., 2022b).

44.3.3 Exploring the toy model #

Figure 44.1 runs the experiment in its smallest interesting form: five features, two dimensions, importance falling by a tenth per feature. Pooled over 20 random starts with the settings shown, the model behaves as the theory predicts at the ends of the range and less tidily in between. At sparsities up to 0.3, every run stores features 1 and 2 on perpendicular axes and drops the other three, as the linear model does at every sparsity tried. At $S = 0.7$, 18 of the 20 runs store four features as two antipodal pairs at right angles. At $S = 0.95$ and $0.99$, all 20 runs store all five features with neighboring arrows 70 to 74 degrees apart, close to the 72 degrees of a regular pentagon. At $S = 0.9$, 18 runs do, and the other two stay at the square of four. Between $S = 0.4$ and $0.6$ the runs disagree: they store two to four features, and which ones varies with the random start, a sign that gradient descent settles in different local minima of the same loss.

44.3.4 Independent and follow-on results #

Adam Scherlis and colleagues, at Redwood Research and the Flatiron Institute, explained the same allocations through capacity, the fraction of an embedding dimension a feature consumes, which Elhage and colleagues had called its dimensionality (Scherlis et al., 2022). In their toy model the optimal allocation of capacity "tends to monosemantically represent the most important features, polysemantically represent less important features (in proportion to their impact on the loss), and entirely ignore the least important features" (Scherlis et al., 2022). Polysemanticity was more prevalent when inputs had higher kurtosis (heavier tails) or sparsity, and the efficient embeddings had a block-semi-orthogonal structure whose small blocks correspond to the polytopes of Elhage and colleagues (Scherlis et al., 2022). Their model differs in two ways that bear on how far the agreement reaches: it uses squared activations to make the mathematics tractable, and it studies computation rather than compression (Scherlis et al., 2022). The paper is a preprint.

Zhongtian Chen, Edmund Lau, Jake Mendel, Susan Wei, and Daniel Murfet, mostly at the University of Melbourne, analyzed the toy model with two hidden dimensions in a 2023 preprint, in the limit where every input has exactly one active feature (Chen et al., 2023b). They proved that regular polygons of four to eight feature directions are critical points of the loss, places where its gradient vanishes, so that training can stall there (Chen et al., 2023b). With six features, they sampled the Bayesian posterior, the distribution over weights that Bayesian inference assigns after $n$ training examples (Section 10.2), and observed it switch from pentagons to hexagons between 600 and 700 examples. The free-energy formula of singular learning theory, which trades a solution's loss against its learning coefficient (an exponent that is smaller when more nearby parameters fit nearly as well), predicted the switch near 600 (Chen et al., 2023b). Their method for classifying posterior samples is, in their words, "not error-free" (Chen et al., 2023b).

They then trained the same model by stochastic gradient descent, gradient descent on small random batches of examples, from a square of feature directions plus noise. The loss fell in plateaus at the critical points they had classified, for example from squares through pentagons, and each drop came with a rise in the estimated learning coefficient (Chen et al., 2023b). The two kinds of transition did not coincide everywhere: the pentagon-to-hexagon switch of the posterior was not observed during gradient descent, which the authors say may occur "just with low probability per SGD step" (Chen et al., 2023b). That gradient descent's transitions have Bayesian counterparts is their Bayesian antecedent hypothesis, a proposal rather than a finding (Chen et al., 2023b). Section 10.2 reports the study briefly as evidence for singular learning theory, whose evidence comes almost entirely from one research community, so it stands as reported, not settled. The square that two runs of Figure 44.1 fail to leave at $S = 0.9$ has the shape of one of the critical points they classify (inference).

Adam Jermyn, Nicholas Schiefer, and Evan Hubinger reported in a preprint that toy models "can be made more monosemantic without increasing the loss by just changing which local minimum the training process finds" (Jermyn et al., 2022). Superposition is thus partly a matter of which solution training reaches, not only of what the loss forces. A note by Tom Henighan and colleagues at Anthropic, on superposition of training examples in a memorizing regime, is recalled in Section 9.2 (Henighan et al., 2023).

44.3.5 What the toy model does not show #

The toy model is evidence that superposition can arise in a trained network under stated conditions, not that real networks meet them. Its features are sparse, independent, and of fixed importance. Elhage and colleagues ask how realistic their toy models are, and judge superposition and the mix of monosemantic and polysemantic neurons "likely to generalize" while the geometry is less certain (Elhage et al., 2022b). The first assumption to fail in practice was independence. Studying a real one-layer transformer, Bricken and colleagues found features that are highly correlated and produce similar effects on the output, which "causes real models to have both denser and more structured superposition" than the toy models showed (Bricken et al., 2023). Section 44.6 shows what correlated, nested, and curved features do to the tools built on the toy picture.

44.4 Computing in superposition #

The toy model stores features and reads them back. A network must also compute with them: combine two features into a third, route one to where it is needed, suppress another. If computation required each feature to have a neuron of its own, superposition would be a storage trick and every computation would have to unpack it first. This section reports the evidence that computation itself can run in superposition, and the bounds on how far.

44.4.1 Computation in the toy model #

Elhage and colleagues gave a first demonstration (Elhage et al., 2022b). A model with a ReLU hidden layer of 40 neurons learned the absolute values of 100 sparse features, made signed for this task, each nonzero value drawn uniformly from $[-1, 1]$. That makes it a task that needs the nonlinearity, since $|f| = \mathrm{ReLU}(f) + \mathrm{ReLU}(-f)$. Once the features were sparse enough, it represented more features than it had neurons while still approximating their absolute values, which the authors read as "the ability of neural networks to perform computation even on data that is represented in superposition" (Elhage et al., 2022b).

44.4.2 One gate with interference #

Why computation can tolerate superposition is easiest to see in one gate. For binary features, the neuron $\mathrm{ReLU}(f_i + f_j - 1)$ outputs 1 when both $f_i$ and $f_j$ are 1 and 0 otherwise: it computes $f_i \wedge f_j$ exactly. A network in superposition cannot give each pair its own clean neuron. Each neuron also hears, faintly, many features it was not built for.

The capacity of one gate is inversely proportional to how often features are active: halving the density doubles the number of features that can share the neuron at the same error rate. This is the price-of-interference argument of Section 44.2 again, now for a computation instead of a read-out. The constructions below arrange many such gates so that the errors of each are averaged away.

44.4.3 Constructions #

Kaarel Hänni, Jake Mendel, Dmitry Vaintrob, and Lawrence Chan, in a 2024 preprint, gave "mathematical models of computation in superposition, where superposition is actively helpful for efficiently accomplishing the task" (Hänni et al., 2024). Their central task, the universal AND, takes $n_{\text{f}}$ sparse binary features and must make the AND of every one of the $n_{\text{f}}(n_{\text{f}}-1)/2$ pairs available (we write $n_{\text{f}}$ where the papers of this section write $m$, which this book keeps for the dimension of an activation). The naive network spends one neuron per pair. Their construction instead has each neuron check whether at least two inputs in a small random subset are active; the AND of a given pair is read off by averaging the neurons whose subsets contain both (Hänni et al., 2024). When the inputs arrive on their own coordinates, the width needed grows only polylogarithmically in $n_{\text{f}}$ (Hänni et al., 2024). The paper's abstract gives $\tilde{O}(n_{\text{f}}^{2/3})$ neurons "even when the input features are themselves in superposition", where $\tilde{O}$ hides logarithmic factors; the theorem in its body states a width of order $\sqrt{n_{\text{f}}}$ for that case (Hänni et al., 2024). Either count grows far more slowly than the number of pairs.

The authors extend the construction to sparse Boolean circuits of low depth, and add "error correction" layers that let a deep network emulate circuits wider than itself by roughly a power of 1.5, of any polynomial depth (Hänni et al., 2024). They also prove that, for inputs on their own coordinates, sufficiently wide randomly initialized one-layer networks already represent all the pairwise ANDs in the read-off sense, which suggests such computation could arise without being designed (Hänni et al., 2024). The construction began as a 2024 forum post by Vaintrob, Mendel, and Hänni (Vaintrob et al., 2024).

The authors state the limits: the features are binary and must be sparse, and "it even appears open to us whether a significant fraction of features are sparse in practice" (Hänni et al., 2024). One of their takeaways matters for every later section. Random networks already make many ANDs linearly decodable, so a concept that a probe or a dictionary can read off may not be used by the model at all, and finding a direction is not strong evidence of its use (Hänni et al., 2024).

44.4.4 Bounds #

Adler and Shavit, at MIT and Red Hat, proved the first lower bounds for computing in superposition, in a 2024 preprint (Adler and Shavit, 2024). Write $n_{\text{out}}$ for the number of features a layer computes. For a class of problems that includes permutations and pairwise ANDs, they show that computing $n_{\text{out}}$ features in superposition requires at least $\Omega(\sqrt{n_{\text{out}}\log n_{\text{out}}})$ neurons and $\Omega(n_{\text{out}}\log n_{\text{out}})$ parameters (Adler and Shavit, 2024). A layer can therefore compute at most of the order of the square of its neuron count, divided by that count's logarithm, of such features (Adler and Shavit, 2024). They also give a construction computing pairwise ANDs with $O(\sqrt{n_{\text{out}}}\log n_{\text{out}})$ neurons, close to the bound (Adler and Shavit, 2024). They write that follow-on work by Adler and colleagues found one of the construction's techniques, routing inputs through superposed "computational channels", emerging "naturally in toy networks trained via standard gradient descent" (Adler and Shavit, 2024). Representing features without computing on them can take only logarithmically many neurons, so the authors report an exponential gap between computing and merely representing (Adler and Shavit, 2024).

The two papers' neuron counts cannot be compared directly, because they count different things under different rules (Table 44.1). Hänni and colleagues compute one fixed function, all pairs at once, and count an output as computed when a linear read-out recovers it within an error $\varepsilon$. Adler and Shavit's lower bound is about a family of functions: the network's parameters must encode which $n_{\text{out}}$ pairs, in which order, out of all possible choices, so the bound counts the information needed to tell the functions apart. Their sharpest bounds also rely on ordered outputs, and they caution that the impossibility results "may overstate the constraints" for unordered tasks (Adler and Shavit, 2024).

44.4.5 Interference weights #

Superposition of features forces superposition of the weights between them. In an informal 2025 note at Anthropic, Olah, Nicholas Turner, and Tom Conerly argued that rewriting such weights as connections between features yields many interference weights: genuine connections that exist because of the packing and do not help the loss (Anthropic, 2025a). They demonstrated them in a toy model and wrote that whether they can be ignored "may be the pivotal question in whether global mechanistic analysis is possible" (Anthropic, 2025a). They also asked readers to take all claims "with significant caveats and low confidence throughout" (Anthropic, 2025a).

A 2026 report by Nicholas Turner, Jeffrey Wu, and Joshua Batson, of the same team, found interference weights in a trained one-layer transformer (Turner et al., 2026). Helpful and harmful connections were "scattered across the entire range" of weight magnitudes, so the largest weights are a poor guide to the circuits that matter; removing the least effective 70 percent of connections worsened the loss by only 0.01 nats (Turner et al., 2026). These are connections of the rewritten model, which has about 100 times as many as the transformer's 2.9 million parameters (Turner et al., 2026). Both are a developer's work on toy or research models.

44.4.6 What it means for the rest of the part #

A circuit drawn over features (Chapter 45) is a simplification of a computation in which every feature also leaks, weakly, into many others; the interference terms are what the drawing leaves out (inference). The constructions show that this leakage can be managed by averaging and error correction, and the bounds say that the parameter count, more than the neuron count, limits how many features a network can compute (Adler and Shavit, 2024). All of this is theory about Boolean features under strong sparsity, or measurement on toy and one-layer models. As of October 2026 we found no study that connects the constructions to the features found in production models.

44.5 Sparse dictionary learning #

If superposition is how networks store features, the features are hidden but not lost. Each activation vector is, on the hypothesis, a sparse sum of feature directions, so the directions might be recovered from activations alone by searching for a set of directions that explains every activation with few active terms. This section teaches the method built for that search, reports what it found, and sets out how its quality is judged.

44.5.1 The objective #

Sparse dictionary learning looks for an overcomplete dictionary, a set of directions with more members than the space has dimensions, such that each data vector is approximately a combination of only a few of them. In interpretability it is done with a sparse autoencoder, a second network trained on the first one's activations (Bricken et al., 2023; Cunningham et al., 2023). Given an activation vector $\vh \in \R^m$, the encoder computes $n_{\text{f}} > m$ nonnegative coefficients and the decoder rebuilds the vector from them:

Training minimizes, over many activation vectors,

where $\|\vf\|_1 = \sum_i |f_i|$ is the L1 norm and $\lambda$ sets the weight of sparsity against reconstruction. The columns of $\mW_{\text{dec}}$ are held at unit length (Bricken et al., 2023), or each latent's penalty is multiplied by its column's length (Templeton et al., 2024). Otherwise training could shrink every $f_i$ and lengthen its column to match, keeping $\hat{\vh}$ while making the penalty as small as it likes (Templeton et al., 2024). Each column of $\mW_{\text{dec}}$ is a candidate feature direction, and the entry $f_i$ says how strongly it is present. The literature calls the entries latents, a word that avoids presupposing that each one is a real feature of the model.

The dictionary is fitted to activations alone, so a latent's effect on the model's outputs is an independent check on it (Bricken et al., 2023). Its encoder is a single layer, like the layers that read the activations inside the model. Bricken and colleagues preferred it to classic dictionary-learning algorithms partly for that reason: iterative methods might be "too strong", recovering features "which the model itself cannot access" (Bricken et al., 2023), the worry of Section 44.2.

44.5.2 What the L1 penalty does #

The L1 penalty is what makes the codes sparse, and it has a known side effect. It charges every unit of every active coefficient, so the cheapest code reports each active feature as weaker than it is. Statisticians call this shrinkage.

The same trade appears in the network's own read-out of Section 44.2, where a bias that silences interference also reduces every active feature by the bias. In a dictionary, the shrinkage is an error of the instrument.

44.5.3 The first dictionaries #

Two groups reported the method on language models in 2023. Bricken and colleagues, at Anthropic, studied the developer's own one-layer transformer with a 512-neuron MLP layer, training dictionaries from 512 to 131,072 latents on activations from 8 billion data points and analyzing one run of 4,096 latents in detail (Bricken et al., 2023). A blinded annotator, one of the authors, scored random latents and random neurons with a rubric: the median neuron scored 0, meaning no hypothesis could be formed, and the median latent interval scored 12, a confident and specific hypothesis (Bricken et al., 2023). Some latents were invisible in the top activations of every neuron, and fixing a base64 latent at a high value made the model write base64 (Bricken et al., 2023). Replacing the MLP's activations with the 4,096-latent reconstruction recovered 79 percent of the loss reduction the MLP layer provides (Bricken et al., 2023).

Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey, from EleutherAI, MATS, the Bristol AI Safety Centre, and Apollo Research, reported the same method in a paper at ICLR (Cunningham et al., 2023). They worked on the residual streams of two open models, Pythia-70M and Pythia-410M (Cunningham et al., 2023). Bricken and colleagues describe the line of work that led to it as independent of their own (Bricken et al., 2023). By an automated measure, a language model's success at predicting a unit's activations from an explanation it wrote, the learned latents were more interpretable than the residual stream's coordinates, random directions, or principal or independent components, two standard statistical ways of finding directions in data (Cunningham et al., 2023). The advantage shrank with depth: it was "comparable to ICA in layer 4", ICA being independent component analysis, and showed "minimal improvement in the final layer" (Cunningham et al., 2023). Patching a few latents, replacing their values with those from a contrasting input (Section 43.4), also localized a known behavior, identifying the indirect object of a sentence, more tightly than patching principal components (Cunningham et al., 2023).

44.5.4 Scale #

In 2024 Adly Templeton and colleagues at Anthropic scaled the method to the developer's Claude 3 Sonnet, "the exact model in production as of the writing of this paper" (Templeton et al., 2024). They trained dictionaries of about 1, 4, and 34 million latents on the residual stream at the model's middle layer (Templeton et al., 2024). Fewer than 300 latents were active on a typical token, the reconstructions explained at least 65 percent of the variance, and about 65 percent of the largest dictionary's latents never activated on a sample of 10 million tokens at the end of training (Templeton et al., 2024). Many latents were multilingual and responded to the same concept in text and in images, although the dictionaries were trained only on text (Templeton et al., 2024).

Some latents were labeled with safety-relevant concepts: security vulnerabilities and backdoors in code, bias, "lying, deception, and power-seeking", sycophancy, and dangerous or criminal content (Templeton et al., 2024). The authors caution against reading much into the existence of such latents, since "there's a difference (for example) between knowing about lies, being capable of lying, and actually lying in the real world" (Templeton et al., 2024). They also report how incomplete the dictionaries were. The model could list all of London's boroughs, yet latents were found for only about 60 percent of them, and whether a concept had a latent tracked how often it appeared in the training data (Templeton et al., 2024). The authors judge it "quite likely that we're orders of magnitude short" of all the features (Templeton et al., 2024).

Leo Gao and colleagues at OpenAI trained a dictionary of 16 million latents on GPT-4's residual stream over 40 billion tokens, a developer's study of its own model, and found clean scaling laws in dictionary size and sparsity (Gao et al., 2024). They also report that "a large fraction of the random activations of features we find, especially in GPT-4, are not yet adequately monosemantic" (Gao et al., 2024). Open dictionaries then made the work repeatable outside the laboratories. Google DeepMind released Gemma Scope, dictionaries for every layer and sublayer of its own Gemma 2 2B and 9B models and some layers of the 27B model (Lieberum et al., 2024). A group at Fudan University released 256 dictionaries covering every layer and sublayer of Llama-3.1-8B, in a preprint (He et al., 2024c).

44.5.5 Replacing the L1 penalty #

Templeton and colleagues name shrinkage among their limitations, writing that it "significantly harms sparse autoencoder performance" (Templeton et al., 2024). Several replacements followed in 2024, each a preprint unless noted. Gated autoencoders, from Senthooran Rajamanoharan and colleagues, separate deciding which latents fire from estimating how strongly, and apply the penalty only to the first. On models of up to 7 billion parameters, in typical settings of their hyperparameters, they "solve shrinkage" and need about half as many active latents for comparable reconstruction (Rajamanoharan et al., 2024a). TopK autoencoders keep only a fixed number of the largest latents and zero the rest, an idea from the $k$-sparse autoencoders of Alireza Makhzani and Brendan Frey (Makhzani and Frey, 2013). Gao and colleagues adopted it because it "removes the need for the L1 penalty"; when they re-optimized the values of the active latents, the values rose for L1-trained autoencoders and not for TopK (Gao et al., 2024). BatchTopK fixes the count per batch instead of per input (Bussmann et al., 2024). JumpReLU autoencoders, from Rajamanoharan and colleagues, replace the ReLU with a learned threshold and train the number of active latents directly, which on Gemma 2 9B gave the best reconstruction at a given sparsity among the methods compared (Rajamanoharan et al., 2024b). TopK and JumpReLU are hard thresholds in the sense of the derivation above; Gemma Scope uses JumpReLU (Lieberum et al., 2024).

44.5.6 How quality is judged #

A dictionary has no test set in the usual sense, because nobody knows the true features; the developer's 2023 report says it had "yet to find anything satisfactory" as a metric (Bricken et al., 2023). Four measures are in common use, and each can mislead.

Reconstruction is the error of $\hat{\vh}$, often reported as the share of variance explained. Sparsity is the average number of active latents per input, called L0. Loss recovered replaces the model's activations with the reconstruction and asks how much of the model's performance survives, measured against the loss when the activations are set to zero. Gao and colleagues show why the last flatters: their 16-million-latent dictionary scores 98.2 percent of loss recovered by that measure, yet substituting it into GPT-4 gives a language-modeling loss "corresponding to 10% of the pretraining compute of GPT-4" (Gao et al., 2024). Zeroing the residual stream is so destructive that almost any reconstruction looks good against it (Gao et al., 2024).

Automated interpretability has a language model write an explanation of a latent from examples where it fires, then predict its activations on new text from the explanation, and scores the agreement (Bricken et al., 2023; Cunningham et al., 2023). Thomas Heap and colleagues at the University of Bristol tested whether such scores can tell a trained transformer from a randomly initialized one, in a 2025 preprint. Across Pythia models of many sizes and several ways of randomizing the weights, dictionaries trained on random transformers produced automated interpretability scores and reconstruction metrics "similar to those from trained models" in many settings (Heap et al., 2025). The aggregate scores missed a qualitative difference, the greater abstractness of trained models' latents in later layers, and the authors recommend random baselines as routine (Heap et al., 2025). Bricken and colleagues had run a random-weights control of their own, by shuffling each weight matrix, and found its latents uninterpretable beyond single tokens when inspected by hand (Bricken et al., 2023). The two results agree on the lesson that the automated averages are weaker evidence than the qualitative inspection behind them (inference).

A 2025 review of open problems by Lee Sharkey and colleagues adds that sparsity is a poor proxy for interpretability and that dictionaries decompose activations, not the mechanisms that act on them (Sharkey et al., 2025). The next two sections ask whether the dictionary found is the only one that fits, and whether it beats simpler tools at tasks that matter.

44.5.7 From neuroscience: sparse codes in cortex #

From neuroscience. Sparse coding, the neuroscientist's name for dictionary learning, is one of the method's roots, and it carries a test of the book's claim that superposition has no counterpart in the older fields. In 1996 Bruno Olshausen and David Field reported, in their title's words, the "emergence of simple-cell receptive field properties by learning a sparse code for natural images" (Olshausen and Field, 1996). Simple cells are neurons of the primary visual cortex, known as V1, and a neuron's receptive field is the pattern of light that drives it. As Ilias Rentzeperis and colleagues summarize the result, a network trained to reconstruct natural images under a constraint of sparse activity develops units with the properties of the receptive fields of V1's simple cells: "bandpass, oriented and spatially localized" (Rentzeperis et al., 2023). The model represents each image as a combination of a few atoms from an overcomplete dictionary, and in its reading of V1 the atoms correspond to neurons' receptive fields and the coefficients to their activity (Rentzeperis et al., 2023). The interpretability papers cite this lineage directly (Elhage et al., 2022b; Cunningham et al., 2023), and Olshausen himself coauthored the 2021 application of dictionary learning to transformers (Yun et al., 2021).

Compressed sensing reached neuroscience too: a 2012 review by Surya Ganguli and Haim Sompolinsky, read here from its abstract, treats it as a tool both for analyzing recordings and for explaining how brains process high-dimensional activity (Ganguli and Sompolinsky, 2012). In 2005 R. Quian Quiroga and colleagues reported, in the words of their abstract, neurons in the human medial temporal lobe "selectively activated by strikingly different pictures of given individuals, landmarks or objects and in some cases even by letter strings with their names" (Quiroga et al., 2005). They read these as "an invariant, sparse and explicit code" (Quiroga et al., 2005). Gurnee and colleagues cite such neurons as the biological analog of monosemantic units (Gurnee et al., 2023), and the multimodal latents found in Claude 3 Sonnet have the same shape (inference).

The shrinkage diagnosis also arrived in both fields. Rentzeperis and colleagues, in PLoS Computational Biology, used biological vision as a test bed for sparsity penalties and found the soft thresholding of the L1 penalty "highly suboptimal" compared with penalties closer to counting active units (Rentzeperis et al., 2023). In their simulations on natural images, holding sparsity fixed, soft thresholding needed about ten times as many units to reconstruct image patches as well as the best such penalty (Rentzeperis et al., 2023). Made as sparse as that penalty, its error was about three times worse (Rentzeperis et al., 2023). They conclude that V1 uses a regularization "closer to the $\ell_0$ pseudo-norm rather than to the $\ell_1$ norm" (Rentzeperis et al., 2023). Interpretability reached the same conclusion for its instrument and moved from L1 to TopK and JumpReLU.

For representation, then, superposition has a counterpart. Sparse and distributed codes are old ideas about brains, which Elhage and colleagues acknowledge (Elhage et al., 2022b), and Olah's distinction between composition and superposition is built on example codes taken from neuroscience (Anthropic, 2023d). The analogy stops in three places. First, the direction of use is reversed, as Elhage and colleagues point out: neuroscience "often thinks of biological neurons as sparse coding their inputs", while interpretability applies sparse coding "the opposite direction, to find features in superposition over neurons" (Elhage et al., 2022b). In V1 the sparse code is a hypothesis about what neurons do; in a network it is an instrument an analyst fits over them. Second, the V1 model was fitted to images and judged by its resemblance to recorded receptive fields, while a network's dictionary is fitted to the activations themselves and judged by its effect on outputs. Third, an interpretability researcher can record every unit, intervene on it exactly, and copy the network (Section 43.6), and so can train two dictionaries on the same complete activations and compare them, by their directions and by their effects under intervention. The comparison by directions is where non-uniqueness appears, in the next section; whether dictionaries converge under intervention had not been tested as of October 2026 (Section 44.8). A recording of a brain would allow the first kind of comparison over the units it captures; the second needs access no recording gives (inference).

The book's reading is that the negative-space entry falls for superposition as a strategy of representation, which neuroscience anticipated, and stands for the non-uniqueness of the features recovered, which was found with the complete records only a trained network allows (inference). The verdict belongs to Chapter 69.

44.6 Why a dictionary is not unique #

A sparse autoencoder returns a dictionary that reconstructs activations sparsely. Is that dictionary the model's own set of features, or one of many that fit about equally well? If it is the model's own, an inspector can enumerate the features of a model and check that a dangerous one is absent, the use Elhage and colleagues hoped superposition research would enable (Elhage et al., 2022b). If it is one of many, what the inspector sees depends on choices the inspector made. This section teaches the mechanisms that make dictionaries differ and reports the evidence on how much they do.

44.6.1 Splitting #

The first sign came with the first dictionaries. Bricken and colleagues found features in clusters that grew with the dictionary. Going from 512 latents to 4,096 and then 16,384, the number of latents specific to base64 text went "from 1 to 3 to many more", a phenomenon they named feature splitting (Bricken et al., 2023). The structure of the refinement "is more complex than a tree": latents at one size "may both split and merge" at the next (Bricken et al., 2023). The authors conjectured that an unlimited dictionary would return some idealized set of features, which smaller dictionaries cover less specifically, like lower resolutions of one picture (Bricken et al., 2023). The results below test that conjecture.

44.6.2 Absorption #

David Chanin and colleagues found a mechanism that bends the dictionary away from the features it is meant to find, in a paper at NeurIPS. When features form a hierarchy, as every token "short" also has the feature "starts with S", "seemingly monosemantic features fail to fire where they should, and instead get 'absorbed' into their children features"; they named this feature absorption and traced it to the sparsity objective (Chanin et al., 2024). In a toy setting with four orthogonal features, an autoencoder recovered each feature exactly when the features fired independently. When one feature fired only together with another, the parent's latent stopped firing on the child's inputs, and the child's latent absorbed the parent's direction (Chanin et al., 2024). The objective itself explains why.

Chanin and colleagues then found the effect in real dictionaries: the Gemma Scope dictionaries for Gemma 2 2B, and their own, including TopK ones, on Qwen2 0.5B and Llama 3.2 1B (Chanin et al., 2024). The task was spelling, which letter a token starts with, and no dictionary latent matched a linear probe at it (Chanin et al., 2024). In one Gemma Scope dictionary, a latent that classified "starts with S" with an F1 score of 0.81, a combined measure of precision and recall, failed to fire on the token "short". There a latent for the word "short" fired instead and carried part of the "starts with S" direction. Ablating it, setting it to zero, had a large effect on the model's spelling of the token, and the effect disappeared once the probe's direction was projected out of the latent (Chanin et al., 2024). Absorption rates rose with sparsity and dictionary width, and the authors conclude that varying size or sparsity "is insufficient to solve this issue" (Chanin et al., 2024). They describe their metric as likely "an under-estimate", and it cannot reach the later layers of the model, where spelling information has moved elsewhere (Chanin et al., 2024).

44.6.3 No canonical units #

Patrick Leask and colleagues, at Durham University, Apollo Research, and elsewhere, tested the conjecture of an idealized dictionary directly in a paper at ICLR (Leask et al., 2025). They call a set of units canonical if it is unique, complete, and atomic, and they test two of the three. Stitching inserts latents from a larger dictionary into a smaller one. Some inserted latents improved reconstruction, which shows the smaller dictionary was missing information, while others duplicated latents it already had (Leask et al., 2025). Meta-autoencoders, autoencoders trained on the decoder directions of another autoencoder, found that latents of a larger dictionary "often decompose into combinations of latents from a smaller SAE": a latent for "Einstein" decomposes into "scientist", "Germany", and "famous person" (Leask et al., 2025). The experiments used dictionaries from 768 to 98,304 latents on one layer of GPT-2 Small and two Gemma Scope dictionaries (Leask et al., 2025). The authors conclude that "there is no single SAE width at which it learns a unique and complete dictionary of atomic features", and they name the limits of their own evidence: two models, and no dictionaries as large as the largest in use (Leask et al., 2025).

Gonçalo Paulo and Nora Belrose, at EleutherAI, tested uniqueness by changing nothing but the random seed, in a 2025 preprint (Paulo and Belrose, 2025). They matched the latents of two dictionaries trained on the same model and data, in the same order, and called a latent shared when its best match had cosine similarity of at least 0.7 for both encoder and decoder directions (Paulo and Belrose, 2025). On one MLP layer of Pythia-160M, with 32,768 latents, 42 percent of latents were shared (Paulo and Belrose, 2025). With nine seeds, about 35 percent of a dictionary's latents still appeared in no other (Paulo and Belrose, 2025). In a dictionary of 131,000 latents on an MLP layer of Llama 3 8B, "only 30% of the features were shared across different seeds" (Paulo and Belrose, 2025). Dictionaries using the TopK activation were more seed-dependent than those trained with the L1 penalty, even at matched sparsity (Paulo and Belrose, 2025). Larger dictionaries overlapped less, and some latents found in only one dictionary were highly interpretable (Paulo and Belrose, 2025). The authors conclude that a dictionary is "a pragmatically useful decomposition of activation space, rather than an exhaustive and universal list of features 'truly used' by the model" (Paulo and Belrose, 2025).

Two further results show the instrument shaping what it finds. In a preprint by Aleksandar Makelov, Georg Lange, and Neel Nanda, autoencoders controlled GPT-2 Small less well than dictionaries built from a task's known variables (Makelov et al., 2024b). They also showed feature occlusion, in which a causally relevant concept is "robustly overshadowed" by slightly larger ones (Makelov et al., 2024b). Sai Sumedh Hindupur and colleagues, at Harvard, showed in a paper at NeurIPS that each kind of autoencoder "imposes structural assumptions about how concepts are encoded", so that "switching architectures can expose entirely new concepts or obscure existing ones" (Hindupur et al., 2025).

44.6.4 Not every feature is a direction #

The dictionary's basic assumption, that features are directions, also has counterexamples. Joshua Engels and colleagues, at MIT, used autoencoders to find multi-dimensional features in GPT-2 and Mistral 7B, including circles on which the days of the week and the months of the year are arranged in order (Engels et al., 2024). The models use these circles for arithmetic on days and months, and interventions on Mistral 7B and Llama 3 8B gave evidence that the circles are "the fundamental unit of computation in these tasks" (Engels et al., 2024). In a capacity model of dictionary scaling, Eric Michaud and colleagues found a regime in which such feature manifolds make dictionaries "learn far fewer features in data than there are latents", leaving open, in a preprint, whether real dictionaries are in it (Michaud et al., 2025). A 2025 report by Wes Gurnee and colleagues at Anthropic, on the developer's own Claude 3.5 Haiku, found that the model tracks the number of characters in a line on a curved manifold (Gurnee et al., 2025). The authors gave the same representation "dual interpretations": a family of discrete features, or one continuous manifold (Gurnee et al., 2025). Both descriptions fit, and the choice between them is the analyst's (inference).

44.6.5 Reading the evidence #

Two positions organize these results. On the first, a model has a definite set of features and dictionaries are noisy estimators of it, as in Bricken and colleagues' idealized dictionary. On the second, activations admit many sparse decompositions and none is the model's own, as Paulo and Belrose conclude. Lewis Smith and colleagues on Google DeepMind's interpretability team wrote in 2025 that current autoencoders are far from capturing a canonical set of concepts and that "it is unclear to us if such 'true concepts' even exist" (Smith et al., 2025b).

Splitting fits both positions. Absorption is a defect of the objective, so it fits the first if a better objective removes it, and the same team reported that a later variant, the Matryoshka autoencoder, "seems to have made substantial strides on feature absorption" (Smith et al., 2025b). Seed dependence and non-atomic latents fit the second more naturally. The first position has one reported case in its favor. In the developer's report on its own Claude 3.5 Haiku above, dictionaries of different sizes learned character-count features "with very similar receptive fields", a featurization the authors call "in some sense canonical", alongside some feature splitting (Gurnee et al., 2025). That comparison covers one task and dictionary sizes only, not seeds, architectures, or interventions. The first position also has a strong reply, which is why the question is open. The shared-latent counts compare directions by cosine similarity. Two dictionaries could divide the same computation differently, one with a merged latent and one with two split ones, and still agree about what the model does when the latents are intervened on. Interventions have been used to rank dictionaries by how cleanly ablating or editing their latents controls named concepts (Karvonen et al., 2024; Makelov et al., 2024b). A test of whether differing dictionaries converge under interchange interventions, which Section 43.5 teaches, has not been run as of October 2026, as far as we found.

44.6.6 What it means for verification #

A feature missing from a dictionary is not evidence that the model lacks it. Templeton and colleagues make the point for concepts without a dedicated latent, whose information may still be present in combinations of others (Templeton et al., 2024). Absorption makes it for latents that look like clean detectors yet fall silent on some of the inputs they should cover (Chanin et al., 2024). Two analysts with different dictionaries can also describe the same model differently and both be right about their dictionaries (inference). For the developer and the public, this matters most when a safety argument rests on an absence. A claim that a model has no feature for deception is a claim about one dictionary, of one size, seed, and architecture, unless shown otherwise (inference). The general question of whether a correct explanation of a network is unique, beyond dictionaries, is Section 47.2's.

44.7 What dictionaries are good for #

A method can be judged by whether its outputs look right or by whether they help with a task someone needed done. From 2025 several groups judged dictionaries the second way, on tasks that can be stated without mentioning interpretability and that have a simpler method to beat. This section reports those tests, the evidence on the other side, and what the two together decide.

44.7.1 Measurement tasks #

The first kind of task asks a tool to measure a concept someone has already named: is this prompt harmful, is this sentence grammatical, does this text mention a city. The baseline is a supervised linear probe, a classifier trained on labeled activations (Section 43.2), or a difference-in-means direction, the average activation on positive examples minus the average on negative ones.

Subhash Kantamneni, Joshua Engels, Senthooran Rajamanoharan, Max Tegmark, and Neel Nanda tested probes restricted to a small subset of dictionary latents against such baselines on 113 binary classification datasets, in a paper at ICML (Kantamneni et al., 2025). They probed Gemma-2-9B with the Gemma Scope dictionaries and repeated the core results on Llama-3.1-8B (Kantamneni et al., 2025). They chose four regimes that should favor a sparse, interpretable basis: scarce data, imbalanced classes, noisy labels, and a shifted distribution. Adding dictionary probes to the methods a practitioner could choose from gave no consistent improvement, and under distribution shift the baselines generalized better (Kantamneni et al., 2025). Dictionary probes "occasionally perform better than baselines on individual datasets", on 2.2 percent of them in the last-token setting (Kantamneni et al., 2025). The tasks where dictionaries first looked useful, such as finding spurious correlations and mislabeled examples, turned out to be achievable "with simple non-SAE baselines as well" (Kantamneni et al., 2025).

Zhengxuan Wu and colleagues at Stanford built a benchmark, AxBench, for detecting and steering concepts in Gemma-2-2B-it and Gemma-2-9B-it, the instruction-tuned versions, in a paper at ICML (Wu et al., 2025b). For detection, averaged over 500 concepts, a difference-in-means direction reached an area under the ROC curve (the probability that a random positive example scores above a random negative one) of 0.942. A dictionary latent with the concept's label reached 0.695, and the best of the dictionary's latents chosen with labeled data reached 0.917 (Wu et al., 2025b). Plain dictionary latents were "significantly outperformed by five supervised methods" (Wu et al., 2025b). The steering half of the benchmark, where prompting beat every method that edits activations, is reported in Section 46.2. The authors add a caution about their own test: the concept labels came from automated descriptions skewed toward token-level concepts, so the dictionaries' poor showing "is at least partially a reflection of the limitations of auto-interpretability" (Wu et al., 2025b).

Google DeepMind's interpretability team reported the third test in an informal progress update on the laboratory's own Gemma 2 9B, in its instruction-tuned assistant version (Smith et al., 2025b). The task was detecting harmful intent in user prompts, with jailbreaks, prompts written to get past a model's refusals, held out of training to test generalization. Linear probes on the model's activations were nearly perfect, with an area under the ROC curve of 1.0 on the training and validation sets and about 0.999 out of distribution (Smith et al., 2025b). Probes on single dictionary latents failed to fit the training set; probes on about 20 latents fitted it but generalized "distinctly worse"; and fine-tuning the dictionaries on chat data "helps, but only closes about half the gap to dense linear probes" (Smith et al., 2025b). Linear probes trained on the dictionary's reconstructions were also worse out of distribution than probes on the raw activations, "suggesting that SAEs are discarding information relevant to the target concept" (Smith et al., 2025b). The team's resulting change of priorities is recalled in Section 43.7.

The null results have a strong counter-reading. Each task names its concept in advance and supplies labels, which is exactly what supervised probes are built for. A dictionary trained on pretraining text may lack latents for a concept that appears mostly in chat, which the half-closed gap suggests (Smith et al., 2025b). The AxBench labels were automated and shallow, and for Gemma-2-2B-it the dictionaries were trained on the base model, since none were available for the instruction-tuned one (Wu et al., 2025b). The models were 2 to 9 billion parameters, far from the frontier. The authors frame their own results this way: Kantamneni and colleagues "cannot discount SAEs' utility on other tasks" (Kantamneni et al., 2025), and the Google DeepMind team writes that "no individual negative result is a strong update" (Smith et al., 2025b). What the results do establish is narrower and firm: on tasks where the concept is known and labeled, the dictionaries tested did not beat cheap supervised directions.

44.7.2 Discovery tasks #

The second kind of task asks a tool to find something nobody named in advance. Here the evidence is mostly from developers about their own models. Templeton and colleagues built probes from the same few examples they had used to find latents, and could neither interpret the probes' top activations nor, in most cases, steer the model with them, where the latents succeeded (Templeton et al., 2024). They limit the claim to "the 'few-shot' regime" and write that "it remains to be seen whether this is a compelling advantage in practice" (Templeton et al., 2024). They also note what an unsupervised method offers: a probe requires guessing the variable in advance, while dictionaries "allow us to uncover abstractions or associations formed by the model that we may not have predicted in advance" (Templeton et al., 2024).

Four applications reported in other chapters fit this kind. In the developer's blind auditing game, a researcher using dictionary features found the training documents behind a planted hidden objective fastest, though the two teams allowed dictionaries had begun as one, and a later semantic search of the training data surfaced the same documents (Marks et al., 2025) (Section 38.2). Dictionaries learned jointly over two models, crosscoders, have been used to find what fine-tuning changed, with measured artifacts that can report a shared concept as new (Lindsey et al., 2024; Minder et al., 2025b) (Section 38.3.4). A dictionary trained on GPT-4o's activations, a developer's study of its own model, found persona latents that control emergent misalignment (Wang et al., 2025e) (Section 32.3.1). And the developer of Claude Sonnet 4.5 used dictionary features, among other directions, to suppress the model's verbalized awareness of being tested (Anthropic, 2025aq) (Section 37.4.3).

Adam Karvonen and colleagues also turned one such use, removing spurious cues from a classifier by ablating latents judged irrelevant, into automated metrics that distinguish dictionary designs, in a preprint (Karvonen et al., 2024). And the Google DeepMind team, despite its negative probing result, found dictionaries "fairly helpful for debugging low quality datasets" (Smith et al., 2025b).

44.7.3 The bridge to circuits #

A dictionary describes one layer's activations. Tracing a computation needs a description of how one layer's features produce the next, and the variant built for that is the transcoder, which is trained to reconstruct a component's output from its input. Gonçalo Paulo, Stepan Shabalin, and Nora Belrose found transcoder features "significantly more interpretable" than autoencoder features trained on the same model and data, in a preprint (Paulo et al., 2025). Transcoders are the unit of the replacement models and attribution graphs of Section 45.4, so the non-uniqueness of this chapter travels with them (inference).

44.7.4 What the evidence decides #

The evidence so far favors dictionaries as tools for finding what to look for, and supervised directions as tools for measuring something already named (Chapter 46) (inference). The positive evidence for discovery comes mostly from developers studying their own models, in single test models and case studies; the negative evidence for measurement comes mostly from groups testing open models against baselines. Whether the division holds on frontier models with objectives that no training document states is open (inference).

For the book's thesis, this chapter tests one of the named novelties: a principal who can read the agent's internals. The reading is real, in that dictionary features change the model's behavior when they are set (Section 44.5.3). It also returns a description that is partly chosen by the reader, through an instrument whose miss rates nobody has measured on a model whose features are known (inference). The nearest older structure is the inspection of an agent through an instrument of unvalidated accuracy, the problem forensic science faced with the polygraph, which Section 46.4 takes up. Most claims here concern the second relationship, an overseer checking a deployed system. The provenance pattern concerns the third: the strongest positive results are developers' reports on proprietary models, and the tests that outsiders can repeat are the ones on open models (inference).

44.8 Settled, contested, missing #

44.9 Exercises #

Further reading #

• Elhage et al. (2022b): the toy model, its phase changes, and its geometry.
• Bricken et al. (2023): dictionary learning on a small transformer, with its measures and their limits.
• Templeton et al. (2024): dictionaries on a production model, by its developer.
• Hänni et al. (2024): computation, not only storage, in superposition.
• Chanin et al. (2024): how the sparsity objective bends the features it finds.
• Kantamneni et al. (2025): dictionaries tested against strong baselines.