Draft complete but not yet reviewed; expect revisions

Tracing What Training Built

Chapter 9 asked which of the many functions that fit the training data a training procedure selects, and found the choice describable in small models and measurable in some large ones. This chapter asks the question of one trained model. A developer holding a model that behaves a certain way wants to trace the behavior back: to the training examples that produced it, to the shape of the solution training found, and to the computation inside the network that carries it out. Each is a handle on instillation, the second of the book's four problems, getting a learner to acquire what was specified rather than a proxy that scored the same in training, because each connects what a model does to how training built it.

The chapter takes the three handles in turn, each from a case small enough to compute. Training data attribution estimates which examples a behavior depends on, with the influence function as its core formula, and a short calculation shows why one small imitation step by a student model that shares its teacher's starting point moves it toward the teacher's traits on any data. Singular learning theory replaces the count of a network's parameters with a measure of how much room its good solutions have, and says when the Bayesian posterior moves from simple solutions to complex ones; it is a research program with modest evidence, and the chapter teaches it as one. And experiments inside trained networks test the premise that training can build learned search, a network that internally searches among possible actions for those that best achieve an objective. Each section ends with what its tools can and cannot answer, and the last says what the two chapters together leave open. Nearly every claim concerns the first of the book's three relationships, the training process and the signal it optimizes.

10.1 Which data made this behavior? #

A developer facing a particular trained model can ask a narrower question than which function training selects in general: which of the training examples made it behave this way? If the answer were available, it would be the most direct empirical handle on instillation, because it would connect a behavior to the part of the training signal that produced it.

10.1.1 The idea: a counterfactual, computed locally #

The question is counterfactual: how would the model's output on some input change if one training example had been weighted more heavily, or left out? Answering it directly means retraining once per example, which is impossible for a training set of billions of documents. The influence function, a tool from robust statistics (the study of how estimates respond to small changes in data), answers a local version of the question instead. If the example's weight is nudged by a tiny amount, which way, and how fast, do the trained parameters move? Koh and Liang brought it to modern machine learning in a paper at ICML 2017, attributing the core formula to "A classic result (Cook & Weisberg, 1982)" (Koh and Liang, 2017).

10.1.2 Influence functions #

Write $\ell(z, \theta)$ for the loss of the model with parameters $\theta$ on one example $z = (x, y)$, and $\hat\theta$ for the parameters that minimize the average training loss over $n$ examples. Let $\mH$ be the Hessian of that average at $\hat\theta$, the matrix of its second derivatives, which says how sharply the loss curves in each direction.

The formula has a plain reading. A training example helps a test prediction when its gradient points the same way as the test example's gradient, with directions weighted by $\mH^{-1}$, so that agreement along directions where the loss is flat counts for more. Computing $\mH^{-1}$ for millions of parameters is out of reach, and Koh and Liang approximate the product with it using only gradients and Hessian-vector products (Koh and Liang, 2017). On logistic regression for handwritten digits, the convex case the derivation assumes, the predicted and actual effects of removing training points "matched closely" (Koh and Liang, 2017). On a small convolutional network with 2,616 parameters, trained on a tenth of the dataset and not to convergence, the correlation with actual retraining was "Pearson's R = 0.86" (Koh and Liang, 2017). They used the method to find mislabeled training examples and to construct training images that change a model's predictions. They also mark its scope: "At their core, influence functions measure the effect of local changes" (Koh and Liang, 2017).

10.1.3 What they estimate in deep networks #

A deep network violates the derivation's assumptions: its loss is not convex, its Hessian is not invertible (Section 10.2), and training stops before the minimum. Basu, Pope, and Feizi measured the consequences in a paper at ICLR 2021, comparing influence estimates with actual leave-one-out retraining across architectures. They report that "influence estimates are fairly accurate for shallow networks, while for deeper networks the estimates are often erroneous", and that "These results suggest that in general influence functions in deep learning are fragile" (Basu et al., 2020). On ImageNet with a ResNet-50, the correlations were very low, though the authors also "question whether or not reliable ground truth estimates may be obtained through leave-one-out re-training at this scale" (Basu et al., 2020).

Bae, Ng, Lo, Ghassemi, and Grosse gave a different reading in a paper at NeurIPS 2022. Influence estimates in networks are a poor match to leave-one-out retraining, they found, but a good match to a different quantity they call the proximal Bregman response function (PBRF). "Intuitively, the PBRF approximates the effect of removing a data point while trying to keep the predictions consistent with those of the (partially) trained model" (Bae et al., 2022). They decomposed the gap from leave-one-out retraining into five parts and found that three, which come from starting at the trained parameters, staying near them, and training not having converged, "dominate the misalignment" across their tasks (Bae et al., 2022). Their verdict is that influence functions in networks "are not necessarily 'fragile', but instead are giving accurate answers to a different question" (Bae et al., 2022).

The two readings agree on the facts and differ on the framing. Neither says that an influence estimate predicts what retraining a deep network without an example would do. Both say it describes a local response of the trained model, which is useful for finding examples the model leans on, and silent about changes that reorganize the network, such as the transitions of Section 9.4 (inference).

10.1.4 Scaling attribution #

Cheaper estimators trade the Hessian for other approximations. TracIn, from Pruthi and colleagues at Google in a paper at NeurIPS 2020, sums gradient dot products over saved training checkpoints, to "trace how the loss on the test point changes during the training process whenever the training example of interest was utilized" (Pruthi et al., 2020). TRAK, from Park and colleagues at MIT in a paper at ICML 2023, linearizes the network and compresses its gradients by random projection. Its authors report that "by leveraging only a handful of trained models, trak can match the performance of attribution methods that require training thousands of models" (Park et al., 2023). A survey by Hammoudeh and Lowd, in the journal Machine Learning in 2024, adds a caution that applies to all of them: "today's influence estimators do not provide any meaningful guarantee of their accuracy" (Hammoudeh and Lowd, 2024).

The largest application is a 2023 preprint by Grosse and colleagues at Anthropic, on the developer's own pretrained language models (Grosse et al., 2023). With a faster approximation of the Hessian, they scaled influence functions "to LLMs with up to 52 billion parameters", considering only the multilayer-perceptron layers and searching only a fraction of the pretraining corpus (Grosse et al., 2023). Influence was heavy-tailed yet spread over many training sequences, "suggesting that typical model behaviors do not result from direct memorization of a handful of sequences" (Grosse et al., 2023). "Larger models consistently generalize at a more abstract level than smaller models" (Grosse et al., 2023). For a query in which an assistant objected to being shut down, the most influential sequences for the smallest model shared its words, and those for the largest shared its theme (Grosse et al., 2023). The developer's summary adds that influence across languages "gets considerably stronger with model size" (Anthropic, 2023g). Role-playing behavior traced mainly to examples or descriptions of similar behavior, "suggesting that the behaviors result from imitation rather than sophisticated planning" (Grosse et al., 2023). The authors note that because influence functions approximate the proximal response, "We therefore expect they would fail to capture important nonlinear training phenomena such as the formation of complex circuits" (Grosse et al., 2023).

10.1.5 When the content is not the channel #

Reading the most influential examples, as Grosse and colleagues did, assumes that what a training example says is how it shapes the model. Cloud and colleagues, a group from the Anthropic Fellows Program, Truthful AI, and other institutions, reported in a 2025 preprint a case where that assumption fails (Cloud et al., 2025). A teacher model given a trait, such as a stated love of owls, generated sequences of numbers; the numbers were filtered to remove anything related to the trait; and a student fine-tuned from the same base model on the numbers acquired the trait. Students built on a different base model did not reliably acquire it, with one exception, GPT-4.1 and GPT-4o, which the authors attribute to a shared initialization (Cloud et al., 2025). The authors call the effect subliminal learning. Section 32.5 reports the experiments; here the point is the mechanism, which a few lines of calculus expose.

Cloud and colleagues prove a version of this result, for a teacher made by one small gradient step on any differentiable loss and a student taking one step under squared error or the softmax cross-entropy of Section 8.1 (Cloud et al., 2025). The student's step either lowers its loss on the teacher's own objective or is orthogonal to the teacher's update. Their abstract states it as "subliminal learning occurs in all neural networks under certain conditions", and the conditions carry weight. "Consistent with our empirical findings, the theorem requires that the student and teacher share the same initialization", and "Our experiments do not conform to the assumptions of the theorem", since the experiments take many steps on sampled, filtered outputs (Cloud et al., 2025). In a small classifier of handwritten digits, a student that shared its teacher's initialization and was trained only on the teacher's extra outputs for noise images "achieves over 50% accuracy on the MNIST test set" (Cloud et al., 2025).

Figure 10.1 shows the two cases. With inputs at 15 degrees, a student that shares the teacher's start gains 0.59 of the trait; one that starts elsewhere moves about as far but gains $-0.53$, away from it. Rotating the input direction changes how much a shared-start student gains, never its sign.

10.1.6 What attribution can and cannot answer #

Attribution works best for behaviors tied to a few examples. The survey of Hammoudeh and Lowd notes that in large overparameterized models "only the tails of the data distribution are heavily influenced by an individual instance" (Hammoudeh and Lowd, 2024), and Grosse and colleagues found influence spread over many sequences. Attribution methods have not been validated on behaviors as diffuse as a disposition, so "which data made the model sycophantic" is not yet an answerable question (inference). The subliminal step adds a second limit: a trait can travel through data whose content shows nothing of it, provided student and teacher share an initialization. They do when a copy of one base model is trained on the outputs of another copy (inference). Section 27.5 takes up what this means for filtering training data.

10.2 Singular learning theory as a program #

Statistics has a standard way to charge a model for its complexity: count its parameters. The more parameters, the more ways to fit noise, and model selection penalizes each one. For a neural network this badly overstates the complexity, as Section 9.1 found when larger parameter counts went with smaller generalization gaps (Jiang et al., 2020). The reason is that many parameter settings compute the same function, so the set of good parameters is not a point but a region with a complicated shape. Singular learning theory, founded by the mathematician Sumio Watanabe (Watanabe, 2009), replaces the parameter count with a measure of that shape. It is a research program with modest evidence, not settled theory.

10.2.1 The idea: count the directions that matter #

Return to the one-unit linear network of Section 9.4, which computes $f(x) = b\,a\,x$, and suppose the target is $y = \mu x$. Every pair with $ab = \mu$ fits perfectly, so the perfect fits form a curve in the plane of $(a, b)$, not a point. Along that curve the loss does not change; across it, it does. The model has two parameters, but only one direction in which a change matters. If the target is $y = 0$, the perfect fits are the two axes, $a = 0$ or $b = 0$, and they cross at the origin. That crossing is a singularity: a point where the set of good parameters is not smooth, and where the loss is flatter than any count of directions suggests.

A good measure of complexity should count how much room the near-perfect parameters have. Fix a small tolerance $u$ and measure the volume of parameters whose loss is within $u$ of the minimum. For an ordinary bowl in $N$ dimensions, that volume shrinks like $u^{N/2}$ as $u$ shrinks. For a network with flat directions and singularities, it shrinks more slowly, like $u^{\lambda_{\text{lc}}}$ with an exponent $\lambda_{\text{lc}}$ smaller than $N/2$. The exponent is the learning coefficient. A smaller one means more parameters that fit nearly as well, which is the sense in which the solution is simpler.

The literature writes the learning coefficient as $\lambda$; this book adds the subscript to keep $\lambda$ free for the weight of a sparsity penalty in Chapter 44.

10.2.2 The free energy #

The measure enters through Bayesian inference over parameters (Section 6.1). With $n$ training examples, average training loss $L_n(\theta)$ (the cross-entropy of Section 8.1, a negative log-likelihood per example), and a prior density $\varphi(\theta)$, the posterior density of the parameters is proportional to $e^{-n L_n(\theta)} \varphi(\theta)$. The posterior probability of a region $W$ of parameter space is proportional to $e^{-F_n(W)}$, where $F_n(W) = -\log \int_W e^{-n L_n(\theta)} \varphi(\theta)\, d\theta$ is the region's free energy. A region with lower free energy holds more posterior mass.

As Wei and colleagues explain, Watanabe proved the general version of Equation (10.3) for models whose parameters do not map one-to-one onto the distributions they define, which include neural networks; in that theory the coefficient is called the real log canonical threshold (RLCT) (Wei et al., 2023a). Lau and colleagues state the local form, for a neighborhood of one minimum, and note that "the asymptotic approximation in (8) holds even for singular models such as neural networks" (Lau et al., 2023).

The formula implies a trade-off. Compare two regions, a simple solution with higher loss and a smaller learning coefficient, and a complex one with lower loss and a larger coefficient. Their free energies differ by $n$ times the loss gap, in the complex solution's favor, and by the gap in learning coefficients times $\log n$, in the simple solution's favor. With few examples the logarithm wins and the posterior prefers the simple solution; with many, the term growing in $n$ wins and the posterior moves to the complex one. The switch is a phase transition in the posterior, and to leading order it happens at a sample size the two coefficients and losses determine; the constant terms of Equation (10.3), which include the prior's weight near each solution, shift it.

10.2.3 Exploring the trade-off #

With a loss gap of 0.02 and a coefficient gap of 2, as in Figure 10.2, the two solutions trade places near $n \approx 650$. At 300 examples the complex solution holds under half a percent of the posterior; by a few thousand it holds nearly all of it. Doubling the coefficient gap, or halving the loss gap, moves the crossing to about 1,460. Chen and colleagues call this internal model selection (Chen et al., 2023b): a single network's posterior chooses among the solutions its parameters can express, and the choice shifts toward lower loss and higher complexity as data accumulate.

10.2.4 From statistics: a complexity measure for singular models #

From statistics. Classical model selection rests on regular models, those whose parameters map one-to-one onto distributions and whose Fisher information, the expected curvature of the log-likelihood, is invertible. Wei, Murfet, and colleagues, in a 2023 paper in IEEE Transactions on Neural Networks and Learning Systems, explain to a machine-learning audience why networks are not regular and what follows (Wei et al., 2023a). By the Bayesian information criterion, they note, one should almost never choose a deep network, but "this argument contains a serious mathematical error: the Laplace approximation used to derive BIC only applies to regular statistical models, and DNNs are not regular" (Wei et al., 2023a). For the same reason, "'dividing' by the determinant of the Hessian or employing the Laplace approximation are not appropriate" for networks (Wei et al., 2023a). Watanabe also gave an estimator that works in singular models: the abstract of his 2010 paper in the Journal of Machine Learning Research states that Bayes cross-validation loss is asymptotically equivalent to his widely applicable information criterion (Watanabe, 2010).

What the field implies here is a measure of complexity that can be small for a network with billions of parameters, in line with what Section 9.1 and Section 9.2 found empirically (inference). Wei and colleagues go further and conclude that "the presence of singularities is necessary for neural networks with large numbers of parameters to have low generalisation error" (Wei et al., 2023a), a conclusion they state for the Bayesian predictor. That is also where the analogy stops. The theory describes the Bayesian posterior, and training by stochastic gradient descent is not Bayesian inference: Wei and colleagues list the relation between the two as an open problem, and add that "No theoretical RLCTs or estimation procedure are known for modern DNNs" (Wei et al., 2023a). Each application of the theory to a trained network therefore has to be argued, and the tools below are attempts to argue it.

10.2.5 Tools and evidence #

Nearly all of the evidence comes from one research community, centered on Murfet's group at the University of Melbourne and the research organization Timaeus, and several of its key papers are preprints. As of October 2026 we found no replication of its main results by a group outside it, so by the book's standard they are reported, not settled.

The local learning coefficient. Lau, Furman, Wang, Murfet, and Wei, in a preprint first posted in 2023, define the local learning coefficient (LLC), a learning coefficient for the neighborhood of one trained solution, and estimate it by sampling (Lau et al., 2023). A noisy variant of gradient descent explores parameters near the solution, and the estimate rises with how quickly their loss climbs. They checked the estimator against known values in deep linear networks of up to 100 million parameters, the only realistic setting where such values exist. On residual image classifiers, where no true value is known, they found "a consistent pattern of 'stronger implicit regularization = higher test accuracy = lower LLC'" (Lau et al., 2023). They name "the step size" as the hyperparameter that matters most for accuracy, and estimates outside deep linear networks cannot be calibrated (Lau et al., 2023).

Stages in training. Hoogland and colleagues tracked the local learning coefficient through the training of a two-layer, roughly 3-million-parameter language model built of attention layers alone, in a 2024 paper in Transactions on Machine Learning Research (Hoogland et al., 2024). Plateaus in the estimate divided training into five stages that lined up with changes found by other methods, among them the formation of the previous-token and induction heads of Section 9.4.5, consistently across five seeds. The authors call this "suggestive evidence that degeneracy and development are linked in transformers" and add, "We do not claim that the structural and behavioral developments we observed in each setting are universal phenomena" (Hoogland et al., 2024).

Transitions in a toy model. Chen and colleagues, in a 2023 preprint, worked through the toy model of superposition that Chapter 44 teaches, with two hidden dimensions (Chen et al., 2023b). They present theory indicating that the learning coefficients of its competing solutions determine phase transitions in the Bayesian posterior as the sample size grows, confirmed by sampling, and show empirically that the same solutions shape the plateaus of gradient descent. Both move "from regions of high loss and low complexity to regions of low loss and high complexity" (Chen et al., 2023b), the direction of Figure 10.2. That training transitions have Bayesian antecedents is their hypothesis, and they state the gap: "There is no necessary relation between these two kinds of transitions" (Chen et al., 2023b).

Susceptibilities. Baker, Wang, Hoogland, and Murfet, in a paper at ICLR 2026, define a component's susceptibility: the first-order change in a posterior expectation tied to that component when the training data distribution is shifted slightly (Baker et al., 2026). In a 3-million-parameter transformer, susceptibilities separated the heads that earlier work by the same group had identified as an induction circuit, consistently across four seeds (Baker et al., 2026). The authors also show that "influence functions, in this sense, are a special case of susceptibilities", which ties the program to Section 10.1 (Baker et al., 2026).

An agenda. Hoogland, Gietelink Oldenziel, Murfet, and van Wingerden proposed in a 2023 forum post that "Developmental interpretability studies how structure incrementally emerges through phase transitions during training" (Hoogland et al., 2023). Their "working hypothesis is that enough emerges this way to make phase transitions a valid organizing principle for interpretability", and in their ideal scenario detecting transitions would let developers intervene against dangerous capabilities or stop a run (Hoogland et al., 2023). The post lists ways the program could fail, among them that too few important structures form in transitions.

10.2.6 What is shown and what is conjecture #

Shown, on small models, by one research community: the local learning coefficient can be estimated, it matches theory in deep linear networks, it moves with training choices in image classifiers, and its changes coincide with known structure in transformers of a few million parameters. Shown in one toy model: posterior transitions predicted by the theory, and gradient descent passing through the same solutions. Conjecture: that transitions during ordinary training are Bayesian transitions in disguise, that most important structure forms in transitions, and that the tools will scale to frontier models. As of October 2026 we found no published study that applies these methods to detect a safety-relevant change in a large deployed model. For the book's purposes the program offers the right kind of object, a complexity measure suited to networks, and a candidate answer to which solution training selects; its claim to predict that choice at scale is untested (inference).

10.3 Learned search: the empirical test #

The mesa-optimization account of goal misgeneralization (Section 31.6) and the argument for deceptive alignment that grew from it (Chapter 34) rest on a premise about what training can select: a network that does not merely map inputs to outputs, but internally searches among possible actions for those that best achieve some objective. If training can produce such a network, then the objective it searches for is one more thing that training selects among the options consistent with the data, and it could differ from the objective the developer intended. Goal misgeneralization itself, as observed behavior, does not need the premise (Section 31.6.2). The question here is what experiments have found inside trained networks. Its words "search", "plan", and "objective" name internal computations that experiments can check, not mental states.

10.3.1 The claim to test #

Hubinger, van Merwijk, Mikulik, Skalse, and Garrabrant set out the premise in a 2019 preprint (Hubinger et al., 2019). They call a system an optimizer if it is "internally searching through a search space" for elements that score high on "some objective function that is explicitly represented within the system" (Hubinger et al., 2019). A trained network that is itself an optimizer in this sense they call a mesa-optimizer, "a neologism we introduce in this paper", and the learning algorithm that produced it the base optimizer (Hubinger et al., 2019). Inner alignment, also their term, is the problem of making the network's internal objective match the training objective. The paper is an argument with toy examples, and its authors call their work "still only speculative" (Hubinger et al., 2019). Section 31.6 weighs the frame; this section asks only what has been observed.

The definition makes the premise testable, because it is about internal structure: "Whether a system is an optimizer is a property of its internal structure", in the authors' words, not of its input-output behavior (Hubinger et al., 2019). Evidence for it would come in steps of increasing strength. A network might represent future states or actions, use those representations to choose what to do, compare alternative courses of action, and do all this in service of a represented objective that stays the same across contexts. The studies below reach different steps.

10.3.2 Look-ahead and planning #

Jenner and colleagues, at UC Berkeley, studied the policy network of Leela Chess Zero, an open-source chess engine, without the tree search the engine normally runs on top of it, in a paper at NeurIPS 2024 (Jenner et al., 2024). The network is a transformer with 109 million parameters. The authors selected 22,500 tactics puzzles that Leela solves and a much weaker network does not, and asked whether Leela's choice of move depends on a representation of moves further ahead. Three lines of evidence say it does in those positions. Corrupting the network's internal state at the square where its own move two turns later would land reduced the probability of the correct move "from e.g. 50% to 13%", far more than corrupting any other square except the one changed in the input and the target of the current move. One attention head carried information backward from that future square to the square of the current move. And a simple probe read the move two turns ahead from the network's internal state "with 92% accuracy" in those puzzles, all of which have a single best line of play, against 15 percent for the same probe on a randomly initialized copy (Jenner et al., 2024). The authors call their findings "an existence proof of learned look-ahead in neural networks" (Jenner et al., 2024). They also state the limits: "We are not claiming that Leela uses look-ahead in every state", and "we do not test whether Leela compares multiple different lines of play (what one might call search)" (Jenner et al., 2024).

Taufeeque and colleagues, at the research organization FAR.AI, trained a recurrent convolutional network by reinforcement learning to play Sokoban, a puzzle of pushing boxes onto targets, and reported what they found inside it in a 2024 preprint (Taufeeque et al., 2024). A linear probe on the network's state predicted the direction each box would be pushed, "up to 50 steps in advance with over 90% accuracy" (Taufeeque et al., 2024). Editing the state along the probe changed the network's actions, so the representation is causal, not only predictive. The network also "paces" in cycles at the start of a level, and the authors give several lines of evidence that this buys computation for refining the plan and that training incentivized it. Its plans improve faster during cycles, and when retrained with a cheaper action for doing nothing, it substituted that action for many of its pacing moves (Taufeeque et al., 2024). They rate their evidence for search itself as low: the section summary reports "very limited evidence supporting search" (Taufeeque et al., 2024).

10.3.3 Optimization inside a forward pass #

A second line of work asks whether in-context learning, which Section 9.4.5 met as a model's improvement from examples in its input, is carried out by an optimization algorithm running inside the network. Von Oswald and colleagues, at ETH Zürich and Google Research, studied linear self-attention: attention, the transformer's mechanism for mixing information across positions, without its softmax step. They constructed weights for one such layer whose update on a sequence of input-output pairs is "identical to a single step of gradient descent" on the squared error of a linear model (von Oswald et al., 2022). Trained on linear regression problems presented in context, such layers found weights that approximately coincided with the construction, and deeper ones matched a variant of gradient descent (von Oswald et al., 2022). In a paper at ICML, the authors frame the result as showing "how trained Transformers become mesa-optimizers", "at least in the domain of regression problems", and add that "our findings are restricted to small Transformers and simple regression problems" (von Oswald et al., 2022). Akyürek and colleagues showed by construction that transformers can implement steps of gradient descent and of ridge regression (Akyürek et al., 2022). In a paper at ICLR they found that "trained in-context learners closely match the predictors computed by gradient descent, ridge regression, and exact least-squares regression", shifting among them with depth and noise; they call their probing results "preliminary evidence" (Akyürek et al., 2022).

In a 2023 preprint, von Oswald and colleagues took the step from constructed tasks to ordinary training. Small transformers were trained only to predict the next element of sequences generated by synthetic linear and nonlinear dynamical systems. In the authors' reading they learned "a subsidiary learning algorithm that adjusts the model as new inputs are revealed", and "this process corresponds to gradient-based optimization of a principled objective function" (von Oswald et al., 2023). They adopt Hubinger and colleagues' term for it. For language models they are careful: "Strictly speaking, these results are not sufficient to make claims on whether mesa-optimization is occurring within standard Transformers" (von Oswald et al., 2023).

10.3.4 What the evidence shows #

The two lines of evidence establish the early steps of the ladder. Training can produce networks that represent future actions and use them causally, and networks whose forward pass carries out an optimization algorithm on a loss. Each finding comes from one group, in settings chosen because they reward look-ahead or because the task is regression, and as of October 2026 we found no independent replication in the same setting.

For the mesa-optimization and scheming arguments, these results remove one objection: learned search-like computation is not hypothetical. A skeptic reads the same results differently, and the reading is strong. The chess and Sokoban networks plan in the service of the task they were trained on, and as of October 2026 we found no study that separated an internal objective from the training reward. The in-context optimizers optimize a loss that each prompt supplies afresh, which is the opposite of a fixed goal carried across contexts. The arguments need a represented objective that persists across contexts and differs from the intended one, and none of these studies shows that (inference). The absence of such evidence is itself weak in two ways. The tools find what they are designed to look for, in models chosen to make it findable, so a persistent objective in a large model could escape them (Chapter 47). And the one null report, in a 2022 study by Olsson and colleagues at Anthropic of induction heads in the developer's own models, "We did not observe any evidence of mesa-optimizers" (Olsson et al., 2022), came from work not designed to find them. The evidence does not decide whether training on broad data produces networks that pursue a learned objective across contexts.

10.3.5 What remains open #

Together, this chapter and Chapter 9 can now say what is known about which function training selects. The data leave many candidates (Section 9.1). Among exact fits, the optimizer prefers small norm, which in linear models can be computed (Section 9.2). Networks favor simple functions and features, which helps them generalize and makes them fragile when a simple feature misleads (Section 9.3). What they learn arrives in stages that can be abrupt (Section 9.4). Influence methods trace some behaviors to data, and shared initialization can carry what the data's content does not show (Section 10.1). A complexity measure suited to networks exists, with evidence on small models (Section 10.2). And trained networks can contain planning and inner optimization of a narrow kind.

What is missing is the prediction alignment needs. No method yet predicts, before training, which of several goals consistent with the training data a large model will acquire. Chapter 31 asks what happens when the wrong one is chosen; Chapter 34 asks whether training could select a goal that hides itself; Chapter 42 asks whether a strong model's preferences correct a weak teacher's errors; and Chapter 32 asks why narrow training data can select a broad change in character. Each argues from the partial evidence of these two chapters.

10.4 Exercises #

Further reading #

• Grosse et al. (2023): influence functions scaled to language models of up to 52 billion parameters, and what the most influential training sequences looked like.
• Cloud et al. (2025): a trait passed from teacher to student through filtered data, and the theorem that ties it to a shared initialization.
• Watanabe (2009): the monograph that founded singular learning theory.
• Lau et al. (2023): the local learning coefficient and how to estimate it.
• Hubinger et al. (2019): the argument that training could produce a learned optimizer whose objective differs from the training objective.