Draft complete but not yet reviewed; expect revisions

What Training Selects

The previous chapter ended with a puzzle. Standard networks can fit labels assigned at random, so their capacity cannot be what makes them generalize on real labels (Section 8.4.5). The puzzle matters for alignment because of what it implies. If a network can represent many functions that fit the training data equally well, then the data and the loss do not decide what the network does on new inputs. The training procedure decides, by favoring some fitting functions over others. That preference is called inductive bias. Instillation, the second of the book's four problems, is getting a learner to acquire what was specified rather than a proxy that scored the same in training. It is a question about inductive bias: of all the behaviors consistent with the training signal, which one did training produce?

This chapter teaches what is known about that choice, starting each time from a case small enough to compute exactly, and it marks how much is not known. It takes four ideas in turn: the data and a held-out check leave many fits standing; past a threshold of size, the optimizer chooses among exact fits; networks favor simple functions; and what they learn can arrive in sudden stages. The choice can be described in small models, measured in some large ones, and predicted in none of the cases alignment cares about most. No theory yet says which goals or values, among those consistent with alignment training data, a large model acquires. Chapter 10 turns from the general question to one trained model, and asks which data made a behavior, what measure of complexity suits a network, and whether training builds learned search. The later chapters that need a prediction (Chapter 31; Chapter 34; Chapter 42; Chapter 32) argue from the partial evidence the two chapters collect. Nearly every claim in the chapter concerns the first of the book's three relationships, the training process and the signal it optimizes.

9.1 Many solutions fit #

A developer trains a model, checks it on held-out data drawn the same way as the training data, and finds it accurate. The previous chapter showed one way that check can mislead: a shortcut that agrees with the label in training earns its accuracy there and fails where the agreement breaks (Section 8.4.2). This section shows that the problem is more general than shortcuts. Even when nothing in the data is misleading, the data and the held-out check together do not determine what the model does on inputs unlike those it was trained on.

9.1.1 The idea: a curve through a few points #

Draw eight points on a page and ask for a curve through them. Infinitely many curves pass through all eight. They agree exactly at the points and can do anything at all between and beyond them. Nothing in the points chooses among them. Whoever draws the curve chooses, by a preference brought to the task: a smooth curve, a short formula, a curve that levels off at the edges.

A trained network is in the same position, with millions of parameters in place of a pencil. When the network has more parameters than it needs to fit its data, many settings fit equally well, and Section 8.3.6 observed that gradient descent is indifferent among them. Which one a run ends in depends on things the loss never mentions: the random starting point, the order of the examples, the learning rate, the architecture. Held-out data drawn like the training data cannot tell the candidates apart either, because on that distribution their scores are nearly the same, even where their individual predictions differ. The random-label experiment of Zhang and colleagues, reported in Section 8.4.5, is the extreme case: networks that generalize on real labels could also fit labels that carry no pattern at all, so the capacity to fit was never the constraint (Zhang et al., 2016).

9.1.2 Underspecification #

The idea needs two quantities. Write $\ell(f(x), y)$ for the loss of a predictor $f$ on one example, and $p$ for the distribution the training and held-out examples are drawn from. The risk of $f$ under $p$ is its expected loss, $L_p(f) = \E_{(x,y) \sim p}[\ell(f(x), y)]$, the quantity that held-out data estimates (Section 8.1.2). Deployment draws its inputs from a distribution $q$ that may differ from $p$.

The usage follows D'Amour and colleagues, who define it this way: "An ML pipeline is underspecified when it can return many predictors with equivalently strong held-out performance in the training domain" (D'Amour et al., 2022). The definition separates two properties. In-distribution equivalence says that every member of $\mathcal{F}_\epsilon$ scores nearly the same under $p$; a held-out test confirms it. Out-of-distribution divergence says that the members score differently under $q$; a held-out test cannot see it, because it samples from $p$. The authors state the consequence plainly: if the pipeline is sensitive only to performance on the training distribution, "it will return an arbitrarily chosen predictor from this class" (D'Amour et al., 2022). They also tie the definition to inductive bias, since members of the near-optimal set can "encode substantially different inductive biases" (D'Amour et al., 2022).

Underspecification is a property of a pipeline, not of a trained model. Any single model is a definite function. The question is how much the procedure that produced it constrained that function where the data did not reach.

9.1.3 Exploring many fits #

Figure 9.1 runs the experiment on the smallest scale that shows it. Over 300 sets of ten networks, the median of the largest disagreement anywhere inside the training interval is 0.10, and the median held-out error, measured on inputs drawn from the interval, is 0.0003. Only one of the 3,000 networks has a held-out error above 0.01; it fits the eight points but climbs steeply between the outermost point and the edge of the interval. By the standard check, nearly every network is equally good. Half a unit outside the interval, at $x = 1.5$, the median spread of the ten predictions is 0.39; at $x = 2$ it is 0.71, and at $x = 3$ it is 1.39, more than half the whole range of the curve that made the data. None of these networks is wrong about the training data. They differ only where the training data is silent, which is where deployment may take them.

The networks also share something the data did not give them. All ten are built of straight pieces, because a sum of hinges is linear between its kinks and beyond its last one, and all ten are the smallest-norm fit for their hidden layer. A different architecture or training rule would fan out differently: the procedure, not the data, decides the shape of the fan.

9.1.4 The evidence at scale #

D'Amour and colleagues, about forty authors most of them at Google, reported underspecification in a 2022 paper in the Journal of Machine Learning Research (D'Amour et al., 2022). Their protocol builds an ensemble by perturbing a small part of a pipeline, most often only the random seed, and confirms that its members perform nearly equally on held-out data from the training distribution. It then compares them on stress tests: evaluations on shifted, stratified, or deliberately constructed data that probe a property the application cares about (D'Amour et al., 2022). Their abstract lists examples "from computer vision, medical imaging, natural language processing, clinical risk prediction based on electronic health records, and medical genomics" (D'Amour et al., 2022). The genomics example is a linear risk score used as a warm-up; the other four are deep networks, several of them medical models from the authors' own organizations.

Three results show the pattern. Fifty ResNet-50 image classifiers, trained on ImageNet "using identical pipelines that differ only in their random seed", varied on some corrupted versions of the test images by an order of magnitude more than on the standard test set (D'Amour et al., 2022). Five pretrainings of the BERT language model, each fine-tuned twenty times, scored nearly alike on standard tasks. Yet the correlation between each model's gendered associations with occupations and official occupational statistics ranged from 0.3 to 0.7 on one task, and the spread came from the pretraining seed as well as the fine-tuning seed (D'Amour et al., 2022). And in a model predicting acute kidney injury from health records, an ensemble whose held-out scores were tightly clustered became "both worse and more widely dispersed" when the records were shifted in time of day and in which laboratory tests were ordered (D'Amour et al., 2022). The authors' summary: "We find evidence of underspecification in all applications, with downstream effects on robustness, fairness, and causal grounding" (D'Amour et al., 2022).

Varying only the seed explores a small part of the near-optimal set, so the authors say their method "underestimates the impact of underspecification"; that varying the optimizer or learning rate too would reveal a far larger set is their conjecture (D'Amour et al., 2022). Several of their significance tests are offered as descriptive, and the dermatology results as exploratory. Their remedy is to state, for each application, extra criteria that select among equally accurate predictors, and to stress-test the properties that matter (D'Amour et al., 2022).

If held-out accuracy cannot pick the right member, perhaps a measure of complexity can. Jiang and colleagues at Google tested that hope in a paper at ICLR 2020. They trained "over 10,000 convolutional networks by systematically varying commonly used hyperparameters" on two small image datasets and compared more than 40 proposed complexity measures against each network's actual generalization gap, holding hyperparameters fixed in turn (Jiang et al., 2020). Measures based on the size of the weights, which classical theory favors, often pointed the wrong way: larger values went with smaller generalization gaps, and so did larger parameter counts. Measures based on sharpness, how fast the loss rises when the weights are perturbed, "perform the best overall" (Jiang et al., 2020). The authors limit the claim to "relatively small architectures, which do not reflect the models used in production" (Jiang et al., 2020). The quantities classical bounds use to rank models ranked these networks poorly, and the better predictors were few (inference).

9.1.5 One question in three chapters #

Underspecification is the shared root of three problems the book takes up later, and stated side by side they are one question (inference). In goal misgeneralization, the near-optimal set contains a policy that pursues the intended goal and one that pursues a correlate of it, and training cannot tell them apart where the two coincided (Chapter 31). In the problem of eliciting latent knowledge, a model could learn either of two ways to report what it knows, which agree on every case a person can label and disagree on the cases that matter (Chapter 47). And weak-to-strong generalization asks whether a strong student trained on a weaker teacher's labels reproduces the teacher's errors on new inputs or follows its intent, when the labels themselves reward the errors (Chapter 42). In each, the data do not select the intended answer, so the outcome depends on what training prefers. The rest of this chapter is about that preference.

The practical lesson is the one D'Amour and colleagues draw. A held-out score certifies membership in the near-optimal set and nothing more. Disagreement among equally accurate models, which Section 8.4.4 met as a signal of uncertainty, is also a cheap detector of underspecification: where models that fit the same data disagree, the data did not settle the answer.

9.2 Double descent and benign overfitting #

The classical rule says that a model with more parameters than examples will overfit. Figure 8.1 showed the rule at work: a polynomial with twelve coefficients passed through twelve noisy points and swung wildly between them. Yet image classifiers usually have more parameters than training examples (Zhang et al., 2016), are trained until they fit those examples almost exactly, and generalize. Language-model pretraining is a different regime, since compute-efficient training stops "significantly before convergence" (Kaplan et al., 2020), and the results below bear on it only by analogy (inference). This section shows how the two observations fit together, in a model simple enough to solve exactly.

9.2.1 The idea: past the threshold, the optimizer chooses #

Take a linear model with $N$ adjustable weights, fitted by least squares to $n$ noisy examples. While $N < n$, no setting of the weights fits every example, and least squares averages over the noise: more examples per weight, less damage. At $N = n$ exactly one setting fits every example, noise included, and nothing averages the noise away; the fit is forced to chase it. This point, where the model can first fit its training data exactly, is the interpolation threshold. Past it, with $N > n$, infinitely many settings fit every example, and the optimizer, not the data, chooses among them.

Gradient descent started from zero weights makes a definite choice. Its steps are combinations of the training inputs, so it never moves the weights in a direction the inputs do not span, and among all exact fits it converges to the one with the smallest norm (Hastie et al., 2022). With many more weights than examples, that smallest fit can spread itself thinly over many directions, so each direction carries only a little of the noise. The error that peaked at the threshold falls again. That second fall is double descent, the name Belkin and colleagues gave the whole curve (Belkin et al., 2019).

9.2.2 Minimum-norm least squares #

Make the model precise. Each example has $N$ features, drawn independently with mean zero and variance one; stack the $n$ training inputs as the rows of a matrix $\mathbf{X}$. The labels are $y = \vx^\T \vw_\star + \text{noise}$, with a true weight vector $\vw_\star$ whose squared length, the signal, is $\|\vw_\star\|^2$, and noise of variance $\sigma^2$. The fitted weights are the minimum-norm least-squares solution $\hat{\vw}$: the shortest vector among those that minimize the squared error on the training data. Its prediction risk is the expected squared distance between its prediction and the true regression function on a fresh input, $R = \E[(\vx_0^\T \hat{\vw} - \vx_0^\T \vw_\star)^2]$, which is the test error above the noise floor. Write $\psi = N/n$ for the number of parameters per training example.

Hastie, Montanari, Rosset, and Tibshirani, in a 2022 paper in the Annals of Statistics, give the risk exactly as $n$ and $N$ grow together with $N/n \to \psi$ (Hastie et al., 2022):

Below the threshold the risk is all variance, the error that comes from fitting noise. Above it, the risk has a bias term, the part of the signal the fit misses, which grows with $\psi$, and a variance term, which shrinks with $\psi$. Both pieces can be seen directly.

The formula says why the second descent happens and also what it does not promise. As $\psi \to \infty$, the risk approaches $\|\vw_\star\|^2$, the null risk of predicting zero everywhere. In this model, where all the signal lives in the observed features, the global minimum lies below the threshold: the authors note that the risk "achieves its global minimum in the underparametrized regime" (Hastie et al., 2022). The second descent beats the peak, not the best small model.

9.2.3 Exploring double descent #

Figure 9.2 draws Equation (9.2). With signal $\|\vw_\star\|^2 = 5$ and noise $\sigma^2 = 1$, the risk is 1.0 at $\psi = 0.5$, infinite at the threshold, 3.5 at $\psi = 2$, and approaches the null risk of 5 from below. Its second dip bottoms out at 3.47, at $\psi = \sqrt{5}/(\sqrt{5} - 1) \approx 1.81$, a local minimum whose location Hastie and colleagues derive (Hastie et al., 2022). Switching to the misspecified model changes the verdict. There the true function uses infinitely many features and the model observes the first $N$; adding features shrinks the part of the signal it cannot see. With the same signal and noise, every model below the threshold does worse than predicting zero, and the lowest risk, 4.62, lies near $\psi = 5.3$, past the threshold. The authors state the general condition: in a misspecified model, "the risk can attain its global minimum in the overparametrized regime" (Hastie et al., 2022).

The ridge buttons add a penalty on the squared weights, the weight decay of Section 8.4.1. Any penalty keeps the fit from chasing noise at the threshold, so the risk there stays finite; a small penalty can leave a rounded bump, as in the misspecified model, and the tuned one removes the peak. Tuned to its best value at each $\psi$, the ridge fit "dominates the min-norm least squares estimator in risk, across all values of" $\psi$ and the signal-to-noise ratio (Hastie et al., 2022). The peak is a property of fitting exactly, not of having many parameters.

9.2.4 From statistics: the trade-off and its revision #

From statistics. Statistics taught for decades that test error is a bias-variance trade-off: a richer model misses less of the signal (lower bias) and fits more of the noise (higher variance), so error is U-shaped in model size. Belkin, Hsu, Ma, and Mandal set that teaching beside practice in a 2019 paper in the Proceedings of the National Academy of Sciences. Of models trained to fit their data exactly they wrote, "Classically, such models would be considered over-fit, and yet they often obtain high accuracy on test data" (Belkin et al., 2019). They showed test error peaking at the interpolation threshold and falling beyond it, for random-feature models, small fully connected networks, and random forests on subsets of standard datasets. They "posit a mechanism": a larger family contains smoother exact fits, and "Choosing the smoothest function that perfectly fits observed data is a form of Occam's razor" (Belkin et al., 2019). They add that regularization can mask the peak, and that for data at the scale of ImageNet "the classical regime of the U-shaped risk curve is more appropriate to understand generalization" (Belkin et al., 2019).

The linear theory then made the picture exact. Hastie and colleagues write that their results "formally verify that this double descent phenomenon occurs even in the simple and fundamental case of least squares regression" (Hastie et al., 2022). Bartlett, Long, Lugosi, and Tsigler, in a 2020 paper in the same journal as Belkin's, asked when the exact fit of noisy data still predicts nearly as well as possible, a case they call benign overfitting. In linear regression with suitably independent features, they found that "the number of directions in parameter space that are unimportant for prediction must significantly exceed the sample size" (Bartlett et al., 2020). The noise is hidden in those many unimportant directions, where it barely affects predictions. The error from the signal must be small too; the paper's bound on it grows with the size of the true weights and is small when the total variance of the features is small compared to the sample size (Bartlett et al., 2020).

What the field implies for this chapter is a change of question. In this regime, generalization is governed by which exact fit the optimizer picks, and the parameter count says little about it (inference). The analogy also has a clear edge. The exact results are for linear models and for random features, where only the last layer is trained; Hastie and colleagues show that, for one class of random nonlinear features, the risk matches the linear case (Hastie et al., 2022). For networks whose every layer is trained, double descent is observed, not derived. Bartlett and colleagues say their own results are "not directly applicable" to the idealized network setting they discuss, and call extending them to deep networks "an important open problem" (Bartlett et al., 2020).

9.2.5 Double descent in deep networks #

Nakkiran and colleagues, at Harvard and OpenAI, found the pattern in modern networks, in a 2021 paper in the Journal of Statistical Mechanics (Nakkiran et al., 2021). Image classifiers on CIFAR-10 and CIFAR-100, and transformer translation models, showed test error first worsening and then improving as width grew, and a large model trained longer could likewise get worse and then better. Near the peak, more data could hurt: "for some models, the performance for 18k samples is worse than for 4k samples" in a translation task (Nakkiran et al., 2021). Two conditions travel with these results. The authors "observe all forms of double descent most strongly in settings with label noise in the train set", labels deliberately corrupted at random, and "many of the phenomena that we highlight often do not occur with optimal early-stopping" (Nakkiran et al., 2021). Their account, a measure of the training procedure's effective complexity that places the peak where the procedure can just fit the training set, is offered as a hypothesis that is "informal in several ways" (Nakkiran et al., 2021).

A note by researchers at Anthropic, on toy models they trained, connects the peak to how a network stores what it learns. Henighan and colleagues trained small autoencoders, networks that compress an input and reconstruct it, on synthetic data sets of varying size. They report that "Overfitting corresponds to storing data points, rather than features, in superposition", that is, packing more items into a layer than it has units, and that "We observe double descent as the model transitions between these regimes" (Henighan et al., 2023). They call the note "a very preliminary investigation" of toy models (Henighan et al., 2023), and Chapter 44 teaches the geometry it relies on. One application runs forward. Wu and Sahai prove, in a stylized linear model, that a strong student fitted exactly to a weak teacher's nearly random labels can still generalize, given enough weakly labeled examples, an account of weak-to-strong generalization through benign overfitting that Section 42.2 reports (Wu and Sahai, 2025).

9.3 Simplicity bias and its failure mode #

Underspecification says that training chooses among many fitting functions. This section asks which way the choice leans. The best-supported answer is that training leans toward simple functions, in several senses of "simple" that turn out to be related. The same lean helps explain why overparameterized networks generalize and why they can fail badly when the simplest available explanation of the data is the wrong one. It is also the hinge of a debate about whether training would favor deceptive models.

9.3.1 The idea: random parameters compute simple functions #

Many parameter settings compute the same function: swap two hidden units and nothing changes, scale one layer up and the next down and nothing changes. So the map from parameters to functions, which Valle-Pérez, Camargo, and Louis call the parameter-function map, sends large regions of parameter space to single functions. Draw the parameters at random and ask how often each function comes out. If every function were equally likely, a network would have no preference among the functions that fit its data, and the many fits of Section 9.1 would leave generalization a matter of luck. They argue, in a paper at ICLR 2019, that the distribution is far from uniform (Valle-Pérez et al., 2019).

Their argument borrows from algorithmic information theory, the study of how much information a single object contains. The Kolmogorov complexity of a string is the length of the shortest program that prints it; it cannot be computed exactly, but a compression algorithm gives an estimate. A result they cite says that for a large class of input-output maps, the probability that a random input produces output $x$ is bounded by $P(x) \le 2^{-a\tilde K(x) + b}$, with $\tilde K$ an estimate of the complexity of $x$ and $a, b$ constants of the map (Valle-Pérez et al., 2019). A simple output can be very probable; a complex one cannot. The authors are careful about what this shows: the arguments "do not prove that maps are biased", but "if there is bias, it will follow Eq. (1)", the bound just given (Valle-Pérez et al., 2019).

They then measured it. A network with 7 Boolean inputs and two hidden layers of 40 rectified linear units computes one of $2^{128}$ possible functions, each written as a 128-bit truth table. Sampling its parameters at random, they found a probability distribution "spanning as many orders of magnitude as the finite sample size allows", "with all high-probability functions having low Lempel-Ziv (LZ) complexity", the compression estimate they used (Valle-Pérez et al., 2019). Their headline claim is strong: "We claim that this intrinsic bias towards simpler functions is the fundamental source of regularization that allows DNNs to generalize", DNNs being deep neural networks (Valle-Pérez et al., 2019).

9.3.2 Exploring the parameter-function map #