Draft complete but not yet reviewed; expect revisions

Learning Machines

The previous chapter treated an agent's objective as given: something that ranks outcomes, from which choices follow (Chapter 7). The systems this book is about are not built that way. Nobody writes their behavior, and nobody writes their objective. A developer chooses a family of functions, a body of examples, and a measure of error, and an optimization procedure adjusts millions or billions of numbers until the error is small. What the resulting system does on inputs it never saw in training is a consequence of that procedure, not of anyone's instruction. The second of the book's four problems, instillation (getting a learner to acquire what was specified rather than a proxy that scored the same on the training data), is a question about what this procedure produces.

One observation runs through the chapter: training sees only the data and the measure of error, never the intention behind them. The chapter builds the procedure from the smallest learning problem, fitting a curve to a dozen points, up to the networks behind today's language models. It then follows the observation outward: to the ways a well-fitted function fails on new inputs, and to the regularities that let developers predict a model's error, though not its abilities, before training it. Why networks generalize at all, and which of many fitting solutions training picks, is the next chapter's subject (Chapter 9). Nearly every claim here concerns the first of the book's three relationships, the training process and the signal it optimizes.

8.1 Learning as fitting a function #

Suppose an engineer needs a program that reads a support ticket and says whether it is urgent. Writing the rules by hand fails quickly: urgency shows in phrasing, context, and combinations of words that no list of keywords captures. What the engineer does have is examples, thousands of past tickets that a person marked urgent or not. Machine learning is the craft of turning such examples into a program. This section sets out its basic vocabulary, which every later chapter uses.

8.1.1 The idea: choose a curve from a family #

Start smaller. A dozen points lie roughly along a curve, and the task is to predict the height of the curve at a new position. One approach picks a family of candidate curves, such as all straight lines $y = wx + b$, and chooses the member that passes closest to the points. The two numbers $w$ and $b$ that select a member are the parameters. A richer family, polynomials of degree $d$, has $d + 1$ parameters and can bend to follow the points more closely. Choosing the parameters from the examples is training, or fitting; using the chosen curve on a new input is inference.

Everything else in machine learning elaborates this picture. The family may be a neural network with billions of parameters, the input an image or a document, and the output a probability for every word that could come next. But the logic stays the same: a family fixed in advance, parameters chosen to match examples, and a chosen member then used on inputs it was not chosen on.

8.1.2 Data, model, and loss #

A dataset $\D$ is a collection of pairs $(x, y)$. The input $x$ is described by features, the measurable quantities a program receives, such as the pixels of an image or the words of a ticket. The output $y$ is the label, the answer the example records. A model is a function $f_\theta$ from inputs to outputs whose behavior depends on a vector of parameters $\theta$; in a neural network the parameters are called weights. A loss measures how far the model's output is from the label. For a numerical label, the usual choice is the squared error, and the average over the dataset is the empirical risk:

Training chooses $\theta$ to make $L(\theta)$ small, a procedure called empirical risk minimization. Predicting a number is regression; predicting one of several categories is classification.

For classification the model first outputs one number per class, called a logit, and turns the vector of logits $\vz$ into probabilities with the softmax,

which makes every entry positive and the entries sum to one, while keeping their order. Write $p_\theta(y \given x)$ for the probability the softmax gives class $y$ on input $x$. The loss for one example is then $-\log p_\theta(y \given x)$, the negative log of the probability the model gave the correct label. Its average over the data is the cross-entropy of Section 6.2.2, so minimizing it is maximum likelihood estimation (Section 6.1.5), and the loss of a language model in Section 6.2.3 is the same quantity with the next token as the label.

The quantity a developer cares about is not $L(\theta)$ on the training data but the loss on new inputs. Practice therefore splits the available examples. The training set is used to fit the parameters. A validation set is used to choose among settings that training does not choose, such as the degree of the polynomial, which are called hyperparameters. A test set is held back and used once, at the end, to estimate the loss on new inputs. Data not used for fitting is called held-out data.

The definition contains an assumption that matters for the rest of the book: new inputs are drawn from the same distribution as the training inputs, each independently of the others. The test set estimates the gap only under that assumption. When deployment inputs differ systematically from the training inputs, a change called distribution shift, the test set says nothing direct about deployment, and Section 8.4 shows what can happen then.

import numpy as np

rng = np.random.default_rng(0)
x = rng.uniform(-1, 1, size=200)                  # inputs
y = 2.0 * x + 0.5 + 0.1 * rng.normal(size=200)    # labels: a line plus noise

w, b = 0.0, 0.0       # the parameters theta
eta = 0.1             # the learning rate
for step in range(500):
    err = (w * x + b) - y            # forward pass: prediction minus label
    loss = np.mean(err ** 2)         # empirical risk
    grad_w = 2 * np.mean(err * x)    # dL/dw
    grad_b = 2 * np.mean(err)        # dL/db
    w -= eta * grad_w                # step against the gradient
    b -= eta * grad_b
print(round(w, 2), round(b, 2), round(loss, 4))   # 1.99 0.49 0.0104

The loop recovers the line that generated the data, a slope near 2 and an intercept near 0.5, and its final loss of about 0.01 is the variance of the noise, the part no line can explain. Every training procedure in this book has this shape: compute the outputs, compute the loss, compute how the loss changes with each parameter, and move each parameter a little in the direction that lowers the loss. Section 8.3 explains the last two steps and how they scale to billions of parameters.

8.1.3 Exploring the fit #

Figure 8.1 runs the experiment. The training error can only fall as the degree rises, because each family contains the one before it. The test error does not. Over 1,000 datasets drawn this way, the median test error is 0.27 for a straight line and lowest at degree 3 (0.085, against the noise floor of 0.0625). It is about 153 at degree 11, where the fit passes through every point and its training error is zero. The training error at degree 11 says nothing about how badly the curve behaves between the points; only held-out data shows it. This U-shaped test curve is the classical picture of overfitting. For networks much larger than their data, the picture is incomplete in a way Section 9.2 takes up.

8.1.4 What the specification is #

The example carries the point this book needs from machine learning. The fitted curve was never told about $\sin(2\pi x)$. It received twelve pairs of numbers and a loss, and the loss rewarded closeness to those pairs and nothing else. Whatever the engineer meant, the specification a trained system actually receives is the data and the loss; the intention behind them reaches it only as far as they express it. This is where instillation begins (Chapter 1). Ngo, Chan, and Mindermann argue, in a paper at ICLR 2024, that models trained like today's most capable ones could learn misaligned internally represented goals that generalize beyond their fine-tuning distributions (Ngo et al., 2024). That is an argument about what training might produce rather than an observation, and Chapter 31 weighs it against the evidence.

8.2 Neural networks #

Polynomials served for one input, but they do not scale. A polynomial of degree 3 in 1,000 inputs has 167,668,501 coefficients, one for every way of choosing up to three inputs to multiply, repeats allowed, and an image or a paragraph has far more than 1,000 numbers in it. The family of functions has to be rich enough to represent what matters in images and language, yet small enough to store and to fit. Neural networks are the family that turned out to meet both demands, and this section builds one from its smallest part.

8.2.1 The idea: compose simple pieces #

The smallest part is a unit. It takes a weighted sum of its inputs, adds a number called its bias, and passes the result through a fixed nonlinear function. The choice this chapter uses is the rectified linear unit, or ReLU, $\max(0, z)$, which passes positive numbers and replaces negative ones by zero. With a single input $x$, a unit computes $\max(0, wx + b)$: zero up to the point $x = -b/w$, then a straight line. Its graph is a hinge.

A sum of hinges with different kink points is a curve made of straight segments, one segment between each pair of neighboring kinks, and with enough kinks it can follow any smooth curve as closely as desired. Figure 8.2 shows this with one layer of units whose kinks are evenly spaced, and with the output weights chosen to make the sum as close as possible to a target curve. With 4 units the largest error is 0.59; with 16 it is 0.044; with 40 units, 121 parameters in all, it is 0.0071. Doubling the number of units divides the error by roughly four, because each straight segment's error shrinks with the square of its length.

8.2.2 Layers, depth, and width #

A layer applies many units to the same input at once. Writing the units' weights as the rows of a matrix $\mW$ and their biases as a vector $\vb$, a layer maps an input vector $\vx$ to

where the activation function $g$, such as the ReLU, acts on each entry separately. The output $\vh$, the vector of the units' values, is called the layer's activations. A network stacks layers, each reading the activations of the one before, and ends with a layer that has no activation function and outputs the prediction, or the logits of Equation (8.2):

The parameters $\theta$ are all the entries of all the $\mW$ and $\vb$. The number of layers is the network's depth, and the number of units in a layer its width. A layer with $n_{\text{in}}$ inputs and $n_{\text{out}}$ units has $n_{\text{in}} n_{\text{out}} + n_{\text{out}}$ parameters. A network that reads a 28-by-28 grayscale image (784 numbers) through one hidden layer of 256 units into 10 class logits therefore has $784 \times 256 + 256 + 256 \times 10 + 10 = 203{,}530$ parameters. Layers between the input and the output are called hidden, because no label says what their units should compute.

The activation function is what makes depth worth having. Without it, each layer is an affine map (a matrix multiplication plus a constant), and a composition of affine maps is again an affine map, so a deep network would be no richer than a single layer. With it, each layer can compute new features of the features the layer below computed. Other activation functions exist. The Gaussian error linear unit, or GELU, is $x\,\Phi(x)$, with $\Phi$ the standard normal distribution function (Hendrycks and Gimpel, 2016b). In a 2016 preprint, its authors describe it as weighting inputs by their value rather than gating them by their sign, as the ReLU does, and report gains over the ReLU on the vision, language, and speech tasks they tested (Hendrycks and Gimpel, 2016b).

How rich is the family? The abstract of a 1989 paper by Cybenko states that finite sums of a fixed function of one variable, applied to affine functions of the input, can uniformly approximate any continuous function of several real variables on the unit hypercube, under mild conditions on that function (Cybenko, 1989). Such a sum is what one hidden layer followed by a weighted sum computes. The abstract adds that arbitrary decision regions can be approximated arbitrarily well with a single hidden layer and any continuous sigmoidal activation, an S-shaped curve rising from 0 to 1 (Cybenko, 1989). A paper by Hornik, Stinchcombe, and White in the same year bears the title "Multilayer feedforward networks are universal approximators" (Hornik et al., 1989). Results of this kind are called universal approximation theorems. They say a large enough network exists; they say neither how many units it needs nor how to find its weights. Goodfellow, Shlens, and Szegedy put the second limit plainly: the theorem "does not say anything about whether a training algorithm will be able to discover a function with all of the desired properties" (Goodfellow et al., 2015). The figure above sidestepped both questions by placing the kinks by hand and solving for the output weights.

8.2.3 Architecture as an assumption #

A network's architecture, the pattern of layers and connections, decides which functions are cheap to represent and easy to reach. Three ideas recur in this book.

A convolutional layer applies the same small set of weights at every position of an image, so that a pattern detected in one place is detected in every place. The sharing builds in an assumption, that what an object is does not depend on where it is. In a toy task where a fully connected network learned to classify shapes by their location, Geirhos and colleagues note that a convolutional network would have been prevented from taking this route by its design (Geirhos et al., 2020). An architecture is a choice of what the network finds easy to learn, a theme Chapter 9 develops.

A residual connection adds a block's input to its output. He and colleagues at Microsoft Research found that deeper plain networks could have higher training error than shallower ones, a degradation that was not caused by overfitting, since a deeper network could in principle copy the shallower one and add layers that do nothing (He et al., 2015a). They let each block of layers compute a correction $F(\vx)$ and output $F(\vx) + \vx$, so that doing nothing is the easy default, and trained networks of up to 152 layers on the ImageNet image-classification benchmark (He et al., 2015a). The residual stream of a transformer, the architecture behind today's language models, is the running sum that every layer reads from and adds to, and it is built on this idea (Section 11.2).

A normalization layer rescales a vector of activations so that its typical size stays fixed as it passes through many layers, which stabilizes training. Root mean square normalization divides each vector by the root of the mean of its squared entries (Zhang and Sennrich, 2019); the transformer chapter uses it.

8.2.4 Where the meaning lives #

It is tempting to read a trained network unit by unit: this unit detects faces, that one detects text. An early study of image classifiers by Szegedy and colleagues, in a paper at ICLR 2014, found that "there is no distinction between individual high level units and random linear combinations of high level units, according to various methods of unit analysis" (Szegedy et al., 2014). The authors read this as suggesting that the space of activations, rather than the individual units, carries the information (Szegedy et al., 2014). A representation spread across many units in this way is called distributed. Whether, and how, the directions of that space correspond to concepts is the question Chapter 44 takes up, and much of the book's account of inspecting a model from the inside depends on its answer.

8.3 Gradient descent and backpropagation #

A network like Equation (8.4) has anywhere from thousands to trillions of parameters, and its loss is a function of all of them at once. No formula gives the minimum, and trying settings at random is hopeless: even a coarse grid of ten values per parameter has $10^{1000}$ points for a network with a thousand parameters. Every network in this book is trained instead by a method that uses one local fact, the slope of the loss at the current parameters, and a way of computing that slope for every parameter at once.

8.3.1 The idea: walk downhill #

Take one parameter and the loss $L(w) = (w - 3)^2$, whose minimum is at $w = 3$. Its derivative $L'(w) = 2(w - 3)$ is positive to the right of the minimum and negative to the left, so stepping against it always moves toward the minimum. The update

with a small positive learning rate $\eta$ does this. Here it moves $w - 3$, the distance to the minimum, to $(1 - 2\eta)(w - 3)$. With $\eta = 0.1$ the distance shrinks by a factor of 0.8 per step. With $\eta = 0.6$ the factor is $-0.2$: each step overshoots to the other side but lands closer. With $\eta = 1.2$ the factor is $-1.4$, and each step overshoots farther than the last, so the iteration diverges.

The same arithmetic holds near the minimum of any smooth loss, because there the loss is close to a parabola. If its second derivative, the curvature, is $L''$, each step multiplies the distance to the minimum by $1 - \eta L''$, so the iteration converges exactly when $0 < \eta < 2/L''$. A step too small crawls; a step too large diverges.

8.3.2 The gradient and the learning rate #

With many parameters, the slope becomes the gradient $\nabla L(\theta)$, the vector of partial derivatives of the loss with respect to every parameter. It points in the direction in which the loss increases fastest, so gradient descent steps the other way:

The difficulty in many dimensions is that different directions curve differently. Take $L(w_1, w_2) = \tfrac12 (w_1^2 + 25 w_2^2)$, a valley that is steep across ($w_2$, curvature 25) and shallow along its floor ($w_1$, curvature 1). Each coordinate obeys its own copy of the one-parameter arithmetic. The steep direction caps the learning rate at $2/25 = 0.08$; at any rate below that, the shallow direction shrinks by a factor of at least $1 - 0.08 = 0.92$ per step, so progress along the floor takes dozens of steps. The ratio of the largest to the smallest curvature, here 25, is the condition number, and it measures how badly one learning rate fits all directions.

One remedy is momentum. Instead of stepping along the current gradient alone, the optimizer keeps a running sum of its past steps, each multiplied by a fixed momentum coefficient below one, and steps along that sum. Along the floor of the valley successive gradients point the same way and the sum builds up speed; across the valley they alternate in sign and largely cancel.

Figure 8.3 shows all three regimes, with the loss starting at 22.6. After 30 steps, a learning rate of 0.01 has reached a loss of 5.5, a rate of 0.07 reaches 0.13, and a rate of 0.085 has diverged to about $1.5 \times 10^4$. At the rate of 0.07, adding momentum with coefficient 0.5 brings the loss after 30 steps to $1.7 \times 10^{-4}$. With coefficient 0.9 the path swings around the minimum, and the loss after 50 steps (0.14) is higher than after 30 (0.012).

8.3.3 Minibatches and adaptive steps #

The empirical risk Equation (8.1) is an average over the dataset, so its gradient is the average of one gradient per example. For a dataset of billions of examples, computing the full average for every step is wasteful. Stochastic gradient descent instead computes the gradient on a small random minibatch of examples at each step. Because a random minibatch's average is an unbiased estimate of the full average, each step points downhill on average, at a small fraction of the cost. One pass through the whole dataset is an epoch. The noise of minibatch gradients is not only a cost: it also decides which of several equally good solutions a run ends in, a fact the next chapter uses.

One refinement, Adam, adapts the step to each parameter. Kingma and Ba introduced it in a paper at ICLR (Kingma and Ba, 2014). It keeps a running average of each parameter's gradient and of its squared gradient, and divides the first by the square root of the second. Each parameter then moves by a step of roughly the size the learning rate sets, whether its gradients are large or small. The authors note that the method is invariant to diagonal rescaling of the gradients, meaning that multiplying one parameter's gradients by a constant does not change its steps (Kingma and Ba, 2014). Kaplan and colleagues, whose scaling study Section 8.5 reports, trained their models with it, switching to a related optimizer, Adafactor, for their largest models because of memory limits (Kaplan et al., 2020).

8.3.4 Backpropagation #

Gradient descent needs $\nabla L(\theta)$ for every parameter at every step. Estimating each partial derivative by nudging its parameter and measuring the change in loss would take one evaluation of the network per parameter, a billion evaluations for a billion parameters. Backpropagation computes all of them in a single backward sweep through the network. Its cost is a small multiple of one evaluation: in the accounting of Kaplan and colleagues, the backward pass takes about twice the computation of the forward pass (Kaplan et al., 2020).

The method is the chain rule of calculus applied to a computation graph, the network written as a sequence of elementary operations, each feeding the next. The forward pass evaluates the operations in order, from the input to the loss, and stores every intermediate value. The backward pass then visits the operations in reverse. At each it multiplies the derivative of the loss with respect to the operation's output, already computed, by the operation's own local derivative, giving the derivative with respect to its inputs. A software engineer will recognize this as reverse-mode automatic differentiation, which deep learning libraries implement for arbitrary programs. The name echoes the title of a 1986 paper by Rumelhart, Hinton, and Williams, "Learning representations by back-propagating errors" (Rumelhart et al., 1986).

8.3.5 The shape of the landscape #

A loss that is not a bowl can have many valleys, and an early worry was that gradient descent would stall in poor ones. Three results suggest the landscape is kinder than feared, at least for the networks they studied. Goodfellow, Vinyals, and Saxe, in a paper at ICLR, evaluated the loss along the straight line from a network's initial parameters to its trained ones, for seven models that included image classifiers and a recurrent model of text. They found that "a variety of state of the art neural networks never encounter any significant obstacles" on that path (Goodfellow et al., 2014). In one generative model the line did pass a secondary minimum with high error, which training itself steered around (Goodfellow et al., 2014). Frankle and colleagues trained two copies of a network from the same intermediate point with different random data orders (Frankle et al., 2019). They found that standard vision models become stable to this noise early in training, after which "the outcome of optimization is determined to a linearly connected region" (Frankle et al., 2019). And Frankle and Carbin found that dense networks contain subnetworks, less than 10 to 20 percent of the original size in their fully connected and convolutional networks for two small image benchmarks, that train in isolation to comparable accuracy when reset to their original initial weights; randomly reinitialized, the same subnetworks no longer matched the original network (Frankle and Carbin, 2018). All three concern networks far smaller than today's language models, and the last two concern image classifiers only.

8.3.6 What the optimizer does not see #

Gradient descent follows the loss and nothing else. Two parameter settings with the same loss are equally good to it, however differently they behave on inputs the loss never evaluates. When a network has more parameters than it needs to fit its data, there are many such settings, and which one a run ends in depends on the starting point, the learning rate, the minibatch noise, and the architecture, none of which the loss mentions. This is the observation the next chapter builds on (Chapter 9): the data and the loss narrow the set of fitted functions, and the training procedure picks from what remains.

8.4 Generalization and its failures #

Low training loss is not the goal. A ticket classifier is worth something only if it labels next month's tickets well, and a language model only if it behaves well on prompts no one wrote during training. Generalization is the name for behavior on new inputs, and this section is about why it can fail while every number the developer measured looked good.

8.4.1 The idea: held-out data and its assumption #

The standard check is the one Figure 8.1 used: hold out data the fit never saw and measure the loss there. Two remedies follow when the held-out loss is much worse than the training loss. Regularization changes the objective to prefer simpler fits. In one form, weight decay, the objective adds a penalty proportional to the sum of the squared weights, which pulls every weight toward zero unless the data pay for it. Early stopping watches the validation loss during training and stops when it begins to rise, before the fit has time to chase the noise.

Both remedies, and the held-out check itself, rest on the assumption stated in Section 8.1: new inputs come from the same distribution as the training inputs. Held-out data drawn the same way can detect overfitting, the gap between fitting particular examples and fitting their distribution. It cannot detect a model that has fitted the distribution well by a rule that stops working when the distribution changes. Under distribution shift, the test set answers a question about a world the model will no longer be in. Even a careful re-collection of a benchmark's test set, following the original procedure, cost a broad range of image classifiers accuracy (Recht et al., 2019); Section 15.7.2 reports that measurement and what it does and does not show.

8.4.2 Shortcuts #

In a 2020 perspective in Nature Machine Intelligence, Geirhos and colleagues argued that many of deep learning's failures are symptoms of one problem, shortcut learning (Geirhos et al., 2020). They define shortcuts as "decision rules that perform well on standard benchmarks but fail to transfer to more challenging testing conditions, such as real-world scenarios" (Geirhos et al., 2020). Their own toy example makes the mechanism plain. A fully connected network learned to tell stars from moons in images where, in both the training data and the held-out test data, stars appeared in two corners and moons in the other two. Tested on images where location no longer predicted the shape, it fell to chance, having learned location rather than shape (Geirhos et al., 2020). The authors observe that a test drawn the same way as the training data cannot separate such rules: classifying by shape, by the number of white pixels, or by location all scored well on it (Geirhos et al., 2020).

The perspective collects cases from practice: a classifier that recognized cows on grass but not on a beach, and a pneumonia detector for chest scans that performed poorly at new hospitals, having learned to identify the hospital that took the scan (Geirhos et al., 2020). It also notes parallels outside machine learning, in comparative psychology and in education, which suggests to the authors that shortcut learning may be a common characteristic of learning systems (Geirhos et al., 2020).

Accuracy alone does not reveal a shortcut, because the shortcut earns the accuracy. Lapuschkin and colleagues, in a 2019 paper in Nature Communications, used an attribution method, a procedure that assigns each input pixel a share of the credit for a decision and draws the shares as a heatmap (Lapuschkin et al., 2019). On an image benchmark, the heatmaps of a classifier with high test accuracy concentrated on a source tag, a small caption that appeared in about one fifth of the horse images. Removing the tag removed the classifier's ability to recognize the horse, and pasting it onto a picture of a car made it answer "horse" (Lapuschkin et al., 2019). The classifier was a Fisher vector model, not a neural network; a deep network fine-tuned on the same data attended to the horse and its rider (Lapuschkin et al., 2019). The authors conclude that "standard performance evaluation metrics can be oblivious to distinguishing these diverse problem solving behaviors" (Lapuschkin et al., 2019). They call such models Clever Hans predictors, after a horse that appeared to answer arithmetic questions, a case Section 36.6 reports.

A shortcut is not a defect of a bad optimizer. Figure 8.4 computes the simplest case exactly. A classifier sees two features: a core feature that the designer intends it to use, which predicts the label correctly 85 percent of the time, and a background cue that, in the training data, agrees with the label 95 percent of the time. The model is logistic regression, a weighted sum of the features passed through the two-class softmax of Section 8.1, at the exact optimum of its training loss.

The optimum weights the background cue about 1.4 times as heavily as the core feature, because in training the cue is the more reliable of the two. Its training accuracy is 96 percent, better than the intended rule's 85. On test data where the cue agrees half the time, its accuracy is 67 percent; where the cue is reversed, 35. Nothing in the training loss, the training accuracy, or a test set drawn like the training set warned of this, because on that distribution the classifier is correct to rely on the cue. Real networks can go further than this optimum: Geirhos and colleagues note decision rules that depend on a single predictive pixel while all other evidence is ignored (Geirhos et al., 2020), a preference for simple cues that the next chapter examines (Section 9.3). Shortcut learning is the supervised ancestor of goal misgeneralization, in which an agent keeps its competence under distribution shift but pursues a goal that only coincided with the intended one during training (Chapter 31).

8.4.3 Adversarial examples #

A second failure needs no change in what the inputs depict, only a deliberate search. Szegedy and colleagues reported, in the ICLR 2014 paper of Section 8.2, that image classifiers which generalize well can be made to misclassify an image by "applying a certain hardly perceptible perturbation, which is found by maximizing the network's prediction error" (Szegedy et al., 2014). They called the perturbed inputs adversarial examples. The perturbations were not accidents of one training run: the same perturbation could cause a different network, trained on a different subset of the data, to misclassify the same input (Szegedy et al., 2014).

Goodfellow, Shlens, and Szegedy, in a paper at ICLR 2015, argued that the cause is not that networks are too nonlinear but that they are too linear (Goodfellow et al., 2015). The argument fits in a few lines.

The authors note that ReLU networks and others are "intentionally designed to behave in very linear ways, so that they are easier to optimize" (Goodfellow et al., 2015). That suggests a cheap attack: compute the gradient of the loss with respect to the input by backpropagation, and move every input by $\epsilon$ in the direction of its sign. The authors call this the fast gradient sign method (Goodfellow et al., 2015). On handwritten digits, with $\epsilon = 0.25$ on pixel values between 0 and 1, it caused a maxout network, a network with a different piecewise linear activation, to misclassify 89.4 percent of adversarial examples. A change of that size is visible, but the authors note that people read such images without difficulty (Goodfellow et al., 2015). Training on such examples as they were generated, an approach called adversarial training, together with wider layers (1,600 units rather than 240) and early stopping on adversarial validation error, lowered that error rate to 17.9 percent (Goodfellow et al., 2015). Madry and colleagues later posed training against the worst perturbation within a budget as a robust optimization problem (Madry et al., 2018), and Section 48.1 teaches that formulation together with the record of defenses and the adaptive attacks that broke them. For language models the inputs are words, the budget has no natural definition, and Chapter 48 takes up what adversarial robustness means there.

8.4.4 Knowing when it is wrong #

A model that cannot avoid every failure could at least signal when it is likely to fail. Hendrycks and Gimpel, in a paper at ICLR, proposed a simple baseline: misclassified and out-of-distribution examples, inputs unlike the training data, tend to receive a lower maximum softmax probability, the probability the model gives its top answer, than correctly classified ones (Hendrycks and Gimpel, 2016a). That difference allows them to be detected across vision, language, and speech tasks (Hendrycks and Gimpel, 2016a). The result is about the ordering of probabilities, not their values. Read on its own, the softmax probability was a poor measure of confidence: random Gaussian noise fed to a classifier of handwritten digits received a predicted class probability of 91 percent (Hendrycks and Gimpel, 2016a). Whether stated probabilities can be taken at face value is the question of calibration (Section 6.4).

Lakshminarayanan, Pritzel, and Blundell, researchers at DeepMind, proposed deep ensembles in a paper at NeurIPS: train several networks from different random initializations, each with a loss that rewards honest probabilities, and average their predictions (Lakshminarayanan et al., 2016). They recommend five networks by default and found that random initialization and random data order were enough to make the members differ (Lakshminarayanan et al., 2016). Networks trained on handwritten digits and tested on handwritten letters, classes they had never seen, gave predictions of higher entropy as members were added, and a network trained on photographs of house numbers did the same on photographs of objects (Lakshminarayanan et al., 2016). Disagreement among models that fit the same data equally well is a signal that the data did not settle the answer, an idea the next chapter makes central.

8.4.5 The puzzle left open #

The account so far explains failure: a model can fit the training data by a rule that does not transfer. It does not explain success. Zhang and colleagues reported, in a 2016 preprint, that convolutional networks trained with stochastic gradient methods "easily fit a random labeling of the training data", with explicit regularization making little qualitative difference (Zhang et al., 2016). A network that can memorize random labels has the capacity to memorize real ones, so its capacity cannot be why it generalizes on real labels instead. Which of the many functions that fit the data training actually finds, and why it is so often one that generalizes, is the subject of Section 9.1.

8.5 Scaling laws #

A large training run is expensive enough to be done once. Before paying for it, a developer wants to know how good the model will be, and how to divide a fixed budget of computation between a larger model and more training data. For one quantity, the loss, there is an answer: over many orders of magnitude, loss has fallen along simple curves as models, data, and computation grew. This section teaches those curves, the evidence for them, and what they do not predict.

8.5.1 The idea: straight lines on logarithmic axes #

A power law is a relation $y = c\,x^{-a}$ for constants $c$ and $a > 0$. Taking logarithms gives $\log y = \log c - a \log x$, a straight line of slope $-a$ when both axes are logarithmic. A power law has no characteristic scale: every doubling of $x$ multiplies $y$ by the same factor, $2^{-a}$. With the exponent of about 0.076 that Kaplan and colleagues measured for language models trained to convergence on datasets large enough not to limit them, doubling the number of parameters multiplies the loss by $2^{-0.076} = 0.95$ (Kaplan et al., 2020). Each doubling buys the same five percent, so the gains are steady but each costs twice the last.

Loss cannot fall forever. Section 6.2.3 showed that the expected loss of a language model cannot go below the entropy of the text it predicts, so a better form is a power law plus a constant,

with $L_\infty$ the irreducible loss that no amount of scale removes and the power-law term the reducible loss. Here $x$ stands for whichever input to training is being scaled, and $x_0$ is a constant in the same units.

The pattern was visible before language models dominated. In a 2017 preprint, Hestness and colleagues at Baidu Research reported power-law scaling of error with training set size in machine translation, language modeling, image processing, and speech recognition, and found that better models shifted the error curve without appearing to change its exponent (Hestness et al., 2017).

8.5.2 The laws for language models #

For language models, the reference study is a 2020 preprint by researchers at OpenAI, Kaplan and colleagues, on transformers they trained themselves, the architecture of Chapter 11. Its abstract reports that the loss "scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude" (Kaplan et al., 2020). It adds that "other architectural details such as network width or depth have minimal effects within a wide range" (Kaplan et al., 2020). Here the model size $N$ counts the parameters outside the embedding layers that turn tokens into vectors, and the dataset size $D$ counts training tokens, the size of the dataset $\D$.

The study's accounting of computation recurs in the later papers below. A forward pass costs about $2N$ arithmetic operations per token, one multiply and one add per parameter. The backward pass costs about twice that, so training costs about $6N$ operations per token and $C \approx 6ND$ operations in all (Kaplan et al., 2020). The count ignores a term that grows with the length of the context, which is small for the models they studied (Kaplan et al., 2020). Computation is counted in floating-point operations, or FLOPs.

Two further findings concern how to spend a budget. Larger models were more sample-efficient: they reached a given loss with fewer training tokens. And for a fixed budget of computation, the authors found it best to train "very large models on a relatively modest amount of data" and stop "significantly before convergence" (Kaplan et al., 2020), with the optimal model size growing as about $C^{0.73}$. The authors were explicit about the limits. They write that they have no solid theoretical understanding of any of their scaling laws, and that their trends must eventually level off, since natural language has nonzero entropy (Kaplan et al., 2020). Their own equations contradict each other several orders of magnitude beyond the scales they studied, which implies that the laws break down before that point (Kaplan et al., 2020).

A companion preprint by Henighan and colleagues, also at OpenAI, fitted power laws plus a constant to transformers they trained on generative image modeling, video, paired images and text, and mathematical problem solving, and found the same smooth improvement in every domain (Henighan et al., 2020). The optimal model size grew with the compute budget as a power with exponent about 0.7 in all of them (Henighan et al., 2020). Section 6.2.3 reported the paper's reading of the constant as the data's entropy and of the power-law term as the model's KL divergence from the data. One result bears on generalization. On mathematics problems harder than any in training, models of different sizes that had reached the same training loss performed about equally, so that size helped extrapolation only through its effect on the training loss (Henighan et al., 2020).

8.5.3 Compute-optimal training #

Two years later, researchers at DeepMind revised the allocation. Hoffmann and colleagues, in a 2022 preprint, trained "over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens" (Hoffmann et al., 2022). They concluded that "the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled" (Hoffmann et al., 2022). They reached the conclusion by three methods, whose exponents Table 8.1 sets beside the earlier one.

Many large models of the time had been trained on about 300 billion tokens, in line with the earlier allocation, the authors observe (Hoffmann et al., 2022). The developers tested their prediction on their own models. With the compute budget of their 280-billion-parameter model Gopher, they trained Chinchilla, with 70 billion parameters and four times as much data (Hoffmann et al., 2022). They report that it "uniformly and significantly outperforms" Gopher and three other large models from other developers on a large range of downstream evaluation tasks (Hoffmann et al., 2022).

The authors give two reasons for the disagreement with Kaplan and colleagues (Hoffmann et al., 2022). The earlier study used the same number of training tokens and the same learning-rate schedule for every model. Its losses for shorter training were therefore taken partway through a schedule tuned for a longer run, which overstated the loss a properly scheduled short run reaches and so understated the effectiveness of training on fewer than 130 billion tokens, pushing the conclusion toward model size growing faster than data. And most of the earlier runs were small, many under 100 million parameters, while most of Hoffmann's had more than 500 million, which mattered because the frontier curves slightly at large sizes. The authors also list their own limits. They had only two comparable training runs at large scale, Chinchilla and Gopher, and assumed a power-law form. Curvature at high budgets suggests they may still overestimate the optimal model size, and all their runs used less than one pass through the data (Hoffmann et al., 2022).

Their third method fits the loss directly as a function of both inputs. In the book's notation it reads

the same loss $L$ as before, now as a function of model size and data. The first term, in the authors' account, corresponds to the entropy of natural text; the second is the cost of having only $N$ parameters; and the third the cost of training on only $D$ tokens (Hoffmann et al., 2022). They report the fitted values $L_\infty = 1.69$, $a_N = 0.34$, and $a_D = 0.28$ (their $E$, $\alpha$, and $\beta$), with $N_c^{a_N} = 406.4$ and $D_c^{a_D} = 410.7$ (their $A$ and $B$) (Hoffmann et al., 2022). Which allocation minimizes this loss on a fixed budget follows in a few lines.

Figure 8.5 draws the fitted law. The frontier is straight because both optimal quantities are powers of $C$, and the readout shows the number of tokens per parameter rising slowly along it, as the exponents $0.45$ and $0.55$ require. Gopher and Chinchilla lie near the same budget line, and the contours show the law placing Chinchilla on a lower loss.

8.5.4 Why power laws? #

The studies above offer no theory that derives their exponents, but solvable models show where power laws can come from. Maloney, Roberts, and Sully, in a 2022 preprint, constructed a statistical model, linear regression on random features of data generated from a latent space, that reproduces the scaling behavior observed in large models (Maloney et al., 2022). The essential ingredient was data whose covariance spectrum has many decades of power-law eigenvalues, so that larger models and datasets resolve more and more of the data's structure; with simpler data, they found no power law (Maloney et al., 2022). The power law in their model also ends: it requires the latent space of the data to be larger than both the model and the dataset (Maloney et al., 2022).

The exponent also depends on which data are used. Sorscher and colleagues, in a paper at NeurIPS 2022, showed in theory that a high-quality pruning metric, a score for which training examples to discard, could make error fall faster than a power law in dataset size, potentially exponentially (Sorscher et al., 2022). They observed better-than-power-law scaling for image classifiers on three benchmarks (Sorscher et al., 2022). They also found that most existing pruning metrics scaled poorly to the largest of the three, ImageNet (Sorscher et al., 2022). A scaling law describes a recipe for choosing data and training, not a law of nature (inference).

8.5.5 The inputs are growing #

Scaling laws turn a forecast of loss into a forecast of the inputs: how fast the computation, data, and methods available for training grow. Sevilla and colleagues, in a paper at IJCNN 2022, estimated the training compute of 121 milestone machine learning systems from 1952 to 2022 and found three trends (Sevilla et al., 2022), given in Table 8.2. The authors note that they assigned systems to the large-scale trend by an intuitive decision, justified after the fact, and that the data admit other interpretations (Sevilla et al., 2022).

Methods improve too. Ho and colleagues, in a paper at NeurIPS 2024, used over 200 evaluations of language models on two standard text benchmarks from 2012 to 2023 (Ho et al., 2024). They found that "the compute required to reach a set performance threshold has halved approximately every 8 months, with a 95% confidence interval of around 5 to 14 months" (Ho et al., 2024). Even so, they found that the growth in compute contributed more to the overall improvement than the improvement in algorithms, and they note that their analysis is limited by noisy benchmark data (Ho et al., 2024). How these input trends feed forecasts of what models can do is Section 36.3's subject.

8.5.6 What a scaling law predicts #

A scaling law predicts the average loss, and the average is a weak guide to any particular behavior. Section 6.2.4 showed that a model can have excellent average loss and wrong rare behavior, because rare contexts barely move the average. The same holds for abilities: a smooth fall in loss says nothing direct about when a model starts to do a specific task correctly. Kaplan and colleagues made the point themselves: "Smooth quantitative change can mask major qualitative improvements" (Kaplan et al., 2020). The next section asks how sudden such changes are.

8.6 Smooth loss, sudden abilities? #

If abilities tracked loss smoothly, the laws of Section 8.5 would forecast them, and a developer could see a dangerous capability approaching from a distance. If abilities appear suddenly at some scale, a model could cross a threshold that no smaller model hinted at. Which of these describes language models is disputed. This section sets out the two readings, the evidence for each, and what the evidence decides.

8.6.1 Emergence as claimed #

Wei and colleagues, most of them at Google, collected cases in a 2022 survey in Transactions on Machine Learning Research. They wrote: "We consider an ability to be emergent if it is not present in smaller models but is present in larger models. Thus, emergent abilities cannot be predicted simply by extrapolating the performance of smaller models" (Wei et al., 2022b). Their examples include eight tasks posed by few-shot prompting, in which a model is shown a few solved examples in its input and asked to solve a new one. On each, performance plotted against training computation stays near random until a threshold and then rises well above it (Wei et al., 2022b). Their general definition of emergence, "when quantitative changes in a system result in qualitative changes in behavior", is one they describe as rooted in the physicist Philip Anderson's 1972 essay "More Is Different" (Wei et al., 2022b; Anderson, 1972).

The survey weighs its own evidence. Its authors note that scoring long answers by exact string match "may disguise compounding incremental improvements as emergence", but they call this at best an incomplete explanation, because emergence also appears on classification tasks, where the model picks among fixed options (Wei et al., 2022b). For six of the emergent tasks, they report that the cross-entropy loss on the correct answer improved even at scales where the task metrics stayed near random (Wei et al., 2022b). And scale is not the only factor: on 14 tasks, PaLM, with 62 billion parameters, performed above random where LaMDA, with 137 billion, and GPT-3, with 175 billion, did not (Wei et al., 2022b). PaLM and LaMDA are Google's own models, so this comparison is partly a developer's report on its own systems.

8.6.2 Emergence as a property of the measurement #

Schaeffer, Miranda, and Koyejo, at Stanford, offered an alternative in a paper at NeurIPS 2023. For a fixed task and model family, they argue, apparent emergence comes from the researcher's choice of metric, not from fundamental changes in the model's behavior with scale (Schaeffer et al., 2023). In their words, "nonlinear or discontinuous metrics produce apparent emergent abilities, whereas linear or continuous metrics produce smooth, continuous, predictable changes in model performance" (Schaeffer et al., 2023). Their mechanism is simple enough to derive.

Figure 8.6 draws the same family scored at different answer lengths. For answers of 10 tokens, exact match stays below 1 percent until about a quarter of the way along the family and passes 50 percent at 0.88; for 20 tokens, it stays below 1 percent until 0.60 and passes 50 percent at 0.95. A small test set makes the step look sharper still. Schaeffer and colleagues name this second cause: with too few test questions, small models appear wholly unable to do a task they do slightly better than chance (Schaeffer et al., 2023).

The authors tested the account three ways. On the InstructGPT and GPT-3 family, the arithmetic abilities that looked emergent for answers of four or five digits under exact accuracy improved smoothly when the same outputs were scored by token edit distance, the number of tokens that must change to make the answer right (Schaeffer et al., 2023). With more test data, every model scored above chance even on accuracy (Schaeffer et al., 2023). In published results on BIG-Bench, a large public collection of language tasks, more than 92 percent of the emergent abilities that another group had annotated appeared under one of two metrics, multiple-choice grade and exact string match (Schaeffer et al., 2023). The LaMDA family's emergent abilities under multiple-choice grade disappeared under the Brier score, a continuous scoring rule (Section 6.4) (Schaeffer et al., 2023). Finally, they produced seemingly emergent abilities in vision networks of several architectures by choosing a sharp metric on purpose (Schaeffer et al., 2023). They add a limit on their own claim: "nothing in this paper should be interpreted as claiming that large language models cannot display emergent abilities" (Schaeffer et al., 2023).

8.6.3 What the evidence decides #

The two readings agree on more than their titles suggest. Both papers found that a smooth quantity, the loss on the correct answer or a continuous metric, kept improving while a thresholded metric stayed flat. They differ on what to call the jump, and on whether every jump has such a smooth precursor.

For the measurement reading, the strongest evidence is that the claimed cases it examined dissolved under a change of metric, and that the authors could manufacture the pattern at will. Its strongest objection is the one Wei and colleagues raised in advance: the measurement account explains why a final answer jumps once its steps are reliable, not why the quality of the steps themselves can rise suddenly (Wei et al., 2022b). A second objection concerns use rather than truth. Many abilities that matter for safety are themselves thresholds, an exploit that works or does not, so a smooth precursor helps forecasting only if someone has measured it for that ability and knows how it maps onto the threshold (inference).

For the emergence reading, the strongest evidence is the breadth of cases and the survey's observation that model scale is not the only route to them. Its strongest objection is that the cases were selected from a very large space: Schaeffer and colleagues estimate about $10^6$ combinations of task, metric, and model family in BIG-Bench alone, so some sharp-looking curves are expected by chance (Schaeffer et al., 2023).

The evidence does not decide whether any ability of a language model is emergent in the strong sense, with no smooth precursor at all. It does show that the cases these two papers examined in detail had one, and that whether a jump appears depends on how an ability is scored. The question for a forecaster is therefore practical: whether a continuous measure exists for the ability in question and has been tracked.

8.6.4 What forecasting achieves #

Owen, at the research group Epoch, tested the practical question directly in a 2024 preprint, extrapolating benchmark performance across many model families from scaled compute: the computation that, spent on compute-optimal training, would give the loss that Hoffmann and colleagues' fitted law (Equation (8.6)) predicts from a model's parameters and tokens (Owen, 2024a). The scores came from prompts with a few solved examples and no step-by-step reasoning (Owen, 2024a). Averaged over the tasks of BIG-Bench Hard, extrapolation across one order of magnitude of scaled compute had an average absolute error of 6 percentage points; for individual BIG-Bench tasks the error was 18 points (Owen, 2024a). Prediction also required that some models already performed better than chance: the study could not confidently predict when tasks still at or below chance would improve (Owen, 2024a). Aggregate performance was the more predictable; individual tasks were less so, though still "significantly more predictable than chance" (Owen, 2024a). A second approach tracks one broad capability directly, the length of software tasks, in human working time, that models complete half the time. Its authors report that on their task suites it has grown exponentially since 2019 (Kwa et al., 2025), and Section 36.3 teaches the method and its limits.

Two later questions depend on this section. Whether a dangerous capability can be seen coming is a question for evaluations and for the case for risk (Chapter 36; Chapter 4). And whether safety evidence gathered on smaller models transfers to larger ones depends on whether the properties measured change smoothly with scale (Chapter 5) (inference). Wei and colleagues note that risks, like abilities, could emerge with scale (Wei et al., 2022b); the measurement account implies that a smooth precursor would then exist to be tracked, if someone defines and measures it (inference).

8.7 Exercises #

Further reading #

• Kaplan et al. (2020): the power laws that turned model training into forecasting.
• Geirhos et al. (2020): one pattern behind many failures of deep networks.
• Szegedy et al. (2014): adversarial examples and distributed representations in one paper.
• Zhang et al. (2016): the random-label experiment that opens the next chapter.
• Sevilla et al. (2022): how fast the computation behind these systems has grown.