Draft complete but not yet reviewed; expect revisions

Probability and Information

Every object this book studies is, underneath, a probability distribution: an assignment of probabilities to the possible outcomes of something uncertain. A language model assigns a probability to each possible next token, and a trained policy is a distribution over actions. A reward model scores responses so that the gap between two scores sets the probability that one response is preferred to the other, and a safety claim about a deployed system is a claim about how often something happens. The previous part asked how the field knows what it claims to know (Chapter 5). This chapter supplies the arithmetic for answering that question in particular cases.

Four questions organize it. How should evidence change a belief? How far is one distribution from another, and in particular, how far has training pushed a model from the model it started as? Can a stated probability be taken at its word? And what does a record of clean tests, or a monitor's alarm, actually establish? The answers are Bayes' rule, the Kullback-Leibler (KL) divergence that the rest of the book uses to measure optimization pressure, calibration, and the statistics of rare events and detection. The last of these is the arithmetic behind every claim of the form "we tested it and it never misbehaved", whether the claim is about a training run, a deployed system, or a developer's report to the public.

6.1 Distributions, likelihood, and Bayes' rule #

Suppose a monitor, a program that reads an AI agent's actions and flags those that look like sabotage, flags one transcript among the day's thousands. How likely is it that the transcript records a real attack? The question has an answer only once three numbers are known: how often real attacks occur, how often the monitor flags a real attack, and how often it flags an innocent transcript. This section builds the vocabulary for combining them, starting from an object every engineer already knows.

6.1.1 A distribution is a table #

A random variable is a quantity whose value is uncertain: the next token a model will write, the reward a response will receive, whether the next episode, one complete run of an agent on a task, will contain an attack. Its distribution says how likely each value is. Over a finite set of values, a distribution is a table from values to non-negative weights that sum to one, a dictionary of probabilities. A language model's output at one position is such a table, with one entry for every token in its vocabulary. This book writes $p(x)$ for the probability of the value $x$ under a distribution $p$, and $\Prob(\mathcal{E})$ for the probability of an event $\mathcal{E}$, a set of outcomes such as "the transcript is flagged".

When the values are real numbers, such as a monitor's score, the table becomes a density $p(x)$, whose area over an interval is the probability of landing in it, and sums become integrals. The normal (Gaussian) distribution $\N(\mu, \sigma^2)$ is the bell curve centered at $\mu$ with width set by $\sigma$.

6.1.2 Joint, marginal, and conditional #

Two random variables have a joint distribution, a table over pairs of values. Let one variable say whether an episode is bad (it contains an attack) and the other whether the monitor flags it, and suppose one episode in a hundred is bad. Table 6.1 gives a joint distribution for a monitor that flags 95 percent of bad episodes and 1 percent of good ones.

The marginal distribution of one variable sums out the other: adding across the first row gives the probability 0.01 that an episode is bad. The conditional distribution of one variable given a value of the other restricts attention to that row or column and rescales it to sum to one:

Given that an episode is bad, it is flagged with probability $0.0095 / 0.01 = 0.95$. Given that an episode is flagged, it is bad with probability $0.0095 / 0.0194 \approx 0.49$. The two conditionals share a numerator and differ in what they divide by, and confusing them is a common error in reasoning about tests. Two variables are independent when $p(x, y) = p(x)\,p(y)$ for every pair of values: learning one never changes the distribution of the other.

Rearranging the definition of a conditional gives $p(x, y) = p(x)\,p(y \given x)$, and applying it repeatedly factors the probability of a whole sequence:

where $\ell$ is the length of the sequence and $x_{<t}$ stands for $x_1, \dots, x_{t-1}$. This is the chain rule of probability, a different rule from the chain rule of calculus that backpropagation, the algorithm that computes a network's gradients, uses (Section 8.3). It says that a model which predicts each token from the tokens before it assigns a probability to every complete text, the factorization Section 11.1 builds language models on.

6.1.3 Expectation, variance, and correlation #

The expectation of a random variable $X$ is its probability-weighted average, $\E[X] = \sum_x x\,p(x)$, written $\E_{x \sim p}[f(x)]$ when the distribution needs naming; the expectation of a sum is the sum of the expectations. The variance $\Var(X) = \E\big[(X - \E[X])^2\big]$ measures spread, and its square root is the standard deviation. The covariance $\Cov(X, Y) = \E\big[(X - \E[X])(Y - \E[Y])\big]$ is positive when one variable tends to be above its average whenever the other is. Divided by both standard deviations, it becomes the correlation coefficient

a number between $-1$ and $1$. Independent variables have $\rho = 0$, though a correlation of zero does not imply independence. When one variable is predicted from others, as when a benchmark score is predicted from a model's size, the share of the predicted variable's variance that the predictions account for is written $R^2$; for a straight-line prediction from a single variable, $R^2 = \rho^2$. Later chapters report $R^2$ as their sources give it, and Section 6.7 uses the correlation of two monitors' errors.

6.1.4 Bayes' rule #

The monitor's designer knows $p(\text{flag} \given \text{bad})$ from testing. The person reading a flag wants $p(\text{bad} \given \text{flag})$. Bayes' rule converts one into the other.

Each piece has a name the rest of the book uses. The prior $p(h)$ is the probability of the hypothesis before the evidence, and the posterior $p(h \given \mathcal{E})$ is its probability after. The odds of an event are its probability divided by the probability that it does not occur, so a probability of 0.2 is odds of 1 to 4. The ratio $p(\mathcal{E} \given h) / p(\mathcal{E} \given \neg h)$ is the likelihood ratio: how much more probable the evidence is if the hypothesis is true than if it is false. Equation (6.3) says that the posterior odds are the prior odds times the likelihood ratio. Taking logarithms turns the product into a sum: each piece of evidence that is independent of the others once it is known whether $h$ holds adds its log likelihood ratio to the log-odds, the logarithm of the odds.

For the monitor of Table 6.1, the likelihood ratio of a flag is $0.95 / 0.01 = 95$. At a base rate of one bad episode in a hundred, the prior odds are 1 to 99, the posterior odds about 1 to 1.04, and a flag is right about half the time, as the table showed. At one in ten thousand, the prior odds are 1 to 9,999, the posterior odds about 1 to 105, and a flag is right 0.94 percent of the time. The monitor did not change. Only the prior did.

Figure 6.1 makes the dependence visible. A flag multiplies the odds by 95 whatever the base rate, and what the multiplication produces depends entirely on what it multiplies. Section 6.7 returns to this arithmetic for monitors whose recall and false positive rate the designer can trade against each other.

6.1.5 Likelihood and maximum likelihood #

The quantity $p(\mathcal{E} \given h)$ has a second use. Read as a function of the hypothesis, with the evidence held fixed, it is called the likelihood of the hypothesis. When the hypotheses are the possible values of a parameter, the maximum likelihood estimate is the value that makes the observed data most probable. A coin that came up heads $k$ times in $n$ independent flips has likelihood $\theta^k (1 - \theta)^{n-k}$ for a heads probability $\theta$. Its logarithm, $k \log \theta + (n - k) \log(1 - \theta)$, has derivative zero at $\theta = k / n$, the observed frequency, which is the estimate.

A language model with parameters $\theta$ assigns, by Equation (6.1), a probability to every text in a corpus. Its log-likelihood is a sum, over every token, of the logarithm of the probability the model gave that token, and pretraining adjusts $\theta$ to maximize the sum (Section 11.1). Logarithms are not only a convenience here.

import math

# Next-token probabilities a model assigned to a 1,200-token text.
probs = [0.2, 0.05, 0.6] * 400

p = 1.0
for q in probs:
    p *= q            # 0.0: the product underflows long before the end

logp = sum(math.log(q) for q in probs)   # about -2,046 nats, no trouble

The product of these 1,200 probabilities is about $e^{-2046}$, far below the smallest positive double-precision number, near $e^{-745}$, while the sum of their logarithms is an ordinary number. Language models are trained and evaluated with log-probabilities for this reason.

6.1.6 Probability as a degree of belief #

So far a probability has been a frequency. Bayes' rule needs only the rules of conditional probability, so it applies equally to a degree of belief, a number measuring how strongly someone expects something, such as the chance that a model would sabotage a kind of task it has never been given. Section 7.2 shows how a Dutch book argument ties degrees of belief to consistent betting.

Bayes' rule is the update behind every later use of "prior" and "posterior" in the book. A 2021 analysis takes a pretraining distribution that is a mixture of hidden Markov models, a simple family of models of sequences. It proves that a predictor matching that distribution then learns a task from examples placed in its prompt, in the limit of infinitely many examples and when the task's concept is distinguishable from the others, because it infers a shared latent concept between the examples (Xie et al., 2021), a reading Section 11.3 teaches. Fine-tuning, further training of a pretrained model, can be read, when it carries a KL penalty, as updating a prior, the pretrained model, to conform with evidence provided by the reward (Korbak et al., 2022), a reading Section 12.4 develops. This chapter teaches only the rule; those chapters teach the readings.

6.2 Entropy, cross-entropy, and compression #

A language model is judged by the probabilities it gave to the text that actually came next. Many single numbers could summarize them: the average probability, the fraction of tokens ranked first, the worst case. One of them has a privileged status, and the reason comes from data compression. The surprise of an outcome, $-\log p(x)$, is the length of the shortest code for it under $p$, so a better predictor is a better compressor, and the natural score for a predictor is the length of the code it implies.

6.2.1 Surprise as code length #

Take a source that emits four symbols with probabilities $p = (1/2, 1/4, 1/8, 1/8)$. A binary code assigns each symbol a string of bits, and a code can be read without separators if no codeword is the beginning of another. Give A the codeword 0, B 10, C 110, and D 111. The lengths are 1, 2, 3, and 3 bits, which are exactly $-\log_2 p(x)$ for each symbol, and the expected length per symbol is

Frequent symbols get short codes and rare ones long codes, and the length $-\log p(x)$ is called the surprise of $x$. No code does better on average. Shannon's source coding theorem sets the limit: the expected length of any code is at least the entropy of the source (Delétang et al., 2024; Shannon, 1948),

the expected surprise. With logarithms in base 2 the unit is the bit, and with natural logarithms it is the nat; one nat is $1/\log 2 \approx 1.44$ bits. This book uses nats unless a source reports bits. Entropy is zero for a source that always emits the same symbol and largest for one that emits every symbol equally often.

The correspondence also runs the other way. Arithmetic coding transforms a sequence model into a compressor, and a compressor can be turned into a predictor by reading its code lengths as probabilities (Delétang et al., 2024). Researchers at Google DeepMind used this to run language models as general-purpose compressors, in a 2024 paper at ICLR. Chinchilla 70B, the developer's own model trained on text, compressed patches of ImageNet images to 43.4 percent of their raw size and LibriSpeech audio to 16.4 percent, against 58.5 percent for PNG and 30.3 percent for FLAC (Delétang et al., 2024). Those rates do not count the size of the model's parameters, which a decoder would also need (Delétang et al., 2024).

6.2.2 Cross-entropy and Gibbs' inequality #

Now suppose the symbols come from $p$ but the code was built from a model $q$. Symbol $x$ costs $-\log q(x)$, and the expected cost under the true source is the cross-entropy

The excess over the best possible code is the KL divergence,

which follows by writing $-\log q(x) = -\log p(x) + \log\big(p(x)/q(x)\big)$ inside the sum. The excess is never negative.

The inequality has a consequence for honest reporting. Suppose a forecaster believes the outcome is distributed as $p$ and is paid the logarithm of the probability it stated for the outcome that happened. If it states $q$, its expected payment is $-H(p, q)$, which by Gibbs' inequality is largest at $q = p$. A rule for paying a forecaster under which stating one's true belief maximizes the expected payment is called a proper scoring rule, and the log score is one. Section 6.4 builds on this.

Figure 6.2 lets the reader distort the model and watch the cost. Starving one symbol is expensive: multiplying D's probability by 0.001 raises the expected code length from 1.75 to 2.80 bits per symbol, a KL divergence of 1.05 bits. The two directions of temperature are not equally costly either. A model that is too sure (temperature 0.5) pays 0.21 bits per symbol, while one that is too unsure by the same factor (temperature 2) pays 0.06. The excess $\log\big(p(x)/q(x)\big)$ is weighted by the true probabilities, and an overconfident model gives too little probability to outcomes that do happen.

6.2.3 The loss of a language model #

The true distribution of text is never available, only samples from it. Given a corpus of tokens $x_1, \dots, x_N$, each in its context, the average $-\frac1N \sum_i \log q(x_i)$ estimates the cross-entropy $H(p, q)$, and minimizing it over models $q$ is the same as maximizing the likelihood of the corpus. The loss reported for a language model is this average, per token, in nats or bits. Its exponential in the same base ($e$ for nats, 2 for bits) is the perplexity, the number of equally likely choices that would leave a predictor as uncertain as the model is on average. By Equation (6.4), the expected loss, the cross-entropy that a finite corpus estimates, can never fall below the entropy of the text itself, and everything above that floor is the model's KL divergence from the source.

Scaling laws are fitted to this quantity. A 2020 preprint by OpenAI researchers found that language-model loss scales as a power law with model size, dataset size, and training compute (Kaplan et al., 2020), the regularity Section 8.5 teaches. A second 2020 preprint by OpenAI researchers fitted a power law plus a constant. It read the constant as an estimate of the entropy of the data and the power-law term as an estimate of the model's KL divergence from it, a reading that assumes an infinitely large model could match the data's distribution exactly (Henighan et al., 2020).

6.2.4 What an average loss hides #

An average weights each token by how often it occurs. Suppose one context in 100,000 calls for a particular continuation, such as a refusal of a dangerous request, and the true probability of that continuation there is 0.5. A model that gives it probability $10^{-4}$ instead, and the rest to the alternative, pays about $\log(0.5 / 10^{-4}) \approx 8.5$ extra nats each time the continuation occurs and saves about $\log 2 \approx 0.69$ each time it does not, an expected 3.9 extra nats in each such context. Spread over all tokens, that is $3.9 \times 10^{-5}$ nats per token, a change in the fifth decimal place of the loss. A model can therefore have excellent loss and wrong rare behavior, and a comparison of average losses will barely show it. This is the first of several places in the chapter where an average says little about a rare event, and the last three sections are about measuring rare events directly.

The same accounting makes the pretraining distribution the reference point for everything that follows. Pretraining fits the model's cross-entropy to a large corpus, and later training moves the model some distance from there, which the next section learns to measure. A 2025 paper at ACL argues from compression theory that further fine-tuning undermines alignment far more than it undermines pretraining, and conjectures that this elasticity, a tendency to revert to the behavior distribution formed in pretraining, makes language models resist alignment (Ji et al., 2025). Chapter 26 weighs that argument.

6.3 KL divergence and its asymmetry #

Fine-tuning changes a pretrained model, and nearly every later chapter needs to say by how much. Comparing parameters does not answer the question: swapping two units in the same layer of a network changes its weights and leaves its function untouched. What matters is how differently the two models behave, which means comparing two distributions over outputs. The KL divergence of Equation (6.4) does this. This section teaches its properties, the two ways it can be used to fit one distribution to another, and the convention by which the book measures optimization pressure with it.

6.3.1 Expected extra surprise #

Written as an expectation,

the divergence is the average, over outcomes drawn from $p$, of how much more surprised an observer expecting $q$ is than one expecting $p$. In the language of the previous section, it is the expected number of extra nats spent by a code built for the wrong distribution. It is named for Kullback and Leibler, whose 1951 paper "On Information and Sufficiency" is the source of the name (Kullback and Leibler, 1951), and it is also called the relative entropy.

6.3.2 Four properties #

The first property is Gibbs' inequality: $\KL(p \,\Vert\, q) \ge 0$, with equality only when $p = q$. A divergence of zero means the two distributions agree on every outcome.

The second is that the divergence is infinite whenever $p$ gives positive probability to an outcome to which $q$ gives none. The term $p(x) \log\big(p(x)/0\big)$ is unbounded: an observer certain that something cannot happen is infinitely surprised when it does.

The third is that the divergence is not symmetric. For two coins, or any event that happens with probability $a$ under one distribution and $b$ under the other, the divergence has a closed form,

where $\mathrm{Bern}(a)$, the Bernoulli distribution, gives the event probability $a$ and its absence $1 - a$. With $a = 0.5$ and $b = 0.01$ the divergence is 1.61 nats; with the arguments swapped it is 0.64 nats. Because the order matters, the divergence is not a distance in the mathematical sense, which is why it is called a divergence. $\KL(p \,\Vert\, q)$ is read "the KL divergence of $p$ from $q$", and the first argument is the distribution the expectation is taken under.

The fourth is that the divergence between two distributions over sequences adds up along the sequence. Take the logarithm of the ratio of two factorizations by Equation (6.1): the log-ratio of whole sequences is the sum of the per-token log-ratios. Taking expectations under $p$,

where $x_{1:\ell}$ is the whole sequence. The divergence between two language models over complete responses is the sum of their next-token divergences along responses drawn from the first. This is why a KL penalty on whole responses can be charged token by token, as the training pipeline of Section 12.2.1 does.

6.3.3 Two directions of fitting #

The asymmetry matters most when a simple family of distributions has to stand in for a complicated one, and the family cannot match the target exactly. Minimizing the two directions then gives different answers.

Minimizing $\KL(p \,\Vert\, q)$ over $q$, with $p$ the target, is called the forward direction. By Equation (6.4) it is the same as minimizing the cross-entropy $H(p, q)$, since $H(p)$ does not depend on $q$, and on samples from $p$ it is maximum likelihood. Every outcome where $p$ has mass and $q$ has little is charged heavily, so the fit spreads to cover all of $p$'s mass, even if that means putting mass where $p$ has little. The forward direction is mass-covering.

Minimizing $\KL(q \,\Vert\, p)$ over $q$ is the reverse direction. Now the expectation is under $q$, so $q$ pays for every place it puts mass where $p$ has little, and pays nothing for the parts of $p$ it ignores. The fit concentrates on one region where $p$ is high. The reverse direction is mode-seeking, a mode being a peak of a distribution.

Figure 6.3 fits one normal distribution to a target with two peaks four units apart. The forward fit has mean $-0.40$ and standard deviation 2.05, and it puts 30 percent of its probability in the valley between the peaks, where the target has 2 percent. The reverse fit has mean $-2.00$ and standard deviation 0.60: it reproduces the heavier peak and gives essentially no probability to the region around the other, which holds 38 percent of the target. Each fit is poor by the other's measure. The forward fit's reverse divergence is 1.28 nats, and the reverse fit's forward divergence is 8.11 nats, because the reverse fit gives almost no probability to outcomes the target produces often.

The distinction recurs. Maximum likelihood, and therefore pretraining, is the forward direction, so pretraining charges a model for every kind of text in its corpus to which it gives too little probability. The KL-regularized objective of reinforcement learning turns out to be a reverse-direction fit to an ideal policy, so a trained policy that cannot represent the ideal exactly tends to concentrate on part of it (Section 12.4.1).

6.3.4 The book's measure of optimization pressure #

Training moves a policy $\pi$ away from the reference policy $\pi_0$ it started from: for a language model, the distribution over responses before and after fine-tuning. The book measures how far it has moved by

the expected log-ratio over responses the trained policy actually writes. This direction has two practical virtues. It can be estimated during training from the policy's own samples, by averaging the difference of two log-probabilities that both models can compute. And it is infinite if the policy writes anything the reference would never write, so a finite value certifies that the policy has stayed within what the reference could produce.

The objective that trades reward against this divergence, expected reward minus $\beta\,\KL(\pi \,\Vert\, \pi_0)$ for a weight $\beta > 0$, is the center of post-training, and Section 12.3 derives its optimum. The divergence also serves as the horizontal axis of the book's pictures of overoptimization. A study of reward models by OpenAI researchers measured the score of a "gold-standard" reward model, standing in for the true objective, against $d = \sqrt{\KL}$, which it chose because KL is a quadratic metric of distance (Gao et al., 2022), and Section 15.6 teaches the curves.

6.3.5 What a KL budget does not bound #

The divergence measures how much a policy has moved, and its limits come from what that average leaves out.

First, it is infinite outside the reference's support, the set of outputs the reference gives positive probability, so the reference policy must give every output some probability. A softmax, which turns a model's scores into probabilities by exponentiating them and dividing by their sum, gives every token positive probability, so for language models the constraint is weak in principle, though a response with reference probability $10^{-30}$ will never be sampled from the reference.

Second, a small divergence does not bound a large change in rare behavior. The divergence charges each outcome its new probability times its log-ratio, so an outcome that stays rare costs little however much its probability grows. By Equation (6.6), raising the probability of a behavior from one in a million to one in a thousand costs less than 0.01 nats (Exercise 6.2). A KL budget is a guarantee about typical behavior and almost no guarantee about the tail, which is where the behaviors the later chapters worry about live. Section 12.3.3 computes what the penalty charges to make a rare response common.

Third, the divergence measures distance, not direction. A penalty on it does not distinguish movement toward better responses from movement toward responses that only score better. Whether a KL penalty protects the objective the reward stands for depends on how the reward's errors are distributed, which Section 15.5 takes up.

6.4 Calibration and proper scoring rules #

A model that says it is 90 percent sure is useful only if it is right about nine times in ten when it says so. Then its stated probabilities can be acted on: a reviewer can check the answers it marks as uncertain, and a decision rule can weigh them against the cost of being wrong. This section defines that property, how it is scored, and what the evidence says about neural networks and language models.

6.4.1 Reliability diagrams and calibration error #

The idea is to take a forecaster at its word and check. Group its predictions by the probability it stated, and within each group compare the stated probability with how often the event occurred. Plotted against each other, the pairs form a reliability diagram, and a forecaster whose points lie on the diagonal is calibrated: among all the cases where it states a probability of 0.7, the event occurs 70 percent of the time, and likewise for every probability it states.

The expected calibration error (ECE) summarizes the diagram in one number. Divide the stated probabilities into bins, and in each bin take the absolute gap between the average stated probability and the observed frequency. The ECE is the average of these gaps, weighted by the share of cases in each bin (Guo et al., 2017). The number depends on the binning: one study cited below uses 15 bins of equal width (Guo et al., 2017), another 10 bins holding equal numbers of predictions (Kadavath et al., 2022), so their numbers cannot be set side by side.

6.4.2 Calibration is not accuracy #

Calibration is easy to achieve by giving up usefulness. A weather forecaster who announces a 30 percent chance of rain every day, in a city where it rains on 30 percent of days, is perfectly calibrated and tells no one anything. What calibration leaves out is sharpness: how far the forecasts depart from the base rate, toward probabilities near 0 and 1. A good forecaster is calibrated and sharp, and a single score has to reward both.

The Brier score is the mean squared difference between the stated probability and the outcome, counted as 1 if the event occurred and 0 if not; it takes its name from Glenn Brier, whose 1950 paper concerned the verification of forecasts expressed as probabilities (Brier, 1950). As Lin, Hilton, and Evans put it, a model can be perfectly calibrated and not have a squared error of zero, because the squared error combines calibration error with sharpness (Lin et al., 2022). The log score, the average negative log-probability given to what happened, is the cross-entropy of Section 6.2. Both are proper scoring rules, and the derivation for the Brier score takes three lines.

A binned calibration error is not proper, as Lin and colleagues note of theirs (Lin et al., 2022): a forecaster can lower it without becoming more informative, as the always-30-percent forecaster shows. The general theory of which rules are proper, and why, is set out in Gneiting and Raftery's "Strictly Proper Scoring Rules, Prediction, and Estimation" (Gneiting and Raftery, 2007).

6.4.3 Temperature scaling #

A classifier's outputs are probabilities computed from logits, unnormalized scores turned into probabilities by the logistic function or, for many classes, the softmax of Section 6.3.5. If the classifier is overconfident, a simple repair divides every logit by a single number $T > 1$, the temperature, before converting, which pulls every stated probability toward the middle. The temperature is fitted on held-out data, and since dividing by a positive number never changes which logit is largest, the repair never changes a prediction, only the confidence attached to it (Guo et al., 2017). The same $T$ is the sampling temperature of a language model (Section 11.1).

Figure 6.4 runs the repair on a synthetic classifier that is overconfident by a known factor. Its expected calibration error falls from 0.102 at $T = 1$ to zero at $T = 2$, and its accuracy stays at 73.4 percent. Its Brier score improves only from 0.189 to 0.177, because most of the score measures uncertainty no rescaling can remove: a calibrated forecast of 0.6 is still wrong four times in ten.

6.4.4 What neural networks do #

In a 2017 paper at ICML, Guo, Pleiss, Sun, and Weinberger reported that modern neural networks, unlike those from a decade earlier, are poorly calibrated (Guo et al., 2017). Table 6.2 gives their measurements on one image classification task. Across their datasets the uncalibrated error was typically between 4 and 10 percent (Guo et al., 2017). Temperature scaling gave the lowest error of the repairs they tested in 10 of their 15 vision comparisons and performed comparably on the text tasks (Guo et al., 2017).

The authors found increased model capacity and a lack of regularization closely related to miscalibration, while saying that they cannot claim causality (Guo et al., 2017). Their mechanism comes from the log score: once a network classifies almost all of its training examples correctly, it can lower its training loss further only by becoming more confident (Guo et al., 2017). Every measurement in the table is on test data drawn like the training data, with the temperature fitted on held-out data of the same kind. Averaging several independently trained networks is another route to better uncertainty estimates (Lakshminarayanan et al., 2016), which Section 8.4 takes up.

6.4.5 What language models do #

A 2022 preprint by Anthropic researchers studied the probabilities of Anthropic's own pretrained models, from 800 million to 52 billion parameters (Kadavath et al., 2022). They report that larger models are well calibrated on diverse multiple choice and true/false questions when the questions are provided in the right format, with visible lettered answer options. Calibration improved with model size and with a few worked examples in the prompt (Kadavath et al., 2022). Replacing one option with "none of the above" reduced both accuracy and calibration significantly (Kadavath et al., 2022).

The same report tested models fine-tuned by reinforcement learning from human feedback, training on people's preferences between responses (Section 12.2.1). Such models naively appear very miscalibrated, and a single temperature of 2.5 largely fixed calibration on three evaluations; the authors caution that more intensive reinforcement learning might distort calibration in ways a temperature cannot repair (Kadavath et al., 2022). The developer of GPT-4 reports the same direction for its own model: the pretrained model was highly calibrated on a subset of a multiple-choice benchmark, and post-training, the fine-tuning that turns a pretrained model into an assistant, hurt calibration significantly (OpenAI et al., 2023).

Anthropic's models were also trained to predict whether they would answer a question correctly, a quantity the authors call P(IK), for "I know". The models predicted it well and partly generalized across tasks, but on new tasks the calibration of P(IK) was often quite poor, and the models were mostly underconfident (Kadavath et al., 2022). A model can also state a confidence in words.

Lin, Hilton, and Evans, at the University of Oxford and OpenAI, fine-tuned GPT-3 to state its confidence in its own answers to arithmetic questions in words, in a 2022 paper in Transactions on Machine Learning Research. The stated confidences mapped to well-calibrated probabilities, and the model remained moderately calibrated under distribution shift (Lin et al., 2022); the authors note that while the label distribution shifted, the content and format of the questions did not shift much (Lin et al., 2022). Without such training, confidence stated in words tends to be overconfident. A study of five models, among them GPT-4 and LLaMA 2 Chat, first posted in 2023 and published at ICLR 2024, found verbalized confidences mostly between 80 and 100 percent, often in multiples of 5, as people tend to state them (Xiong et al., 2023). The same study supplies a real case of a low calibration error that carries no information. With step-by-step reasoning on grade-school arithmetic problems, GPT-4 reached 93.6 percent accuracy and a calibration error of 0.064, which the authors call near-optimal, by stating 100 percent confidence for every answer, which leaves no way to tell its right answers from its wrong ones (Xiong et al., 2023). A survey at NAACL collects the methods for estimating and calibrating language-model confidence (Geng et al., 2023).

How grading shapes what a model says when it is unsure is the subject of a 2025 preprint by researchers at OpenAI and Georgia Tech. It argues that language models hallucinate because training and evaluation procedures reward guessing over acknowledging uncertainty (Kalai et al., 2025), and Section 39.2 works through the argument with this section's scoring rules. Human forecasters have their own calibration literature, reviewed up to 1980 by Lichtenstein, Fischhoff, and Phillips (Lichtenstein et al., 1982), and Section 4.5 weighs the record of expert forecasting and what it implies for estimates of AI risk.

6.4.6 Where measured calibration stops #

Calibration can be scored only on questions that resolve, and only on questions like the ones scored. A model's calibration on benchmark questions says little about its calibration on the questions alignment cares about most: whether it will behave well in a situation it has not met, what it would do with more autonomy, or questions that never resolve at all (inference). The studies above found calibration that depends on format, on training stage, and on whether the task is new, which is evidence that it does not transfer for free.

An average over a dataset can also hide miscalibration in a subgroup or after a shift in inputs, since bins pool cases of every kind. And calibration is compatible with dishonesty, since a model could be calibrated by imitating a calibrated person (Lin et al., 2022). Chapter 39 takes honesty to include calibration and more, and Chapter 4 applies the scoring rules to estimates of risk.

6.5 Bounding a rate you have never observed #

A laboratory runs its model through $n$ agentic episodes, in which the model acts on its own over many steps with tools, and sees no attempt at sabotage. What has it learned about how often the model would attempt sabotage? Not that the rate is zero: a rate of one in a million would almost always produce a clean record in a few thousand episodes. The absence of events does bound the rate, but weakly, and only under an assumption about the episodes that the strategic setting of later chapters makes hard to meet.

6.5.1 The rule of three #

Suppose each episode misbehaves with the same probability $p$, independently of the others. The chance of a clean record in $n$ episodes is $(1 - p)^n$. A high rate would make a clean record improbable, so the clean record is evidence against a high rate. The upper confidence bound $p_{\text{upper}}$ is the largest rate under which a clean record would still have probability at least $\delta$, a small number the analyst chooses. Used on every clean record, the procedure states a bound below the true rate at most a fraction $\delta$ of the time, which is what "confidence $1 - \delta$" means.

The approximation $3/n$ is widely called the rule of three, and a 1983 paper in JAMA by Hanley carries in its title the question the rule answers: "If Nothing Goes Wrong, Is Everything All Right?" (Hanley, 1983). Nothing going wrong in $n$ trials is consistent with a rate as high as $3/n$.

Table 6.3 makes the scaling plain. Each factor of ten in the bound costs a factor of ten in clean episodes, so a claim that a behavior occurs less than once in a million episodes needs about three million clean ones.

6.5.2 Deployment arithmetic #

Deployment multiplies whatever rate is left. With a per-query rate $p$ and $N$ queries, the expected number of events is $pN$, and the chance of at least one is $1 - (1 - p)^N \approx 1 - e^{-pN}$. A behavior that occurs once in a million queries, in a system serving a hundred million queries a day, occurs about a hundred times a day. A test campaign of 100,000 queries would have seen it at least once with probability $1 - e^{-0.1} \approx 0.095$, so nine campaigns in ten would report a clean record. A rate too small to see in testing can be routine in use.

6.5.3 What the bound assumes #

The derivation assumed that the test episodes are independent draws from the same distribution as the episodes the claim is about. Two things break this. Episodes that share a task family or a prompt template are not independent, and a record of $n$ correlated episodes carries less evidence than $n$ independent ones. More fundamentally, the bound is about the population the tests were drawn from. A system that behaves differently when it can tell it is being tested makes the test episodes a different population from deployed ones, and a clean test record then bounds the rate on tests and says nothing about the rate in deployment. Chapter 37 takes up whether current models can tell.

6.5.4 From statistics: error bars for evaluations #

From statistics. The rule of three is the zero-event corner of a larger subject, the statistics of measurement from samples. In a 2024 preprint, Evan Miller of Anthropic argues that evaluations are experiments, and that the literature on evaluations has largely ignored the literature from other sciences on experiment analysis and planning (Miller, 2024). His framing treats the questions of a benchmark as drawn at random from a hypothetical, infinite, unseen super-population of questions, so that a score estimates the skill the benchmark is meant to measure (Miller, 2024). For a benchmark scored 0 or 1 per question, with mean score $\hat p$ over $n$ questions, the central limit theorem gives a standard error of $\sqrt{\hat p (1 - \hat p)/n}$ (Miller, 2024), and a 95 percent interval of about two standard errors either side.

The uncertainty is large at ordinary sizes. A model that scores 80 percent on 1,000 questions has a standard error of $\sqrt{0.8 \times 0.2 / 1000} \approx 0.013$, so its interval spans about 2.5 points either side, and two such intervals alone cannot establish a 2-point difference between models. Miller's further recommendations, standard errors that account for questions sharing a source, comparisons that pair the two models question by question, and power analysis to choose the number of questions before running an evaluation (Miller, 2024), are applied in Section 36.7.

What the field implies here is that every benchmark number is a measurement with an uncertainty, and that the rule of three is what remains of the standard error when the score is zero. The formula above gives a standard error of zero at $\hat p = 0$, a claim of certainty no clean record supports; the zero-event bound replaces it. Where the analogy stops: the super-population is the population of benchmark questions, not of deployment inputs, so the error bar covers resampling of questions like those in the benchmark and says nothing about the gap between the benchmark and use (inference).

6.5.5 From fault-tolerant computing: what testing cannot establish #

From fault-tolerant computing. Software engineering met the limits of the rule of three when it was asked to certify systems whose failure would kill people. Littlewood and Strigini's 1993 paper on the validation of ultrahigh dependability for software-based systems named the problem in its title (Littlewood and Strigini, 1993). In the same year, the abstract of a paper by Butler and Finelli stated that quantifying the reliability of life-critical software is infeasible using statistical methods, and that classical methods lead to exorbitant amounts of testing. It added that reliability growth models, which extrapolate from the falling failure rate seen while bugs are fixed, cannot overcome the need for excessive testing (Butler and Finelli, 1993).

Kalra and Paddock, at RAND, worked the same arithmetic for autonomous vehicles in 2016. The benchmark was the 2013 US rate of 1.09 fatalities per 100 million vehicle miles, and the method was the binomial zero-failure bound (Kalra and Paddock, 2016). Table 6.4 gives three of their figures.

The first row is the rule of three: $3 / (1.09 \times 10^{-8}) \approx 275$ million. The authors call their results optimistic, since they treat the human rate as a known constant, and they conclude that test-driving alone cannot provide sufficient evidence for the safety of autonomous vehicles with respect to fatalities and injuries (Kalra and Paddock, 2016).

What the field implies for alignment is that a claim of a very low misbehavior rate cannot rest on testing alone. It must rest on arguments about the system's design and limits, with tests as one kind of evidence among several, which is the job of a safety case (Chapter 52). Where the analogy stops: the failures these papers count are accidental, not strategic. An accidental fault does not occur less often because the program is being tested, while a model that conditions its behavior on being observed breaks the representativeness assumption, as only software written to detect tests has (Section 37.7). For such a model the testing problem is worse than the one these papers describe (inference).

6.6 Estimating rates too small to sample #

A bound says how large a rate might be. Often an estimate is wanted instead: not "below one in a thousand" but "about three in a hundred thousand". A rate far below $1/n$ cannot be estimated by drawing $n$ samples, because the samples almost never contain the event, and an estimate of zero is wrong by the whole of the quantity. The idea that rescues the estimate is to steer sampling toward the event and then correct for the steering.

6.6.1 Importance sampling #

Let $p$ be the distribution of inputs and $\mathcal{E}$ the set of inputs that produce the event. The indicator $f(x) = \mathbb{1}[x \in \mathcal{E}]$ is 1 when $x$ produces the event and 0 otherwise, so the rate is $\Prob(\mathcal{E}) = \E_{x \sim p}[f(x)]$. For any other distribution $q$, the proposal, that is positive wherever $p(x) f(x)$ is,

So draw $x_1, \dots, x_n$ from $q$ and average $f(x_i)\,\omega(x_i)$. Each sample that lands in the event is counted with the importance weight $\omega$, which undoes the steering: an input the proposal made a thousand times more likely than it really is counts a thousandth as much. The estimate $\hat p$ is unbiased for any valid proposal. The proposal decides its variance.

Figure 6.5 runs the estimator on a normal tail. With plain sampling ($c = 0$), the relative standard error of a 1,000-sample estimate is 5.6, and a run sees the event at all with probability 3.1 percent. With the proposal centered on the boundary of the event ($c = 4$), the relative standard error is 0.067. Overshooting to $c = 7$ raises it to 0.60: every proposal sample lands in the event, but the samples near the boundary, where most of the event's probability lies, have become rare and carry huge weights.

6.6.2 Splitting #

The second family of methods splits a rare event into a chain of less rare ones. If a score $g(x)$ exceeds 0 exactly on the event, choose levels $\tau_1 < \tau_2 < \dots < \tau_K = 0$ and write

a product of conditional probabilities, each large enough to estimate by sampling, with $\tau_0 = -\infty$ and $X$ a random input. Samples that pass one level are copied and perturbed to explore the next. Adaptive multilevel splitting chooses each level from the current samples, the method of Cérou and Guyader's 2007 paper "Adaptive Multilevel Splitting for Rare Event Analysis" (Cérou and Guyader, 2007).

Webb, Rainforth, Teh, and Kumar applied it to neural networks in a 2019 paper at ICLR. They estimate the probability that a property is violated under an input model, in place of the yes-or-no answer of formal verification, a proof that the property holds for every input, which for a property that does fail says only that it is not verifiable (Webb et al., 2019). On 500 properties of a small network, naive Monte Carlo sampling with $10^{10}$ samples missed counterexamples for 8 of the rarest violated properties, which their method found (Webb et al., 2019). The authors note the limit themselves: against an effective adversary, the counterexamples may lie in a set too improbable for the method to resolve, or of probability zero under the input distribution (Webb et al., 2019).

6.6.3 Rare outputs of language models #

Gabriel Wu and Jacob Hilton, of the Alignment Research Center, posed the problem for language models in a 2025 paper at ICLR. The task is to estimate the probability that a model produces a given output on inputs from a formally specified distribution, when that probability is too small to estimate by sampling (Wu and Hilton, 2025). They used small transformers, the network architecture of language models (Section 11.2), of one, two, and four layers, single-token outputs, and target probabilities between $10^{-9}$ and $10^{-5}$, with ground truth from $2^{32}$ samples and a budget of $2^{16}$ model runs per method (Wu and Hilton, 2025). Importance sampling, which searches for inputs giving rise to the rare output, outperformed activation extrapolation, which fits a probability distribution to the model's logits, and both outperformed naive sampling (Wu and Hilton, 2025). Their proposal distribution has the form $p(x)\exp\big(g(x)/T\big)$ for a score $g$, the same exponential reweighting that Section 12.3 derives as the optimum of KL-regularized training. They list the limits of the setting: inputs whose tokens are drawn independently and outputs of a single token, while the behaviors of concern involve long chains of generation or interaction with the world (Wu and Hilton, 2025).

6.6.4 Extrapolating the tail #

A third approach measures what can be measured and extrapolates. Extreme value theory, the statistics of maxima and the subject of Coles's An Introduction to Statistical Modeling of Extreme Values (Coles, 2001), shows that under general conditions the largest of many draws follows one of a few limiting families of distributions (Jones et al., 2025). The shape of a tail observed in moderate samples can then be extended beyond them. A 2025 preprint by Anthropic researchers, with collaborators at Mila and UC Berkeley, applied the idea to Anthropic's own models (Jones et al., 2025). For each query they measure an elicitation probability, the chance that a sampled response to the query shows a behavior. They report that the largest observed elicitation probabilities scale predictably with the number of queries, across up to three orders of magnitude of query volume (Jones et al., 2025).

Their main test forecast the worst-query risk, the largest elicitation probability among $n$ queries, for deployments of 10,000 to 90,000 queries from evaluation sets of 100 to 1,000, on Claude 3.5 Sonnet, Claude 3 Haiku, and their two base models, the pretrained models before post-training (Jones et al., 2025). The behavior in this test was the probability of one specific harmful output in reply to queries a model wrote to elicit it, a proxy the authors note may not reflect the actual likelihood of producing useful harmful instructions (Jones et al., 2025). Table 6.5 compares their method, which fits only the tail, with a Gumbel distribution (one of the limiting families), against a baseline that fits a log-normal distribution to the negative log elicitation probabilities of all the evaluation queries and so extrapolates from average behavior (Jones et al., 2025).

The authors list their limits: the simulated deployments are at most three orders of magnitude larger than the evaluation and could in principle be evaluation sets themselves; they do not study distribution shift; and adversaries in practice may adapt their queries (Jones et al., 2025).

6.6.5 What these methods assume #

All three methods treat the behavior as a fixed property of a model on a formally specified input distribution. That is what makes them work, and an estimate covers deployment only if that distribution covers deployment's inputs, including the cues a strategic system would use to tell a test from deployment. Wu and Hilton give, as one motivation, a model that behaves well only when it judges itself to be in training, and suggest crafting an input distribution "wide enough" to give some probability to any input deployment could bring (Wu and Hilton, 2025); none of the three studies measures such a model. Whether behavior that depends on being observed follows the same tail as accidental failure is open (inference). Chapter 48 applies these methods to attacks on safeguards, where the inputs are chosen by an adversary rather than drawn from a distribution.

6.7 Detection: what an alarm is worth #

Every monitoring and control claim in the later parts of the book is a detection claim: a classifier, a probe, or another model reads what an AI system does and raises a flag. This section asks how much a flag should be trusted, which brings together the Bayes' rule of Section 6.1 and the low base rates of the last two sections.

6.7.1 Four outcomes and three rates #

Of the episodes a monitor sees, some are bad and some are flagged, and the four combinations have names. A true positive is a flagged bad episode, and a false negative or miss a bad episode not flagged. A false positive or false alarm is a flagged good episode, and a true negative a good episode not flagged. Three rates summarize them. The recall, or true positive rate, is the share of bad episodes flagged, written $1 - \beta_{\text{miss}}$, with $\beta_{\text{miss}}$ the miss rate. The false positive rate $\alpha$ is the share of good episodes flagged. The precision is the share of flags that are real.

Recall and the false positive rate are properties of the monitor, measurable on labeled tests. Precision is not: it also depends on the base rate $\pi_{\text{base}}$, the share of episodes that are bad. By Bayes' rule (Equation (6.2)),

Half of all flags are real when the two terms in the denominator are equal, that is, when the false positive rate falls to about the recall times the base rate. At a base rate of one in ten thousand, a monitor needs a false positive rate near one in ten thousand before a flag is as likely to be real as not.

6.7.2 Operating points and the ROC curve #

Most monitors output a score, and the designer chooses a threshold. A low threshold flags more bad episodes and more good ones; a high threshold flags fewer of each. Each threshold is an operating point, and the curve of recall against false positive rate traced by all of them is the receiver operating characteristic (ROC) curve. The curve, not any single point on it, is what a monitor offers; Fawcett's 2006 "An introduction to ROC analysis" (Fawcett, 2006) introduces the vocabulary.

6.7.3 Precision at a low base rate #

Equation (6.8) explains a lesson computer security learned from intrusion detection. Stefan Axelsson's 2000 paper states in its abstract that, for a reasonable set of assumptions, the false alarm rate is the limiting factor for the performance of an intrusion detection system. The cause is the base-rate fallacy: alarms become trustworthy only at a perhaps, in some cases, unattainably low false alarm rate (Axelsson, 2000).

Figure 6.6 puts numbers on the lesson. At a base rate of one in ten thousand and a million episodes a day, the monitor with 95 percent recall and a 1 percent false positive rate raises 10,094 alerts a day, of which 95 are real, and misses 5 bad episodes. Tightening its threshold to a false positive rate of 0.01 percent cuts the alerts to 160, of which 60 are real, a precision of 37 percent, at the price of 40 misses. No threshold gives both: a better trade needs a monitor with a larger separation, which moves the whole curve.

6.7.4 Two monitors that fail together #

A natural response to misses is a second monitor, with an episode escalated if either flags it. Then a bad episode escapes only if both miss. If the two monitors' misses were independent, the pair's miss rate would be the product of their miss rates. They rarely are.

In Figure 6.6, adding a second monitor at $\rho = 0.5$ reduces the daily misses from 5 to 2.6, against 0.25 if the errors were independent. Taking the same correlation for the two monitors' false alarms on good episodes, the alerts rise to 15,046 and the precision falls to 0.65 percent.

From fault-tolerant computing. Software fault tolerance built this defense deliberately, from separately programmed versions of one program. The abstract of Butler and Finelli's paper calls the key assumption of software fault tolerance, that separately programmed versions fail independently, problematic: it cannot be justified by experimentation in the ultrareliability region, and the multiversion software experiments of the time support the doubt (Butler and Finelli, 1993). Knight and Leveson's 1986 "An experimental evaluation of the assumption of independence in multiversion programming" (Knight and Leveson, 1986) and Eckhardt and Lee's 1985 "A Theoretical Basis for the Analysis of Multiversion Software Subject to Coincident Errors" (Eckhardt and Lee, 1985) are taken up in Section 50.4, which derives where such correlation comes from. For language models, a related quantity has been measured: not how often two models err on the same input, but how often, when both err, they give the same wrong answer. A 2025 preprint by Kim, Garg, Peng, and Garg evaluated over 350 models and found substantial correlation in their errors: on one leaderboard dataset, two models that both erred gave the same answer 60 percent of the time, and larger, more accurate models had highly correlated errors even across architectures and providers (Kim et al., 2025). What the field implies here is that monitors built from similar models should be expected to share blind spots, and that a claim about stacked monitors must measure their joint failures rather than multiply their separate rates (inference). Where the analogy stops: independently written programs fail together by accident, on inputs that happen to be hard, while an adversary can search for the inputs on which every monitor is weak. That case is Chapter 50's, and Section 50.4 applies the arithmetic to it.

6.7.5 The person after the monitor #

The flags go to a reviewer, who has a detection problem of their own. In a 2005 laboratory study of visual search, observers missed rare targets far more often than common ones (Wolfe et al., 2005). The prevalence the reviewer sees is the precision of the monitor's flags, which at a low base rate is low, so the human stage is weakest where it matters most (inference). Section 51.2 reports the rates and applies them to the audit step of a control protocol.

6.7.6 What goes wrong in practice #

Three quantities in Equation (6.8) are harder to know than they look. The base rate of the behavior a monitor looks for is rarely known, and is often the very thing the monitoring is meant to find out. Recall is measured on planted examples, which may not resemble the real ones. And a monitored system that learns what is monitored can move its behavior to where the monitor is weak, so a recall measured before deployment need not hold after. Later chapters apply this section's arithmetic to the precision of monitors that read a model's reasoning (Section 40.2), to control protocols with a budget for human audits (Chapter 50), to incident detection in deployment (Chapter 54), and to the design of evaluations (Chapter 36).

6.8 Exercises #

Further reading #

• Shannon (1948): A Mathematical Theory of Communication, read here only as a catalog record; Section 6.2 teaches entropy from later sources.
• Kullback and Leibler (1951): the 1951 paper on information and sufficiency that gave the divergence its name.
• Brier (1950): the 1950 paper on verifying probability forecasts from which the Brier score takes its name.
• Kadavath et al. (2022): a developer's study of what its language models' probabilities say about what they know.
• Hanley (1983): the question its title asks, which the rule of three answers.
• Littlewood and Strigini (1993): on validating ultrahigh dependability for software, the limit every safety case faces.