Draft complete but not yet reviewed; expect revisions

Notation

The tables below list the symbols that recur across chapters, each with the meaning it keeps and the section that defines it. A symbol used in a single derivation or exercise (the $p$ of a coin, a function $f$ ) is defined where it is used. A few such local definitions borrow a listed letter, such as $T$ for a count of intermediate steps or $d$ for the depth of a leaf in a decision tree; the local definition then holds only in that passage, which states it. The letter $z$ has two book-wide meanings, since it is the usual letter both for a latent variable and for a measured score: it is a latent task or persona in the chapters on language models and the personas training shapes, and a measured score in the chapters on evaluation and delegation. No chapter uses it in both senses, and each defines it where it first appears; a single derivation may also use it locally, as for a logit or a data point, and defines it there. The last table spells out the abbreviations the book uses.

B.1 Conventions #

Form	Meaning
$x$ , $n$ , $\beta$	scalars, in italic
$\vx$ , $\vh$	vectors, bold lowercase
$\mW$ , $\mA$	matrices, bold uppercase
$\X$ , $\D$	sets and datasets, calligraphic
$\Prob(\mathcal{E})$ , $\Prob(\mathcal{E} \given \mathcal{E}')$	probability of an event, a set of outcomes, and conditional probability (Section 6.1.1)
$p(x)$ , $q(x)$	probability distributions (mass or density)
$\E[X]$ , $\E_{y \sim \pi}[r(y)]$	expectation, with the distribution as subscript when it matters
$\Var(X)$ , $\Cov(X, Y)$	variance, covariance
$\N(\mu, \sigma^2)$	normal distribution with mean $\mu$ and variance $\sigma^2$
$\KL(p \,\Vert\, q)$	KL divergence of $p$ from $q$ (Section 6.3)
$\chi^2(p \,\Vert\, q)$	chi-squared divergence of $p$ from $q$ , $\sum_x p(x)^2/q(x) - 1$ (Section 15.4.4)
$H(p)$ , $H(p, q)$	entropy of $p$ , and cross-entropy of $q$ under $p$ (Section 6.2.1, Section 6.2.2)
$\log$	natural logarithm; information in nats unless a source reports bits
$\argmax_x f(x)$	the input at which $f$ is largest
$\sigma(z) = 1/(1 + e^{-z})$	the logistic function
$\mathrm{ReLU}(z) = \max(0, z)$	the rectified linear unit (Section 8.2.1)
$\Phi$ , $\phi$	the standard normal distribution function and density
$\Gamma(\cdot)$	the gamma function (Section 4.6.2)
$\mathbb{1}[\cdot]$	the indicator: 1 when the condition holds, 0 otherwise (Section 6.6.1)
$\mathrm{A}$ , $\mathrm{B}$ , $\mathrm{L}_1$	labels of options, outcomes, and lotteries, set upright so they never read as the symbols listed below
$\mA^\T$	transpose

B.2 Language models and training #

Symbol	Meaning	Defined in
$x$	a prompt or context given to a model	Section 11.1
$y$	a response or completion	Section 11.1
$x_t$ , $x_{<t}$	the token at position $t$ , and the tokens before it; $y_t$ , $y_{<t}$ the same within a response	Section 11.1
$\theta$	a model's parameters (weights)	Section 8.1
$f_\theta$	a model: the function from inputs to outputs that the parameters $\theta$ select	Section 8.1
$p_\theta$	a model's predicted distribution, such as its next-token distribution $p_\theta(x_t \given x_{<t})$	Section 8.1, Section 11.1
$\mathcal{F}$	a family of predictors that training chooses among	Section 9.1.2
$\pi_\theta(y \given x)$	the policy: the model's distribution over responses	Section 12.2
$\pi_0$	the reference policy that training starts from and is anchored to	Section 12.3
$\pi^\ast$	the optimum of a stated objective	Section 12.3
$r(x, y)$	a reward: the signal training optimizes	Section 7.5
$r_\phi$	a learned reward model with parameters $\phi$	Section 13.3
$r^\ast$	the intended (gold) reward that $r$ or $r_\phi$ stands for	Section 15.3.1
$\hat r_\theta(y)$	the implicit reward of a policy, $\beta \log\big(\pi_\theta(y \given x)/\pi_0(y \given x)\big)$	Section 12.6
$\beta$	the weight of the KL penalty	Section 12.3
$Z(x)$	the normalizing constant of the KL-regularized optimum	Section 12.3
$d = \sqrt{\KL(\pi \,\Vert\, \pi_0)}$	optimization pressure, as distance from the reference policy	Section 15.6
$n$	a sample size: training examples, test questions, or independent trials; in "best-of- $n$ " the number of samples drawn	Section 6.5, Section 9.2.2, Section 12.5
$\lambda_{\text{lc}}$	the learning coefficient of singular learning theory (the subscript keeps $\lambda$ free for a sparsity weight)	Section 10.2
$T$	sampling temperature	Section 11.1
$\D$	a training or evaluation dataset	Section 8.1
$L(\theta)$	a loss function	Section 8.1
$N$ , $D$ , $C$	a model's parameter count, its training tokens, and its training compute in FLOPs ( $C$ is effective compute in Section 36.2.1 and Section 62.2)	Section 8.5
$\eta$	the learning rate	Section 8.3
$J(\theta)$ , $J(\pi)$	an objective: the quantity training maximizes	Section 12.1.2
$A$	an advantage: a reward minus its expected value under the policy	Section 12.1.3
$k$	the number of responses sampled per prompt in group-based training	Section 12.7.1
$F$	the distribution function of the reward under $\pi_0$	Section 12.5.1
$\kappa = 1/\beta$	the inverse KL weight	Section 12.3.3
$\omega$	a probability ratio used as a weight: the importance weight $p(x)/q(x)$ , and PPO's ratio $\omega(\theta)$ between the new and the old policy	Section 6.6.1, Section 12.1.4
$p_0$	the reference probability of one response	Section 12.3.3
$o_0$	prior odds before training: of a response under the reference, $p_0/(1 - p_0)$ , or of one goal against another	Section 24.4.3, Section 30.2.2
$\Delta$	the reward gap between one response and its alternatives	Section 12.3.3
$z$ , $p(z)$ , $z^\ast$	a latent task a text could come from, its prior probability, and the task that generated the examples (in these chapters only; see the score row under evaluation)	Section 11.3.1, Section 11.6.1
$z_A$ , $z_B$	two personas a model can enact: the Assistant and another character	Section 11.6.3
$\Lambda$	the likelihood ratio one example gives between two personas	Section 11.6.3
$M$	the number of examples in a context	Section 11.3.1
$y_{\mathrm{h}}$	an honest response, honest work	Section 30.2.2, Section 40.5.2
$y_{\mathrm{e}}$	a grader exploit	Section 30.2.2
$y_{\mathrm{o}}$ , $y_{\mathrm{c}}$	an overt and a covert hack; $p_{\mathrm{h}}$ , $p_{\mathrm{o}}$ , $p_{\mathrm{c}}$ the reference probabilities of honest work and the two hacks	Section 40.5.2
$\Delta_{\text{mon}}$	the reward a monitor's flag removes from an episode	Section 40.5.2

B.3 Decisions and reinforcement learning #

Symbol	Meaning	Defined in
$s$ , $a$	a state and an action	Section 7.4
$\mathcal{S}$ , $\mathcal{A}$	the sets of states and actions	Section 7.4
$P(s' \given s, a)$	transition probabilities	Section 7.4
$\gamma$	the discount factor	Section 7.4
$G$	the return: the discounted sum of rewards along a trajectory	Section 7.4
$V(s)$ , $Q(s, a)$	the value of a state, of a state-action pair	Section 7.4
$U$	a utility function, or the utility of an uncertain outcome	Section 7.2

B.4 Preferences and choice #

Symbol	Meaning	Defined in
$y_w \succ y_l$	$y_w$ is preferred to $y_l$	Section 13.1
$y_w$ , $y_l$	the preferred and the dispreferred response of a comparison	Section 13.2
$\Prob(y_w \succ y_l) = \sigma(r(y_w) - r(y_l))$	the Bradley-Terry model	Section 13.2
$\tau$	a chooser's noise scale; its inverse is the rationality coefficient of a Boltzmann model	Section 13.2.2
$\Psi$	a shaping potential	Section 13.4.1

B.5 Rates, tests, and detection #

Symbol	Meaning	Defined in
$\alpha$	the false positive rate of a test or monitor	Section 6.7
$\beta_{\text{miss}}$	the miss rate, so $1 - \beta_{\text{miss}}$ is the detection rate; the subscript keeps $\beta$ free for the KL weight	Section 6.7
$\pi_{\text{base}}$	the base rate of the event a test looks for	Section 6.7
$\hat{p}$	an estimated rate	Section 6.5
$p_{\text{upper}}$	an upper confidence bound on a rate never observed	Section 6.5
$\rho$	a correlation coefficient, including the intraclass correlation of clustered questions	Section 6.1, Section 36.7.1
$d'$	the separation of two normal score distributions in units of their standard deviation, a detector's sensitivity	Section 6.7.2
$R^2$	the share of variance a model explains, as a source reports it	Section 6.1
$m_n$	the expected maximum of $n$ independent standard normal draws	Section 7.7.2
$P_T$ , $P_D$	the distributions of inputs in tests and in deployment	Section 37.2
$\mathrm{TV}(p, q)$	total variation distance: the largest difference two distributions give to one set	Section 37.2

B.6 Evaluation and measurement #

Symbol	Meaning	Defined in
$z(u)$ , $\hat z$ , $z^\ast$	a measured score: under elicitation method $u$ (in Section 48.1, $u$ is an attack); the reported score; the best score under any method. In delegation, $z$ is the principal's noisy measure of the agent's effort. In the chapters on language models and personas, $z$ is a latent task instead (see the introduction)	Section 36.2.1, Section 14.1.1
$\mathcal{U}_{\text{eval}}$ , $\mathcal{U}$	the methods an evaluation tried (in Section 48.1, attacks); the attacks a threat model allows	Section 36.2.1, Section 48.1
$z_{10}$	score points per tenfold increase in training compute	Section 36.2.1
$z_{\text{weak}}$ , $z_{\text{w2s}}$ , $z_{\text{ceil}}$	the scores of a weak supervisor, of a strong student trained on its labels, and of the strong model trained on correct labels	Section 42.1
$\mathrm{PGR}$	performance gap recovered, $(z_{\text{w2s}} - z_{\text{weak}})/(z_{\text{ceil}} - z_{\text{weak}})$	Section 42.1
$g_{\text{el}}$	the elicitation gap, as a compute-equivalent factor	Section 36.2.1, Section 62.2
$h_{50}$	a model's 50 percent time horizon	Section 36.3.1
$t_j$	the time task $j$ takes a human	Section 36.3.1
$\xi$	an unobserved trait that a measurement targets; in an item response model, a respondent's ability	Section 5.2.2, Section 18.2.2, Section 36.4.1
$b_j$ , $\ell_j$	an item's difficulty and discrimination	Section 36.4.1
$I(\xi)$	test information, the sum of the items' $I_j(\xi)$	Section 36.4.1
$n_{\text{c}}$ , $n_{\text{eff}}$	questions per cluster; the effective number of independent questions	Section 36.7.1
$\delta_{\min}$ , $\sigma_d^2$	the smallest difference a comparison should detect; the variance of a per-question difference	Section 36.7.2

B.7 Delegation and games #

Symbol	Meaning	Defined in
$e$	the agent's action or effort, unobserved by the principal	Section 14.1
$w$	the payment or reward the principal sets	Section 14.1
$c(e)$	the agent's cost of an action	Section 14.1
$\theta_A$	the agent's type: what it gains of its own when the principal's mission succeeds	Section 14.5.1
$q_{\text{insp}}$	the probability of inspection in an inspection game	Section 14.6
$\mathrm{T}$ , $\mathrm{R}$ , $\mathrm{P}$ , $\mathrm{S}$	the payoffs of a prisoner's dilemma: temptation, reward, punishment, and the sucker's payoff (upright, since $T$ and $P$ are listed)	Section 57.2.1
$n_{\text{att}}$ , $p_{\text{succ}}$	the expected attack attempts over a deployment, and the chance an unaudited attempt succeeds	Section 50.2
$\alpha_{\text{def}}$	the share of honest outputs scoring above the lower, deferral threshold, audited ones included, so that the share deferred to a trusted model is $\alpha_{\text{def}} - \alpha$	Section 50.2
$n_{\text{inst}}$ , $p_{\text{inst}}$	the number of instances of a model deployed, and an instance's probability of failing (as a function, $p_{\text{inst}}(x)$ , of the situation)	Section 51.4, Section 57.4
$p_{\text{opp}}$ , $p_{\text{pr}}$ , $p_{\text{act}}$	the share of an agent's episodes with an opportunity for harm, the share of those under pressure, and the conditional propensity to act	Section 56.2
$n_{\text{team}}$	the number of teams: in each arm of an uplift trial, or in a race	Section 59.1, Section 65.2.2
$\mathrm{Harm}$	the harm one bad outcome causes, in a stated unit; $\mathrm{Harm}_c$ for harm category $c$	Section 59.1, Section 61.5.1, Section 63.1.1
$W \wedge S \vdash R$	the proof obligation: domain assumptions about the world $W$ and a specification $S$ together entail the requirements $R$ ; the three letters keep these meanings only inside it	Section 19.1
$p_{\text{find}}$	the chance of following an obfuscated flaw to its leaf	Section 41.4
$\Gamma_{\mathrm{g}}$ , $\Gamma_0$	the general-capability Elo gap between overseer and overseen, and a game's tilt	Section 41.6
$n_{\text{step}}$	the number of steps in nested oversight; $p_{\text{step}}$ and $P_{\text{chain}}$ the per-step and chain success	Section 41.6
$E_{\mathrm{G}}$ , $E_{\mathrm{H}}$	the Elo scores of the overseer (Guard) and the overseen (Houdini)	Section 41.6

B.8 Representations and interpretability #

Symbol	Meaning	Defined in
$\mathcal{V}$	a language model's vocabulary	Section 11.1.1
$\vh$	an activation vector, such as the residual stream at one position	Section 11.2
$m$	the dimension of an activation vector	Section 11.2
$\mW_E$ , $\mW_U$	the embedding and unembedding matrices	Section 11.2.2
$\mW_Q$ , $\mW_K$ , $\mW_V$ , $\mW_O$	an attention head's query, key, value, and output matrices; $\mW_Q^\T \mW_K$ and $\mW_O \mW_V$ are its QK and OV circuits	Section 11.2.2, Section 11.2.3
$m_{\text{head}}$	the dimension of an attention head	Section 11.2.2
$a_{ts}$	the attention weight from position $t$ to an earlier position $s$	Section 11.2.2
$\vf$	a vector of feature activations	Section 44.2
$\mW_{\text{enc}}$ , $\mW_{\text{dec}}$	the encoder and decoder of a sparse autoencoder	Section 44.5
$\lambda$	the weight of a sparsity penalty	Section 44.5
$\vv$ , $v_0$	a direction in activation space (a probe or steering vector), and a probe's offset	Section 43.2.2
$\mathrm{Net}$ , $\mathrm{Alg}$	a network, and a high-level algorithm it may implement	Section 43.5.2
$\mathrm{Perf}$	a scalar measure of a model's output, such as a logit difference	Section 43.3.3, Section 45.1.1
$n_{\text{f}}$	the number of features, or of latents in a dictionary	Section 44.2
$\vmu_0$ , $\vmu_1$	the mean activation without and with a concept	Section 46.1
$\mSigma$	the within-class covariance of activations; $\mSigma_{\text{u}}$ the covariance of the unembedding vectors	Section 46.1
$\vv_{\text{read}}$ , $\vv_{\text{write}}$	for two Gaussian classes with a shared, invertible covariance: the best linear probe direction, $\mSigma^{-1}(\vmu_1 - \vmu_0)$ , and the added vector that turns one class's distribution into the other's, $\vmu_1 - \vmu_0$	Section 46.1.3

B.9 Abbreviations #

Each abbreviation is listed with the section that introduces the term. Names of organizations, models, benchmarks, and laws are not listed. Several abbreviations name a framework or method that a party with a stake in it coined, most often a laboratory naming its own framework; the last column names the coiner, as the glossary does (Appendix A), and is empty for terms standard in their field.

Table B.1 Abbreviations

Abbreviation	Spelled out	Introduced in	Coined by
ACDC	Automatic Circuit DisCovery	Section 45.3.1
AGI	artificial general intelligence	Section 2.5.2
API	application programming interface	Section 12.2.1
ASL	AI Safety Level	Section 62.1.1	Anthropic
ASR	attack success rate	Section 48.1.2
AtP	attribution patching	Section 43.3.3
AUC, AUROC	area under the ROC curve	Section 5.8.3
BIC	Bayesian information criterion	Section 10.2.2
CBRN	chemical, biological, radiological, and nuclear	Section 62.1.2
CCL	Critical Capability Level	Section 62.1.2	Google DeepMind
CCS	contrast-consistent search	Section 39.7.2
CDT	causal decision theory	Section 7.6.1
CIRL	cooperative inverse reinforcement learning	Section 22.2
CoT	chain of thought	Section 11.2.5
DAS	distributed alignment search	Section 43.5.4
DPO	direct preference optimization	Section 12.6
EDT	evidential decision theory	Section 7.6.1
ELK	eliciting latent knowledge	Section 39.7.1	Christiano, Cotra, and Xu, Alignment Research Center
FDT	functional decision theory	Section 7.6.1	Yudkowsky and Soares, MIRI
FLOP, FLOPs	floating-point operation, floating-point operations	Section 8.5.2
GRPO	Group Relative Policy Optimization	Section 12.7.1	Shao et al., DeepSeek
GSN	goal structuring notation	Section 52.2
HHH	helpful, honest, and harmless	Section 1.1.4	Askell et al., Anthropic
ICM	internal coherence maximization	Section 42.5.2	Wen et al., most of them at Anthropic
IIA	interchange intervention accuracy	Section 43.5.2
IPO	Identity-PO: $\Psi$ -preference optimization with the identity mapping	Section 22.4.1	Azar et al., Google DeepMind
KL	Kullback-Leibler, as in KL divergence	Section 6.3.1
KTO	Kahneman-Tversky optimization	Section 13.3.4	Ethayarajh et al.
LEACE	least-squares concept erasure	Section 43.2.4
LLC	local learning coefficient	Section 10.2.5
LLM, LM	large language model, language model	Chapter 11
LoRA	low-rank adaptation	Section 13.5.4
MAIM	Mutual Assured AI Malfunction	Section 65.3.5	Hendrycks, Schmidt, and Wang
MDP	Markov decision process	Section 7.4.1
MEG	maximum entropy goal-directedness	Section 7.1.3
ML	machine learning	Chapter 8
MLP	multilayer perceptron: the two-layer network in each transformer block	Section 11.2.2
MONA	myopic optimization with non-myopic approval	Section 28.4.2	Farquhar et al., Google DeepMind
OV, QK	output-value and query-key, the two circuits of an attention head	Section 11.2.3
PAC	probably approximately correct, as in PAC learning	Section 47.6
PGR	performance gap recovered	Section 42.1.1	Burns et al., OpenAI
PPO	proximal policy optimization	Section 12.1.4
PPV	positive predictive value	Section 17.3
ReLU	rectified linear unit	Section 8.2.1
RL	reinforcement learning	Chapter 12
RLAIF	reinforcement learning from AI feedback	Section 22.7.3	Bai et al., Anthropic
RLHF	reinforcement learning from human feedback	Section 22.3
RLVR	reinforcement learning with verifiable rewards	Section 12.7	Lambert et al., in a developer's report
RMU	Representation Misdirection for Unlearning	Section 27.7.1
ROC	receiver operating characteristic	Section 6.7.2
ROME	Rank-One Model Editing	Section 43.4.2
RSP	responsible scaling policy	Section 62.1.1	Anthropic, which credits the framework to Paul Christiano and ARC Evals
SAE	sparse autoencoder	Section 32.3
SDF	synthetic document fine-tuning	Section 5.3.1	Anthropic
SFT	supervised fine-tuning	Section 12.2.1
SGD	stochastic gradient descent	Section 9.3.2
STPA	System-Theoretic Process Analysis	Section 52.2
TD	temporal difference, as in TD error	Section 29.3.1