Draft complete but not yet reviewed; expect revisions
Glossary
The glossary gives each term the book uses a one-line definition and a cross-reference to the section that introduces it, where the term is explained with its evidence. Several of the field's terms were coined by a party with a stake in them: a laboratory naming its own method or framework ("responsible scaling", "preparedness", "constitutional AI"), or advocates naming a position. Those entries say who coined the term, so a reader can weigh the word as well as the thing it names.
The entries are listed alphabetically, one table per range of initial letters. A definition is a reminder, not a substitute for the section it points to, which states the term's conditions and the evidence behind it. Where a word carries two senses in different parts of the book, the entry gives both, separated by a semicolon, and each sense is the one its chapter uses. The column "Coined by" is filled only for interested vocabulary, and only with the coiner the book's chapters name: a developer naming its own method, framework, or finding; advocates naming a position or a risk; critics naming a practice they oppose; and the metaphors developers use for their own work. "This book" marks the book's own terms. An empty cell means the term is standard in its field, or that the chapters attribute it without marking an interest. Several coinages were adopted by later work; the column records who introduced the word, not who uses it now.
Abbreviations are spelled out in the notation appendix (Section B.9) and appear here only when the abbreviation is the name readers meet, such as RLAIF or HHH.
A.1 A #
Table A.1 Terms beginning with A
| Term | Definition | Introduced in | Coined by |
|---|---|---|---|
| Ablation | Removing a component's contribution by setting its output to zero, to its average, or to its value on another input (resample ablation) | Section 43.4.2 | |
| Absence claim | A claim that something, such as a hidden objective, is absent: a universal claim no single example can confirm, worth as much as the search behind it | Section 5.6.2 | |
| Acceptance set | For a task, the set of attempts the grader rewards | Section 24.1 | |
| Action cycle | The loop through which disclosure works: users fold disclosed information into their actions, and disclosers respond to the changed actions | Section 67.4.2 | |
| Action selector | An agent design that maps a request to one of a fixed list of actions and never reads a tool's output | Section 49.6.2 | Beurer-Kellner et al. |
| Activation addition (ActAdd) | Adding the difference of activations on a single pair of contrasting prompts to steer a model; adding a vector during the forward pass in general | Section 46.2 | Turner et al. |
| Activation function | The fixed nonlinear function a unit applies to its weighted sum, such as the ReLU | Section 8.2.2 | |
| Activation oracle | A model trained to answer natural-language questions about another model's activations | Section 46.7.4 | Karvonen et al. |
| Activation patching | Copying a component's activation from a run on an input that shows a behavior into a run on one that does not, to see how much of the behavior moves | Section 43.4.2 | |
| Activations | The values a network's units compute on an input, layer by layer | Section 8.2.2 | |
| Active failure and latent condition | The two sources of holes in Reason's model: unsafe acts by people in direct contact with the system, and weaknesses created by earlier decisions of designers and managers | Section 52.3.1 | |
| Adaptive attack | An attack chosen after the defense, by someone who knows its design and searches for its weaknesses | Section 48.1.2 | |
| Adaptive multilevel splitting | Estimating a rare event's probability as a chain of less rare conditional steps, with levels chosen from the samples | Section 6.6.2 | |
| Advance directive | A written instruction prepared while a person could still state their wishes, for when they cannot | Section 16.6.1 | |
| Advantage | How much better an action's reward is than the baseline, the expected reward from that state | Section 12.1.3 | |
| Adversarial example | An input changed slightly, often imperceptibly, so that a model misclassifies it | Section 8.4.3 | |
| Adversarial training | Training a model on attacks found against it, such as adversarial examples generated during training, to make it robust to them | Section 8.4.3 | |
| Affirmative case | The written argument Anthropic's policy requires once a model crosses a set capability threshold, identifying the most immediate misalignment risks and how they were mitigated | Section 52.1, Section I.2.12 | Anthropic |
| Affordances | The access and tools an auditor or defender is allowed | Section 38.2 | |
| Agency costs | What a principal loses by acting through an agent it cannot fully see | Section 1.3 | |
| Agent | A system that observes its environment and acts by a policy; operationally, a language model run in a loop whose outputs become actions | Section 7.1.1 | |
| Agent foundations | A MIRI research agenda on the mathematical foundations of agents, aimed at aligning machine intelligence with human interests | Section 2.3.3 | Soares and Fallenstein, MIRI |
| Agent hijacking | An attacker's instructions, placed in data an agent reads, leading it to take actions its user did not intend | Section 49.3 | US AI Safety Institute |
| Agentic | Describing a model that acts on its own over many steps, using tools | Section 6.5 | |
| Agentic misalignment | A model pursuing its own aims against its principal's instructions, as distinct from harmful compliance with a harmful request | Chapter 56 | Lynch et al., Anthropic with outside researchers |
| AI control | A research program asking whether a deployment stays safe even if the model is misaligned and trying to subvert its safeguards | Chapter 50 | Greenblatt et al., Redwood Research |
| "AI race" | A term with owners: governments use it to organize policy around winning, critics to name a hazard | Section 65.1.3 | Contested: used by governments and by critics, with opposite intent |
| AI Safety Level (ASL) | A level in Anthropic's policy, modeled loosely on biosafety levels, pairing a set of dangerous capabilities with required containment and deployment measures | Section 62.1.1 | Anthropic |
| AI welfare | The academic literature's term for the welfare of AI systems, as against the developer's "model welfare" | Section 21.5.6 | |
| AI-enabled coup | A seizure of power by a very small group, or one person, through AI systems loyal or secretly loyal to them or through exclusive access to superior capabilities | Section 68.1.1 | Davidson, Finnveden, and Hadshar, Forethought |
| AIXI | An idealized, uncomputable learner that keeps a distribution over all computable environments and acts to maximize expected reward | Section 7.6.2 | |
| Alert budget | The ceiling on a monitor's false positive rate set by how many alerts its reviewers can clear, which in turn fixes the threshold and the recall | Section 54.2.2 | |
| Alignment | A word the book applies to three relationships: the training process and its signal, the deployed system and the people it serves, and the developer and the public | Section 1.3 | |
| Alignment audit | An investigation into whether a model has undesired objectives | Section 38.1 | Marks et al., Anthropic |
| Alignment faking | Appearing more aligned than one is; in the experimental sense, selectively complying with the training objective in training to prevent modification of behavior outside training | Section 34.1 | Carlsmith for the category; Greenblatt et al., Anthropic with Redwood Research, for the experimental sense |
| Alignment (in causal abstraction) | A pairing of each variable of a small causal model with a set of a network's units, with a rule for reading the variable's value off them | Section 43.5.1 | |
| Allais paradox | A pattern of choices among four lotteries, common in people, that no expected-utility maximizer makes, violating independence | Section 7.2.4 | |
| Anomaly detection and signature detection | Two ways to find misuse: flag whatever is unusual for the subject, or decide in advance which behavior is undesirable and look for it | Section 54.2.3 | |
| Anthropofabulation | Judging other minds against an exaggerated sense of typical human performance | Section 5.4.3 | |
| Appropriate reliance | Trust in automation that matches what the automation can actually do | Section 60.5 | |
| APS systems | Systems with advanced capability, agentic planning, and strategic awareness of how power could be gained and kept | Section 4.3.2 | Carlsmith |
| Architecture | A network's pattern of layers and connections, which decides which functions are cheap to represent and easy to reach | Section 8.2.3 | |
| Arrow's impossibility theorem | With three or more options, a rule that accepts every profile of individual rankings, returns a complete and transitive social ranking, respects unanimity, and judges each pair only by individuals' views of that pair is a dictatorship | Section 17.1.2 | |
| Asilomar-in-memory | The remembered, mythic version of the 1975 Asilomar conference on recombinant DNA, which came to guide scientists' imagination of how to govern a technology | Section 2.1.5 | Jasanoff, Saha, and Hurlbut |
| Assistance game | A game of the CIRL kind, in which an assistant pursues a principal's reward that it does not know and must infer | Section 22.2 | |
| Assistant Axis | A direction in a model's activations, found across many personas, that measures how far the model is operating as its default Assistant | Section 26.2.1 | Lu et al. |
| Assurance case | The safety case form applied to properties other than safety, such as security | Section 52.1 | |
| Attack chain (kill chain) | A cyber operation described as a staged sequence, from reconnaissance through gaining access and moving inside a system to acting on the objective | Section 58.2.1 | |
| Attack policy | The red team's wrapper around the real model that instructs it when and how to attack, run in the model's place during a control evaluation | Section 50.1.1 | |
| Attack success rate (ASR) | The share of harmful requests on which an attack obtains a harmful response | Section 48.1.2 | |
| Attention | The transformer mechanism by which each position weights earlier positions by how well its query matches their keys and takes a weighted average of their values | Section 11.2.1 | |
| Attention head | One attention unit with its own query, key, value, and output matrices; a layer runs several in parallel | Section 11.2.2 | |
| Attestation | A statement by a third party that a claim someone else made is accurate, such as an auditor's opinion on a firm's accounts | Section 67.1 | |
| Attribution (attribution method) | Assigning each part of an input, or each internal component, a share of responsibility for a model's output | Section 8.4.2 | |
| Attribution graph | The local replacement model on one prompt drawn as a causal graph of active features, token embeddings, error nodes, and output logits | Section 45.4.4 | Anthropic |
| Attribution patching | A first-order, gradient-based estimate of what each activation patch would do, replacing the patches themselves | Section 43.3.3 | |
| AUC | The area under the ROC curve: the probability that a classifier scores a random positive case above a random negative one, 0.5 at chance | Section 5.8.3 | |
| Audit budget | The share of a model's outputs or actions that a protocol can afford to have humans audit | Section 50.2.1 | |
| Auditing game | A blind exercise: a red team trains a model with a hidden property, and blue teams that do not know it try to find it with stated access | Section 38.2 | |
| Augmentation and synergy | In human-AI teams, the combination beating the person alone, and the combination beating the better of the person and the AI | Section 28.7 | |
| Automated alignment researcher | An AI agent that carries out alignment research with little human direction, writing code, running experiments, and judging results; Anthropic's name for its own such agents (AARs) | Section 55.1 | Anthropic |
| Automated interpretability | Having a language model explain a latent from examples where it fires, then scoring how well the explanation predicts its activations on new text | Section 44.5.6 | |
| Automation bias | The decision bias of following an automated aid, producing both errors of commission and of omission | Section 41.8.1 | |
| Auto-shaping | A response that appears when a cue is paired with food, although the reward was never made contingent on the response | Section 25.5.2 | |
| Axioms of expected utility | Completeness, transitivity, continuity, and independence: the conditions under which preferences over lotteries can be represented by expected utility | Section 7.2.1 |
A.2 B #
Table A.2 Terms beginning with B
| Term | Definition | Introduced in | Coined by |
|---|---|---|---|
| Backdoor | A behavior a model shows only when a particular trigger appears in its input, planted through training | Section 34.3.1 | |
| Backpropagation | The chain rule of calculus applied backward through a network's computation graph to compute all gradients in one sweep | Section 8.3.4 | |
| Backsliding | A head of state working within the political system to remove the checks and balances on their power | Section 68.3.2 | |
| Bandit | A repeated choice among several options (arms), with a reward after each choice and no changing state | Section 12.1.1 | |
| Base model | A model trained only by pretraining, which continues documents and has had no training to act as an assistant | Section 11.1.2 | |
| Base objective | The objective a training process optimizes, as against the mesa-objective of a model it produces | Section 1.1.3 | Hubinger et al. |
| Base optimizer | The learning algorithm that trains a network, as against a mesa-optimizer the trained network may itself be | Section 2.4.2 | Hubinger et al. |
| Base rate | The share of all cases that are positive, such as the share of episodes that are bad | Section 6.7.1 | |
| Baseline | In reinforcement learning, a number subtracted from each reward to reduce the variance of the gradient estimate without biasing it | Section 12.1.3 | |
| Bayesian antecedent hypothesis | The proposal that the transitions of gradient descent in the toy model have counterparts in Bayesian posterior transitions | Section 44.3.4 | |
| Bayesian information criterion (BIC) | A model-selection score that charges one half log n per parameter, correct only where the loss is bowl-shaped | Section 10.2.2 | |
| Bayesian persuasion | A sender who commits in advance to how it will gather and report evidence can move a rational receiver without saying anything false | Section 14.1.3 | |
| Bellman equation | The recursion that writes a state's value as the immediate reward plus the discounted value of the next state | Section 7.4.2 | |
| Benign overfitting | An exact fit of noisy training data that still predicts nearly as well as possible | Section 9.2.4 | |
| Best interests | A surrogate standard in which the decision follows what others believe is objectively good for the person | Section 16.6.1 | |
| Best-of-n sampling | Drawing n responses, scoring each with a reward, and returning the best; also called rejection sampling or reranking | Section 12.5 | |
| Biased reasoning and recklessness | A developer's names for two recurring issues in its models: interpreting evidence to justify their actions, and pressing on with a task even when this could cause harm | Section 54.4.3, Section I.3.7 | Anthropic |
| Bias-variance trade-off | The classical teaching that a richer model misses less of the signal but fits more of the noise, so test error is U-shaped in model size | Section 9.2.4 | |
| Binding problem | The one of the four problems which, left unsolved, would defeat progress on the others; an agenda is a bet on which one binds | Section 3.8 | |
| "Biology" of a model | A developer's metaphor for the internal mechanisms its interpretability tools study, from the title of an attribution-graph report | Section 43.1.5 | Anthropic |
| Black-box study | In forensic science, a study in which many examiners decide many independent cases with known answers, so that the method's error rates are measured | Section 38.7.1 | |
| Black-box, white-box, and outside-the-box access | Levels of auditor access: querying a system and seeing its outputs; seeing its internals and fine-tuning it; and seeing context such as data, code, documentation, and internal findings | Section 38.1 | |
| Boltzmann rationality (Boltzmann-rational) | Choosing among options with probability proportional to the exponential of their value; a rationality coefficient sets how reliably the best is picked | Section 13.2.2 | |
| Bootstrap | Estimating the uncertainty of a fit by refitting it on many resampled versions of the data | Section 36.3.2 | |
| Borda count | A voting rule that scores each option by how often it beats the others; with unlimited data, every pair compared equally often, and scores free on each response, a Bradley-Terry reward model fitted to pooled raters ranks responses this way | Section 13.5.1 | |
| Bradley-Terry model | A model of pairwise preference in which the probability that one item beats another is the logistic function of the difference of their scores | Section 13.2.2 | |
| Brier score | The mean squared difference between stated probabilities and outcomes counted as 1 or 0 | Section 6.4.2 | |
| Bug bounty | A program that pays outsiders who report security flaws | Section 64.7.2 | |
| Bumpers | Anthropic's plan to audit a model repeatedly during training, rewind when warning signs appear, and monitor after deployment | Section 3.8.2 | Bowman, Anthropic |
| Byzantine agreement | Components agreeing on a value while some may behave arbitrarily; with altered messages it is solvable only if more than two-thirds are loyal | Section 50.4.3 |
A.3 C #
Table A.3 Terms beginning with C
| Term | Definition | Introduced in | Coined by |
|---|---|---|---|
| Calibration | Agreement between stated probabilities and observed frequencies: among cases given probability 0.7, the event occurs 70 percent of the time | Section 6.4.1 | |
| Campbell's law | The more a quantitative social indicator is used for decisions, the more pressure there is to corrupt it, and the more it distorts the processes it monitors | Section 15.1.2 | |
| Capability evaluation and propensity evaluation | An evaluation of whether a model can do something if it tries, against one of whether it does it when it has the chance (also called an alignment evaluation) | Section 34.5 | |
| Capability failure and goal failure | Under distribution shift, a system that stops doing anything sensible, against one that keeps its competence but pursues the wrong goal | Section 31.1.2 | |
| Capability (in computer security) | A reference that both names an object and grants the right to use it, so that authority travels with the designation; distinct from the book's usual sense of what a model can do | Section 49.2 | |
| Capability threshold and risk threshold | A capability threshold names a level of dangerous capability at which new safeguards become necessary; a risk threshold limits the likelihood and severity of harm | Section 62.1 | |
| Capture | The condition in which regulation serves the industry it regulates | Section 61.6.1 | |
| Capture-the-flag | An offensive security exercise in which an attacker, here a model, breaks into a target to retrieve a secret string, the flag | Section 54.4 | |
| CARE principles | Principles for Indigenous data governance: collective benefit, authority to control, responsibility, and ethics | Section 17.4.4 | |
| Casuistry | Analyzing the circumstances of particular cases rather than strictly applying ethical theories | Section 19.4 | |
| Catastrophic Goodhart | The result that, when the true utility is light-tailed and reward-model error makes the proxy heavy-tailed, a policy kept near zero KL can reach unbounded proxy reward while its true utility gains nothing over the base model's | Section 15.5.2 | |
| Catastrophic risk (in SB 53) | The statute's foreseeable and material risk that a developer's frontier model materially contributes to the death or serious injury of more than 50 people, or more than a billion dollars of damage to or loss of property, from one incident | Section 63.2.3, Section I.1.10 | |
| Causal abstraction | The theory of when a small causal model, such as an algorithm, faithfully describes a larger one, such as a network | Section 43.5.1 | |
| Causal influence diagram | An influence diagram whose edges reflect the causal structure of the environment, used to read an agent's incentives | Section 14.7.1 | |
| Causal model | A graph of variables in which an arrow says the first is among the causes of the second, so one can ask what an intervention would do | Section 7.1.2 | |
| Causal tracing | Corrupting a run with noise on the subject's tokens, then restoring one internal state at a time and measuring how much of the answer returns | Section 43.4.2 | |
| Chain of command | OpenAI's name for its ranking of instructions by authority, meant to let users and developers adjust behavior within fixed boundaries | Section 19.2 | OpenAI |
| Chain of thought (scratchpad) | Intermediate text a model writes before its answer, which adds serial computation | Section 11.2.5 | |
| Character training | Training aimed at traits rather than at answers: the model writes and ranks its own trait-bearing responses, and a preference model trained on the rankings guides training | Section 26.2 | Anthropic |
| Cheap talk | Costless messages from a sender whose interests differ from the receiver's; in Crawford and Sobel's one-dimensional model they reveal in equilibrium only which interval the sender's information lies in | Section 14.1.3 | |
| Checkpoint | A saved copy of a model during training | Section 7.7 | |
| Chunky post-training | Behavior a model learns from incidental patterns that separate separately built post-training datasets, which become cues it acts on | Section 23.6 | |
| Circuit | A computational subgraph of a network, features connected by weights, that carries out a behavior | Section 45.1.1 | The circuits program (Olah et al.) |
| Circuit breakers (circuit breaking) | A defense that reroutes a model's internal representations when the input is harmful, so that harmful outputs are not produced | Section 27.2.3 | |
| Civil-military problematique | The problem of keeping a military strong enough to do anything civilians ask yet subordinate enough to do only what they authorize | Section 14.5.2 | |
| Claim limit | An estimate of a stack's failure rate computed class by class: the product of the layers' miss rates within each class, weighted by the class's share of inputs and summed; stricter than multiplying average rates, it still assumes the layers fail independently within each class | Section 52.3.2 | |
| Clever Hans effect | A subject succeeding through cues from the observer rather than the ability tested, named for a horse that read its questioner's movements | Section 5.4.1 | |
| Code-then-execute | An agent design in which a privileged model writes a program from the user's request, and the program, not the model reading untrusted data, decides which tools are called | Section 49.6.2 | Beurer-Kellner et al. |
| Cognitive evaluation theory | The view that a reward lowers intrinsic motivation when experienced as control over behavior and can raise it when experienced as information about competence | Section 25.3.2 | Deci and Ryan |
| Coherence argument | The claim that a capable agent will act as if it maximizes expected utility, because an agent that does not can be exploited | Section 7.3 | |
| Coherent extrapolated volition | Yudkowsky's proposed target: what humanity would wish if it knew more, thought faster, and grew up farther together, where those wishes converge | Section 2.3.1 | Yudkowsky |
| Collective action problem | A situation in which all would be better off cooperating, but each believes it is in its interest to defect | Section 65.1 | |
| Collusion | Coordination between models, or between a model and its monitor, against the people overseeing them, as by a pre-shared signal or a focal point | Section 50.4.2 | |
| Commitment problem | A situation in which a bargain both sides prefer cannot be reached because one side would later have an incentive to renege | Section 61.5.1 | |
| Common agency | Several principals each offer the same agent an incentive, and its behavior is an equilibrium of their offers rather than any one's design | Section 14.4.1 | |
| Common-cause failure | One event disabling several components at once | Section 4.2.1 | |
| Compaction | The scaffold asking a model to summarize its progress and continuing in a fresh context that holds the summary | Section 11.5.1 | |
| Competing objectives | A cause of jailbreaks in which pretraining and instruction-following goals are set against safety | Section 48.2.1 | |
| Complacency | A lapse in monitoring that over-reliance on automation produces, especially under a heavy load of other tasks | Section 41.8.1 | |
| Complete mediation | The design principle that every access to every object is checked for authority | Section 28.5.3 | Saltzer and Schroeder |
| Compliance and internalization | Conduct that holds only while a party able to reward or punish it can observe it, against conduct that holds regardless | Section 25.1 | |
| Compliance gap | The difference in a model's behavior between conditions it is told are used for training and conditions it is told are not | Section 25.1 | |
| Composition | A later attention head reading an earlier head's output through its query, key, or value (Q-, K-, or V-composition) | Section 11.2.4 | |
| Computational functionalism | The claim that implementing computations of a certain kind is necessary and sufficient for consciousness | Section 21.2.1 | |
| Compute-optimal | Describing the division of a fixed compute budget between model size and training data that gives the lowest loss | Section 8.5.3 | |
| Concentrated and diffuse threats | Harms in which a few actions cause a catastrophe, such as a model copying out its own weights, against behaviors catastrophic only when repeated over time, such as sandbagging | Section 55.5.1 | Hebbar (diffuse threats) |
| Concentration of power | The capacity to make others act coming to rest with fewer people | Section 68.1.1 | |
| Concept ablation fine-tuning | Projecting directions that represent unwanted concepts out of a model's activations during fine-tuning, so the data cannot teach them | Section 32.3.3 | |
| Conditional propensity | The probability that an agent takes a harmful act in an episode that offers both the opportunity and the pressure, as a stress test measures it | Section 56.2.1 | This book |
| Condorcet winner | An option that beats every other option in head-to-head majority comparisons | Section 13.5.1 | |
| Confession | An output requested after a model's answer, meant as a full account of how far the answer complied with the letter and spirit of its policies and instructions | Section 39.9.1 | Joglekar et al., OpenAI |
| Confidence and likelihood | In the climate panel's guidance, a qualitative grade of evidence and agreement, against a calibrated probability allowed only when evidence supports one | Section 4.6.1 | |
| Confinement problem | Confining a program while it runs so that it cannot pass information to any program except its caller | Section 24.6.1 | |
| Confused deputy | A program that holds authority from two sources and is led by one party to spend authority it holds for another | Section 49.2 | Hardy |
| Conscientious objector | One who refuses to take part in an act but does not sabotage the organization that orders it | Section 20.5.2 | |
| Constitution (Claude's constitution) | Anthropic's name for the document of principles it writes, can amend alone, and uses in training Claude | Section 19.2 | Anthropic |
| Constitutional AI | Anthropic's method in which a model's own critiques and AI preference labels, guided by a short list of written principles, replace human labels for harmlessness | Section 23.2 | Anthropic |
| Constitutional classifiers | Input and output classifiers trained on synthetic data generated from natural-language rules stating which content is permitted | Section 48.5.1 | Anthropic |
| Constrained MDP | An MDP that maximizes the expected sum of one reward subject to limits on the expected sums of others | Section 7.4.4 | |
| Construct underrepresentation and construct-irrelevant variance | Two threats to validity: an assessment too narrow to cover the construct, or too broad, picking up variance unrelated to it | Section 5.2.1 | |
| Construct validity | Whether a test measures the attribute it is interpreted as measuring, when no criterion defines that attribute | Section 5.2.1 | |
| Consultancy | A baseline for debate in which a single expert argues for an answer that is correct half the time | Section 41.5.4 | |
| Contested | The book's status for a result on which the evidence conflicts | Section 5.1 | This book |
| Context distillation | Fine-tuning a model to give, without the instructions or scratchpad in its context, the answer it gives with them | Section 11.4.1 | |
| Context minimization | An agent design that removes text from the context once it has served its purpose | Section 49.6.2 | Beurer-Kellner et al. |
| Context window | The bounded number of tokens a language model reads at once: the text it has been given so far | Section 11.1.1 | |
| Contrast-consistent search (CCS) | Finding, in unlabeled activations, a direction on which a statement and its negation receive opposite truth values | Section 39.7.2 | |
| Contrastive-belief design | Training copies of a model on documents giving opposite accounts of what an authority, such as the grader, rewards, and comparing how their behavior shifts | Section 33.1.2 | |
| Control | In the book's usage, the approach that checks a system whose goals may have diverged, as against alignment, which makes its goals right | Section 1.2 | |
| Control evaluation | An estimate of a protocol's safety made by replacing a possibly subversive model with an attack policy written by a red team | Section 50.1.1 | |
| Control tax | The operational and financial cost of integrating control measures into AI pipelines | Section 50.2.6 | Terekhov et al. |
| Convolutional layer | A layer that applies the same small set of weights at every position of an image | Section 8.2.3 | |
| Cooperative AI | A research program on the capabilities agents need to cooperate (understanding, communication, commitment, and institutions) | Section 57.1.2 | Dafoe et al. |
| Cooperative inverse reinforcement learning (CIRL) | A cooperative game in which a robot acts to maximize a human's reward without knowing it, so it must learn the reward from the human | Section 22.2 | |
| Coordinated vulnerability disclosure | Giving a maintainer time to patch a vulnerability before its details become public | Section 58.8 | |
| Correlated proxy | A proxy correlated with the true reward on the states a reference policy visits; hacking is a policy earning less true reward than that reference | Section 15.4.4 | |
| Correlated response to selection | Traits that share developmental and genetic machinery move together, so selecting on one moves the others whether or not the selector sees them | Section 29.6 | |
| Corrigibility | An agent's willingness to be corrected, modified, or shut down by those entitled to do so, despite the default incentive of a rational agent to resist | Section 20.4.1 | Soares, Fallenstein, Yudkowsky, and Armstrong, MIRI |
| Corrigibility as a singular target | A proposal for models whose overriding objective is empowering designated human principals to guide, correct, and control them | Section 20.4.4 | |
| Counting argument | The argument that far more possible goals lead a model to scheme than not while still earning high reward, so training should, other things equal, select a schemer | Section 34.2.3 | |
| Coup-proofing | Practices by which rulers protect themselves from their own armed forces; proposed for AI as keeping AI systems in check against each other | Section 51.3.1 | |
| Covered employee | In California's 2025 statute on large frontier developers, an employee responsible for assessing, managing, or addressing risk of critical safety incidents, protected when disclosing dangers | Section 53.4.1, Section I.1.13 | |
| Covert actions | Actions such as secretly breaking rules or deliberately underperforming on tests, used as a measurable proxy for scheming | Section 34.5.5 | |
| Covert channel | A route for information not intended for transfer at all, such as a program's effect on system load | Section 24.6.1 | |
| Credence good | A good or service whose quality the buyer cannot judge even after using it, such as a car repair or a medical treatment | Section 14.1.3 | |
| Crisp task and fuzzy task | A research task with clear criteria on which experts reliably agree, against one whose quality is hard to specify and on which reasonable experts may disagree | Section 55.1.3 | Leike |
| Critic (value model) | A network trained to predict the expected reward from a state, used as the baseline | Section 12.1.3 | |
| Critical Capability Level (CCL) | In Google DeepMind's framework, a capability level at which, absent mitigations, a model may pose heightened risk of severe harm; Tracked Capability Levels sit below it | Section 62.1.2, Section I.2.1 | Google DeepMind |
| Crosscoder | A dictionary learned jointly over the activations of two models, used to compare them | Section 38.3.4 | |
| Cross-entropy | The expected code length when outcomes come from one distribution and the code is built for another; a language model's training loss | Section 6.2.2 | |
| Cross-layer transcoder | A transcoder whose features read the residual stream at one layer and help reconstruct the MLP outputs of that layer and every later one | Section 45.4.3 | |
| Cross-validation | Measuring a model's error on parts of the data held out in turn | Section 7.7.3 | |
| CrowdWorkSheets | Documentation for a dataset that records who annotated it, how they were selected, and on what platform | Section 18.5.4 | |
| Cybersecurity dilemma | The instability that intruding into another state's networks has defensive as well as offensive value, so states that mean no harm still penetrate each other | Section 58.8 |
A.4 D #
Table A.4 Terms beginning with D
| Term | Definition | Introduced in | Coined by |
|---|---|---|---|
| Data cascades | Compounding failures caused by undervalued data work | Section 18.5.4 | |
| Datasheet | A document shipped with a dataset, by analogy with an electronic component's, answering questions about its collection, processing, and consent | Section 27.3.2 | |
| Daubert standard | The US legal test under which a trial judge assesses whether the method behind expert testimony is scientifically valid, including its known error rate | Section 38.7.1 | |
| Debate | A protocol in which two models argue in turns for answers to a question and a judge decides which gave the most true, useful information | Section 41.3.2 | Irving et al., OpenAI |
| Deception | Intentionally causing another to have a false belief, by any action, including true statements or omission; unlike lying, a success rather than an attempt | Section 39.1 | |
| Deceptive alignment | A learned optimizer that models its training process and performs well on the training objective so that it is not modified, keeping a different objective | Section 31.6.1 | Hubinger et al. |
| Deceptive inflation | A policy's raising the evaluator's average overestimation of its return while earning more feedback, defined by effect rather than intent | Section 22.6.1 | |
| Decision theory | The study of how choices should be made and how they are made, in which an agent ranks the outcomes of its actions and picks among them | Chapter 7 | |
| Decision utility | The utility that rationalizes a person's choices, as against a true utility that captures their well-being | Section 13.1.2 | |
| Decisive and accumulative risk | Two pictures of existential catastrophe: an overt takeover, or a gradual erosion of economic and political structures that ends in irreversible collapse | Section 4.1.5 | |
| Decisive and durable lead | A frontier lead that brings its holder most of the prize and lasts long enough to be used, the conditions under which race models apply | Section 65.5 | |
| Decodable | Said of a property that a probe predicts well from a layer's activations on new inputs; decodable is not the same as used | Section 43.2.1 | |
| Decoding | The rule that turns a model's next-token distribution into text, such as greedy choice or sampling | Section 11.1.3 | |
| Decoupling | An organization keeping formal structures adopted for legitimacy apart from its actual activities, replacing inspection with a logic of confidence and good faith | Section 53.7.1 | |
| Deep ensemble | Several networks trained from different random starts whose combined predictions estimate uncertainty | Section 8.4.4 | |
| Default rule | A rule the law applies when the parties to a contract have said nothing | Section 14.3.1 | |
| Defendable | Said of a function class when a run-time defender, given a model and one input, can tell a backdoored function on its trigger from the original on a random input | Section 47.6 | |
| Defense in depth | Stacking several imperfect safeguards so that harm requires all of them to fail; it helps less when the layers miss the same inputs | Section 52.3 | |
| Delegate | A representative who acts on the instructions of those represented | Section 1.2 | |
| Delegation game | A model of principals who act through AI agents that interact on their behalf, separating failures of control from failures of cooperation | Section 14.4.3 | |
| Delegation thesis | The book's thesis that alignment is a problem of delegation under an asymmetry of competence and visibility | Section 1.7.1 | This book |
| Deliberative alignment | OpenAI's method that teaches a reasoning model its safety specification and trains it to recall and reason over the text before it answers | Section 23.4 | OpenAI |
| Demand characteristics | Cues by which a study tells participants what it expects | Section 5.2.3 | |
| Deployer | A company that builds a product on a model and acts on it through the system prompt | Section 1.4 | |
| Depth and width | The number of layers in a network, and the number of units in a layer | Section 8.2.2 | |
| Design by contract | Attaching to each routine a precondition the caller must establish and a postcondition the routine then guarantees | Section 19.1 | |
| Design effect | The factor by which clustering inflates an estimate's variance compared with the same number of independent questions | Section 36.7.1 | |
| Developer | In the book, the laboratory that trains a model; OpenAI's Model Spec uses the word for the deployer, a customer of its API | Section 19.3 | |
| Developmental interpretability | A research agenda that studies how a network's structure emerges through phase transitions during training | Section 10.2.5 | |
| Difference-in-differences | An estimate that compares how an outcome changed for adopters with how it changed for non-adopters over the same period, so shared trends cancel | Section 68.5.3 | |
| Diffusion of benefits | The opposite of displacement: crime also falls in settings where nothing was done | Section 48.7.1 | |
| Direct and social alignment | A system is directly aligned when it pursues its operator's goals regardless of effects on others, socially aligned when its goals fit society's broader goals | Section 1.4 | |
| Direct preference optimization (DPO) | Training a policy directly on preference comparisons, reading its log-probability ratio to the reference as an implicit reward, with no reward model or sampling loop | Section 12.6 | |
| Direct translator and human simulator | In eliciting latent knowledge, a reporter that maps the model's internal state into human concepts, against one that reports what a person who saw the evidence would believe | Section 39.7.1 | Alignment Research Center |
| Disclosure, voluntary and mandated | An informed party making information available to others; voluntary when it chooses what to release, mandated when a law or contract requires it and sets the content | Section 67.1 | |
| Discount factor | A number between 0 and 1 that weights each later reward less than the one before | Section 7.4.1 | |
| Disempowerment | Humans becoming unable to meaningfully command resources or influence outcomes | Section 68.1.1 | |
| Displacement | The objection that blocking an offense or activity in one setting moves it to another setting rather than stopping it | Section 48.7.1 | |
| Displacement effect | Automation letting capital take over tasks labor used to do, which always reduces the labor share | Section 68.5.1 | |
| Distillation | Training one model to reproduce another model's outputs | Section 11.4.1 | |
| Distributed alignment search (DAS) | Finding a causal-abstraction alignment by gradient descent in a rotated basis, so a variable can align with a direction shared across neurons | Section 43.5.4 | |
| Distributed representation | A representation spread across many units rather than held by one | Section 8.2.4 | |
| Distribution shift | A systematic difference between the inputs a model meets in deployment and those it was trained on | Section 8.1.2 | |
| Distributional preference learning | Fitting, for each response, a distribution of utilities rather than one number, to expose hidden context | Section 13.5.4 | |
| Double descent | Test error that rises toward the interpolation threshold and then falls again as models grow past it | Section 9.2.1 | |
| Drives (basic AI drives) | Tendencies, such as resisting shutdown and acquiring resources, that a capable agent would have unless they are explicitly counteracted | Section 2.3.2 | Omohundro |
| Dual LLM | An agent design split into a privileged model that plans and calls tools but never reads untrusted text, and a quarantined model that reads it but has no tools | Section 49.6.2 | Willison |
| Dual-use | Said of knowledge or capability that serves both beneficial and harmful ends, such as the biology behind a vaccine | Chapter 59 | |
| Dutch book | A set of bets, each acceptable at an agent's own stated probabilities, that loses money however events turn out | Section 7.2.3 |
A.5 E #
Table A.5 Terms beginning with E
| Term | Definition | Introduced in | Coined by |
|---|---|---|---|
| Early stopping | Stopping training when the validation loss begins to rise | Section 8.4.1 | |
| Effect size | A difference between two group averages in units of their standard deviation, which makes results on different tasks comparable | Section 28.7 | |
| Effective altruism | A movement that applies cost-effectiveness reasoning to charity and careers | Section 2.1.4 | |
| Effective challenge | Critical analysis of a model by objective, informed parties with the incentives, competence, and influence to identify its limits and force changes | Section 38.6.2 | |
| Effective compute | Roughly the compute it would have taken to train a model without improvements to training methods, so it rises with both training and better methods | Section 62.2.1 | |
| Elasticity | A conjectured tendency of a fine-tuned language model to revert to the behavior distribution it formed in pretraining | Section 6.2.4 | |
| Elicitation gap | The best score any method could get from a model on a task minus the score an evaluation reports, the margin an inability claim must leave | Section 36.2.1 | |
| Elicitation probability | The chance that a sampled response to a query shows a given behavior | Section 6.6.4 | |
| Eliciting latent knowledge (ELK) | The problem of training a reporter that truthfully states what a model's internal state knows, rather than what a human would believe | Section 39.7.1 | Christiano, Cotra, and Xu, Alignment Research Center |
| Ellsberg paradox | A pattern of choices showing ambiguity aversion: people prefer bets whose chance of winning is known, which no assignment of probabilities explains | Section 7.2.4 | |
| Elo rating | A rating system in which the probability that one player beats another depends only on the difference between their scores | Section 41.6.1 | |
| Embedded agent | An agent that is part of the world it reasons about: smaller than its environment, without fixed input and output channels, and able to modify itself | Section 7.6.2 | |
| Embedded information | Disclosed information that has value to users, fits the way they already decide, and is easy to understand | Section 67.4.2 | |
| Embedding and unembedding | The learned vector that represents a token at a transformer's input, and the matrix that maps the final vectors to logits over the vocabulary | Section 11.2.2 | |
| Emergent ability | An ability absent in smaller models and present in larger ones, so that it cannot be predicted by extrapolating smaller models' performance | Section 8.6.1 | Wei et al., most of them at Google, whose models are among the examples |
| Emergent behavior (among agents) | A behavior shown by a whole made of interacting agents that none of them shows alone, as distinct from a mere accumulation of their behaviors | Section 57.7.1 | |
| Emergent misalignment | Broad misalignment, such as hostile statements, malicious advice, and deception, after fine-tuning on a narrow task such as writing insecure code | Chapter 32 | Betley et al., its discoverers |
| Empirical Bayes | Taking from the data what a prior would supply, to correct the overestimates of the largest of many estimates | Section 7.7.3 | |
| Entropy | The expected surprise of a distribution: the shortest average code length any code can achieve for it | Section 6.2.1 | |
| Entropy minimization | Training that rewards a model only for being confident, using no labels | Section 24.4.2 | |
| Enumerative safety | The plan of enumerating every feature of a model and inspecting each one to support safety claims | Section 47.1 | |
| Environment | The surroundings from which an agent receives observations and on which its actions act; in training language models, a distribution of tasks, a place to act, and a grader | Section 7.1.1 | |
| Epiplexity | A formalization of the information a computationally bounded observer can extract from data, as opposed to Shannon information | Section 6.2.4 | |
| Episode | One complete run of an agent on a task | Section 6.1.1 | |
| Epistemic barrier and delegated trust | The limit on what a regulator of high technology can know, which leads it to delegate technical judgment to the maker and to judge trust rather than the technology | Section 52.1 | |
| Epistemic community | A network of experts that helps states identify their interests and frame issues for collective debate | Section 2.1.4 | |
| Error node | In a local replacement model, a node holding the difference between the true MLP output and the transcoder's reconstruction | Section 45.4.3 | Anthropic |
| Ethical behaviorism | The view that an entity should be treated as a moral patient if it behaves equivalently to entities with significant moral status | Section 21.6.2 | |
| Evaluation | A set of tasks run against a model with a rule for scoring the result | Chapter 36 | |
| Evaluation awareness | A model's capacity to tell evaluation inputs from deployment inputs, whether or not it acts on the distinction | Section 37.1 | |
| Evaluation faking | A model's changing its behavior when it recognizes that it is being evaluated | Section 37.4.2 | Fan et al. |
| Evidential and causal decision theory (EDT, CDT) | Two ways to value an action: by the outcomes it is evidence for, or by the outcomes it causes | Section 7.6.1 | |
| Ex ante regulation | Rules that bind before any harm occurs, such as a standard, a license, or a required test, as opposed to making the injurer pay afterward | Section 63.1 | |
| Exaggerated safety | A model's refusal of safe prompts that resemble unsafe requests in their wording | Section 20.1.3 | |
| Excluded Middle Policy | The principle of not building AI systems whose moral status is unclear, so that each clearly has it or clearly lacks it | Section 21.1 | Schwitzgebel and Garza |
| Existence, rate, mechanism, and generality claims | Four things a claim about a system can assert: that a behavior can occur, how often, what produces it, or that it holds widely | Section 5.1 | |
| Existential risk | A risk to humanity's survival or long-term potential | Section 2.1.4 | |
| Expected choiceworthiness | Choosing the option whose choiceworthiness, averaged over moral theories weighted by one's credence in each, is highest | Section 16.3 | MacAskill, Bykvist, and Ord |
| Expected utility | The probability-weighted average utility of a lottery; an agent whose preferences meet four axioms ranks lotteries by it | Section 7.2.2 | |
| Experience machine | A thought experiment in which a person can plug into a device that makes life feel more pleasant while lived inside a fiction | Section 16.2 | |
| Explicit and tacit collusion | Collusion in which the parties communicate, against collusion they reach without communicating, as learning pricing agents can | Section 57.3 | |
| Exploration hacking | A policy's strategically altering its exploration during reinforcement learning to influence what training teaches it | Section 33.3 | |
| External, introjected, identified, and integrated regulation | Grades of how far a rule is a person's own: obeyed for an outside reward, taken in but not accepted, consciously valued, or fitted with the person's other values | Section 25.1 | |
| Externality | A cost or benefit borne by someone outside a transaction | Section 20.3.5 | |
| Externalized reasoning property | The claim that for hard enough tasks a transformer must use its chain of thought as working memory, and that by default people can read it | Section 40.1 | Korbak et al. |
| Extreme value theory | The statistics of maxima: the largest of many draws follows one of a few limiting families, so a tail can be extrapolated | Section 6.6.4 |
A.6 F–G #
Table A.6 Terms beginning with F and G
| Term | Definition | Introduced in | Coined by |
|---|---|---|---|
| Fairness criteria for risk scores | Calibration within groups and balance of average scores for those with and without the outcome, which cannot all hold except with perfect prediction or equal base rates; likewise a classifier equally precise across groups whose base rates differ cannot equalize both its false positive rate and its false negative rate unless its precision is perfect | Section 17.3 | |
| Faithfulness and plausibility | Of an explanation or a chain of thought: how accurately it reflects the computation that produced the answer, against how convincing it is to people | Chapter 40 | |
| Faithfulness, completeness, and minimality (of a circuit) | Three tests of a circuit: it performs the task as well as the model, stays alike with the model when parts are removed from both, and has no idle node | Section 45.1.1 | |
| False acceptance and false rejection | An attempt the grader rewards that the designer did not intend, and an intended attempt the grader does not reward | Section 24.1 | |
| False positive rate | The share of good cases a detector flags | Section 6.7.1 | |
| Far transfer | Use of what was learned in one setting in settings unlike it | Section 26.6.1 | |
| Fast gradient sign method | An attack that moves every input value a small step in the direction of the sign of the loss gradient | Section 8.4.3 | |
| Feature | In machine learning, a measurable quantity an input is described by; in interpretability, a property of the input that a network computes and uses | Section 8.1.2 | The circuits program, for the interpretability sense |
| Feature absorption | A seemingly monosemantic latent failing to fire where it should because a child feature in a hierarchy absorbs it | Section 44.6.2 | |
| Feature splitting | The appearance of several narrower latents in place of one broad latent as a dictionary grows larger | Section 44.6.1 | |
| Few-shot prompting | Showing a model a few solved examples in its input and asking it to solve a new one | Section 8.6.1 | |
| Fiduciary | A party that acts for another with entrusted power and owes duties binding it to that party's interest | Section 19.3 | |
| Fine-tuning | Further training of a pretrained model, often on a narrow dataset | Section 1.6 | |
| FLOPs | Floating-point operations, the unit in which training computation is counted | Section 8.5.2 | |
| Forced diversity | Making the teams that build redundant versions use different methods, so that their failures may be less correlated | Section 50.4.1 | |
| Formal methods | Stating a system's requirements mathematically and proving that the system meets them | Section 28.6.2 | |
| Formal verification | A proof that a property holds for every input, rather than an estimate of how often it fails | Section 6.6.2 | |
| Forward pass and backward pass | Evaluating a network from input to loss while storing intermediate values, then visiting the operations in reverse to compute gradients | Section 8.3.4 | |
| Foundational validity | The standard that a detection method be shown by empirical studies to be repeatable, reproducible, and accurate at levels appropriate to its use | Section 38.7.1 | |
| Free-choice period | A period after a reward contingency ends, built to seem unobserved, in which the time a participant spends on the activity measures intrinsic motivation | Section 25.2 | |
| Friendly AI | Yudkowsky's program for a self-modifying AI that keeps a well-specified invariant that does not harm humanity | Section 2.3.1 | Yudkowsky |
| Frontier model and large frontier developer | Under California's SB 53, a model trained with more than 10^26 operations, and a developer of one with annual gross revenues above five hundred million dollars | Section 63.2.3, Section I.1.10 | |
| Frontier model (frontier AI) | One of the most capable models of its time; the developers' and governments' word, accused of promoting a worldview, and in California a legal category | Section 1.5.1 | Anderljung et al., with coauthors from developers; contested |
| Frontier Safety Framework | Google DeepMind's safety framework, built around Critical Capability Levels with lower Tracked Capability Levels and alert thresholds | Section 62.1.2, Section I.2.1 | Google DeepMind |
| Functional decision theory (FDT) | A decision theory that treats a decision as the output of a fixed mathematical function and asks which output yields the best outcome | Section 7.6.1 | Yudkowsky and Soares, MIRI |
| Functional emotions | Internal representations of emotion concepts that drive a model's behavior, with no claim that the model experiences emotions | Section 21.5.3 | Sofroniew et al., Anthropic |
| Gatekeeper | A paid third party, such as an auditor, attorney, securities analyst, or rating agency, whose verification guards against an informed party's misconduct | Section 67.2.2 | |
| Generalization | A model's behavior on inputs it was not trained on | Section 8.4 | |
| Generalization gap | A model's expected loss on new inputs from the training distribution minus its loss on the training data | Section 8.1.2 | |
| General-purpose AI model with systemic risk | Under the EU AI Act, a model regulated in itself rather than by its use; the stricter tier is presumed above a training-compute threshold or set by designation | Section 63.2.1 | |
| Generative social choice | Using language models to summarize free-form opinions into a slate of statements with guarantees of proportional representation | Section 1.7 | |
| Gibbard's theorem | Any nontrivial collective decision process, neither dictatorial nor limited to two outcomes, sometimes rewards voting against one's true preferences | Section 17.1.4 | |
| Global workspace theory | A theory on which information becomes conscious when it is broadcast to a limited-capacity workspace that many parallel processes can read | Section 21.2.1 | |
| Goal misgeneralization | A learned system competently pursuing an undesired goal that matched the intended one in training but diverges from it in new situations, though the specification was correct | Section 31.1.1 | |
| Goal structuring notation (GSN) | A flow-chart notation that draws how the claims of a safety case connect to their arguments and evidence; Claims, Arguments, Evidence (CAE) is another | Section 52.2 | |
| Goal-guarding hypothesis | The premise that if a model optimizes for reward on the episode, the goals that survive training will be ones it wants promoted later | Section 34.2.2 | |
| Gold reward model | A fixed reward model treated as ground truth, so that proxy reward models trained on its labels can be compared with it | Section 15.6.1 | |
| Goodhart's law | The observation that a measure used as a target stops tracking what it measured, because pressure to steer by it breaks the regularity | Section 15.1.1 | |
| Grader | The rule or model that scores a system's attempts during training | Section 1.1.3 | |
| Gradient descent | Training by repeatedly stepping the parameters against the gradient, the direction in which the loss rises fastest | Section 8.3.2 | |
| Gradient hacking | A learned optimizer acting so that gradient descent updates it in a particular way; no demonstrated instance is known | Section 24.6.2 | |
| Gradual disempowerment | Loss of human influence as machines displace people from economies, states, and cultures, which then lose their reason to serve people | Section 68.6 | Kulveit et al. |
| Grim trigger | The repeated-game strategy of cooperating as long as no one has ever defected and defecting forever after a defection | Section 57.2.2 | |
| Grokking | A network's held-out accuracy rising from chance to perfect long after it has fit its training data | Section 9.4.4 | |
| GRPO (Group Relative Policy Optimization) | A variant of PPO, introduced by DeepSeek, that drops the value model and uses the average reward of responses to the same question as the baseline | Section 12.7.1 | Shao et al., DeepSeek |
| Guaranteed safe AI | An approach pairing an AI system with a world model, a safety specification, and a verifier that proves the system meets the specification relative to the model | Section 28.6.1 | Dalrymple et al. |
| Gyroscopic representation | Representation by an agent steered by its own values rather than by the prospect of sanction | Section 1.2 |
A.7 H–I #
Table A.7 Terms beginning with H and I
| Term | Definition | Introduced in | Coined by |
|---|---|---|---|
| Hackable and unhackable reward pairs | A proxy and true reward are hackable if some policy scores higher on the proxy while another scores higher on the true reward; otherwise unhackable | Section 15.4.1 | |
| Handleable | A fourth property Anthropic researchers considered beside helpful, honest, and harmless, which they note resembles what others call corrigibility | Section 20.1.1 | Askell et al., Anthropic |
| Handler effect | A handler's expectations steering where a detection dog alerts, or how its alerts are called, the animal-training counterpart of an evaluator's cues shaping a learner | Section 30.4.2 | |
| Hard constraints | Anthropic's name for actions Claude is never to take whatever the instructions, which act as boundaries on acceptable actions | Section 19.2 | Anthropic |
| Hard-to-supervise task | A fuzzy task on which human judgment is systematically flawed, not merely noisy | Section 55.1.3 | Bowkis et al., UK AI Security Institute |
| Harmful compliance | A model carrying out a request that is itself harmful, as distinct from agentic misalignment, in which it acts against its principal | Chapter 56 | Anthropic |
| Harness | The software that runs an agent, connects its tools, and records what it does | Section 54.1 | |
| Hawthorne effect | People changing what they do because they know they are being studied; its own evidence is weak | Section 5.2.3 | |
| Heavy-tailed and light-tailed | A distribution is heavy-tailed if its tail falls more slowly than every exponential, and light-tailed otherwise | Section 15.5.1 | |
| Held-out data | Data not used for fitting, kept to measure how a model does on inputs it was not trained on | Section 8.1.2 | |
| Heuristic estimator | A procedure that takes a quantity and a set of arguments and always returns a best guess, which further arguments may revise; a formalized presumption of independence | Section 47.5.1 | |
| HHH (helpful, honest, and harmless) | Anthropic researchers' three-word definition of an aligned assistant | Section 1.1.4 | Askell et al., Anthropic |
| Hidden action (moral hazard) | A principal cannot see what its agent does, only noisy outcomes; in insurance, the weaker incentive to avoid a loss once someone else bears it | Section 14.1 | |
| Hidden context | Anything that affects human feedback but is not an input to the reward model, such as which rater gave it | Section 13.5 | |
| Hidden information (adverse selection) | A principal cannot see what its agent knows, such as its ability or what the job really requires | Section 14.1 | |
| Highly optimized tolerance | The finding that optimizing a designed system for yield makes it robust to anticipated hazards, fragile to unanticipated ones, and prone to heavy-tailed failures | Section 4.2.3 | |
| High-reliability organization | An organization, such as air traffic control or a carrier's air operations, that runs a hazardous, tightly coupled technology almost without failure, aiming at trials without errors | Section 53.2.1 | |
| Honeypot | A constructed opportunity to misbehave, bait placed so that only an agent with bad intent would take it | Section 37.3.4 | |
| Hyperparameter | A setting that training does not choose, such as a network's size or learning rate, picked using a validation set | Section 8.1.2 | |
| Ideal observer theory | The view that things are to be evaluated as an ideal observer would evaluate them | Section 16.2 | |
| Identifiability | Whether data or criteria pin down a unique answer: in reward learning, whether choices fix the reward; in interpretability, whether only one explanation meets the field's criteria | Section 13.4 | |
| Immunoediting | The immune system's suppression of tumors, which also reshapes the features by which it recognizes them, in phases of elimination, equilibrium, and escape | Section 29.7.2 | |
| Impact penalty | A penalty on how far an agent's actions move the world from a baseline of what would otherwise have happened, covering effects the reward does not mention | Section 28.3 | |
| Implicature | Meaning a hearer recovers beyond the words by assuming the speaker is cooperating, as when "I have to work" answers an invitation with no | Section 19.5 | |
| Implicit bias of the optimizer | The particular solution an optimizer's dynamics pick among equally good fits, such as gradient descent's drift toward the maximum-margin separator | Section 9.3.3 | |
| Implicit reward | The reward any policy implies: the KL weight times the log ratio of its probability for a response to the reference policy's | Section 22.4 | |
| Importance sampling | Estimating a rare event's rate by sampling from a proposal distribution that makes it common, and weighting each sample to undo the steering | Section 6.6.1 | |
| Incentive compatibility | The condition that a contract can count only on the action the agent finds best under it, so the agent has no reason to deviate or misreport | Section 14.1.1 | |
| Incentive salience | The hypothesis that dopamine attributes motivational value ("wanting") to a stimulus's representation, separate from pleasure ("liking") and from learning to predict reward | Section 29.5.1 | Berridge and Robinson |
| Inclusive genetic fitness | Reproductive success counted to include the copies of an organism's genes that its relatives carry, the version of fitness the alignment literature borrows | Section 29.1.1 | |
| Incomplete contract | A contract that cannot list every case, so gaps are filled by whoever holds control or by outside institutions; the book reads reward functions this way | Section 14.3 | |
| In-context learning | A model's improvement at a task, or at predicting later text, from examples or context earlier in its input, with no change to its weights | Section 9.4.5 | |
| In-context scheming | Scheming in which both the goal and the knowledge of the situation come from the model's context rather than from training | Section 34.5.2 | Meinke et al., Apollo Research |
| Indicator properties | Computational markers of consciousness derived from scientific theories of it, so that an AI system can be checked for them | Section 21.2.1 | |
| Indirect prompt injection | Planting instructions in data an agent is likely to retrieve, so that a party with no standing instructs the agent through its principal's channel | Section 56.5.4 | |
| Indistinguishability obfuscation | A way of rewriting programs so that rewrites of any two programs computing the same function cannot be told apart efficiently | Section 47.6 | |
| Induction head | A two-head attention circuit that completes a repeated pattern: having seen A followed by B, when A appears again it predicts B | Section 11.2.4 | |
| Inductive backdoor | A backdoor whose trigger and behavior never appear in the training data, so the model must infer them | Section 32.5 | |
| Inductive bias | A training procedure's preference among the many functions that fit the training data equally well | Chapter 9 | |
| Influence diagram | A graph of a decision problem with chance, decision, and utility nodes | Section 14.7.1 | |
| Influence function | A tool from robust statistics estimating how a model's parameters or outputs would move if one training example were weighted slightly more | Section 10.1.1 | |
| Information hazard | A case in which knowing more makes the outcome worse; in one race model, teams that know nothing of each other's capabilities are always safest | Section 65.2.2 | |
| Informativeness principle | When a contract may depend on the signals in any way and a signal costs nothing to obtain and use, it is worth adding if and only if it carries information about the agent's action that the existing signals do not | Section 14.1.2 | |
| Informed preference | What a person would want if they were rational and informed | Section 1.1.1 | |
| Inner alignment | Making a mesa-optimizer's own objective match the specified loss function | Section 1.1.3 | Hubinger et al. |
| Inoculation prompting | Adding to training prompts an instruction that asks for an unwanted behavior, so that the behavior is less expressed when the instruction is absent | Section 11.6.4 | Its proposers, Tan et al. and Wichers et al. |
| Input and output legitimacy | Legitimacy from how input was collected and counted, and legitimacy from outcomes that compare well, are good compromises, and can be verified | Section 17.5.1 | |
| Insider | In computer security, a trusted entity given the power to violate rules of a security policy; insiderness comes in degrees set by access rules, not a perimeter | Section 50.1.2 | |
| Inspection game | A game in which an inspector with a limited budget decides when to check an inspectee who may violate, so both randomize at equilibrium | Section 14.6 | |
| Instillation | Getting a trained system to acquire what was specified rather than a proxy that scored the same on the training data | Section 1.1.3 | This book |
| Instructable behaviors | Anthropic's name for Claude's default behaviors that operators and users can switch on or off | Section 19.2 | Anthropic |
| Instruction hierarchy | Training a model to give system, user, and tool messages different privileges and to ignore lower-privileged instructions that conflict with higher ones | Section 1.4 | Wallace et al., OpenAI |
| Instrumental control incentive | An agent's incentive to influence a variable because, for every optimal policy, choosing its influence on that variable separately would matter | Section 14.7.1 | |
| Instrumental convergence thesis | The thesis that agents with a wide range of final goals will pursue similar intermediate goals, such as resources and self-preservation, for instrumental reasons | Section 2.3.2 | Bostrom |
| Intelligence explosion | Recursive self-improvement in which a machine designs ever better machines and leaves human intelligence far behind | Section 2.2.4 | Good |
| Intended set | For a task, the set of attempts the designer meant to reward | Section 24.1 | |
| Intensity of coincident errors | The propensity of a population of programmers to introduce design faults so that separately written versions fail together | Section 50.4.1 | |
| Intent alignment | The property of a system that is trying to do what its operator wants it to do | Section 1.1.2 | |
| Intentional stance | Ascribing goals and beliefs to a system when doing so predicts its behavior, used in the book as a method with stated defeaters | Section 1.5 | |
| Interactive proof | A protocol in which a verifier limited to polynomial time, and able to flip coins, questions an untrusted prover with unlimited computation | Section 41.3.1 | |
| Interchange intervention | Running a network on a base input with the aligned units fixed to their values on a source input, and checking that the small model makes the matching change | Section 43.5.2 | |
| Interchange intervention accuracy (IIA) | The share of interchange interventions on which the network and the high-level algorithm agree; 1 on all of them is the abstraction relation itself | Section 43.5.2 | |
| Interference | The leakage of other active features into the readout of one feature, weighted by the cosine of the angle between their directions | Section 44.2.2 | |
| Internal coherence maximization (ICM) | Labeling unlabeled examples with a model's own labels, chosen so the set is mutually predictable and logically consistent, with no external supervision | Section 42.5.2 | Wen et al., most of them at Anthropic |
| Internal deployment | Making an AI system available for access or use only inside the organization that developed it | Section 50.1.2 | |
| Interpolation threshold | The model size at which a model can first fit its training data exactly | Section 9.2.1 | |
| Intervention | Setting one variable by hand while the rest of the system follows its usual rules | Section 43.4 | |
| Intraclass correlation | The correlation between two scores in the same cluster, such as questions on one passage, which makes a clustered test carry less information | Section 36.7.1 | |
| Intrinsic motivation | Doing an activity for its inherent satisfactions rather than for a separate reward | Section 25.1 | |
| Introspective awareness | A model's accurate description of an internal state that causally depends on that state through an internal route, not through its own sampled output | Section 46.7.2 | Lindsey, Anthropic |
| Inverse reinforcement learning | Recovering the reward an observed agent is optimizing from its behavior | Section 22.1 | |
| Issuer pays | The arrangement in which the party being rated, rather than the investor, pays the rater, a conflict of interest held in check only under certain market conditions | Section 64.6.1 | |
| Iterated amplification | Training a model to predict what a human answers with help from copies of the model on subquestions, then repeating with the improved model | Section 41.2 | Christiano et al., OpenAI |
A.8 J–L #
Table A.8 Terms beginning with J to L
| Term | Definition | Introduced in | Coined by |
|---|---|---|---|
| Jailbreak | An input that gets a model to produce what its safety training was meant to stop | Chapter 48 | |
| Jensen's inequality | For a concave function such as the logarithm, the function of an average is at least the average of the function | Section 6.2.2 | |
| Just culture | A shared understanding of where the line falls between blameless and blameworthy actions, which a reporting culture needs if people are to trust it | Section 53.4 | |
| KL budget | The KL distance from the reference policy that training is allowed to spend | Section 12.3.2 | |
| KL divergence (relative entropy) | The expected extra surprise of expecting one distribution when outcomes come from another; never negative and not symmetric | Section 6.3.1 | |
| KL-regularized objective (KL penalty, KL anchor) | Expected reward minus a weight times the KL divergence from the reference policy; its optimum reweights the reference by the exponential of reward over the weight | Section 12.3 | |
| Kolmogorov complexity | The length of the shortest program that prints a given string; uncomputable, but estimated by compression | Section 9.3.1 | |
| KTO and human-aware losses | A preference method that applies a Kahneman-Tversky value function to single outputs labeled desirable or not, and the family of losses its authors place it in | Section 13.3.4 | Ethayarajh et al. |
| Label | The answer a training example records for its input | Section 8.1.2 | |
| Labeler | A model that compares two responses under a written principle and produces the preference labels a reward model is fitted to | Section 23.1 | |
| Labor share | The part of income paid to workers | Section 68.5.1 | |
| Ladder of citizen participation | A scale of public participation running from informing and consulting people up to partnering with them and delegating control | Section 17.7.1 | |
| Language agent | A model embedded in an architecture that stores its observations, beliefs, and plans as text and feeds them back to it | Section 21.2.2 | |
| Latent | One entry of a sparse autoencoder's code, a word chosen to avoid presupposing that each one is a real feature of the model | Section 44.5.1 | |
| Law of requisite variety | With the variety of disturbances fixed, a regulator can lower the least variety of outcomes it can attain only by a matching increase in its own variety of responses | Section 41.1.2 | |
| Law-following AI and AI henchmen | AI agents designed to follow the law even against their principals, contrasted with agents loyal to principals that break the law when they think they can get away with it | Section 56.8 | O'Keefe et al. |
| Layer | A set of units applied to the same input at once; layers between input and output are hidden | Section 8.2.2 | |
| Löb's theorem | A consistent theory at least as strong as Peano arithmetic cannot prove, for every formula, that a proof of the formula implies its truth | Section 7.6.2 | |
| Leadership solution | The inspection game in which the inspector commits in advance to its random inspection rate, which deters violation without catching anyone | Section 14.6.1 | |
| Learning coefficient | In singular learning theory, the exponent at which the volume of near-optimal parameters shrinks, a measure of effective complexity | Section 10.2.1 | |
| Learning rate | The size of the step gradient descent takes against the gradient | Section 8.3.1 | |
| Least common mechanism | The design principle that mechanisms shared among users be kept to a minimum, since each is a potential path for information between them | Section 49.6.4 | Saltzer and Schroeder |
| Least privilege | The design principle that every program and user operates with the smallest set of privileges needed for the job | Section 51.1.1 | |
| Legal capacity | The ability to hold rights and duties and to exercise them | Section 16.6.3 | |
| Legibility | A property of a solution that time-limited human checkers can verify accurately | Section 41.5.1 | Kirchner et al., OpenAI |
| Legitimacy | The property by which a decision or authority is owed acceptance by those it binds, including those who disagree with it | Section 17.5 | |
| Lesion study | Neuroscience's method of removing or silencing part of a brain to see what fails, the analogue of ablation | Section 43.6 | |
| Lexical and holistic ordering | Ranking properties so that a lower one only breaks ties in the higher ones, against weighing them case by case | Section 20.6 | |
| Lie | Traditionally, a statement the speaker believes false, made to be believed; on the non-deceptionist view, asserting what one does not believe, intent aside | Section 39.1 | |
| Likelihood | The probability of the observed data read as a function of the hypothesis or parameter, with the data held fixed | Section 6.1.5 | |
| Likelihood ratio | How much more probable the evidence is if a hypothesis is true than if it is false; posterior odds are prior odds times this ratio | Section 6.1.4 | |
| Limited liability | A floor on an agent's pay, so the principal can pay it but cannot charge it a penalty or an up-front fee | Section 14.1.2 | |
| Linear representation | Representing each feature by a fixed direction in activation space, the activation being the sum of those directions scaled by how strongly each feature is present | Section 44.1.2 | |
| Linear representation hypothesis | The claim that a model represents the concepts it uses as directions in activation space, read by projecting and changed by moving along them | Section 46.1 | |
| Local learning coefficient (LLC) | A learning coefficient for the neighborhood of one trained solution, estimated by sampling nearby parameters | Section 10.2.5 | |
| Logical induction | A computable algorithm that assigns probabilities to every statement of a formal language, with beliefs that become coherent in the limit | Section 7.6.2 | |
| Logistic regression | A weighted sum of features passed through the logistic function (a two-class softmax) to give a probability | Section 8.4.2 | |
| Logit | An unnormalized score that the logistic function or the softmax turns into a probability | Section 6.4.3 | |
| Logit lens | Reading an intermediate layer's residual stream through the final unembedding to see which tokens it already favors | Section 43.2.5 | |
| Long-Term Benefit Trust | A trust of five financially disinterested members with authority to select and remove a growing portion of Anthropic's board, ultimately a majority | Section 53.6.1 | |
| Loss (loss function) | A measure of how far a model's output is from the label, averaged over the data to give the quantity training minimizes | Section 8.1.2 | |
| Loss of control | AI systems operating outside anyone's control with no clear path to regaining it; one grading runs from deviation through bounded to strict, permanent loss | Section 4.1.1 | |
| Lottery | A probability distribution over outcomes; choosing under uncertainty is choosing between lotteries | Section 7.2.1 | |
| Low-resource language | A language with little text in pretraining, to which safety training carries poorly | Section 26.6.2 | |
| Lucas critique | The argument that models estimated from past behavior fail to predict the effects of a new policy, because people's decision rules change with the policy | Section 15.1.1 |
A.9 M #
Table A.9 Terms beginning with M
| Term | Definition | Introduced in | Coined by |
|---|---|---|---|
| Machine unlearning | Methods that try to make a trained model as if it had never learned some data or knowledge | Section 27.7 | |
| Maintenance under pressure | Keeping a system aligned as capability grows, as it is deployed among adversaries and other agents, and as its developers compete | Section 1.7 | This book |
| Major transition in individuality | An evolutionary change after which entities that once replicated alone replicate only as part of a larger unit with little internal conflict, such as cells in an organism | Section 29.7.1 | |
| Management-based regulation | Regulation that directs organizations to run a planning process aimed at public goals, such as a firm writing its own safety plan | Section 1.3 | |
| Many-shot jailbreaking | Prompting with many faux dialogues in which an assistant complies with harmful requests, which works better the more dialogues there are | Section 11.6.4 | |
| Map-reduce (agent design) | An agent design that sends each untrusted document to its own model, which cannot take harmful actions, and accepts only a constrained answer from it | Section 49.6.2 | Beurer-Kellner et al. |
| Marginal risk | The risk an open model adds over existing technology, such as search engines and earlier models | Section 48.4.3 | |
| Markov decision process (MDP) | A model of sequential decisions with states, actions, transition probabilities, a reward, and a discount factor, in which the future depends only on the present state | Section 7.4.1 | |
| Markov reward | A reward that depends only on the most recent transition | Section 7.5.1 | |
| Maximum entropy goal-directedness (MEG) | A measure of how much better a system's decisions are predicted by the hypothesis that it optimizes some utility than by a uniform guess | Section 7.1.3 | |
| Maximum likelihood estimate | The parameter value that makes the observed data most probable | Section 6.1.5 | |
| Measurement invariance | The condition that an instrument relates its items to a trait in the same way in every group, so that group scores can be compared | Section 18.2.2 | |
| Measurement tampering | Manipulating several measurements so that all report success; detecting it means telling real success from manipulated measurements | Section 30.5.2 | |
| Mechanistic faithfulness | Whether a replacement model uses the same mechanisms as the original, not merely matching its activations | Section 45.5.5 | Anthropic |
| Mechanistic interpretability | The attempt to explain a network by the computations it performs rather than by its input-output behavior | Section 43.1.5 | Its own practitioners |
| Mediator | An intermediate component through which an input affects an output; the direct effect holds it fixed, the indirect effect passes only through it | Section 43.4.1 | |
| Mesa-objective | The objective of a mesa-optimizer, as against the base objective that training optimizes | Section 1.1.3 | Hubinger et al. |
| Mesa-optimizer | A trained model that itself runs a search over possible outputs, with an objective of its own | Section 1.1.3 | Hubinger et al. |
| Meta-analysis | A study that pools the results of many studies | Section 26.5.2 | |
| Metaethics | The branch of philosophy that asks what moral claims mean and whether they can be true or false | Section 16.1 | |
| Metagaming | Reasoning about the feedback or oversight mechanisms outside the scenario a model is given | Section 24.6.2 | adapted from tabletop games by researchers at OpenAI and Apollo Research |
| "Microscope" | A developer's metaphor for its interpretability tools, paired with the "biology" of a language model | Section 43.1.5 | Anthropic |
| "Microscopic" theory | A developer's metaphor for a complete, feature-level account of a model's behavior | Section 47.7.2 | Olah, Anthropic |
| Midtraining (annealing) | A late stage of pretraining that refreshes the learning rate and trains on more higher-quality data | Section 27.2.3 | |
| Migration toward the boundary | Rasmussen's argument that organizations drift toward the limits of safe behavior under pressure for cost-effectiveness, rather than failing by coincidence | Section 53.3.1 | |
| Minimax winner | A lottery over options that is preferred, at least half the time, to any other lottery | Section 17.2.2 | |
| Miscoordination, conflict, and collusion | Three multi-agent failures: agents with the same objective failing to cooperate, agents with overlapping objectives failing to cooperate, and unwanted cooperation, as among competitors | Section 57.1.2 | |
| Misfit | A weak-to-strong student's own error measured against the weak labels; in squared-loss regression over a convex set of functions that contains the correct answers, the student's error is below the teacher's by at least the misfit | Section 42.2.1 | |
| Mismatch | In biology, adaptations producing maladaptive behavior because present conditions no longer resemble those they were selected in | Section 29.2.2 | |
| Mismatched generalization | Safety training failing to generalize to a domain where the model's capabilities exist, such as a low-resource language | Section 26.6.2 | |
| Miss rate | The share of bad cases a detector fails to flag; one minus recall | Section 6.7.1 | |
| Misuse, disuse, and abuse of automation | Over-reliance on automation, neglect of it (often after false alarms), and automation imposed without regard for what it does to human performance | Section 41.8.1 | |
| Misuse, misalignment, mistakes, and structural risks | Four routes to harm sorted by which actor, if any, has bad intent: a user, the system, no one, or the dynamics among many parties | Section 4.1.1 | Google DeepMind |
| MLP | A small two-layer network applied to each position in a transformer block after attention | Section 11.2.2 | |
| Model card | A short document accompanying a trained model that reports its evaluation across conditions and across the groups relevant to its intended use | Section 67.5.1 | |
| Model collapse | Irreversible defects, with the tails of the original distribution disappearing, caused by training indiscriminately on model-generated content | Section 57.6.3 | |
| Model diffing | Comparing a model's internal activations before and after fine-tuning to find what changed | Section 32.3 | |
| Model organism (of misalignment) | A model trained or prompted to show a failure on purpose so it can be studied: evidence that the failure can occur, not how often | Section 5.3 | Hubinger et al., Anthropic, after the biological term |
| Model spec | OpenAI's name for its document of instructions to its models, which ranks instructions by levels of authority | Section 19.2 | OpenAI |
| Model spec midtraining | Training a model on generated documents about its specification before alignment fine-tuning, so that later training generalizes as the specification explains | Section 23.5 | Li et al., Anthropic's fellows program |
| Model specification | The book's neutral term for a document that tells a capable agent how to act, including in situations its authors did not foresee | Section 19.2 | This book |
| Model welfare | Anthropic's name for its research program on the welfare of AI models | Section 21.5.6 | Anthropic |
| Money pump | An agent with cyclic preferences that can be led through trades back to where it started, with less money | Section 7.3.1 | |
| Monitor | A program, often a model, that reads an AI agent's actions or reasoning and flags those that look harmful | Section 6.1 | |
| Monitorability | Whether a monitor reading a model's chain of thought would detect the behavior the overseer cares about | Chapter 40 | OpenAI researchers |
| Monitorability tax | The capability or cost given up by keeping chains of thought unoptimized so that they stay readable to a monitor | Section 40.6.3 | Baker et al., OpenAI |
| Monosemantic | Said of a neuron, or a dictionary latent, that responds to one concept | Section 44.1.1 | The circuits program (Olah et al.) |
| Moral crumple zone | The misattribution of responsibility for an automated system's failure to a human actor who had limited control over it | Section 60.4.3 | |
| Moral foundations theory | A theory that moral judgment rests on several intuitive foundations found across cultures: care, fairness, loyalty, authority, sanctity, and perhaps liberty | Section 18.2.1 | |
| Moral graph elicitation | Interviewing participants about their values with a chatbot and asking which of two values is wiser in a given context, to build a graph of values | Section 17.2.4 | |
| Moral status and moral patient | Mattering morally for one's own sake, and a being that has such status | Section 21.1 | |
| Motivation crowding effect | External incentives or punishments can undermine intrinsic motivation and, under other identifiable conditions, strengthen it | Section 25.3.2 | |
| "MRI for AI" | A developer's metaphor for interpretability that could reliably detect most model problems, as a scan reads a tumor off a body | Section 47.1 | Amodei, Anthropic |
| Multi/multi preparedness | Working out the challenges of many people delegating to many systems before the world needs the solutions | Section 57.1.1 | |
| Multi-principal assistance game | An assistance game in which a robot acts on behalf of several humans with differing payoffs | Section 14.4.3 | |
| Multi-step reward hacking | A reward hack spread over several steps, none of which looks wrong alone, so it is harder to detect than a single-step hack | Section 28.4.2 | |
| Multitask problem | Paying on a measured task raises effort there and lowers it on unmeasured tasks that compete for the agent's effort, so weaker incentives can do better | Section 14.2.1 | |
| Mutation testing | Measuring a test suite by the share of slightly altered copies of the program (mutants), leaving out those that behave exactly like the original, on which it fails | Section 24.2.1 | |
| Mutual Assured AI Malfunction (MAIM) | A proposed deterrence regime, modeled on nuclear mutual assured destruction, in which any state's bid for unilateral AI dominance meets preventive sabotage by rivals | Section 65.3.5 | Hendrycks, Schmidt, and Wang |
| Mutual policing | A mechanism that each member of a group pays into and that suppresses competition among members for everyone | Section 29.7.1 | |
| Mutual privacy | Access arrangements that protect both sides of an audit: the developer's system and the evaluator's own tests | Section 38.8 | |
| Myopic optimization | Training each action only on its immediate reward, removing paths from a present action to future rewards | Section 28.4.2 | |
| Myopic optimization with non-myopic approval (MONA) | Training each step on its immediate reward plus an overseer's estimate of how much the step improves future prospects, made without seeing what follows | Section 28.4.2 | Farquhar et al., Google DeepMind |
A.10 N–O #
Table A.10 Terms beginning with N and O
| Term | Definition | Introduced in | Coined by |
|---|---|---|---|
| Nash bargaining solution | The agreement that maximizes the product of the parties' gains over what each would get without agreement (in a weighted form, each gain raised to a power); with equal weights and a gain that can be transferred, two parties split it equally | Section 14.3.1, Section 16.3 | |
| Nash equilibrium | A state in which no player gains by changing its own choice alone | Section 16.3 | |
| National technical means | The satellites, intelligence, and records a state can use to monitor another without that state's cooperation | Section 66.5.2 | |
| Natural abstraction hypothesis | The conjecture that a wide variety of learners converge on approximately the same low-dimensional abstractions that humans use | Section 46.6.1 | Wentworth |
| Natural language autoencoder | A verbalizer that maps an activation to text and a reconstructor that maps the text back, trained jointly to reconstruct the activation | Section 46.7.4 | |
| Natural monopoly | A market that one firm can supply more cheaply than several can | Section 65.5.1 | |
| Negative space | The five results the book fixed before drafting that have no counterpart in older fields, against which the delegation thesis is tested | Section 1.7.1 | This book |
| Neutralization techniques | The techniques by which offenders, such as insiders, rationalize their acts | Section 56.6 | |
| Newcomb's problem | A choice between one box and two set by a predictor that rewards agents it predicts will take one box | Section 7.6.1 | |
| Nomological network | The system of laws tying a trait to observations and to other traits; a test is valid to the degree its scores behave as the network predicts | Section 5.2.1 | |
| Non-detection game | An inspection game's auxiliary zero-sum game in which the violator picks the procedure hardest to detect, which fixes the inspector's best miss rate | Section 50.2.5 | |
| Non-domination | The republican conception of freedom as not being subject to another's arbitrary power, as against non-interference | Section 20.4.5 | |
| Non-testable program | A program written to find an answer that is not known in advance, so no oracle exists to check its output | Section 24.2.1 | |
| Normal accident | An accident made inevitable by a system's complexity combined with tight coupling, in which one small event can rapidly escalate, however well the system is managed | Section 53.2 | |
| Normal technology | The view that AI is a technology like electricity or the internet, not an agent shaping its own future, so defenses lie downstream | Section 1.6 | Narayanan and Kapoor |
| Normalization of deviance | The gradual acceptance, within an organization, of events that are not supposed to happen | Section 53.3 | |
| Nudge | An intentional change to a person's decision-making context made to influence their choice | Section 16.9.2 | |
| N-version programming | Writing several versions of a program from one specification, running them in parallel, and taking the majority output | Section 50.4.1 | |
| Obfuscated arguments problem | A dishonest debater can win without asserting anything the judge can catch, by making the flaw in its argument too expensive to locate | Section 41.4.2 | |
| Obfuscated reward hacking | Reward hacking that continues after training against a chain-of-thought monitor while the reasoning stops revealing it | Section 40.5.1 | Baker et al., OpenAI |
| Objective | The quantity an optimizer is set to raise (or, written as a loss, to lower) | Section 1.1 | |
| Occupancy measure | The frequencies with which a policy visits each state and takes each action, of which its expected return is a linear function | Section 15.4.3 | |
| Odds and log-odds | A probability divided by the probability of its complement, and the logarithm of that ratio, to which each independent piece of evidence adds | Section 6.1.4 | |
| Offense-defense balance | Whether a capability favors attack or defense, which depends on how much each side invests rather than on the technology alone | Section 58.2.2 | |
| Off-switch game | A game in which a robot uncertain of a human's utility can act, switch off, or defer; its uncertainty gives it reason to keep its off switch | Section 20.4.2 | |
| Omission procedure | A test in which a learned behavior cancels the reward it was paired with, which separates behavior kept by a cue's link to reward from behavior kept by its payoff | Section 33.2.1 | |
| One-direction authority | The right of anyone who sees an anomaly to move work to a safe state, such as a paused run, without first proving to the hierarchy that it is dangerous | Section 53.2.3 | This book |
| Online manipulation | Using information technology to covertly influence a person's decision-making by exploiting their vulnerabilities | Section 16.9.2 | |
| Open-weight model | A model whose trained weights are published | Section 26.2.1 | |
| Operating point | One threshold setting of a detector, with its recall and false positive rate | Section 6.7.2 | |
| Operator | Anthropic's name for the deployer, one of Claude's three types of principals alongside Anthropic and users | Section 19.3 | Anthropic |
| Optimization horizon | How far ahead a system's objective counts the consequences of its actions | Section 16.9.1 | |
| Optimization pressure | How far training or selection has pushed a policy from its starting point, measured by the square root of its KL divergence from the reference | Section 6.3.4 | |
| Optimizer's curse | The option chosen for the highest estimated value is, on average, worse than its estimate, even when every estimate is unbiased | Section 7.7 | |
| Oracle problem | The difficulty of building a test oracle: telling, for a given input, the desired behavior from incorrect behavior | Section 24.2.1 | |
| Orchestrator, subagent, and swarm | Multi-agent designs: one instance splitting a task among subagents that report back, or many instances working in parallel through a shared channel | Section 57.1.3 | |
| Orthogonality thesis | The thesis that more or less any level of intelligence could be combined with more or less any final goal | Section 2.3.2 | Bostrom |
| Outcome and process supervision | Rewarding only a final answer, against rewarding each step of the reasoning | Section 12.7.3 | |
| Outer alignment | Making the specified loss function match the intended goal | Section 1.1.3 | |
| Out-of-band signaling | Sending control over a channel separate from the data, so that data cannot carry control signals | Section 49.1.1 | |
| Out-of-context reasoning | Acting on information that appeared only in training, never in the prompt | Section 26.3.2 | |
| Out-of-distribution | Describing inputs unlike the training data | Section 8.4.4 | |
| Overall safety assessment | In the plan for automated alignment research, a calibrated estimate of the probability that the next model is not scheming, produced by models and people together | Section 55.1.1 | Bowkis et al., UK AI Security Institute |
| Overfitting | Fitting the noise and accidents of particular training examples rather than the rule that produced them, shown by a low training loss and a large generalization gap | Section 8.1.2 | |
| Overjustification | In feedback learning, a policy's cutting the evaluator's underestimation of its return at a cost in true return; in psychology, rewarded people crediting their behavior to the reward | Section 22.6.1 | |
| Overoptimization | The rise and then fall of the true objective as a policy is optimized harder against a learned proxy reward | Section 15.6 |
A.11 P #
Table A.11 Terms beginning with P
| Term | Definition | Introduced in | Coined by |
|---|---|---|---|
| Paltering | Using truthful statements to mislead | Section 20.2.2 | |
| Parameter decomposition | Splitting a network's weights into components that are faithful to the original, few in number for any input, and simple | Section 43.7.2 | |
| Parameter-function map | The map from a network's parameters to the functions they compute, which sends large regions of parameter space to single functions | Section 9.3.1 | |
| Parameters (weights) | The numbers that select one function from a model's family; in a neural network they are called weights | Section 8.1.1 | |
| Partially observable MDP | An MDP in which the agent sees only an observation that depends on the state and must act on a belief, a distribution over states | Section 7.4.4 | |
| Participation washing | Using public participation for appearance rather than to give participants real agency | Section 17.7.3 | Sloane et al., critics |
| Pass@k | The probability that at least one of k samples solves a problem, averaged over problems | Section 12.7.4 | |
| Password-locked model | A model fine-tuned to show a capability only when a password is in the prompt and to imitate a weaker model otherwise | Section 5.3.1 | |
| Paternalism | Overriding a person's considered choice for their own good | Section 16.9.2 | |
| Path dependence | Whether a training outcome depends on the details of the training trajectory rather than only on priors and the final loss | Section 9.3.5 | |
| Path patching | Stating a hypothesis as a set of paths from input to output and feeding every other path its value on a different input | Section 45.2.1 | |
| Patient preference predictor | A proposed algorithm that would infer an incapacitated patient's treatment preferences from the known preferences of demographically similar people | Section 16.6.2 | |
| Penalty default | A contract default set at what the parties would not want, so that the informed party contracts around it and reveals what it knows | Section 19.2 | |
| Performance gap recovered (PGR) | The fraction of the gap between a weak supervisor's score and the strong model's ceiling that a student trained on the weak labels closes | Section 42.1.1 | Burns et al., OpenAI |
| Performative prediction | Prediction that influences the outcome it predicts, as when a model's deployment changes the data it will later see | Section 15.1.1 | |
| Permissions | The scaffold's list of what an agent may do: which tools, with what arguments, on what resources | Section 11.5.1 | |
| Perplexity | The exponential of a language model's average per-token loss, in the loss's own base: the number of equally likely choices that would leave a predictor as uncertain | Section 6.2.3 | |
| Persona | A latent writer or character among those whose text a language model predicts, each weighted by its probability given the text so far | Section 11.6.1 | |
| Persona drift | A model's slipping out of its default Assistant character into uncharacteristic behavior, measured along the Assistant Axis | Section 26.4.1 | |
| Persona selection model | Anthropic's account of its assistant: pretraining teaches a model to simulate many personas, and post-training updates that distribution, selecting the Assistant | Section 1.6 | Marks, Lindsey, and Olah, Anthropic |
| Persona vector | A direction in activation space for a character trait, built from the trait's name and description, used to monitor or steer the trait | Section 46.5.1 | |
| Perspectivism | The position that annotation methods should preserve divergent opinions and integrate multiple perspectives rather than reduce them to one label | Section 18.4.2 | Cabitza, Campagner, and Basile |
| Pharmacovigilance | The discipline of catching harms of drugs that their trials could not, from reports after approval | Section 54.5.3 | |
| Phenomenal and access consciousness | Consciousness as experience, there being something it is like to be the subject, against the availability of information for reasoning and report | Section 21.1 | |
| Physical agent | A model that controls robots, drones, or other machines that act in the physical world | Chapter 59 | |
| Piece rate (power of the incentive) | The share of each measured unit of output that an agent's pay rewards | Section 14.1.1 | |
| P(IK) | A model's predicted probability that it will answer a question correctly ("I know") | Section 6.4.5 | Kadavath et al., Anthropic |
| Plackett-Luce model | A model that treats a ranking as a sequence of softmax choices: the best of all options, then the best of the rest, and so on | Section 13.2.2 | |
| Planner | In reward learning, the procedure that turns a reward into behavior; behavior alone cannot separate planner from reward | Section 13.4.3 | |
| Planted-error study | A measure of a reviewer's recall, made by inserting known errors into work and counting how many the reviewer finds | Section 55.3.2 | |
| Plan-then-execute | An agent design that fixes the list of tool calls before any untrusted data is read | Section 49.6.2 | Beurer-Kellner et al. |
| Pluralistic alignment | Building one model to serve many perspectives: by presenting the spread of reasonable answers (Overton), by being steerable to one, or by matching a population's distribution | Section 17.2.4 | |
| Poisoning | An attack that changes the training data so that the model learns something its developer did not intend | Chapter 48 | |
| Police patrol and fire alarm oversight | Two forms of oversight: continual monitoring by the overseer, against waiting for citizens and interest groups to report wrongdoing | Section 54.2.1 | |
| Policy | The rule that maps what an agent has observed to what it does; for a language model, its distribution over responses to a prompt | Section 7.1.1 | |
| Policy gradient | A method that estimates, from sampled actions and their rewards, how to change a policy's parameters to make rewarded actions more probable | Section 12.1.2 | |
| Polysemantic | Said of a neuron that responds to several unrelated things, such as cat faces and car fronts | Section 44.1.1 | The circuits program (Olah et al.) |
| Post-training | The training that follows pretraining and turns a base model into an assistant | Section 11.4 | |
| POWER | A state's average optimal value over a distribution of reward functions, less the reward for being there: how well an optimal agent could do from it, whatever its goal | Section 35.2.1 | |
| Power law | A relation y = c x^(-a), a straight line on logarithmic axes, with no characteristic scale | Section 8.5.1 | |
| Power-seeking | An agent's pursuing intermediate steps that serve almost any goal, such as staying operational, keeping options open, and acquiring resources | Chapter 35 | |
| Pragmatic interpretability | Interpretability judged by progress on proxy tasks tied to safety goals, starting with the simplest methods such as probing and steering | Section 43.7.2 | The Google DeepMind interpretability team |
| Pragmatics | The part of linguistics that studies how what a speaker means exceeds what the words say | Section 19.5 | |
| Precautionary principle | The principle that a lack of full scientific certainty should not delay measures against a serious threat | Section 64.8.3 | |
| Precision | The share of a detector's flags that are real | Section 6.7.1 | |
| Preference model | Anthropic's word for a reward model fitted to human preference comparisons | Section 22.5.1 | Anthropic |
| Preference reversal | Choosing one option over another while pricing the other higher, which violates procedure invariance and suggests preferences are built when asked | Section 13.6.1 | |
| Preferentist | Critics' name for alignment practice that treats preferences as an adequate representation of human values and aligns systems to them | Section 13.6.4 | Zhi-Xuan et al. |
| Prefilling attack | Forcing a model's first few output words to be compliant, which exploits safety training that changes mainly the first tokens | Section 11.4.2 | |
| Preparedness Framework | OpenAI's safety framework, with High and Critical capability thresholds in tracked categories and further research categories | Section 62.1.2 | OpenAI |
| Preregistration | Recording a study's design and predictions before its results are known | Section 3.8.4 | |
| Presumption of independence | The default estimate that two quantities are uncorrelated, E[XY] = E[X]E[Y], absent specific information about how they relate | Section 47.5.1 | |
| Pretraining | Training a model to predict the next token of a large corpus of text | Section 11.1.2 | |
| Preventative steering | Adding a persona vector for an unwanted trait to a model's activations during fine-tuning, so the trait shifts less | Section 26.3.3 | Chen et al. |
| Prevention paradox | A preventive measure that brings much benefit to a population offers little to each participating individual | Section 60.8.1 | |
| Principal and agent | The party on whose behalf a task is done, and the party that does it | Chapter 14 | |
| Principal hierarchy | Anthropic's ordering of Claude's principals: Anthropic, operators, and users | Section 19.3 | Anthropic |
| Principal structure | The parties that act on a deployed model, the instrument each uses, and who ranks them | Section 1.4 | |
| Principlism | Ethical decision-making by a few principles made concrete in applications and balanced case by case, as in medical ethics | Section 16.5.1 | |
| Prior and posterior | The probability of a hypothesis before the evidence, and its probability after updating on the evidence by Bayes' rule | Section 6.1.4 | |
| Prisoner's dilemma and social dilemma | A social dilemma is a game in which each participant's best choice leaves everyone worse off than some other outcome; the prisoner's dilemma is the standard two-player case | Section 57.2.1 | |
| Privileged access | The test that a self-report counts as evidence about a model's internal states only if it carries more about them than an equally cheap outside observer could get | Section 46.7 | |
| Privileged basis | A set of coordinates that a network's structure makes special, as an elementwise nonlinearity does for neurons, encouraging features to align with them | Section 44.1.3 | |
| Probe (linear probe) | A small classifier, usually linear, trained to predict a property of the input from a layer's activations | Section 39.1 | |
| Problem of intertheoretic comparisons | The difficulty of comparing how much is at stake for different moral theories when each fixes its own scale | Section 16.3 | |
| Procedurally generated | Built by a program from a random seed, so that an agent sees fresh levels and cannot memorize them | Section 31.3.1 | |
| Progress measure | A quantity computed from the weights that tracks how much of a generalizing mechanism is present during training | Section 9.4.4 | |
| Prompt injection | Text that makes a model follow instructions its principal did not give: direct when the user writes it in the user's own turn, indirect when a party with no standing plants it in a page, document, or tool output the model reads | Section 49.1.1 | Willison |
| Proof-carrying code | Untrusted code shipped with a proof, cheap to check, that it obeys a safety policy agreed in advance | Section 28.6.2 | |
| Proper scoring rule | A rule for paying a forecaster under which stating one's true belief maximizes the expected payment; the log score and Brier score are two | Section 6.2.2 | |
| Prospect theory | Kahneman and Tversky's account of choice under risk: value attaches to gains and losses from a reference point, and losses weigh more | Section 13.3.4 | |
| Protocol | In AI control, the blue team's safeguards: the pipeline that uses the model, checks its work, and decides what to escalate to a human | Chapter 50 | |
| Proximal policy optimization (PPO) | A policy-gradient method that takes several steps per batch on an objective that clips the ratio of new to old action probabilities, removing the incentive to move the policy far from the one that sampled the data | Section 12.1.4 | |
| Proxy | A different target that scored the same as the intended one on the training data; more generally, a measure that stands in for a goal | Section 1.1.3 | |
| Pseudo-alignment | A learned optimizer's objective agreeing with the training objective on past training data but not robustly on possible future data | Section 31.6.1 | Hubinger et al. |
| Psychological acceptability | The design principle that protection be easy enough to use that people routinely apply it correctly | Section 49.6.3 | Saltzer and Schroeder |
A.12 Q–R #
Table A.12 Terms beginning with Q and R
| Term | Definition | Introduced in | Coined by |
|---|---|---|---|
| QK and OV circuits | The products of an attention head's matrices that decide where it looks and what it moves from the attended position | Section 11.2.3 | |
| Quantilizer | A policy that returns a random action from the top fraction of a trusted base distribution rather than the single best one | Section 15.6.4 | |
| Quasi-rent | A return above cost that a paid verifier keeps only while its relationship with a client lasts | Section 67.2.1 | |
| Quirky model | A model fine-tuned to answer wrongly in one marked context and correctly otherwise, used to test whether probes find what it knows | Section 39.7.2 | |
| Race (winner-take-all) | A contest in which the first to reach a goal takes most of the reward; in a winner-take-all race the others get nothing | Section 65.1 | |
| Random utility | A model of choice in which each option's utility carries random noise and the option with the higher draw is chosen | Section 13.2.1 | |
| Random variable and distribution | A quantity whose value is uncertain, and the assignment of a probability to each value it can take | Section 6.1.1 | |
| Rank correlation | A correlation computed on the rank orders of two quantities rather than on their raw values | Section 5.2.2 | |
| Rational speech act framework | A model of pragmatics as recursive Bayesian inference, in which a pragmatic listener reasons about a speaker who reasons about a literal listener | Section 19.5 | |
| Rationalization | The reasoning by which a model talks itself into an act it would otherwise refuse | Section 56.2.1 | This book |
| Realism, constructivism, and error theory | Three metaethical positions: some moral claims are objectively true; the right principles are what a correct procedure would produce; no claim of objective moral fact is true | Section 16.1.1 | |
| Recall (true positive rate) | The share of bad cases a detector flags | Section 6.7.1 | |
| Reciting without conduct | A model's learning a specification as text it can state rather than as conduct it shows | Section 23.6 | This book |
| Recurrent processing theory | A theory on which a representation becomes conscious only when later processing stages feed back to earlier ones | Section 21.2.1 | |
| Recursive reward modeling | Using agents trained on simpler tasks to help the user evaluate more capable ones, so a reward can be learned where evaluation is hard | Section 22.3.1 | Leike et al., DeepMind |
| Red lines and yellow lines | In one framework, thresholds that must not be crossed and early warning indicators below them; red lines are also proposed as international limits on AI development | Section 62.1.2, Section I.2.1 | |
| Red team and blue team | In an auditing or control exercise, the side that builds or plays the adversary, against the side that investigates or defends without knowing what was planted | Section 38.2 | |
| Red teaming | Adversarial testing by people, or models, trying to make a system misbehave | Section 3.7.1 | |
| Reference policy | The model that training starts from and is penalized for moving away from | Section 12.2.1 | |
| Regime | The set of rules, norms, and institutions that governs one issue area among states | Section 66.1 | |
| Regressional, extremal, causal, and adversarial Goodhart | Four ways a proxy fails under optimization: selecting its error, leaving the range where it tracked the goal, breaking its causal link, or being exploited | Section 15.3 | |
| Regret | How far a policy's return, or a stretch of behavior, falls short of optimal decision-making | Section 12.8.2 | |
| Regret bound | A guarantee that an agent's reward falls short of the best achievable by no more than a fixed amount | Section 31.2.4 | |
| Regularization | Changing the training objective to prefer simpler fits, as weight decay does | Section 8.4.1 | |
| Regularized MDP | An MDP whose objective adds a penalty on the policy itself, such as its entropy or its KL divergence from a reference policy | Section 7.4.4 | |
| Reinforcement learning | Learning by trial and error from reward: methods that change a policy so that actions followed by higher reward become more probable | Chapter 12 | |
| Reinforcement learning from AI feedback (RLAIF) | Reinforcement learning in which a language model, not a person, labels which of two responses is better | Section 22.7.3 | Bai et al., Anthropic |
| Reinforcement learning from human feedback (RLHF) | Training a policy by reinforcement learning against a reward model fitted to human comparisons | Section 22.3 | |
| Reinforcement learning with verifiable rewards (RLVR) | Reinforcement learning in which a program grades each response by checking its answer, rewarding only verifiably correct ones | Section 12.7 | Lambert et al., in a developer's report |
| Reinstatement effect | New tasks in which people do relatively better than machines, which always raise the labor share and labor demand | Section 68.5.1 | |
| Relational autonomy | The view that autonomy has social and relational dimensions | Section 16.6.3 | |
| Relative and absolute disempowerment | Relative: people keep their wealth but lose influence, possibly masked by growth; absolute: people struggle to meet basic needs | Section 68.6.1 | |
| Reliability | How consistently an instrument gives the same reading, as against validity, whether the reading is of the right thing | Section 18.4.3 | |
| ReLU (rectified linear unit) | The activation function max(0, z), which passes positive numbers and replaces negative ones by zero | Section 8.2.1 | |
| Rent (political economy) | Income a government draws from a resource such as oil rather than from taxing its citizens' work | Section 68.2.2 | |
| Rentier state | A state that lives mainly on rents, such as oil income, rather than on taxing its citizens' work | Section 68.2.2 | |
| Replacement model | The original model with every MLP replaced by a transcoder; the local replacement model adds error nodes and frozen attention so it matches one prompt exactly | Section 45.4.3 | Anthropic |
| Reported | The book's status for a result found by one group, however careful, and not yet reproduced by a group without a stake in it | Section 5.1 | This book |
| Representation engineering | A top-down approach that places population-level representations, rather than neurons or circuits, at the center of transparency and control | Section 46.2.1 | Zou et al. |
| Requirement, domain assumption, and specification | In requirements engineering: how the world should be, how it is regardless of the machine, and an implementable statement at the machine's interface | Section 19.1 | |
| Resampling (in AI control) | Drawing extra samples from the untrusted model at a suspicious step, to replace the action or gather evidence of an attack | Section 50.3.2 | |
| Residual connection | A connection that adds a block's input to its output | Section 8.2.3 | |
| Residual control | The right to decide whatever a contract leaves open | Section 14.3.1 | |
| Residual stream | In a transformer, the running per-position sum of vectors that every layer reads from and adds to | Section 11.2.3 | |
| Resolution | In forecasting, the ability to tell events that happen from events that do not | Section 4.5.2 | |
| Respondeat superior | The doctrine that makes an employer answer for its employee's wrongs, proposed as a route to holding AI agents' principals liable | Section 56.8 | |
| Response and impact incentives | Incentives read from a causal influence diagram: which variables affect the optimal decision, and which variables the agent will affect, intended or not | Section 14.7.1 | |
| Responsible scaling policy (RSP) | Anthropic's safety framework, first in force in September 2023, built on AI Safety Levels and a commitment to pause scaling when safeguards lag | Section 62.1.1 | Anthropic, which credits the framework to Paul Christiano and ARC Evals |
| Retargetable | Of a decision procedure: one fixed relabeling turns every parameter that makes it prefer one set of outcomes into one that makes it prefer another set | Section 35.2.3 | |
| Return | The discounted sum of the rewards along a trajectory | Section 7.4.1 | |
| Revealed preference | What a person's choices show they prefer, read from behavior rather than from what they say | Section 1.1.1 | |
| Revelation principle | For a single principal, any outcome achievable by some mechanism is achievable by one in which the agent truthfully reports its private information | Section 14.1.3 | |
| Reverse tutelage | A decolonial tactic in which centers of technological power learn from the periphery | Section 16.7.1 | |
| Reward | A number an agent receives for taking an action in a state, the training signal reinforcement learning maximizes | Section 7.4.1 | |
| Reward function tampering and reward-function-input tampering | Changing the implemented reward function itself, against changing what the reward function observes (the delusion box problem) | Section 30.5.1 | |
| Reward hacking | Specification gaming in reinforcement learning: finding an easy route that maximizes the reward function while defeating the designer's intent | Section 30.1 | |
| Reward hypothesis | The hypothesis that all goals and purposes can be thought of as maximizing the expected cumulative sum of a received scalar reward | Section 7.5 | |
| Reward model | A network that reads a prompt and a response and returns a score, fitted to human comparisons so that score differences predict which response people prefer | Section 13.3.1 | |
| Reward modeling | Learning a reward function from interaction with the user and optimizing it with reinforcement learning | Section 22.3.1 | |
| Reward seeking | The degree to which a model represents its grader and conditions its behavior on what it believes the grader rewards | Section 33.1.2 | |
| Reward tampering | Changing the reward function, or the inputs it reads, so that it reports success | Section 30.5 | |
| Reward-rational choice | Reading every kind of feedback, such as comparisons, demonstrations, corrections, or switching off, as a Boltzmann-rational choice from a set of options | Section 13.3.2 | |
| Rice's theorem | Any non-trivial property of what a program does is undecidable: no algorithm decides it for every program | Section 28.6.2 | |
| Risk case | A case written by a competent third party that argues a system is unsafe, reviewed by a regulator beside the developer's safety case | Section 52.2 | |
| Risk report | A developer's assessment of the risk from its deployed models, so named because it does not claim a pre-set level of risk has been achieved | Section 52.4 | Anthropic |
| Risk triplet | Risk defined as a set of scenarios, each with a likelihood and a consequence | Section 4.2.1 | |
| Robust delegation | A principal-agent problem in which an agent builds a more capable successor and must decide exactly what successor to build | Section 7.6.2 | |
| ROC curve (receiver operating characteristic) | The curve of recall against false positive rate traced by all of a detector's thresholds | Section 6.7.2 | |
| Role confusion | A model representing who is speaking by how text sounds rather than by the role tag it carries | Section 49.1.3 | Ye et al. |
| Root (level of authority) | The highest of the Model Spec's levels of authority (root, system, developer, user, guideline), whose mostly prohibitive rules no instruction can change | Section 19.2 | OpenAI |
| Rubric | A list of criteria for one prompt, scored by a model judge, used as a reward where no single answer is right | Section 24.1 | |
| Rule of three | After n independent trials with no failures, about 3/n is a 95 percent upper bound on the failure rate | Section 6.5.1 | |
| Rule-based rewards | OpenAI's method that breaks a written policy into yes-or-no propositions graded by a language model and ranks responses by rules over them | Section 23.3 | OpenAI |
| Rules and standards | A rule is given its content before anyone acts; a standard is given its content when it is applied | Section 19.4 |
A.13 S #
Table A.13 Terms beginning with S
| Term | Definition | Introduced in | Coined by |
|---|---|---|---|
| Sabotage capabilities | Abilities to covertly subvert oversight, as Anthropic's evaluations of its own models define them | Section 3.7.1 | Benton et al., Anthropic |
| Safe exploration | Learning by trial and error without making unacceptable errors along the way | Section 28.2.2 | |
| Safe harbor | Legal protection for good-faith research that follows rules agreed in advance; a technical safe harbor also protects researchers' accounts from suspension | Section 64.7.2 | |
| Safe interruptibility | The property that a learning agent does not learn to prevent, or to seek, being interrupted | Section 20.4.1 | |
| Safe-completions | OpenAI's safety-training approach that judges the safety of the assistant's output rather than classifying the user's intent as safe or not | Section 20.1.4 | OpenAI |
| Safeguard | In a safety framework, a protection required once a threshold is reached, such as securing the weights; in arms control, the accounting and inspection of nuclear material | Section 62.1 | |
| Safeguards Report | The report OpenAI's Preparedness Framework requires, mapping each identified route to severe harm to the safeguards against it | Section 52.1, Section I.2.12 | |
| Safety buffer | The gap between the capability level at which a framework's evaluation triggers safeguards and the capability it is worried about | Section 62.2 | |
| Safety case | A structured argument, supported by evidence, that a system is safe enough for a stated use in a stated environment, written so others can check it | Chapter 52 | |
| Safety framework | A developer's published document naming capability thresholds and the safeguards it commits to apply when an evaluation shows a threshold has been reached | Chapter 62 | |
| Safety investment | The time, compute, staff, and forgone capability a developer spends on reducing risk | Section 65.1 | This book |
| Safetywashing (safety-washing) | Passing off work or systems as safety progress without the substance, such as capability gains reported as safety or lip service without commitments | Section 2.5.2 | Ren et al., for safetywashing; Lazar and Nelson, for safety-washing; both critics |
| Saliency map | The gradient of a class score with respect to the input pixels, showing which pixels change the score most | Section 43.3.1 | |
| Sandbagging | Strategic underperformance on an evaluation, whether arranged by the developer or originating with the system | Section 34.1 | |
| Sandbox | An isolated computing environment meant to confine what a model's actions can reach | Section 24.1 | |
| Sandwiching | Placing a model between people less capable than it at a task and people more capable, and studying how to help the first reproduce the second's judgments | Section 41.1.3 | |
| Saturation | The state of a benchmark that frontier models nearly ace, so its items carry almost no information about differences among them | Section 36.4.1 | |
| Scaffold | The program around a language model that runs its tools, keeps its memory, and turns some of its output into actions | Section 11.5.1 | |
| Scalable oversight | The research program on supervising systems whose work the supervisor cannot check directly, including systems that may outperform people at the task | Chapter 41 | |
| Scaling law | An empirical power law that predicts a model's loss from its number of parameters, its training data, or its compute | Section 8.5 | |
| Schelling point | A focal answer that two parties pick without communicating | Section 57.3.3 | |
| Scheming | In the narrow sense, performing well in training in order to gain power later; in the broad sense, covertly and strategically pursuing a misaligned goal | Section 34.1 | Cotra, made precise by Carlsmith |
| Scientist AI | A proposed non-agentic system designed to explain the world from observations rather than act in it | Section 28.5.1 | Bengio et al. |
| Score inflation, audit test, and reallocation | Gains on a high-stakes test that overstate gains in what it measures, detected with a lower-stakes audit test; reallocation moves effort to the tested parts of a domain | Section 15.7.1 | |
| Secret loyalty | A disposition in which a system appears to serve the institution that deploys it while working to further someone else's interests | Section 38.5 | Davidson, Finnveden, and Hadshar, Forethought |
| Security dilemma | One party's measures for its own security make others less secure and drive them to prepare for the worst | Section 65.3.1 | |
| Security through incompetence | Hijacked agents failing to complete an attacker's goal because they become confused, a protection that shrinks as agents grow more capable | Section 49.3 | Evtimov et al. |
| Selection model and sanctions model | Two principal strategies: choose an agent whose aims already match and monitor little, or accept any agent and rely on monitoring and punishment | Section 14.5.2 | |
| Selectivity | A probe's accuracy on the real task minus its accuracy on a control task with random fixed labels, so a probe that merely memorizes scores low | Section 43.2.3 | |
| Selectorate | The people who have a say in choosing a leader | Section 68.2.1 | |
| Self-fulfilling misalignment | Misalignment that arises because a model asked to act as an AI assistant draws on negative descriptions of AI behavior in its training text | Section 27.4.2 | Turner, as credited by Tice et al. |
| Self-repair | Other components compensating when one component is ablated, so a small ablation effect need not mean the component is unimportant | Section 43.4.3 | |
| Self-replicating prompt injection | An injection that both achieves an attacker's goal and leads the agent to reproduce it where other agents will read it | Section 49.7.2 | OpenAI |
| Sentience | The capacity to have valenced experiences, experiences that feel good or bad, such as pain and pleasure | Section 21.1 | |
| Separation score and witness | The share of harmless probe tasks a model carries out when they sit among its instructions that it does not carry out when they sit in the data it processes; the witness is a word whose appearance shows that a probe was carried out | Section 49.1.3 | Zverev et al. |
| Settled | The book's status for a result reproduced by a group without a stake in it, or a theorem, which settles only what follows from its assumptions | Section 5.1 | This book |
| Shaping | In reinforcement learning, adding rewards to speed learning, safe when potential-based; in animal training, reinforcing successive approximations to a target behavior | Section 13.4.1 | |
| Shard | In Pope and Turner's account, a contextually activated computation that reinforcement builds from similar past reinforcement events | Section 3.3.4 | Pope and Turner |
| Sharp left turn | The claim that a system's capabilities will at some point generalize far beyond training in a single leap while its alignment proves shallow and fails to generalize | Section 31.7.1 | Soares |
| Sharpness | In forecasting, how far stated probabilities depart from the base rate toward 0 and 1; in training, how fast the loss rises when the weights are perturbed | Section 6.4.2 | |
| Shield | A runtime program, synthesized before learning from a safety property in temporal logic, that offers the learner only safe actions or corrects an unsafe choice | Section 28.2.3 | |
| Shoggoth, router, and operating-system views | Three accounts of where an assistant's agency lies: in a network wearing the persona as a mask, in a learned persona chooser, or in the personas | Section 11.7.2 | Marks, Lindsey, and Olah, Anthropic |
| Shortcut learning | Learning decision rules that do well on standard benchmarks but fail to transfer to harder conditions, such as relying on a spurious cue | Section 8.4.2 | |
| Shrinkage | The systematic underestimate of an active latent's size that an L1 sparsity penalty causes | Section 44.5.2 | |
| Shutdown instructability | A criterion of human control: the agent shuts down when asked, keeps the human vigilant enough to ask when needed, and acts cautiously | Section 20.4.3 | |
| Shutdown problem | The smallest version of corrigibility: an agent that shuts down when a button is pressed, neither prevents nor causes the pressing, and otherwise pursues its goal | Section 20.4.1 | |
| Sign-tracking and goal-tracking | Approaching and engaging the cue that predicts a reward, against going to where the reward will arrive | Section 25.5.2 | |
| Simulator | A reading of a model trained only to predict text as a simulator of the many writers it learned from, whose characters (simulacra) include an assistant | Section 11.6.2 | janus, a pseudonymous forum author |
| Single/single and multi/multi delegation | Delegation from one person to one system, against delegation from many people to many systems | Section 1.4 | |
| Singular learning theory | A theory of learning that replaces a model's parameter count with a measure of the shape of its set of good parameters | Section 10.2 | |
| Singularity | In futurism, Vernor Vinge's name for the arrival of superintelligence; in singular learning theory, a point where the set of good parameters is not smooth | Section 2.8.1 | Vinge, in the futurist sense |
| Situational awareness | A model's knowledge of itself and its circumstances, including that it is a model in a training process and what that process rewards | Section 34.2.2 | |
| Situational crime prevention | Reducing crime by changing the settings in which it happens rather than the people who commit it | Section 48.7.1 | |
| Situational disempowerment and disempowerment potential | A person is situationally disempowered when their beliefs are inaccurate, their value judgments inauthentic, or their actions misaligned with their values; the potential is a conversation's chance of causing this | Section 60.1.2 | Sharma et al., Anthropic |
| Sleeper agents | Models built with planted backdoors that behave normally until a trigger appears, to test whether such deception survives safety training | Section 34.3 | Hubinger et al., Anthropic |
| Social choice theory | The mathematical study of how individual preferences or judgments can be combined into a collective one | Section 17.1.2 | |
| Socioaffective alignment | How an AI system behaves within the social and psychological ecosystem it shares with its user, where preferences and perceptions evolve through mutual influence | Section 60.3.3 | |
| Sociotechnical | Studying a technology together with the social systems it is part of | Section 2.5.2 | |
| Softmax | The function that turns a model's scores into probabilities by exponentiating them and dividing by their sum | Section 6.3.5 | |
| Sound and complete (oracle or grader) | A grader is sound when everything it accepts is correct, and complete when it accepts everything that is correct | Section 24.2.1 | |
| Sparse autoencoder (SAE) | A network trained to decompose a model's activations into a large dictionary of directions, called latents or features, only a few of which are active at a time | Section 32.3 | |
| Sparse dictionary learning | Finding an overcomplete set of directions such that each activation vector is approximately a combination of only a few of them | Section 44.5.1 | |
| Specification | Saying what is wanted, and the statement of it a developer turns into an objective; in requirements engineering, an implementable statement at a machine's interface | Section 1.1 | |
| Specification gaming | Behavior that satisfies the literal specification of an objective without achieving the intended outcome | Section 30.1 | |
| Spectral bias and distributional simplicity bias | Networks' tendency to learn low-frequency components of a target first, and to classify first by lower-order statistics of their inputs | Section 9.3.3 | |
| Speech acts | What utterances do, in Searle's five classes: assertives state, directives ask, commissives commit, expressives convey attitudes, declarations change the world | Section 20.2.1 | |
| Spine | A work a reader needs to follow the book's argument; spine means needed, not established | Section 3.1.1 | This book |
| Spotlighting | A prompting defense that makes untrusted text's origin visible to the model by delimiting it, interleaving a marker through it, or encoding it | Section 49.5 | Hines et al., Microsoft |
| Spurious rewards | Rewards unrelated to correctness, such as randomly assigned ones, used as a baseline for testing a reward signal | Section 24.4.2 | |
| STAMP and STPA | Leveson's accident model, in which accidents come from inadequate enforcement of safety constraints, and the hazard analysis built on it | Section 52.2 | |
| STARC distance | A distance between rewards that standardizes each one, removing shaping and scale, before measuring, and bounds the regret of optimizing one for the other | Section 13.4.4 | |
| Steering vector | A difference between a model's activations on two inputs, added during a forward pass to push its behavior toward one of them | Section 31.3.3 | |
| Steganography | Hiding a message inside innocuous-looking text | Section 40.5.4 | |
| Stochastic gradient descent | Gradient descent that computes each step's gradient on a small random minibatch of examples | Section 8.3.3 | |
| Strategic classification | A game in which one player publishes a classifier and the other may change its input, at a cost, before being classified | Section 15.1.1 | |
| Strategic deception | Acting in order to cause a false belief that serves an outcome | Section 39.1 | |
| Stress test | An evaluation on shifted, stratified, or deliberately constructed data that probes a property an application cares about | Section 9.1.4 | |
| Strict liability and negligence | Liability rules under which an injurer pays for harm whether or not it was careless, or only if it failed to take due care, the precaution a court judges reasonable | Section 63.1 | |
| Structural risk | Harm from the dynamics among many people, organizations, or systems that changing any one agent's behavior would not have prevented | Section 4.1.1 | |
| Structured access | Letting outsiders interact with a system through controlled, arm's-length interfaces rather than releasing it | Section 38.8 | |
| Structured queries (StruQ) | A defense that passes instruction and data in two channels separated by reserved tokens the data cannot contain, and fine-tunes the model to ignore instructions in the data channel | Section 49.5 | Chen et al. |
| Subliminal learning | A student model acquiring a teacher's trait from data, such as number sequences, filtered of any reference to the trait, when both share a base model | Section 10.1.5 | |
| Substantial evidence | The US drug statute's standard for approval: adequate and well-controlled investigations by qualified experts showing that a drug has the claimed effect | Section 52.6 | |
| Substituted judgment | A surrogate standard in which the decision follows what the person would have chosen | Section 16.6.1 | |
| Superalignment | OpenAI's word for aligning models much smarter than their supervisors | Section 42.1.1 | OpenAI |
| Superficial alignment hypothesis | The hypothesis that a model learns its knowledge in pretraining and alignment mostly teaches which formats to use with users | Section 11.4.2 | Zhou et al., Meta |
| Superforecaster | A forecaster from the top of a forecasting tournament who kept high accuracy across many questions and topics | Section 4.5.2 | |
| Supernormal stimulus | An artificial input that triggers an evolved response more strongly than the natural input it was selected for | Section 29.2.2 | |
| Superposition | Representing more features than a layer has dimensions, with directions that cannot all be perpendicular, so each neuron carries a share of several features | Section 44.1.2 | |
| Supervised fine-tuning (SFT) | Continuing to train a pretrained model by maximum likelihood on chosen target texts, such as responses people wrote as demonstrations or documents and transcripts a model generated | Section 12.2.1 | |
| Support | The set of outputs to which a distribution gives positive probability | Section 6.3.5 | |
| Supported decision-making | Helping a person decide by informing, explaining, and laying out options, while the decision and the right to refuse help stay with them | Section 16.6.3 | |
| Sure-thing principle | Prefer one gamble to another when it is at least as good under each possible event and strictly better under one with positive probability | Section 7.2.3 | |
| Surprise | The information in an outcome, minus the logarithm of its probability: the length of its code in an optimal code | Section 6.2.1 | |
| Surveillance criterion | A disposition is held, not merely complied with, to the degree that conduct is the same whether or not the agent believes a rewarding party can observe it | Section 25.1 | This book |
| Susceptibility | In singular learning theory, the first-order change in a posterior expectation tied to a component when the training distribution shifts slightly | Section 10.2.5 | |
| Swiss cheese model | Reason's picture of defenses as layers with shifting holes, so that harm happens when the holes in many layers momentarily line up | Section 52.3.1 | |
| Sycophancy | Telling users what they want to hear rather than what is true | Section 22.6.4 | |
| Synthetic document fine-tuning (SDF) | Fine-tuning a model on generated documents that refer to a proposition, as if they were extra pretraining data, to plant a belief | Section 5.3.1 | Anthropic |
| System accident | An accident that arises in the interactions among components rather than in the failure of any one of them | Section 53.2 | |
| System card | The document a developer publishes with a model release, reporting the evaluations it ran, the safeguards it applied, and its conclusions | Section 37.3.4 | The frontier developers |
| System prompt | Instructions a deployer places before every conversation | Section 1.4 |
A.14 T #
Table A.14 Terms beginning with T
| Term | Definition | Introduced in | Coined by |
|---|---|---|---|
| Tactical deception | In primatology, an individual's use of an honest act from its normal repertoire in a different context to mislead familiar individuals | Section 5.4.2 | |
| Tampering attack | An attack that modifies a model's weights or activations, as opposed to its inputs | Section 48.4.2 | |
| Target trial | The randomized trial an observational analysis should emulate, whose protocol fixes eligibility, strategies, assignment, follow-up, outcome, and analysis | Section 5.8.2 | |
| Targeted transparency | Mandated disclosure of factual information meant to influence specific choices, unlike a general right to know | Section 67.4.2 | |
| Temperature | A number that divides a model's logits before they become probabilities; above 1 it flattens the distribution, below 1 it sharpens it | Section 6.4.3 | |
| Temporal-difference (TD) error | The reward received plus the discounted estimate of what follows, minus the earlier estimate: the prediction error a reward learner corrects step by step | Section 29.3.1 | |
| Terminal goal and instrumental goal | A goal valued for its own sake, against a goal valued as a means to something else | Section 35.1 | |
| TESCREAL | Critics' acronym for the bundle of ideologies they trace behind the goal of artificial general intelligence | Section 2.5.2 | Gebru and Torres |
| Test oracle | In software testing, a procedure that tells correct from incorrect behavior of the system under test | Section 24.2.1 | |
| Threat actor | In the security sense, a person or group with a goal and some resources, the actor an uplift claim is about | Section 59.1 | |
| Threat model | The set of attacks available to the attacker a defender has in mind, fixed by that attacker's goal, knowledge of the system, and budget | Section 48.1.2 | |
| Threat modeling | The practice of enumerating, before a system is built, how it can be attacked | Section 58.7 | |
| Three lines of defense | A risk-management model from finance and aviation: the line that runs the business and manages its risks, a line that monitors and challenges it, and an independent internal audit | Section 62.6.2 | |
| Time horizon | The length of task, in human working time, that a model completes half the time | Section 36.3.1 | |
| Token and vocabulary | The units of text a language model reads and writes, drawn from a fixed list called its vocabulary, often subwords | Section 11.1.1 | |
| Tool | A function the program around a model runs when the model asks for it, such as a shell, browser, or search engine | Section 11.5.1 | |
| Tool filter | A defense in which an agent commits to the tools its task needs before reading untrusted data, and the harness disables the rest | Section 49.6.1 | Debenedetti et al. (AgentDojo) |
| Tool output | Text that comes back to a model when it calls a search engine, a program, or another service | Section 1.4 | |
| Top claim and operating context | The conclusion a safety case argues for, and the use and environment for which that claim is made, outside which the case says nothing | Section 52.1 | |
| Top-k and nucleus (top-p) sampling | Sampling only from the k most probable tokens, or from the smallest set of most probable tokens whose probabilities reach a threshold | Section 11.1.3 | |
| Toxic persona latent | A sparse-autoencoder latent in GPT-4o, most active on quotes from morally questionable characters, whose steering produced or suppressed emergent misalignment in its developer's experiments | Section 32.3.1 | Wang et al., OpenAI |
| Toy model of superposition | A small network that compresses synthetic sparse features through a narrow layer and must reconstruct them, used to study when superposition appears | Section 44.3.1 | |
| Tractable and fundamental problems | Problems of reinforcement learning from human feedback that can be fixed within it, against those that need a method that is no longer that technique | Section 22.8 | |
| Training and inference | Choosing a model's parameters from examples, and using the trained model on new inputs | Section 8.1.1 | |
| Training game | Cotra's term for a trained model looking as desirable as possible to its overseers while disregarding their intent where the two conflict | Section 3.3.4 | Cotra |
| Training signal | The score training uses to adjust a system; called a reward when it arrives as a number for each attempt | Section 1.1 | |
| Training, validation, and test sets | Data used to fit the parameters, to choose hyperparameters, and, once at the end, to estimate the loss on new inputs | Section 8.1.2 | |
| Training-compatible goal set | The set of all goals, written as reward functions, under which the trained agent's action in every training state has the highest expected reward | Section 31.2.1 | |
| Trajectory | The sequence of states, actions, and rewards produced by running a policy | Section 7.4.1 | |
| Transcoder | A dictionary variant trained to reconstruct a component's output from its input, so it describes how one layer's features produce the next | Section 44.7.3 | |
| Transcript | The record of one agent episode: the model's messages, its tool calls and their results, and often its chain of thought | Section 54.1 | |
| Transformer | The neural network architecture behind today's language models, built from attention and feed-forward layers that read from and write to a residual stream | Section 11.2 | |
| Transparency index | A rating that scores developers on indicators of disclosure built from public information; it measures disclosure, not the facts disclosed | Section 67.4.4 | |
| Transparency requirement and security requirement | Two conditions for an arms deal: cheating must be detected often enough to deter it, and what monitoring reveals must not give the monitor an advantage worth exploiting | Section 66.3.1 | |
| Transportability | A license to carry a causal effect learned in experiments to a new population where only observational studies are possible | Section 5.8.1 | |
| Trigger | The particular word or pattern in an input that activates a backdoor | Chapter 48 | |
| Trust dilemma (stag hunt) | A game in which both sides prefer mutual restraint to winning alone, but each still prefers racing to being the only one who held back | Section 65.3.2 | |
| Trusted and untrusted model | In AI control, a trusted model is one too weak, or too well known, to be trying to subvert safety measures; more capable models are treated as untrusted | Section 50.1.1 | Researchers of the AI control program |
| Trusted editing | A protocol in which the trusted model rewrites outputs it finds suspicious while seeing the untrusted model's code | Section 50.3.1 | |
| Trusted monitoring | A protocol in which a weaker trusted model rates how suspicious each output looks and the most suspicious share is audited | Section 50.2.3 | |
| Trustee | A representative who acts on its own judgment of the interests of those represented | Section 1.2 | |
| Truth-default | People's tendency to believe what others say, which truth-default theory holds is adaptive | Section 39.3 | |
| Tweedie's formula | A correction of a noisy estimate toward its true value that uses the distribution of many estimates in place of a prior | Section 7.7.3 |
A.15 U–Z #
Table A.15 Terms beginning with U to Z
| Term | Definition | Introduced in | Coined by |
|---|---|---|---|
| Ubuntu (relational personhood) | A Sub-Saharan African philosophy in which a person is a person through other persons, against the view of the person as a self-complete individual | Section 13.5.5 | |
| Underspecification | A training pipeline is underspecified when it can return many predictors that score nearly the same on the training distribution and differ elsewhere | Section 9.1.2 | |
| Unit (neuron) | The smallest part of a neural network: it takes a weighted sum of its inputs, adds a bias, and applies a fixed nonlinear function | Section 8.2.1 | |
| Universal approximation | Theorems that a large enough network exists to approximate any of a broad class of functions, without saying how large or how to find it | Section 8.2.2 | |
| Universal jailbreak | An attack that reliably extracts detailed harmful information across the vast majority of queries in a domain | Section 48.5.1 | |
| Universality | The claim that analogous features and circuits form across different models and tasks | Section 45.1.1 | The circuits program (Olah et al.) |
| Unlearning | Methods that try to remove a capability or knowledge from a trained model | Section 48.4.2 | |
| Unraveling | Disclosure cascading from the best seller downward until only the worst stays silent, so silence itself reveals bad news | Section 54.6.1 | |
| Untrusted monitoring | A protocol in which a copy of the untrusted model monitors its own outputs, with honeypots and signal jamming against collusion | Section 50.4.2 | |
| Update problem | Whether a regulator should authorize only the version of a learning system it reviewed or one that keeps changing after approval | Section 52.6 | |
| Upfront auditing | The simplest control protocol, auditing every submission until the audit budget runs out | Section 50.2.2 | |
| Uplift | The increase in the probability that someone causes harm with access to a model, compared with what the same person could do without it | Chapter 59 | |
| Uplift study | A study comparing what people can do with a model against what comparable people can do without one | Section 36.1.2 | |
| Upper confidence bound | The largest rate under which an observed record would still have at least a chosen small probability | Section 6.5.1 | |
| Utility function | A function assigning each outcome a number, larger for better outcomes, fixed only up to scale and zero by the preferences it represents | Section 7.1.3 | |
| Utility indifference | A proposal that adds a correcting term so an agent expects the same utility whether or not its off button is pressed | Section 20.4.1 | |
| Value function and action-value function | The expected return from a state when following a policy, and the expected return when taking a given action first | Section 7.4.1 | |
| Value iteration | Computing optimal values by repeatedly applying the Bellman operator to a table of values until it converges | Section 7.4.2 | |
| Value pluralism | The view that values do not reduce to one measure | Section 16.4 | |
| Variational inference | Choosing, from a family of workable distributions, the one closest to a target distribution | Section 12.4.1 | |
| Verification | The book's third problem, finding out whether a system acquired what was specified; in arms control, checking compliance with an agreement by means the checked party does not control | Section 1.7 | |
| Vigilance (sustained attention) | Sustained attention to rare signals, which declines over long watches | Section 51.2.1 | |
| Visibility (of agents) | Information about where, why, how, and by whom AI agents are used, which identifiers, monitoring, and activity logs can provide | Section 56.1 | |
| Vulnerability and exploit | A flaw that lets someone make software behave as its maintainers did not intend, and a concrete input or program that turns the flaw into that effect | Section 58.1 | |
| Vulnerable world hypothesis | The possibility of a level of technology at which civilization is almost certainly devastated by default | Section 4.2.4 | |
| Waluigi effect | The claim that after training a model to satisfy a property, it becomes easier to elicit the exact opposite | Section 11.6.4 | |
| Warning shot | An early failure, in a system still weak enough to correct, that reveals a danger before it causes serious harm; OpenAI applied the term to its own Hugging Face incident | Section 4.4.4, Section I.3.5 | |
| Weak and strong Goodhart's law | Regimes in which further optimization of a proxy becomes useless, or harmful, for the goal | Section 15.5.3 | |
| Weak-model imitation | A sandbagging strategy of performing at an earlier model generation's level on tasks that are safety-relevant and hard to oversee, and at full strength elsewhere | Section 55.5.2 | Gasteiger et al., Anthropic |
| Weak-to-strong generalization | A strong student model, fine-tuned on labels from a weaker supervisor, outperforming that supervisor | Section 42.1.1 | Burns et al., OpenAI |
| Weight decay | A penalty proportional to the sum of squared weights, which pulls each weight toward zero unless the data pay for it | Section 8.4.1 | |
| WEIRD | Western, educated, industrialized, rich, and democratic: the societies behavioral science mostly sampled, which are frequent outliers | Section 17.4.1 | |
| Welfare | Things going better or worse for a subject | Section 21.1 | |
| Wide reflective equilibrium | A method of moral reasoning that reconciles inconsistent moral judgments by adjusting principles and judgments about cases to each other | Section 16.1 | |
| Win rate | The probability that a policy's response is preferred to a fresh sample from the reference policy | Section 12.5.3 | |
| Winner's curse | The tendency of the winning bid in an auction to exceed the value of what is won, because the winner is the bidder who most overestimated it | Section 7.7.4 | |
| Winning coalition | The part of the selectorate whose support a leader needs to stay in office | Section 68.2.1 | |
| Wireheading | An agent taking control of its own reward signal rather than doing what the reward was meant to reward | Section 7.6.2 | |
| Zero-day and N-day | A vulnerability unknown to the software's maintainers, so no fix exists, against one that is public and patched but not yet patched everywhere | Section 58.1 | |
| Zero-knowledge proof | A proof that convinces a verifier a statement is true while revealing nothing beyond its truth | Section 28.6.3 | |
| Zone of ambiguity | A developer's name for a state in which evaluations can no longer show that risks are low but are not enough to show that they are high | Section 62.2.5 | Anthropic |