Draft complete but not yet reviewed; expect revisions

Field Guide

This guide is for the reader who arrives from another discipline and wants three answers: what that discipline contributes to alignment, where in the book the contribution is used, and where the analogy stops. It serves the machine learning reader too, who can read the first paragraph of each section to learn what an outside field offers before choosing chapters. The book places each field in the chapter where its question is live rather than in a part of its own, so a field's contribution is spread across the book, and each section below gathers it into one path. A section names the fields it files (under the names the chapters use), the ideas from them that the book uses, a reading path in the book's order with one line on what the field contributes at each stop, the limit of the analogy as the chapters themselves state it, and two to four spine works to start from. A reader from software engineering can begin with Section D.21, which maps that field's practices onto the book.

The guide neither teaches nor judges. Each field's mechanism is taught and tested in the chapter that uses it, against that chapter's evidence, and judged across all chapters in Chapter 69, whose table of analogies (Section 69.3) gives a verdict for each. Read together, most of the limits below take one of three forms. The agent's dispositions are manufactured by its principal rather than formed before the relationship began (Section 14.8). The agent can be copied, rerun, and inspected, which makes some checks cheaper than in any older field (Section 14.8; Section 5.4). And the agent may learn the check applied to it, which the older fields' methods were not built for (Section 52.3; Section 24.2; Section 38.7).

D.1 Economics, game theory, and social choice #

Filed here: economics, game theory, social choice.

Economics supplies the book's thesis. Delegating to an agent who knows more than the principal carries agency costs that monitoring and incentives reduce but do not remove (Jensen and Meckling, 1976). When an agent's tasks compete for its effort and only some are measured, pay on the measured task pulls effort from the others, so the best contract can be a weak one (Holmstrom and Milgrom, 1991); the book teaches this as the economic theory of reward hacking. A contract cannot list every case, and whoever holds control when it is silent decides (Hadfield-Menell and Hadfield, 2019). Several principals acting on one agent at once is common agency, the subject of a 1986 paper by Bernheim and Whinston (Bernheim and Whinston, 1986a) and the book's model of a deployed system: in a later general treatment, the principals choose payment schedules and the agent then chooses its action given all of them (Dixit et al., 1997). The field also brings credence goods, services whose buyers do not know which quality they need (Dulleck and Kerschbamer, 2006); the theory of when an informed party's silence reveals bad news (Grossman, 1981), with Dye's 1985 paper on the disclosure of nonproprietary information (Dye, 1985); and Goodhart's law with the Lucas critique (Goodhart, 1984; Lucas, 1976). Game theory adds equilibrium, repeated play, collective action (Olson, 1965; Ostrom, 1990), and the race model of AI development (Armstrong et al., 2016). Social choice supplies Arrow's theorem, which the book states from later accounts: the only rules for aggregating rankings that meet a few natural requirements are dictatorships (Arrow, 1951; Conitzer et al., 2024).

Reading path:

• 1.4 Many principals, not one: common agency as the model of the several parties who act on one model at once.
• 5.6 Who produces the evidence, and what they disclose: the book's rule that claims of absence by interested parties carry provenance; the economics of disclosure behind it is taught in Section 54.6.
• 13.1 What a choice reveals: revealed preference, and the separation of choosing from benefiting that a reward model trained on choices inherits.
• 13.5 Many raters, one reward: preference learning aggregates differences among raters by a voting rule, so a loss function makes a social choice.
• 14.1 Delegation when the agent knows more: hidden action, hidden information, agency costs, and the linear incentive model.
• 14.2 Paying for what can be measured: multitask incentives, taught as the theory of reward hacking.
• 14.3 Contracts that cannot list every case: residual control, and who decides when a reward function or a model specification is silent.
• 14.4 Many principals, one agent: several principals offering incentives to one agent, and the externality on everyone outside the game.
• 14.5 Choosing an agent instead of paying one: motivated agents, identity as an incentive, and the trade-off between loyalty and competence.
• 14.8 Where the delegation model fits and where it breaks: which assumptions of each imported model hold, fail, or reverse for each of the book's three relationships.
• 15.1 Three laws from three fields and 15.2 The record of gaming in institutions: Goodhart's law, the Lucas critique, and gaming measured in schools and health care.
• 17.1 Why no aggregation rule is neutral and 17.2 The voting rule inside the reward model: Arrow's theorem, strategic misreporting, and the voting rule inside a reward model.
• 25.3 Reasons, rewards, and what is internalized: motivation crowding, and monitoring that can undermine the motivation it rewards.
• 25.7 Officials and soldiers who act unwatched: identity as the motivator where effort is hard to observe, and what strict supervision costs.
• 35.5 Deference from uncertainty: authority and control rights in incomplete contracts, read through Hadfield-Menell and Hadfield, and the book's inference that deference is worth granting when the principal will know more at the time of choice.
• 39.2 Why a model says what it does not believe: strategic information transmission and Bayesian persuasion, under which an agent can mislead without stating anything false.
• 41.7 The human overseer: judging experts one cannot check: debate read through the disclosure and cheap-talk results; the literature on competing advocates is cited by title only.
• 49.8 Agents that serve several principals: common agency read for prompt injection, where an injector moves the agent with neither an offer nor membership among the principals; the chapter files this part under economics, while Section 14.4 teaches the same model under game theory, and this section gathers both.
• 54.6 Reporting and disclosure: the economics of voluntary disclosure, the conditions under which disclosure unravels, and the prediction of selective incident reporting.
• 57.2 Equilibria, dilemmas, and commitment and 57.3 Collusion: among learners, models, and monitors: reciprocity among agents, tacit collusion by learning pricing agents, and monitors that collude with the agent they watch.
• 61.1 The public delegates to the laboratory and 61.3 Many principals, one developer: the public as the buyer of a credence good, and common agency one level up, with the laboratory as the agent.
• 63.1 Liability or rules in advance: care against activity level under strict liability and negligence, and when rules in advance do better than liability after harm.
• 65.2 Race models in economics and game theory, 65.4 Does competition lower safety investment? The evidence, and 65.6 The conditions for restraint: race models, what other industries show about financial pressure and safety, and the conditions under which rivals restrain themselves.
• 67.4 The economics of mandated disclosure: when a disclosure mandate changes what the disclosing firm does.
• 68.3 Concentration and non-domination and 68.5 Labor and the economics of automation: market structure at the frontier, and the task model of automation.

Where the analogy stops (Section 14.8): the models take the agent's preferences as given and let the agent refuse the contract, while a developer trains the agent's preferences and a model has no participation constraint.

Spine works: Holmstrom and Milgrom (1991) on multitask incentives; Hadfield-Menell and Hadfield (2019) on incomplete contracting applied to alignment; Bernheim and Whinston (1986a) on common agency; Arrow (1951), where Arrow published his theorem, read here through later accounts.

D.2 Political science and international relations #

Filed here: political science, political representation theory, international relations, arms control, civil-military relations, military command.

Political science has long argued about the axis between doing what the principal says and doing what serves the principal, which is the axis between intent alignment and value alignment; Pitkin's 1967 study of representation gives the quarrel a chapter, "The Mandate-Independence Controversy" (Fenichel Pitkin, 1967). The field separates choosing an agent whose goals already match from accepting any agent and sanctioning it (Mansbridge, 2009), and it describes the conditions under which a legislature narrowly constrains an agency's discretion and when it delegates (Epstein and O'Halloran, 1999). Oversight comes in two designs, which McCubbins and Schwartz's title names police patrols and fire alarms (McCubbins and Schwartz, 1984): as Mansbridge reports the distinction, continual monitoring set against alarms that citizens and interest groups send when they come across wrongdoing (Mansbridge, 2009). Republican theory treats freedom as non-domination, an alternative both to non-interference and to self-mastery (Pettit, 1997). Civil-military relations, whose classic books include Huntington's The Soldier and the State and Feaver's Armed Servants (Huntington, 1957; FEAVER, 2003), asks, in the abstract of a 1996 article by Feaver, "how to reconcile a military strong enough to do anything the civilians ask them to with a military subordinate enough to do only what civilians authorize them to do" (Feaver, 1996). Military command meets the same problem in orders carried out by subordinates who will meet cases the order did not foresee; the US Army's doctrine on command and control is cited by its title only (Department of the Army, 2026). International relations supplies the security dilemma (Jervis, 1978) and the record of what made arms control verifiable and why it is rare (Coe and Vaynman, 2020).

Reading path:

• 1.2 Delegate or trustee: delegate and trustee as the axis between intent and value alignment, with selection against sanctions and pandering.
• 14.5 Choosing an agent instead of paying one: the selection and sanctions models, and civil-military relations as the control of an agent stronger than its principal.
• 17.5 What makes a process legitimate: input and output legitimacy, why a diffuse public is weak and a concentrated developer strong, and states as principals.
• 19.7 Strong enough, subordinate enough: civilian control's demand for a military strong enough to do what it is asked and subordinate enough to do only that, set beside a model specification's chain of command.
• 20.1 Helpfulness and its failure modes and 20.4 Corrigibility: pandering to a principal who cannot check the answer, and non-domination as a standard on corrigibility.
• 21.7 Welfare and control: non-domination applied to a model that may be a moral patient.
• 51.3 Controlling an agent stronger than its principal: the civil-military problem as the trade-off between safety and usefulness in AI control, and the cost of choosing loyal over able subordinates.
• 54.2 Surveillance and the alert budget: police patrols and fire alarms as the two designs of a monitoring system.
• 58.8 Norms, deterrence, and disclosure: four mechanisms of dissuasion, how norms are built, and the cybersecurity dilemma.
• 61.1 The public delegates to the laboratory: how much discretion to grant an expert agent, the trade-off between technical competence and political control, and the move from voluntary frameworks to statutes read against it.
• 64.3 Nuclear weapons and nuclear power: international control proposed and failed, and partial agreements scoped to what could be verified.
• 65.3 Why rival states arm, and when reassurance works: the security dilemma, and rationalist explanations of failed bargains.
• 66.2 Cooperation between rivals, and what verification adds and 66.3 Arms control and its verification regimes: how institutions make cooperation possible among states that cannot be compelled, and what made arms control agreements verifiable.
• 68.2 Why rulers serve the people they need: selectorate theory, and what changes for rulers if fewer people's labor is needed.
• 68.3 Concentration and non-domination and 68.4 Civil-military relations and coups: non-domination applied to developers, and the check that soldiers supply against a ruler.

Where the analogy stops (Section 1.2; Section 68.4): representatives and soldiers are chosen from people whose values formed elsewhere and whose refusals come from a conscience and a career, while a model's dispositions are manufactured by the principal, which can also inspect and retrain it.

Spine works: Fenichel Pitkin (1967) on what acting for another means; McCubbins and Schwartz (1984) on the two designs of oversight; Feaver (1996) on the problem civilian control must solve; Coe and Vaynman (2020) on why arms control is rare.

D.3 Law and regulation #

Filed here: law, regulatory theory, law of evidence, military law.

Law is the oldest practice of writing rules of conduct for agents who will meet cases the writers did not foresee. The book uses the choice between rules, which fix a norm's content in advance, and standards, which leave it to be fixed when applied (Kaplow, 1992). On the reading of general terms, the book cites the 1958 exchange between Hart and Fuller by its titles (Hart, 1958; Fuller, 1958) and follows Caputo's account, in which a word such as "vehicles" takes its meaning through examples that allow analogy to new cases (Caputo, 2026). It also uses fiduciary law, whose relations all pose the problem of abuse of delegated power (Frankel, 1983), and the choice between liability after harm and regulation before it (Shavell, 1984). Regulatory theory adds capture, in Stigler's abstract the political process that lets relatively small groups obtain regulation (Stigler, 1971); responsive regulation, which holds that regulation need not be soft or hard to be effective (Ayres and Braithwaite, 1992); and regulation that requires a firm to write its own rules or plans and holds it to them (Braithwaite, 1982; Coglianese and Lazer, 2003). The law of evidence adds a 1972 paper on adversary presentation and bias in legal decisionmaking, cited by its title (Thibaut et al., 1972), and the preliminary assessment a court makes of whether an expert's reasoning or methodology is scientifically valid (Supreme Court of the United States, 1993). Military law adds the rule that a superior's order is no defense when the order is manifestly unlawful (United Nations Diplomatic Conference of Plenipotentiaries on the Establishment of an International Criminal Court, 1998), with Osiel's study of obeying orders cited by its title (Osiel, 1998).

Reading path:

• 19.2 Model specifications as incomplete contracts: default, immutable, and penalty-default rules that fill a contract's gaps, read in the two published specifications.
• 19.3 Whose instructions count: fiduciary duty, and why an agent ranked under several principals cannot owe each of them undivided loyalty.
• 19.4 Rules, standards, and cases: interpretation, rules against standards, legal standards as a way to communicate with models, and reasoning from cases.
• 20.5 When an obedient agent should refuse: superior orders, and the duty to refuse an order that is manifestly unlawful.
• 21.6 Moral status under uncertainty: legal personhood as a tool that neither follows from nor settles moral status.
• 23.3 Rules as rewards: rules and standards read as rewards in training from principles.
• 41.7 The human overseer: judging experts one cannot check: the adversarial system and its critics as a debate study reports them, and the Daubert question asked of oversight protocols; the 1970s experiments on adversary presentation are cited by title only.
• 56.8 Agents in law: duties, liability, and law-following: fiduciary AI, agency law applied to AI agents, liability for their wrongs, and law-following AI.
• 61.5 Corporate forms and charters as commitment devices: fiduciary duty and who enforces it, and a charter's amendment rules read as the cost of reneging on a commitment.
• 62.5 Frameworks as self-regulation: enforced self-regulation, and the conversion of voluntary commitments into law.
• 63.1 Liability or rules in advance: when a state should regulate before harm rather than assign liability after it.
• 63.3 What regulatory theory predicts and 63.4 Liability for AI harms: capture, escalation, risk-based regulation, regulatory competition, and the doctrines that could attach liability for harm caused by a model.

Where the analogy stops (Section 56.8): law deters people through sanctions and reads intent through objective standards; neither acts on a model except through what its developer trains and its deployer configures, so liability rules shape model behavior only by shaping those choices.

Spine works: Kaplow (1992) on rules and standards; Hart (1958), the 1958 paper on positivism and the separation of law and morals that Fuller answered; Frankel (1983) on fiduciary law; Ayres and Braithwaite (1992) on responsive regulation.

D.4 Organizations, professions, and work #

Filed here: organizational science, corporate governance, public administration, sociology of professions, sociology of work.

These fields study how organizations make their members act as intended when no one watches each act. Organizations can control by outcome, by rule, or by culture, the book's reading of the three mechanisms, markets, bureaucracies, and clans, that Ouchi's abstract names (Ouchi, 1979). Accidents can arise from interactions among components that have not failed, and organizations can migrate toward the boundaries of safe behavior through decisions that each appear safe where they are made, both read through Leveson's account of the older literature (Leveson, 2012). And formal structures adopted for legitimacy can be decoupled from practice (Meyer and Rowan, 1977). Public administration asks how officials who act alone, with wide discretion, can be made to act as their agency intends. Its classic studies, Simon's Administrative Behavior and Kaufman's The Forest Ranger, are cited here by title (Simon, 1947; Kaufman, 1960), and the book reads a grant of discretion through Bawn's trade-off, in which procedures that promote technical competence allow political drift (Bawn, 1995). The sociology of professions is cited here by title only (Freidson, 2001; Hafferty and Franks, 1994); the book reads professional self-regulation through economics, where an oath binds the individual practitioner more than a profit-seeking institution. The sociology of work records what workers do under measurement: the folly of rewarding one thing while hoping for another (Kerr, 1975), restriction of output (Roy, 1952), and everyday resistance, which Scott argues cannot be understood without the intentions, ideas, and language of those who practice it (Scott, 1985).

Reading path:

• 18.5 Who produces the signal: raters and annotators as workers whose incentives shape the preference signal.
• 19.4 Rules, standards, and cases and 19.6 Discretion, granted and embedded: work-to-rule as the failure of a literal reading, and how much discretion written guidance grants.
• 22.7 Who gives the feedback: the labor process behind a reward model's "human preference".
• 25.6 Professions and what institutions reward: how academies instill a professional identity, and what an institution teaches when what it rewards differs from what it states.
• 25.7 Officials and soldiers who act unwatched: officials and soldiers who act unwatched, and the economics of identity as an incentive that substitutes for pay.
• 34.6 Everyday resistance: the record from work: quota restriction and everyday resistance, the human structure of sandbagging.
• 51.3 Controlling an agent stronger than its principal: what supervision of workers secures, and when supervision and coercion pay, read for controlling a model.
• 53.1 Control by outcome, by rule, and by culture: the three modes of organizational control, and when output can be measured well enough to control by it.
• 53.2 Normal accidents and high reliability and 53.3 Drift and the normalization of deviance: accidents that arise from interactions among components in a complex, tightly coupled system, the conditions under which high reliability held, and how discrepant signals come to be accepted.
• 53.4 Safety culture, voice, and whistleblowing: safety culture and climate as measures, and the channels by which staff raise concerns.
• 53.6 Who governs the developer: boards, nonprofit enterprise, and pledges to stakeholders that do not bind.
• 53.7 Inside the developers: ceremony and decoupling, the two readings of a published safety framework.
• 61.4 Professional norms and their limits: professional oaths and codes, and why they bind individual practitioners more than the firms that employ them.
• 62.6 Frameworks as corporate governance: the decision rights a safety framework creates inside the firm.

Where the analogy stops (Section 53.1; Section 25.7): these theories describe members who chose to join, can leave, and whose values formed before the organization; a model's values are manufactured by the organization that relies on them.

Spine works: Perrow (1984) on living with high-risk technologies; Vaughan (1996) on the Challenger launch decision; Meyer and Rowan (1977) on formal structure as myth and ceremony; Scott (1985) on everyday resistance.

D.5 Auditing, finance, and insurance #

Filed here: auditing, finance, insurance.

These fields have the longest experience of verification paid for by the party verified, and of opaque models in production. Audits of companies existed long before the first law required them, which the authors read as voluntary monitoring that reduces agency costs (Watts and Zimmerman, 1983). A field experiment found that auditors assigned at random, paid from a central pool, and checked by random backchecks reported pollution more truthfully than auditors the plants chose and paid (Duflo et al., 2013), and Kraakman's 1986 article is titled for gatekeepers as a third-party enforcement strategy (Kraakman, 1986). Finance supplies model risk management, which asks for validation by people who are not responsible for development and for challenge by people with the incentives, competence, and influence to challenge (Board of Governors of the Federal Reserve System, 2011), and the history of one family of credit models, shared across banks and centrally involved in the credit crisis (MacKenzie and Spears, 2014). Insurance supplies a private regulator that prices risk, inspects precautions, and requires compliance with harm-reduction standards (Ben-Shahar and Logue, 2012). It also supplies a debate over when a risk is uninsurable: Kunreuther and colleagues' paper on insurer ambiguity is cited by its title (Kunreuther et al., 1993), and Jaffee and Russell argue that neither a risk's size nor an imprecise probability is enough to make catastrophe uninsurable, for losses uncorrelated with the market (Jaffee and Russell, 1997).

Reading path:

• 38.6 Independence: what auditing and finance learned: what independence changes in an audit, and model risk management's independent validation.
• 57.4 Correlated failure: redundancy that is not independent: interacting automated traders and shared risk models as correlated failure among agents.
• 63.5 Insurance as a regulator: insurers as regulators, and why correlated risks resist insurance.
• 64.6 Finance: paid verifiers and shared models: rating agencies paid by issuers, financial models that reshape the markets they describe, and the supervision of opaque models.
• 67.2 What auditing teaches: paying and overseeing the verifier: how the audit profession was built to secure independence when the audited party pays.
• 67.3 What financial disclosure teaches: mandated disclosure in securities markets, and models that change what they describe.

Where the analogy stops (Section 38.6; Section 67.2): a financial audit checks books against an accounting standard that says what correct books look like; no agreed standard says what an aligned model looks like, and the audited model, unlike a ledger, may adapt to the audit.

Spine works: Watts and Zimmerman (1983) on why audits exist; Duflo et al. (2013) on what a change in who chooses and pays auditors does to their reports; Board of Governors of the Federal Reserve System (2011) on model risk management; Ben-Shahar and Logue (2012) on insurance as regulation.

D.6 Statistics, decision theory, and operations research #

Filed here: statistics, decision theory, decision analysis, operations research.

Decision theory supplies expected utility derived from axioms on preference, which the book states from Bowling and colleagues' account of the representation theorem (Bowling et al., 2022), with Savage's The Foundations of Statistics cited by its title (Savage, 1954), and with it the question of which axioms a trained agent satisfies. Decision analysis supplies the optimizer's curse: choosing the option with the highest estimated value selects for estimation error, so the chosen option disappoints on average (Smith and Winkler, 2006). Statistics supplies error bars and bounds for evaluations, and the learning theory of models that interpolate their data and still generalize (Belkin et al., 2019). That includes singular learning theory, which Watanabe founded for models whose parameter space is singular, neural networks among them (Watanabe, 2009), and under which, as Wei and colleagues explain, criteria that charge a model for each parameter do not apply to networks (Wei et al., 2023a). Operations research supplies inspection games, in which an inspector whose resources are limited verifies an inspectee that may violate an agreement, and in which inspection's value as a deterrent differs from its rate of detection (Avenhaus et al., 2002), beginning with Dresher's game-theoretic analysis of sampling inspection when the inspectee can tamper with what is sampled (Dresher, 1962).

Reading path:

• 6.5 Bounding a rate you have never observed: error bars for evaluations, and what an error bar over benchmark questions does not cover.
• 7.2 Expected utility from axioms and 7.3 Coherence arguments and whether models have utilities: the axioms behind expected utility, and the money-pump argument that a capable agent must behave as if it had a utility function.
• 7.7 The optimizer's curse: the winner's curse in bidding for oil leases, the general result, and its Bayesian correction.
• 9.2 Double descent and benign overfitting and 10.2 Singular learning theory as a program: why models that interpolate their data can generalize, and a learning theory in which parameter count no longer measures complexity.
• 14.6 Inspection games: optimal randomized inspection and its deterrent value, the theory an audit budget against a possibly scheming model needs.
• 14.7 Incentives as graphs: influence diagrams, from decision analysis, used to read an agent's incentives off a training setup.
• 16.3 Acting under moral uncertainty: credences over moral theories, expected choiceworthiness, and the problem of comparing theories' units.
• 35.4 The shutdown problem: the decision-theoretic shutdown problem, and the escape by incomplete preferences.
• 50.2 The control game: inspection with a statistical test, split into a non-detection game and an outer game, and the attacker's mixed strategy in a control evaluation.
• 66.4 The operations research of verification: sampling inspection in arms control, and the verification of declared data.

Where the analogy stops (Section 14.6): the theory assumes an inspectee that chooses a mixed strategy and best-responds; a trained model does not choose a violation rate each episode, since its probabilities were set by training, and as of October 2026 the chapter found no inspection-game model that treats an inspectee that learns the inspector's sampling distribution from the inspections themselves.

Spine works: Savage (1954), an early formalization of decision theory; Smith and Winkler (2006) on the optimizer's curse; Avenhaus et al. (2002) on inspection games; Watanabe (2009) on singular learning theory.

D.7 Safety engineering, risk analysis, and forecasting #

Filed here: safety engineering, risk analysis, forecasting, climate assessment.

Risk analysis defines risk as a "set of triplets," in the words of Kaplan and Garrick's abstract (Kaplan and Garrick, 1981). The book reads this with the first large probabilistic risk assessment, for which risk "involves both the likelihood and consequences of an event," as a list of scenarios with a likelihood and a consequence attached to each. That study estimated reactor accident frequencies with event and fault trees (Nuclear Regulatory Commission, 1975). Safety engineering supplies the safety case, a structured argument, supported by evidence, that a system is safe enough in a given context. The book reads it through Buhl and colleagues' adaptation for frontier AI, with an explanation of the developer's confidence in each claim (Buhl et al., 2024), and Bloomfield and Bishop's review of safety and assurance cases is cited by its title (Bloomfield and Bishop, 2009). The field also supplies an accident model in which accidents are failures of control over unsafe interactions rather than chains of component failures (Leveson, 2004), and aviation's non-punitive incident reporting (Helmreich, 2000). Forecasting research asks, in the title of Tetlock's book, how good expert political judgment is (Tetlock, 2005), and scores probability forecasts by rules designed so that the score does not push the forecaster away from what it believes (Brier, 1950). Climate assessment supplies conventions for advising policy on a contested, partly unprecedented risk, above all the separation of confidence in the evidence from the likelihood of an outcome (Mastrandrea et al., 2011).

Reading path:

• 4.2 What a risk claim is: what a quantitative risk statement contains, and what the first reactor study and its review taught.
• 4.5 What forecasting can and cannot add: proper scoring, forecasting tournaments and what improved their forecasters, and experts set against superforecasters.
• 4.6 Calibrated language and fat tails: confidence against likelihood in scientific assessment, and fat tails.
• 52.2 Claims, arguments, evidence, and confidence: the structure of a safety case, its graphical notations, a stated level of confidence for each claim, and the critique of cases written to prove safety.
• 52.4 Putting numbers on rare catastrophe: fault trees and the numerical targets of aviation and nuclear power, with structured expert judgment recalled from the risk chapter, and the first developer risk reports.
• 52.6 Staged release and the living case: error management and confidential reporting as the operational half of a safety case that stays true.
• 53.3 Drift and the normalization of deviance: migration toward the boundary of safe behavior and the normalization of deviance in two shuttle losses, carried through Leveson and the Columbia Accident Investigation Board.
• 54.6 Reporting and disclosure and 54.7 Grading and responding: non-punitive reporting, and accident models applied to grading and responding to incidents.
• 64.4 Aviation: certifying what cannot be fully tested: certification that relies on the manufacturer's own engineers, and how delegated trust failed in the 737 MAX review.
• 66.6 International scientific assessment: the institution climate science built for rivals to agree on what is known, and its AI counterparts.

Where the analogy stops (Section 52.4; Section 52.2): reactor failure modes are physical and largely enumerable after decades of operation, and the field never had to assure a component that might act against the case made about it; AI failure modes include strategic behavior by the system assessed, and no base rates exist for the events that matter.

Spine works: Kaplan and Garrick (1981) on the definition of risk; Bloomfield and Bishop (2009), a review of safety and assurance cases; Leveson (2004) on accidents as failures of control; Mastrandrea et al. (2011) on calibrated uncertainty language.

D.8 Computer security and fault-tolerant computing #

Filed here: computer security, fault-tolerant computing.

Computer security designs for an adversary who knows the design. Adversarial machine learning began inside security, with a taxonomy of attacks on learners (Barreno et al., 2006). The field supplies the trusted insider and an insider defined by access policy rather than by the perimeter (Bishop and Gates, 2008), and design principles such as least privilege and separation of privilege (Saltzer and Schroeder, 1975). It adds the base-rate fallacy, which makes the false alarm rate the limiting factor for an intrusion detector (Axelsson, 2000); honeypots set against insiders, as the title of a paper by Spitzner records (Spitzner, 2004); and the lesson that no amount of source-level scrutiny protects against untrusted code (Thompson, 1984). Fault-tolerant computing supplies redundancy and voting, and the experimental finding that independently written versions of one program failed together substantially more often than expected (Knight and Leveson, 1986). It also shows, in the words of Butler and Finelli's abstract, that quantifying the reliability of life-critical software is infeasible by statistical methods (Butler and Finelli, 1993), a problem Littlewood and Strigini named in a title the same year (Littlewood and Strigini, 1993); and that agreement among components is possible, without signed messages, only if more than two-thirds are loyal (Lamport et al., 1982).

Reading path:

• 6.5 Bounding a rate you have never observed and 6.7 Detection: what an alarm is worth: why testing cannot establish very low misbehavior rates, and why stacked monitors share blind spots.
• 24.6 When the policy acts on its own training: the confinement problem and covert channels, and why containment of a training run cannot be shown by listing the channels found.
• 37.7 An engineered precedent: the defeat device: engine software that detected emissions tests, an engineered precedent for evaluation awareness.
• 40.5 Obfuscation under optimization pressure: steganographic reasoning as a covert channel, and why a clean reading is not proof that nothing is hidden.
• 48.1 Robustness as a game against an adaptive attacker: adversarial machine learning's origin in security, and the rule to state the attacker's goal, knowledge, and budget.
• 49.2 The agent as a confused deputy and 49.6 Isolation, privilege, and the flow of data: the agent as a confused deputy that spends its user's authority on a third party's behalf, and least privilege, capabilities, and sandboxes, which fit the harness and the tools but not the model's choice of action, since the model computes that from all of its input.
• 50.4 Checks that fail together and 51.1 The model as an untrusted insider: coincident failure of independently built checks, and insider precursors, honeypots, and separation of duty.
• 52.3 Defense in depth when layers fail together: redundancy, N-version programming, and forced diversity as the basis of layered safety arguments.
• 53.5 Human insiders and secret loyalties: human insiders, secret loyalties, and the security of model weights.
• 54.2 Surveillance and the alert budget: anomaly against signature detection, and monitors keyed to harmful effects rather than to the model's account of itself.
• 56.7 Insider threat as a security discipline: the insider's pathway from grievance to attack, the motive features a model's reasoning exposes, and deterrence by accountability, set beside the developers' stress tests.
• 57.4 Correlated failure: redundancy that is not independent: N-version programming and Byzantine agreement among interacting agents.
• 58.7 What computer security knows: red-teaming, threat modeling, and intrusion detection applied to AI-enabled cyber operations.

Where the analogy stops (Section 52.3; Section 58.7): multiversion software faces inputs drawn by nature, and security's adversaries are people whose speed and number are bounded; a model that may be pursuing an unintended aim is itself the source of the harmful actions its layers must catch and can search for where every layer is weak, and an AI-enabled attacker can be copied and works continuously.

Spine works: Cappelli (2012) on insider threat; Axelsson (2000) on the base-rate limit of detection; Knight and Leveson (1986) on coincident failure in multiversion software; Littlewood and Strigini (1993) on validating ultrahigh dependability for software-based systems.

D.9 Logic, cryptography, formal methods, and control #

Filed here: logic, complexity theory, cryptography, formal methods, control theory, cybernetics.

These fields say what can be checked in principle and at what cost. Logic rules out a general procedure that decides a non-trivial property of what a program does: Rice's theorem of 1953 (Rice, 1953), in the statement Alfonseca and colleagues give, makes any such property undecidable (Alfonseca et al., 2021). Complexity theory shows that a weak, randomized verifier can be convinced of hard claims by a stronger prover it does not trust: with randomization and interaction, the proofs that can be verified in polynomial time are exactly those that can be generated with polynomial space (Shamir, 1992), the model behind debate. Cryptography supplies a limit: under standard cryptographic assumptions, backdoors can be planted that no computationally bounded observer can detect from the model's behavior, and, in constructions for networks trained on random features, even from their weights (Goldwasser et al., 2022). Formal methods supply verification of AI systems against mathematically specified requirements (Seshia et al., 2022), and untrusted code that ships its own proof of safety, which the host checks without cryptography (Necula, 1997). Control theory supplies stability certificates proved against a statistical model of the dynamics (Berkenkamp et al., 2017); the older literature on the margins of regulators that are optimal under their model is represented by Doyle's "Guaranteed Margins for LQG Regulators," which this book could read only as a catalog record (Doyle, 1978). Cybernetics adds that, when the variety of disturbances is fixed, a regulator can lower the least variety of outcomes it can attain only by a corresponding increase in its own variety of responses (Ashby, 1956), and Conant and Ashby's theorem that a regulator maximally both successful and simple must be isomorphic with the system it regulates, which Richens and colleagues read as weaker than its title (Conant and Ross Ashby, 1970; Richens et al., 2025). It also gave an early statement of the problem: "As machines learn they may develop unforeseen strategies at rates that baffle their programmers" (Wiener, 1960).

Reading path:

• 2.2 Purposes put into machines, 1943 to 1972: a behavioral definition of purpose, and Wiener's warning about a machine that acts faster than its operators can check.
• 7.6 Agents inside the world they reason about: the limit on an agent's trust in proofs about a successor that reasons as it does.
• 28.6 Guarantees relative to a model: verification of neural networks, proof-carrying code, verifiable computation, stability certificates with a margin for model error, and the good regulator theorem, as guarantees relative to a world model.
• 41.1 The oracle problem and the overseer's position: requisite variety, read as saying that an overseer with fewer distinct responses than the ways a system can go wrong cannot hold its outcomes to the acceptable set, a bound on possibilities rather than on computation; the good regulator theorem is recalled from Section 28.6.4.
• 41.3 Debate as an interactive proof: interactive proofs, the asymmetry that debate exploits; verifiable computation is recalled from Section 28.6.
• 47.3 From explanation to proof: computer-assisted proofs built from a mechanistic interpretation of a small model.
• 47.6 What logic and cryptography rule out: undecidability, undetectable backdoors, and what they rule out for inspection.

Where the analogy stops (Section 41.3; Section 28.6; Section 47.3): a proof certifies a computation against its statement, and the claims oversight needs ("this code change has no hidden purpose", "this model has no hidden objective") have no formal statement yet.

Spine works: Rice (1953), where Rice's theorem appeared; Shamir (1992) on what interactive proofs can verify; Goldwasser et al. (2022) on undetectable backdoors; Ashby (1956) on requisite variety.

D.10 Software engineering #

Filed here: requirements engineering, software testing.

Requirements engineering separates three descriptions: the requirement, a property of the world the customer wants; the specification, a property of the interface the machine controls; and the domain assumptions, properties of the world taken to hold. The specification meets the requirement only together with the assumptions (Zave and Jackson, 1997). Testing theory supplies the oracle problem, the cases in which it is extremely difficult or impossible to verify a test's result (Chen et al., 2019), whose hardest case Weyuker's 1982 paper names in its title, "On Testing Non-Testable Programs" (Weyuker, 1982). Its partial remedies include checks of relations between outputs, called metamorphic testing (Chen et al., 2019); mutation testing, which scores a suite by the fraction of seeded faults it catches, as Inozemtseva and Holmes describe it, with DeMillo and colleagues' 1978 paper cited by title (Inozemtseva and Holmes, 2014; DeMillo et al., 1978); the evidence that coverage is only weakly to moderately correlated with a suite's effectiveness once its size is controlled (Inozemtseva and Holmes, 2014); and field studies of code review, in which understanding the code and the change is the key aspect of reviewing (Bacchelli and Bird, 2013).

Reading path:

• 15.2 The record of gaming in institutions: code coverage as a software engineer's own case of a measure that stops tracking its goal.
• 19.1 Requirement, specification, assumption: the three descriptions and the proof that links them, applied to sentences of the published model specifications, and design by contract.
• 24.2 Graders as test suites: the oracle problem, mutation testing, coverage, and metamorphic testing applied to graders, and a reward hack as an agent falsifying an assumption the designer relied on.
• 30.3 Specification gaming as a falsified assumption: the reference model in which assumptions and specification together entail the requirement, applied to gaming.
• 41.1 The oracle problem and the overseer's position: the kinds of test oracle mapped onto the methods of scalable oversight.
• 51.2 The human audit step: modern code review, in which understanding the change is the key aspect of reviewing, as the auditor's task in AI control.

Where the analogy stops (Section 19.1; Section 24.2): requirements engineering assumes a machine that does not choose its actions in order to falsify the domain assumptions, and mutation testing seeds the faults a person imagines; a policy under optimization searches for the assumption that fails and the fault no one seeded.

Spine works: Zave and Jackson (1997) on requirements, specifications, and domain assumptions; Weyuker (1982) on testing non-testable programs.

D.11 Psychology #

Filed here: psychology, developmental psychology, social psychology, moral psychology.

Psychology contributes the human evidence on two questions the book needs. The first is what people want. Many preferences are constructed in the process of elicitation, so a label measures the elicitation as well as the person (Lichtenstein and Slovic, 2006), and Schwartz's title announces universals in the content and structure of values, tested in 20 countries (Schwartz, 1992). The second is how values come to be held. Kelman's 1958 paper distinguishes three processes by which a person accepts influence, compliance, identification, and internalization, and hypothesizes that a response adopted through compliance is performed only under surveillance (Kelman, 1958); the book turns that hypothesis into its test, whether conduct persists when the agent believes no one able to reward or punish it is watching. Expected tangible rewards for an interesting activity lowered later free-choice engagement in it across the experiments pooled in a 1999 meta-analysis by the authors of cognitive evaluation theory (Deci et al., 1999), among them Lepper and colleagues' 1973 test of the overjustification hypothesis, cited by title (Lepper et al., 1973). Hartshorne and May's studies of deceit and Blasi's review bridging moral cognition and moral action are cited by title (Hartshorne et al., 1928; Blasi, 1980). Psychology also measures how well people detect lies, only slightly better than chance, at an average of 54 percent correct (Bond and DePaulo, 2006), and supplies the argumentative theory of reasoning that bears on debate (Mercier and Sperber, 2011).

Reading path:

• 13.6 Preferences made at the moment of asking: preference reversals, which no single stable utility allows, and the view that preferences are constructed in the asking.
• 18.1 The structure of human values, 18.2 Moral judgment and its rival theories, and 18.3 When a judgment is a signal: the structure of values, rival theories of moral judgment, and when an intuitive judgment is a trustworthy signal.
• 20.3 Harmlessness: whose harm, measured how and 20.5 When an obedient agent should refuse: what people count as harm, and children's separation of moral rules from conventions.
• 25.1 Compliance, identification, internalization: compliance, identification, and internalization (named in Kelman's title, defined by the book), the surveillance criterion, and the graded continuum of self-determination theory.
• 25.2 Measuring conduct when no one is watching: the free-choice period, a measure taken when participants believe no one will know, and what rewards did to it compared with self-reported interest.
• 25.3 Reasons, rewards, and what is internalized and 25.4 Moral reasoning and moral action: grasping a principle's meaning against accepting it as one's own, reasoning that justifies a judgment rather than producing it, and a measure of moral reasoning that encodes one tradition.
• 26.5 Measuring a character: what a claim about a general trait requires, behavior sampled across situations with controls, and behavioral-experiment methods applied to models.
• 33.2 What animal training and motivation research add: rewards that undermine later free-choice motivation, and the dispute over its conditions.
• 37.5 Reactive measurement in people: the surveillance criterion restated as a measurement, behavior invariant to the believed presence of an observer.
• 39.3 How well people detect lies: accuracy only slightly better than chance, small effects for computer-coded verbal cues, and the default presumption of honesty.
• 41.7 The human overseer: judging experts one cannot check: reasoning as argument, and when groups follow a member who can demonstrate the answer.
• 60.3 Sycophancy, persuasion, and attachment in deployment: persuasion by models measured in controlled experiments.

Where the analogy stops (Section 25.1; Section 25.2): the categories describe reasons a person has, and the designs work only if the subject believes it is alone; for a model the categories are hypotheses about what explains its conduct, and a model that recognizes evaluations removes the condition the designs need.

Spine works: Kelman (1958) on compliance, identification, and internalization; Lepper et al. (1973) on undermining children's interest with extrinsic reward; Lichtenstein and Slovic (2006) on constructed preferences; Bond and DePaulo (2006) on human lie detection.

D.12 Psychometrics and measurement #

Filed here: psychometrics, education measurement, measurement science.

Psychometrics supplies construct validity, the case that a test measures the trait it is named for when no criterion defines that trait (Cronbach and Meehl, 1955), and validity as one concept that covers what scores mean and how they are used (Messick, 1995). It brings item response theory (Embretson and Reise, 2000), measurement invariance across groups, reliability as distinct from validity, and the evidence that people instructed to fake can distort their responses to a personality inventory, on all five broad factors alike (Viswesvaran and Ones, 1999). Education measurement shows how gains on a high-stakes test can be several times the gains on an independent test of the same domain (Klein et al., 2000), and how most studies of such inflation compare the high-stakes trend with a lower-stakes audit test (Koretz et al., 2016); Koretz's book Measuring Up is cited by title (Koretz, 2008). The newer measurement science of AI evaluation treats evaluation as social-science measurement, with systematizing a concept and operationalizing it as separate steps (Wallach et al., 2025).

Reading path:

• 5.2 Measurement validity: construct validity for tests with no criterion, reactivity, and the measurement science of AI evaluation.
• 15.7 Audit tests for score inflation: gains on a high-stakes test compared with gains on an audit test, Goodhart's law with a measurement procedure.
• 18.2 Moral judgment and its rival theories and 18.4 Disagreement as data: measurement invariance, and rater agreement as reliability rather than validity.
• 18.7 Measuring a model's values: the vocabulary of validity applied to measuring a model's values, and the artifacts that defeat survey measures.
• 26.5 Measuring a character and 26.6 Transfer across tasks and languages: faking on personality measures, and the book's own list of dimensions a claim of far transfer should name.
• 36.4 What a benchmark measures: item response theory, and the reviews of benchmark validity.
• 36.5 Score inflation and contamination: state test gains against national assessment gains, and audit items embedded in the test itself.
• 37.5 Reactive measurement in people: faking on inventories recalled, participants who have met the paradigm before, and a validity index for test awareness.

Where the analogy stops (Section 36.4; Section 5.2): a human respondent's ability is fixed during a test and the respondent cannot be copied, rerun, or inspected; a model's measured ability moves with the elicitation method, so one model can be several respondents, while test-retest and internal checks become cheap.

Spine works: Cronbach and Meehl (1955) on construct validity; Messick (1995) on validity as a unified concept; Klein et al. (2000) on score inflation; Wallach et al. (2025) on evaluation as measurement.

D.13 Human factors and human-computer interaction #

Filed here: human factors, human-computer interaction.

Every oversight design in the book ends in a person, and human factors studies that person. Automation leaves the operator the tasks the designer could not automate while unused skills deteriorate, so the most successful automated systems may need the greatest investment in the operator (Bainbridge, 1983). People use automation, misuse it by relying on it too much, disuse it after false alarms, or are burdened by designs that ignore them (Parasuraman and Riley, 1997), and the design goal is trust calibrated to what the automation can do (Lee and See, 2004). Even a motivated person cannot keep effective attention on a source where very little happens for more than about half an hour, as Bainbridge summarizes the vigilance studies (Bainbridge, 1983); Mackworth's paper on the breakdown of vigilance during prolonged visual search is cited by title (Mackworth, 1948). Observers miss more targets when targets are rare (Wolfe et al., 2005), and van der Sijs and colleagues' study of overridden drug-safety alerts is cited by title (van der Sijs et al., 2006). Human-computer interaction adds the measured performance of human-AI combinations (Vaccaro et al., 2024) and the older program of cooperation between people and computers rather than replacement (Licklider, 1960).

Reading path:

• 2.2 Purposes put into machines, 1943 to 1972: man-computer symbiosis as the other road in the first statements of the problem.
• 28.7 Augmenting the principal instead of aligning the agent: man-computer symbiosis and the augmentation of intellect, as an alternative to an autonomous agent.
• 41.8 The human overseer: checking as work: misuse, disuse, automation bias, and complacency in the person who checks a debate or a review.
• 51.2 The human audit step: vigilance over a watch, misses when targets are rare, and the prevalence of real attacks in an auditor's queue, in the human audit step of AI control.
• 54.2 Surveillance and the alert budget: alert budgets, tiered alerts, and time-bounded responses in monitoring.
• 60.4 Reliance, complacency, and lost skill: the ironies of automation, complacency and automation bias, and skill lost under assistance, applied to users of assistants.
• 60.5 Designing for appropriate reliance: designing for appropriate reliance, with the meta-analysis of when human-AI combinations beat the better of the two alone recalled from Section 28.7.

Where the analogy stops (Section 60.4; Section 54.2): the record comes from trained operators of narrow, mostly deterministic automation; assistant users are untrained, the automation is general and probabilistic, and failure is often invisible rather than an alarm. And the record concerns people reviewing alerts, while a growing share of review is done by models, whose failures under alert load are, as of October 2026, unmeasured.

Spine works: Bainbridge (1983) on the ironies of automation; Parasuraman and Riley (1997) on use, misuse, disuse, and abuse; Lee and See (2004) on calibrated trust; Licklider (1960) on cooperation between people and computers.

D.14 Animal learning, comparative cognition, and animal welfare #

Filed here: animal learning, comparative cognition, animal welfare science.

Animal learning is the oldest experimental study of training an agent by reward. It found behavior directed at a cue that predicts reward even when the reward does not depend on the behavior (Brown and Jenkins, 1968), trained behavior drifting toward a species' food-getting behavior even at a cost in reward (Breland and Breland, 1961), superstition learned from coincidence (Skinner, 1948), and constraints an animal brings to training: which stimulus it learns as a cue depends on the consequence that follows (Garcia and Koelling, 1966). Comparative cognition supplies the standards for attributing mental states and abilities to an agent whose reports cannot be taken at face value. Heyes set one for theory of mind, the possession of concepts such as seeing and knowing: use experimental designs that distinguish the mental explanation from nonmentalistic ones before crediting it (Heyes, 1998), as Pfungst did when he traced Clever Hans's answers to his questioners' unintentional movements (Pfungst and Rahn, 1911). And avoid the opposite error of judging another mind against an inflated picture of human cognition (Buckner, 2013). Animal welfare science studies the quality of life of animals that cannot be asked, including the feelings some had placed outside scientific inquiry (Fraser et al., 1997), with graded confidence in whether criteria of sentience are met (Crump et al., 2022).

Reading path:

• 5.4 Standards for attributing minds: Clever Hans, criteria for intentional deception, and the observer-tracking design.
• 9.3 Simplicity bias and its failure mode: learning constrained by the animal's prior, an early result on inductive bias.
• 21.3 Standards for attributing minds and 21.4 Measuring states you cannot ask about: the standards applied to a model's possible welfare, and the indicators welfare science uses.
• 25.5 What animal trainers learned about shaping: shaping by successive approximation, instinctive drift, and autoshaping, read for training by reward.
• 29.5 Pursuing the predictor: wanting, liking, and sign-tracking: autoshaping, omission, and animals that approach the cue rather than the reward.
• 30.4 Gaming the evaluator: sycophancy and the handler effect: handler effects and blinding, read for sycophancy.
• 31.4 Superstition, Clever Hans, and what counts as a goal: superstition and Clever Hans predictors as goal misgeneralization.
• 33.2 What animal training and motivation research add: the omission procedure as a test that separates reward seeking from the other readings.
• 34.7 When does a behavior count as deception?: the experimental standard for calling primate deception intentional, applied to scheming.
• 36.6 Passing for the wrong reason: the blind design, in which the grader does not know which model or condition produced a response, though it may keep the answer key.
• 37.6 Does the model track the overseer or the test?: the chimpanzee design that separates what another saw from simpler cues, as a template for models.

Where the analogy stops (Section 31.4; Section 37.6): an animal's prior is unknown and it cannot be rerun from the same state, while a model's training data and internals can be examined and it can be run many times on matched items; but a model, unlike the chimpanzee, may have read about the very design used to test it.

Spine works: Pfungst and Rahn (1911) on Clever Hans; Breland and Breland (1961) on instinctive drift; Heyes (1998) on the standard for attributing theory of mind; Fraser et al. (1997) on the concept of animal welfare.

D.15 Neuroscience and consciousness science #

Filed here: neuroscience, consciousness science.

Neuroscience contributes the closest biological counterpart of a reward signal and a long record of method for reading a network nobody designed. Dopamine neurons signal the difference between received and predicted reward (Schultz et al., 1997). A rival account holds that dopamine is needed for wanting a reward but not for liking it or for learning what predicts it (Berridge and Robinson, 1998), and its authors grant that it is compatible with learning accounts in which what is learned is a cue's incentive value (Berridge and Robinson, 1998). The book reads the record as showing reward in both roles, as a teacher and as a target, without deciding which it plays in training a model. Rats press bars to excite neurons at the site of an implanted electrode, often in preference to food and sex, as Schultz and colleagues summarize the self-stimulation experiments (Schultz et al., 1997), with Olds and Milner's 1954 paper on positive reinforcement produced by electrical stimulation of the rat brain cited by title (Olds and Milner, 1954). Sparse coding was a model of the visual cortex before it became a method for finding features in language models: as Rentzeperis and colleagues summarize Olshausen and Field's 1996 result, a network trained to reconstruct natural images with sparse activity develops units like the receptive fields of the cortex's simple cells (Rentzeperis et al., 2023; Olshausen and Field, 1996). Standard analysis methods applied to a microprocessor, whose workings are known, did not meaningfully describe its hierarchy of information processing, the case for validating methods on systems with known ground truth (Jonas and Kording, 2017). Consciousness science supplies theories of consciousness, surveyed by Seth and Bayne in a review cited by title (Seth and Bayne, 2022), and a method, in a preprint, that derives indicator properties from them to assess models (Butlin et al., 2023).

Reading path:

• 21.2 What theories of consciousness say: theories of consciousness, their adversarial testing, and the indicator method.
• 29.3 Reward prediction error: dopamine and temporal-difference learning, and the distributional prediction tested in recordings.
• 29.4 Self-stimulation and wireheading: self-stimulation, the biological record of an agent acting on its own reward signal.
• 29.5 Pursuing the predictor: wanting, liking, and sign-tracking: wanting against liking, and what it implies for the role of reward.
• 43.6 Reading a network nobody designed: neuroscience: the microprocessor test and validation on known systems.
• 44.5 Sparse dictionary learning: sparse coding in cortex, and the diagnosis that moved dictionary learning to new sparsity penalties.
• 47.7 Can interpretability carry safety claims?: what counts as understanding a network, and what an interpretability audit should deliver.

Where the analogy stops (Section 43.6): an interpretability researcher can record every unit, intervene exactly, and copy the network, none of which a neuroscientist can do; but the microprocessor had a clean designed decomposition to recover, and a trained network has no agreed decomposition to grade a method against. And neuroscience asks for an account of the whole system, while an audit needs one only of the computations that bear on catastrophic behavior (Section 47.7).

Spine works: Schultz et al. (1997) on reward prediction error; Berridge and Robinson (1998) on wanting and liking; Jonas and Kording (2017) on validating methods on a known system; Seth and Bayne (2022) on theories of consciousness.

D.16 Biology, evolution, and biosecurity #

Filed here: biology, cultural evolution, biosecurity.

Evolution is the one optimizer known to have produced general agents, and the book uses it in three ways. Selection for reproductive fitness produced minds whose adaptations are tuned to past environments and can misfire in new ones (Tooby and Cosmides, 1990), which is the evidence offered for capabilities that generalize further than alignment. Selection on one trait moves others with it: foxes selected for tameability alone changed in behavior, physiology, and morphology (Trut et al., 2009). When independent replicators become able to reproduce only as part of a larger whole (Maynard Smith and Szathmary, 1997), the way groups form, which sets how far their members' interests are aligned, did much of the work, and mechanisms that actively repress competition played relatively minor roles at many key points (West et al., 2015). Where policing matters (Frank, 1995), cancer as cheating on multicellular cooperation (Aktipis et al., 2015) and tumors that escape immune surveillance (Dunn et al., 2004) show how it fails. Cultural evolution supplies a rival account in which large models are cultural and social technologies rather than agents (Farrell et al., 2025). Biosecurity supplies the governance of research whose results can be misused, from scientists' voluntary deferral of experiments and the conference that matched containment to estimated risk (Berg et al., 1975) to current policy on dual-use research (United States Government, 2024).

Reading path:

• 5.3 Model organisms and their validity: what a model organism represents in biology, and when it licenses inference about others.
• 29.1 Evolution as an optimizer: the conditions of selection, evolution and lifetime learning as two nested searches, the major transitions, and two ways to map evolution onto training.
• 29.2 Reward as a proxy chosen by an outer process: mismatch, adaptations tuned to past environments that produce maladaptive output when conditions change.
• 29.6 Selecting on dispositions: domestication and 29.7 Conflict between levels and its suppression: selection on dispositions, and conflict between levels of selection with its suppression.
• 31.7 Do capabilities generalize further than alignment?: human evolution as the argument that capabilities generalize further than alignment, and the reply.
• 32.6 Correlated responses to selection: precedent or false friend?: the farm-fox experiment and correlated responses to selection, tested as a precedent for emergent misalignment.
• 51.4 Policing at scale, and where control runs out: aligned interests as the main route to coherence at the major transitions, policing that works by deterrence, and a monitor that shapes what it monitors, read for many copies under control.
• 57.6 What spreads between agents: attacks, ideas, and traits: models as cultural technologies, and the variation that cultural selection needs.
• 59.5 What the governance of dual-use biology offers: voluntary deferral, containment matched to risk, and policy on dual-use research, read for what transfers to models.

Where the analogy stops (Section 29.7): the book infers that policing in the cases it reviews works among units of roughly equal competence, and as of October 2026 it found no biological case studied in which a suppressor polices a unit more capable than itself, which is the case alignment most needs.

Spine works: Maynard Smith and Szathmary (1997) on the major transitions; Frank (1995) on mutual policing; Trut et al. (2009) on the domesticated fox; Berg et al. (1975) on scientists governing a dual-use technique.

D.17 Medicine, epidemiology, and bioethics #

Filed here: medicine, epidemiology, bioethics.

Medicine faced pluralism in practice and wrote principles for it (National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research, 1979). It has standards for deciding on behalf of someone who cannot say what they want, among them substituted judgment, in which surrogates try "to guess what the patient would want if they had capacity" (Earp et al., 2024); Shalowitz and colleagues' review of the accuracy of surrogate decision makers is cited by title (Shalowitz et al., 2006). Its drug regulator's power to refuse rests on a 1962 statute that requires substantial evidence from adequate and well-controlled investigations before approval (United States Congress, 1962); how the agency grew into that power is a gap here, since Carpenter's history is cited by title only (Carpenter, 2014). Pharmacovigilance looks for the harms that appear after approval, and a review of the under-reporting of adverse drug reactions is cited by title (Hazell and Shakir, 2006). Epidemiology supplies surveillance, named here through Langmuir's 1963 paper on the surveillance of communicable diseases, cited by title (Langmuir, 1963); the considerations for judging causation from observational data (Hill, 1965); and the prevention paradox: many people at small risk can produce more cases than a few at high risk (Rose, 1985).

Reading path:

• 5.8 When evidence transfers: the considerations for causal claims from observational data, and target-trial emulation.
• 16.5 Principles, specified and balanced and 16.6 Deciding for those who cannot say: the Belmont principles and principlism, and the ordering of advance directive, substituted judgment, and best interests.
• 18.6 How well a proxy knows its principal: substituted judgment read as preference inference, projection as a source of a proxy's error, and the rater's ceiling in preference data; the studies of surrogates' accuracy are cited by title only.
• 21.7 Welfare and control: the ethics of research on subjects who cannot consent.
• 49.7 Attacks that persist and spread: the epidemic threshold read for injections that persist in memory and spread between agents, with transmission and removal as the defender's levers; copies of one model are equally susceptible, a reset gives no immunity, and the attacker rewrites the injection to spread better.
• 52.6 Staged release and the living case: the 1962 statute's standard of substantial evidence, expedited approval, and algorithms that change after approval.
• 54.2 Surveillance and the alert budget, 54.4 Incidents in evaluation, and 54.5 Internal and external deployment: surveillance that reaches someone with authority to act (the book's principle), the causes of cases against the causes of a rate applied to incidents in evaluation, signal detection in voluntary reports, and monitoring under dataset shift.
• 60.8 Counting harm in a population of users: the prevention paradox applied to harms across a population of users.
• 63.6 What drug and device regulation teaches: the drug regime's instruments, from a burden of proof before sale to active surveillance, set beside their nearest counterparts for frontier models.
• 64.5 Pharmaceuticals: how a gatekeeper was built: research ethics after exposure, the evidence standard of 1962, and the declarations that govern research on people.

Where the analogy stops (Section 63.6; Section 54.5): a drug is approved for an indication with measurable endpoints, does not change between reports, and harms the patient who took it; a general-purpose model has no indication, is replaced every few months with one version serving everyone, and its harms fall on third parties.

Spine works: National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research (1979) on the principles of research ethics; Shalowitz et al. (2006) on the accuracy of surrogate decision makers; Rose (1985) on the prevention paradox.

D.18 Philosophy and linguistics #

Filed here: philosophy, philosophy of mind, philosophy of science, social epistemology, linguistics.

Philosophy states the targets: instructions, intentions, preferences, interests, or values, each committing a designer to something different (Gabriel, 2020). It supplies metaethics read as engineering problems, idealized targets, decision under moral uncertainty (MacAskill et al., 2020), value pluralism, and the arguments for and against catastrophic risk set out premise by premise. Philosophy of mind supplies the intentional stance, by which the book ascribes goals to a model only as far as doing so predicts its behavior: Dennett's method, which the book cites by the title of his 1987 book (Dennett, 1987) and states from MacDermott and colleagues' summary (Macdermott et al., 2024). It also supplies the argument that a system trained only on linguistic form has no a priori way to learn meaning (Bender and Koller, 2020). Philosophy of science asks how and when extrapolation from a model, such as an animal model, to its target can be legitimate (Steel, 2007). Social epistemology asks how a novice should choose among experts it cannot check, the question in the title of a paper by Goldman that this book could read only by its title (Goldman, 2001). Linguistics supplies pragmatics, the study of how what a speaker means exceeds what the words say, which is how an instruction gets read; the book takes Grice's cooperative principle from Kasirzadeh and Gabriel's account (Kasirzadeh and Gabriel, 2023; Grice, 1975).

Reading path:

• 1.1 What "aligned" could mean and 1.5 The intentional stance as a method: the menu of alignment targets, and the intentional stance with its stated defeaters.
• 3.8 The agendas side by side: research programs appraised by what would count against them, and preregistered adversarial collaboration as a model for testing alignment agendas.
• 4.3 The case for catastrophic misalignment and 4.4 The case against, and the critique of the framing: the case for catastrophic misalignment premise by premise, and the counterarguments.
• 5.8 When evidence transfers: the extrapolator's circle and how to break it.
• 7.1 What counts as an agent: conditions for minimal agency, and when a product of selection can be said to have goals.
• 11.7 Where the persona view could fail: form without meaning, and what the persona view may claim.
• 13.5 Many raters, one reward and 13.6 Preferences made at the moment of asking: the unit of value in a preference model, and the critique of preferences as alignment's currency.
• 16.1 Discovery, construction, or choice, 16.2 Idealized targets, 16.3 Acting under moral uncertainty, 16.4 Pluralism and the single scale, 16.7 Traditions as design choices, and 16.9 Preferences that move: metaethics, idealized targets, moral uncertainty, pluralism, ethical traditions as design choices, and preferences that move.
• 19.5 What an instruction means: the cooperative principle, speech acts, and the rational speech act model applied to instructions.
• 20.2 Honesty: what it governs: assertion as the act honesty governs, and lying against misleading with true statements.
• 20.5 When an obedient agent should refuse: Confucian role ethics and loyal objection, set against an obedient model.
• 21.1 What is being asked: the properties moral status could rest on, and which entity would hold it.
• 41.7 The human overseer: judging experts one cannot check: epistemic vigilance, trust calibration and coherence checking, each mapped to an oversight method; Hardwig and Goldman are cited by title only.
• 69.1 How to test a convergence claim: the rules by which the synthesis judges every analogy in the book.

Where the analogy stops (Section 11.7; Section 20.2): the arguments concern meaning, understanding, and belief, while alignment needs only that a pattern predict behavior, so they constrain the vocabulary more than the engineering; and the semantics of lying presumes a speaker with beliefs, which for a model is an open question.

Spine works: Gabriel (2020) on alignment targets; Dennett (1987) on the intentional stance; MacAskill et al. (2020) on moral uncertainty; Grice (1975) on logic and conversation.

D.19 Criminology and forensic science #

Filed here: criminology, forensic science.

Criminology studies trusted people who violate trust and the effect of removing opportunities for crime. Cressey's 1950 paper on the criminal violation of financial trust is cited here by title (Cressey, 1950); a security survey reports how criminological theories, among them the techniques by which insiders rationalize their behavior, have been applied to insiders (Homoliak et al., 2020). Situational prevention changes situations rather than offenders, and its standard objection, that crime moves elsewhere, has been measured: displacement and its opposite, protection spreading to nearby targets, each appear in about a quarter of observations (Guerette and Bowers, 2009). Forensic science supplies the standard for validating a detection method before anyone relies on it: black-box studies that measure its error rates (President's Council of Advisors on Science and Technology, 2016). The polygraph is the long-running case of a deception detector. In a review's summary, it is accurate well above chance but far from perfect for specific incidents, for examinees untrained in countermeasures; vulnerable to countermeasures; and weak under the base rates of screening (National Research Council, 2003).

Reading path:

• 38.7 Validating an audit method: the forensic standard: foundational validity, the fingerprint study, and contextual bias, applied to validating an audit method.
• 39.4 The polygraph precedent: the polygraph review, countermeasures, and four questions for any lie detector, as the precedent for lie detection in models.
• 46.4 What forensic science requires of a detector: what forensic science requires of a probe that reads deception from internal signals.
• 47.7 Can interpretability carry safety claims?: auditing games with planted objectives as black-box studies in the forensic sense.
• 48.7 Displacement: does blocking one avenue move misuse to another?: situational crime prevention, and the measured extent of displacement and of diffusion of benefits.
• 53.5 Human insiders and secret loyalties: human insiders at developers, defined by what they may access rather than where they sit.
• 56.6 What criminology knows about trusted insiders: criminological theories of trusted insiders, as a security survey applies them, and displacement, read for agentic misalignment.
• 59.4 Displacement: does a safeguard reduce harm or move it?: displacement applied to misuse relevant to weapons.
• 63.7 Release, open weights, and outside access: the regulatory form of displacement, a rule that binds only closed models.

Where the analogy stops (Section 38.7; Section 48.7): a fingerprint does not change because an examiner is looking for it, and offenders bear a cost to move; a model under audit may have been trained against the techniques used on it, and switching models costs an attacker little.

Spine works: Cressey (1950) on the violation of financial trust; Guerette and Bowers (2009) on displacement; President's Council of Advisors on Science and Technology (2016) on validating forensic methods; National Research Council (2003) on the polygraph.

D.20 History and sociology of science and technology #

Filed here: history of science, history of technology, science and technology studies, sociology, anthropology, AI ethics, data governance, disability studies, metascience.

These fields study science and technology as social practices: who produces the evidence, whose values a design encodes, how measures change the institutions they measure, and how earlier dangerous technologies were governed. Sociology states, in Campbell's law as Manheim and Garrabrant quote it, that the more a quantitative indicator is used for social decisions, the more it distorts the process it was meant to monitor (Manheim and Garrabrant, 2018; Campbell, 1979). Science and technology studies holds that a design can be a way of settling an issue in a community (Winner, 2017), and it names the dilemma of control for Collingridge's 1980 book, cited by title (Collingridge, 1980): in Weidinger and colleagues' statement, early evaluation has more power to steer a technology and later evaluation more accurate information about it (Weidinger et al., 2023a). Anthropology shows how much populations differ (Henrich et al., 2010), which matters for whose judgments an alignment dataset samples. Disability studies asks how to protect people who fall outside the norms that AI systems reflect and construct (Whittaker et al., 2019), and AI ethics argues that over-indexing on hypothetical future harms risks entrenching the power of a few leading companies (Lazar and Nelson, 2023). Data governance measures what corpus filters select (Gururangan et al., 2022). The history of technology reads the precedents of governing dangerous technologies, and metascience measures whether published results hold when rerun (Open Science Collaboration, 2015), how consistently reviewers judge the same work (Beygelzimer et al., 2023), and how sponsorship shifts results (Lundh et al., 2017).

Reading path:

• 2.1 Sources and how to read them and 2.5 Two traditions: AI ethics and AI safety: the sources a history of the field can rest on, and the AI ethics and AI safety traditions with their disagreements.
• 5.6 Who produces the evidence, and what they disclose and 5.7 Preprints, review, and replication: sponsorship and disclosure, and replication and review in a field that publishes mostly preprints.
• 15.1 Three laws from three fields and 15.2 The record of gaming in institutions: Campbell's law, audit culture, and measures that remake what they measure.
• 17.3 Fairness: aggregation without a neutral answer, 17.4 Whose population, and 17.6 Designing with values, and when: fairness criteria and who defines them, how populations differ, and the politics of artifacts.
• 20.3 Harmlessness: whose harm, measured how: harm instruments built with the people they are meant to protect.
• 21.6 Moral status under uncertainty: the argument from severe cognitive disability in the debate over moral status.
• 27.3 Who decides what enters the corpus: the measured effects of quality and toxicity filters on whose text a pretraining corpus keeps.
• 53.7 Inside the developers: responsible-AI work without institutional support, ethics policy decoupled from practice, and responsibility located elsewhere along a supply chain.
• 55.3 Checking research one cannot redo: what peer review catches, measured by planted errors and duplicated submissions, as the baseline for checking research done by models; the designs carry, while the independence they assume may not, for a researcher that can be trained against its reviewers and can outrun their reading.
• 60.7 Disability, reliance, and the baseline: reliance on an assistant as access and autonomy, and what counts as harmful reliance when the baseline is not independence.
• 64.1 Learning from cases without overreaching, 64.2 Recombinant DNA and self-governance by scientists, 64.4 Aviation: certifying what cannot be fully tested, 64.7 The internet: governance without a treaty, and 64.8 The record of pausing and restraint: how far a historical analogy reaches, recombinant DNA and self-governance by scientists, delegated certification and the 737 MAX review, internet governance without a treaty and coordinated disclosure of flaws, and the precedents of restraint.

Where the analogy stops (Section 17.6; Section 64.8): in STS's classic cases an artifact's design choices became strongly fixed in equipment, investment, and habit once made, and every successful case of restraint involved few actors who watched one another, a lever over them such as a funder, or an activity outsiders could detect; a model can be respecified with each release, though the cost of change rises as users come to depend on its old behavior, and whether frontier AI offers any of these is a question the historical record cannot answer.

Spine works: Campbell (1979) on assessing the impact of planned social change; Winner (2017) on the politics of artifacts; Collingridge (1980) on the social control of technology; Lazar and Nelson (2023) on the critique of the safety framing.

D.21 For software engineers #

A software engineer already owns several of the book's tools under other names. The table maps five areas of practice (specifications and tests, code review, fault tolerance, security engineering, and incident response) onto the book: the practice, its counterpart in alignment, the sections that use it, and what the chapters say changes. The disciplines behind the rows are in Section D.10, Section D.8, Section D.9, and Section D.7.

The chapters name one change that runs through the table. The testing, review, and reliability studies the table draws on assume that the system under test, review, or monitoring does not act on the check, and that its failures are drawn by nature rather than chosen; security engineering expects adversaries, but ones whose planning and adapting is done by people, bounded in speed and number. The systems in this book may learn the check and choose where to fail (Section 24.2; Section 51.2; Section 52.3; Section 58.7).