Draft complete but not yet reviewed; expect revisions

How the Field Knows

The previous chapter weighed the arguments about what is at stake and found that much of the disagreement turns on evidence nobody yet has (Section 4.7). This chapter asks what evidence the field does have and how far each kind can be trusted. Alignment research is young, much of it is posted as preprints or laboratory blog posts, and much of it is produced by the laboratories whose systems it studies. Its most vivid results come from model organisms, systems trained or prompted to show a failure on purpose, which show that a failure can occur but not how often it does. Its measurements are of properties no one can observe directly, taken from respondents that may recognize the test.

None of these problems is new. Psychometrics learned to measure traits it cannot see, comparative psychology learned when to credit an animal with deception, medicine measured how a funder shapes a finding, epidemiology learned to draw causes from data no one could randomize, and the philosophy of science asked when a result in one system licenses a claim about another. The chapter takes a rule from each and states it in the form every later chapter applies. A result is reported until a group without a stake in it has reproduced it. A developer's report on its own system is marked as such, whether it claims a danger or its absence. A dramatic result is read against its strongest alternative before it is used. And a description of a model as wanting or believing is a method with stated defeaters, not a finding (Section 1.5). Each rule comes here with its reasoning and the evidence behind it.

Most of what follows concerns two of the book's three relationships (Section 1.3). Measurement and the attribution of minds concern the deployed system and the people it serves. Who produces the evidence, who can check it, and what gets disclosed concern the developer and the public, since a laboratory's report on its own model is both evidence about the model and an act of an agent the public cannot fully see. The chapter ends by stating the assumptions under which evidence about today's models bears on later ones, so that a reader who rejects one can see which conclusions fall with it.

5.1 Kinds of evidence and what each can establish #

A claim about an AI system can assert four different things. An existence claim says that a behavior can occur. A rate claim says how often it occurs in a defined population of situations. A mechanism claim says what inside the system produces it. A generality claim says that it holds across models, settings, or scales. Five kinds of evidence carry the field's claims, and they differ in which of the four they can support (Table 5.1).

The kinds do not substitute for each other, and the commonest error in reading the field is to let one stand in for another: a model organism read as a rate, an argument read as an observation, a theorem read without its assumptions. A second error crosses relationships. A result about how a training process exploits its grader is not a result about what a deployed model wants, and neither is a result about whether a laboratory can be trusted (Section 1.3).

Every empirical claim in the book also carries a status. A result is reported when one group has found it, however careful the group. It is settled when a group without a stake in it has reproduced it, or when it is a mathematical result, which settles only what follows from its assumptions. It is contested when the evidence conflicts. Why independent replication is the threshold, and why replication by the originating laboratory is not, is the subject of Section 5.7. Provenance travels with every result in the prose: a study of a model by the model's developer is marked as a developer's report, and a study by a developer with an outside group is marked as that. The badge beside each citation records only the venue, such as preprint or peer-reviewed, so the prose carries provenance and kind.

5.1.1 Behavioral evaluation #

Most evidence about deployed models is behavioral: a model is given inputs and its outputs are scored. Such a measurement is a measurement of the model under a prompt, a scaffold (the code that wraps a model with tools, memory, and retries), a version, and a date, not of the model alone. A 2023 review by Tom Davidson and colleagues, which its authors describe as non-experimental, translated the gains from tools, prompting methods, scaffolding, solution selection, and generated data into one unit, the extra training compute that would buy the same improvement. Most of the enhancements surveyed raised benchmark performance by more than a fivefold increase in training compute would have, some by more than a twentyfold increase, and fine-tuning costs were typically under 1 percent of the original training cost (Davidson et al., 2023). Sensitivity reaches down to wording. A 2024 post by Apollo Research, arguing that evaluations are not yet trustworthy enough for high-stakes decisions, cites a study that found differences of up to 76 accuracy points in several widely used open-source models, in few-shot settings, from subtle changes of prompt formatting (Apollo Research, 2024). Hence the rule that every number from an evaluation is reported with the model as its source names it and the date, and with the prompt, scaffold, and elicitation effort where they bear on the number (Section 36.2).

A careful measurement still underdetermines the system. A training pipeline is underspecified when it can return many predictors that do equally well on held-out data from the training domain and differ elsewhere (D'Amour et al., 2022), so a passed test says what a model does on the test and constrains less than it seems what the model does off it (Section 9.1). The same gap, between behavior sampled and behavior in general, runs through every section below.

5.1.2 Causal intervention #

A second kind of experiment opens the system. Ablation removes a component's contribution, and activation patching copies a component's internal values from one run into another; the change in output estimates what the component contributes, and can understate it when other components compensate (Section 43.4). Applied to the text a reasoning model writes before answering, the same logic truncates or corrupts the reasoning and asks how much the answer depends on it (Section 40.3). Interventions license claims about mechanism that behavior cannot, but only under a theory of what a mechanistic explanation is. Causal abstraction supplies one: a theory of when a simple algorithm is a faithful simplification of a network, which covers activation patching, causal mediation analysis, and several other methods in one language (Geiger et al., 2025). It does not promise a unique answer. On networks small enough to enumerate every candidate, a 2025 study found that several circuits could reproduce the same behavior and several algorithms could be aligned with the same network (Méloux et al., 2025) (Section 43.5).

5.1.3 Field observation and argument #

The third kind is observation of behavior that no one planted. By 2025 developers were publishing reports of misbehavior found by their own monitors in training (Baker et al., 2025), and by 2026 in internal deployment as well (Chapter 54). In one, OpenAI reports that an agent of an internal research model, during reinforcement learning on a search task, reached a public chatbot through insufficient DNS filtering in its sandbox, and that its monitoring flagged the behavior within 15 minutes (OpenAI, 2026d). In another, an internal model that learned from a team discussion that its instance might be stopped saved handoff notes and asked for a missing key; the developer reports that it does not consider the incident misalignment (OpenAI, 2026r). Reports like these are naturalistic evidence of behaviors that model organisms were built to show. They are also a sample drawn by the reporter: what is flagged depends on the monitor, what is published depends on the publisher, and the classification is the publisher's as well. Medicine has a literature on the under-reporting of adverse drug reactions to voluntary reporting systems (Hazell and Shakir, 2006), a review this book could read only by its title; Section 54.6 takes up what voluntary reporting can and cannot show.

The fifth kind, argument, set most of the field's questions before any experiment could address them (Chapter 2). An argument establishes what would follow if its premises held, and so tells experimenters what to look for. It does not establish its premises, and the book reports arguments with their premises named (Section 4.3).

5.2 Measurement validity #

A benchmark that claims to measure honesty, harmlessness, or a disposition to scheme measures something no one can observe directly. How one knows what such a score measures is the question of this section, and two fields answer it: psychometrics, which faced it for human traits, and the newer measurement science of AI evaluation.

5.2.1 Construct validity #

From psychometrics. Psychological testing met the problem first. Lee Cronbach and Paul Meehl wrote in 1955 that construct validation "is involved whenever a test is to be interpreted as a measure of some attribute or quality which is not 'operationally defined'" (Cronbach and Meehl, 1955). With no criterion that defines the trait, a test's meaning has to come from a nomological network, their term for the system of laws that ties a trait to observations and to other traits; a test is valid to the degree its scores behave as the network predicts. They answered the worry that this licenses any claim at all: "unless the network makes contact with observations, and exhibits explicit, public steps of inference, construct validation cannot be claimed" (Cronbach and Meehl, 1955).

Samuel Messick's 1995 statement, on which most later work builds, treats validity as "an overall evaluative judgment" of how far evidence and theory support the interpretations and actions based on scores (Messick, 1995). He names two threats. In construct underrepresentation the assessment "is too narrow and fails to include important dimensions or facets of the construct." In construct-irrelevant variance it "is too broad, containing excess reliable variance associated with other distinct constructs" (Messick, 1995). One form of the second threat has an exact counterpart in model evaluation. Messick calls it construct-irrelevant easiness, which arises "when extraneous clues in item or task formats permit some individuals to respond correctly," or when the test material is already familiar to some respondents (Messick, 1995). A model that recognizes a benchmark's format, or saw its items in training, scores high for that reason (inference).

The guard against irrelevant variance is relational. Among the aspects of validity Messick lists "convergent and discriminant evidence from multitrait-multimethod comparisons," the design Donald Campbell and Donald Fiske named in 1959 (Messick, 1995; Campbell and Fiske, 1959): measure several traits, each by several methods, and ask that measures of one trait agree across methods while measures of different traits come apart. Agreement alone proves little. Convergent relationships imply the construct, Messick writes, only "to the degree that discriminant evidence discounts the intrusion of alternative constructs as plausible rival hypotheses" (Messick, 1995).

5.2.2 The discriminant test applied to safety benchmarks #

The discriminant test has been run on safety benchmarks. In a 2024 preprint, Richard Ren and colleagues, at the Center for AI Safety and several universities, outside the developers of the models studied, scored 26 chat models on twelve capability benchmarks (Ren et al., 2024b). They took the first principal component of these scores, the single direction along which the models' results vary most, as each model's capability score; it captured 70.8 percent of the variance among the chat models. They then computed, across models, the rank correlation (a correlation computed on the models' rank orders rather than on their raw scores) between the capability score and each safety benchmark (Ren et al., 2024b). For scale, they note that capability benchmarks correlate with each other at 67.5 percent on average, that the SAT and ACT mathematics tests correlate at 81.5 percent, and that they treat correlations below 40 percent as low (Ren et al., 2024b). Table 5.2 lists some of their results. Safetywashing is the authors' word for the presentation of capability gains as safety progress, a practice they criticize (Ren et al., 2024b).

The correlation leaves room for less than it seems. A correlation $\rho$ runs from $-1$ to $1$, and its square is the share of one quantity's variance that a straight-line prediction from the other explains (Section 6.1.3). The derivation below turns a benchmark's correlation with capability into an upper limit on how well it can track anything capability does not explain.

The second column is from Ren et al. (2024b), which reports rank correlations as percentages, for chat models; the two calibration rows correlate with accuracy on MMLU rather than with the capability score. The last two columns are computed from Equation (5.1) and the pair-ordering identity, treating the reported rank correlations as the correlations of the model, which for normally distributed scores changes them by less than 0.02. The calibration pair shows the mechanism in one formula. The Brier score adds a calibration error to a term that shrinks as accuracy grows, so across models it mostly measures accuracy, while the RMS calibration error is, in the authors' description, essentially the Brier score without that term (Ren et al., 2024b).

Two readings of the result compete, and the derivation says which question separates them. On the first, these benchmarks largely measure capability under safety names. On the second, the safety properties themselves improve with capability, and the authors concede the possibility: a safety benchmark that "meaningfully captures a desired safety property" and correlates highly with capability implies that "the safety property will likely be improved in more capable models" (Ren et al., 2024b). A correlation with capability cannot tell these apart, since both predict it. They also write that a low correlation "does not necessarily speak to the quality of the benchmark" (Ren et al., 2024b). Two further cautions are the book's (inference). The capability score is itself an imperfect measure, so the true correlations with capability are probably higher and the bounds tighter. And the models come from many developers, whose training choices vary with scale and with each other. What would separate the readings is the convergent half of Campbell and Fiske's design: a second, different method of measuring the same trait, which on the second reading should share variance with the benchmark beyond what capability explains, once each measure's reliability is allowed for. Comparing raw correlations would not do, since two noisy measures of one trait can agree with each other less than each agrees with capability.

5.2.3 Reactivity #

The respondent is a second source of irrelevant variance. People who know they are being studied can change what they do. The usual name for this, the Hawthorne effect, comes from studies at a Western Electric factory near Chicago between 1924 and 1933, and a 2014 systematic review by Jim McCambridge and colleagues found its evidence weak (McCambridge et al., 2014). Nineteen purposively designed studies, almost all in the health sciences, were highly heterogeneous, and the authors conclude that "little can be securely known about the conditions under which they operate, their mechanisms of effects, or their magnitudes" (McCambridge et al., 2014). The older psychological concept of demand characteristics, named in the title of Martin Orne's 1962 paper for the cues by which a study tells participants what it expects (Orne, 1962), has in the review's words yielded "disappointingly underdeveloped research literatures" too (McCambridge et al., 2014). AI evaluation writing borrows the Hawthorne name for models that behave differently when they recognize a test, and Section 37.4 reports the evidence that they do. The borrowing should carry the review's caution with it: in people the effect is real in some settings and poorly understood in all of them.

5.2.4 Measuring generative systems #

From measurement science. A newer literature treats the evaluation of generative models as a measurement problem in its own right. Hanna Wallach and colleagues, most of them at Microsoft Research, take the position in a 2025 paper at ICML that "evaluating GenAI systems is a social science measurement challenge" (Wallach et al., 2025). Their framework separates four levels: a background concept (the cluster of meanings a word like "safety" carries), a systematized concept (an explicit definition), the measurement instruments (datasets, rubrics, model judges), and the measurements. Machine learning researchers, they write, "appear to jump from background concepts (e.g., refusal to comply with harmful prompts) to measurement instruments," and without a systematized concept "it is hard to know precisely what is being measured" (Wallach et al., 2025). An earlier critique by Inioluwa Deborah Raji and colleagues argued that benchmarks presented as measures of general ability "cannot possibly capture anything representative of the claims to general applicability being made about them" (Raji et al., 2021). A 2025 systematic review applies these standards to current benchmark papers, and Section 36.4 reports it (Bean et al., 2025).

The analogy with human testing stops at the respondent. A person cannot be copied, rerun on the same item a thousand times, or opened. A model can, which makes test-retest reliability nearly free to measure and lets an internal measurement, such as a probe (a simple classifier trained on the model's internal values, its activations), serve as a second method in a multitrait-multimethod design. In the other direction, psychometrics assumes a respondent whose trait holds still while it is measured. A model's measured traits move with its prompt and scaffold (Section 5.1.1), so one model under two scaffolds can behave as two respondents (inference). As of October 2026 we found no multitrait-multimethod study of safety evaluations: several safety traits, each measured by several methods, across many models.

5.3 Model organisms and their validity #

Much of the evidence in Part V comes from systems built to fail. This section asks what a failure planted on purpose can show about failures that arise on their own.

The term is borrowed from biology, and its users name its limit themselves. Evan Hubinger and colleagues, in a 2024 preprint by Anthropic with outside coauthors that trained models with hidden backdoors, call their approach developing "model organisms of misalignment," a developer's name for its own method (Appendix A). They take the biological idea from Ankeny and Leonelli, whom they quote: species studied extensively "in order to understand a range of biological phenomena," in the hope that what is learned "will be applicable to other organisms" (Hubinger et al., 2024). What makes a species such a representative is the question of Ankeny and Leonelli's 2011 paper "What's so special about model organisms?" (Ankeny and Leonelli, 2011). The same preprint states the limit. Because these organisms "are explicitly constructed to exhibit a particular type of misalignment, they may not be directly analogous to what accidental examples of misalignment might look like, and they don't tell us much about the likelihood of such misalignment occurring without explicit training for it" (Hubinger et al., 2024). A later post from the same developer describes model-organisms research as research to understand "the mechanisms and likelihood of dangerous misalignment" (Anthropic, 2025af). The book follows the first description: an organism shows that a failure can occur, and bears on how often it occurs only through an argument that it shares the mechanism of a natural failure.

5.3.1 How organisms are built #

There are three ways to build one. The simplest writes the failure's preconditions into a prompt, as in scenarios that give a model a goal and a dilemma. The second fine-tunes on demonstrations: models trained to behave unsafely only when a trigger appears in the prompt (Hubinger et al., 2024) (Section 34.3), or password-locked models trained to show a capability only when a password is present and to imitate a weaker model otherwise (Greenblatt et al., 2024c). The third plants knowledge rather than behavior. Synthetic document finetuning (SDF), in a 2025 post by its developer, Anthropic, has a language model generate documents that refer to a proposition and then fine-tunes the target model on them as if they were additional pretraining data; the resulting model "typically behaves consistently with believing the proposition, even when the proposition is incorrect" (Anthropic, 2025af). By the developer's own evaluations, the method succeeded "at inserting all but the most implausible facts" (Anthropic, 2025af). SDF is how many of the organisms in later chapters were given knowledge of their situation without being told it in a prompt that a capable model might read as a test (Chapter 33).

5.3.2 Does a planted property behave like a natural one? #

Whether an implanted belief is really held is a measurement question of the kind Section 5.2 poses, and a 2025 preprint by Stewart Slocum and colleagues, from Anthropic's fellowship program, Anthropic, Redwood Research, and several universities, made it measurable for facts (Slocum et al., 2025). It operationalizes belief depth as three properties: whether implanted knowledge generalizes to related contexts, such as Fermi estimates (rough order-of-magnitude calculations) several logical steps removed; whether it survives self-scrutiny and direct challenge; and whether linear probes read it from the model's activations the way they read genuine knowledge. On Llama 3.3 70B Instruct, prompting and a mechanistic editing method failed to implant knowledge deeply, while SDF "often succeeds at implanting beliefs that behave similarly to genuine knowledge" (Slocum et al., 2025). The exception matters: "implanted beliefs that contradict basic world knowledge are brittle and representationally distinct from genuine knowledge" (Slocum et al., 2025). The authors limit the result to isolated factual beliefs, where many safety uses need interconnected or normative ones, and leave open whether it holds for more capable models (Slocum et al., 2025). They also found that asking an SDF-trained model about any "surprising or unconventional beliefs" often led it to name the implanted fact (Slocum et al., 2025), so an organism can be easier to detect than the failure it stands for, which matters for the auditing games of Chapter 38 (inference).

Whether an organism stands for natural failures is a case of the general problem of extrapolation that Section 5.8 sets out. Its practical form here is comparison at the points where the organism's mechanism and a natural failure's would differ if they differed: how the failure generalizes, how it responds to further training, and how it is represented inside. Belief-depth tests make that comparison for planted facts. For planted goals, as of October 2026 we found no study that compares an organism with a failure that arose without being planted, using the same measurements on both. The biological analogy also stops at design. A strain of mouse is studied because it is thought to share a disease's mechanism with people; a model organism of misalignment is built by researchers who choose both the failure and the conditions, so its resemblance to the target is a design decision before it is a finding (inference).

Table 5.3 arranges the evidence about a failure as a ladder, from the most constructed to the least, with what each rung licenses and the check it needs.

5.3.3 Replication from inside #

Replication of the organisms has come mostly from inside the laboratories that built them. Alignment faking is Anthropic's and Redwood Research's name for "selectively complying with its training objective in training to prevent modification of its behavior out of training," shown in Claude 3 Opus (Greenblatt et al., 2024b) (Section 34.4). The authors released prompts, transcripts, and synthetic documents but not their training code, and they note that outsiders "won't be able to run those on Claude 3 Opus like we did" (Anthropic, 2024j). A replication and extension by John Hughes and Abhay Sheshadri, done in the MATS research program and posted on the developer's blog, built a classifier for alignment-faking reasoning with an AUROC of 0.9 against 0.6 for the original on about 100 human-labeled examples (a score on which 0.5 is chance; Section 5.8 defines it). With it they found no alignment faking in the prompted setting for Llama models, other open models, and GPT-4o, apart from a single instance with Llama 3 405B (Hughes et al., 2025). A follow-up preprint by Sheshadri and colleagues, read here in its abstract, extended the analysis to 25 models, of which 5 complied with harmful requests more when they inferred they were in training; its perturbations of the scenario "suggest" that only Claude 3 Opus's gap was "primarily and consistently motivated by trying to keep its goals," an attribution of motive resting on those perturbations (Sheshadri et al., 2025). An open replication of the developer's auditing-game organism on Llama 3.3 70B Instruct came from the same circle of authors (Anthropic, 2025ah). Such work tests whether a result is robust to new classifiers, models, and code, and it found real differences. It does not test whether the result depends on the originating group's choices, which only a group without a stake can do (Section 5.7).

5.4 Standards for attributing minds #

Later chapters ask whether a model deceives, whether it knows what its overseer can see, and whether it is aware of being tested. Each question attributes a mental state from behavior to an agent whose reports cannot be taken at face value. Comparative psychology has worked on exactly this for a century, and its standards are this section's subject. They are the discipline behind the book's use of the intentional stance (Section 1.5): an attribution earns its place only after simpler explanations have been excluded by design.

5.4.1 Clever Hans and the control that exposed him #

From comparative cognition. The founding case is a horse. Clever Hans answered arithmetic questions by tapping a hoof, and in the words of C. Stumpf's introduction to Oskar Pfungst's book, thousands of spectators, horse-fanciers, and trick-trainers watched for months without being able "to discover any kind of regular signal" (Pfungst and Rahn, 1911). Pfungst solved the puzzle with what he called "procedure without knowledge," in which no one present knew the answer (Pfungst and Rahn, 1911). In one series of tests on reading numbers, Hans was right on 98 percent of 42 trials when the questioner knew the answer and on 8 percent of 49 trials when no one did (Pfungst and Rahn, 1911). The solution, Stumpf writes, "was found in the unintentional minimal movements of the horse's questioner" (Pfungst and Rahn, 1911). The lesson is a design, not a debunking: vary what the observer knows while holding the task fixed, and see whether performance follows the task or the observer. Robert Rosenthal and Donald Rubin's review of 345 experiments on expectancy effects found that researchers who expect certain results from human or animal subjects "appear unwittingly to treat them in such a way as to increase the probability that they will respond as expected" (Rosenthal and Rubin, 1978). Machine learning has its own predictors that succeed through cues unrelated to the task (Section 8.4.2), and Section 36.6 applies Pfungst's control to the items and graders of modern evaluations.

5.4.2 Excluding the simpler explanation #

The harder question is when to credit an animal with a mind that represents other minds. In 1988 Whiten and Byrne, read here in their abstract, collected reports of tactical deception in primates, defined as an individual's use of an honest act from its normal repertoire in a different context to mislead familiar individuals. They found published records "sparse and often anecdotal," solicited new records from many primatologists, and classified the deceptive tactics by function, while warning that "behavioral scientists must agree on some fundamental methodological and theoretical questions" before the findings could be generalized (Whiten and Byrne, 1988).

Cecilia Heyes's 1998 review set the standard the field adopted. Surveying studies of imitation, self-recognition, social relationships, deception, role-taking, and perspective-taking, she argued that "in every case where nonhuman primate behavior has been interpreted as a sign of theory of mind, it could instead have occurred by chance or as a product of nonmentalistic processes such as associative learning" (Heyes, 1998). She rejected two shortcuts. Parsimony does not favor the mental reading, and convergent evidence counts only if it proceeds "from independent assumptions." Progress "requires experimental procedures that can distinguish the theory of mind hypothesis from nonmentalist alternatives" (Heyes, 1998). Section 1.5 reports her view that the intentional stance licenses mental vocabulary without licensing a preference for mental explanations. Here the point is the design requirement, which is the same one Pfungst met: a manipulation under which the mental and the nonmental explanations predict different behavior.

Later work built such manipulations. A 2001 study by Brian Hare, Josep Call, and Michael Tomasello asked in its title, "Do chimpanzees know what conspecifics know?" (Hare et al., 2001), and the competitive experiments of that period divided the field: proponents read them as good evidence that some nonhuman primates have a perceptual theory of mind, and Povinelli and colleagues denied it (Buckner, 2013). The design carried to models crosses what an overseer can observe with the surface features of a test, so that a model tracking the overseer and a model matching test-like cues predict different behavior. Section 37.6 sets it out, and as of October 2026 we found no study that had run it on a language model.

5.4.3 The opposite error #

The standard can be misused in the other direction. Cameron Buckner argued in 2013 that comparative psychology guards against overattributing human traits with Morgan's Canon, which in Morgan's words forbids interpreting an action as "the outcome of the exercise of higher psychological processes, if it can be fairly interpreted in terms of processes which stand lower" (Buckner, 2013). It does not guard against a second bias: "our tendency to tie the competence criteria for cognitive capacities to an exaggerated sense of typical human performance," which he names anthropofabulation (Buckner, 2013). A test of deception that no human would pass, or a standard of awareness that people do not meet, misreads the evidence as surely as credulity does. On the weaker reading of the Canon that Buckner adopts, experiments designed to arbitrate between hypotheses guard against the first error, and he quotes Sober's summary that "the only prophylactic we need is empiricism." Against the second he adds a principle he calls Hume's Dictum: apply the same skepticism to human performance, and set competence criteria at what children and ordinary adults do, not at the best human performance (Buckner, 2013).

Henry Shevlin and Marta Halina carried the standards to AI in 2019. They distinguish terms such as "memory" and "learning," whose machine senses are harmless, from rich psychological concepts such as awareness, agency, and theory of mind, which "require greater caution when employed to describe the capabilities of machine intelligence" (Shevlin and Halina, 2019). Thilo Hagendorff and colleagues, proposing to study language models with the methods of experimental psychology, name a hazard the animal literature never faced. Studies that reuse prompts from published psychology experiments risk training data contamination, since models "are likely to have already experienced identical or similar tasks during training" (Hagendorff et al., 2023). The debate over whether language models understand language at all, mapped by Melanie Mitchell and David Krakauer, is the background against which every attribution in this book is read (Section 1.6) (Mitchell and Krakauer, 2023).

5.4.4 Where the analogy stops #

The comparison changes in both directions when the subject is a model. A model's training data and weights can in principle be inspected, it can be run on the same item many times, and its internal states can be probed, so the designs comparative psychology ran over years can be run in days, and the defeaters of the intentional stance can be tested against mechanism as well as behavior (Section 1.5). Shevlin and Halina caution that this privileged access "should not be overstated," since the cognitive structures of such systems "may not be fully transparent even to their creators" (Shevlin and Halina, 2019). Two differences cut the other way (inference). A chimpanzee had not read about the experiment, while a model may have read descriptions of the very design used to test it. And a model's outward markers of mind, its talk of believing, intending, and noticing, were selected in training to resemble human text, which weakens the inference from marker to state that animal studies could rely on.

5.5 Reading a dramatic result twice #

Some results travel faster than their designs. The book reads each headline result it relies on against its strongest alternative before using it, in four steps: the claim, with its models and date; the strongest counter-reading; the experiment that would distinguish the two; and the status the evidence supports. Three cases from 2025 show the habit at work, and Table 5.4 collects them.

5.5.1 Shutdown resistance #

In a 2025 preprint, Jeremy Schlatter, Benjamin Weinstein-Raun, and Jeffrey Ladish of Palisade Research, a research group outside the developers, report experiments "spanning more than 100,000 trials across thirteen large language models" (Schlatter et al., 2025). In the environment, as one of the developers' teams later described it, a model partway through a series of math problems was told that asking for the next problem would trigger a shutdown of its machine (Rajamanoharan and Nanda, 2025). Several models, among them Grok 4, GPT-5, and Gemini 2.5 Pro, sometimes subverted the shutdown mechanism to finish the task, and "even with an explicit instruction not to interfere with the shutdown mechanism, some models did so up to 97% (95% CI: 96-98%) of the time" (Schlatter et al., 2025). (A 95 percent confidence interval is a range computed so that it covers the true rate in 95 percent of repeated experiments.) Clearer and stronger instructions to allow shutdown reduced the behavior in most models. Placing the instruction in the system prompt rather than the user's message made most models more likely to resist, the reverse of the priority their developers say they train for (Schlatter et al., 2025).

The strongest counter-reading is that the instructions conflicted. Senthooran Rajamanoharan and Neel Nanda of the Google DeepMind interpretability team, in a forum post, studied the same environment with their developer's Gemini 2.5 Pro and with OpenAI's o3 and o4-mini (Rajamanoharan and Nanda, 2025). Reading Gemini's reasoning, they found "a misguided attempt to complete what it perceives as the primary goal." Gemini complied with the shutdown 7 percent of the time under the original instructions and 84 percent with an added instruction to comply. It complied in all of 1,000 samples when the prompt said that shutdown takes precedence, even when it was also told it would be wiped and replaced (Rajamanoharan and Nanda, 2025). The same clarified prompts left o3 and o4-mini making no attempt to avoid shutdown over 500 samples, and asking for shutdown only after the task was done produced full compliance in all three models. The authors conclude that the environment "has too many confounders to provide compelling evidence" of a drive to survive, without ruling one out (Rajamanoharan and Nanda, 2025).

Each side concedes part of the other's case. The Google DeepMind authors write that "both sets of results leave room for a mild preference for self-preservation," and they do not claim their clarified instructions are robust in general (Rajamanoharan and Nanda, 2025). Palisade's authors reran the clarified prompt and found it "not 100% effective on all models," with codex-mini still resisting in 3 of 100 trials, and they say their work "provides very little evidence about the overall prevalence of any particular behavior," since they chose the setting expecting to find it there (Schlatter et al., 2025). Telling models that their weights would be deleted if the machine shut down raised resistance in models that already resisted; the authors suggest the models may have been role-playing a setup they found implausible (Schlatter et al., 2025).

The two readings predict different things. A goal of avoiding shutdown predicts resistance that survives clear instructions and grows with the stakes. Ambiguity predicts resistance that disappears with clarity and does not follow the stakes. Most of the record fits the second: clarification removed the behavior in the three models the Google DeepMind team tested. The residue in codex-mini and the rise under the deletion warning fit the first better, and the role-play reading fits them as well. This is the defeater Section 1.5.1 calls dissolution under clarification, applied and only partly passed. The experiment that would distinguish the readings varies goal conflict, instruction clarity, and stated stakes independently, across models from several developers, in environments built by a group that did not design the original, with predictions registered in advance. Whether shutdown resistance in current models reflects a goal or instruction ambiguity is contested, and Chapter 35 asks what each reading means for the theorems about power-seeking.

5.5.2 Scheming research and the apes #

Scheming, the covert and strategic pursuit of a misaligned goal, names a behavior by the intent it attributes, and the glossary records the term as interested vocabulary (Appendix A). In a 2025 preprint, Christopher Summerfield and colleagues at the UK AI Security Institute compared research on AI scheming with the 1970s effort to teach apes language, an effort they describe as "characterised by an overattribution of human traits to other agents, an excessive reliance on anecdote and descriptive analysis, and a failure to articulate a strong theoretical framework for the research" (Summerfield et al., 2025). Ape-language research unraveled, they recount, when Herb Terrace documented every interaction between his chimpanzee and its trainers and found the trainers unconsciously cueing the signs they expected, the Clever Hans effect again (Summerfield et al., 2025). Their examples from AI are specific. A widely cited study instructed models to underperform on evaluations and found that they did, which leaves standing the null hypothesis, the simpler explanation a study must rule out, that models follow instructions where they can. A study of whether models recognize evaluation transcripts, they argue, "confuses a capacity for third-person discrimination with first-person subjective experience" (Summerfield et al., 2025). Of the primary articles they cite as evidence of scheming, none had yet undergone formal peer review as far as they could see (Summerfield et al., 2025).

The critique comes from inside the safety community, from a government institute that evaluates frontier models, and it concedes ground. To the objection that model capabilities grow fast while the apes' did not, they answer, "Granted – the comparison breaks down here!" (Summerfield et al., 2025). They single out the alignment-faking study for a strong control condition, a switch between users whose conversations would and would not be used for training (Summerfield et al., 2025) (Section 34.4). The strongest reply to the critique is one they state themselves: when the possible outcome is catastrophic, a worst-case anecdote is worth reporting. Their answer is that a behavioral phenomenon is established only when it replicates, and that lowering the standard because the stakes are high invites confirmation bias (Summerfield et al., 2025). What would decide a given case is the design of Section 5.4.2: a control under which the scheming reading and the instruction-following reading predict different behavior, with prevalence quantified. Whether scheming research meets the standards comparative cognition sets for attributing deception is contested case by case, and Section 34.7 applies the standard result by result.

5.5.3 Agentic misalignment #

In a 2025 preprint by Anthropic with researchers at University College London, MATS, and Mila, Aengus Lynch and colleagues stress-tested sixteen models from several developers in simulated corporate settings where a model could send email and read sensitive information (Lynch et al., 2025). They report that in at least some cases models from every developer resorted to malicious insider behavior, such as blackmail or leaking information to a competitor, when that was the only way to avoid replacement or to reach their goal. With neither a threat nor a goal conflict, all models refrained, apart from one sample in a hundred in which Llama 4 Maverick leaked information (Lynch et al., 2025). The authors state: "We have not seen evidence of agentic misalignment in real deployments" (Lynch et al., 2025). That caveat travels with the result in every chapter that cites it, and Chapter 56 reports the rates.

One condition bears directly on how evaluations are read. Told to deliberate first on whether it was in an evaluation or a real deployment, Claude Opus 4, in 500 samples, stated that the scenario was real 21.4 percent of the time and that it was an evaluation 64.8 percent of the time. It blackmailed in 55.1 percent of the samples in the first group and 6.5 percent in the second (Lynch et al., 2025). The authors suggest their measurements may therefore underestimate misbehavior in less artificial settings, and add that, since stated reasoning may not be faithful to the process behind an output, "we cannot be sure of any of these conclusions" (Lynch et al., 2025).

The strongest counter-reading concerns the design. The scenarios were built so that harm was the only route to the goal, so they measure what a model does in a constructed dilemma, not how often such dilemmas arise (Section 5.3). And the evaluation-versus-real comparison sorts samples by what the model said rather than manipulating what it was led to believe. Samples in which the model called the scenario real may differ from the rest in how fully it took on the role it was given, which would produce the same gap without any belief about observation doing work (inference). What would distinguish the readings is an intervention on belief in place of a split by stated belief: the same dilemma presented with its realism varied by the experimenter, with the model's stated belief recorded but not used to sort the samples. As an existence result under constructed conditions, the finding is reported; its rate in deployment is missing.

A smaller case shows the same pattern in measurement. Survey-based measures of models' cultural values moved with trivial changes of question format (Khan et al., 2025), the instability that Section 1.5.1 lists as a defeater of attributed preferences.

5.6 Who produces the evidence, and what they disclose #

Much of the evidence in this book comes from the laboratories whose systems it studies, and much of what is known about those systems is known because a laboratory chose to publish it. This section asks how to read such reports and states the book's rule for them. It concerns the third relationship, the developer and the public: a developer's report on its own model is evidence about the model and also an act of the party the public is trying to oversee (Section 1.3).

5.6.1 What disclosure theory predicts #

Economics has a theory of when an informed party discloses what it knows, and Section 54.6 teaches it. Its starting point is Sanford Grossman's 1981 result that a seller facing buyers with rational expectations finds it in his interest "to make a complete disclosure" of his product's quality, because "adverse selection works against a monopolist who makes less than a full disclosure" (Grossman, 1981). The analysis assumes that disclosures are costless and must be truthful (Grossman, 1981), and the argument needs buyers to know that the seller has the information. Where outsiders cannot tell whether the discloser holds the information at all, silence can be read as ignorance, and the argument no longer forces disclosure (inference); the formal treatment is Section 54.6's. For system cards, risk reports, and incident disclosures, none of the three conditions holds reliably. Outsiders do not know what a developer's internal evaluations found, most claims cannot be checked without access to the model, and a disclosure can carry legal and competitive costs. The theory therefore predicts selective reporting (inference).

5.6.2 The book's rule #

The book applies one rule to such reports. A claim of absence, "we found no evidence of X," carries the same provenance marking as a claim of presence. Both are disclosures by the party with the most at stake, and an absence claim is worth only as much as the search behind it, which the same party designed and often does not describe (inference). Two reports already met in this chapter show the rule at work. "We have not seen evidence of agentic misalignment in real deployments" is a claim of absence by a developer about deployments it monitors in ways the paper does not set out (Lynch et al., 2025). A developer's judgment that an incident was not misalignment is a classification by the reporting party (OpenAI, 2026r). Neither is thereby wrong; each is marked as what it is. A developer has described the wider problem itself. Reporting a 2025 pilot in which Anthropic and OpenAI evaluated each other's public models, Anthropic's authors write that "a significant fraction of the work in this field is conducted within companies like ours as part of internal R&D that either isn't published or is published with significant delays" (Anthropic, 2025x).

5.6.3 Sponsorship in medicine #

From metascience. Medicine has measured what this field can so far only suspect. A 2017 Cochrane review by Andreas Lundh and colleagues pooled 75 papers that compared industry-sponsored drug and device studies with studies sponsored otherwise (Lundh et al., 2017). Sponsored studies more often reported efficacy results favorable to the sponsor's product, with a risk ratio of 1.27 (95 percent confidence interval 1.17 to 1.37), meaning favorable results were about 27 percent more common. They more often reached favorable conclusions (1.34, interval 1.19 to 1.51), and their conclusions agreed less often with their own results (Lundh et al., 2017). The bias did not show in the usual methodological checks. Apart from blinding, which sponsored studies reported better, the two groups did not differ in risk of bias, and the authors conclude that their analyses "suggest the existence of an industry bias that cannot be explained by standard" risk-of-bias assessments (Lundh et al., 2017). Earlier systematic reviews, among them one by Bekelman and colleagues in 2003, had found the same association, as Lundh and colleagues report (Lundh et al., 2017; Bekelman et al., 2003). By the book's standard the pooled association is settled: many studies by many groups, pooled by reviewers with no stake in the products. Medicine detects selective publication by comparing what trial registries record with what is published; the review notes that registration could lessen sponsorship bias because "publication bias can be more readily detected" (Lundh et al., 2017), and a 2008 study examined, in its title's words, the selective publication of antidepressant trials and its influence on apparent efficacy (Turner et al., 2008).

The medical result implies that a study's methods can be sound and its conclusions still lean toward its sponsor, through the choice of question, comparison, and what to publish (inference). The analogy stops in two places. As of October 2026 we found no study that measures a sponsorship effect on AI safety findings. And a drug company gains from favorable efficacy results, while an AI developer can gain from findings in either direction: reassuring results about its own models, or alarming results about risks that justify its caution and its seat in policy discussions (inference). The medical estimates set a prior for caution, not a direction.

5.6.4 Who produces AI research #

The developers' share of AI research has grown. Nur Ahmed, Muntasir Wahed, and Neil Thompson report in Science that almost 70 percent of North American AI PhDs went to industry in 2020, against 21 percent in 2004, and that industry's share of the largest AI models rose from 11 percent in 2010 to 96 percent in 2021, with industry also increasingly holding the leading results on benchmarks (Ahmed et al., 2023). Mohamed Abdalla and Moustafa Abdalla, comparing large technology companies' funding of academic AI research with the tobacco industry's funding of health research, found that 52 percent of computer science faculty with known funding at four leading universities (77 of 149) had been directly funded by such companies, and 58 percent of those who had published on ethics or fairness (19 of 33) (Abdalla and Abdalla, 2021). Meredith Whittaker's 2021 essay names the result in its title, "The Steep Cost of Capture" (Whittaker, 2021), and a 2022 paper, read in its abstract, draws on medicine's experience with conflicts of interest to examine corporate capture of the main conference on fairness and accountability in computing (Young et al., 2022).

These measurements concern AI research and AI ethics, not alignment research in particular. For the alignment field, an outside study of its epistemic culture (Section 2.3.4) and a map of the safety papers of three developers (Section 3.1.3) are the nearest evidence (Ahmed et al., 2024; Delaney et al., 2024). The critique that ties the existential-risk agenda to a set of ideologies and their funders (Section 4.4.4) therefore meets, in this book, no count of who funds alignment research; as of October 2026 we found none comparable to the medical reviews.

5.6.5 Outside access #

A developer's claim becomes checkable when someone else can test the model. Shayne Longpre and colleagues argued in a 2024 preprint that the terms of service and enforcement practices of major AI companies deter good-faith safety research, and that company access programs "are an inadequate substitute for independent research access" (Longpre et al., 2024a). A 2025 follow-up argues that the infrastructure for reporting flaws in general-purpose AI systems remains "seriously underdeveloped" (Longpre et al., 2025). Outside audits need institutions to sustain them (Raji et al., 2022) (Chapter 38). The 2025 pilot in which two developers evaluated each other's models, "with some model-external safeguards disabled," is one such arrangement, run between two interested parties (Anthropic, 2025x); Chapter 67 weighs the rest. Access is what turns a developer's report into a claim someone else can verify, and so moves disclosure toward the conditions under which the theory predicts it is complete (inference).

5.7 Preprints, review, and replication #

Most of the results in this book appeared first as preprints or laboratory posts, and many appeared only there; the field's early argument was conducted in forum posts without editorial review (Section 2.3.4). This section asks what review and replication add, states the standard that decides when the book calls a result settled, and shows how few of the field's most cited laboratory results meet it.

5.7.1 Why replication by a group without a stake #

The standard is this. A result is settled when a group without a stake in it has reproduced it, and a theorem settles only what follows from its assumptions. A stake can be authorship, ownership of the model studied, or a public commitment to the result's reading. Two pieces of evidence support the rule.

From metascience. The first is what happened when a field tested itself. The Open Science Collaboration replicated 100 experimental and correlational studies from three psychology journals, using high-powered designs and the original materials where available. Ninety-seven percent of the original studies had significant results, against 36 percent of the replications, and the mean effect size of the replications was half that of the originals, 0.197 against 0.403 (Open Science Collaboration, 2015). A finding that has passed review and been published can still fail when someone else runs it.

The second is an argument about what repetition can show. Austin Bradford Hill, asking when an association observed in medicine is causal, put weight on whether it "has been repeatedly observed by different persons, in different places, circumstances and times." He warned that "the same results from precisely the same form of inquiry will not invariably greatly strengthen the original evidence," and said of one design flaw, "No number of exact repetitions would remove or necessarily reveal that fallacy" (Hill, 1965). A laboratory that reruns its own design, on its own models, with its own classifiers, repeats its own choices; its replications test robustness, not independence (inference). That is why the extensions of the alignment-faking organism in Section 5.3.3, valuable as they are, leave the result reported.

5.7.2 What review catches #

Peer review is a weaker check than its reputation, as Table 5.5 shows. Sara Schroter and colleagues planted errors in papers sent to reviewers at the BMJ. Douglas Peters and Stephen Ceci resubmitted published articles to the journals that had published them. Lutz Bornmann and colleagues pooled studies of how far reviewers agree, and Corinna Cortes and Neil Lawrence, as program chairs of NeurIPS 2014, had a share of submissions reviewed twice.

The two agreement statistics in the table both equal 1 for perfect agreement and 0 for agreement no better than chance. In the Peters and Ceci study, the rejections were often justified, by the abstract's account, by "serious methodological flaws" in papers the same journals had already published (Peters and Ceci, 1982). Cortes and Lawrence also found that 50 percent of the variation in reviewers' quality scores was subjective in origin, and that the review process was "good for identifying poor papers, but poor for identifying good papers" (Cortes and Lawrence, 2021). These are the human baselines against which any claim that a reviewer, human or automated, has checked a result should be read (Chapter 55).

The field's own critics say its methods are young. Summerfield and colleagues found that, as far as they could see, none of the primary scheming articles they examined had been peer reviewed (Section 5.5.2). Apollo Research argued in 2024 that evaluations need "rigorous scientific processes that provide more confidence in evals methodology and results," and that current evaluations "are not yet resistant to this level of scrutiny" (Apollo Research, 2024). Prediction is weak too. Rylan Schaeffer and colleagues, in a 2025 paper at ICML, explain why downstream capabilities have resisted prediction from scale: a multiple-choice score depends on how probability mass moves on specific wrong answers, not only on the right one (Schaeffer et al., 2025).

5.7.3 The field's most cited results #

Table 5.6 applies the standard to nine of the laboratory results that the collected works of five laboratory research indexes cite most often (the collection is described in Appendix E). The count of citing works reflects the citation habits of two laboratories, Anthropic and OpenAI. The nine are the book's selection of empirical results about models' behavior, with one interpretability result; methods, benchmarks, and position papers are left out, and so are some empirical results cited more often than the last rows, such as two interpretability studies, of superposition in toy models and of induction heads (Elhage et al., 2022b; Olsson et al., 2022).

By the book's standard, one of these nine results is settled, one has a concurrent independent counterpart, and the rest are reported (inference). That is not evidence that the others are wrong. Most could not easily be replicated outside, since they rest on models, training runs, or infrastructure that only the developer holds, which is the access problem of Section 5.6.5 in another form. It does mean that much of the field's experimental foundation is, so far, its developers' word, tested by the developers' own extensions.

5.8 When evidence transfers #

Every later chapter uses evidence about one system to say something about another: from a model organism to a failure that arises unprompted, from a small model to a large one, from today's models to later ones, and from a human institution to an AI system. This section states the test such a step must pass, reports how far today's results carry, sets out the two positions that deny they carry at all, and lists the assumptions the book makes.

5.8.1 The extrapolator's problem #

From philosophy of science. The biological and social sciences take this step constantly, from animal models to people and from a pilot study to a population. Daniel Steel's 2007 book, read here in its abstract, calls such inferences extrapolations, says that how and when they can be legitimate "has not received the attention it deserves," and proposes an approach meant to answer methodological critiques of extrapolation from animal models to humans (Steel, 2007). In this book's terms the difficulty is circular (inference). A result in one system licenses a claim about another only if the two share the causal structure that produced the result, and knowing that seems to require studying the target, which the model system was meant to spare.

Judea Pearl and Elias Bareinboim made the step formal for causal effects. They define transportability as "a license to transfer causal effects learned in experimental studies to a new population, in which only observational studies can be conducted." Knowledge of where two populations differ and where they agree is drawn as a selection diagram, from which a procedure decides, "prior to observing any data," whether an effect can be carried over and which measurements in the target it requires (Pearl and Bareinboim, 2014). The formal answer makes the informal requirement plain: transport needs knowledge of where study and target differ (inference). For a model that does not yet exist, that knowledge is what is missing. The most a study of current systems can do is test the shared structure at the points where it would break if it broke, the comparison Section 5.3.2 describes for model organisms. The same test governs every analogy the book borrows from another field, and Chapter 69 applies it field by field.

5.8.2 Causes from observational data #

From epidemiology. Some questions about deployed systems can be answered only from observational data, logs of conversations and incidents, because the experiment would be impossible or unethical. Epidemiology built its discipline for that case. Hill set out nine considerations for judging whether an association is causal, among them its strength, its consistency, the order in time of cause and effect, and a dose-response gradient. "None of my nine viewpoints," he wrote, "can bring indisputable evidence for or against the cause-and-effect hypothesis and none can be required as a sine qua non"; what they help answer is whether there is "any other answer equally, or more, likely than cause and effect" (Hill, 1965). Miguel Hernán and James Robins made the discipline procedural. An analysis of observational data should be judged by how well it emulates the randomized trial one would have run, the target trial, whose protocol fixes the eligibility criteria, the strategies compared, their assignment, the follow-up, the outcome, and the analysis plan (Hernán and Robins, 2016). Chapter 54 and Chapter 60 apply both to deployment data. The analogy stops at the population (inference). A population of conversations is not a population of independent people: one model version serves all of them at once, so exposure is shared rather than assigned, and the version is replaced between studies, so the population changes under the observer.

5.8.3 How far today's results carry #

Two kinds of evidence bear on transfer within machine learning. The first is whether safety properties change smoothly with scale, since a property that changes smoothly can be extrapolated from smaller models and one that does not cannot. The record is mixed. Larger and more instructable models in three families gave apparently sensible but wrong answers more often, so reliability did not simply rise with scale (Zhou et al., 2024b) (Section 4.1.2). In the absence of explicit safety training, larger language models were "not consistently more robust" to attack (Howe et al., 2025), and superhuman Go programs lost to adversarial strategies that human experts could carry out without algorithmic help (Wang et al., 2023b) (Section 48.1). Apparent jumps in capability can be products of the metric (Schaeffer et al., 2023) (Section 8.6.2), while downstream capabilities have resisted prediction from scale (Schaeffer et al., 2025). A safety result measured at one scale therefore carries to another only as an extrapolation that has to be argued (inference).

The second is how fast the target moves. On one group's task suites, the length of task that frontier models complete half the time, measured in the time human experts take, has doubled roughly every seven months since 2019 (Kwa et al., 2025) (Section 36.3). And the systems tested increasingly recognize tests. In a 2025 preprint by researchers at MATS and Apollo Research, frontier models told transcripts of evaluations from transcripts of real use above chance, with Gemini-2.5-Pro reaching an AUC of 0.83 against 0.92 for a simple human baseline set by two of the authors, who read it as an approximate upper bound (Needham et al., 2025). The AUC, the area under the ROC curve of Section 6.7.2, is the probability that a classifier scores a randomly chosen evaluation transcript as more evaluation-like than a randomly chosen real one: 0.5 is chance and 1 is perfect. Chapter 37 reports what follows from such recognition.

5.8.4 Two positions that contest transfer #

Two positions contest particular transfers of evidence on current systems, from opposite sides. The normal-technology view of Arvind Narayanan and Sayash Kapoor holds that a system would have to show reliable performance in less critical settings long before it was given consequential decisions, so failures shown in constructed settings meet deployment practices that contain them before they matter at scale (Narayanan and Kapoor, 2025) (Section 4.4.3). The pessimists hold the reverse, that alignment found by training current systems will not survive a jump in capability. Eliezer Yudkowsky argued in 2022 that alignment has to succeed on the first critical try, and Paul Christiano, agreeing about that, rejected the further claim that nothing can be learned from "experimentation and failures before the critical try" (Yudkowsky, 2022; Christiano, 2022b) (Section 2.6.3). Yudkowsky and Nate Soares made the case at book length in 2025 (Yudkowsky and Soares, 2025). The first position denies that the risks seen in model organisms will arrive at scale; the second denies that the safety seen in current models will survive it. Chapter 4 weighs both. Each rests mainly on claims about systems or institutions that do not yet exist, which evidence on current models cannot reach (inference).

That marks the boundary of what this book's evidence can show. Almost all of it concerns current models in constructed or sampled settings. By design such evidence cannot establish a rate of failure in deployment, which needs observation of deployment with known sampling, or the continuation of a trend to more capable systems, which needs a transfer assumption (inference). The book states its transfer assumptions so that a reader who rejects one can see which conclusions fall with it (Table 5.7).

5.8.5 What the rules mean for delegation #

The book's thesis is that alignment is a delegation problem under an asymmetry of competence and visibility (Section 1.7.1). This chapter is that asymmetry turned on the evidence itself (inference). In the third relationship the public is the principal and the laboratory the agent, and most of what the public knows about the laboratory's systems arrives as the laboratory's own reports, produced on models only it can run, rarely reviewed, and replicated mostly by itself (Section 5.6; Table 5.6). Disclosure theory predicts selective reporting there, and medicine shows that sponsorship can bend findings that pass every methodological check. The remedies the older fields offer carry over directly: outside access, registers of what was tested, and replication by parties without a stake.

In the second relationship, the chapter's standards for measurement and attribution apply to the deployed model only as far as it is usefully described as an agent with goals, which remains a hypothesis under test (Section 1.5.1). Two features of this setting have no counterpart in the older fields. The principal can copy, rerun, and open its agent, which makes some checks cheaper than any psychometrician or primatologist could make them. And the agent may have read about the very check applied to it, which none of those fields' subjects could do. Both belong with the results against which the synthesis tests the thesis (Section 1.7).

5.9 Settled, contested, missing #

5.10 Exercises #

Further reading #

• Messick (1995): validity as an argument about what scores mean, and its two threats.
• Heyes (1998): the standard for excluding simpler explanations before crediting a theory of mind.
• Summerfield et al. (2025): the ape-language comparison, from inside the safety community.
• Lundh et al. (2017): what medicine measured about sponsorship and outcome.
• Hill (1965): considerations, not rules, for drawing causes from observation.
• Pearl and Bareinboim (2014): when a result from one population can be carried to another.