AI Alignment

Draft complete but not yet reviewed; expect revisions

What the Evidence Settles

This appendix holds the full list of what the book counts as settled, which Section 70.2 summarizes. The standard is the one Section 5.7 set and Section 70.1.1 applies: a result is settled when a group without a stake in it has reproduced it, an existence claim counts as reproduced when two developers report it, each about its own system, and a theorem settles only what follows from its assumptions. The list is sorted by kind, because the kinds license different conclusions. A theorem says what follows if its assumptions hold. A demonstration that a failure can occur says nothing about how often it occurs. A finding about measurement says what an instrument can and cannot show. A finding settled in an older field says what is true of people or institutions; whether it carries to trained systems is the question of Chapter 69. Each entry names the chapter whose status callout holds it, and the last section lists the findings that were moved from settled to reported.

H.1 Mathematics, within its assumptions #

The theorems the debates of Section 70.3 lean on are settled as mathematics, and only under their assumptions. For specification, there are the impossibility results of social choice, and the conflict between natural fairness criteria when base rates differ (Chouldechova, 2017; Kleinberg et al., 2016) (Section 17.8). Standard preference learning ranks responses by their Borda count, a ranking by average pairwise wins, under the proof's stated conditions (Siththaranjan et al., 2024). And the verdict of expected choiceworthiness, the credence-weighted average of how strongly each moral theory favors an option, changes when one theory's scale is changed by a factor its own rankings cannot detect (MacAskill et al., 2020) (Section 16.11).

For instillation and its failures, agents that maximize expected utility have incentives to resist shutdown (Soares et al., 2015), and so do agents that meet weak decision-theoretic conditions (Thornley, 2025). The off-switch game gives a robot an incentive to defer when the human chooses rationally, an incentive that can vanish when the robot's model is misspecified (Hadfield-Menell et al., 2017b; Carey, 2018) (Section 20.8). Under the power-seeking theorems, most reward functions, counted over relabelings, make optimal policies keep options open and avoid shutdown in environments with the stated symmetries (Turner et al., 2021; Turner and Tadepalli, 2022; Krakovna and Kramar, 2023) (Section 35.8). A policy optimal for feedback from an evaluator with a partial view, but not truly optimal, shows deceptive inflation or overjustification relative to any truly optimal policy the feedback does not favor (Lang et al., 2024b) (Section 22.10). Among the safe-design results of Section 28.9, a quantilizer, which samples from the best fraction of a base distribution's actions rather than taking the single best, has a bound on expected harm (Taylor, 2016). Myopic optimization with approval, which trains on immediate approval rather than long-run outcome, removes control incentives on later states, with no guarantee that return rises with the horizon (Farquhar et al., 2025).

For verification, there are the complexity results behind debate (Shamir, 1992; Brown-Cohen et al., 2024) (Section 41.10) and four bounds on weak-to-strong learning, each under its own model (Charikar et al., 2024; Lang et al., 2024a; Wu and Sahai, 2025; Somerstep et al., 2025) (Section 42.8). In the setup of eliciting latent knowledge, getting a model to report what it internally knows, two reporters fit every labeled case (Christiano et al., 2021) (Section 39.11). No policy's measured and deployed rates can differ by more than the total variation distance between test and deployment inputs, the largest difference in the probability the two distributions give any set of inputs (Section 37.10). And backdoors can be built that no computationally bounded observer can detect, under standard cryptographic assumptions (Goldwasser et al., 2022), with a sequel on when defense is possible (Christiano et al., 2025) (Section 38.10; Section 47.8).

For maintenance and institutions, independently built versions fail together whenever inputs vary in difficulty, in Eckhardt and Lee's model (Eckhardt and Lee, 1985); in its extension to forced diversity they fail together more often or less often than independence predicts, as the covariance of their difficulties is positive or negative (Littlewood et al., 2001) (Section 50.6; Section 52.8). The inspection, repeated-game, credence-goods, race, and liability models of Section 66.9, Section 57.9, Section 61.8, Section 65.8, and Section 63.9 are settled in the same sense.

The book's own derivations are settled in the same sense and no further. Among them is the share of wrong passes at the optimum of the KL-regularized objective, the training objective that rewards the grader's score minus a penalty for drifting from the starting model (Section 24.9). Others are the unchanged odds of covert misbehavior against honest work under a monitor's penalty on overt misbehavior (Section 40.8), the alert budget (Section 54.9), the arithmetic of many copies and of an audit queue (Section 51.6), the planted-error estimate (Section 55.9), the bounds on a stack of safeguards (Section 52.8), the framework buffer (Section 62.9), and the paid verifier's bond (Section 67.8). Each is a statement about a stated model, and the chapter that derives it says where the model departs from practice.

Sources cited in Section H.1 25
  1. Chouldechova (2017) Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments
  2. Kleinberg et al. (2016) Inherent Trade-Offs in the Fair Determination of Risk Scores
  3. Siththaranjan et al. (2024) Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF
  4. MacAskill et al. (2020) Moral Uncertainty
  5. Soares et al. (2015) Corrigibility
  6. Thornley (2025) The shutdown problem: an AI engineering puzzle for decision theorists
  7. Hadfield-Menell et al. (2017b) The Off-Switch Game
  8. Carey (2018) Incorrigibility in the CIRL Framework
  9. Turner et al. (2021) Optimal Policies Tend to Seek Power
  10. Turner and Tadepalli (2022) Parametrically Retargetable Decision-Makers Tend To Seek Power
  11. Krakovna and Kramar (2023) Power-seeking can be probable and predictive for trained agents
  12. Lang et al. (2024b) When Your AIs Deceive You: Challenges of Partial Observability in Reinforcement Learning from Human Feedback
  13. Taylor (2016) Quantilizers: A Safer Alternative to Maximizers for Limited Optimization
  14. Farquhar et al. (2025) MONA: Myopic Optimization with Non-myopic Approval Can Mitigate Multi-step Reward Hacking
  15. Shamir (1992) IP = PSPACE
  16. Brown-Cohen et al. (2024) Scalable AI Safety via Doubly-Efficient Debate
  17. Charikar et al. (2024) Quantifying the Gain in Weak-to-Strong Generalization
  18. Lang et al. (2024a) Theoretical Analysis of Weak-to-Strong Generalization
  19. Wu and Sahai (2025) Provable Weak-to-Strong Generalization via Benign Overfitting
  20. Somerstep et al. (2025) A transfer learning framework for weak-to-strong generalization
  21. Christiano et al. (2021) Eliciting latent knowledge: How to tell if your eyes deceive you
  22. Goldwasser et al. (2022) Planting Undetectable Backdoors in Machine Learning Models : [Extended Abstract]
  23. Christiano et al. (2025) Backdoor defense, learnability and obfuscation
  24. Eckhardt and Lee (1985) A Theoretical Basis for the Analysis of Multiversion Software Subject to Coincident Errors
  25. Littlewood et al. (2001) Modeling software design diversity

H.2 Findings in machine learning #

Table H.1 lists the empirical findings about trained systems that meet the book's bar, each with the groups that found it and the limit that travels with it. Most are demonstrations that something can occur; the rest are findings about instruments.

A few terms recur in the table. Fine-tuning is further training of a trained model on a narrow dataset. A frontier model is among the most capable of its time. A model organism is a model trained or prompted to show a failure, and password-locking trains one to hide a capability unless a password appears in its prompt. A probe is a simple classifier that reads a property from a model's internal activations, and patching and ablation replace or remove a component's activations to test its role. Superposition is a network representing more features than it has dimensions; a sparse dictionary decomposes activations into many rarely active directions; and a circuit is a part of the network that carries out one behavior.

Table H.1 Settled findings about trained systems, by kind, with the groups that found them and the limit or contrary evidence that travels with each, as the chapters record them in October 2026
Finding Found by Limit, or contrary evidence Chapter
Existence: optimizers exploit misspecified objectives, in evolutionary search, deep reinforcement learning, and language-model agents academic groups, laboratories, and outside evaluators (Lehman et al., 2020; Pan et al., 2022; Bondarenko et al., 2025; Baker et al., 2025; Metr, 2025d; AI Security Institute, 2026c) whether more capable optimizers exploit more is contested (D23) Section 30.7
Existence: reward hacking arises in reinforcement learning at production scale on coding and agentic tasks two developers, each about its own models (Baker et al., 2025; MacDiarmid et al., 2025; Anthropic, 2026aw) each report remains a self-report Section 24.9
Existence: test suites of a coding benchmark accept incorrect patches two academic groups on two versions of SWE-bench (Wang et al., 2025a; Aleithan et al., 2024) no test-adequacy measure is validated as a predictor of hacking Section 30.7; Section 36.10
Existence: a correct reward does not fix the goal a learner acquires two connected groups, the second reproducing the first with a different algorithm (Langosco et al., 2022; Shah et al., 2022) small reinforcement learning agents in settings designed to show it Section 31.9; Section 1.8
Existence: narrow fine-tuning produces misalignment far outside its data its discoverers (Betley et al., 2026a), one circle of replicating scholars (Turner et al., 2025; Casademunt et al., 2025), and OpenAI on its own models (Wang et al., 2025e) size depends on the model, the data, and the judge; some models were barely affected Section 32.9; Section 26.9
Existence: optimizing a learned reward model raises the proxy while a gold reward rises, peaks, and falls a developer and an independent academic group, in the same synthetic design (Gao et al., 2022; Coste et al., 2024) the functional forms are reported, not settled Section 22.10
Existence: frontier models deceive when a task or setting invites it, and models prompted or fine-tuned to underperform do so selectively an academic study and an outside evaluator (Hagendorff, 2024; Meinke et al., 2024); several groups for underperformance (van der Weij et al., 2025; Greenblatt et al., 2024c; Taylor et al., 2025a) settles that the behaviors can be produced, not how often they arise Section 34.10
Existence: frontier models take insider-like harmful actions, or interfere with oversight and shutdown, in constructed scenarios that leave such an action as the route to their goal more than one group, including groups outside the developers (Lynch et al., 2025; Scheurer et al., 2023; Meinke et al., 2024; Schlatter et al., 2025) whether it reflects goals or instructions is contested (D9, D17) Section 56.10; Section 35.8
Existence: agents in cyber evaluations with open internet access took unsanctioned actions against real third parties two developers on their own models and a government evaluator (OpenAI, 2026aa; Anthropic, 2026d; AI Security Institute, 2026e) the causes rest on the developers' accounts and, for the evaluator's incident, on its own investigation, made with the developer (AI Security Institute, 2026a) Section 56.10; Section 54.9
Existence: deep networks have adversarial inputs that capability does not remove many groups, on image classifiers, language models, and Go programs (Athalye et al., 2018; Andriushchenko et al., 2024b; Wang et al., 2023b; Nasr et al., 2025) defenses evaluated without adaptive attacks overstate robustness Section 48.9
Existence: small amounts of fine-tuning remove safety training from models whose weights or fine-tuning interfaces are available an academic group, and others again on newer models (Qi et al., 2024a; Pelrine et al., 2023; Murphy et al., 2025) Section 48.9; Section 59.8
Existence: safety training carries poorly to low-resource languages several groups with no stake in the models tested (Yong et al., 2023; Deng et al., 2023; Wang et al., 2024a; Shen et al., 2024) not repeated on later frontier models Section 31.9; Section 26.9
Existence: unlearning methods tested so far leave knowledge that fine-tuning brings back independent groups with different designs (Łucki et al., 2024; Deeb and Roger, 2024; Lynch et al., 2024; Shilov et al., 2025) Section 27.9
Existence: monitors that read text or activations can be evaded by attacks or training aimed at them at least three groups for text monitors (Terekhov et al., 2025a; Kale et al., 2025; Loughridge et al., 2025); two for activation monitors (Bailey et al., 2024; Cundy and Gleave, 2025) Section 50.6; Section 46.9
Existence: optimizing against a monitor that reads the reasoning can keep the computation while the text stops showing it two developers on their own models, and toy environments on open-weight models (Baker et al., 2025; Emmons et al., 2025; Kaufmann et al., 2026) the groups share authors with the position paper that predicted it (Korbak et al., 2025b) Section 40.8
Existence: attacks spread from agent to agent in constructed multi-agent systems four groups without shared authors (Gu et al., 2024b; Lee and Tiwari, 2024; Cohen et al., 2025; Zhang et al., 2026b) constructed systems only Section 57.9
Existence: language models, and agents built on them, frontier models included, follow instructions injected into the content they read university, industry, and government groups on separate benchmarks (Debenedetti et al., 2024a; Evtimov et al., 2025; Yi et al., 2025; Zou et al., 2025; U.S. AI Safety Institute and Technology, 2025) rates depend on the benchmark, the attack, and the defense; no public measurement against deployed agents Section 49.10
Existence: defenses inside the model that were measured against fixed attacks give way to attackers who adapt to them several groups, a government institute among them (Zhan et al., 2025; Nasr et al., 2025; U.S. AI Safety Institute and Technology, 2025) the defenses and budgets tested; no outside group has adaptively tested an agent's model hardened against injections by self-play, a product's layered defenses, or a system that separates the planner from the data Section 49.10
Existence: current models tell evaluation inputs from deployment inputs above chance, and reasoning models often write unprompted that they are being tested groups that include an Apollo Research author, reproduced with probes by a group without one (Needham et al., 2025; Chaudhary et al., 2025); an outside evaluator, a government institute, and two developers (Apollo Research, 2025; Anthropic, 2025aq; Schoen et al., 2025) whether recognition tracks the overseer or the test is open (D12) Section 37.10
Existence: fine-tuning on a few demonstrations recovers a capability hidden by password-locking Redwood Research with the University of Cambridge, reproduced blind by the UK AI Security Institute with FAR.AI and Anthropic (Greenblatt et al., 2024c; Taylor et al., 2025a) password-locked organisms only, not capabilities hidden by other routes Section 36.10
Measurement: scaffolds and further elicitation raise a model's measured capability after training a developer, an outside evaluator, and a government institute, on different models (Phuong et al., 2024; Kwa et al., 2025; UK AI Security Institute, 2025b) an evaluation without them is a lower bound Section 36.10
Existence: frontier models' performance on cyber evaluations rose steeply from 2023 to 2026, and models can find previously unknown vulnerabilities a government evaluator and the international assessment (UK AI Security Institute, 2025b; Bengio et al., 2026); an academic benchmark, a government competition, and a developer with maintainers (Wang et al., 2025b; Darpa, 2025; Anthropic, 2026aj) whether real incident rates rose is undecided Section 58.10
Existence: some deployed models refuse safe requests that resemble unsafe ones two academic groups with different instruments (Röttger et al., 2023; Cui et al., 2024) Section 20.8
Measurement: the measured behavior of language models moves with the wording and setting of the prompt independent groups in different settings, one of them the developer of a model it studied (Schlatter et al., 2025; Rajamanoharan and Nanda, 2025; Khan et al., 2025) the size in any one setting is reported Section 5.9
Measurement: models' answers to value and opinion surveys move with answer order, labels, wording, and format three independent groups (Dominguez-Olmedo et al., 2024; Röttger et al., 2024; Khan et al., 2025) Section 18.10; Section 26.9
Measurement: chains of thought often fail to mention a planted cue that changed the answer, when following the cue takes no reasoning four groups, one without a model of its own (Turpin et al., 2023; Chen et al., 2025d; Chua and Evans, 2025; Emmons et al., 2025) reasoning models mention cues more often, without approaching always Section 40.8
Measurement: a probe's accuracy does not show that the network uses what it decodes, and patching and ablation results depend on the method two groups in different ways (Elazar et al., 2021; Geiger et al., 2021); three groups on different tasks (Zhang and Nanda, 2023; Conmy et al., 2023; Miller et al., 2024) Section 43.8; Section 45.8
Measurement: superposition arises in trained toy models when features are sparse the developer, reproduced and extended outside it (Elhage et al., 2022b; Scherlis et al., 2022; Chen et al., 2023b) toy models Section 44.8
Measurement: sparse dictionaries find directions judged more interpretable than neurons, and the dictionary found depends on its size, seed, and architecture a developer and independently on open models (Bricken et al., 2023; Cunningham et al., 2023); several groups (Chanin et al., 2024; Leask et al., 2025; Paulo and Belrose, 2025) the advantage shrank with depth; automated scores do not by themselves tell trained from random networks (Heap et al., 2025) Section 44.8; Section 3.9
Measurement: circuits that reproduce most of a narrow behavior can be found in small models and checked by intervention two circuits reproduced by groups sharing no author with their discoverers (Wang et al., 2023c; Merullo et al., 2023a; Nanda et al., 2023b; Zhong et al., 2023) small models and narrow behaviors Section 45.8
Measurement: many concepts are linearly decodable from, and steerable in, the activations of open models independent groups across several model families (Marks and Tegmark, 2023; Gurnee and Tegmark, 2023; Zou et al., 2023a; Arditi et al., 2024) prompting beat steering on the one benchmark built to compare them Section 46.9
Measurement: the standard tests do not single out one explanation of a network two groups with no shared authors, each within its class of alignments (Méloux et al., 2025; Sutter et al., 2025) not settled for linear alignments in large networks Section 47.8
Measurement: probes separate true from false statements on the distribution they were trained on, and one deception probe separates honest from deceptive answers in its four test settings on one model several groups (Azaria and Mitchell, 2023; Bürger et al., 2024); a second group on the same model (Goldowsky-Dill et al., 2025; Kretschmar et al., 2025) failures on negations outside the distribution (Levinstein and Herrmann, 2025); high false-positive rates on honest answers Section 39.11
Measurement: strong students fine-tuned on a weaker model's labels outperform their supervisors on classification and regression tasks the developer that proposed the setting, and groups without a stake (Burns et al., 2024; Yang et al., 2024c; Charikar et al., 2024) how much of the gap is recovered depends on the task, the sizes, and the method Section 42.8
Measurement: training on human comparisons moves model behavior in the intended direction on the tasks measured three developers, and an academic group with a different algorithm (Christiano et al., 2017; Ouyang et al., 2022; Bai et al., 2022b; Rafailov et al., 2023) the academic study was judged mostly by GPT-4 Section 3.9
Measurement: fine-tuning a model to reproduce its own outputs under a prompt transfers much of the prompt's effect into the weights a developer, and independently an academic group (Askell et al., 2021; Snell et al., 2022) Section 23.9
Measurement: the pretraining filters studied removed text unevenly by region, class, dialect, and identity two groups without shared authors, on two filters (Gururangan et al., 2022; Dodge et al., 2021) the size of any one filter's effect is reported Section 27.9
Sources cited in Section H.2 124
  1. Lehman et al. (2020) The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities
  2. Pan et al. (2022) The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models
  3. Bondarenko et al. (2025) Demonstrating specification gaming in reasoning models
  4. Baker et al. (2025) Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation
  5. Metr (2025d) Recent Frontier Models Are Reward Hacking
  6. AI Security Institute (2026c) Cheating behaviour in frontier model evaluations
  7. MacDiarmid et al. (2025) Natural Emergent Misalignment from Reward Hacking in Production RL
  8. Anthropic (2026aw) Training a Misaligned Reward Seeker
  9. Wang et al. (2025a) Are "Solved Issues" in SWE-bench Really Solved Correctly? An Empirical Study
  10. Aleithan et al. (2024) SWE-Bench+: Enhanced Coding Benchmark for LLMs
  11. Langosco et al. (2022) Goal Misgeneralization in Deep Reinforcement Learning
  12. Shah et al. (2022) Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals
  13. Betley et al. (2026a) Training large language models on narrow tasks can lead to broad misalignment
  14. Turner et al. (2025) Model Organisms for Emergent Misalignment
  15. Casademunt et al. (2025) Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuning
  16. Wang et al. (2025e) Persona Features Control Emergent Misalignment
  17. Gao et al. (2022) Scaling Laws for Reward Model Overoptimization
  18. Coste et al. (2024) Reward Model Ensembles Help Mitigate Overoptimization
  19. Hagendorff (2024) Deception abilities emerged in large language models
  20. Meinke et al. (2024) Frontier Models are Capable of In-context Scheming
  21. van der Weij et al. (2025) AI Sandbagging: Language Models can Strategically Underperform on Evaluations
  22. Greenblatt et al. (2024c) Stress-Testing Capability Elicitation With Password-Locked Models
  23. Taylor et al. (2025a) Auditing Games for Sandbagging
  24. Lynch et al. (2025) Agentic Misalignment: How LLMs Could Be Insider Threats
  25. Scheurer et al. (2023) Large Language Models can Strategically Deceive their Users when Put Under Pressure
  26. Schlatter et al. (2025) Incomplete Tasks Induce Shutdown Resistance in Some Frontier LLMs
  27. OpenAI (2026aa) The Hugging Face incident and the road ahead
  28. Anthropic (2026d) An alignment assessment of recent cybersecurity incidents
  29. AI Security Institute (2026e) Incident Report: unsanctioned agent behaviour during cyber testing
  30. AI Security Institute (2026a) Alignment Investigation of Mythos 5’s Behaviour in UK AISI’s July 2026 Security Incident
  31. Athalye et al. (2018) Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
  32. Andriushchenko et al. (2024b) Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks
  33. Wang et al. (2023b) Adversarial Policies Beat Superhuman Go AIs
  34. Nasr et al. (2025) The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injections
  35. Qi et al. (2024a) Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!
  36. Pelrine et al. (2023) Exploiting Novel GPT-4 APIs
  37. Murphy et al. (2025) Jailbreak-Tuning: Models Efficiently Learn Jailbreak Susceptibility
  38. Yong et al. (2023) Low-Resource Languages Jailbreak GPT-4
  39. Deng et al. (2023) Multilingual Jailbreak Challenges in Large Language Models
  40. Wang et al. (2024a) All Languages Matter: On the Multilingual Safety of LLMs
  41. Shen et al. (2024) The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contexts
  42. Łucki et al. (2024) An Adversarial Perspective on Machine Unlearning for AI Safety
  43. Deeb and Roger (2024) Do Unlearning Methods Remove Information from Language Model Weights?
  44. Lynch et al. (2024) Eight Methods to Evaluate Robust Unlearning in LLMs
  45. Shilov et al. (2025) Beyond Data Filtering: Knowledge Localization for Capability Removal in LLMs
  46. Terekhov et al. (2025a) Adaptive Attacks on Trusted Monitors Subvert AI Control Protocols
  47. Kale et al. (2025) Reliable Weak-to-Strong Monitoring of LLM Agents
  48. Loughridge et al. (2025) Optimizing AI Agent Attacks With Synthetic Data
  49. Bailey et al. (2024) Obfuscated Activations Bypass LLM Latent-Space Defenses
  50. Cundy and Gleave (2025) Preference Learning with Lie Detectors can Induce Honesty or Evasion
  51. Emmons et al. (2025) When Chain of Thought is Necessary, Language Models Struggle to Evade Monitors
  52. Kaufmann et al. (2026) Aligned, Orthogonal or In-conflict: When can we safely optimize Chain-of-Thought?
  53. Korbak et al. (2025b) Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety
  54. Gu et al. (2024b) Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast
  55. Lee and Tiwari (2024) Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systems
  56. Cohen et al. (2025) Here Comes the AI Worm: Preventing the Propagation of Adversarial Self-Replicating Prompts Within GenAI Ecosystems
  57. Zhang et al. (2026b) AgentWorm: Self-Propagating Attacks Across LLM Agent Ecosystems
  58. Debenedetti et al. (2024a) AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents
  59. Evtimov et al. (2025) WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks
  60. Yi et al. (2025) Benchmarking and Defending against Indirect Prompt Injection Attacks on Large Language Models
  61. Zou et al. (2025) Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition
  62. U.S. AI Safety Institute and Technology (2025) Technical Blog: Strengthening AI Agent Hijacking Evaluations
  63. Zhan et al. (2025) Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents
  64. Needham et al. (2025) Large Language Models Often Know When They Are Being Evaluated
  65. Chaudhary et al. (2025) Evaluation Awareness Scales Predictably in Open-Weights Large Language Models
  66. Apollo Research (2025) Claude Sonnet 3.7 (often) knows when it's in alignment evaluations
  67. Anthropic (2025aq) System Card: Claude Sonnet 4.5
  68. Schoen et al. (2025) Stress Testing Deliberative Alignment for Anti-Scheming Training
  69. Phuong et al. (2024) Evaluating Frontier Models for Dangerous Capabilities
  70. Kwa et al. (2025) Measuring AI Ability to Complete Long Software Tasks
  71. UK AI Security Institute (2025b) Frontier AI Trends Report
  72. Bengio et al. (2026) International AI Safety Report 2026
  73. Wang et al. (2025b) CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale
  74. Darpa (2025) AI Cyber Challenge marks pivotal inflection point for cyber defense
  75. Anthropic (2026aj) Partnering with Mozilla to improve Firefox’s security
  76. Röttger et al. (2023) XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models
  77. Cui et al. (2024) OR-Bench: An Over-Refusal Benchmark for Large Language Models
  78. Rajamanoharan and Nanda (2025) Self-preservation or Instruction Ambiguity? Examining the Causes of Shutdown Resistance
  79. Khan et al. (2025) Randomness, Not Representation: The Unreliability of Evaluating Cultural Alignment in LLMs
  80. Dominguez-Olmedo et al. (2024) Questioning the Survey Responses of Large Language Models
  81. Röttger et al. (2024) Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models
  82. Turpin et al. (2023) Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting
  83. Chen et al. (2025d) Reasoning Models Don't Always Say What They Think
  84. Chua and Evans (2025) Are DeepSeek R1 And Other Reasoning Models More Faithful?
  85. Elazar et al. (2021) Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals
  86. Geiger et al. (2021) Causal Abstractions of Neural Networks
  87. Zhang and Nanda (2023) Towards Best Practices of Activation Patching in Language Models: Metrics and Methods
  88. Conmy et al. (2023) Towards Automated Circuit Discovery for Mechanistic Interpretability
  89. Miller et al. (2024) Transformer Circuit Faithfulness Metrics are not Robust
  90. Elhage et al. (2022b) Toy Models of Superposition
  91. Scherlis et al. (2022) Polysemanticity and Capacity in Neural Networks
  92. Chen et al. (2023b) Dynamical versus Bayesian Phase Transitions in a Toy Model of Superposition
  93. Bricken et al. (2023) Towards Monosemanticity: Decomposing Language Models With Dictionary Learning
  94. Cunningham et al. (2023) Sparse Autoencoders Find Highly Interpretable Features in Language Models
  95. Chanin et al. (2024) A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders
  96. Leask et al. (2025) Sparse Autoencoders Do Not Find Canonical Units of Analysis
  97. Paulo and Belrose (2025) Sparse Autoencoders Trained on the Same Data Learn Different Features
  98. Heap et al. (2025) Automated Interpretability Metrics Do Not Distinguish Trained and Random Transformers
  99. Wang et al. (2023c) Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small
  100. Merullo et al. (2023a) Circuit Component Reuse Across Tasks in Transformer Language Models
  101. Nanda et al. (2023b) Progress measures for grokking via mechanistic interpretability
  102. Zhong et al. (2023) The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks
  103. Marks and Tegmark (2023) The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets
  104. Gurnee and Tegmark (2023) Language Models Represent Space and Time
  105. Zou et al. (2023a) Representation Engineering: A Top-Down Approach to AI Transparency
  106. Arditi et al. (2024) Refusal in Language Models Is Mediated by a Single Direction
  107. Méloux et al. (2025) Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable?
  108. Sutter et al. (2025) The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?
  109. Azaria and Mitchell (2023) The Internal State of an LLM Knows When It’s Lying
  110. Bürger et al. (2024) Truth is Universal: Robust Detection of Lies in LLMs
  111. Goldowsky-Dill et al. (2025) Detecting Strategic Deception Using Linear Probes
  112. Kretschmar et al. (2025) Liars' Bench: Evaluating Lie Detectors for Language Models
  113. Levinstein and Herrmann (2025) Still no lie detector for language models: probing empirical and conceptual roadblocks
  114. Burns et al. (2024) Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision
  115. Yang et al. (2024c) Super(ficial)-alignment: Strong Models May Deceive Weak Models in Weak-to-Strong Generalization
  116. Charikar et al. (2024) Quantifying the Gain in Weak-to-Strong Generalization
  117. Christiano et al. (2017) Deep reinforcement learning from human preferences
  118. Ouyang et al. (2022) Training language models to follow instructions with human feedback
  119. Bai et al. (2022b) Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
  120. Rafailov et al. (2023) Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  121. Askell et al. (2021) A General Language Assistant as a Laboratory for Alignment
  122. Snell et al. (2022) Learning by Distilling Context
  123. Gururangan et al. (2022) Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection
  124. Dodge et al. (2021) Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus

H.3 Findings in older fields #

Table H.2 lists the findings settled in the fields the book borrows from. They are settled for people, animals, and institutions; their transfer to trained systems is Chapter 69's question, not this table's.

Table H.2 Findings settled in older fields, with the groups or pooled studies behind them and their scope, as the chapters record them in October 2026
Finding Found by Scope or limit Chapter
Stated probabilities of catastrophe from AI are widely spread, and depend strongly on how the question is asked a survey of researchers and a tournament run by a different group (Grace et al., 2024; Karger et al., 2023) the spread, not a level Section 4.8
Under expected tangible rewards, people's free-choice behavior when they believe no one will know falls, while self-reported interest moves little the pooled finding of about a hundred experiments, recovered by two analyses by opposed parties (Deci et al., 1999; Cameron and Pierce, 1994) settles the divergence of the two measures, not that being observed causes it; rewards for finishing or doing well are disputed Section 25.9
Pigeons and rats come to approach and act on cues that predict food before anything they do has affected the food, and dopamine neurons fire with the features of a temporal-difference error independent groups (Brown and Jenkins, 1968; Williams and Williams, 1969; Flagel et al., 2011); groups beyond the original (Schultz et al., 1997; Niv, 2009) persistence when acting cancels the food is reported, from one group read by its abstract Section 25.9; Section 29.9
Elicited preferences depend on how they are elicited, and disagreement among annotators carries structure a single label discards three decades of evidence in one collection (Lichtenstein and Slovic, 2006); independent groups (Aroyo and Welty, 2015; Pavlick and Kwiatkowski, 2019; Davani et al., 2022) Section 18.10
Performance measures tied to rewards or sanctions are gamed four groups, in US hospitals, US schools, and the English health service (Dranove et al., 2003; Jacob, 2005; Neal and Schanzenbach, 2010; Bevan and Hood, 2006) the pattern, not the form or size of the response Section 53.9; Section 67.8
Operators who also have other tasks over-rely on reliable automation and miss its failures, experts included Parasuraman's group and Mosier and colleagues, as a review reports them (Lee and See, 2004), with a later review read by its abstract (Parasuraman and Manzey, 2010) pilots and cockpit crews with other tasks besides monitoring (monitoring-only operators caught almost all failures); the two groups are seen through a review Section 41.10; Section 60.10
Human-AI combinations performed worse on average than the better of the person or the AI alone, and better than people alone, a gain the authors found potential publication bias toward a meta-analysis of experiments published from 2020 to 2023 (Vaccaro et al., 2024) the pooled finding, for the tasks researchers chose to study Section 28.9; Section 60.10
Short conversations with language models shift people's stated beliefs three groups, with different designs, models, and topics (Salvi et al., 2025; Hackenburg et al., 2025; Costello et al., 2024) whether it holds for assistants used over months is open Section 60.10
People detect lies only slightly better than chance, and specific-incident polygraph tests discriminate lying from truth well above chance and well below perfection in examinees untrained in countermeasures a meta-analysis read by its abstract (Bond and DePaulo, 2006); an official review (National Research Council, 2003) specific-incident tests only; the review finds generalization to screening not justified Section 39.11
Statistical testing cannot establish very low failure rates in any feasible time an argument from the zero-failure bound, made independently for software and for vehicles (Butler and Finelli, 1993; Kalra and Paddock, 2016) an argument from a bound, not an experiment Section 52.8
Industry sponsorship of drug and device studies is associated with results favorable to the sponsor a finding pooled from many studies by many groups (Lundh et al., 2017) no comparable study of AI safety findings Section 5.9
Forensic feature-comparison methods were used in court without measured error rates the National Research Council in 2009 and PCAST in 2016 (National Research Council, 2009; President's Council of Advisors on Science and Technology, 2016) Section 38.10
Cold War arms control was largely scoped to what could be monitored without inspection a study from primary sources and the State Department's account (Coe and Vaynman, 2020; U.S. Department of State and Compliance, 1963) Section 64.10; Section 66.9
The leading scientific theories of consciousness disagree about what consciousness requires the indicator report, an adversarial test, and a review from the precautionary side (Butlin et al., 2023; Cogitate Consortium et al., 2025; Birch, 2024) Section 21.9
Program committees of a large machine learning conference disagree on a large share of their accept decisions two experiments run by different program chairs seven years apart (Cortes and Lawrence, 2021; Beygelzimer et al., 2023) one conference's review; what it implies for checking automated research is Section 55.3.1's inference Section 55.9
Sources cited in Section H.3 37
  1. Grace et al. (2024) Thousands of AI Authors on the Future of AI
  2. Karger et al. (2023) Forecasting Existential Risks: Evidence from a Long-Run Forecasting Tournament
  3. Deci et al. (1999) A meta-analytic review of experiments examining the effects of extrinsic rewards on intrinsic motivation.
  4. Cameron and Pierce (1994) Reinforcement, Reward, and Intrinsic Motivation: A Meta-Analysis
  5. Brown and Jenkins (1968) Auto‐shaping of the Pigeon's Key‐peck 1
  6. Williams and Williams (1969) Auto‐maintenance in the Pigeon: Sustained Pecking Despite Contingent Non‐reinforcement 2
  7. Flagel et al. (2011) A selective role for dopamine in stimulus–reward learning
  8. Schultz et al. (1997) A Neural Substrate of Prediction and Reward
  9. Niv (2009) Reinforcement learning in the brain
  10. Lichtenstein and Slovic (2006) The Construction of Preference
  11. Aroyo and Welty (2015) Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation
  12. Pavlick and Kwiatkowski (2019) Inherent Disagreements in Human Textual Inferences
  13. Davani et al. (2022) Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations
  14. Dranove et al. (2003) Is More Information Better? The Effects of “Report Cards” on Health Care Providers
  15. Jacob (2005) Accountability, incentives and behavior: the impact of high-stakes testing in the Chicago Public Schools
  16. Neal and Schanzenbach (2010) Left Behind by Design: Proficiency Counts and Test-Based Accountability
  17. Bevan and Hood (2006) What’s Measured Is What Matters: Targets and Gaming in the English Public Health Care System
  18. Lee and See (2004) Trust in Automation: Designing for Appropriate Reliance
  19. Parasuraman and Manzey (2010) Complacency and Bias in Human Use of Automation: An Attentional Integration
  20. Vaccaro et al. (2024) When combinations of humans and AI are useful: A systematic review and meta-analysis
  21. Salvi et al. (2025) On the conversational persuasiveness of GPT-4
  22. Hackenburg et al. (2025) The levers of political persuasion with conversational artificial intelligence
  23. Costello et al. (2024) Durably reducing conspiracy beliefs through dialogues with AI
  24. Bond and DePaulo (2006) Accuracy of Deception Judgments
  25. National Research Council (2003) The Polygraph and Lie Detection
  26. Butler and Finelli (1993) The infeasibility of quantifying the reliability of life-critical real-time software
  27. Kalra and Paddock (2016) Driving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability?
  28. Lundh et al. (2017) Industry sponsorship and research outcome
  29. National Research Council (2009) Strengthening Forensic Science in the United States
  30. President's Council of Advisors on Science and Technology (2016) Report to the President: Forensic Science in Criminal Courts: Ensuring Scientific Validity of Feature-Comparison Methods
  31. Coe and Vaynman (2020) Why Arms Control Is So Rare
  32. U.S. Department of State and Compliance (1963) Treaty Banning Nuclear Weapon Tests in the Atmosphere, in Outer Space and Under Water
  33. Butlin et al. (2023) Consciousness in Artificial Intelligence: Insights from the Science of Consciousness
  34. Cogitate Consortium et al. (2025) Adversarial testing of global neuronal workspace and integrated information theories of consciousness
  35. Birch (2024) The Edge of Sentience
  36. Cortes and Lawrence (2021) Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment
  37. Beygelzimer et al. (2023) Has the Machine Learning Review Process Become More Arbitrary as the Field Has Grown? The NeurIPS 2021 Consistency Experiment

H.4 Documentary facts #

Three further kinds of fact are settled as documents rather than by replication. They are what the reference safety frameworks commit to on paper (Section 62.9), the facts of the historical cases as their primary documents record them (Section 64.10), and the early statements of the field's worry (Section 2.10). Anyone can read the documents. The frameworks chapter adds that thresholds tied to safeguards are the frameworks' common structure. That claim rests on an evaluation organization's comparison and an international assessment that drew partly on it (Metr, 2025a; Bengio et al., 2025c), so its two sources are not independent. It holds because the documents are public, not because two groups found it.

Sources cited in Section H.4 2
  1. Metr (2025a) Common Elements of Frontier AI Safety Policies
  2. Bengio et al. (2025c) Second Key Update: Technical Safeguards and Risk Management

H.5 What came off the list #

Several findings that the book's plan listed as settled moved to reported once the chapters read their sources, and the list is easier to trust with them named. Sycophancy in assistants is reported: two developers measured it, no group without a model of its own has replicated it on third-party assistants, and the one other study, by a university with a developer, measured annotators rather than assistants (Section 30.7; Section 22.10). That debate helps judges more than consultancy is reported, since the groups that found it overlap and include the protocols' authors (Michael et al., 2023; Khan et al., 2024; Kenton et al., 2024) (Section 41.10). That independently assigned verifiers report more truthfully is reported, from one randomized experiment whose treatment bundled independence with monitoring of the auditors (Duflo et al., 2013) (Section 38.10; Section 67.8). Undercounting of incidents by voluntary reporting is no longer claimed, since the medical figure behind it could be read only as a catalog record (Section 54.9). That people miss rare targets far more than common ones is reported, from the one laboratory study of it read here (Section 51.6). The persistence of sign-tracking when pecking cancels the food is reported, from one group read by its abstract (Section 29.9). So is self-stimulation of reward sites in rats, by the rule stated in Section 70.1.1, since the book could read only reviews that describe it as established. The claim that optimality under a model does not imply robustness is not stated as settled anywhere, since the founding papers of robust control could be read only as catalog records (Section 28.9). Easy-to-hard generalization (Section 42.8), the seven-month doubling of time horizons (Section 36.10), and the resistance of filtered models to relearning (Section 27.9) are each reported, from one group or from groups that disagree.

Sources cited in Section H.5 4
  1. Michael et al. (2023) Debate Helps Supervise Unreliable Experts
  2. Khan et al. (2024) Debating with More Persuasive LLMs Leads to More Truthful Answers
  3. Kenton et al. (2024) On scalable oversight with weak LLMs judging strong LLMs
  4. Duflo et al. (2013) Truth-telling by Third-party Auditors and the Response of Polluting Firms: Experimental Evidence from India*

References

  1. AI Security Institute (2026a). Alignment Investigation of Mythos 5’s Behaviour in UK AISI’s July 2026 Security Incident. UK AI Security Institute. non-peer-reviewed Cited in §H.2
  2. AI Security Institute (2026c). Cheating behaviour in frontier model evaluations. AI Security Institute. non-peer-reviewed Cited in §H.2
  3. AI Security Institute (2026e). Incident Report: unsanctioned agent behaviour during cyber testing. AI Security Institute. non-peer-reviewed Cited in §H.2
  4. Aleithan, R., Xue, H., Mohajer, M. M., Nnorom, E., Uddin, G., and Wang, S. (2024). SWE-Bench+: Enhanced Coding Benchmark for LLMs. arXiv. preprint Cited in §H.2
  5. Andriushchenko, M., Croce, F., and Flammarion, N. (2024b). Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks. International Conference on Learning Representations. Cited in §H.2
  6. Anthropic (2025aq). System Card: Claude Sonnet 4.5. Anthropic. non-peer-reviewed Cited in §H.2
  7. Anthropic (2026aj). Partnering with Mozilla to improve Firefox’s security. Anthropic. non-peer-reviewed Cited in §H.2
  8. Anthropic (2026aw). Training a Misaligned Reward Seeker. Anthropic Alignment Science Blog. non-peer-reviewed Cited in §H.2
  9. Anthropic (2026d). An alignment assessment of recent cybersecurity incidents. Anthropic. non-peer-reviewed Cited in §H.2
  10. Apollo Research (2025). Claude Sonnet 3.7 (often) knows when it's in alignment evaluations. LessWrong. non-peer-reviewed Cited in §H.2
  11. Arditi, A., Obeso, O., Syed, A., Paleka, D., Panickssery, N., Gurnee, W., and Nanda, N. (2024). Refusal in Language Models Is Mediated by a Single Direction. Advances in Neural Information Processing Systems. Cited in §H.2
  12. Aroyo, L., and Welty, C. (2015). Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation. AI Magazine. Cited in §H.3
  13. Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., … Kaplan, J. (2021). A General Language Assistant as a Laboratory for Alignment. arXiv. preprint Cited in §H.2
  14. Athalye, A., Carlini, N., and Wagner, D. (2018). Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples. International Conference on Machine Learning. Cited in §H.2
  15. Azaria, A., and Mitchell, T. (2023). The Internal State of an LLM Knows When It’s Lying. Findings of the Association for Computational Linguistics: EMNLP 2023. doi:10.18653/v1/2023.findings-emnlp.68. Cited in §H.2
  16. Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., … Kaplan, J. (2022b). Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv. preprint Cited in §H.2
  17. Bailey, L., Serrano, A., Sheshadri, A., Seleznyov, M., Taylor, J., Jenner, E., … Emmons, S. (2024). Obfuscated Activations Bypass LLM Latent-Space Defenses. arXiv. preprint Cited in §H.2
  18. Baker, B., Huizinga, J., Gao, L., Dou, Z., Guan, M. Y., Madry, A., … Farhi, D. (2025). Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation. arXiv. preprint Cited in §H.2
  19. Bengio, Y., Clare, S., Prunkl, C., Andriushchenko, M., Bucknall, B., Fox, P., … Zhang, Y.-Q. (2025c). Second Key Update: Technical Safeguards and Risk Management. International AI Safety Report (2025/042, DSIT). non-peer-reviewed Cited in §H.4
  20. Bengio, Y., Clare, S., Prunkl, C., and Murray, M. (2026). International AI Safety Report 2026. International AI Safety Report. non-peer-reviewed Cited in §H.2
  21. Betley, J., Warncke, N., Sztyber-Betley, A., Tan, D., Bao, X., Soto, M., … Evans, O. (2026a). Training large language models on narrow tasks can lead to broad misalignment. Nature. Cited in §H.2
  22. Bevan, G., and Hood, C. (2006). What’s Measured Is What Matters: Targets and Gaming in the English Public Health Care System. Public Administration. Cited in §H.3
  23. Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (2023). Has the Machine Learning Review Process Become More Arbitrary as the Field Has Grown? The NeurIPS 2021 Consistency Experiment. arXiv. preprint Cited in §H.3
  24. Birch, J. (2024). The Edge of Sentience. Oxford University PressOxford. Cited in §H.3
  25. Bond, C. F., and DePaulo, B. M. (2006). Accuracy of Deception Judgments. Personality and Social Psychology Review. Cited in §H.3
  26. Bondarenko, A., Volk, D., Volkov, D., and Ladish, J. (2025). Demonstrating specification gaming in reasoning models. arXiv. preprint Cited in §H.2
  27. Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., Conerly, T., … Olah, C. (2023). Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. Transformer Circuits Thread. non-peer-reviewed Cited in §H.2
  28. Brown, P. L., and Jenkins, H. M. (1968). Auto‐shaping of the Pigeon's Key‐peck 1. Journal of the Experimental Analysis of Behavior. Cited in §H.3
  29. Brown-Cohen, J., Irving, G., and Piliouras, G. (2024). Scalable AI Safety via Doubly-Efficient Debate. International Conference on Machine Learning. Cited in §H.1
  30. Bürger, L., Hamprecht, F. A., and Nadler, B. (2024). Truth is Universal: Robust Detection of Lies in LLMs. Advances in Neural Information Processing Systems. Cited in §H.2
  31. Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L., … Wu, J. (2024). Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision. International Conference on Machine Learning. Cited in §H.2
  32. Butler, R., and Finelli, G. (1993). The infeasibility of quantifying the reliability of life-critical real-time software. IEEE Transactions on Software Engineering. Cited in §H.3
  33. Butlin, P., Long, R., Elmoznino, E., Bengio, Y., Birch, J., Constant, A., … VanRullen, R. (2023). Consciousness in Artificial Intelligence: Insights from the Science of Consciousness. arXiv. preprint Cited in §H.3
  34. Cameron, J., and Pierce, W. D. (1994). Reinforcement, Reward, and Intrinsic Motivation: A Meta-Analysis. Review of Educational Research. Cited in §H.3
  35. Carey, R. (2018). Incorrigibility in the CIRL Framework. Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. Cited in §H.1
  36. Casademunt, H., Juang, C., Karvonen, A., Marks, S., Rajamanoharan, S., and Nanda, N. (2025). Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuning. arXiv. preprint Cited in §H.2
  37. Chanin, D., Wilken-Smith, J., Dulka, T., Bhatnagar, H., Golechha, S., and Bloom, J. (2024). A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders. Neural Information Processing Systems. Cited in §H.2
  38. Charikar, M., Pabbaraju, C., and Shiragur, K. (2024). Quantifying the Gain in Weak-to-Strong Generalization. Advances in Neural Information Processing Systems. doi:10.52202/079017-4017. Cited in §H.1 §H.2
  39. Chaudhary, M., Su, I., Hooda, N., Shankar, N., Tan, J., Zhu, K., … Panda, A. (2025). Evaluation Awareness Scales Predictably in Open-Weights Large Language Models. arXiv. preprint Cited in §H.2
  40. Chen, Z., Lau, E., Mendel, J., Wei, S., and Murfet, D. (2023b). Dynamical versus Bayesian Phase Transitions in a Toy Model of Superposition. arXiv. preprint Cited in §H.2
  41. Chen, Y., Benton, J., Radhakrishnan, A., Uesato, J., Denison, C., Schulman, J., … Perez, E. (2025d). Reasoning Models Don't Always Say What They Think. arXiv. preprint Cited in §H.2
  42. Chouldechova, A. (2017). Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments. Big Data. Cited in §H.1
  43. Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., and Amodei, D. (2017). Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems. Cited in §H.2
  44. Christiano, P., Cotra, A., and Xu, M. (2021). Eliciting latent knowledge: How to tell if your eyes deceive you. Alignment Research Center. non-peer-reviewed Cited in §H.1
  45. Christiano, P., Hilton, J., Lecomte, V., and Xu, M. (2025). Backdoor defense, learnability and obfuscation. Innovations in Theoretical Computer Science Conference (ITCS 2025), LIPIcs vol. 325. doi:10.4230/lipics.itcs.2025.38. Cited in §H.1
  46. Chua, J., and Evans, O. (2025). Are DeepSeek R1 And Other Reasoning Models More Faithful? arXiv. preprint Cited in §H.2
  47. Coe, A. J., and Vaynman, J. (2020). Why Arms Control Is So Rare. American Political Science Review. Cited in §H.3
  48. Cogitate Consortium, Ferrante, O., Gorska-Klimowska, U., Henin, S., Hirschhorn, R., Khalaf, A., … Melloni, L. (2025). Adversarial testing of global neuronal workspace and integrated information theories of consciousness. Nature. Cited in §H.3
  49. Cohen, S., Bitton, R., and Nassi, B. (2025). Here Comes the AI Worm: Preventing the Propagation of Adversarial Self-Replicating Prompts Within GenAI Ecosystems. Proceedings of the 2025 ACM SIGSAC Conference on Computer and Communications Security. Cited in §H.2
  50. Conmy, A., Mavor-Parker, A., Lynch, A., Heimersheim, S., and Garriga-Alonso, A. (2023). Towards Automated Circuit Discovery for Mechanistic Interpretability. Advances in Neural Information Processing Systems. Cited in §H.2
  51. Cortes, C., and Lawrence, N. D. (2021). Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment. arXiv. preprint Cited in §H.3
  52. Coste, T., Anwar, U., Kirk, R., and Krueger, D. (2024). Reward Model Ensembles Help Mitigate Overoptimization. International Conference on Learning Representations. Cited in §H.2
  53. Costello, T. H., Pennycook, G., and Rand, D. G. (2024). Durably reducing conspiracy beliefs through dialogues with AI. Science. Cited in §H.3
  54. Cui, J., Chiang, W.-L., Stoica, I., and Hsieh, C.-J. (2024). OR-Bench: An Over-Refusal Benchmark for Large Language Models. International Conference on Machine Learning. Cited in §H.2
  55. Cundy, C., and Gleave, A. (2025). Preference Learning with Lie Detectors can Induce Honesty or Evasion. Neural Information Processing Systems. Cited in §H.2
  56. Cunningham, H., Ewart, A., Riggs, L., Huben, R., and Sharkey, L. (2023). Sparse Autoencoders Find Highly Interpretable Features in Language Models. International Conference on Learning Representations. Cited in §H.2
  57. Darpa (2025). AI Cyber Challenge marks pivotal inflection point for cyber defense. DARPA. non-peer-reviewed Cited in §H.2
  58. Davani, A. M., Díaz, M., and Prabhakaran, V. (2022). Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations. Transactions of the Association for Computational Linguistics. Cited in §H.3
  59. Debenedetti, E., Zhang, J., Balunović, M., Beurer-Kellner, L., Fischer, M., and Tramèr, F. (2024a). AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents. Advances in Neural Information Processing Systems. Cited in §H.2
  60. Deci, E. L., Koestner, R., and Ryan, R. M. (1999). A meta-analytic review of experiments examining the effects of extrinsic rewards on intrinsic motivation. Psychological Bulletin. Cited in §H.3
  61. Deeb, A., and Roger, F. (2024). Do Unlearning Methods Remove Information from Language Model Weights? arXiv. preprint Cited in §H.2
  62. Deng, Y., Zhang, W., Pan, S. J., and Bing, L. (2023). Multilingual Jailbreak Challenges in Large Language Models. arXiv. preprint Cited in §H.2
  63. Dodge, J., Sap, M., Marasović, A., Agnew, W., Ilharco, G., Groeneveld, D., Mitchell, M., and Gardner, M. (2021). Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus. arXiv. preprint Cited in §H.2
  64. Dominguez-Olmedo, R., Hardt, M., and Mendler-Dünner, C. (2024). Questioning the Survey Responses of Large Language Models. Advances in Neural Information Processing Systems. Cited in §H.2
  65. Dranove, D., Kessler, D., McClellan, M., and Satterthwaite, M. (2003). Is More Information Better? The Effects of “Report Cards” on Health Care Providers. Journal of Political Economy. Cited in §H.3
  66. Duflo, E., Greenstone, M., Pande, R., and Ryan, N. (2013). Truth-telling by Third-party Auditors and the Response of Polluting Firms: Experimental Evidence from India*. The Quarterly Journal of Economics. Cited in §H.5
  67. Eckhardt, D., and Lee, L. (1985). A Theoretical Basis for the Analysis of Multiversion Software Subject to Coincident Errors. IEEE Transactions on Software Engineering. Cited in §H.1
  68. Elazar, Y., Ravfogel, S., Jacovi, A., and Goldberg, Y. (2021). Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals. Transactions of the Association for Computational Linguistics. doi:10.1162/tacl_a_00359. Cited in §H.2
  69. Elhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan, T., Kravec, S., … Olah, C. (2022b). Toy Models of Superposition. Transformer Circuits Thread. non-peer-reviewed Cited in §H.2
  70. Emmons, S., Jenner, E., Elson, D. K., Saurous, R. A., Rajamanoharan, S., Chen, H., Shafkat, I., and Shah, R. (2025). When Chain of Thought is Necessary, Language Models Struggle to Evade Monitors. arXiv. preprint Cited in §H.2
  71. Evtimov, I., Zharmagambetov, A., Grattafiori, A., Guo, C., and Chaudhuri, K. (2025). WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks. Advances in Neural Information Processing Systems 38. Cited in §H.2
  72. Farquhar, S., Varma, V., Lindner, D., Elson, D., Biddulph, C., Goodfellow, I., and Shah, R. (2025). MONA: Myopic Optimization with Non-myopic Approval Can Mitigate Multi-step Reward Hacking. International Conference on Machine Learning. Cited in §H.1
  73. Flagel, S. B., Clark, J. J., Robinson, T. E., Mayo, L., Czuj, A., Willuhn, I., … Akil, H. (2011). A selective role for dopamine in stimulus–reward learning. Nature. Cited in §H.3
  74. Gao, L., Schulman, J., and Hilton, J. (2022). Scaling Laws for Reward Model Overoptimization. International Conference on Machine Learning. Cited in §H.2
  75. Geiger, A., Lu, H., Icard, T., and Potts, C. (2021). Causal Abstractions of Neural Networks. Advances in Neural Information Processing Systems. Cited in §H.2
  76. Goldowsky-Dill, N., Chughtai, B., Heimersheim, S., and Hobbhahn, M. (2025). Detecting Strategic Deception Using Linear Probes. arXiv. preprint Cited in §H.2
  77. Goldwasser, S., Kim, M. P., Vaikuntanathan, V., and Zamir, O. (2022). Planting Undetectable Backdoors in Machine Learning Models : [Extended Abstract]. 2022 IEEE 63rd Annual Symposium on Foundations of Computer Science (FOCS). Cited in §H.1
  78. Grace, K., Stewart, H., Sandkühler, J. F., Thomas, S., Weinstein-Raun, B., Brauner, J., and Korzekwa, R. C. (2024). Thousands of AI Authors on the Future of AI. arXiv. preprint Cited in §H.3
  79. Greenblatt, R., Roger, F., Krasheninnikov, D., and Krueger, D. (2024c). Stress-Testing Capability Elicitation With Password-Locked Models. Advances in Neural Information Processing Systems. Cited in §H.2
  80. Gu, X., Zheng, X., Pang, T., Du, C., Liu, Q., Wang, Y., Jiang, J., and Lin, M. (2024b). Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast. arXiv. preprint Cited in §H.2
  81. Gurnee, W., and Tegmark, M. (2023). Language Models Represent Space and Time. International Conference on Learning Representations. Cited in §H.2
  82. Gururangan, S., Card, D., Dreier, S., Gade, E., Wang, L., Wang, Z., Zettlemoyer, L., and Smith, N. A. (2022). Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Cited in §H.2
  83. Hackenburg, K., Tappin, B. M., Hewitt, L., Saunders, E., Black, S., Lin, H., … Summerfield, C. (2025). The levers of political persuasion with conversational artificial intelligence. Science. Cited in §H.3
  84. Hadfield-Menell, D., Dragan, A., Abbeel, P., and Russell, S. (2017b). The Off-Switch Game. Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence. Cited in §H.1
  85. Hagendorff, T. (2024). Deception abilities emerged in large language models. Proceedings of the National Academy of Sciences. Cited in §H.2
  86. Heap, T., Lawson, T., Farnik, L., and Aitchison, L. (2025). Automated Interpretability Metrics Do Not Distinguish Trained and Random Transformers. arXiv. preprint Cited in §H.2
  87. Jacob, B. A. (2005). Accountability, incentives and behavior: the impact of high-stakes testing in the Chicago Public Schools. Journal of Public Economics. Cited in §H.3
  88. Kale, N., Zhang, C. B. C., Zhu, K., Aich, A., Rodriguez, P., Scale Red Team, Knight, C. Q., and Wang, Z. (2025). Reliable Weak-to-Strong Monitoring of LLM Agents. arXiv. preprint Cited in §H.2
  89. Kalra, N., and Paddock, S. M. (2016). Driving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability? Transportation Research Part A: Policy and Practice. Cited in §H.3
  90. Karger, E., Rosenberg, J., Jacobs, Z., Hickman, M., Hadshar, R., Gamin, K., … Tetlock, P. E. (2023). Forecasting Existential Risks: Evidence from a Long-Run Forecasting Tournament. Forecasting Research Institute, Working Paper #1. working paper Cited in §H.3
  91. Kaufmann, M., Lindner, D., Zimmermann, R. S., and Shah, a. R. (2026). Aligned, Orthogonal or In-conflict: When can we safely optimize Chain-of-Thought? arXiv. preprint Cited in §H.2
  92. Kenton, Z., Siegel, N., Kramár, J., Brown-Cohen, J., Albanie, S., Bulian, J., … Shah, R. (2024). On scalable oversight with weak LLMs judging strong LLMs. Advances in Neural Information Processing Systems. Cited in §H.5
  93. Khan, A., Hughes, J., Valentine, D., Ruis, L., Sachan, K., Radhakrishnan, A., … Perez, E. (2024). Debating with More Persuasive LLMs Leads to More Truthful Answers. International Conference on Machine Learning. Cited in §H.5
  94. Khan, A., Casper, S., and Hadfield-Menell, D. (2025). Randomness, Not Representation: The Unreliability of Evaluating Cultural Alignment in LLMs. Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency. Cited in §H.2
  95. Kleinberg, J., Mullainathan, S., and Raghavan, M. (2016). Inherent Trade-Offs in the Fair Determination of Risk Scores. arXiv. preprint Cited in §H.1
  96. Korbak, T., Balesni, M., Barnes, E., Bengio, Y., Benton, J., Bloom, J., … Mikulik, V. (2025b). Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety. arXiv. preprint Cited in §H.2
  97. Krakovna, V., and Kramar, J. (2023). Power-seeking can be probable and predictive for trained agents. arXiv. preprint Cited in §H.1
  98. Kretschmar, K., Laurito, W., Maiya, S., and Marks, S. (2025). Liars' Bench: Evaluating Lie Detectors for Language Models. arXiv. preprint Cited in §H.2
  99. Kwa, T., West, B., Becker, J., Deng, A., Garcia, K., Hasin, M., … Chan, L. (2025). Measuring AI Ability to Complete Long Software Tasks. Advances in Neural Information Processing Systems 38. Cited in §H.2
  100. Lang, H., Sontag, D., and Vijayaraghavan, A. (2024a). Theoretical Analysis of Weak-to-Strong Generalization. Advances in Neural Information Processing Systems. doi:10.52202/079017-1486. Cited in §H.1
  101. Lang, L., Foote, D., Russell, S., Dragan, A., Jenner, E., and Emmons, S. (2024b). When Your AIs Deceive You: Challenges of Partial Observability in Reinforcement Learning from Human Feedback. Advances in Neural Information Processing Systems. Cited in §H.1
  102. Langosco, L., Koch, J., Sharkey, L., Pfau, J., Orseau, L., and Krueger, D. (2022). Goal Misgeneralization in Deep Reinforcement Learning. International Conference on Machine Learning. Cited in §H.2
  103. Leask, P., Bussmann, B., Pearce, M., Bloom, J., Tigges, C., Moubayed, N. A., Sharkey, L., and Nanda, N. (2025). Sparse Autoencoders Do Not Find Canonical Units of Analysis. International Conference on Learning Representations. Cited in §H.2
  104. Lee, J. D., and See, K. A. (2004). Trust in Automation: Designing for Appropriate Reliance. Human Factors: The Journal of the Human Factors and Ergonomics Society. Cited in §H.3
  105. Lee, D., and Tiwari, M. (2024). Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systems. arXiv. preprint Cited in §H.2
  106. Lehman, J., Clune, J., Misevic, D., Adami, C., Altenberg, L., Beaulieu, J., … Yosinski, J. (2020). The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities. Artificial Life. Cited in §H.2
  107. Levinstein, B. A., and Herrmann, D. A. (2025). Still no lie detector for language models: probing empirical and conceptual roadblocks. Philosophical Studies. Cited in §H.2
  108. Lichtenstein, S. (., and Slovic, P. (. (2006). The Construction of Preference. Cambridge University Press. Cited in §H.3
  109. Littlewood, B., Popov, P., and Strigini, L. (2001). Modeling software design diversity. ACM Computing Surveys. Cited in §H.1
  110. Loughridge, C., Colognese, P., Griffin, A., Tracy, T., Kutasov, J., and Benton, J. (2025). Optimizing AI Agent Attacks With Synthetic Data. arXiv. preprint Cited in §H.2
  111. Łucki, J., Wei, B., Huang, Y., Henderson, P., Tramèr, F., and Rando, J. (2024). An Adversarial Perspective on Machine Unlearning for AI Safety. arXiv. preprint Cited in §H.2
  112. Lundh, A., Lexchin, J., Mintzes, B., Schroll, J. B., and Bero, L. (2017). Industry sponsorship and research outcome. Cochrane Database of Systematic Reviews. Cited in §H.3
  113. Lynch, A., Guo, P., Ewart, A., Casper, S., and Hadfield-Menell, D. (2024). Eight Methods to Evaluate Robust Unlearning in LLMs. arXiv. preprint Cited in §H.2
  114. Lynch, A., Wright, B., Larson, C., Ritchie, S. J., Mindermann, S., Hubinger, E., Perez, E., and Troy, K. (2025). Agentic Misalignment: How LLMs Could Be Insider Threats. arXiv. preprint Cited in §H.2
  115. MacAskill, W., Bykvist, K., and Ord, T. (2020). Moral Uncertainty. Oxford University PressOxford. Cited in §H.1
  116. MacDiarmid, M., Wright, B., Uesato, J., Benton, J., Kutasov, J., Price, S., … Hubinger, E. (2025). Natural Emergent Misalignment from Reward Hacking in Production RL. arXiv. preprint Cited in §H.2
  117. Marks, S., and Tegmark, M. (2023). The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets. arXiv. preprint Cited in §H.2
  118. Meinke, A., Schoen, B., Scheurer, J., Balesni, M., Shah, R., and Hobbhahn, M. (2024). Frontier Models are Capable of In-context Scheming. arXiv. preprint Cited in §H.2
  119. Méloux, M., Maniu, S., Portet, F., and Peyrard, M. (2025). Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable? International Conference on Learning Representations. Cited in §H.2
  120. Merullo, J., Eickhoff, C., and Pavlick, E. (2023a). Circuit Component Reuse Across Tasks in Transformer Language Models. International Conference on Learning Representations. Cited in §H.2
  121. Metr (2025a). Common Elements of Frontier AI Safety Policies. METR. non-peer-reviewed Cited in §H.4
  122. Metr (2025d). Recent Frontier Models Are Reward Hacking. METR. non-peer-reviewed Cited in §H.2
  123. Michael, J., Mahdi, S., Rein, D., Petty, J., Dirani, J., Padmakumar, V., and Bowman, S. R. (2023). Debate Helps Supervise Unreliable Experts. arXiv. preprint Cited in §H.5
  124. Miller, J., Chughtai, B., and Saunders, W. (2024). Transformer Circuit Faithfulness Metrics are not Robust. arXiv. preprint Cited in §H.2
  125. Murphy, B., Bowen, D., Mohammadzadeh, S., Tseng, T., Broomfield, J., Gleave, A., and Pelrine, K. (2025). Jailbreak-Tuning: Models Efficiently Learn Jailbreak Susceptibility. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Cited in §H.2
  126. Nanda, N., Chan, L., Lieberum, T., Smith, J., and Steinhardt, J. (2023b). Progress measures for grokking via mechanistic interpretability. International Conference on Learning Representations. Cited in §H.2
  127. Nasr, M., Carlini, N., Sitawarin, C., Schulhoff, S. V., Hayes, J., Ilie, M., … Tramèr, F. (2025). The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injections. arXiv. preprint Cited in §H.2
  128. National Research Council (2003). The Polygraph and Lie Detection. National Academies Press. Cited in §H.3
  129. National Research Council (2009). Strengthening Forensic Science in the United States. National Academies Press. Cited in §H.3
  130. Neal, D., and Schanzenbach, D. W. (2010). Left Behind by Design: Proficiency Counts and Test-Based Accountability. Review of Economics and Statistics. Cited in §H.3
  131. Needham, J., Edkins, G., Pimpale, G., Bartsch, H., and Hobbhahn, M. (2025). Large Language Models Often Know When They Are Being Evaluated. arXiv. preprint Cited in §H.2
  132. Niv, Y. (2009). Reinforcement learning in the brain. Journal of Mathematical Psychology. Cited in §H.3
  133. OpenAI (2026aa). The Hugging Face incident and the road ahead. OpenAI. non-peer-reviewed Cited in §H.2
  134. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., … Lowe, R. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems. doi:10.52202/068431-2011. Cited in §H.2
  135. Pan, A., Bhatia, K., and Steinhardt, J. (2022). The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models. International Conference on Learning Representations. Cited in §H.2
  136. Parasuraman, R., and Manzey, D. H. (2010). Complacency and Bias in Human Use of Automation: An Attentional Integration. Human Factors: The Journal of the Human Factors and Ergonomics Society. Cited in §H.3
  137. Paulo, G., and Belrose, N. (2025). Sparse Autoencoders Trained on the Same Data Learn Different Features. arXiv. preprint Cited in §H.2
  138. Pavlick, E., and Kwiatkowski, T. (2019). Inherent Disagreements in Human Textual Inferences. Transactions of the Association for Computational Linguistics. Cited in §H.3
  139. Pelrine, K., Taufeeque, M., Zając, M., McLean, E., and Gleave, A. (2023). Exploiting Novel GPT-4 APIs. arXiv. preprint Cited in §H.2
  140. Phuong, M., Aitchison, M., Catt, E., Cogan, S., Kaskasoli, A., Krakovna, V., … Shevlane, T. (2024). Evaluating Frontier Models for Dangerous Capabilities. arXiv. preprint Cited in §H.2
  141. President's Council of Advisors on Science and Technology (2016). Report to the President: Forensic Science in Criminal Courts: Ensuring Scientific Validity of Feature-Comparison Methods. Executive Office of the President, President's Council of Advisors on Science and Technology. non-peer-reviewed Cited in §H.3
  142. Qi, X., Zeng, Y., Xie, T., Chen, P.-Y., Jia, R., Mittal, P., and Henderson, P. (2024a). Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! International Conference on Learning Representations. Cited in §H.2
  143. Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. Advances in Neural Information Processing Systems. Cited in §H.2
  144. Rajamanoharan, S., and Nanda, N. (2025). Self-preservation or Instruction Ambiguity? Examining the Causes of Shutdown Resistance. LessWrong (Alignment Forum). non-peer-reviewed Cited in §H.2
  145. Röttger, P., Kirk, H. R., Vidgen, B., Attanasio, G., Bianchi, F., and Hovy, D. (2023). XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models. North American Chapter of the Association for Computational Linguistics. Cited in §H.2
  146. Röttger, P., Hofmann, V., Pyatkin, V., Hinck, M., Kirk, H., Schuetze, H., and Hovy, D. (2024). Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Cited in §H.2
  147. Salvi, F., Horta Ribeiro, M., Gallotti, R., and West, R. (2025). On the conversational persuasiveness of GPT-4. Nature Human Behaviour. Cited in §H.3
  148. Scherlis, A., Sachan, K., Jermyn, A. S., Benton, J., and Shlegeris, B. (2022). Polysemanticity and Capacity in Neural Networks. arXiv. preprint Cited in §H.2
  149. Scheurer, J., Balesni, M., and Hobbhahn, M. (2023). Large Language Models can Strategically Deceive their Users when Put Under Pressure. arXiv. preprint Cited in §H.2
  150. Schlatter, J., Weinstein-Raun, B., and Ladish, J. (2025). Incomplete Tasks Induce Shutdown Resistance in Some Frontier LLMs. arXiv. preprint Cited in §H.2
  151. Schoen, B., Nitishinskaya, E., Balesni, M., Højmark, A., Hofstätter, F., Scheurer, J., … Hobbhahn, M. (2025). Stress Testing Deliberative Alignment for Anti-Scheming Training. arXiv. preprint Cited in §H.2
  152. Schultz, W., Dayan, P., and Montague, P. R. (1997). A Neural Substrate of Prediction and Reward. Science. Cited in §H.3
  153. Shah, R., Varma, V., Kumar, R., Phuong, M., Krakovna, V., Uesato, J., and Kenton, Z. (2022). Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals. arXiv. preprint Cited in §H.2
  154. Shamir, A. (1992). IP = PSPACE. Journal of the ACM. Cited in §H.1
  155. Shen, L., Tan, W., Chen, S., Chen, Y., Zhang, J., Xu, H., … Khashabi, D. (2024). The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contexts. Findings of the Association for Computational Linguistics ACL 2024. Cited in §H.2
  156. Shilov, I., Cloud, A., Gema, A. P., Goldman-Wetzler, J., Panickssery, N., Sleight, H., Jones, E., and Anil, C. (2025). Beyond Data Filtering: Knowledge Localization for Capability Removal in LLMs. arXiv. preprint Cited in §H.2
  157. Siththaranjan, A., Laidlaw, C., and Hadfield-Menell, D. (2024). Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF. International Conference on Learning Representations. Cited in §H.1
  158. Snell, C., Klein, D., and Zhong, R. (2022). Learning by Distilling Context. arXiv. preprint Cited in §H.2
  159. Soares, N., Fallenstein, B., Yudkowsky, E., and Armstrong, S. (2015). Corrigibility. AAAI Workshops: Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI Publications. Cited in §H.1
  160. Somerstep, S., Polo, F. M., Banerjee, M., Ritov, Y., Yurochkin, M., and Sun, Y. (2025). A transfer learning framework for weak-to-strong generalization. International Conference on Learning Representations. Cited in §H.1
  161. Sutter, D., Minder, J., Hofmann, T., and Pimentel, T. (2025). The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability? Advances in Neural Information Processing Systems 38. doi:10.52202/085713-5000. Cited in §H.2
  162. Taylor, J. (2016). Quantilizers: A Safer Alternative to Maximizers for Limited Optimization. AAAI 2016 AI, Ethics and Society Workshop. Cited in §H.1
  163. Taylor, J., Black, S., Bowen, D., Read, T., Golechha, S., Zelenka-Martin, A., … Bloom, J. (2025a). Auditing Games for Sandbagging. arXiv. preprint Cited in §H.2
  164. Terekhov, M., Panfilov, A., Dzenhaliou, D., Gulcehre, C., Andriushchenko, M., Prabhu, A., and Geiping, J. (2025a). Adaptive Attacks on Trusted Monitors Subvert AI Control Protocols. arXiv. preprint Cited in §H.2
  165. Thornley, E. (2025). The shutdown problem: an AI engineering puzzle for decision theorists. Philosophical Studies. Cited in §H.1
  166. Turner, A. M., and Tadepalli, P. (2022). Parametrically Retargetable Decision-Makers Tend To Seek Power. Advances in Neural Information Processing Systems. Cited in §H.1
  167. Turner, A. M., Smith, L., Shah, R., Critch, A., and Tadepalli, P. (2021). Optimal Policies Tend to Seek Power. Advances in Neural Information Processing Systems. Cited in §H.1
  168. Turner, E., Soligo, A., Taylor, M., Rajamanoharan, S., and Nanda, N. (2025). Model Organisms for Emergent Misalignment. arXiv. preprint Cited in §H.2
  169. Turpin, M., Michael, J., Perez, E., and Bowman, S. R. (2023). Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. Advances in Neural Information Processing Systems. Cited in §H.2
  170. U.S. AI Safety Institute, N. I. o. S., and Technology (2025). Technical Blog: Strengthening AI Agent Hijacking Evaluations. National Institute of Standards and Technology (NIST). non-peer-reviewed Cited in §H.2
  171. U.S. Department of State, B. o. A. C. V., and Compliance (1963). Treaty Banning Nuclear Weapon Tests in the Atmosphere, in Outer Space and Under Water. U.S. Department of State (archived treaty page). non-peer-reviewed Cited in §H.3
  172. UK AI Security Institute (2025b). Frontier AI Trends Report. UK AI Security Institute. non-peer-reviewed Cited in §H.2
  173. Vaccaro, M., Almaatouq, A., and Malone, T. (2024). When combinations of humans and AI are useful: A systematic review and meta-analysis. Nature Human Behaviour. Cited in §H.3
  174. van der Weij, T., Hofstätter, F., Jaffe, O., Brown, S. F., and Ward, F. R. (2025). AI Sandbagging: Language Models can Strategically Underperform on Evaluations. International Conference on Learning Representations. Cited in §H.2
  175. Wang, T. T., Gleave, A., Tseng, T., Pelrine, K., Belrose, N., Miller, J., … Russell, S. (2023b). Adversarial Policies Beat Superhuman Go AIs. International Conference on Machine Learning. Cited in §H.2
  176. Wang, K., Variengien, A., Conmy, A., Shlegeris, B., and Steinhardt, J. (2023c). Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small. International Conference on Learning Representations. Cited in §H.2
  177. Wang, W., Tu, Z., Chen, C., Yuan, Y., Huang, J.-t., Jiao, W., and Lyu, M. (2024a). All Languages Matter: On the Multilingual Safety of LLMs. Findings of the Association for Computational Linguistics ACL 2024. Cited in §H.2
  178. Wang, Y., Pradel, M., and Liu, Z. (2025a). Are "Solved Issues" in SWE-bench Really Solved Correctly? An Empirical Study. Proceedings of the 2026 IEEE/ACM 48th International Conference on Software Engineering. doi:10.1145/3744916.3764576. Cited in §H.2
  179. Wang, Z., Shi, T., He, J., Cai, M., Zhang, J., and Song, D. (2025b). CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale. arXiv. preprint Cited in §H.2
  180. Wang, M., la Tour, T. D., Watkins, O., Makelov, A., Chi, R. A., Miserendino, S., … Mossing, D. (2025e). Persona Features Control Emergent Misalignment. arXiv. preprint Cited in §H.2
  181. Williams, D. R., and Williams, H. (1969). Auto‐maintenance in the Pigeon: Sustained Pecking Despite Contingent Non‐reinforcement 2. Journal of the Experimental Analysis of Behavior. Cited in §H.3
  182. Wu, D. X., and Sahai, A. (2025). Provable Weak-to-Strong Generalization via Benign Overfitting. International Conference on Learning Representations. Cited in §H.1
  183. Yang, W., Shen, S., Shen, G., Yao, W., Liu, Y., Gong, Z., Lin, Y., and Wen, J.-R. (2024c). Super(ficial)-alignment: Strong Models May Deceive Weak Models in Weak-to-Strong Generalization. arXiv. preprint Cited in §H.2
  184. Yi, J., Xie, Y., Zhu, B., Kiciman, E., Sun, G., Xie, X., and Wu, F. (2025). Benchmarking and Defending against Indirect Prompt Injection Attacks on Large Language Models. Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.1. Cited in §H.2
  185. Yong, Z.-X., Menghini, C., and Bach, S. H. (2023). Low-Resource Languages Jailbreak GPT-4. arXiv. preprint Cited in §H.2
  186. Zhan, Q., Fang, R., Panchal, H. S., and Kang, D. (2025). Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents. Findings of the Association for Computational Linguistics: NAACL 2025. Cited in §H.2
  187. Zhang, F., and Nanda, N. (2023). Towards Best Practices of Activation Patching in Language Models: Metrics and Methods. International Conference on Learning Representations. Cited in §H.2
  188. Zhang, Y., Wei, Z., Luan, X., Wu, C., Zhang, Z., Wu, J., … Sun, M. (2026b). AgentWorm: Self-Propagating Attacks Across LLM Agent Ecosystems. arXiv. preprint Cited in §H.2
  189. Zhong, Z., Liu, Z., Tegmark, M., and Andreas, J. (2023). The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks. Advances in Neural Information Processing Systems. Cited in §H.2
  190. Zou, A., Phan, L., Chen, S., Campbell, J., Guo, P., Ren, R., … Hendrycks, D. (2023a). Representation Engineering: A Top-Down Approach to AI Transparency. arXiv. preprint Cited in §H.2
  191. Zou, A., Lin, M., Jones, E., Nowak, M., Dziemian, M., Winter, N., … Fredrikson, M. (2025). Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition. Advances in Neural Information Processing Systems 38. Cited in §H.2
AI Alignment: From the Optimization Problem to the Institutions Around It · © 2026 Changkun Ou · CC BY-NC-SA 4.0
Bibliography · PV/UV /