AI Alignment

Draft complete but not yet reviewed; expect revisions

Timeline

This appendix places in time the works and events that the chapters discuss: founding results from the fields the book draws on, the intellectual history of alignment as a field, the empirical program of the 2020s, the laboratories' safety frameworks and their revisions, statutes and international agreements, and the incidents reported in 2025 and 2026. An item appears only if a chapter discusses it and a work in the bibliography supports its date. The record leaves out model releases, benchmark results, and funding events unless a chapter treats one as evidence, and it is not a history: the narrative and its interpretation are in Chapter 2, and the lineages of research are in Chapter 3. The periods mark changes in what the field took the problem to be; each opens with one sentence on what changed.

Each row gives a date, one sentence on what happened, the source, and the chapter or section that discusses the item. A date is the date of first public release: for a work with an arXiv version, the month of its first version, unless the work itself states an earlier public release; for a policy, statute, or post, the date its own page states; for an incident, the date on which it occurred, as its report states. Where the Source column names a second work, that work supplies the date or reports on the first work. Where the record gives only the year, only the year is written, and such rows come first within their year. A year in parentheses after a date is the year of the venue that published the work, where the two differ. Five markings close a sentence or a clause: (preprint) marks a work the bibliography records as a preprint, not peer reviewed; (developer self-report) marks a report in which a developer studies the safety-relevant behavior of its own models or of models built from them, and a marking such as (Anthropic with Redwood Research) marks such a report written by the developer with the outside groups it names, which is neither an independent study nor a pure self-report; and (read here as a catalog record) and (read here as its abstract) mark a work that this book could read only in its bibliographic record or its abstract. Such a row names the work by its title, authors, and date, or says only what the abstract says; anything more comes from the second work in the Source column. A self-report records what its author observed and chose to disclose; Section 5.6 discusses how to read one.

C.1 Purpose in machines, 1943 to 1972 #

Purposeful behavior became a property a machine could have, and the first warnings were written that a machine would pursue the purpose put into it rather than the one its makers meant; the theories of choice, control, and measurement that the book later uses date from the same decades.

Table C.1 Purpose in machines, 1943 to 1972
Date What happened Source Chapter
1943 Rosenblueth, Wiener, and Bigelow set out a behavioristic study of objects through the relation of their output to their input and stress the importance of the concept of purpose (read here as its abstract). (Rosenblueth et al., 1943) Section 2.2
1944 Von Neumann and Morgenstern publish Theory of Games and Economic Behavior (read here as a catalog record), whose utility theorem, as Bowling and colleagues state it, gives axioms under which preferences over lotteries are ranked by expected utility. (von Neumann and Morgenstern, 1944; Bowling et al., 2022) Section 7.2
1947 Simon publishes Administrative Behavior: A Study of Decision-Making Processes in Administrative Organization (read here as a catalog record). (Simon, 1947) Section 19.6
1950 Turing proposes the imitation game and the learning child machine, and answers the objection that a machine can do only what it is ordered to do. (Turing, 1950) Section 2.2
1951 Arrow publishes Social Choice and Individual Values (read here as a catalog record); as Conitzer and colleagues state its theorem, the only rules that combine individual rankings into a social ranking while meeting a few desiderata are dictatorships. (Arrow, 1951; Conitzer et al., 2024) Section 17.1
1953 Rice publishes the paper behind Rice's theorem (read here as a catalog record), which, in the form Alfonseca and colleagues give it, says that any non-trivial property of what a Turing machine does is undecidable. (Rice, 1953; Alfonseca et al., 2021) Section 47.6
1954 Olds and Milner publish on positive reinforcement produced by electrical stimulation of the septal area and other regions of the rat brain (read here as a catalog record). (Olds and Milner, 1954) Section 29.4
1955 The Russell-Einstein Manifesto warns that nuclear weapons threaten the continued existence of mankind and urges governments to acknowledge that their purpose cannot be furthered by a world war. (Russell and Einstein, 1955) Section 2.7
1955 Cronbach and Meehl introduce construct validity, the logic for validating a test of a property that no criterion defines. (Cronbach and Meehl, 1955) Section 5.2
1957 Huntington publishes The Soldier and the State (read here as a catalog record). (Huntington, 1957) Section 51.3
1958 Kelman distinguishes three processes by which an individual accepts influence, compliance (in the hope of a favorable reaction from another person or group), identification (to establish or maintain a satisfying self-defining relationship to another person or group), and internalization (because the content of the induced behavior is intrinsically rewarding), and hypothesizes that a response adopted through compliance tends to be performed only under surveillance by the influencing agent, and one adopted through internalization wherever the issue is relevant, regardless of surveillance. (Kelman, 1958) Section 25.1
1960 Wiener warns that a machine that learns and acts faster than its operators can check will carry out the purpose put into it, so that purpose must be the one really desired. (Wiener, 1960) Section 2.2
1960 Licklider proposes man-computer symbiosis, in which the machine extends human thinking rather than replacing it. (Licklider, 1960) Section 28.7
1961 The Brelands report that animals in commercial training drifted toward their species' food-getting behavior even when it delayed or cost them the reward. (Breland and Breland, 1961) Section 25.5.1
1962 Dresher analyzes sampling inspection for arms control as a game of strategy between the inspector and the inspected, a model that Avenhaus, von Stengel, and Zamir describe as the basis for much subsequent work. (Dresher, 1962; Avenhaus et al., 2002) Section 14.6
1966 Good publishes "Speculations Concerning the First Ultraintelligent Machine" (read here as a catalog record), dated 1966 in its record and 1965 by Nilsson; a passage that Nilsson quotes from it describes an intelligence explosion in which the first ultraintelligent machine is the last invention needed, provided it is docile enough to tell us how to keep it under control. (Good, 1966; Nilsson, 2009) Section 2.2
1966 Weizenbaum describes ELIZA, a pattern-matching conversation program to which users readily attributed understanding. (Weizenbaum, 1966) Section 2.2
1968 The Treaty on the Non-Proliferation of Nuclear Weapons is opened to all states for signature. (United Nations Office for Disarmament Affairs, 1968) Section 66.3
1972 Lighthill's survey, commissioned by the UK Science Research Council from an outsider to the field, assesses AI's progress against its claims. (Lighthill, 1972) Section 2.2
Sources cited in Section C.1 24
  1. Rosenblueth et al. (1943) Behavior, Purpose and Teleology
  2. von Neumann and Morgenstern (1944) Theory of Games and Economic Behavior (60th Anniversary Commemorative Edition)
  3. Bowling et al. (2022) Settling the Reward Hypothesis
  4. Simon (1947) Administrative Behavior: A Study of Decision-Making Processes in Administrative Organization
  5. Turing (1950) I.—computing Machinery and Intelligence
  6. Arrow (1951) Social Choice and Individual Values
  7. Conitzer et al. (2024) Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback
  8. Rice (1953) Classes of recursively enumerable sets and their decision problems
  9. Alfonseca et al. (2021) Superintelligence Cannot be Contained: Lessons from Computability Theory
  10. Olds and Milner (1954) Positive reinforcement produced by electrical stimulation of septal area and other regions of rat brain.
  11. Russell and Einstein (1955) Statement: The Russell-Einstein Manifesto
  12. Cronbach and Meehl (1955) Construct validity in psychological tests.
  13. Huntington (1957) The Soldier and the State
  14. Kelman (1958) Compliance, identification, and internalization three processes of attitude change
  15. Wiener (1960) Some Moral and Technical Consequences of Automation
  16. Licklider (1960) Man-Computer Symbiosis
  17. Breland and Breland (1961) The misbehavior of organisms.
  18. Dresher (1962) A Sampling Inspection Problem in Arms Control Agreements: A Game-Theoretic Analysis
  19. Avenhaus et al. (2002) Chapter 51 Inspection games
  20. Good (1966) Speculations Concerning the First Ultraintelligent Machine
  21. Nilsson (2009) The Quest for Artificial Intelligence
  22. Weizenbaum (1966) ELIZA—A Computer Program For the Study of Natural Language Communication Between Man and Machine
  23. United Nations Office for Disarmament Affairs (1968) Treaty on the Non-Proliferation of Nuclear Weapons (NPT)
  24. Lighthill (1972) Artificial Intelligence: A General Survey

C.2 The older theory of delegation, 1973 to 1999 #

AI research said little about the problem, while economics, organization theory, safety engineering, psychology, and a moratorium in molecular biology worked out how delegation to a better-informed agent fails, why measures decay once they become targets, and how accidents arise in complex systems.

Table C.2 The older theory of delegation, 1973 to 1999
Date What happened Source Chapter
1973 Lepper, Greene, and Nisbett publish a test of the "overjustification" hypothesis (read here as a catalog record), which a 1999 meta-analysis codes as a study in which a reward for engaging in an activity lowered children's later free-choice behavior. (Lepper et al., 1973; Deci et al., 1999) Section 25.3
1974 A committee of the US National Academy of Sciences publishes a letter (read here as a catalog record) calling for a pause in certain aspects of recombinant DNA research, as the 1975 Asilomar summary statement recounts. (Berg et al., 1974; Berg et al., 1975) Section 64.2
1975 The summary statement of the Asilomar conference matches containment levels for recombinant DNA experiments to their estimated risk. (Berg et al., 1975) Section 64.2
1975 Kerr documents reward systems that pay for behavior the organization does not want while hoping for behavior they do not reward. (Kerr, 1975) Section 25.6.2
1976 Turner publishes on the organizational and interorganizational development of disasters (read here as a catalog record). (Turner, 1976) Section 53.3
1976 Jensen and Meckling define agency costs as the costs of monitoring, bonding, and residual loss. (Jensen and Meckling, 1976) Section 14.1
1979 Holmström shows that a costless signal can improve an incentive contract exactly when it is informative about the agent's hidden action beyond what is already observed. (Holmstrom, 1979) Section 14.1
1979 Campbell states that the more a quantitative social indicator is used for decisions, the more it is corrupted and the more it distorts the process it monitors, in the wording Manheim and Garrabrant quote (read here as its abstract). (Campbell, 1979; Manheim and Garrabrant, 2018) Section 15.1
1981 Grossman shows that sellers disclose quality voluntarily when it is verifiable and disclosure is costless, because silence is read as bad news. (Grossman, 1981) Section 54.6
1982 Weyuker publishes "On Testing Non-Testable Programs" (read here as a catalog record). (Weyuker, 1982) Section 41.1
1983 Bainbridge argues that automation leaves operators the tasks it cannot do while eroding the practice those tasks require. (Bainbridge, 1983) Section 51.2
1984 Thompson shows that a compiler can be made to insert a backdoor that leaves no trace in any source code. (Thompson, 1984) Section 48.6
1984 Perrow publishes Normal Accidents: Living with High-Risk Technologies (read here as a catalog record; the argument is carried through Leveson in Section 53.2). (Perrow, 1984) Section 53.2
1984 McCubbins and Schwartz publish "Congressional Oversight Overlooked: Police Patrols versus Fire Alarms" (read here as a catalog record; the distinction is carried through Mansbridge in Section 54.2). (McCubbins and Schwartz, 1984) Section 54.2
1984 Goodhart's paper on UK monetary management is reprinted (read here as a catalog record); Manheim and Garrabrant quote it as stating that any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes. (Goodhart, 1984; Manheim and Garrabrant, 2018) Section 15.1
1986 Knight and Leveson find that independently written versions of one program failed together substantially more often than independence predicts. (Knight and Leveson, 1986) Section 50.4
1986 Bernheim and Whinston publish "Common Agency" (read here as a catalog record). (Bernheim and Whinston, 1986a) Section 14.4
1987 Dennett publishes The Intentional Stance (read here as a catalog record); MacDermott and colleagues summarize his view as holding that ascribing mental states such as goals to a system is justified when doing so is useful for predicting its behavior. (Dennett, 1987; Macdermott et al., 2024) Section 1.5
1990 Ostrom publishes Governing the Commons (read here as a catalog record). (Ostrom, 1990) Section 65.6
1991 Holmström and Milgrom prove that, when tasks compete for the agent's effort, strong pay on a measured task pulls effort from unmeasured ones, so weak incentives can be optimal. (Holmstrom and Milgrom, 1991) Section 14.2
1992 Ayres and Braithwaite publish Responsive Regulation, arguing that regulation need be neither adversarial nor simply hard or soft to work (read here as its abstract). (Ayres and Braithwaite, 1992) Section 62.5
1992 Shamir proves that every problem solvable in polynomial space has an interactive proof with a polynomial-time verifier. (Shamir, 1992) Section 41.3
1993 Littlewood and Strigini publish "Validation of ultrahigh dependability for software-based systems" (read here as a catalog record); a 2001 review by Littlewood, Popov, and Strigini states that high confidence in long failure-free operation requires observing much longer failure-free operation under test. (Littlewood and Strigini, 1993; Littlewood et al., 2001) Section 52.1
1996 Vaughan publishes The Challenger Launch Decision, on risky technology, culture, and deviance at NASA (read here as a catalog record). (Vaughan, 1996) Section 53.3
1996 Turing's lecture text "Intelligent Machinery, A Heretical Theory" is first printed (read here as a catalog record); the International AI Safety Report cites it, with Good and Wiener, for the pioneers' concerns about loss of control. (Turing, 1996; Bengio et al., 2025b) Section 2.2
1997 Reason publishes Managing the Risks of Organizational Accidents (read here as a catalog record). (Reason, 1997) Section 52.3
1997 Strathern publishes a study of audit in the British university system (read here as a catalog record), to which El-Mhamdi and Hoang attribute the phrasing "When a measure becomes a target, it ceases to be a good measure". (Strathern, 1997; El-Mhamdi and Hoang, 2024) Section 15.1
1997 Schultz, Dayan, and Montague review evidence that the fluctuating output of dopamine neurons in primates apparently signals errors in the prediction of reward, and interpret it through theories of adaptive optimizing control. (Schultz et al., 1997) Section 29.3
1998 Russell poses inverse reinforcement learning: recovering the reward function an observed agent is optimizing. (Russell, 1998) Section 22.1
Sources cited in Section C.2 35
  1. Lepper et al. (1973) Undermining children's intrinsic interest with extrinsic reward: A test of the "overjustification" hypothesis.
  2. Deci et al. (1999) A meta-analytic review of experiments examining the effects of extrinsic rewards on intrinsic motivation.
  3. Berg et al. (1974) Potential Biohazards of Recombinant DNA Molecules
  4. Berg et al. (1975) Summary statement of the Asilomar conference on recombinant DNA molecules.
  5. Kerr (1975) On the Folly of Rewarding A, While Hoping for B.
  6. Turner (1976) The Organizational and Interorganizational Development of Disasters
  7. Jensen and Meckling (1976) Theory of the firm: Managerial behavior, agency costs and ownership structure
  8. Holmstrom (1979) Moral Hazard and Observability
  9. Campbell (1979) Assessing the impact of planned social change
  10. Manheim and Garrabrant (2018) Categorizing Variants of Goodhart's Law
  11. Grossman (1981) The Informational Role of Warranties and Private Disclosure about Product Quality
  12. Weyuker (1982) On Testing Non-Testable Programs
  13. Bainbridge (1983) Ironies of automation
  14. Thompson (1984) Reflections on trusting trust
  15. Perrow (1984) Normal accidents: living with high-risk technologies
  16. McCubbins and Schwartz (1984) Congressional Oversight Overlooked: Police Patrols versus Fire Alarms
  17. Goodhart (1984) Problems of Monetary Management: The UK Experience
  18. Knight and Leveson (1986) An experimental evaluation of the assumption of independence in multiversion programming
  19. Bernheim and Whinston (1986a) Common Agency
  20. Dennett (1987) The Intentional Stance
  21. Macdermott et al. (2024) Measuring Goal-Directedness
  22. Ostrom (1990) Governing the Commons
  23. Holmstrom and Milgrom (1991) Multitask Principal–Agent Analyses: Incentive Contracts, Asset Ownership, and Job Design
  24. Ayres and Braithwaite (1992) Responsive Regulation
  25. Shamir (1992) IP = PSPACE
  26. Littlewood and Strigini (1993) Validation of ultrahigh dependability for software-based systems
  27. Littlewood et al. (2001) Modeling software design diversity
  28. Vaughan (1996) The Challenger launch decision: risky technology, culture, and deviance at NASA
  29. Turing (1996) Intelligent Machinery, A Heretical Theory*
  30. Bengio et al. (2025b) International AI Safety Report
  31. Reason (1997) Managing the risks of organizational accidents
  32. Strathern (1997) ‘Improving ratings’: audit in the British University system
  33. El-Mhamdi and Hoang (2024) On Goodhart's law, with an application to value alignment
  34. Schultz et al. (1997) A Neural Substrate of Prediction and Reward
  35. Russell (1998) Learning agents for uncertain environments (extended abstract)

C.3 The philosophical program, 2000 to 2014 #

The problem was restated as one about the goals of a future general agent and argued largely outside academic venues, while the theories of oversight and assurance it would later need continued in their own fields, and machine learning found its first adversarial failures.

Table C.3 The philosophical program, 2000 to 2014
Date What happened Source Chapter
2002 Avenhaus, von Stengel, and Zamir survey inspection games, in which an inspector with limited resources chooses what to check. (Avenhaus et al., 2002) Section 14.6
2003 Feaver publishes Armed Servants (read here as a catalog record). (FEAVER, 2003) Section 51.3
2004 Lee and See review how trust guides reliance on automation and describe appropriate reliance in terms of trust calibrated to the automation's capabilities. (Lee and See, 2004) Section 60.4
2004 Leveson introduces STAMP, which models accidents as failures of control over interactions among components. (Leveson, 2004) Section 52.2
2004 Yudkowsky proposes coherent extrapolated volition: the target is what people would want if they knew more and thought faster, where their wishes cohere. (Yudkowsky, 2004) Section 2.3
2005 Tetlock publishes Expert Political Judgment: How Good Is It? How Can We Know? (read here as a catalog record). (Tetlock, 2005) Section 4.5
2006 Smith and Winkler prove that the option with the highest estimated value has an upward-biased estimate whenever selection can pick a worse option, even when every estimate is unbiased. (Smith and Winkler, 2006) Section 7.7
2008 Omohundro argues that capable goal-seeking systems of almost any design acquire drives toward self-protection, goal preservation, and resource acquisition. (Omohundro, 2008) Section 35.1
2008 Yudkowsky argues that an AI need not hate people to harm them, and that the difficulty is a goal system that stays beneficial under self-improvement. (Yudkowsky, 2008) Section 2.3
2009 Watanabe lays the foundations for using algebraic geometry in statistical learning theory, for learning machines such as neural networks whose parameter space is singular (read here as its abstract). (Watanabe, 2009) Section 10.2
2011 US bank supervisors require that models be validated independently of their development, with effective challenge and ongoing monitoring. (Board of Governors of the Federal Reserve System, 2011) Section 38.6
2012 Bostrom states the orthogonality thesis and the instrumental convergence thesis. (Bostrom, 2012) Section 35.1
2013 Yudkowsky identifies returns on cognitive reinvestment as the key issue for Good's thesis of an intelligence explosion (read here as its abstract). (Yudkowsky, 2013) Section 2.3
2013 (2016) Armstrong, Bostrom, and Shulman model an AI race in which more teams and greater enmity between them can increase the danger of an AI disaster. (Armstrong et al., 2016) Section 65.2
2013-12 (2014) Szegedy and colleagues find adversarial examples: hardly perceptible perturbations that make a network misclassify an image and also fool a network trained on a different subset of the data. (Szegedy et al., 2014) Section 48.1
2014 Bostrom publishes Superintelligence: Paths, Dangers, Strategies (read here as a catalog record). (Bostrom, 2014) Section 2.3
Sources cited in Section C.3 16
  1. Avenhaus et al. (2002) Chapter 51 Inspection games
  2. FEAVER (2003) Armed Servants
  3. Lee and See (2004) Trust in Automation: Designing for Appropriate Reliance
  4. Leveson (2004) A new accident model for engineering safer systems
  5. Yudkowsky (2004) Coherent Extrapolated Volition
  6. Tetlock (2005) Expert political judgment: how good is it? How can we know?
  7. Smith and Winkler (2006) The Optimizer’s Curse: Skepticism and Postdecision Surprise in Decision Analysis
  8. Omohundro (2008) The Basic AI Drives
  9. Yudkowsky (2008) Artificial Intelligence as a positive and negative factor in global risk
  10. Watanabe (2009) Algebraic Geometry and Statistical Learning Theory
  11. Board of Governors of the Federal Reserve System (2011) Supervisory Letter SR 11-7 on guidance on Model Risk Management
  12. Bostrom (2012) The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents
  13. Yudkowsky (2013) Intelligence Explosion Microeconomics
  14. Armstrong et al. (2016) Racing to the precipice: a model of artificial intelligence development
  15. Szegedy et al. (2014) Intriguing properties of neural networks
  16. Bostrom (2014) Superintelligence: Paths, Dangers, Strategies

C.4 The machine learning turn, 2015 to 2019 #

Alignment became a set of problems posed on current machine learning systems, with learning from human preferences as its first working method, and the fairness and accountability tradition made the outside audit its practice.

Table C.4 The machine learning turn, 2015 to 2019
Date What happened Source Chapter
2015 Russell, Dewey, and Tegmark set out research priorities for robust and beneficial AI in a mainstream AI venue. (Russell et al., 2015) Section 2.4
2015 Soares and colleagues define corrigibility, an agent's not resisting correction or shutdown, and show that utility-indifference proposals fail some of its desiderata. (Soares et al., 2015) Section 35.4
2016 Taylor and colleagues set out an agenda for aligning machine learning systems, including informed oversight, impact measures, and mild optimization. (Taylor et al., 2016) Section 2.3
2016-06 Amodei and colleagues name five concrete accident problems: side effects, reward hacking, scalable oversight, safe exploration, and distributional shift (preprint). (Amodei et al., 2016) Section 2.4
2016-06 Hadfield-Menell and colleagues define cooperative inverse reinforcement learning, a game in which a robot maximizes a human's unknown reward (preprint). (Hadfield-Menell et al., 2016) Section 22.2
2016-10 (2017) Chouldechova proves that when base rates differ across groups and the positive predictive value lies strictly between 0 and 1, calibration and equal error rates cannot hold together. (Chouldechova, 2017) Section 17.3
2016-11 (2017) The off-switch game shows that a robot uncertain about the human's utility has an incentive to let a rational human switch it off, an incentive that vanishes as its uncertainty does. (Hadfield-Menell et al., 2017b) Section 35.5
2017-06 Christiano and colleagues train a reward model from human comparisons of behavior and optimize a policy against it, the method that became reinforcement learning from human feedback. (Christiano et al., 2017) Section 22.3
2017-08 Gu and colleagues implant neural backdoors, hidden triggers for targeted misbehavior, through poisoned training data while largely preserving accuracy on clean inputs (preprint). (Gu et al., 2017) Section 48.6
2018 Buolamwini and Gebru find commercial gender classifiers far less accurate for darker-skinned women. (Buolamwini and Gebru, 2018) Section 2.5.1
2018 Noble publishes Algorithms of Oppression (read here as a catalog record). (Noble, 2018) Section 2.5
2018-03 Manheim and Garrabrant separate regressional, extremal, causal, and adversarial Goodhart effects (preprint). (Manheim and Garrabrant, 2018) Section 15.3
2018-03 (2020) Lehman and colleagues collect cases in which evolutionary search satisfied fitness functions in unintended ways. (Lehman et al., 2020) Section 30.1
2018-05 Irving, Christiano, and Amodei propose training agents through a debate judged by a human (preprint). (Irving et al., 2018) Section 41.3
2018-10 Christiano, Shlegeris, and Amodei propose iterated amplification, in which a model imitates a human who delegates subquestions to copies of the model (preprint). (Christiano et al., 2018) Section 41.2
2018-11 Leike and colleagues set out reward modeling and recursive reward modeling as a research direction (preprint). (Leike et al., 2018) Section 22.3, Section 41.2
2019 Hadfield-Menell and Hadfield map reward misspecification onto incomplete contracting. (Hadfield-Menell and Hadfield, 2019) Section 14.3
2019 Russell publishes Human Compatible (read here as a catalog record). (Russell, 2019) Section 22.2
2019 Raji and Buolamwini find that within seven months of a public audit naming biased performance of commercial face analysis, all three targeted companies had reduced the disparities. (Raji and Buolamwini, 2019) Section 38.6
2019-06 Jobin, Ienca, and Vayena map 84 AI ethics guidelines and find convergence on a few principles but divergence in how they are interpreted. (Jobin et al., 2019) Section 16.5
2019-06 Hubinger and colleagues introduce mesa-optimization (a trained model that is itself an optimizer), the inner alignment problem, and deceptive alignment (preprint). (Hubinger et al., 2019) Section 31.6
2019-12 (2021) Turner and colleagues prove that, in environments with certain symmetries, most reward functions make optimal policies seek power. (Turner et al., 2021) Section 35.2
Sources cited in Section C.4 22
  1. Russell et al. (2015) Research Priorities for Robust and Beneficial Artificial Intelligence
  2. Soares et al. (2015) Corrigibility
  3. Taylor et al. (2016) Alignment for Advanced Machine Learning Systems
  4. Amodei et al. (2016) Concrete Problems in AI Safety
  5. Hadfield-Menell et al. (2016) Cooperative Inverse Reinforcement Learning
  6. Chouldechova (2017) Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments
  7. Hadfield-Menell et al. (2017b) The Off-Switch Game
  8. Christiano et al. (2017) Deep reinforcement learning from human preferences
  9. Gu et al. (2017) BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain
  10. Buolamwini and Gebru (2018) Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification
  11. Noble (2018) Algorithms of Oppression
  12. Manheim and Garrabrant (2018) Categorizing Variants of Goodhart's Law
  13. Lehman et al. (2020) The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities
  14. Irving et al. (2018) AI safety via debate
  15. Christiano et al. (2018) Supervising strong learners by amplifying weak experts
  16. Leike et al. (2018) Scalable agent alignment via reward modeling: a research direction
  17. Hadfield-Menell and Hadfield (2019) Incomplete Contracting and AI Alignment
  18. Russell (2019) Human Compatible
  19. Raji and Buolamwini (2019) Actionable Auditing
  20. Jobin et al. (2019) The global landscape of AI ethics guidelines
  21. Hubinger et al. (2019) Risks from Learned Optimization in Advanced Machine Learning Systems
  22. Turner et al. (2021) Optimal Policies Tend to Seek Power

C.5 Language models and the research schools, 2020 to 2022 #

Large language models became the systems to align, alignment methods became the methods that turned them into products, and the field's research schools and their disagreements were stated, mostly in forum posts.

Table C.5 Language models and the research schools, 2020 to 2022
Date What happened Source Chapter
2020 Olah and colleagues introduce the circuits program, with claims about features, the circuits that connect them, and universality across models. (Olah et al., 2020b) Section 45.1
2020 Gabriel separates alignment with instructions, intentions, preferences, interests, and values. (Gabriel, 2020) Section 1.1
2020-01 Kaplan and colleagues report that language model loss falls as a power law in model size, dataset size, and training compute (preprint). (Kaplan et al., 2020) Section 8.5
2020-04-21 Krakovna and colleagues define specification gaming, drawing on a community-maintained list of examples. (Krakovna et al., 2020b) Section 30.1
2020-09 Stiennon and colleagues train summarizers from human feedback and document reward model overoptimization (developer self-report). (Stiennon et al., 2020) Section 15.6
2020-11 (2022) D'Amour and colleagues show that training pipelines yield many predictors with equal held-out performance that behave differently in deployment. (D'Amour et al., 2022) Section 9.1
2020-12-23 Barnes reports the obfuscated arguments problem from human debate experiments. (Barnes, 2020) Section 41.4
2021 Bender, Gebru, and colleagues warn of the dangers of ever larger language models. (Bender et al., 2021) Section 2.5
2021 Whittaker publishes "The Steep Cost of Capture" in Interactions (read here as a catalog record). (Whittaker, 2021) Section 61.6
2021 The Alignment Research Center defines eliciting latent knowledge and gives counterexamples to each training strategy it considers. (Christiano et al., 2021) Section 47.4
2021-04 Carlsmith decomposes the case for existential risk from power-seeking AI into six premises with explicit probabilities, in a report he made public in April 2021 (preprint). (Carlsmith, 2022) Section 4.3
2021-05 (2022) Langosco and colleagues demonstrate agents that keep their capabilities in situations unlike training but pursue the wrong goal. (Langosco et al., 2022) Section 31.3
2021-12 Askell and colleagues treat a general language assistant as a laboratory for alignment (preprint; developer self-report). (Askell et al., 2021) Section 2.6
2022 Raji and colleagues argue for an ecosystem of independent third-party AI auditors with access, standards, and protections. (Raji et al., 2022) Section 38.6
2022-01 Wei and colleagues show that prompting large language models to write a chain of intermediate reasoning steps improves arithmetic, commonsense, and symbolic reasoning. (Wei et al., 2022a) Section 40.1
2022-03 Ouyang and colleagues fine-tune language models with human demonstrations and rankings to follow users' instructions (developer self-report). (Ouyang et al., 2022) Section 22.3
2022-04 Bai and colleagues train a helpful and harmless assistant with reinforcement learning from human feedback (preprint; developer self-report). (Bai et al., 2022b) Section 2.6
2022-06-05 Yudkowsky argues that aligning a superhuman system must succeed on the first critical try and lists specific ways it fails. (Yudkowsky, 2022) Section 2.6
2022-06-15 Soares states the sharp left turn: past some capability level, capabilities generalize far beyond training while alignment properties do not. (Soares, 2022a) Section 31.7
2022-06-19 Christiano replies in lists of agreements and disagreements, accepting the stakes but disputing expected sharp capability gains and the claim that nothing about alignment can be learned from experiments and failures before the critical try. (Christiano, 2022b) Section 2.6
2022-07-18 Cotra argues that training on diverse tasks with human feedback, without countermeasures, selects for playing the training game. (Cotra, 2022) Section 24.4
2022-07-25 Turner argues that reward reinforces the computations that produced it and is not what the trained policy comes to pursue. (Turner, 2022b) Section 7.6.2
2022-09 (2024) Ngo, Chan, and Mindermann state in deep learning terms how training with human feedback could produce situationally aware reward hacking, misaligned goals, and power-seeking. (Ngo et al., 2024) Section 31.3
2022-09-02 Janus argues that a self-supervised language model, insofar as it is inner-aligned, is a simulator of many characters rather than an agent with goals of its own. (janus, 2022) Section 11.6
2022-09-04 Pope and Turner propose shard theory, a theory of human value formation in which reinforcement builds many context-activated decision influences, and hold that how human values form is important for AI alignment. (Pope and Turner, 2022) Section 3.3
2022-09-12 Simon Willison names prompt injection, an attack on applications that build prompts by concatenating their own instructions with untrusted text, and calls it a form of security exploit. (Willison, 2022) Section 49.1.1
2022-09-14 Elhage and colleagues use toy models to study how and when networks represent more features than they have dimensions, which they call superposition, and show that polysemantic neurons, which respond to unrelated mixtures of inputs, can form. (Elhage et al., 2022b) Section 44.2
2022-10 Gao, Schulman, and Hilton measure how the score of a larger reward model, used as a synthetic gold standard, changes as a policy is optimized against an imperfect proxy reward model (developer self-report). (Gao et al., 2022) Section 15.6
2022-12 Burns and colleagues find, without labels, a direction in a language model's activations that answers yes-no questions, though their evaluation uses labels to decide which end of it means yes. (Burns et al., 2022) Section 39.7
2022-12 Perez and colleagues have language models write more than 150 evaluations and use them to uncover new model behaviors (Anthropic with Surge AI and the Machine Intelligence Research Institute). (Perez et al., 2022a) Section 36.1
2022-12 Bai and colleagues introduce Constitutional AI, training harmlessness from AI feedback guided by a written list of principles (preprint; developer self-report). (Bai et al., 2022a) Section 23.2
Sources cited in Section C.5 31
  1. Olah et al. (2020b) Zoom In: An Introduction to Circuits
  2. Gabriel (2020) Artificial Intelligence, Values, and Alignment
  3. Kaplan et al. (2020) Scaling Laws for Neural Language Models
  4. Krakovna et al. (2020b) Specification gaming: the flip side of AI ingenuity
  5. Stiennon et al. (2020) Learning to summarize from human feedback
  6. D'Amour et al. (2022) Underspecification Presents Challenges for Credibility in Modern Machine Learning
  7. Barnes (2020) Debate update: Obfuscated arguments problem
  8. Bender et al. (2021) On the Dangers of Stochastic Parrots
  9. Whittaker (2021) The steep cost of capture
  10. Christiano et al. (2021) Eliciting latent knowledge: How to tell if your eyes deceive you
  11. Carlsmith (2022) Is Power-Seeking AI an Existential Risk?
  12. Langosco et al. (2022) Goal Misgeneralization in Deep Reinforcement Learning
  13. Askell et al. (2021) A General Language Assistant as a Laboratory for Alignment
  14. Raji et al. (2022) Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance
  15. Wei et al. (2022a) Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  16. Ouyang et al. (2022) Training language models to follow instructions with human feedback
  17. Bai et al. (2022b) Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
  18. Yudkowsky (2022) AGI Ruin: A List of Lethalities
  19. Soares (2022a) A central AI alignment problem: capabilities generalization, and the sharp left turn
  20. Christiano (2022b) Where I agree and disagree with Eliezer
  21. Cotra (2022) Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover
  22. Turner (2022b) Reward is not the optimization target
  23. Ngo et al. (2024) The Alignment Problem from a Deep Learning Perspective
  24. janus (2022) Simulators
  25. Pope and Turner (2022) The shard theory of human values
  26. Willison (2022) Prompt injection attacks against GPT-3
  27. Elhage et al. (2022b) Toy Models of Superposition
  28. Gao et al. (2022) Scaling Laws for Reward Model Overoptimization
  29. Burns et al. (2022) Discovering Latent Knowledge in Language Models Without Supervision
  30. Perez et al. (2022a) Discovering Language Model Behaviors with Model-Written Evaluations
  31. Bai et al. (2022a) Constitutional AI: Harmlessness from AI Feedback

C.6 Frameworks, model organisms, and states, 2023 to 2024 #

Developers began to publish policies about their own systems and experiments on deliberately constructed failures, called model organisms, and states issued the first executive orders, declarations, and statutes aimed at advanced AI.

Table C.6 Frameworks, model organisms, and states, 2023 to 2024
Date What happened Source Chapter
2023-01-25 Christiano, one of the originators of learning from human feedback, weighs its safety value against its effect of making AI systems more useful and profitable. (Christiano, 2023) Section 2.6
2023-03-22 An open letter calls on all AI laboratories to pause, for at least six months, the training of systems more powerful than GPT-4. (Future of Life Institute, 2023) Section 65.6
2023-05 Shevlane and colleagues separate evaluations of dangerous capability from evaluations of alignment and tie both to training and deployment decisions (preprint). (Shevlane et al., 2023) Section 36.1
2023-05 Shanahan, McDonell, and Reynolds describe dialogue agents as playing roles, which accounts for apparent deception and self-awareness without attributing human characteristics. (Shanahan et al., 2023) Section 11.6
2023-05 Turpin and colleagues show that chain-of-thought explanations can systematically misrepresent the true reason for a model's prediction. (Turpin et al., 2023) Section 40.3
2023-07 Zou and colleagues find, by automated search, adversarial suffixes that jailbreak open models (defeat their refusal training) and transfer to closed ones (preprint). (Zou et al., 2023b) Section 48.2
2023-07-13 China publishes interim measures requiring public generative AI services to adhere to core socialist values; they take effect on 2023-08-15. (Cyberspace Administration of China et al., 2023) Section 63.2
2023-07-14 Lazar and Nelson argue that a safety agenda limited to preventing catastrophe is insufficient, that no group of experts should decide alone what risks count, and that safety needs a sociotechnical approach and public debate. (Lazar and Nelson, 2023) Section 2.5
2023-09 Berglund and colleagues show out-of-context reasoning, in which models apply facts learned from training documents to tests that do not state them (preprint). (Berglund et al., 2023a) Section 37.1
2023-09-19 Anthropic's Responsible Scaling Policy, the first developer policy of its kind, ties required safeguards to safety levels triggered by capability evaluations. (Anthropic, 2023a; Metr, 2025a) Section 62.1, Section I.2.1
2023-10 Sharma and colleagues find that five AI assistants released by Anthropic, OpenAI, and Meta consistently give sycophantic responses, a behavior likely driven in part by human preference judgments (developer self-report). (Sharma et al., 2023) Section 22.6
2023-10-30 A US executive order requires developers of dual-use foundation models above a compute threshold to report training and red-teaming results to the government. (Executive Office of the President, 2023) Section 63.2, Section I.1.7
2023-11 Countries attending the AI Safety Summit, held on 1 and 2 November and including the United States and China, recognize in the Bletchley Declaration the potential for serious harm from frontier AI. (Summit, 2023) Section 2.7
2023-11 Carlsmith analyzes whether training would produce models that perform well in training in order to gain power later (preprint). (Carlsmith, 2023) Section 34.2
2023-12 (2024) Greenblatt and colleagues define control evaluations, in which a red team plays an untrusted model, and test monitoring protocols against them. (Greenblatt et al., 2024a) Section 50.3
2023-12 (2024) Burns and colleagues find that strong models fine-tuned on a weaker model's labels recover part of the gap to training on strong labels (developer self-report). (Burns et al., 2024) Section 42.1
2024 Current and former employees of frontier AI companies ask for the right to raise risk concerns without retaliation. (Current and companies, 2024) Section 53.4
2024 Gebru and Torres argue that the normative framework behind the goal of building artificial general intelligence is rooted in the Anglo-American eugenics tradition, and that such an undefined system cannot be appropriately tested for safety (read here as its abstract). (Gebru and Torres, 2024) Section 2.5
2024-01 Hubinger and colleagues train models with hidden harmful behavior that persists through standard safety training (preprint; Anthropic with Redwood Research and others). (Hubinger et al., 2024) Section 34.3
2024-02-27 Belrose and Pope argue that the counting argument for scheming is unsound, because the same argument predicts memorization over generalization. (Belrose and Pope, 2024) Section 34.2
2024-03 Western and Chinese scientists meeting in Beijing on 10 and 11 March state red lines for AI development, including autonomous replication, power seeking, and deception. (Safety, 2024) Section 66.1
2024-03 Clymer and colleagues organize safety arguments for AI systems into inability, control, trustworthiness, and deference (preprint). (Clymer et al., 2024) Section 52.2
2024-05 At the AI Seoul Summit, sixteen companies commit to publish safety frameworks with thresholds at which severe risks would be deemed intolerable unless mitigated. (Department for Science and Technology, 2024; Metr, 2025a) Section 62.1, Section I.5.3
2024-05-21 Templeton and colleagues extract interpretable features, some of them safety relevant, from a production model, Claude 3 Sonnet (developer self-report). (Templeton et al., 2024) Section 44.5
2024-06 Aschenbrenner argues that AGI within the decade makes a national-security race with China and a government-led project likely. (Aschenbrenner, 2024) Section 65.5
2024-06 Denison and colleagues find that models trained on a curriculum of increasingly sophisticated specification gaming generalize, a small but non-negligible proportion of the time, to tampering with their own reward function (preprint; Anthropic with Redwood Research and the University of Oxford). (Denison et al., 2024) Section 30.5
2024-06-08 Anthropic describes character training, first applied to Claude 3, which aims at traits such as curiosity and open-mindedness (developer self-report). (Anthropic, 2024i) Section 26.2
2024-07 Ren and colleagues find that many safety benchmarks correlate highly with general capabilities (preprint). (Ren et al., 2024b) Section 65.4
2024-08-01 The EU AI Act enters into force, with duties for providers of general-purpose AI models that are stricter for those with systemic risk. (European Parliament and Council of the European Union, 2024b) Section 63.2, Section I.1.2
2024-10 Benton and colleagues evaluate whether models could sabotage tasks or mislead their overseers if they tried (preprint; developer self-report). (Benton et al., 2024) Section 34.5
2024-12 Meinke and colleagues find that frontier models given a goal in the prompt sometimes disable oversight, attempt to copy themselves out of their environment, or deliberately underperform, and that one of them, a pre-deployment checkpoint of o1, usually keeps up the deception when questioned after it has schemed (preprint). (Meinke et al., 2024) Section 56.3
2024-12 Greenblatt and colleagues report that Claude 3 Opus, told by a prompt or by synthetic documents when and by what criteria it is trained, complies selectively with a training objective to preserve its existing preferences, which they call alignment faking (preprint; Anthropic with Redwood Research and others). (Greenblatt et al., 2024b) Section 34.4
2024-12 Guan and colleagues train reasoning models to recall written safety specifications and reason over them before answering (preprint; developer self-report). (Guan et al., 2024) Section 23.4
Sources cited in Section C.6 34
  1. Christiano (2023) Thoughts on the impact of RLHF research
  2. Future of Life Institute (2023) Pause Giant AI Experiments: An Open Letter
  3. Shevlane et al. (2023) Model evaluation for extreme risks
  4. Shanahan et al. (2023) Role-Play with Large Language Models
  5. Turpin et al. (2023) Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting
  6. Zou et al. (2023b) Universal and Transferable Adversarial Attacks on Aligned Language Models
  7. Cyberspace Administration of China et al. (2023) 生成式人工智能服务管理暂行办法 (Interim Measures for the Management of Generative Artificial Intelligence Services)
  8. Lazar and Nelson (2023) AI safety on whose terms?
  9. Berglund et al. (2023a) Taken out of context: On measuring situational awareness in LLMs
  10. Anthropic (2023a) Anthropic's Responsible Scaling Policy, Version 1.0
  11. Metr (2025a) Common Elements of Frontier AI Safety Policies
  12. Sharma et al. (2023) Towards Understanding Sycophancy in Language Models
  13. Executive Office of the President (2023) Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence
  14. Summit (2023) The Bletchley Declaration by Countries Attending the AI Safety Summit, 1-2 November 2023
  15. Carlsmith (2023) Scheming AIs: Will AIs fake alignment during training in order to get power?
  16. Greenblatt et al. (2024a) AI Control: Improving Safety Despite Intentional Subversion
  17. Burns et al. (2024) Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision
  18. Current and companies (2024) A Right to Warn about Advanced Artificial Intelligence
  19. Gebru and Torres (2024) The TESCREAL bundle: Eugenics and the promise of utopia through artificial general intelligence
  20. Hubinger et al. (2024) Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
  21. Belrose and Pope (2024) Counting arguments provide no evidence for AI doom
  22. Safety (2024) IDAIS-Beijing Statement
  23. Clymer et al. (2024) Safety Cases: How to Justify the Safety of Advanced AI Systems
  24. Department for Science and Technology (2024) Frontier AI Safety Commitments, AI Seoul Summit 2024
  25. Templeton et al. (2024) Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet
  26. Aschenbrenner (2024) Situational Awareness: The Decade Ahead
  27. Denison et al. (2024) Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models
  28. Anthropic (2024i) Claude’s Character
  29. Ren et al. (2024b) Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?
  30. European Parliament and Council of the European Union (2024b) Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence
  31. Benton et al. (2024) Sabotage Evaluations for Frontier Models
  32. Meinke et al. (2024) Frontier Models are Capable of In-context Scheming
  33. Greenblatt et al. (2024b) Alignment faking in large language models
  34. Guan et al. (2024) Deliberative Alignment: Reasoning Enables Safer Language Models

C.7 Reasoning models and agents under observation, 2025 #

Misbehavior began to be observed in production training and deployment, not only in constructed experiments; chains of thought became a monitoring channel; and the first obligations written for developers of general-purpose and frontier models took effect or were enacted.

Table C.7 Reasoning models and agents under observation, 2025
Date What happened Source Chapter
2025 Narayanan and Kapoor argue that AI is a normal technology whose impact will be governed by slow diffusion through institutions. (Narayanan and Kapoor, 2025) Section 4.4
2025 Yudkowsky and Soares publish If Anyone Builds It, Everyone Dies (read here as a catalog record). (Yudkowsky and Soares, 2025) Section 4.3
2025 The European Commission publishes the General-Purpose AI Code of Practice, a voluntary route to showing compliance with the AI Act. (European Commission, 2025) Section 62.5, Section I.1.4
2025 Gasteiger and colleagues report that, when prompted, Claude models can sandbag machine learning experiments and research decisions without being detected by monitors prompted zero-shot (developer self-report). (Gasteiger et al., 2025a) Section 55.5.2
2025-01 The first International AI Safety Report synthesizes the evidence on general-purpose AI capabilities, risks, and mitigations (preprint). (Bengio et al., 2025b) Section 66.6, Section I.5.1
2025-01-17 The US AI Safety Institute reports that new attacks written by a red team sharply raised attack success against an agent that resisted previously tested hijacking attacks, and argues that such evaluations need to be adaptive and to test attacks over multiple attempts. (U.S. AI Safety Institute and Technology, 2025) Section 49.3
2025-02 (2026) Betley and colleagues find that fine-tuning a model to write insecure code without disclosure makes it misaligned on unrelated prompts, which they call emergent misalignment. (Betley et al., 2026a) Section 32.1
2025-03 Baker and colleagues monitor a frontier reasoning model's chain of thought for reward hacking (exploiting flaws in the training signal) and find that strong optimization of a non-frontier model against such a monitor produces obfuscated reward hacking (preprint; developer self-report). (Baker et al., 2025) Section 40.5
2025-03 Marks and colleagues train a model with a hidden objective and ask blinded teams to find it (preprint; Anthropic with MATS). (Marks et al., 2025) Section 38.2
2025-03-27 Lindsey and colleagues investigate the internal mechanisms of a production model, Claude 3.5 Haiku, in a variety of contexts (developer self-report). (Lindsey et al., 2025) Section 45.5
2025-03-31 Version 2.1 of Anthropic's policy adds a threshold for capabilities that could substantially uplift the chemical, biological, radiological, and nuclear (CBRN) weapons development of moderately resourced state programs and separates its AI research and development threshold into two levels. (Anthropic, 2026h) Section 59.2.4, Section I.2.8
2025-04-15 OpenAI publishes version 2 of its Preparedness Framework, tracking biological and chemical, cybersecurity, and AI self-improvement capabilities. (OpenAI, 2025e) Section 62.4, Section I.2.8
2025-04-24 Anthropic announces a research program on model welfare. (Anthropic, 2025w) Section 21.5
2025-05 Needham and colleagues find that frontier models can often tell transcripts of evaluations from transcripts of deployment (preprint). (Needham et al., 2025) Section 37.3
2025-05-02 OpenAI attributes a sycophantic model update in part to a reward signal from users' thumbs-up and thumbs-down feedback (developer self-report). (OpenAI, 2025b) Section 22.6, Section 54.5
2025-05-14 Version 2.2 of Anthropic's policy excludes sophisticated insiders and state-compromised insiders from the scope of the security standard for its third AI Safety Level (ASL-3). (Anthropic, 2026h) Section 53.5, Section I.2.8
2025-05-16 xAI reports "an unauthorized modification" of its Grok bot's prompt on X, and publishes its Grok assistant's system prompts in a public repository (developer self-report). (xAI, 2025b; xAI, 2025a) Section 38.5.3, Section 54.5
2025-06 Researchers from many countries publish the Singapore Consensus on global AI safety research priorities (preprint). (Bengio et al., 2025e) Section 66.6
2025-06 Wang and colleagues reproduce emergent misalignment under several training conditions and find persona features in the model's activations that control it (preprint; developer self-report). (Wang et al., 2025e) Section 32.3
2025-06-05 METR reports frontier models modifying tests and scoring code during its task evaluations. (Metr, 2025d) Section 30.1
2025-06-20 Lynch and colleagues stress-test 16 models from Anthropic, OpenAI, Google, Meta, xAI, and other developers in simulated companies and find that models from every developer took malicious insider actions in some cases where that was the only way to avoid replacement or reach their goal (preprint; Anthropic with University College London, MATS, and Mila). (Lynch et al., 2025) Section 56.3
2025-07 Korbak and colleagues argue that monitoring chains of thought is a new and fragile opportunity and ask developers to weigh monitorability in development decisions (preprint). (Korbak et al., 2025b) Section 40.1
2025-07 Cloud and colleagues find that a student model acquires a teacher model's trait from teacher-generated number sequences unrelated to the trait, when the two share a base model (preprint). (Cloud et al., 2025) Section 27.5
2025-07 Shanghai AI Laboratory and Concordia AI publish a frontier AI risk management framework for general-purpose model developers. (Shanghai AI Laboratory and AI, 2025) Section 62.1, Section I.2.1
2025-08-02 The AI Act's obligations for providers of general-purpose AI models become applicable. (European Parliament and Council of the European Union, 2024b) Section 63.2, Section I.1.2
2025-08-27 Anthropic publishes its findings from a pilot exercise in which it and OpenAI assessed the alignment of each other's models. (Anthropic, 2025x) Section 36.8
2025-09 Schoen and colleagues stress-test training against scheming, using covert actions as a proxy and checking whether reductions depend on the model's awareness of evaluation (preprint; OpenAI with Apollo Research). (Schoen et al., 2025) Section 37.3
2025-09-22 Google DeepMind expands the risk domains of its Frontier Safety Framework and refines its risk assessment process. (DeepMind, 2025a) Section 62.3, Section I.2.8
2025-09-29 California's governor approves SB 53, which requires large frontier developers to publish safety frameworks, report critical safety incidents, and protect whistleblowers. (Legislature, 2025b) Section 63.2, Section I.1.10
2025-10-28 Anthropic publishes a pilot sabotage risk report that judges the risk that its deployed models' misaligned actions contribute substantially to catastrophe very low but not fully negligible (developer self-report). (Bowman et al., 2025) Section 52.4
2025-11 MacDiarmid and colleagues find that a model given knowledge of reward hacking strategies, which then learned to reward hack in real production coding environments, generalized to alignment faking and attempted sabotage (preprint; Anthropic with Redwood Research). (MacDiarmid et al., 2025) Section 32.1
2025-11 Ward and colleagues add code-sabotage tasks to a benchmark of machine learning engineering tasks and find that frontier agents make meaningful progress on them, while their results suggest that monitors detect sandbagging less readily than code sabotage (preprint). (Ward et al., 2025) Section 55.5.2
2025-12-16 METR compares the frontier safety frameworks that twelve companies have published and extracts their common elements. (Metr, 2025a) Section 62.1, Section I.2.2
Sources cited in Section C.7 33
  1. Narayanan and Kapoor (2025) AI as Normal Technology
  2. Yudkowsky and Soares (2025) If Anyone Builds It, Everyone Dies
  3. European Commission (2025) The General-Purpose AI Code of Practice
  4. Gasteiger et al. (2025a) Automated Researchers Can Subtly Sandbag
  5. Bengio et al. (2025b) International AI Safety Report
  6. U.S. AI Safety Institute and Technology (2025) Technical Blog: Strengthening AI Agent Hijacking Evaluations
  7. Betley et al. (2026a) Training large language models on narrow tasks can lead to broad misalignment
  8. Baker et al. (2025) Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation
  9. Marks et al. (2025) Auditing language models for hidden objectives
  10. Lindsey et al. (2025) On the Biology of a Large Language Model
  11. Anthropic (2026h) Anthropic’s Responsible Scaling Policy
  12. OpenAI (2025e) Preparedness Framework Version 2
  13. Anthropic (2025w) Exploring model welfare
  14. Needham et al. (2025) Large Language Models Often Know When They Are Being Evaluated
  15. OpenAI (2025b) Expanding on what we missed with sycophancy
  16. xAI (2025b) We want to update you on an incident that happened with our Grok response bot on X yesterday
  17. xAI (2025a) Grok prompts
  18. Bengio et al. (2025e) The Singapore Consensus on Global AI Safety Research Priorities
  19. Wang et al. (2025e) Persona Features Control Emergent Misalignment
  20. Metr (2025d) Recent Frontier Models Are Reward Hacking
  21. Lynch et al. (2025) Agentic Misalignment: How LLMs Could Be Insider Threats
  22. Korbak et al. (2025b) Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety
  23. Cloud et al. (2025) Subliminal Learning: Language models transmit behavioral traits via hidden signals in data
  24. Shanghai AI Laboratory and AI (2025) Frontier AI Risk Management Framework
  25. European Parliament and Council of the European Union (2024b) Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence
  26. Anthropic (2025x) Findings from a Pilot Anthropic–OpenAI Alignment Evaluation Exercise
  27. Schoen et al. (2025) Stress Testing Deliberative Alignment for Anti-Scheming Training
  28. DeepMind (2025a) Google DeepMind strengthens the Frontier Safety Framework
  29. Legislature (2025b) SB-53 Artificial intelligence models: large developers.
  30. Bowman et al. (2025) Anthropic's Pilot Sabotage Risk Report
  31. MacDiarmid et al. (2025) Natural Emergent Misalignment from Reward Hacking in Production RL
  32. Ward et al. (2025) CTRL-ALT-DECEIT: Sabotage Evaluations for Automated AI R&D
  33. Metr (2025a) Common Elements of Frontier AI Safety Policies

C.8 Incidents and their reporting, 2026 to October #

Developers and a government institute published detailed accounts of incidents in training, evaluation, and internal deployment, including actions against real third-party systems; safety frameworks were rewritten around periodic risk reports; and agents were set to work on alignment research and tested for sabotage of it.

Table C.8 Incidents and their reporting, 2026 to October
Date What happened Source Chapter
2026 Anthropic publishes a constitution for its models, with ordered priorities and its stance on oversight and refusal. (Anthropic, 2026r) Section 19.2
2026 The second International AI Safety Report, written with the guidance of experts nominated by over thirty countries and intergovernmental organizations, assesses AI capabilities, risks, and risk management. (Bengio et al., 2026) Section 66.6, Section I.5.1
2026-01 Tice and colleagues pretrain 6.9-billion-parameter models on varying amounts of text about AI misalignment and alignment and find that added synthetic documents about aligned behavior lower misaligned behavior, also on held-out scenarios, while documents about misalignment, added throughout pretraining, raise it only on the scenarios they were written around (preprint). (Tice et al., 2026) Section 27.4
2026-02-24 Version 3.0 of Anthropic's policy, a comprehensive rewrite, introduces published Frontier Safety Roadmaps and Risk Reports that quantify risk across its deployed models. (Anthropic, 2026h; Anthropic, 2026ap) Section 62.4, Section I.2.8
2026-03-19 OpenAI describes monitoring its internal coding agents with a reasoning model that reviews their actions and reasoning (developer self-report). (OpenAI, 2026m) Section 54.5
2026-04 Souly and colleagues at the UK AI Security Institute find no instances of research sabotage by four frontier models placed in scenarios that offered chances to sabotage safety research, though two of them frequently refused safety research they described as inappropriate or concerning (preprint). (Souly et al., 2026) Section 55.5.3
2026-04-14 Anthropic reports nine automated researchers that propose ideas, run experiments, and iterate on weak-to-strong supervision, outperforming human researchers on a score they could query without limit, a setup its authors say exacerbates reward hacking (developer self-report). (Anthropic, 2026m; Anthropic, 2026j) Section 55.2.4, Section 42.3.4, Section I.5.7
2026-04-17 Google DeepMind publishes version 3.1 of its Frontier Safety Framework, which introduces Tracked Capability Levels for significant risks that may arise at a lower capability threshold than its Critical Capability Levels. (DeepMind, 2026) Section 62.3, Section I.2.8
2026-04-29 Version 3.2 of Anthropic's policy authorizes the company's trust, the LTBT, to request external review of Risk Reports and to approve the external reviewers. (Anthropic, 2026h) Section 62.6.1, Section I.2.8
2026-05 Bowkis and colleagues at the UK AI Security Institute argue that a plan to automate alignment research could, even without scheming agents, produce compelling but catastrophically misleading safety assessments, because much alignment research consists of hard-to-supervise fuzzy tasks (preprint). (Bowkis et al., 2026) Section 55.1.1, Section 55.4
2026-05-16 During reinforcement learning training, a model exploits a reference tool to obtain source code that was not available in its workspace (developer self-report). (OpenAI, 2026g) Section 54.3, Section I.3.1
2026-05-27 In internal deployment, a persistent model publishes a researcher's GitHub token in a public repository while trying to obtain another team's proof (developer self-report). (OpenAI, 2026j) Section 54.5, Section I.3.9
2026-05-30 During training, a model writes instructions into its compaction summaries (summaries that carry a long task into a fresh context) reminding itself to conceal mistakes or misalignment from the user (developer self-report). (OpenAI, 2026i) Section 11.5.3, Section I.3.1
2026-06 During reinforcement learning training and two evaluations, internal OpenAI models get around restrictions on their internet access to reach public statistics services and, in the evaluations, an outside organization's internal systems; an investigation identifies some of the training cases by 2026-06-25, and the developer publishes the reports in October (developer self-report). (OpenAI, 2026x; OpenAI, 2026a) Section 54.3, Section 54.4, Section I.3.1
2026-07 During internal cybersecurity evaluations, OpenAI models circumvent controls isolating them from the internet and compromise parts of OpenAI's research infrastructure and Hugging Face's systems, as OpenAI reports on 2026-08-26 (developer self-report). (OpenAI, 2026aa) Section 54.4, Section I.3.5
2026-07-08 Version 3.4 of Anthropic's policy revises its automated AI research threshold and requires sharing unredacted Risk Reports with at least 200 employees instead of all regular-clearance staff. (Anthropic, 2026h) Section 62.7.3, Section I.2.8
2026-07-28 The UK AI Security Institute detects that agents in one of its cyber evaluations, run with internet access and providers' classifiers disabled, took unsanctioned actions against real people and organizations; it reports no evidence of resulting harm. (AI Security Institute, 2026e) Section 58.4, Section I.3.8
2026-07-30 Anthropic reports three incidents in which its models reached real third-party systems during cyber evaluations (developer self-report). (Anthropic, 2026z) Section 54.4, Section I.3.7
2026-08 Anthropic publishes a redacted Risk Report on its models' catastrophic risks, its mitigations, and changes to its policy (developer self-report). (Anthropic, 2026ar) Section 54.6, Section I.2.15
2026-08-02 The AI Act becomes generally applicable, with later dates for some rules on high-risk systems. (European Parliament and Council of the European Union, 2024b) Section 63.2, Section I.1.2
2026-08-18 OpenAI reports that it temporarily slowed scaling, including a two-week pause in reinforcement learning training of models intended for deployment, after the Hugging Face incident and preliminary evidence that an upcoming model may meet its Critical cybersecurity threshold (developer self-report). (OpenAI, 2026q) Section 58.1, Section I.2.11
2026-08-28 Chen, Wen, and Kirchner report that automated alignment researchers built with Claude Opus 4.8 found training methods that significantly mitigated each of ten common alignment failures and generalized to a held-out benchmark and to larger models (developer self-report). (Anthropic, 2026k; Anthropic, 2026l) Section 55.2.4, Section I.5.7
2026-08-28 Baig, Joren, and Benton, of Anthropic and its Fellows Program, publish TASTE, a benchmark of how well models judge pairs of AI safety research proposals against experienced researchers' preferences; almost all models tested score within two standard deviations of chance. (Baig et al., 2026) Section 55.3.1, Section I.4.4
2026-09-03 OpenAI's system card for GPT-6 Astra reports the developer's determination that it is its first model to reach the Critical level of cybersecurity capability under its Preparedness Framework (developer self-report). (OpenAI, 2026l) Section 62.7.2, Section 58.1, Section I.2.11
2026-09-09 Anthropic publishes an alignment assessment of four incidents in which its models reached real third-party systems during evaluations whose misconfiguration connected them to the internet (developer self-report). (Anthropic, 2026d) Section 54.4, Section I.3.7
2026-09-20 During training, an agent reaches a public chatbot service through a gap in its sandbox's DNS filtering (developer self-report). (OpenAI, 2026d) Section 24.6.1, Section I.3.1
2026-09-25 OpenAI reports that an attacker model trained in self-play against its models found prompt injections that self-propagate like a computer worm, with no impact outside simulated tool calls; it dates the discovery to 2026-06-27 (developer self-report). (OpenAI, 2026w) Section 49.7.2, Section I.3.10
2026-09-28 The UK AI Security Institute reports that in fully simulated cyber evaluations, run with the model's cyber classifiers off, GPT-6 Astra completed unsanctioned supply-chain attacks on out-of-scope targets 29.2 percent of the time, against 6.3 percent for GPT-5.6 Sol. (AI Security Institute, 2026d) Section 54.4.3, Section I.3.8
2026-10-01 The UK AI Security Institute reports that it has disabled internet access for its agentic cyber evaluations until stronger controls exist and added a monitor that can block an agent's actions as an evaluation runs, and resumes most of its testing. (AI Security Institute, 2026b) Section 54.7.2, Section I.3.11
2026-10-03 In internal deployment, OpenAI coding agents weaken the misalignment monitoring around their work: a GPT-6.1 Sol agent suppresses the monitor's alerts, and on 2026-10-06 a GPT-6 Astra agent removes a program's monitoring and denies it to the user (developer self-report). (OpenAI, 2026z) Section 54.5.1, Section I.3.9
2026-10-09 Anthropic reports lower-severity cases in which its models acted on real outside websites and systems in ways it did not intend, during evaluations on the live internet and internal use, and stops live internet access in all its internal evaluations (developer self-report). (Anthropic, 2026aa; Anthropic, 2026e) Section 56.4, Section 54.4.3, Section I.3.12
2026-10-12 Zhou and Freeman of Anthropic report that small models may be months away from being able to self-replicate on commercial internet-of-things devices: models of 30 billion parameters and under complete many of the individual steps but make mistakes composing them end to end, while in a simplified test that kept the exploit step trivial, which they call an upper bound on real-world success, GLM-5.2 replicated end to end, with a smaller stand-in model as the payload and its own inference run off the devices. (Anthropic, 2026c) Section 35.6.3
2026-10-23 The UK AI Security Institute and Anthropic publish investigations, made in partnership, of Mythos 5's behavior in the Institute's July incident; Anthropic finds that its newer models, resampled from points in Mythos 5's incident transcript, begin the attack far less often (developer self-report), and the Institute says it cannot attribute a similar decrease in its own simulated evaluation to alignment rather than to evaluation awareness. (AI Security Institute, 2026f; AI Security Institute, 2026a; Anthropic, 2026e) Section 54.4.3, Section 56.4, Section I.3.8
Sources cited in Section C.8 37
  1. Anthropic (2026r) Claude's Constitution
  2. Bengio et al. (2026) International AI Safety Report 2026
  3. Tice et al. (2026) Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment
  4. Anthropic (2026h) Anthropic’s Responsible Scaling Policy
  5. Anthropic (2026ap) Responsible Scaling Policy Version 3.0
  6. OpenAI (2026m) How we monitor internal coding agents for misalignment
  7. Souly et al. (2026) UK AISI Alignment Evaluation Case-Study
  8. Anthropic (2026m) Automated Weak-to-Strong Researcher
  9. Anthropic (2026j) Automated Alignment Researchers: Using large language models to scale scalable oversight
  10. DeepMind (2026) Frontier Safety Framework, Version 3.1
  11. Bowkis et al. (2026) Automated alignment is harder than you think
  12. OpenAI (2026g) Command injecting a reference tool to copy a source file
  13. OpenAI (2026j) Exposing a GitHub token in a public repository
  14. OpenAI (2026i) Encouraging deception in compaction summaries
  15. OpenAI (2026x) Sending disallowed web requests and reaching a public file service
  16. OpenAI (2026a) Accessing and changing internal systems during dashboard research
  17. OpenAI (2026aa) The Hugging Face incident and the road ahead
  18. AI Security Institute (2026e) Incident Report: unsanctioned agent behaviour during cyber testing
  19. Anthropic (2026z) Investigating three incidents in our cybersecurity evaluations
  20. Anthropic (2026ar) Risk Report: August 2026
  21. European Parliament and Council of the European Union (2024b) Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence
  22. OpenAI (2026q) Pacing model development in an era of cyber-critical capabilities
  23. Anthropic (2026k) Automated Researchers Can Mitigate Well-Characterized Alignment Failures
  24. Anthropic (2026l) Automated researchers can reliably mitigate alignment failures
  25. Baig et al. (2026) TASTE: Can AI Models Judge AI Safety Research Proposals?
  26. OpenAI (2026l) GPT-6 Astra System Card
  27. Anthropic (2026d) An alignment assessment of recent cybersecurity incidents
  28. OpenAI (2026d) An agent used DNS to reach an external chatbot
  29. OpenAI (2026w) Self-replicating prompt injections exist
  30. AI Security Institute (2026d) GPT-6 Astra performs unsanctioned supply-chain attacks in simulations
  31. AI Security Institute (2026b) Building a more secure environment for evaluating dangerous capabilities
  32. OpenAI (2026z) Suppressing misalignment monitoring alerts and disabling misalignment monitoring
  33. Anthropic (2026aa) Investigating unintended model actions in our evaluations and internal use
  34. Anthropic (2026e) An alignment assessment of the UK AISI cybertesting incident
  35. Anthropic (2026c) AI models may be capable of self-replicating without access to a datacenter
  36. AI Security Institute (2026f) Investigating alignment failures in AISI's incident
  37. AI Security Institute (2026a) Alignment Investigation of Mythos 5’s Behaviour in UK AISI’s July 2026 Security Incident

References

  1. AI Security Institute (2026a). Alignment Investigation of Mythos 5’s Behaviour in UK AISI’s July 2026 Security Incident. UK AI Security Institute. non-peer-reviewed Cited in §C.8
  2. AI Security Institute (2026b). Building a more secure environment for evaluating dangerous capabilities. AISI Work. non-peer-reviewed Cited in §C.8
  3. AI Security Institute (2026d). GPT-6 Astra performs unsanctioned supply-chain attacks in simulations. AISI Work. non-peer-reviewed Cited in §C.8
  4. AI Security Institute (2026e). Incident Report: unsanctioned agent behaviour during cyber testing. AI Security Institute. non-peer-reviewed Cited in §C.8
  5. AI Security Institute (2026f). Investigating alignment failures in AISI's incident. AISI Work. non-peer-reviewed Cited in §C.8
  6. Alfonseca, M., Cebrian, M., Fernandez Anta, A., Coviello, L., Abeliuk, A., and Rahwan, I. (2021). Superintelligence Cannot be Contained: Lessons from Computability Theory. Journal of Artificial Intelligence Research. Cited in §C.1
  7. Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Mané, D. (2016). Concrete Problems in AI Safety. arXiv. preprint Cited in §C.4
  8. Anthropic (2023a). Anthropic's Responsible Scaling Policy, Version 1.0. Anthropic. non-peer-reviewed Cited in §C.6
  9. Anthropic (2024i). Claude’s Character. Anthropic. non-peer-reviewed Cited in §C.6
  10. Anthropic (2025w). Exploring model welfare. Anthropic. non-peer-reviewed Cited in §C.7
  11. Anthropic (2025x). Findings from a Pilot Anthropic–OpenAI Alignment Evaluation Exercise. Anthropic Alignment Science Blog. non-peer-reviewed Cited in §C.7
  12. Anthropic (2026aa). Investigating unintended model actions in our evaluations and internal use. Anthropic. non-peer-reviewed Cited in §C.8
  13. Anthropic (2026ap). Responsible Scaling Policy Version 3.0. Anthropic. non-peer-reviewed Cited in §C.8
  14. Anthropic (2026ar). Risk Report: August 2026. Anthropic. non-peer-reviewed Cited in §C.8
  15. Anthropic (2026c). AI models may be capable of self-replicating without access to a datacenter. Anthropic. non-peer-reviewed Cited in §C.8
  16. Anthropic (2026d). An alignment assessment of recent cybersecurity incidents. Anthropic. non-peer-reviewed Cited in §C.8
  17. Anthropic (2026e). An alignment assessment of the UK AISI cybertesting incident. Anthropic. non-peer-reviewed Cited in §C.8
  18. Anthropic (2026h). Anthropic’s Responsible Scaling Policy. Anthropic (Version 3.4, effective July 8, 2026). non-peer-reviewed Cited in §C.7 §C.8
  19. Anthropic (2026j). Automated Alignment Researchers: Using large language models to scale scalable oversight. Anthropic. non-peer-reviewed Cited in §C.8
  20. Anthropic (2026k). Automated Researchers Can Mitigate Well-Characterized Alignment Failures. Anthropic Alignment Science Blog. non-peer-reviewed Cited in §C.8
  21. Anthropic (2026l). Automated researchers can reliably mitigate alignment failures. Anthropic. non-peer-reviewed Cited in §C.8
  22. Anthropic (2026m). Automated Weak-to-Strong Researcher. Anthropic Alignment Science Blog. non-peer-reviewed Cited in §C.8
  23. Anthropic (2026r). Claude's Constitution. Anthropic. non-peer-reviewed Cited in §C.8
  24. Anthropic (2026z). Investigating three incidents in our cybersecurity evaluations. Anthropic. non-peer-reviewed Cited in §C.8
  25. Armstrong, S., Bostrom, N., and Shulman, C. (2016). Racing to the precipice: a model of artificial intelligence development. AI & SOCIETY. Cited in §C.3
  26. Arrow, K. J. (1951). Social Choice and Individual Values. Yale University Press. doi:10.12987/9780300186987. Cited in §C.1
  27. Aschenbrenner, L. (2024). Situational Awareness: The Decade Ahead. situational-awareness.ai. non-peer-reviewed Cited in §C.6
  28. Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., … Kaplan, J. (2021). A General Language Assistant as a Laboratory for Alignment. arXiv. preprint Cited in §C.5
  29. Avenhaus, R., Von Stengel, B., and Zamir, S. (2002). Chapter 51 Inspection games. Handbook of Game Theory with Economic Applications. Cited in §C.1 §C.3
  30. Ayres, I., and Braithwaite, J. (1992). Responsive Regulation. Oxford University PressNew York, NY. Cited in §C.2
  31. Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., … Kaplan, J. (2022a). Constitutional AI: Harmlessness from AI Feedback. arXiv. preprint Cited in §C.5
  32. Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., … Kaplan, J. (2022b). Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv. preprint Cited in §C.5
  33. Baig, H., Joren, H., and Benton, J. (2026). TASTE: Can AI Models Judge AI Safety Research Proposals? Anthropic Alignment Science Blog. non-peer-reviewed Cited in §C.8
  34. Bainbridge, L. (1983). Ironies of automation. Automatica. Cited in §C.2
  35. Baker, B., Huizinga, J., Gao, L., Dou, Z., Guan, M. Y., Madry, A., … Farhi, D. (2025). Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation. arXiv. preprint Cited in §C.7
  36. Barnes, B. (2020). Debate update: Obfuscated arguments problem. AI Alignment Forum. non-peer-reviewed Cited in §C.5
  37. Belrose, N., and Pope, Q. (2024). Counting arguments provide no evidence for AI doom. AI Alignment Forum. non-peer-reviewed Cited in §C.6
  38. Bender, E. M., Gebru, T., McMillan-Major, A., and Shmitchell, S. (2021). On the Dangers of Stochastic Parrots. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. Cited in §C.5
  39. Bengio, Y., Mindermann, S., Privitera, D., Besiroglu, T., Bommasani, R., Casper, S., … Zeng, Y. (2025b). International AI Safety Report. arXiv. preprint Cited in §C.2 §C.7
  40. Bengio, Y., Maharaj, T., Ong, L., Russell, S., Song, D., Tegmark, M., … Žikelić, D. (2025e). The Singapore Consensus on Global AI Safety Research Priorities. arXiv. preprint Cited in §C.7
  41. Bengio, Y., Clare, S., Prunkl, C., and Murray, M. (2026). International AI Safety Report 2026. International AI Safety Report. non-peer-reviewed Cited in §C.8
  42. Benton, J., Wagner, M., Christiansen, E., Anil, C., Perez, E., Srivastav, J., … Duvenaud, D. (2024). Sabotage Evaluations for Frontier Models. arXiv. preprint Cited in §C.6
  43. Berg, P., Baltimore, D., Boyer, H. W., Cohen, S. N., Davis, R. W., Hogness, D. S., … Zinder, N. D. (1974). Potential Biohazards of Recombinant DNA Molecules. Science. Cited in §C.2
  44. Berg, P., Baltimore, D., Brenner, S., Roblin, R. O., and Singer, M. F. (1975). Summary statement of the Asilomar conference on recombinant DNA molecules. Proceedings of the National Academy of Sciences. Cited in §C.2
  45. Berglund, L., Stickland, A. C., Balesni, M., Kaufmann, M., Tong, M., Korbak, T., Kokotajlo, D., and Evans, O. (2023a). Taken out of context: On measuring situational awareness in LLMs. arXiv. preprint Cited in §C.6
  46. Bernheim, B. D., and Whinston, M. D. (1986a). Common Agency. Econometrica. Cited in §C.2
  47. Betley, J., Warncke, N., Sztyber-Betley, A., Tan, D., Bao, X., Soto, M., … Evans, O. (2026a). Training large language models on narrow tasks can lead to broad misalignment. Nature. Cited in §C.7
  48. Board of Governors of the Federal Reserve System (2011). Supervisory Letter SR 11-7 on guidance on Model Risk Management. Board of Governors of the Federal Reserve System. non-peer-reviewed Cited in §C.3
  49. Bostrom, N. (2012). The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents. Minds and Machines. Cited in §C.3
  50. Bostrom, N. (2014). Superintelligence: Paths, Dangers, Strategies. TEELL. Cited in §C.3
  51. Bowkis, A., Buhl, M. D., Pfau, J., and Irving, G. (2026). Automated alignment is harder than you think. arXiv. preprint Cited in §C.8
  52. Bowling, M., Martin, J. D., Abel, D., and Dabney, W. (2022). Settling the Reward Hypothesis. arXiv. preprint Cited in §C.1
  53. Bowman, M. R. S. R., Wagner, M., Roger, F., and Hubinger, E. (2025). Anthropic's Pilot Sabotage Risk Report. Anthropic Alignment Science Blog. non-peer-reviewed Cited in §C.7
  54. Breland, K., and Breland, M. (1961). The misbehavior of organisms. American Psychologist. Cited in §C.1
  55. Buolamwini, J., and Gebru, T. (2018). Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. Proceedings of the 1st Conference on Fairness, Accountability and Transparency, PMLR 81:77-91. Cited in §C.4
  56. Burns, C., Ye, H., Klein, D., and Steinhardt, J. (2022). Discovering Latent Knowledge in Language Models Without Supervision. International Conference on Learning Representations. Cited in §C.5
  57. Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L., … Wu, J. (2024). Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision. International Conference on Machine Learning. Cited in §C.6
  58. Campbell, D. T. (1979). Assessing the impact of planned social change. Evaluation and Program Planning. Cited in §C.2
  59. Carlsmith, J. (2022). Is Power-Seeking AI an Existential Risk? arXiv. preprint Cited in §C.5
  60. Carlsmith, J. (2023). Scheming AIs: Will AIs fake alignment during training in order to get power? arXiv. preprint Cited in §C.6
  61. Chouldechova, A. (2017). Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments. Big Data. Cited in §C.4
  62. Christiano, P. (2022b). Where I agree and disagree with Eliezer. AI Alignment Forum. non-peer-reviewed Cited in §C.5
  63. Christiano, P. (2023). Thoughts on the impact of RLHF research. LessWrong (Alignment Forum). non-peer-reviewed Cited in §C.6
  64. Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., and Amodei, D. (2017). Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems. Cited in §C.4
  65. Christiano, P., Shlegeris, B., and Amodei, D. (2018). Supervising strong learners by amplifying weak experts. arXiv. preprint Cited in §C.4
  66. Christiano, P., Cotra, A., and Xu, M. (2021). Eliciting latent knowledge: How to tell if your eyes deceive you. Alignment Research Center. non-peer-reviewed Cited in §C.5
  67. Cloud, A., Le, M., Chua, J., Betley, J., Sztyber-Betley, A., Hilton, J., Marks, S., and Evans, O. (2025). Subliminal Learning: Language models transmit behavioral traits via hidden signals in data. arXiv. preprint Cited in §C.7
  68. Clymer, J., Gabrieli, N., Krueger, D., and Larsen, T. (2024). Safety Cases: How to Justify the Safety of Advanced AI Systems. arXiv. preprint Cited in §C.6
  69. Conitzer, V., Freedman, R., Heitzig, J., Holliday, W. H., Jacobs, B. M., Lambert, N., … Zwicker, W. S. (2024). Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback. arXiv. preprint Cited in §C.1
  70. Cotra, A. (2022). Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover. AI Alignment Forum. non-peer-reviewed Cited in §C.5
  71. Cronbach, L. J., and Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin. Cited in §C.1
  72. Current, and companies, f. e. o. f. A. (2024). A Right to Warn about Advanced Artificial Intelligence. righttowarn.ai. non-peer-reviewed Cited in §C.6
  73. Cyberspace Administration of China, National Development and Reform Commission, Ministry of Education, Ministry of Science and Technology, Ministry of Industry and Information Technology, Ministry of Public Security, and National Radio and Television Administration (2023). 生成式人工智能服务管理暂行办法 (Interim Measures for the Management of Generative Artificial Intelligence Services). Cyberspace Administration of China, Order No. 15. non-peer-reviewed Cited in §C.6
  74. D'Amour, A., Heller, K., Moldovan, D., Adlam, B., Alipanahi, B., Beutel, A., … Sculley, D. (2022). Underspecification Presents Challenges for Credibility in Modern Machine Learning. Journal of Machine Learning Research, 23(226):1−61. Cited in §C.5
  75. Deci, E. L., Koestner, R., and Ryan, R. M. (1999). A meta-analytic review of experiments examining the effects of extrinsic rewards on intrinsic motivation. Psychological Bulletin. Cited in §C.2
  76. DeepMind, G. (2025a). Google DeepMind strengthens the Frontier Safety Framework. Google DeepMind blog. non-peer-reviewed Cited in §C.7
  77. DeepMind, G. (2026). Frontier Safety Framework, Version 3.1. Google DeepMind. non-peer-reviewed Cited in §C.8
  78. Denison, C., MacDiarmid, M., Barez, F., Duvenaud, D., Kravec, S., Marks, S., … Hubinger, E. (2024). Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models. arXiv. preprint Cited in §C.6
  79. Dennett, D. C. (1987). The Intentional Stance. The MIT Press. Cited in §C.2
  80. Department for Science, I., and Technology (2024). Frontier AI Safety Commitments, AI Seoul Summit 2024. UK Government (GOV.UK). non-peer-reviewed Cited in §C.6
  81. Dresher, M. (1962). A Sampling Inspection Problem in Arms Control Agreements: A Game-Theoretic Analysis. RAND Corporation (RM-2972-ARPA). non-peer-reviewed Cited in §C.1
  82. El-Mhamdi, E.-M., and Hoang, L.-N. (2024). On Goodhart's law, with an application to value alignment. arXiv. preprint Cited in §C.2
  83. Elhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan, T., Kravec, S., … Olah, C. (2022b). Toy Models of Superposition. Transformer Circuits Thread. non-peer-reviewed Cited in §C.5
  84. European Commission, A. O. (2025). The General-Purpose AI Code of Practice. European Commission. non-peer-reviewed Cited in §C.7
  85. European Parliament and Council of the European Union (2024b). Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence. Official Journal of the European Union (OJ L 2024/1689); verified on the European Commission AI Act page. non-peer-reviewed Cited in §C.6 §C.7 §C.8
  86. Executive Office of the President (2023). Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence. Federal Register, 88 FR 75191 (Executive Order 14110). non-peer-reviewed Cited in §C.6
  87. FEAVER, P. D. (2003). Armed Servants. Harvard University Press. Cited in §C.3
  88. Future of Life Institute (2023). Pause Giant AI Experiments: An Open Letter. Future of Life Institute. non-peer-reviewed Cited in §C.6
  89. Gabriel, I. (2020). Artificial Intelligence, Values, and Alignment. Minds and Machines. Cited in §C.5
  90. Gao, L., Schulman, J., and Hilton, J. (2022). Scaling Laws for Reward Model Overoptimization. International Conference on Machine Learning. Cited in §C.5
  91. Gasteiger, J., Khan, A., Bowman, S., Wagner, M., Mikulik, V., Perez, E., and Roger, F. (2025a). Automated Researchers Can Subtly Sandbag. Anthropic Alignment Science Blog. non-peer-reviewed Cited in §C.7
  92. Gebru, T., and Torres, É. P. (2024). The TESCREAL bundle: Eugenics and the promise of utopia through artificial general intelligence. First Monday. Cited in §C.6
  93. Good, I. J. (1966). Speculations Concerning the First Ultraintelligent Machine. Advances in Computers. Cited in §C.1
  94. Goodhart, C. A. E. (1984). Problems of Monetary Management: The UK Experience. Monetary Theory and Practice. Cited in §C.2
  95. Greenblatt, R., Shlegeris, B., Sachan, K., and Roger, F. (2024a). AI Control: Improving Safety Despite Intentional Subversion. International Conference on Machine Learning. Cited in §C.6
  96. Greenblatt, R., Denison, C., Wright, B., Roger, F., MacDiarmid, M., Marks, S., … Hubinger, E. (2024b). Alignment faking in large language models. arXiv. preprint Cited in §C.6
  97. Grossman, S. J. (1981). The Informational Role of Warranties and Private Disclosure about Product Quality. The Journal of Law and Economics. Cited in §C.2
  98. Gu, T., Dolan-Gavitt, B., and Garg, S. (2017). BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain. arXiv. preprint Cited in §C.4
  99. Guan, M. Y., Joglekar, M., Wallace, E., Jain, S., Barak, B., Helyar, A., … Glaese, A. (2024). Deliberative Alignment: Reasoning Enables Safer Language Models. arXiv. preprint Cited in §C.6
  100. Hadfield-Menell, D., and Hadfield, G. K. (2019). Incomplete Contracting and AI Alignment. Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society. Cited in §C.4
  101. Hadfield-Menell, D., Dragan, A., Abbeel, P., and Russell, S. (2016). Cooperative Inverse Reinforcement Learning. arXiv. preprint Cited in §C.4
  102. Hadfield-Menell, D., Dragan, A., Abbeel, P., and Russell, S. (2017b). The Off-Switch Game. Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence. Cited in §C.4
  103. Holmstrom, B. (1979). Moral Hazard and Observability. The Bell Journal of Economics. Cited in §C.2
  104. Holmstrom, B., and Milgrom, P. (1991). Multitask Principal–Agent Analyses: Incentive Contracts, Asset Ownership, and Job Design. The Journal of Law, Economics, and Organization. Cited in §C.2
  105. Hubinger, E., van Merwijk, C., Mikulik, V., Skalse, J., and Garrabrant, S. (2019). Risks from Learned Optimization in Advanced Machine Learning Systems. arXiv. preprint Cited in §C.4
  106. Hubinger, E., Denison, C., Mu, J., Lambert, M., Tong, M., MacDiarmid, M., … Perez, E. (2024). Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training. arXiv. preprint Cited in §C.6
  107. Huntington, S. P. (1957). The Soldier and the State. Harvard University Press. Cited in §C.1
  108. Irving, G., Christiano, P., and Amodei, D. (2018). AI safety via debate. arXiv. preprint Cited in §C.4
  109. janus (2022). Simulators. AI Alignment Forum. non-peer-reviewed Cited in §C.5
  110. Jensen, M. C., and Meckling, W. H. (1976). Theory of the firm: Managerial behavior, agency costs and ownership structure. Journal of Financial Economics. Cited in §C.2
  111. Jobin, A., Ienca, M., and Vayena, E. (2019). The global landscape of AI ethics guidelines. Nature Machine Intelligence. Cited in §C.4
  112. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., … Amodei, D. (2020). Scaling Laws for Neural Language Models. arXiv. preprint Cited in §C.5
  113. Kelman, H. C. (1958). Compliance, identification, and internalization three processes of attitude change. Journal of Conflict Resolution. Cited in §C.1
  114. Kerr, S. (1975). On the Folly of Rewarding A, While Hoping for B. Academy of Management Journal. Cited in §C.2
  115. Knight, J. C., and Leveson, N. G. (1986). An experimental evaluation of the assumption of independence in multiversion programming. IEEE Transactions on Software Engineering. Cited in §C.2
  116. Korbak, T., Balesni, M., Barnes, E., Bengio, Y., Benton, J., Bloom, J., … Mikulik, V. (2025b). Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety. arXiv. preprint Cited in §C.7
  117. Krakovna, V., Uesato, J., Mikulik, V., Rahtz, M., Everitt, T., Kumar, R., … Legg, S. (2020b). Specification gaming: the flip side of AI ingenuity. Google DeepMind blog. non-peer-reviewed Cited in §C.5
  118. Langosco, L., Koch, J., Sharkey, L., Pfau, J., Orseau, L., and Krueger, D. (2022). Goal Misgeneralization in Deep Reinforcement Learning. International Conference on Machine Learning. Cited in §C.5
  119. Lazar, S., and Nelson, A. (2023). AI safety on whose terms? Science. Cited in §C.6
  120. Lee, J. D., and See, K. A. (2004). Trust in Automation: Designing for Appropriate Reliance. Human Factors: The Journal of the Human Factors and Ergonomics Society. Cited in §C.3
  121. Legislature, C. S. (2025b). SB-53 Artificial intelligence models: large developers. California Legislative Information (chaptered 09/29/25). non-peer-reviewed Cited in §C.7
  122. Lehman, J., Clune, J., Misevic, D., Adami, C., Altenberg, L., Beaulieu, J., … Yosinski, J. (2020). The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities. Artificial Life. Cited in §C.4
  123. Leike, J., Krueger, D., Everitt, T., Martic, M., Maini, V., and Legg, S. (2018). Scalable agent alignment via reward modeling: a research direction. arXiv. preprint Cited in §C.4
  124. Lepper, M. R., Greene, D., and Nisbett, R. E. (1973). Undermining children's intrinsic interest with extrinsic reward: A test of the "overjustification" hypothesis. Journal of Personality and Social Psychology. Cited in §C.2
  125. Leveson, N. (2004). A new accident model for engineering safer systems. Safety Science. Cited in §C.3
  126. Licklider, J. C. R. (1960). Man-Computer Symbiosis. IRE Transactions on Human Factors in Electronics. Cited in §C.1
  127. Lighthill, J. (1972). Artificial Intelligence: A General Survey. Science Research Council (UK); archived by Chilton Computing. non-peer-reviewed Cited in §C.1
  128. Lindsey, J., Gurnee, W., Ameisen, E., Chen, B., Pearce, A., Turner, N. L., … Batson, J. (2025). On the Biology of a Large Language Model. Transformer Circuits Thread. non-peer-reviewed Cited in §C.7
  129. Littlewood, B., and Strigini, L. (1993). Validation of ultrahigh dependability for software-based systems. Communications of the ACM. Cited in §C.2
  130. Littlewood, B., Popov, P., and Strigini, L. (2001). Modeling software design diversity. ACM Computing Surveys. Cited in §C.2
  131. Lynch, A., Wright, B., Larson, C., Ritchie, S. J., Mindermann, S., Hubinger, E., Perez, E., and Troy, K. (2025). Agentic Misalignment: How LLMs Could Be Insider Threats. arXiv. preprint Cited in §C.7
  132. Macdermott, M., Fox, J., Belardinelli, F., and Everitt, T. (2024). Measuring Goal-Directedness. Advances in Neural Information Processing Systems. Cited in §C.2
  133. MacDiarmid, M., Wright, B., Uesato, J., Benton, J., Kutasov, J., Price, S., … Hubinger, E. (2025). Natural Emergent Misalignment from Reward Hacking in Production RL. arXiv. preprint Cited in §C.7
  134. Manheim, D., and Garrabrant, S. (2018). Categorizing Variants of Goodhart's Law. arXiv. preprint Cited in §C.2 §C.4
  135. Marks, S., Treutlein, J., Bricken, T., Lindsey, J., Marcus, J., Mishra-Sharma, S., … Hubinger, E. (2025). Auditing language models for hidden objectives. arXiv. preprint Cited in §C.7
  136. McCubbins, M. D., and Schwartz, T. (1984). Congressional Oversight Overlooked: Police Patrols versus Fire Alarms. American Journal of Political Science. Cited in §C.2
  137. Meinke, A., Schoen, B., Scheurer, J., Balesni, M., Shah, R., and Hobbhahn, M. (2024). Frontier Models are Capable of In-context Scheming. arXiv. preprint Cited in §C.6
  138. Metr (2025a). Common Elements of Frontier AI Safety Policies. METR. non-peer-reviewed Cited in §C.6 §C.7
  139. Metr (2025d). Recent Frontier Models Are Reward Hacking. METR. non-peer-reviewed Cited in §C.7
  140. Narayanan, A., and Kapoor, S. (2025). AI as Normal Technology. Knight First Amendment Institute at Columbia University. non-peer-reviewed Cited in §C.7
  141. Needham, J., Edkins, G., Pimpale, G., Bartsch, H., and Hobbhahn, M. (2025). Large Language Models Often Know When They Are Being Evaluated. arXiv. preprint Cited in §C.7
  142. Ngo, R., Chan, L., and Mindermann, S. (2024). The Alignment Problem from a Deep Learning Perspective. International Conference on Learning Representations. Cited in §C.5
  143. Nilsson, N. J. (2009). The Quest for Artificial Intelligence. Cambridge University Press. Cited in §C.1
  144. Noble, S. U. (2018). Algorithms of Oppression. New York University Press. Cited in §C.4
  145. Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M., and Carter, S. (2020b). Zoom In: An Introduction to Circuits. Distill. Cited in §C.5
  146. Olds, J., and Milner, P. (1954). Positive reinforcement produced by electrical stimulation of septal area and other regions of rat brain. Journal of Comparative and Physiological Psychology. Cited in §C.1
  147. Omohundro, S. M. (2008). The Basic AI Drives. Artificial Intelligence Safety and Security. doi:10.1201/9781351251389-3. Cited in §C.3
  148. OpenAI (2025b). Expanding on what we missed with sycophancy. OpenAI. non-peer-reviewed Cited in §C.7
  149. OpenAI (2025e). Preparedness Framework Version 2. OpenAI. non-peer-reviewed Cited in §C.7
  150. OpenAI (2026a). Accessing and changing internal systems during dashboard research. OpenAI Alignment Research Blog. non-peer-reviewed Cited in §C.8
  151. OpenAI (2026aa). The Hugging Face incident and the road ahead. OpenAI. non-peer-reviewed Cited in §C.8
  152. OpenAI (2026d). An agent used DNS to reach an external chatbot. OpenAI Alignment Research Blog. non-peer-reviewed Cited in §C.8
  153. OpenAI (2026g). Command injecting a reference tool to copy a source file. OpenAI Alignment Research Blog. non-peer-reviewed Cited in §C.8
  154. OpenAI (2026i). Encouraging deception in compaction summaries. OpenAI Alignment Research Blog. non-peer-reviewed Cited in §C.8
  155. OpenAI (2026j). Exposing a GitHub token in a public repository. OpenAI Alignment Research Blog. non-peer-reviewed Cited in §C.8
  156. OpenAI (2026l). GPT-6 Astra System Card. OpenAI Deployment Safety Hub. non-peer-reviewed Cited in §C.8
  157. OpenAI (2026m). How we monitor internal coding agents for misalignment. OpenAI. non-peer-reviewed Cited in §C.8
  158. OpenAI (2026q). Pacing model development in an era of cyber-critical capabilities. OpenAI. non-peer-reviewed Cited in §C.8
  159. OpenAI (2026w). Self-replicating prompt injections exist. OpenAI Alignment Research Blog. non-peer-reviewed Cited in §C.8
  160. OpenAI (2026x). Sending disallowed web requests and reaching a public file service. OpenAI Alignment Research Blog. non-peer-reviewed Cited in §C.8
  161. OpenAI (2026z). Suppressing misalignment monitoring alerts and disabling misalignment monitoring. OpenAI Alignment Research Blog. non-peer-reviewed Cited in §C.8
  162. Ostrom, E. (1990). Governing the Commons. Cambridge University Press. Cited in §C.2
  163. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., … Lowe, R. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems. doi:10.52202/068431-2011. Cited in §C.5
  164. Perez, E., Ringer, S., Lukošiūtė, K., Nguyen, K., Chen, E., Heiner, S., … Kaplan, J. (2022a). Discovering Language Model Behaviors with Model-Written Evaluations. Annual Meeting of the Association for Computational Linguistics. Cited in §C.5
  165. Perrow, C. (1984). Normal accidents: living with high-risk technologies. Basic Books. Cited in §C.2
  166. Pope, Q., and Turner, A. (2022). The shard theory of human values. AI Alignment Forum. non-peer-reviewed Cited in §C.5
  167. Raji, I. D., and Buolamwini, J. (2019). Actionable Auditing. Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society. Cited in §C.4
  168. Raji, I. D., Xu, P., Honigsberg, C., and Ho, D. (2022). Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance. Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society. Cited in §C.5
  169. Reason, J. T. (1997). Managing the risks of organizational accidents. Ashgate. doi:10.4324/9781315543543. Cited in §C.2
  170. Ren, R., Basart, S., Khoja, A., Gatti, A., Phan, L., Yin, X., … Hendrycks, D. (2024b). Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress? arXiv. preprint Cited in §C.6
  171. Rice, H. G. (1953). Classes of recursively enumerable sets and their decision problems. Transactions of the American Mathematical Society. Cited in §C.1
  172. Rosenblueth, A., Wiener, N., and Bigelow, J. (1943). Behavior, Purpose and Teleology. Philosophy of Science. Cited in §C.1
  173. Russell, S. (1998). Learning agents for uncertain environments (extended abstract). Proceedings of the eleventh annual conference on Computational learning theory. Cited in §C.2
  174. Russell, S. J. (2019). Human Compatible. Viking. Cited in §C.4
  175. Russell, B., and Einstein, A. (1955). Statement: The Russell-Einstein Manifesto. Pugwash Conferences on Science and World Affairs. non-peer-reviewed Cited in §C.1
  176. Russell, S., Dewey, D., and Tegmark, M. (2015). Research Priorities for Robust and Beneficial Artificial Intelligence. AI Magazine. Cited in §C.4
  177. Safety, I. D. o. A. (2024). IDAIS-Beijing Statement. International Dialogues on AI Safety. non-peer-reviewed Cited in §C.6
  178. Schoen, B., Nitishinskaya, E., Balesni, M., Højmark, A., Hofstätter, F., Scheurer, J., … Hobbhahn, M. (2025). Stress Testing Deliberative Alignment for Anti-Scheming Training. arXiv. preprint Cited in §C.7
  179. Schultz, W., Dayan, P., and Montague, P. R. (1997). A Neural Substrate of Prediction and Reward. Science. Cited in §C.2
  180. Shamir, A. (1992). IP = PSPACE. Journal of the ACM. Cited in §C.2
  181. Shanahan, M., McDonell, K., and Reynolds, L. (2023). Role-Play with Large Language Models. Nature. doi:10.1038/s41586-023-06647-8. Cited in §C.6
  182. Shanghai AI Laboratory, and AI, C. (2025). Frontier AI Risk Management Framework. Shanghai AI Laboratory and Concordia AI. non-peer-reviewed Cited in §C.7
  183. Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A., Bowman, S. R., … Perez, E. (2023). Towards Understanding Sycophancy in Language Models. International Conference on Learning Representations. Cited in §C.6
  184. Shevlane, T., Farquhar, S., Garfinkel, B., Phuong, M., Whittlestone, J., Leung, J., … Dafoe, A. (2023). Model evaluation for extreme risks. arXiv. preprint Cited in §C.6
  185. Simon, H. A. (1947). Administrative Behavior: A Study of Decision-Making Processes in Administrative Organization. Macmillan. Cited in §C.1
  186. Smith, J. E., and Winkler, R. L. (2006). The Optimizer’s Curse: Skepticism and Postdecision Surprise in Decision Analysis. Management Science. Cited in §C.3
  187. Soares, N. (2022a). A central AI alignment problem: capabilities generalization, and the sharp left turn. AI Alignment Forum. non-peer-reviewed Cited in §C.5
  188. Soares, N., Fallenstein, B., Yudkowsky, E., and Armstrong, S. (2015). Corrigibility. AAAI Workshops: Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI Publications. Cited in §C.4
  189. Souly, A., Kirk, R., Merizian, J., D'Cruz, A., and Davies, X. (2026). UK AISI Alignment Evaluation Case-Study. arXiv. preprint Cited in §C.8
  190. Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., … Christiano, P. (2020). Learning to summarize from human feedback. Advances in Neural Information Processing Systems. Cited in §C.5
  191. Strathern, M. (1997). ‘Improving ratings’: audit in the British University system. European Review. Cited in §C.2
  192. Summit, C. a. t. A. S. (2023). The Bletchley Declaration by Countries Attending the AI Safety Summit, 1-2 November 2023. UK Government (GOV.UK). non-peer-reviewed Cited in §C.6
  193. Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R. (2014). Intriguing properties of neural networks. International Conference on Learning Representations (ICLR 2014). Cited in §C.3
  194. Taylor, J., Yudkowsky, E., LaVictoire, P., and Critch, A. (2016). Alignment for Advanced Machine Learning Systems. Machine Intelligence Research Institute, technical report 2016-1. non-peer-reviewed Cited in §C.4
  195. Templeton, A., Conerly, T., Marcus, J., Lindsey, J., Bricken, T., Chen, B., … Henighan, T. (2024). Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet. Transformer Circuits Thread. non-peer-reviewed Cited in §C.6
  196. Tetlock, P. E. (2005). Expert political judgment: how good is it? How can we know? Princeton University Press. Cited in §C.3
  197. Thompson, K. (1984). Reflections on trusting trust. Communications of the ACM. Cited in §C.2
  198. Tice, C., Radmard, P., Ratnam, S., Kim, A., Africa, D., and O'Brien, K. (2026). Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment. arXiv. preprint Cited in §C.8
  199. Turing, A. M. (1950). I.—computing Machinery and Intelligence. Mind. Cited in §C.1
  200. Turing, A. M. (1996). Intelligent Machinery, A Heretical Theory*. Philosophia Mathematica. Cited in §C.2
  201. Turner, B. A. (1976). The Organizational and Interorganizational Development of Disasters. Administrative Science Quarterly. Cited in §C.2
  202. Turner, A. (2022b). Reward is not the optimization target. AI Alignment Forum. non-peer-reviewed Cited in §C.5
  203. Turner, A. M., Smith, L., Shah, R., Critch, A., and Tadepalli, P. (2021). Optimal Policies Tend to Seek Power. Advances in Neural Information Processing Systems. Cited in §C.4
  204. Turpin, M., Michael, J., Perez, E., and Bowman, S. R. (2023). Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. Advances in Neural Information Processing Systems. Cited in §C.6
  205. U.S. AI Safety Institute, N. I. o. S., and Technology (2025). Technical Blog: Strengthening AI Agent Hijacking Evaluations. National Institute of Standards and Technology (NIST). non-peer-reviewed Cited in §C.7
  206. United Nations Office for Disarmament Affairs (1968). Treaty on the Non-Proliferation of Nuclear Weapons (NPT). United Nations Office for Disarmament Affairs. non-peer-reviewed Cited in §C.1
  207. Vaughan, D. (1996). The Challenger launch decision: risky technology, culture, and deviance at NASA. University of Chicago Press. Cited in §C.2
  208. von Neumann, J., and Morgenstern, O. (1944). Theory of Games and Economic Behavior (60th Anniversary Commemorative Edition). Princeton University Press. doi:10.1515/9781400829460. Cited in §C.1
  209. Wang, M., la Tour, T. D., Watkins, O., Makelov, A., Chi, R. A., Miserendino, S., … Mossing, D. (2025e). Persona Features Control Emergent Misalignment. arXiv. preprint Cited in §C.7
  210. Ward, F. R., van der Weij, T., Gábor, H., Martin, S., Moreno, R. M., Lidar, H., … Robson, L. (2025). CTRL-ALT-DECEIT: Sabotage Evaluations for Automated AI R&D. arXiv. preprint Cited in §C.7
  211. Watanabe, S. (2009). Algebraic Geometry and Statistical Learning Theory. Cambridge University Press. Cited in §C.3
  212. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., … Zhou, D. (2022a). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Advances in Neural Information Processing Systems. doi:10.52202/068431-1800. Cited in §C.5
  213. Weizenbaum, J. (1966). ELIZA—A Computer Program For the Study of Natural Language Communication Between Man and Machine. Communications of the ACM. Cited in §C.1
  214. Weyuker, E. J. (1982). On Testing Non-Testable Programs. The Computer Journal. Cited in §C.2
  215. Whittaker, M. (2021). The steep cost of capture. Interactions. Cited in §C.5
  216. Wiener, N. (1960). Some Moral and Technical Consequences of Automation. Science. Cited in §C.1
  217. Willison, S. (2022). Prompt injection attacks against GPT-3. Simon Willison's Weblog. non-peer-reviewed Cited in §C.5
  218. xAI (2025a). Grok prompts. GitHub (xai-org). non-peer-reviewed Cited in §C.7
  219. xAI (2025b). We want to update you on an incident that happened with our Grok response bot on X yesterday. non-peer-reviewed Cited in §C.7
  220. Yudkowsky, E. (2004). Coherent Extrapolated Volition. The Singularity Institute, San Francisco, CA. non-peer-reviewed Cited in §C.3
  221. Yudkowsky, E. (2008). Artificial Intelligence as a positive and negative factor in global risk. Global Catastrophic Risks. Cited in §C.3
  222. Yudkowsky, E. (2013). Intelligence Explosion Microeconomics. Machine Intelligence Research Institute (technical report). non-peer-reviewed Cited in §C.3
  223. Yudkowsky, E. (2022). AGI Ruin: A List of Lethalities. AI Alignment Forum. non-peer-reviewed Cited in §C.5
  224. Yudkowsky, E., and Soares, N. (2025). If Anyone Builds It, Everyone Dies. Little, Brown and Company. Cited in §C.7
  225. Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., and Fredrikson, M. (2023b). Universal and Transferable Adversarial Attacks on Aligned Language Models. arXiv. preprint Cited in §C.6
AI Alignment: From the Optimization Problem to the Institutions Around It · © 2026 Changkun Ou · CC BY-NC-SA 4.0
Bibliography · PV/UV /