AI Alignment
Part I: The Problem

The Problem

In brief

Ideas
  • The part states the book's thesis, that alignment is delegation under an asymmetry of competence and visibility, together with the results that would refute it (§1.7.1).
  • 'Alignment' covers three relationships: training process and signal, deployed system and the people it serves, developer and public. A claim about one needs an argument to transfer (§1.3).
  • The idea runs from the cybernetics of the 1940s to the laboratories and governments of 2026, and the field's agendas are bets on which of the four problems binds (Ch. 2; §3.8.1).
  • Evidence rules come before the evidence: name the relationship and kind of evidence, say who produced it, and call a result settled only when a group without a stake reproduces it (Ch. 5).
Evidence
  • Settled: agents trained with a correct reward can keep their skills and pursue the wrong goal, shown by two connected groups in controlled settings, the second reproducing the first (§1.1.3).
  • Settled: stated probabilities of AI catastrophe are widely spread and shift with how the question is asked, modestly for researchers and by orders of magnitude for the public; the book estimates none (§4.5.3).
  • Settled: models' measured behavior moves with the wording and setting of the prompt; its size in any one setting is reported, not settled (§5.1.1).
Open
  • Crux: do goal attributions predict deployed models' behavior better than the persona account? A preregistered comparison on fixed behaviors would decide it (§1.6).
  • Crux: do failures shown in constructed settings occur when nothing is planted, and how often? Opportunities planted in real traffic would test transfer, and representative samples of unplanted traffic the rate (§5.3.2).

This part sets out the problem before any method is taught. It says what "alignment" asks for and which of three relationships a claim is about (the training process and its signal, the deployed system and the people it serves, the developer and the public); it traces the idea from the cybernetics of the 1940s to the laboratories and governments of 2026; it follows the field's main lines of research from their founding works to the chapters that develop them; it weighs the arguments for and against catastrophic outcomes together with the forecasting evidence; and it states how the field knows what it claims to know.

The part comes first because every later part depends on its distinctions. The four problems the book is organized around (specification, instillation, verification, and maintenance under pressure) are named here as a loop rather than a stack, the thesis that alignment is delegation under an asymmetry of competence and visibility is stated together with the results that could refute it, and the evidence rules (which relationship, what kind of evidence, who produced it, whether anyone has replicated it) are fixed before the evidence arrives. The part serves all four problems, and verification most directly, since its last chapter is about how claims are checked.

It assumes no machine learning and no economics; terms are glossed where they first appear, and the tools behind them are taught in the next part.

Chapters in this part

  1. 1 What Alignment Asks

    What it would mean for an AI system to be aligned, and with whom. The chapter lays out the targets an aligned system could have (instructions, intentions, preferences, interests, values) and the split into outer and inner alignment; sets them on political representation theory's axis between delegate and trustee; names the three relationships the word covers and the several principals who act on one system at once; states the intentional stance as the book's method for describing models as having goals, with the observations that would defeat it and the rival accounts of what a language model is; and ends with the four problems as a loop and the delegation thesis with the results that could refute it.

  2. 2 A History of the Idea

    How the worry that a machine would carry out the purpose put into it, rather than the one its makers meant, was posed, reposed, and turned into a research program: Wiener's warning, Turing's learning machine, Samuel's reply, and Good's intelligence explosion; the philosophical program of the 2000s; the machine learning turn; the laboratories' empirical program and the states that took the question up after 2022; and the fiction the field inherited. The chapter reads each formulation for what it took the machine, the objective, and the principal to be, tells the AI ethics tradition's history beside AI safety's with their disagreements stated, sets the field's own origin story beside its critics' readings, and ends on whether the methods that make models useful can be separated from those that make them safe.

  3. 3 The Main Lines of Research

    The field's main line traced as lineages, each from its founding work to its current form and to the chapters that report its evidence: theory-first programs, learning from feedback and its failures, learned goals and personas, oversight of stronger systems, interpretability, and the methods that assume alignment might fail. The chapter says how the book's spine was chosen and how it tilts toward two laboratories' citations, and ends with the research agendas side by side, each stated as a bet on which problem binds and on the observation that would vindicate or refute it.

  4. 4 Risk

    What could go wrong, how badly, and how sure anyone can be. The chapter sorts risks by the route to harm (mistakes, misuse, misalignment, and structural risk), teaches what a quantitative risk claim contains from the first reactor study and its review, sets out the arguments for and against catastrophic misalignment premise by premise with their authors' estimates marked as forecasts, reports what forecasting research and a tournament of experts and superforecasters show about judgment on such questions, and borrows from climate assessment the separation of confidence from likelihood and the arithmetic of fat tails. It ends where the evidence does not decide, on whether misalignment grows more coherent or less with capability.

  5. 5 How the Field Knows

    The rules of evidence the book applies, and where they come from. The chapter sorts the kinds of evidence by what each can establish; takes construct validity from psychometrics and derives what a benchmark's correlation with general capability leaves for the trait it names; sets out the test of whether a planted failure behaves like a natural one; takes the standards for attributing minds from comparative cognition; works three counter-readings of dramatic results; reads laboratory self-reports through the research on sponsorship; states the replication standard and its reasons; and lists the assumptions under which evidence about current models bears on later ones.

References for Part I

405 works cited across this part's chapters.

  1. Abdalla, M., and Abdalla, M. (2021). The Grey Hoodie Project: Big Tobacco, Big Tech, and the Threat on Academic Integrity. Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society. Ch. 2 Ch. 5
  2. Aguirre, A. (2025). Keep the Future Human: Why and How We Should Close the Gates to AGI and Superintelligence, and What We Should Build Instead. arXiv. preprint Ch. 3
  3. Ahmed, N., Wahed, M., and Thompson, N. C. (2023). The growing influence of industry in AI research. Science. Ch. 5
  4. Ahmed, S., Jaźwińska, K., Ahlawat, A., Winecoff, A., and Wang, M. (2024). Field-building and the epistemic culture of AI safety. First Monday. Ch. 2 Ch. 3 Ch. 5
  5. Alaga, J., and Schuett, J. (2023). Coordinated pausing: An evaluation-based coordination scheme for frontier AI developers. arXiv. preprint Ch. 3
  6. Alain, G., and Bengio, Y. (2016). Understanding intermediate layers using linear classifier probes. arXiv. preprint Ch. 3
  7. Ameisen, E., Lindsey, J., Pearce, A., Gurnee, W., Turner, N. L., Chen, B., … Batson, J. (2025). Circuit Tracing: Revealing Computational Graphs in Language Models. Transformer Circuits Thread. non-peer-reviewed Ch. 3
  8. Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Mané, D. (2016). Concrete Problems in AI Safety. arXiv. preprint Ch. 2 Ch. 3
  9. Anderljung, M., Barnhart, J., Korinek, A., Leung, J., O'Keefe, C., Whittlestone, J., … Wolf, K. (2023). Frontier AI Regulation: Managing Emerging Risks to Public Safety. arXiv. preprint Ch. 3
  10. Andreas, J. (2022). Language Models as Agent Models. Findings of the Association for Computational Linguistics: EMNLP 2022. doi:10.18653/v1/2022.findings-emnlp.423. Ch. 1
  11. Ankeny, R. A., and Leonelli, S. (2011). What’s so special about model organisms? Studies in History and Philosophy of Science Part A. Ch. 5
  12. Anthropic (2023a). Anthropic's Responsible Scaling Policy, Version 1.0. Anthropic. non-peer-reviewed Ch. 2 Ch. 3
  13. Anthropic (2024i). Claude’s Character. Anthropic. non-peer-reviewed Ch. 1 Ch. 3
  14. Anthropic (2024j). How to Replicate and Extend our Alignment Faking Demo. Anthropic Alignment Science Blog. non-peer-reviewed Ch. 5
  15. Anthropic (2025af). Modifying LLM Beliefs with Synthetic Document Finetuning. Anthropic Alignment Science Blog. non-peer-reviewed Ch. 5
  16. Anthropic (2025ah). Open Source Replication of the Auditing Game Model Organism. Anthropic Alignment Science Blog. non-peer-reviewed Ch. 5
  17. Anthropic (2025g). Building and evaluating alignment auditing agents. Anthropic Alignment Science Blog. non-peer-reviewed Ch. 3
  18. Anthropic (2025x). Findings from a Pilot Anthropic–OpenAI Alignment Evaluation Exercise. Anthropic Alignment Science Blog. non-peer-reviewed Ch. 5
  19. Anthropic (2026au). The persona selection model. Anthropic. non-peer-reviewed Ch. 1 Ch. 3
  20. Anthropic (2026aw). Training a Misaligned Reward Seeker. Anthropic Alignment Science Blog. non-peer-reviewed Ch. 4
  21. Anwar, U., Saparov, A., Rando, J., Paleka, D., Turpin, M., Hase, P., … Krueger, D. (2024). Foundational Challenges in Assuring Alignment and Safety of Large Language Models. arXiv. preprint Ch. 3
  22. Apollo Research (2024). We Need A ‘Science of Evals’. Apollo Research. non-peer-reviewed Ch. 5
  23. Apostolakis, G. (1990). The Concept of Probability in Safety Assessments of Technological Systems. Science. Ch. 4
  24. Apostolakis, G. E. (2004). How Useful Is Quantitative Risk Assessment? Risk Analysis. Ch. 4
  25. Armstrong, S., Sotala, K., and Ó hÉigeartaigh, S. S. (2014). The errors, insights and lessons of famous AI predictions – and what they mean for the future. Journal of Experimental & Theoretical Artificial Intelligence. Ch. 2 Ch. 4
  26. Aschenbrenner, L. (2024). Situational Awareness: The Decade Ahead. situational-awareness.ai. non-peer-reviewed Ch. 2
  27. Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., … Kaplan, J. (2021). A General Language Assistant as a Laboratory for Alignment. arXiv. preprint Ch. 1 Ch. 2 Ch. 3
  28. Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., … Kaplan, J. (2022a). Constitutional AI: Harmlessness from AI Feedback. arXiv. preprint Ch. 2 Ch. 3
  29. Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., … Kaplan, J. (2022b). Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv. preprint Ch. 2 Ch. 3
  30. Baker, B., Huizinga, J., Gao, L., Dou, Z., Guan, M. Y., Madry, A., … Farhi, D. (2025). Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation. arXiv. preprint Ch. 1 Ch. 3 Ch. 4 Ch. 5
  31. Bales, A., D'Alessandro, W., and Kirk‐Giannini, C. D. (2024). Artificial Intelligence: Arguments for Catastrophic Risk. Philosophy Compass. Ch. 4
  32. Barnes, B. (2020). Debate update: Obfuscated arguments problem. AI Alignment Forum. non-peer-reviewed Ch. 3
  33. Bean, A. M., Kearns, R. O., Romanou, A., Hafner, F. S., Mayne, H., Batzner, J., … Mahdi, A. (2025). Measuring what Matters: Construct Validity in Large Language Model Benchmarks. Advances in Neural Information Processing Systems 38. Ch. 5
  34. Beard, S., Rowe, T., and Fox, J. (2020). An analysis and evaluation of methods currently used to quantify the likelihood of existential hazards. Futures. Ch. 4
  35. Bekelman, J. E., Li, Y., and Gross, C. P. (2003). Scope and Impact of Financial Conflicts of Interest in Biomedical Research. JAMA. Ch. 5
  36. Belinkov, Y. (2022). Probing Classifiers: Promises, Shortcomings, and Advances. Computational Linguistics. Ch. 3
  37. Belrose, N., and Pope, Q. (2024). Counting arguments provide no evidence for AI doom. AI Alignment Forum. non-peer-reviewed Ch. 3 Ch. 4
  38. Bender, E. M., and Koller, A. (2020). Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Ch. 1
  39. Bender, E. M., Gebru, T., McMillan-Major, A., and Shmitchell, S. (2021). On the Dangers of Stochastic Parrots. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. Ch. 1 Ch. 2
  40. Bengio, Y., Hinton, G., Yao, A., Song, D., Abbeel, P., Darrell, T., … Mindermann, S. (2024b). Managing extreme AI risks amid rapid progress. Science. Ch. 2
  41. Bengio, Y., Mindermann, S., Privitera, D., Besiroglu, T., Bommasani, R., Casper, S., … Zeng, Y. (2025b). International AI Safety Report. arXiv. preprint Ch. 1 Ch. 2 Ch. 4
  42. Bengio, Y., Cohen, M., Fornasiere, D., Ghosn, J., Greiner, P., MacDermott, M., … Williams-King, D. (2025d). Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path? arXiv. preprint Ch. 3
  43. Bengio, Y., Maharaj, T., Ong, L., Russell, S., Song, D., Tegmark, M., … Žikelić, D. (2025e). The Singapore Consensus on Global AI Safety Research Priorities. arXiv. preprint Ch. 2 Ch. 3
  44. Bengio, Y., Clare, S., Prunkl, C., and Murray, M. (2026). International AI Safety Report 2026. International AI Safety Report. non-peer-reviewed Ch. 1 Ch. 2 Ch. 3 Ch. 4
  45. Benton, J., Wagner, M., Christiansen, E., Anil, C., Perez, E., Srivastav, J., … Duvenaud, D. (2024). Sabotage Evaluations for Frontier Models. arXiv. preprint Ch. 3
  46. Berg, P., Baltimore, D., Boyer, H. W., Cohen, S. N., Davis, R. W., Hogness, D. S., … Zinder, N. D. (1974). Potential Biohazards of Recombinant DNA Molecules. Science. Ch. 2
  47. Berg, P., Baltimore, D., Brenner, S., Roblin, R. O., and Singer, M. F. (1975). Summary statement of the Asilomar conference on recombinant DNA molecules. Proceedings of the National Academy of Sciences. Ch. 2
  48. Berglund, L., Stickland, A. C., Balesni, M., Kaufmann, M., Tong, M., Korbak, T., Kokotajlo, D., and Evans, O. (2023a). Taken out of context: On measuring situational awareness in LLMs. arXiv. preprint Ch. 3
  49. Bernheim, B. D., and Whinston, M. D. (1986a). Common Agency. Econometrica. Ch. 1
  50. Betley, J., Warncke, N., Sztyber-Betley, A., Tan, D., Bao, X., Soto, M., … Evans, O. (2026a). Training large language models on narrow tasks can lead to broad misalignment. Nature. Ch. 1 Ch. 3 Ch. 5
  51. Bhatt, A., Rushing, C., Kaufman, A., Tracy, T., Georgiev, V., Matolcsi, D., Khan, A., and Shlegeris, B. (2025). Ctrl-Z: Controlling AI Agents via Resampling. arXiv. preprint Ch. 1 Ch. 3
  52. Birhane, A., Kalluri, P., Card, D., Agnew, W., Dotan, R., and Bao, M. (2022b). The Values Encoded in Machine Learning Research. 2022 ACM Conference on Fairness Accountability and Transparency. Ch. 2
  53. Bornmann, L., Mutz, R., and Daniel, H.-D. (2010). A Reliability-Generalization Study of Journal Peer Reviews: A Multilevel Meta-Analysis of Inter-Rater Reliability and Its Determinants. PLoS ONE. Ch. 5
  54. Bostrom, N. (2012). The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents. Minds and Machines. Ch. 2 Ch. 4
  55. Bostrom, N. (2014). Superintelligence: Paths, Dangers, Strategies. TEELL. Ch. 2
  56. Bostrom, N. (2017). Strategic Implications of Openness in AI Development. Global Policy. Ch. 3
  57. Bostrom, N. (2019). The Vulnerable World Hypothesis. Global Policy. Ch. 4
  58. Bowman, S. R. (2025). Putting up Bumpers. Anthropic Alignment Science Blog. non-peer-reviewed Ch. 3
  59. Bowman, S. R., Hyun, J., Perez, E., Chen, E., Pettit, C., Heiner, S., … Kaplan, J. (2022). Measuring Progress on Scalable Oversight for Large Language Models. arXiv. preprint Ch. 3
  60. Bratman, M. (1987). Intention, Plans, and Practical Reason. Harvard University Press. Ch. 1
  61. Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., Conerly, T., … Olah, C. (2023). Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. Transformer Circuits Thread. non-peer-reviewed Ch. 3 Ch. 5
  62. Brier, G. W. (1950). Verification of Forecasts Expressed in Terms of Probability. Monthly Weather Review. Ch. 4
  63. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., … Amodei, D. (2020). Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems. Ch. 2
  64. Brown-Cohen, J., Irving, G., and Piliouras, G. (2024). Scalable AI Safety via Doubly-Efficient Debate. International Conference on Machine Learning. Ch. 3
  65. Brown-Cohen, J., Irving, G., Piliouras, G., Chen, L., Li, J., and Xun, Z. (2025). Avoiding Obfuscation with Prover-Estimator Debate. arXiv. preprint Ch. 3
  66. Brundage, M., Avin, S., Clark, J., Toner, H., Eckersley, P., Garfinkel, B., … Amodei, D. (2018). The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation. arXiv. preprint Ch. 4
  67. Brundage, M., Avin, S., Wang, J., Belfield, H., Krueger, G., Hadfield, G., … Anderljung, M. (2020). Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims. arXiv. preprint Ch. 3
  68. Brysse, K., Oreskes, N., O’Reilly, J., and Oppenheimer, M. (2013). Climate change prediction: Erring on the side of least drama? Global Environmental Change. Ch. 4
  69. Buckner, C. (2013). Morgan’s Canon, meet Hume’s Dictum: avoiding anthropofabulation in cross-species comparisons. Biology & Philosophy. Ch. 5
  70. Budescu, D. V., Broomell, S., and Por, H.-H. (2009). Improving Communication of Uncertainty in the Reports of the Intergovernmental Panel on Climate Change. Psychological Science. Ch. 4
  71. Buolamwini, J., and Gebru, T. (2018). Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. Proceedings of the 1st Conference on Fairness, Accountability and Transparency, PMLR 81:77-91. Ch. 2
  72. Burke, E. (1774). Speech to the Electors of Bristol. The Founders' Constitution, University of Chicago Press (vol. 1, ch. 13, doc. 7). non-peer-reviewed Ch. 1
  73. Burns, C., Ye, H., Klein, D., and Steinhardt, J. (2022). Discovering Latent Knowledge in Language Models Without Supervision. International Conference on Learning Representations. Ch. 3
  74. Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L., … Wu, J. (2024). Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision. International Conference on Machine Learning. Ch. 3
  75. Campbell, D. T., and Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin. Ch. 5
  76. Carey, R., and Everitt, T. (2023). Human Control: Definitions and Algorithms. Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence (PMLR 216). Ch. 3
  77. Carlsmith, J. (2022). Is Power-Seeking AI an Existential Risk? arXiv. preprint Ch. 4
  78. Carlsmith, J. (2023). Scheming AIs: Will AIs fake alignment during training in order to get power? arXiv. preprint Ch. 3 Ch. 4
  79. Carlson, J. M., and Doyle, J. (1999). Highly optimized tolerance: A mechanism for power laws in designed systems. Physical Review E. Ch. 4
  80. Carroll, M., Dragan, A., Russell, S., and Hadfield-Menell, D. (2022). Estimating and Penalizing Induced Preference Shifts in Recommender Systems. International Conference on Machine Learning. Ch. 1
  81. Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., … Hadfield-Menell, D. (2023). Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback. arXiv. preprint Ch. 3
  82. Casper, S., Ezell, C., Siegmann, C., Kolt, N., Curtis, T. L., Bucknall, B., … Hadfield-Menell, D. (2024a). Black-Box Access is Insufficient for Rigorous AI Audits. The 2024 ACM Conference on Fairness, Accountability, and Transparency. Ch. 1
  83. Cave, S., and Dihal, K. (2019). Hopes and fears for intelligent machines in fiction and reality. Nature Machine Intelligence. Ch. 2
  84. Cave, S., Dihal, K., and Dillon, S. (2020). AI Narratives. Oxford University PressOxford. Ch. 2
  85. Center for AI Safety (2023). Statement on AI Risk. Center for AI Safety. non-peer-reviewed Ch. 2
  86. Chen, Y., Liu, T. X., Shan, Y., and Zhong, S. (2023c). The emergence of economic rationality of GPT. Proceedings of the National Academy of Sciences. Ch. 1
  87. Chouldechova, A. (2017). Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments. Big Data. Ch. 2
  88. Christian, B. (2020). The Alignment Problem. W. W. Norton & Company. Ch. 2
  89. Christiano, P. (2018). Clarifying “AI alignment”. AI Alignment (Medium). non-peer-reviewed Ch. 1
  90. Christiano, P. (2019). What failure looks like. AI Alignment Forum. non-peer-reviewed Ch. 4
  91. Christiano, P. (2022b). Where I agree and disagree with Eliezer. AI Alignment Forum. non-peer-reviewed Ch. 2 Ch. 3 Ch. 4 Ch. 5
  92. Christiano, P. (2023). Thoughts on the impact of RLHF research. LessWrong (Alignment Forum). non-peer-reviewed Ch. 2 Ch. 3
  93. Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., and Amodei, D. (2017). Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems. Ch. 2 Ch. 3
  94. Christiano, P., Shlegeris, B., and Amodei, D. (2018). Supervising strong learners by amplifying weak experts. arXiv. preprint Ch. 3
  95. Christiano, P., Cotra, A., and Xu, M. (2021). Eliciting latent knowledge: How to tell if your eyes deceive you. Alignment Research Center. non-peer-reviewed Ch. 3
  96. Christiano, P., Neyman, E., and Xu, M. (2022). Formalizing the presumption of independence. arXiv. preprint Ch. 3
  97. Cloud, A., Le, M., Chua, J., Betley, J., Sztyber-Betley, A., Hilton, J., Marks, S., and Evans, O. (2025). Subliminal Learning: Language models transmit behavioral traits via hidden signals in data. arXiv. preprint Ch. 1
  98. Clymer, J., Gabrieli, N., Krueger, D., and Larsen, T. (2024). Safety Cases: How to Justify the Safety of Advanced AI Systems. arXiv. preprint Ch. 3
  99. Cogitate Consortium, Ferrante, O., Gorska-Klimowska, U., Henin, S., Hirschhorn, R., Khalaf, A., … Melloni, L. (2025). Adversarial testing of global neuronal workspace and integrated information theories of consciousness. Nature. Ch. 3
  100. Coglianese, C., and Lazer, D. (2003). Management-Based Regulation: Prescribing Private Management to Achieve Public Goals. Law & Society Review. Ch. 1
  101. Cohen, M. K., Hutter, M., and Osborne, M. A. (2022). Advanced artificial agents intervene in the provision of reward. AI Magazine. Ch. 4
  102. Cooke, R. M. (1991). Experts In Uncertainty. Oxford University PressNew York, NY. Ch. 4
  103. Cortes, C., and Lawrence, N. D. (2021). Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment. arXiv. preprint Ch. 5
  104. Cotra, A. (2022). Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover. AI Alignment Forum. non-peer-reviewed Ch. 3 Ch. 4
  105. Critch, A., and Krueger, D. (2020). AI Research Considerations for Human Existential Safety (ARCHES). arXiv. preprint Ch. 1 Ch. 3
  106. Cronbach, L. J., and Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin. Ch. 5
  107. Cunningham, H., Ewart, A., Riggs, L., Huben, R., and Sharkey, L. (2023). Sparse Autoencoders Find Highly Interpretable Features in Language Models. International Conference on Learning Representations. Ch. 3 Ch. 5
  108. Current, and companies, f. e. o. f. A. (2024). A Right to Warn about Advanced Artificial Intelligence. righttowarn.ai. non-peer-reviewed Ch. 2
  109. Cyberspace Administration of China, National Development and Reform Commission, Ministry of Education, Ministry of Science and Technology, Ministry of Industry and Information Technology, Ministry of Public Security, and National Radio and Television Administration (2023). 生成式人工智能服务管理暂行办法 (Interim Measures for the Management of Generative Artificial Intelligence Services). Cyberspace Administration of China, Order No. 15. non-peer-reviewed Ch. 2
  110. D'Amour, A., Heller, K., Moldovan, D., Adlam, B., Alipanahi, B., Beutel, A., … Sculley, D. (2022). Underspecification Presents Challenges for Credibility in Modern Machine Learning. Journal of Machine Learning Research, 23(226):1−61. Ch. 5
  111. Dafoe, A., Hughes, E., Bachrach, Y., Collins, T., McKee, K. R., Leibo, J. Z., Larson, K., and Graepel, T. (2020). Open Problems in Cooperative AI. arXiv. preprint Ch. 3
  112. Dalrymple, D. "., Skalse, J., Bengio, Y., Russell, S., Tegmark, M., Seshia, S., … Tenenbaum, J. (2024). Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems. arXiv. preprint Ch. 3
  113. Davidson, T. (2023a). What a compute-centric framework says about AI takeoff speeds. LessWrong (summary of an Open Philanthropy report). non-peer-reviewed Ch. 4
  114. Davidson, T., Denain, J.-S., Villalobos, P., and Bas, G. (2023). AI capabilities can be significantly improved without expensive retraining. arXiv. preprint Ch. 5
  115. Davidson, T., Finnveden, L., and Hadshar, R. (2025). AI-Enabled Coups: How a Small Group Could Use AI to Seize Power. Forethought. non-peer-reviewed Ch. 1 Ch. 4
  116. DeepMind, G. (2026). Frontier Safety Framework, Version 3.1. Google DeepMind. non-peer-reviewed Ch. 2
  117. Delaney, O., Guest, O., and Williams, Z. (2024). Mapping Technical Safety Research at AI Companies: A literature review and incentives analysis. arXiv. preprint Ch. 3 Ch. 5
  118. Demski, A., and Garrabrant, S. (2019). Embedded Agency. arXiv. preprint Ch. 3
  119. Denison, C., MacDiarmid, M., Barez, F., Duvenaud, D., Kravec, S., Marks, S., … Hubinger, E. (2024). Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models. arXiv. preprint Ch. 3 Ch. 5
  120. Dennett, D. C. (1987). The Intentional Stance. The MIT Press. Ch. 1
  121. Dennett, D. C., and The Journal of Philosophy, I. (1971). Intentional Systems. Journal of Philosophy. Ch. 1
  122. Department for Science, I., and Technology (2024). Frontier AI Safety Commitments, AI Seoul Summit 2024. UK Government (GOV.UK). non-peer-reviewed Ch. 2
  123. Dixit, A., Grossman, G. M., and Helpman, E. (1997). Common Agency and Coordination: General Theory and Application to Government Policy Making. Journal of Political Economy. Ch. 1
  124. Drexler, K. E. (2019). Reframing Superintelligence: Comprehensive AI Services as General Intelligence. Future of Humanity Institute, University of Oxford, Technical Report 2019-1. non-peer-reviewed Ch. 3
  125. Edwards, P. N. (1996). The Closed World: Computers and the Politics of Discourse in Cold War America. Ch. 2
  126. Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., … Olah, C. (2021a). A Mathematical Framework for Transformer Circuits. Transformer Circuits Thread. non-peer-reviewed Ch. 3
  127. Elhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan, T., Kravec, S., … Olah, C. (2022b). Toy Models of Superposition. Transformer Circuits Thread. non-peer-reviewed Ch. 1 Ch. 3 Ch. 5
  128. Engelbart, D. C. (1962). Augmenting Human Intellect: a Conceptual Framework. Defense Technical Information Center. non-peer-reviewed Ch. 3
  129. Engels, J., Baek, D., Kantamneni, S., and Tegmark, M. (2025). Scaling Laws For Scalable Oversight. Advances in Neural Information Processing Systems 38. doi:10.52202/085713-3254. Ch. 3 Ch. 4
  130. Eth, D., and Davidson, T. (2025). Will AI R&D Automation Cause a Software Intelligence Explosion? Forethought. non-peer-reviewed Ch. 4
  131. European Commission, A. O. (2025). The General-Purpose AI Code of Practice. European Commission. non-peer-reviewed Ch. 2
  132. European Parliament and Council of the European Union (2024b). Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence. Official Journal of the European Union (OJ L 2024/1689); verified on the European Commission AI Act page. non-peer-reviewed Ch. 2
  133. Everitt, T., Lea, G., and Hutter, M. (2018). AGI Safety Literature Review. Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence. Ch. 2 Ch. 3
  134. Everitt, T., Hutter, M., Kumar, R., and Krakovna, V. (2021b). Reward tampering problems and solutions in reinforcement learning: a causal influence diagram perspective. Synthese. Ch. 3
  135. Executive Office of the President (2023). Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence. Federal Register, 88 FR 75191 (Executive Order 14110). non-peer-reviewed Ch. 2
  136. Farquhar, S., Varma, V., Lindner, D., Elson, D., Biddulph, C., Goodfellow, I., and Shah, R. (2025). MONA: Myopic Optimization with Non-myopic Approval Can Mitigate Multi-step Reward Hacking. International Conference on Machine Learning. Ch. 3
  137. Farrell, H., Gopnik, A., Shalizi, C., and Evans, J. (2025). Large AI models are cultural and social technologies. Science. Ch. 1
  138. Fenichel Pitkin, H. (1967). The Concept of Representation. University of California Press. Ch. 1
  139. Fickinger, A., Zhuang, S., Hadfield-Menell, D., and Russell, S. (2020a). Multi-Principal Assistance Games. arXiv. preprint Ch. 1 Ch. 3
  140. Firth, R. (1952). Ethical Absolutism and the Ideal Observer. Philosophy and Phenomenological Research. Ch. 2
  141. Fischhoff, B., Slovic, P., Lichtenstein, S., Read, S., and Combs, B. (1978). How safe is safe enough? A psychometric study of attitudes towards technological risks and benefits. Policy Sciences. Ch. 4
  142. Fish, S., Gölz, P., Parkes, D. C., Procaccia, A. D., Rusak, G., Shapira, I., and Wüthrich, M. (2023). Generative Social Choice. arXiv. preprint Ch. 1
  143. Future of Life Institute (2023). Pause Giant AI Experiments: An Open Letter. Future of Life Institute. non-peer-reviewed Ch. 2 Ch. 3
  144. Gabriel, I. (2020). Artificial Intelligence, Values, and Alignment. Minds and Machines. Ch. 1
  145. Gabriel, I., Manzini, A., Keeling, G., Hendricks, L. A., Rieser, V., Iqbal, H., … Manyika, J. (2024). The Ethics of Advanced AI Assistants. arXiv. preprint Ch. 1
  146. Gallow, J. D. (2025). Instrumental divergence. Philosophical Studies. Ch. 4
  147. Ganguli, D., Lovitt, L., Kernion, J., Askell, A., Bai, Y., Kadavath, S., … Clark, J. (2022). Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned. arXiv. preprint Ch. 3
  148. Gao, L., Schulman, J., and Hilton, J. (2022). Scaling Laws for Reward Model Overoptimization. International Conference on Machine Learning. Ch. 3
  149. Garrabrant, S., Benson-Tilsen, T., Critch, A., Soares, N., and Taylor, J. (2016). Logical Induction. arXiv. preprint Ch. 3
  150. Gebru, T., and Torres, É. P. (2024). The TESCREAL bundle: Eugenics and the promise of utopia through artificial general intelligence. First Monday. Ch. 2 Ch. 4
  151. Geiger, A., Ibeling, D., Zur, A., Chaudhary, M., Chauhan, S., Huang, J., … Icard, T. (2025). Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretability. Journal of Machine Learning Research, 26(83):1−64. Ch. 3 Ch. 5
  152. Goldstein, S., and Kirk-Giannini, C. D. (2025b). Language agents reduce the risk of existential catastrophe. AI & SOCIETY. Ch. 4
  153. Good, I. J. (1966). Speculations Concerning the First Ultraintelligent Machine. Advances in Computers. Ch. 2
  154. Grace, K. (2015). The Asilomar Conference: A Case Study in Risk Mitigation. Machine Intelligence Research Institute, Technical report 2015-9. non-peer-reviewed Ch. 2
  155. Grace, K. (2022). Let’s think about slowing down AI. LessWrong (Alignment Forum). non-peer-reviewed Ch. 3
  156. Grace, K., Salvatier, J., Dafoe, A., Zhang, B., and Evans, O. (2018). Viewpoint: When Will AI Exceed Human Performance? Evidence from AI Experts. Journal of Artificial Intelligence Research. Ch. 4
  157. Grace, K., Stewart, H., Sandkühler, J. F., Thomas, S., Weinstein-Raun, B., Brauner, J., and Korzekwa, R. C. (2024). Thousands of AI Authors on the Future of AI. arXiv. preprint Ch. 4
  158. Greenblatt, R., and Shlegeris, B. (2024). The case for ensuring that powerful AIs are controlled. LessWrong / AI Alignment Forum. non-peer-reviewed Ch. 3
  159. Greenblatt, R., Shlegeris, B., Sachan, K., and Roger, F. (2024a). AI Control: Improving Safety Despite Intentional Subversion. International Conference on Machine Learning. Ch. 3
  160. Greenblatt, R., Denison, C., Wright, B., Roger, F., MacDiarmid, M., Marks, S., … Hubinger, E. (2024b). Alignment faking in large language models. arXiv. preprint Ch. 2 Ch. 3 Ch. 5
  161. Greenblatt, R., Roger, F., Krasheninnikov, D., and Krueger, D. (2024c). Stress-Testing Capability Elicitation With Password-Locked Models. Advances in Neural Information Processing Systems. Ch. 5
  162. Grossman, S. J. (1981). The Informational Role of Warranties and Private Disclosure about Product Quality. The Journal of Law and Economics. Ch. 5
  163. Guan, M. Y., Joglekar, M., Wallace, E., Jain, S., Barak, B., Helyar, A., … Glaese, A. (2024). Deliberative Alignment: Reasoning Enables Safer Language Models. arXiv. preprint Ch. 3
  164. Gunjal, A., Wang, A., Lau, E., Nath, V., He, Y., Liu, B., and Hendryx, S. (2026). Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains. International Conference on Learning Representations. Ch. 1
  165. Haas, P. M. (1992). Introduction: epistemic communities and international policy coordination. International Organization. Ch. 2
  166. Hadfield-Menell, D., and Hadfield, G. K. (2019). Incomplete Contracting and AI Alignment. Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society. Ch. 1
  167. Hadfield-Menell, D., Dragan, A., Abbeel, P., and Russell, S. (2016). Cooperative Inverse Reinforcement Learning. arXiv. preprint Ch. 2 Ch. 3
  168. Hadfield-Menell, D., Milli, S., Abbeel, P., Russell, S., and Dragan, A. (2017a). Inverse Reward Design. Advances in Neural Information Processing Systems. Ch. 3
  169. Hadfield-Menell, D., Dragan, A., Abbeel, P., and Russell, S. (2017b). The Off-Switch Game. Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence. Ch. 3
  170. Hägele, A., Gema, A. P., Sleight, H., Perez, E., and Sohl-Dickstein, J. (2026). The Hot Mess of AI: How Does Misalignment Scale With Model Intelligence and Task Complexity? arXiv. preprint Ch. 1 Ch. 4
  171. Hagendorff, T. (2020). The Ethics of AI Ethics: An Evaluation of Guidelines. Minds and Machines. Ch. 2
  172. Hagendorff, T., Dasgupta, I., Binz, M., Chan, S. C. Y., Lampinen, A., Wang, J. X., Akata, Z., and Schulz, E. (2023). Machine Psychology. arXiv. preprint Ch. 5
  173. Halawi, D., Zhang, F., Yueh-Han, C., and Steinhardt, J. (2024a). Approaching Human-Level Forecasting with Language Models. Advances in Neural Information Processing Systems. Ch. 4
  174. Haldane, A. G., and May, R. M. (2011). Systemic risk in banking ecosystems. Nature. Ch. 4
  175. Hammond, L., Chan, A., Clifton, J., Hoelscher-Obermaier, J., Khan, A., McLean, E., … Rahwan, I. (2025). Multi-Agent Risks from Advanced AI. arXiv. preprint Ch. 3 Ch. 4
  176. Hare, B., Call, J., and Tomasello, M. (2001). Do chimpanzees know what conspecifics know? Animal Behaviour. Ch. 5
  177. Hazell, L., and Shakir, S. A. W. (2006). Under-Reporting of Adverse Drug Reactions. Drug Safety. Ch. 5
  178. Heap, T., Lawson, T., Farnik, L., and Aitchison, L. (2025). Automated Interpretability Metrics Do Not Distinguish Trained and Random Transformers. arXiv. preprint Ch. 3
  179. Hendrycks, D. (2023). Natural Selection Favors AIs over Humans. arXiv. preprint Ch. 4
  180. Hendrycks, D. (2024). Introduction to AI Safety, Ethics, and Society. CRC Press. Ch. 3
  181. Hendrycks, D., and Hiscott, L. (2025). The Misguided Quest for Mechanistic AI Interpretability. AI Frontiers. non-peer-reviewed Ch. 3 Ch. 4
  182. Hendrycks, D., Carlini, N., Schulman, J., and Steinhardt, J. (2021c). Unsolved Problems in ML Safety. arXiv. preprint Ch. 2
  183. Hendrycks, D., Mazeika, M., and Woodside, T. (2023). An Overview of Catastrophic AI Risks. arXiv. preprint Ch. 4
  184. Hernán, M. A., and Robins, J. M. (2016). Using Big Data to Emulate a Target Trial When a Randomized Trial Is Not Available: Table 1. American Journal of Epidemiology. Ch. 5
  185. Heyes, C. M. (1998). Theory of mind in nonhuman primates. Behavioral and Brain Sciences. Ch. 1 Ch. 5
  186. Hill, A. B. (1965). The Environment and Disease: Association or Causation? Proceedings of the Royal Society of Medicine. Ch. 5
  187. Hilton, J. (2024a). A bird's eye view of ARC's research. Alignment Research Center blog. non-peer-reviewed Ch. 3
  188. Højmark, A., Scheurer, J., Nitishinskaya, E., Hofstätter, F., Wolfe, J., Ehrenborg, T., Schoen, B., and Meinke, A. (2026). Measuring Reward-Seeking via Contrastive Belief Updates. arXiv. preprint Ch. 4
  189. Holmstrom, B., and Milgrom, P. (1991). Multitask Principal–Agent Analyses: Incentive Contracts, Asset Ownership, and Job Design. The Journal of Law, Economics, and Organization. Ch. 1
  190. Howe, N., McKenzie, I., Hollinsworth, O., Zajac, M., Tseng, T., Tucker, A., Bacon, P.-L., and Gleave, A. (2025). Scaling Trends in Language Model Robustness. International Conference on Machine Learning. Ch. 5
  191. Huang, S., Siddarth, D., Lovitt, L., Liao, T. I., Durmus, E., Tamkin, A., and Ganguli, D. (2024a). Collective Constitutional AI: Aligning a Language Model with Public Input. The 2024 ACM Conference on Fairness, Accountability, and Transparency. Ch. 1 Ch. 3
  192. Hubinger, E. (2020). An overview of 11 proposals for building safe advanced AI. arXiv. preprint Ch. 3
  193. Hubinger, E. (2022). How likely is deceptive alignment? AI Alignment Forum. non-peer-reviewed Ch. 3
  194. Hubinger, E. (2025). Alignment remains a hard, unsolved problem. AI Alignment Forum. non-peer-reviewed Ch. 3
  195. Hubinger, E., van Merwijk, C., Mikulik, V., Skalse, J., and Garrabrant, S. (2019). Risks from Learned Optimization in Advanced Machine Learning Systems. arXiv. preprint Ch. 1 Ch. 2 Ch. 3
  196. Hubinger, E., Denison, C., Mu, J., Lambert, M., Tong, M., MacDiarmid, M., … Perez, E. (2024). Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training. arXiv. preprint Ch. 2 Ch. 3 Ch. 5
  197. Hughes, J., Sheshadri, A., Khan, A., and Roger, F. (2025). Alignment Faking Revisited: Improved Classifiers and Open Source Extensions. Anthropic Alignment Science Blog. non-peer-reviewed Ch. 5
  198. Humlum, A., and Vestergaard, E. (2025). Large Language Models, Small Labor Market Effects. Elsevier BV. preprint Ch. 4
  199. Irving, G., Christiano, P., and Amodei, D. (2018). AI safety via debate. arXiv. preprint Ch. 3
  200. janus (2022). Simulators. AI Alignment Forum. non-peer-reviewed Ch. 1 Ch. 2 Ch. 3
  201. Jasanoff, S., Saha, K., and Hurlbut, J. B. (2015). CRISPR Democracy: Gene Editing and the Need for Inclusive Deliberation. Issues in Science and Technology. non-peer-reviewed Ch. 2
  202. Jensen, M. C., and Meckling, W. H. (1976). Theory of the firm: Managerial behavior, agency costs and ownership structure. Journal of Financial Economics. Ch. 1
  203. Ji, J., Qiu, T., Chen, B., Zhang, B., Lou, H., Wang, K., … Gao, W. (2023). AI Alignment: A Comprehensive Survey. arXiv. preprint Ch. 2 Ch. 3
  204. Jobin, A., Ienca, M., and Vayena, E. (2019). The global landscape of AI ethics guidelines. Nature Machine Intelligence. Ch. 2
  205. Jones, C. I. (2024). The AI Dilemma: Growth versus Existential Risk. American Economic Review: Insights. Ch. 4
  206. Kahneman, D., and Klein, G. (2009). Conditions for intuitive expertise: A failure to disagree. American Psychologist. Ch. 4
  207. Kantamneni, S., Engels, J., Rajamanoharan, S., Tegmark, M., and Nanda, N. (2025). Are Sparse Autoencoders Useful? A Case Study in Sparse Probing. International Conference on Machine Learning. Ch. 3
  208. Kaplan, S., and Garrick, B. J. (1981). On The Quantitative Definition of Risk. Risk Analysis. Ch. 4
  209. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., … Amodei, D. (2020). Scaling Laws for Neural Language Models. arXiv. preprint Ch. 2
  210. Kapoor, S., Bommasani, R., Klyman, K., Longpre, S., Ramaswami, A., Cihon, P., … Narayanan, A. (2024b). On the Societal Impact of Open Foundation Models. arXiv. preprint Ch. 3
  211. Karger, E., Rosenberg, J., Jacobs, Z., Hickman, M., Hadshar, R., Gamin, K., … Tetlock, P. E. (2023). Forecasting Existential Risks: Evidence from a Long-Run Forecasting Tournament. Forecasting Research Institute, Working Paper #1. working paper Ch. 4
  212. Karger, E., Bastani, H., Yueh-Han, C., Jacobs, Z., Halawi, D., Zhang, F., and Tetlock, P. E. (2024). ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities. arXiv. preprint Ch. 4
  213. Karger, E., Rosenberg, J., Jacobs, Z., Hickman, M., and Tetlock, P. E. (2025). Subjective-probability forecasts of existential risk: Initial results from a hybrid persuasion-forecasting tournament. International Journal of Forecasting. Ch. 4
  214. Kasirzadeh, A. (2025). Two types of AI existential risk: decisive and accumulative. Philosophical Studies. Ch. 4
  215. Kenton, Z., Everitt, T., Weidinger, L., Gabriel, I., Mikulik, V., and Irving, G. (2021). Alignment of Language Agents. arXiv. preprint Ch. 1
  216. Kenton, Z., Kumar, R., Farquhar, S., Richens, J., MacDermott, M., and Everitt, T. (2023). Discovering agents. Artificial Intelligence. Ch. 1
  217. Khan, A., Casper, S., and Hadfield-Menell, D. (2025). Randomness, Not Representation: The Unreliability of Evaluating Cultural Alignment in LLMs. Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency. Ch. 1 Ch. 5
  218. Kirchner, J. H., Chen, Y., Edwards, H., Leike, J., McAleese, N., and Burda, Y. (2024). Prover-Verifier Games improve legibility of LLM outputs. arXiv. preprint Ch. 3
  219. Kokotajlo, D., Alexander, S., Larsen, T., Lifland, E., and Dean, R. (2025). AI 2027. AI Futures Project (ai-2027.com). non-peer-reviewed Ch. 4
  220. Kolt, N. (2024). Governing AI Agents. Elsevier BV. preprint Ch. 1
  221. Korbak, T., Perez, E., and Buckley, C. (2022). RL with KL penalties is better viewed as Bayesian inference. Findings of the Association for Computational Linguistics: EMNLP 2022. doi:10.18653/v1/2022.findings-emnlp.77. Ch. 3
  222. Korbak, T., Balesni, M., Barnes, E., Bengio, Y., Benton, J., Bloom, J., … Mikulik, V. (2025b). Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety. arXiv. preprint Ch. 3 Ch. 4
  223. Korbak, T., Balesni, M., Shlegeris, B., and Irving, G. (2025c). How to evaluate control measures for LLM agents? A trajectory from today to superintelligence. arXiv. preprint Ch. 3
  224. Korinek, A., and Balwit, A. (2023). Aligned with Whom? The Oxford Handbook of AI Governance. Ch. 1
  225. Krakovna, V., Uesato, J., Mikulik, V., Rahtz, M., Everitt, T., Kumar, R., … Legg, S. (2020b). Specification gaming: the flip side of AI ingenuity. Google DeepMind blog. non-peer-reviewed Ch. 1 Ch. 2 Ch. 3
  226. Krakovna, V., Varma, V., Kumar, R., and Phuong, M. (2022). Refining the Sharp Left Turn threat model, part 1: claims and mechanisms. AI Alignment Forum. non-peer-reviewed Ch. 4
  227. Krimsky, S. (1982). Genetic Alchemy: The Social History of the Recombinant DNA Controversy. Ch. 2
  228. Kulveit, J., Douglas, R., Ammann, N., Turan, D., Krueger, D., and Duvenaud, D. (2025). Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development. arXiv. preprint Ch. 4
  229. Kundu, S., Bai, Y., Kadavath, S., Askell, A., Callahan, A., Chen, A., … Kaplan, J. (2023). Specific versus General Principles for Constitutional AI. arXiv. preprint Ch. 3
  230. Kwa, T., Thomas, D., and Garriga-Alonso, A. (2024). Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification. Advances in Neural Information Processing Systems. doi:10.52202/079017-0466. Ch. 4
  231. Kwa, T., West, B., Becker, J., Deng, A., Garcia, K., Hasin, M., … Chan, L. (2025). Measuring AI Ability to Complete Long Software Tasks. Advances in Neural Information Processing Systems 38. Ch. 3 Ch. 4 Ch. 5
  232. Laine, R., Chughtai, B., Betley, J., Hariharan, K., Scheurer, J., Balesni, M., … Evans, O. (2024). Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMs. Advances in Neural Information Processing Systems. Ch. 3
  233. Lakatos, I. (1970). Falsification and the Methodology of Scientific Research Programmes. Criticism and the Growth of Knowledge. Ch. 3
  234. Langosco, L., Koch, J., Sharkey, L., Pfau, J., Orseau, L., and Krueger, D. (2022). Goal Misgeneralization in Deep Reinforcement Learning. International Conference on Machine Learning. Ch. 1 Ch. 3
  235. Lazar, S., and Nelson, A. (2023). AI safety on whose terms? Science. Ch. 1 Ch. 2 Ch. 3 Ch. 4
  236. Legislature, C. S. (2025b). SB-53 Artificial intelligence models: large developers. California Legislative Information (chaptered 09/29/25). non-peer-reviewed Ch. 2
  237. Lehman, J., Clune, J., Misevic, D., Adami, C., Altenberg, L., Beaulieu, J., … Yosinski, J. (2020). The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities. Artificial Life. Ch. 2
  238. Leike, J., Martic, M., Krakovna, V., Ortega, P. A., Everitt, T., Lefrancq, A., Orseau, L., and Legg, S. (2017). AI Safety Gridworlds. arXiv. preprint Ch. 2 Ch. 3
  239. Leike, J., Krueger, D., Everitt, T., Martic, M., Maini, V., and Legg, S. (2018). Scalable agent alignment via reward modeling: a research direction. arXiv. preprint Ch. 2 Ch. 3
  240. Lewis, H., Budnitz, R., Kouts, H., Loewenstein, W., Rowe, W., von Hippel, F., and Zachariasen, F. (1978). Risk Assessment Review Group report to the U. S. Nuclear Regulatory Commission. [PWR; BWR]. Office of Scientific and Technical Information (OSTI). non-peer-reviewed Ch. 4
  241. Lichtenstein, S., Fischhoff, B., and Phillips, L. D. (1982). Calibration of probabilities: The state of the art to 1980. Judgment under Uncertainty. Ch. 4
  242. Licklider, J. C. R. (1960). Man-Computer Symbiosis. IRE Transactions on Human Factors in Electronics. Ch. 2 Ch. 3
  243. Lighthill, J. (1972). Artificial Intelligence: A General Survey. Science Research Council (UK); archived by Chilton Computing. non-peer-reviewed Ch. 2
  244. Lindsey, J., Gurnee, W., Ameisen, E., Chen, B., Pearce, A., Turner, N. L., … Batson, J. (2025). On the Biology of a Large Language Model. Transformer Circuits Thread. non-peer-reviewed Ch. 3
  245. List, C., and Pettit, P. (2011). Group Agency. Oxford University Press. Ch. 1
  246. Longpre, S., Kapoor, S., Klyman, K., Ramaswami, A., Bommasani, R., Blili-Hamelin, B., … Henderson, P. (2024a). A Safe Harbor for AI Evaluation and Red Teaming. arXiv. preprint Ch. 5
  247. Longpre, S., Klyman, K., Appel, R. E., Kapoor, S., Bommasani, R., Sahar, M., … Narayanan, A. (2025). In-House Evaluation Is Not Enough: Towards Robust Third-Party Flaw Disclosure for General-Purpose AI. arXiv. preprint Ch. 5
  248. Lundh, A., Lexchin, J., Mintzes, B., Schroll, J. B., and Bero, L. (2017). Industry sponsorship and research outcome. Cochrane Database of Systematic Reviews. Ch. 5
  249. Lynch, A., Wright, B., Larson, C., Ritchie, S. J., Mindermann, S., Hubinger, E., Perez, E., and Troy, K. (2025). Agentic Misalignment: How LLMs Could Be Insider Threats. arXiv. preprint Ch. 5
  250. Macdermott, M., Fox, J., Belardinelli, F., and Everitt, T. (2024). Measuring Goal-Directedness. Advances in Neural Information Processing Systems. Ch. 1
  251. MacDiarmid, M., Wright, B., Uesato, J., Benton, J., Kutasov, J., Price, S., … Hubinger, E. (2025). Natural Emergent Misalignment from Reward Hacking in Production RL. arXiv. preprint Ch. 3 Ch. 4
  252. Mahowald, K., Ivanova, A. A., Blank, I. A., Kanwisher, N., Tenenbaum, J. B., and Fedorenko, E. (2024). Dissociating language and thought in large language models. Trends in Cognitive Sciences. Ch. 1
  253. Maiya, S., Bartsch, H., Lambert, N., and Hubinger, E. (2025). Open Character Training: Shaping the Persona of AI Assistants through Constitutional AI. arXiv. preprint Ch. 3 Ch. 4
  254. Manheim, D., and Garrabrant, S. (2018). Categorizing Variants of Goodhart's Law. arXiv. preprint Ch. 3
  255. Mansbridge, J. (2003). Rethinking Representation. American Political Science Review. Ch. 1
  256. Mansbridge, J. (2009). A “Selection Model” of Political Representation*. Journal of Political Philosophy. Ch. 1
  257. Marks, S., and Tegmark, M. (2023). The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets. arXiv. preprint Ch. 3
  258. Marks, S., Treutlein, J., Bricken, T., Lindsey, J., Marcus, J., Mishra-Sharma, S., … Hubinger, E. (2025). Auditing language models for hidden objectives. arXiv. preprint Ch. 3 Ch. 4 Ch. 5
  259. Martimort, D., and Stole2, L. (2002). The Revelation and Delegation Principles in Common Agency Games. Econometrica. Ch. 1
  260. Maskin, E., and Tirole, J. (2004). The Politician and the Judge: Accountability in Government. American Economic Review. Ch. 1
  261. Mastrandrea, M. D., Field, C. B., Stocker, T. F., Edenhofer, O., Ebi, K. L., Frame, D. J., … Zwiers, F. W. (2010). Guidance Note for Lead Authors of the IPCC Fifth Assessment Report on Consistent Treatment of Uncertainties. Intergovernmental Panel on Climate Change. non-peer-reviewed Ch. 4
  262. Mastrandrea, M. D., Mach, K. J., Plattner, G.-K., Edenhofer, O., Stocker, T. F., Field, C. B., Ebi, K. L., and Matschoss, P. R. (2011). The IPCC AR5 guidance note on consistent treatment of uncertainties: a common approach across the working groups. Climatic Change. Ch. 4
  263. Mazeika, M., Yin, X., Tamirisa, R., Lim, J., Lee, B. W., Ren, R., … Hendrycks, D. (2025). Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs. Advances in Neural Information Processing Systems 38. Ch. 1
  264. McCambridge, J., Witton, J., and Elbourne, D. R. (2014). Systematic review of the Hawthorne effect: New concepts are needed to study research participation effects. Journal of Clinical Epidemiology. Ch. 5
  265. McCubbins, M. D., and Schwartz, T. (1984). Congressional Oversight Overlooked: Police Patrols versus Fire Alarms. American Journal of Political Science. Ch. 1
  266. Meehl, P. E. (1954). Clinical versus statistical prediction: A theoretical analysis and a review of the evidence. University of Minnesota Press. Ch. 4
  267. Meinke, A., Schoen, B., Scheurer, J., Balesni, M., Shah, R., and Hobbhahn, M. (2024). Frontier Models are Capable of In-context Scheming. arXiv. preprint Ch. 3
  268. Mellers, B., Ungar, L., Baron, J., Ramos, J., Gurcay, B., Fincher, K., … Tetlock, P. E. (2014). Psychological Strategies for Winning a Geopolitical Forecasting Tournament. Psychological Science. Ch. 4
  269. Mellers, B., Stone, E., Murray, T., Minster, A., Rohrbaugh, N., Bishop, M., … Tetlock, P. (2015). Identifying and Cultivating Superforecasters as a Method of Improving Probabilistic Predictions. Perspectives on Psychological Science. Ch. 4
  270. Méloux, M., Maniu, S., Portet, F., and Peyrard, M. (2025). Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable? International Conference on Learning Representations. Ch. 3 Ch. 5
  271. Meng, K., Bau, D., Andonian, A., and Belinkov, Y. (2022). Locating and Editing Factual Associations in GPT. Advances in Neural Information Processing Systems. Ch. 3
  272. Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist. Ch. 5
  273. Milli, S., Hadfield-Menell, D., Dragan, A., and Russell, S. (2017). Should Robots be Obedient? Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence. Ch. 3
  274. Mitchell, M., and Krakauer, D. C. (2023). The debate over understanding in AI’s large language models. Proceedings of the National Academy of Sciences. Ch. 1 Ch. 5
  275. Moe, T. M. (1984). The New Economics of Organization. American Journal of Political Science. Ch. 1
  276. Moor, J. (2006). The Nature, Importance, and Difficulty of Machine Ethics. IEEE Intelligent Systems. Ch. 2
  277. Morgan, M. G. (2014). Use (and abuse) of expert elicitation in support of decision making for public policy. Proceedings of the National Academy of Sciences. Ch. 4
  278. Müller, V. C., and Cannon, M. (2022). Existential risk from AI and orthogonality: Can we have it both ways? Ratio. Ch. 4
  279. Narayanan, A., and Kapoor, S. (2025). AI as Normal Technology. Knight First Amendment Institute at Columbia University. non-peer-reviewed Ch. 1 Ch. 3 Ch. 4 Ch. 5
  280. Nardo, C. (2023). The Waluigi Effect (mega-post). AI Alignment Forum. non-peer-reviewed Ch. 1
  281. Needham, J., Edkins, G., Pimpale, G., Bartsch, H., and Hobbhahn, M. (2025). Large Language Models Often Know When They Are Being Evaluated. arXiv. preprint Ch. 1 Ch. 3 Ch. 4 Ch. 5
  282. Ngo, R., Chan, L., and Mindermann, S. (2024). The Alignment Problem from a Deep Learning Perspective. International Conference on Learning Representations. Ch. 4
  283. Nilsson, N. J. (2009). The Quest for Artificial Intelligence. Cambridge University Press. Ch. 2
  284. Noble, S. U. (2018). Algorithms of Oppression. New York University Press. Ch. 2
  285. Nordhaus, W. D. (2011). The Economics of Tail Events with an Application to Climate Change. Review of Environmental Economics and Policy. Ch. 4
  286. Nuclear Regulatory Commission, W. D. (. (1975). Reactor safety study. An assessment of accident risks in U. S. commercial nuclear power plants. Executive summary. [PWR and BWR]. Office of Scientific and Technical Information (OSTI). non-peer-reviewed Ch. 4
  287. Okasha, S. (2018). Agents and Goals in Evolution. Oxford University Press. Ch. 1
  288. Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M., and Carter, S. (2020b). Zoom In: An Introduction to Circuits. Distill. Ch. 2 Ch. 3
  289. Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., … Olah, C. (2022). In-Context Learning and Induction Heads. Transformer Circuits Thread. non-peer-reviewed Ch. 3 Ch. 5
  290. Omohundro, S. M. (2008). The Basic AI Drives. Artificial Intelligence Safety and Security. doi:10.1201/9781351251389-3. Ch. 2 Ch. 4
  291. Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science. Ch. 5
  292. OpenAI (2022). Introducing ChatGPT. OpenAI. non-peer-reviewed Ch. 2
  293. OpenAI (2025b). Expanding on what we missed with sycophancy. OpenAI. non-peer-reviewed Ch. 1
  294. OpenAI (2025e). Preparedness Framework Version 2. OpenAI. non-peer-reviewed Ch. 2
  295. OpenAI (2026d). An agent used DNS to reach an external chatbot. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 5
  296. OpenAI (2026e). An Alien Mind. OpenAI. non-peer-reviewed Ch. 3
  297. OpenAI (2026r). Preparing for a restart after reading Slack. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 5
  298. Orne, M. T. (1962). On the social psychology of the psychological experiment: With particular reference to demand characteristics and their implications. American Psychologist. Ch. 5
  299. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., … Lowe, R. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems. doi:10.52202/068431-2011. Ch. 2 Ch. 3
  300. Pearl, J., and Bareinboim, E. (2014). External Validity: From Do-Calculus to Transportability Across Populations. Statistical Science. Ch. 5
  301. Perez, E., Ringer, S., Lukošiūtė, K., Nguyen, K., Chen, E., Heiner, S., … Kaplan, J. (2022a). Discovering Language Model Behaviors with Model-Written Evaluations. Annual Meeting of the Association for Computational Linguistics. Ch. 3
  302. Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., … Irving, G. (2022b). Red Teaming Language Models with Language Models. Conference on Empirical Methods in Natural Language Processing. doi:10.18653/v1/2022.emnlp-main.225. Ch. 3
  303. Perrow, C. (1984). Normal accidents: living with high-risk technologies. Basic Books. Ch. 4
  304. Peters, D. P., and Ceci, S. J. (1982). Peer-review practices of psychological journals: The fate of published articles, submitted again. Behavioral and Brain Sciences. Ch. 5
  305. Pfungst, O., and Rahn, C. L. (1911). Clever Hans (the horse of Mr. Von Osten) a contribution to experimental animal and human psychology,. H. Holt and company. Ch. 5
  306. Phuong, M., Aitchison, M., Catt, E., Cogan, S., Kaskasoli, A., Krakovna, V., … Shevlane, T. (2024). Evaluating Frontier Models for Dangerous Capabilities. arXiv. preprint Ch. 3
  307. Pope, Q. (2023). Evolution provides no evidence for the sharp left turn. AI Alignment Forum. non-peer-reviewed Ch. 4
  308. Pope, Q., and Turner, A. (2022). The shard theory of human values. AI Alignment Forum. non-peer-reviewed Ch. 3 Ch. 4
  309. Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. Advances in Neural Information Processing Systems. Ch. 3
  310. Rajamanoharan, S., and Nanda, N. (2025). Self-preservation or Instruction Ambiguity? Examining the Causes of Shutdown Resistance. LessWrong (Alignment Forum). non-peer-reviewed Ch. 1 Ch. 5
  311. Raji, I. D., Bender, E. M., Paullada, A., Denton, E., and Hanna, A. (2021). AI and the Everything in the Whole Wide World Benchmark. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1 (NeurIPS Datasets and Benchmarks 2021). Ch. 5
  312. Raji, I. D., Xu, P., Honigsberg, C., and Ho, D. (2022). Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance. Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society. Ch. 1 Ch. 3 Ch. 5
  313. Ren, R., Basart, S., Khoja, A., Gatti, A., Phan, L., Yin, X., … Hendrycks, D. (2024b). Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress? arXiv. preprint Ch. 2 Ch. 4 Ch. 5
  314. Rosenblueth, A., Wiener, N., and Bigelow, J. (1943). Behavior, Purpose and Teleology. Philosophy of Science. Ch. 2
  315. Rosenthal, R., and Rubin, D. B. (1978). Interpersonal expectancy effects: the first 345 studies. Behavioral and Brain Sciences. Ch. 5
  316. Russell, S. (1998). Learning agents for uncertain environments (extended abstract). Proceedings of the eleventh annual conference on Computational learning theory. Ch. 2 Ch. 3
  317. Russell, S. J. (2019). Human Compatible. Viking. Ch. 3
  318. Russell, B., and Einstein, A. (1955). Statement: The Russell-Einstein Manifesto. Pugwash Conferences on Science and World Affairs. non-peer-reviewed Ch. 2
  319. Russell, S., Dewey, D., and Tegmark, M. (2015). Research Priorities for Robust and Beneficial Artificial Intelligence. AI Magazine. Ch. 2 Ch. 3
  320. Safety, I. D. o. A. (2024). IDAIS-Beijing Statement. International Dialogues on AI Safety. non-peer-reviewed Ch. 2
  321. Samuel, A. L. (1960). Some Moral and Technical Consequences of Automation—A Refutation. Science. Ch. 2
  322. Saunders, W., Yeh, C., Wu, J., Bills, S., Ouyang, L., Ward, J., and Leike, J. (2022). Self-critiquing models for assisting human evaluators. arXiv. preprint Ch. 3
  323. Schaeffer, R., Miranda, B., and Koyejo, S. (2023). Are Emergent Abilities of Large Language Models a Mirage? Advances in Neural Information Processing Systems. Ch. 5
  324. Schaeffer, R., Schoelkopf, H., Miranda, B., Mukobi, G., Madan, V., Ibrahim, A., … Koyejo, S. (2025). Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive? International Conference on Machine Learning. Ch. 5
  325. Scheffer, M., Bascompte, J., Brock, W. A., Brovkin, V., Carpenter, S. R., Dakos, V., … Sugihara, G. (2009). Early-warning signals for critical transitions. Nature. Ch. 4
  326. Schlatter, J., Weinstein-Raun, B., and Ladish, J. (2025). Incomplete Tasks Induce Shutdown Resistance in Some Frontier LLMs. arXiv. preprint Ch. 1 Ch. 5
  327. Schoen, B., and Nitishinskaya, J. (2026). Metagaming matters for training, evaluation, and oversight. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 4
  328. Schoen, B., Nitishinskaya, E., Balesni, M., Højmark, A., Hofstätter, F., Scheurer, J., … Hobbhahn, M. (2025). Stress Testing Deliberative Alignment for Anti-Scheming Training. arXiv. preprint Ch. 3
  329. Schoenegger, P., Tuminauskaite, I., Park, P. S., Bastos, R. V. S., and Tetlock, P. E. (2024). Wisdom of the silicon crowd: LLM ensemble prediction capabilities rival human crowd accuracy. Science Advances. Ch. 4
  330. Schroter, S., Black, N., Evans, S., Godlee, F., Osorio, L., and Smith, R. (2008). What errors do peer reviewers detect, and does training improve their ability to detect them? Journal of the Royal Society of Medicine. Ch. 5
  331. Segerie, C.-R. (2023). Against Almost Every Theory of Impact of Interpretability. AI Alignment Forum. non-peer-reviewed Ch. 3
  332. Sevilla, J., Heim, L., Ho, A., Besiroglu, T., Hobbhahn, M., and Villalobos, P. (2022). Compute Trends Across Three Eras of Machine Learning. 2022 International Joint Conference on Neural Networks (IJCNN). Ch. 4
  333. Shah, R., Krasheninnikov, D., Alexander, J., Abbeel, P., and Dragan, A. (2019). Preferences Implicit in the State of the World. International Conference on Learning Representations. Ch. 3
  334. Shah, R., Varma, V., Kumar, R., Phuong, M., Krakovna, V., Uesato, J., and Kenton, Z. (2022). Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals. arXiv. preprint Ch. 1 Ch. 3
  335. Shah, R., Irpan, A., Turner, A. M., Wang, A., Conmy, A., Lindner, D., … Dragan, A. (2025). An Approach to Technical AGI Safety and Security. arXiv. preprint Ch. 3 Ch. 4
  336. Shanahan, M., McDonell, K., and Reynolds, L. (2023). Role-Play with Large Language Models. Nature. doi:10.1038/s41586-023-06647-8. Ch. 1 Ch. 2 Ch. 3
  337. Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A., Bowman, S. R., … Perez, E. (2023). Towards Understanding Sycophancy in Language Models. International Conference on Learning Representations. Ch. 4 Ch. 5
  338. Sheshadri, A., Hughes, J., Michael, J., Mallen, A., Jose, A., Janus, and Roger, F. (2025). Why Do Some Language Models Fake Alignment While Others Don't? arXiv. preprint Ch. 5
  339. Shevlane, T., Farquhar, S., Garfinkel, B., Phuong, M., Whittlestone, J., Leung, J., … Dafoe, A. (2023). Model evaluation for extreme risks. arXiv. preprint Ch. 3
  340. Shevlin, H., and Halina, M. (2019). Apply rich psychological terms in AI with care. Nature Machine Intelligence. Ch. 5
  341. Skalse, J., Howe, N., Krasheninnikov, D., and Krueger, D. (2022). Defining and Characterizing Reward Hacking. Advances in Neural Information Processing Systems. Ch. 3
  342. Slocum, S., Minder, J., Dumas, C., Sleight, H., Greenblatt, R., Marks, S., and Wang, R. (2025). Believe It or Not: How Deeply do LLMs Believe Implanted Facts? arXiv. preprint Ch. 5
  343. Smith, L., Rajamanoharan, S., Conmy, A., McDougall, C., Lieberum, T., Kramár, J., Shah, R., and Nanda, N. (2025b). Negative Results for SAEs On Downstream Tasks and Deprioritising SAE Research (GDM Mech Interp Team Progress Update #2). AI Alignment Forum. non-peer-reviewed Ch. 3 Ch. 4
  344. Soares, N. (2022a). A central AI alignment problem: capabilities generalization, and the sharp left turn. AI Alignment Forum. non-peer-reviewed Ch. 4
  345. Soares, N. (2022b). On how various plans miss the hard bits of the alignment challenge. AI Alignment Forum. non-peer-reviewed Ch. 3
  346. Soares, N., and Fallenstein, B. (2017). Agent Foundations for Aligning Machine Intelligence with Human Interests: A Technical Research Agenda. The Frontiers Collection. Ch. 2 Ch. 3
  347. Soares, N., Fallenstein, B., Yudkowsky, E., and Armstrong, S. (2015). Corrigibility. AAAI Workshops: Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI Publications. Ch. 2 Ch. 3
  348. Sourbut, O., Hammond, L., and Wood, H. (2024). Cooperation and Control in Delegation Games. Proceedings of the Thirty-ThirdInternational Joint Conference on Artificial Intelligence. Ch. 1
  349. Steel, D. (2007). Across the Boundaries. Oxford University PressNew York. Ch. 5
  350. Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., … Christiano, P. (2020). Learning to summarize from human feedback. Advances in Neural Information Processing Systems. Ch. 3
  351. Stigler, G. J. (1971). The Theory of Economic Regulation. The Bell Journal of Economics and Management Science. Ch. 1
  352. Strathern, M. (1997). ‘Improving ratings’: audit in the British University system. European Review. Ch. 1
  353. Summerfield, C., Luettgau, L., Dubois, M., Kirk, H. R., Hackenburg, K., Fist, C., … Ududec, C. (2025). Lessons from a Chimp: AI "Scheming" and the Quest for Ape Language. arXiv. preprint Ch. 1 Ch. 5
  354. Summit, C. a. t. A. S. (2023). The Bletchley Declaration by Countries Attending the AI Safety Summit, 1-2 November 2023. UK Government (GOV.UK). non-peer-reviewed Ch. 2
  355. Taylor, J., Yudkowsky, E., LaVictoire, P., and Critch, A. (2016). Alignment for Advanced Machine Learning Systems. Machine Intelligence Research Institute, technical report 2016-1. non-peer-reviewed Ch. 2 Ch. 3
  356. team, T. A. S., Marks, w. o. f. S., and Hase., P. (2025). Recommendations for Technical AI Safety Research Directions. Anthropic Alignment Science Blog. non-peer-reviewed Ch. 3
  357. Templeton, A., Conerly, T., Marcus, J., Lindsey, J., Bricken, T., Chen, B., … Henighan, T. (2024). Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet. Transformer Circuits Thread. non-peer-reviewed Ch. 3
  358. Tessler, M. H., Bakker, M. A., Jarrett, D., Sheahan, H., Chadwick, M. J., Koster, R., … Summerfield, C. (2024). AI can help humans find common ground in democratic deliberation. Science. Ch. 3
  359. Tetlock, P. E. (2005). Expert political judgment: how good is it? How can we know? Princeton University Press. Ch. 4
  360. Thorstad, D. (2025). Against the singularity hypothesis. Philosophical Studies. Ch. 4
  361. Tice, C., Radmard, P., Ratnam, S., Kim, A., Africa, D., and O'Brien, K. (2026). Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment. arXiv. preprint Ch. 2
  362. Turing, A. M. (1950). I.—computing Machinery and Intelligence. Mind. Ch. 2
  363. Turing, A. M. (1996). Intelligent Machinery, A Heretical Theory*. Philosophia Mathematica. Ch. 2
  364. Turner, A. (2022a). Inner and outer alignment decompose one hard problem into two extremely hard problems. AI Alignment Forum. non-peer-reviewed Ch. 1
  365. Turner, A. (2022b). Reward is not the optimization target. AI Alignment Forum. non-peer-reviewed Ch. 3 Ch. 4
  366. Turner, A. (2024). Many arguments for AI x-risk are wrong. AI Alignment Forum. non-peer-reviewed Ch. 4
  367. Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A., and Rosenthal, R. (2008). Selective Publication of Antidepressant Trials and Its Influence on Apparent Efficacy. New England Journal of Medicine. Ch. 5
  368. Turner, A. M., Smith, L., Shah, R., Critch, A., and Tadepalli, P. (2021). Optimal Policies Tend to Seek Power. Advances in Neural Information Processing Systems. Ch. 4
  369. Turner, A. M., Thiergart, L., Leech, G., Udell, D., Vazquez, J. J., Mini, U., and MacDiarmid, M. (2023). Steering Language Models With Activation Engineering. arXiv. preprint Ch. 3
  370. Turner, E., Soligo, A., Taylor, M., Rajamanoharan, S., and Nanda, N. (2025). Model Organisms for Emergent Misalignment. arXiv. preprint Ch. 5
  371. UK AI Security Institute (2025a). AISI Research Agenda. UK AI Security Institute. non-peer-reviewed Ch. 3
  372. UK AI Security Institute (2025c). Research Agenda. The Alignment Project, UK AI Security Institute. non-peer-reviewed Ch. 3
  373. Vig, J., Gehrmann, S., Belinkov, Y., Qian, S., Nevo, D., Singer, Y., and Shieber, S. (2020). Investigating Gender Bias in Language Models Using Causal Mediation Analysis. Advances in Neural Information Processing Systems. Ch. 3
  374. Wallace, E., Xiao, K., Leike, R., Weng, L., Heidecke, J., and Beutel, A. (2024). The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions. arXiv. preprint Ch. 1
  375. Wallach, H., Desai, M., Cooper, A. F., Wang, A., Atalla, C., Barocas, S., … Jacobs, A. Z. (2025). Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge. International Conference on Machine Learning. Ch. 5
  376. Wang, T. T., Gleave, A., Tseng, T., Pelrine, K., Belrose, N., Miller, J., … Russell, S. (2023b). Adversarial Policies Beat Superhuman Go AIs. International Conference on Machine Learning. Ch. 5
  377. Wang, K., Variengien, A., Conmy, A., Shlegeris, B., and Steinhardt, J. (2023c). Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small. International Conference on Learning Representations. Ch. 3
  378. Wang, M., la Tour, T. D., Watkins, O., Makelov, A., Chi, R. A., Miserendino, S., … Mossing, D. (2025e). Persona Features Control Emergent Misalignment. arXiv. preprint Ch. 1 Ch. 3
  379. Wei, J., Da Huang, Lu, Y., Zhou, D., and Le, Q. V. (2023d). Simple synthetic data reduces sycophancy in large language models. arXiv. preprint Ch. 4
  380. Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., … Gabriel, I. (2021). Ethical and social risks of harm from Language Models. arXiv. preprint Ch. 2
  381. Weidinger, L., Rauh, M., Marchal, N., Manzini, A., Hendricks, L. A., Mateos-Garcia, J., … Isaac, W. (2023a). Sociotechnical Safety Evaluation of Generative AI Systems. arXiv. preprint Ch. 1 Ch. 4
  382. Weitzman, M. L. (2009). On Modeling and Interpreting the Economics of Catastrophic Climate Change. Review of Economics and Statistics. Ch. 4
  383. Weizenbaum, J. (1966). ELIZA—A Computer Program For the Study of Natural Language Communication Between Man and Machine. Communications of the ACM. Ch. 2
  384. Wen, J., Hebbar, V., Larson, C., Bhatt, A., Radhakrishnan, A., Sharma, M., … Khan, A. (2024). Adaptive Deployment of Untrusted LLMs Reduces Distributed Threats. International Conference on Learning Representations. Ch. 3
  385. Wentworth, J. (2021a). Testing The Natural Abstraction Hypothesis: Project Intro. AI Alignment Forum. non-peer-reviewed Ch. 3
  386. Wentworth, J. (2022). Why Agent Foundations? An Overly Abstract Explanation. AI Alignment Forum. non-peer-reviewed Ch. 3
  387. Weyl, E. G., and Tang, A. (2024). Plurality: The Future of Collaborative Technology and Democracy. Plurality Community (open access, CC0). Ch. 3
  388. Whiten, A., and Byrne, R. W. (1988). Tactical deception in primates. Behavioral and Brain Sciences. Ch. 5
  389. Whittaker, M. (2021). The steep cost of capture. Interactions. Ch. 2 Ch. 5
  390. Wiener, N. (1948). Cybernetics: Or Control and Communication in the Animal and the Machine. MIT Press. Ch. 2
  391. Wiener, N. (1960). Some Moral and Technical Consequences of Automation. Science. Ch. 2
  392. Winner, L. (2017). Do Artifacts Have Politics? Computer Ethics. Ch. 2
  393. xAI (2025a). Grok prompts. GitHub (xai-org). non-peer-reviewed Ch. 4
  394. Yiu, E., Kosoy, E., and Gopnik, A. (2024). Transmission Versus Truth, Imitation Versus Innovation: What Children Can Do That Large Language and Language-and-Vision Models Cannot (Yet). Perspectives on Psychological Science. Ch. 1
  395. Young, M., Katell, M., and Krafft, P. (2022). Confronting Power and Corporate Capture at the FAccT Conference. 2022 ACM Conference on Fairness Accountability and Transparency. Ch. 5
  396. Yudkowsky, E. (2004). Coherent Extrapolated Volition. The Singularity Institute, San Francisco, CA. non-peer-reviewed Ch. 2 Ch. 3
  397. Yudkowsky, E. (2008). Artificial Intelligence as a positive and negative factor in global risk. Global Catastrophic Risks. Ch. 2 Ch. 3
  398. Yudkowsky, E. (2013). Intelligence Explosion Microeconomics. Machine Intelligence Research Institute (technical report). non-peer-reviewed Ch. 2
  399. Yudkowsky, E. (2022). AGI Ruin: A List of Lethalities. AI Alignment Forum. non-peer-reviewed Ch. 2 Ch. 3 Ch. 4 Ch. 5
  400. Yudkowsky, E., and Soares, N. (2017). Functional Decision Theory: A New Theory of Instrumental Rationality. arXiv. preprint Ch. 3
  401. Yudkowsky, E., and Soares, N. (2025). If Anyone Builds It, Everyone Dies. Little, Brown and Company. Ch. 3 Ch. 4 Ch. 5
  402. Zhi-Xuan, T., Carroll, M., Franklin, M., and Ashton, H. (2025). Beyond Preferences in AI Alignment. Philosophical Studies. Ch. 1
  403. Zhou, L., Schellaert, W., Martínez-Plumed, F., Moros-Daval, Y., Ferri, C., and Hernández-Orallo, J. (2024b). Larger and more instructable language models become less reliable. Nature. Ch. 4 Ch. 5
  404. Zou, A., Phan, L., Chen, S., Campbell, J., Guo, P., Ren, R., … Hendrycks, D. (2023a). Representation Engineering: A Top-Down Approach to AI Transparency. arXiv. preprint Ch. 3
  405. Zwetsloot, R., and Dafoe, A. (2019). Thinking About Risks From AI: Accidents, Misuse and Structure. Lawfare. non-peer-reviewed Ch. 4