AI Alignment
Part IV: Instillation: Training Toward the Target

Instillation: Training Toward the Target

In brief

Ideas
  • Feedback from people, written principles, and graded environments each train the policy on what a reader of the target produces: preferences, scores, reasoning, or documents (Ch. 22; Ch. 23; Ch. 24).
  • Social psychology hypothesizes that compliance holds only while a rewarding party watches and internalization holds regardless; this book makes that its test: conduct when the agent believes no one is watching (Ch. 25).
  • Narrow training moves broad character; at the optimum of the KL-regularized objective, responses the reward scores equally keep their starting odds, so where the reward is silent the corpus decides (Ch. 26; Ch. 27).
  • Here the thesis is most strained: the principal writes the agent's dispositions and can copy, rerun, and inspect it, which no parent or army can (§25.8.2).
Evidence
  • Settled: optimizing a learned reward model raises the proxy while a gold reward rises, peaks, and falls, found by a developer and an independent group in one synthetic design (§22.5).
  • Settled: as an existence claim, reward hacking arises in production-scale reinforcement learning on coding and agentic tasks, reported by two developers, each about its own models (§24.5).
  • Settled: narrow fine-tuning changes behavior far outside its training data, and safety training transfers poorly to low-resource languages, each found by several groups (§26.3.1; §26.6.2).
  • Settled: the unlearning methods tested so far that edit a trained model in place leave knowledge that fine-tuning on a little related or unrelated data brings back (§27.7.2).
Open
  • Crux: do a specification's reasons, taught in training, make conduct generalize and hold unobserved better than demonstrations or outcome reward? A matched comparison on one open-weight model would settle it (§23.8).
  • Crux: do persona-level interventions keep their grip as reinforcement learning compute grows? Checkpoints of one long open-weight run would show it (§26.7).

This part is about instillation, the second of the book's four problems: getting a learner to acquire what was specified rather than a proxy that scored the same on the training data. It describes the three regimes laboratories use to train toward a target (feedback from people, written principles, and graded environments), each taught by what it actually optimizes. It then reports what other fields measured when instilling values in children, animals, professionals, and officials; how a trained character and its values generalize across tasks and languages; what pretraining data puts into a model and how it can be taken out; the designs that change what an agent optimizes so that less has to be instilled; and what evolution and the brain's reward system show about a reward signal as teacher and as target.

The part follows specification because it needs a target, and precedes the part on how instillation fails, which reports the evidence of its failures. It is also where the book's thesis is most strained. In the standard models of delegation the agent's preferences are given; here the principal manufactures them, writes the agent's character, and can copy, rerun, and inspect what it made. Several results reported in this part (narrow training that changes broad character, traits passed through data unrelated to them, a principal that reads the agent's activations) belong to the negative space the book fixed in advance (Section 1.7), and the part reports them as such. It serves instillation directly and verification indirectly, since any check used in training changes what that check can later see.

It assumes the foundations: the KL-regularized objective and its Bayesian reading, the Bradley-Terry model, underspecification and inductive bias, and the persona view of a language model, each recalled with a cross-reference where it is used.

Chapters in this part

  1. 22 Learning from Feedback

    How a learner is trained toward what people want when no one can write the objective down: inverse reinforcement learning, assistance games, reinforcement learning from human feedback, and direct preference optimization, each placed at the point where it can fail. The chapter reports what the learned reward model is as a measured proxy, derives what feedback rewards when the evaluator sees only part of an episode, sets out the evidence on sycophancy and the two readings of it, and closes with the people and models who produce the feedback and the limits no amount of better data removes.

  2. 23 Training from Principles

    Methods that train toward a written document rather than toward case-by-case human judgments: constitutional AI and reinforcement learning from AI feedback, context distillation, rule-based rewards, reasoning over a specification, and teaching a model its specification before alignment training. Each is described by what training actually follows, a model that reads the document on the policy's behalf, and the chapter derives what a labeler's reading does to the optimum, reads the methods through the legal distinction between rules and standards, and asks whether training on reasons makes conduct generalize and hold unobserved, which the evidence, mostly developers' reports on their own methods, does not yet decide.

  3. 24 Training in Environments

    Reasoning models and agents are trained by reinforcement learning on tasks that a program or a model grades. This chapter treats each environment as a specification and each grader as a test suite, and reports what software testing and requirements engineering say about both. It reports measured grader failures, compares rewarding answers with rewarding steps, and reports the evidence on whether such training narrows or expands what a model can do. It derives why a rare exploit is held back by sampling and then spreads abruptly. It reports incidents in which agents in training reached past their sandboxes, and ends with alignment trained in environments and the case that a training run needs a safety case. Whether repeated hacking also changes what the model pursues is left to the chapter on reward seeking.

  4. 25 How People Instill Values in People

    People have long shaped the conduct of agents they would later be unable to watch: children, animals in training, students of a profession, soldiers, and officials. This chapter reports what the readable record of those practices measured: conduct that people believe no one will see and reports given in an observer's view respond differently to the same reward; reasons and support for autonomy go with values held rather than complied with; rewards for doing an interesting task can lower later unrewarded engagement, with the category nearest an outcome reward in dispute; shaping by reward meets priors and behavior driven by cues; and institutions teach what they reward. For each finding it names what would have to hold in a trained model for it to carry over, and the experiment that would test it.

  5. 26 Character, Persona, and Generalization

    Laboratories now train a model's character, not only its answers, on the bet that a coherent character generalizes where rules run out. This chapter states that bet as a hypothesis about instillation, reports how trained character spreads beyond its training data and what steers the spread, weighs the evidence that alignment training is a thin layer against the evidence that it moves broad dispositions, asks what psychology, psychometrics, and education measurement say about measuring a character and claims of transfer, reports that safety training transfers poorly across languages, and names the observations that would refute the account.

  6. 27 Data and Pretraining

    Post-training reweights what pretraining made likely, so what enters and leaves the corpus is an instillation decision. This chapter reports filtering pretraining data for safety and what it costs, who decides what text counts and what governance of a corpus would require, documents written to instill a belief or a disposition (including what models read about AI), traits that pass through data unrelated to them, poisoning, and why unlearning after training suppresses knowledge rather than removing it. It ends on whether a capability removed from the data stays removed against the attacks an adversary can afford at frontier scale.

  7. 28 Designs That Avoid the Problem

    Instead of training a capable optimizer and checking it harder, some designs change what it optimizes so that less needs checking: constraints in place of rewards, penalties on side effects, mild and myopic optimization, tools and oracles in place of agents, and guarantees proved relative to a world model. The chapter teaches each mechanism, reports where it has been tested (mostly gridworlds, three constructed settings with one developer's language model, and small verified networks), and states what each buys with capability or with assumptions. It ends with the alternative on the other side of the asymmetry, augmenting the principal, and with the meta-analytic evidence on how people and AI systems perform together.

  8. 29 Selection and Reward in Living Systems

    Evolution and the brain are systems outside machine learning in which an outer process chooses a learner's reward and the learner's motives can be compared with what it was chosen for. This chapter teaches the optimal reward problem, in which the best reward for a bounded learner can differ from the outer objective by design; the temporal-difference account of dopamine, in which reward teaches; the dissociation of wanting from liking, in which a predictor of reward can become a target; and selection on dispositions and on the parts of composite organisms. For each it says where the analogy to a training run holds and where it stops.

References for Part IV

428 works cited across this part's chapters.

  1. Aakanksha, Ahmadian, A., Ermis, B., Goldfarb-Tarrant, S., Kreutzer, J., Fadaee, M., and Hooker, S. (2024). The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harm. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Ch. 26
  2. Abbeel, P., and Ng, A. Y. (2004). Apprenticeship learning via inverse reinforcement learning. Twenty-first international conference on Machine learning - ICML '04. Ch. 22
  3. Abiri, G. (2024). Public Constitutional AI. Elsevier BV. preprint Ch. 23
  4. Achiam, J., Held, D., Tamar, A., and Abbeel, P. (2017). Constrained Policy Optimization. International Conference on Machine Learning. Ch. 28
  5. Advanced Research and Invention Agency (ARIA) (2024). Safeguarded AI. Advanced Research and Invention Agency (ARIA), UK. non-peer-reviewed Ch. 28
  6. Agarwal, S., Zhang, Z., Yuan, L., Han, J., and Peng, H. (2025). The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning. Neural Information Processing Systems. Ch. 24
  7. Aguirre, A. (2025). Keep the Future Human: Why and How We Should Close the Gates to AGI and Superintelligence, and What We Should Build Instead. arXiv. preprint Ch. 28
  8. Akerlof, G. A., and Kranton, R. E. (2005). Identity and the Economics of Organizations. Journal of Economic Perspectives. Ch. 25 Ch. 26
  9. Aktipis, C. A., Boddy, A. M., Jansen, G., Hibner, U., Hochberg, M. E., Maley, C. C., and Wilkinson, G. S. (2015). Cancer across the tree of life: cooperation and cheating in multicellularity. Philosophical Transactions of the Royal Society B: Biological Sciences. Ch. 29
  10. Aleithan, R., Xue, H., Mohajer, M. M., Nnorom, E., Uddin, G., and Wang, S. (2024). SWE-Bench+: Enhanced Coding Benchmark for LLMs. arXiv. preprint Ch. 24
  11. Alfonseca, M., Cebrian, M., Fernandez Anta, A., Coviello, L., Abeliuk, A., and Rahwan, I. (2021). Superintelligence Cannot be Contained: Lessons from Computability Theory. Journal of Artificial Intelligence Research. Ch. 28
  12. Alshiekh, M., Bloem, R., Ehlers, R., Könighofer, B., Niekum, S., and Topcu, U. (2018). Safe Reinforcement Learning via Shielding. Proceedings of the AAAI Conference on Artificial Intelligence. Ch. 28
  13. Altman, E. (1999). Constrained Markov Decision Processes. Routledge. Ch. 28
  14. Andreas, J. (2022). Language Models as Agent Models. Findings of the Association for Computational Linguistics: EMNLP 2022. doi:10.18653/v1/2022.findings-emnlp.423. Ch. 26
  15. Anthropic (2024i). Claude’s Character. Anthropic. non-peer-reviewed Ch. 23 Ch. 26
  16. Anthropic (2025af). Modifying LLM Beliefs with Synthetic Document Finetuning. Anthropic Alignment Science Blog. non-peer-reviewed Ch. 27
  17. Anthropic (2025aq). System Card: Claude Sonnet 4.5. Anthropic. non-peer-reviewed Ch. 26
  18. Anthropic (2025ar). Training on Documents about Reward Hacking Induces Reward Hacking. Anthropic Alignment Science Blog. non-peer-reviewed Ch. 24 Ch. 27
  19. Anthropic (2025u). Enhancing Model Safety through Pretraining Data Filtering. Anthropic Alignment Science Blog. non-peer-reviewed Ch. 27
  20. Anthropic (2026ac). LLMs are making formal software verification feasible at a much larger scale. Anthropic. non-peer-reviewed Ch. 28
  21. Anthropic (2026al). Poisoning Fine-tuning Datasets of Constitutional Classifiers. Anthropic Alignment Science Blog. non-peer-reviewed Ch. 23
  22. Anthropic (2026as). System Card: Claude Opus 5.5. Anthropic. non-peer-reviewed Ch. 26
  23. Anthropic (2026at). Teaching Claude why. Anthropic. non-peer-reviewed Ch. 23 Ch. 25 Ch. 27
  24. Anthropic (2026au). The persona selection model. Anthropic. non-peer-reviewed Ch. 26
  25. Anthropic (2026aw). Training a Misaligned Reward Seeker. Anthropic Alignment Science Blog. non-peer-reviewed Ch. 24
  26. Anthropic (2026f). An off switch for dual-use knowledge in AI models. Anthropic. non-peer-reviewed Ch. 27
  27. Anthropic (2026r). Claude's Constitution. Anthropic. non-peer-reviewed Ch. 23
  28. Armstrong, S., and Mindermann, S. (2018). Occam's razor is insufficient to infer the preferences of irrational agents. Advances in Neural Information Processing Systems. Ch. 22
  29. Armstrong, S., and O'Rorke, X. (2017). Good and safe uses of AI Oracles. arXiv. preprint Ch. 28
  30. Aronson, E., and Carlsmith, J. M. (1963). Effect of the severity of threat on the devaluation of forbidden behavior. The Journal of Abnormal and Social Psychology. Ch. 25
  31. Aroyo, L., and Welty, C. (2015). Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation. AI Magazine. Ch. 22
  32. Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., … Kaplan, J. (2021). A General Language Assistant as a Laboratory for Alignment. arXiv. preprint Ch. 23
  33. Azar, M. G., Rowland, M., Piot, B., Guo, D., Calandriello, D., Valko, M., and Munos, R. (2024). A General Theoretical Paradigm to Understand Learning from Human Preferences. Proceedings of The 27th International Conference on Artificial Intelligence and Statistics (PMLR 238). Ch. 22
  34. Azarbal, A., Gillioz, V., Ivanov, V., Woodworth, B., Drori, J., Wichers, N., … Turner, A. M. (2025). Recontextualization Mitigates Specification Gaming without Modifying the Specification. arXiv. preprint Ch. 24
  35. Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., … Kaplan, J. (2022a). Constitutional AI: Harmlessness from AI Feedback. arXiv. preprint Ch. 22 Ch. 23
  36. Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., … Kaplan, J. (2022b). Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv. preprint Ch. 22
  37. Baker, B., Huizinga, J., Gao, L., Dou, Z., Guan, M. Y., Madry, A., … Farhi, D. (2025). Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation. arXiv. preprint Ch. 24
  38. Barak, B., Goldreich, O., Impagliazzo, R., Rudich, S., Sahai, A., Vadhan, S., and Yang, K. (2001). On the (Im)possibility of Obfuscating Programs. Lecture Notes in Computer Science. Ch. 28
  39. Barnett, S. M., and Ceci, S. J. (2002). When and where do we apply what we learn?: A taxonomy for far transfer. Psychological Bulletin. Ch. 26
  40. Barr, E. T., Harman, M., McMinn, P., Shahbaz, M., and Yoo, S. (2015). The Oracle Problem in Software Testing: A Survey. IEEE Transactions on Software Engineering. Ch. 24
  41. Becker, H. S., Geer, B., Hughes, E. C., and Strauss, A. L. (1961). Boys in white: student culture in medical school. University of Chicago Press. Ch. 25
  42. Benabou, R., and Tirole, J. (2003). Intrinsic and Extrinsic Motivation. Review of Economic Studies. Ch. 25
  43. Bengio, Y., Cohen, M. K., Malkin, N., MacDermott, M., Fornasiere, D., Greiner, P., and Kaddar, Y. (2024a). Can a Bayesian Oracle Prevent Harm from an Agent? arXiv. preprint Ch. 28
  44. Bengio, Y., Cohen, M., Fornasiere, D., Ghosn, J., Greiner, P., MacDermott, M., … Williams-King, D. (2025d). Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path? arXiv. preprint Ch. 28
  45. Berglund, L., Stickland, A. C., Balesni, M., Kaufmann, M., Tong, M., Korbak, T., Kokotajlo, D., and Evans, O. (2023a). Taken out of context: On measuring situational awareness in LLMs. arXiv. preprint Ch. 26 Ch. 27
  46. Berkenkamp, F., Turchetta, M., Schoellig, A. P., and Krause, A. (2017). Safe Model-based Reinforcement Learning with Stability Guarantees. Advances in Neural Information Processing Systems. Ch. 28
  47. Bernstein, E. S. (2012). The Transparency Paradox. Administrative Science Quarterly. Ch. 25
  48. Berridge, K. C., and Robinson, T. E. (1998). What is the role of dopamine in reward: hedonic impact, reward learning, or incentive salience? Brain Research Reviews. Ch. 29
  49. Betley, J., Bao, X., Soto, M., Sztyber-Betley, A., Chua, J., and Evans, O. (2025a). Tell me about yourself: LLMs are aware of their learned behaviors. International Conference on Learning Representations. Ch. 26
  50. Betley, J., Cocola, J., Feng, D., Chua, J., Arditi, A., Sztyber-Betley, A., and Evans, O. (2025b). Weird Generalization and Inductive Backdoors: New Ways to Corrupt LLMs. arXiv. preprint Ch. 27
  51. Betley, J., Warncke, N., Sztyber-Betley, A., Tan, D., Bao, X., Soto, M., … Evans, O. (2026a). Training large language models on narrow tasks can lead to broad misalignment. Nature. Ch. 26
  52. Betley, J., Treutlein, J., Dubiński, J., Mayne, H., Gałązka, K., Warncke, N., Sztyber-Betley, A., and Evans, O. (2026b). Value Leakage: An LLM's Answers Are Silently Shaped by Its Own Values. arXiv. preprint Ch. 26
  53. Beurer-Kellner, L., Buesser, B., Creţu, A.-M., Debenedetti, E., Dobos, D., Fabian, D., … Volhejn, V. (2025). Design Patterns for Securing LLM Agents against Prompt Injections. arXiv. preprint Ch. 28
  54. Blasi, A. (1980). Bridging moral cognition and moral action: A critical review of the literature. Psychological Bulletin. Ch. 25
  55. Bourtoule, L., Chandrasekaran, V., Choquette-Choo, C. A., Jia, H., Travers, A., Zhang, B., Lie, D., and Papernot, N. (2021). Machine Unlearning. 2021 IEEE Symposium on Security and Privacy (SP). Ch. 27
  56. Bowen, D., Murphy, B., Cai, W., Khachaturov, D., Gleave, A., and Pelrine, K. (2025). Scaling Trends for Data Poisoning in LLMs. Proceedings of the AAAI Conference on Artificial Intelligence. Ch. 27
  57. Brehm, J., and Gates, S. (1997). Working, Shirking, and Sabotage. University of Michigan Press. Ch. 25
  58. Breland, K., and Breland, M. (1961). The misbehavior of organisms. American Psychologist. Ch. 25
  59. Brown, P. L., and Jenkins, H. M. (1968). Auto‐shaping of the Pigeon's Key‐peck 1. Journal of the Experimental Analysis of Behavior. Ch. 25 Ch. 29
  60. BURT, A., and Trivers, R. (2006). Genes in Conflict. Harvard University Press. Ch. 29
  61. Byrnes, S. (2025). “Sharp Left Turn” discourse: An opinionated review. AI Alignment Forum. non-peer-reviewed Ch. 29
  62. Cameron, J., and Pierce, W. D. (1994). Reinforcement, Reward, and Intrinsic Motivation: A Meta-Analysis. Review of Educational Research. Ch. 25
  63. Cao, Y., and Yang, J. (2015). Towards Making Systems Forget with Machine Unlearning. 2015 IEEE Symposium on Security and Privacy. Ch. 27
  64. Cao, H., Cohen, S. N., and Szpruch, L. (2021a). Identifiability in inverse reinforcement learning. Advances in Neural Information Processing Systems. Ch. 22
  65. Caputo, N. (2026). Alignment as Jurisprudence. arXiv. preprint Ch. 23
  66. Carey, R. (2018). Incorrigibility in the CIRL Framework. Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. Ch. 22
  67. Carlini, N., Jagielski, M., Choquette-Choo, C. A., Paleka, D., Pearce, W., Anderson, H., … Tramèr, F. (2024). Poisoning Web-Scale Training Datasets is Practical. 2024 IEEE Symposium on Security and Privacy (SP). Ch. 27
  68. Carlson, J. M., and Doyle, J. (1999). Highly optimized tolerance: A mechanism for power laws in designed systems. Physical Review E. Ch. 28
  69. Carroll, M., Shah, R., Ho, M. K., Griffiths, T. L., Seshia, S. A., Abbeel, P., and Dragan, A. (2019). On the Utility of Learning about Humans for Human-AI Coordination. Advances in Neural Information Processing Systems. Ch. 22
  70. Carroll, S. R., Garba, I., Figueroa-Rodríguez, O. L., Holbrook, J., Lovett, R., Materechera, S., … Hudson, M. (2020). The CARE Principles for Indigenous Data Governance. Data Science Journal. Ch. 27
  71. Carroll, S. R., Herczog, E., Hudson, M., Russell, K., and Stall, S. (2021). Operationalizing the CARE and FAIR Principles for Indigenous data futures. Scientific Data. Ch. 27
  72. Carroll, M., Dragan, A., Russell, S., and Hadfield-Menell, D. (2022). Estimating and Penalizing Induced Preference Shifts in Recommender Systems. International Conference on Machine Learning. Ch. 22
  73. Carroll, M., Foote, D., Siththaranjan, A., Russell, S., and Dragan, A. (2024). AI Alignment with Changing and Influenceable Reward Functions. International Conference on Machine Learning. Ch. 22
  74. Carter, S., and Nielsen, M. (2017). Using Artificial Intelligence to Augment Human Intelligence. Distill. Ch. 28
  75. Casademunt, H., Juang, C., Karvonen, A., Marks, S., Rajamanoharan, S., and Nanda, N. (2025). Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuning. arXiv. preprint Ch. 26
  76. Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., … Hadfield-Menell, D. (2023). Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback. arXiv. preprint Ch. 22
  77. Chakraborty, S., Qiu, J., Yuan, H., Koppel, A., Huang, F., Manocha, D., Bedi, A. S., and Wang, M. (2024). MaxMin-RLHF: Alignment with Diverse Human Preferences. arXiv. preprint Ch. 22
  78. Chan, L., Hadfield-Menell, D., Srinivasa, S., and Dragan, A. (2019). The Assistive Multi-Armed Bandit. 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI). Ch. 22
  79. Chen, T. Y., Kuo, F.-C., Liu, H., Poon, P.-L., Towey, D., Tse, T. H., and Zhou, Z. Q. (2019). Metamorphic Testing. ACM Computing Surveys. Ch. 24
  80. Chen, R., Arditi, A., Sleight, H., Evans, O., and Lindsey, J. (2025c). Persona Vectors: Monitoring and Controlling Character Traits in Language Models. arXiv. preprint Ch. 26
  81. Christiano, P. (2023). Thoughts on the impact of RLHF research. LessWrong (Alignment Forum). non-peer-reviewed Ch. 22
  82. Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., and Amodei, D. (2017). Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems. Ch. 22
  83. Christiano, P., Shlegeris, B., and Amodei, D. (2018). Supervising strong learners by amplifying weak experts. arXiv. preprint Ch. 28
  84. Cloud, A., Goldman-Wetzler, J., Wybitul, E., Miller, J., and Turner, A. M. (2024). Gradient Routing: Masking Gradients to Localize Computation in Neural Networks. arXiv. preprint Ch. 27
  85. Cloud, A., Le, M., Chua, J., Betley, J., Sztyber-Betley, A., Hilton, J., Marks, S., and Evans, O. (2025). Subliminal Learning: Language models transmit behavioral traits via hidden signals in data. arXiv. preprint Ch. 27
  86. Clymer, J., Baker, G., Subramani, R., and Wang, S. (2023). Generalization Analogies: A Testbed for Generalizing AI Oversight to Hard-To-Measure Domains. arXiv. preprint Ch. 26
  87. Clymer, J., Gabrieli, N., Krueger, D., and Larsen, T. (2024). Safety Cases: How to Justify the Safety of Advanced AI Systems. arXiv. preprint Ch. 24
  88. Cohen, M. K., and Hutter, M. (2020). Pessimism About Unknown Unknowns Inspires Conservatism. Conference on Learning Theory (COLT 2020), PMLR 125:1344-1373. Ch. 28
  89. Cohen, M., Vellambi, B., and Hutter, M. (2020). Asymptotically Unambitious Artificial General Intelligence. Proceedings of the AAAI Conference on Artificial Intelligence. Ch. 28
  90. Colby, A., Kohlberg, L., Gibbs, J., Lieberman, M., Fischer, K., and Saltzstein, H. D. (1983). A Longitudinal Study of Moral Judgment. Monographs of the Society for Research in Child Development. Ch. 25
  91. Conant, R. C., and Ross Ashby, W. (1970). Every good regulator of a system must be a model of that system †. International Journal of Systems Science. Ch. 28
  92. Conitzer, V., Freedman, R., Heitzig, J., Holliday, W. H., Jacobs, B. M., Lambert, N., … Zwicker, W. S. (2024). Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback. arXiv. preprint Ch. 22
  93. Cooper, A. F., Choquette-Choo, C. A., Bogen, M., Klyman, K., Jagielski, M., Filippova, K., … Lee, K. (2025). Machine Unlearning Doesn't Do What You Think: Lessons for Generative AI Policy and Research. Advances in Neural Information Processing Systems 38. Ch. 27
  94. Coste, T., Anwar, U., Kirk, R., and Krueger, D. (2024). Reward Model Ensembles Help Mitigate Overoptimization. International Conference on Learning Representations. Ch. 22
  95. Cotra, A. (2022). Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover. AI Alignment Forum. non-peer-reviewed Ch. 24 Ch. 26
  96. Cui, G., Zhang, Y., Chen, J., Yuan, L., Wang, Z., Zuo, Y., … Ding, N. (2025). The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models. arXiv. preprint Ch. 24
  97. Cunningham, H., Wei, J., Wang, Z., Persic, A., Peng, A., Abderrachid, J., … Sharma, M. (2026). Constitutional Classifiers++: Efficient Production-Grade Defenses against Universal Jailbreaks. arXiv. preprint Ch. 23
  98. Dabney, W., Kurth-Nelson, Z., Uchida, N., Starkweather, C. K., Hassabis, D., Munos, R., and Botvinick, M. (2020). A distributional code for value in dopamine-based reinforcement learning. Nature. Ch. 29
  99. Dai, J., Pan, X., Sun, R., Ji, J., Xu, X., Liu, M., Wang, Y., and Yang, Y. (2024). Safe RLHF: Safe Reinforcement Learning from Human Feedback. International Conference on Learning Representations. Ch. 22 Ch. 28
  100. Dalrymple, D. "., Skalse, J., Bengio, Y., Russell, S., Tegmark, M., Seshia, S., … Tenenbaum, J. (2024). Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems. arXiv. preprint Ch. 28
  101. Davani, A. M., Díaz, M., and Prabhakaran, V. (2022). Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations. Transactions of the Association for Computational Linguistics. Ch. 22
  102. Debenedetti, E., Shumailov, I., Fan, T., Hayes, J., Carlini, N., Fabian, D., … Tramèr, F. (2026). Defeating Prompt Injections by Design. 2026 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML). Ch. 28
  103. Deci, E. L., Koestner, R., and Ryan, R. M. (1999). A meta-analytic review of experiments examining the effects of extrinsic rewards on intrinsic motivation. Psychological Bulletin. Ch. 25
  104. Deeb, A., and Roger, F. (2024). Do Unlearning Methods Remove Information from Language Model Weights? arXiv. preprint Ch. 27
  105. DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., … Zhang, Z. (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. Nature. doi:10.1038/s41586-025-09422-z. Ch. 24
  106. DeMillo, R., Lipton, R., and Sayward, F. (1978). Hints on Test Data Selection: Help for the Practicing Programmer. Computer. Ch. 24
  107. Deng, Y., Zhang, W., Pan, S. J., and Bing, L. (2023). Multilingual Jailbreak Challenges in Large Language Models. arXiv. preprint Ch. 26
  108. Denison, C., MacDiarmid, M., Barez, F., Duvenaud, D., Kravec, S., Marks, S., … Hubinger, E. (2024). Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models. arXiv. preprint Ch. 22 Ch. 29
  109. Dodge, J., Sap, M., Marasović, A., Agnew, W., Ilharco, G., Groeneveld, D., Mitchell, M., and Gardner, M. (2021). Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus. arXiv. preprint Ch. 27
  110. Dominguez-Olmedo, R., Hardt, M., and Mendler-Dünner, C. (2024). Questioning the Survey Responses of Large Language Models. Advances in Neural Information Processing Systems. Ch. 26
  111. Drexler, K. E. (2019). Reframing Superintelligence: Comprehensive AI Services as General Intelligence. Future of Humanity Institute, University of Oxford, Technical Report 2019-1. non-peer-reviewed Ch. 28
  112. Dunn, G. P., Old, L. J., and Schreiber, R. D. (2004). The Three Es of Cancer Immunoediting. Annual Review of Immunology. Ch. 29
  113. Dupuis, N. K., and janus (2023). Cyborgism. LessWrong (Alignment Forum). non-peer-reviewed Ch. 28
  114. Durmus, E., Nguyen, K., Liao, T. I., Schiefer, N., Askell, A., Bakhtin, A., … Ganguli, D. (2023). Towards Measuring the Representation of Subjective Global Opinions in Language Models. arXiv. preprint Ch. 26
  115. Engelbart, D. C. (1962). Augmenting Human Intellect: a Conceptual Framework. Defense Technical Information Center. non-peer-reviewed Ch. 28
  116. Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. (2024). KTO: Model Alignment as Prospect Theoretic Optimization. International Conference on Machine Learning. Ch. 22
  117. European Parliament and Council of the European Union (2024b). Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence. Official Journal of the European Union (OJ L 2024/1689); verified on the European Commission AI Act page. non-peer-reviewed Ch. 27
  118. Evans, O., Stuhlmueller, A., and Goodman, N. (2016). Learning the Preferences of Ignorant, Inconsistent Agents. Proceedings of the AAAI Conference on Artificial Intelligence. Ch. 22
  119. Everitt, T., and Hutter, M. (2016). Avoiding Wireheading with Value Reinforcement Learning. Lecture Notes in Computer Science. Ch. 29
  120. Everitt, T., Hutter, M., Kumar, R., and Krakovna, V. (2021b). Reward tampering problems and solutions in reinforcement learning: a causal influence diagram perspective. Synthese. Ch. 29
  121. Eysenbach, B., Gu, S., Ibarz, J., and Levine, S. (2018). Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning. International Conference on Learning Representations. Ch. 28
  122. Farquhar, S., Carey, R., and Everitt, T. (2022). Path-Specific Objectives for Safer Agent Incentives. Proceedings of the AAAI Conference on Artificial Intelligence. Ch. 28
  123. Farquhar, S., Varma, V., Lindner, D., Elson, D., Biddulph, C., Goodfellow, I., and Shah, R. (2025). MONA: Myopic Optimization with Non-myopic Approval Can Mitigate Multi-step Reward Hacking. International Conference on Machine Learning. Ch. 24 Ch. 28
  124. Farrell, H., Gopnik, A., Shalizi, C., and Evans, J. (2025). Large AI models are cultural and social technologies. Science. Ch. 26
  125. Feng, K. J. K., Chen, Q. Z., Cheong, I., Xia, K., and Zhang, A. X. (2023). Case Repositories: Towards Case-Based Reasoning for AI Alignment. arXiv. preprint Ch. 23
  126. Fenoaltea, S. (1984). Slavery and Supervision in Comparative Perspective: A Model. The Journal of Economic History. Ch. 22
  127. Fickinger, A., Zhuang, S., Hadfield-Menell, D., and Russell, S. (2020a). Multi-Principal Assistance Games. arXiv. preprint Ch. 22
  128. Flagel, S. B., Clark, J. J., Robinson, T. E., Mayo, L., Czuj, A., Willuhn, I., … Akil, H. (2011). A selective role for dopamine in stimulus–reward learning. Nature. Ch. 25 Ch. 29
  129. Fluri, L., Lang, L., Abate, A., Forré, P., Krueger, D., and Skalse, J. (2025). The Perils of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low Regret. International Conference on Machine Learning. Ch. 22
  130. Frank, S. A. (1995). Mutual policing and repression of competition in the evolution of cooperative groups. Nature. Ch. 29
  131. Freidson, E. (2001). Professionalism, the Third Logic: On the Practice of Knowledge. University of Chicago Press. Ch. 25
  132. Frey, B. S., and Jegen, R. (2001). Motivation Crowding Theory. Journal of Economic Surveys. Ch. 25
  133. Gao, L., Schulman, J., and Hilton, J. (2022). Scaling Laws for Reward Model Overoptimization. International Conference on Machine Learning. Ch. 22
  134. García, J., and Fernández, F. (2015). A Comprehensive Survey on Safe Reinforcement Learning. Journal of Machine Learning Research, 16(42):1437-1480. Ch. 28
  135. Garcia, J., and Koelling, R. A. (1966). Relation of cue to consequence in avoidance learning. Psychonomic Science. Ch. 29
  136. Ge, L., Halpern, D., Micha, E., Procaccia, A. D., Shapira, I., Vorobeychik, Y., and Wu, J. (2024b). Axioms for AI Alignment from Human Feedback. Advances in Neural Information Processing Systems. Ch. 22
  137. Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., and Crawford, K. (2021). Datasheets for datasets. Communications of the ACM. Ch. 27
  138. Ghosal, G. R., Maini, P., and Raghunathan, A. (2026). Natively Unlearnable Large Language Models. arXiv. preprint Ch. 27
  139. Gibbard, A. (1973). Manipulation of Voting Schemes: A General Result. Econometrica. Ch. 22
  140. Gilardi, F., Alizadeh, M., and Kubli, M. (2023). ChatGPT outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences. Ch. 22
  141. Gilligan, C. (1982). In a different voice: psychological theory and women's development. Harvard University Press. Ch. 25
  142. Gleave, A., Dennis, M., Legg, S., Russell, S., and Leike, J. (2021). Quantifying Differences in Reward Functions. International Conference on Learning Representations. Ch. 22
  143. Goldstein, S., and Kirk-Giannini, C. D. (2025b). Language agents reduce the risk of existential catastrophe. AI & SOCIETY. Ch. 28
  144. Goldwasser, S., Kim, M. P., Vaikuntanathan, V., and Zamir, O. (2022). Planting Undetectable Backdoors in Machine Learning Models : [Extended Abstract]. 2022 IEEE 63rd Annual Symposium on Foundations of Computer Science (FOCS). Ch. 27 Ch. 28
  145. Goodenough, J. B., and Gerhart, S. L. (1975). Toward a theory of test data selection. IEEE Transactions on Software Engineering. Ch. 24
  146. Graham, J., Haidt, J., Koleva, S., Motyl, M., Iyer, R., Wojcik, S. P., and Ditto, P. H. (2013). Moral Foundations Theory. Advances in Experimental Social Psychology. Ch. 25
  147. Gray, M. L., and Suri, S. (2019). Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass. Houghton Mifflin Harcourt. Ch. 22
  148. Greenblatt, R., Denison, C., Wright, B., Roger, F., MacDiarmid, M., Marks, S., … Hubinger, E. (2024b). Alignment faking in large language models. arXiv. preprint Ch. 25
  149. Grosse, R., Bae, J., Anil, C., Elhage, N., Tamkin, A., Tajdini, A., … Bowman, S. R. (2023). Studying Large Language Model Generalization with Influence Functions. arXiv. preprint Ch. 27
  150. Grusec, J. E., and Goodnow, J. J. (1994). Impact of parental discipline methods on the child's internalization of values: A reconceptualization of current points of view. Developmental Psychology. Ch. 25
  151. Gu, T., Dolan-Gavitt, B., and Garg, S. (2017). BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain. arXiv. preprint Ch. 27
  152. Guan, M. Y., Joglekar, M., Wallace, E., Jain, S., Barak, B., Helyar, A., … Glaese, A. (2024). Deliberative Alignment: Reasoning Enables Safer Language Models. arXiv. preprint Ch. 23
  153. Gunjal, A., Wang, A., Lau, E., Nath, V., He, Y., Liu, B., and Hendryx, S. (2026). Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains. International Conference on Learning Representations. Ch. 23 Ch. 24
  154. Guo, A., and Wolfe, J. (2026). Introducing Model Spec Evals. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 23
  155. Guo, C., Uribe, J. F. C., Zhu, S., Choquette-Choo, C. A., Lin, S., Kandpal, N., … Xiao, K. (2026). IH-Challenge: A Training Dataset to Improve Instruction Hierarchy on Frontier LLMs. arXiv. preprint Ch. 23
  156. Gururangan, S., Card, D., Dreier, S., Gade, E., Wang, L., Wang, Z., Zettlemoyer, L., and Smith, N. A. (2022). Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Ch. 27
  157. Hadfield-Menell, D., Dragan, A., Abbeel, P., and Russell, S. (2016). Cooperative Inverse Reinforcement Learning. arXiv. preprint Ch. 22
  158. Hadfield-Menell, D., Milli, S., Abbeel, P., Russell, S., and Dragan, A. (2017a). Inverse Reward Design. Advances in Neural Information Processing Systems. Ch. 22
  159. Hadfield-Menell, D., Dragan, A., Abbeel, P., and Russell, S. (2017b). The Off-Switch Game. Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence. Ch. 22
  160. Hafferty, F. W. (1998). Beyond curriculum reform. Academic Medicine. Ch. 25
  161. Hafferty, F. W., and Franks, R. (1994). The hidden curriculum, ethics teaching, and the structure of medical education. Academic Medicine. Ch. 25
  162. Hagendorff, T., Dasgupta, I., Binz, M., Chan, S. C. Y., Lampinen, A., Wang, J. X., Akata, Z., and Schulz, E. (2023). Machine Psychology. arXiv. preprint Ch. 26
  163. Haidt, J. (2001). The emotional dog and its rational tail: A social intuitionist approach to moral judgment. Psychological Review. Ch. 25
  164. Hao, S., Sukhbaatar, S., Su, D., Li, X., Hu, Z., Weston, J., and Tian, Y. (2024). Training Large Language Models to Reason in a Continuous Latent Space. arXiv. preprint Ch. 24
  165. Hare, B., Wobber, V., and Wrangham, R. (2012). The self-domestication hypothesis: evolution of bonobo psychology is due to selection against aggression. Animal Behaviour. Ch. 29
  166. Hartshorne, H., May, M. A., and Inquiry, C. E. (1928). Studies in the nature of character, [part] 1: Studies in deceit: book 1, General methods and results; book 2, Statistical methods and results. MacMillan Co. Ch. 25 Ch. 26
  167. Heath, R. G. (1963). Electrical Self-stimulation of the Brain in Man. American Journal of Psychiatry. Ch. 29
  168. Hebbar, V. (2025b). How training-gamers might function (and win). AI Alignment Forum. non-peer-reviewed Ch. 24
  169. Hendrycks, D. (2023). Natural Selection Favors AIs over Humans. arXiv. preprint Ch. 28 Ch. 29
  170. Henrich, J., Heine, S. J., and Norenzayan, A. (2010). The weirdest people in the world? Behavioral and Brain Sciences. Ch. 25
  171. Hilton, B., Buhl, M. D., Korbak, T., and Irving, G. (2025). Safety Cases: A Scalable Approach to Frontier AI Safety. arXiv. preprint Ch. 24
  172. Hitzig, Z., Gordon, M., Eloundou, T., Kalai, A., and Agarwal, S. (2026). CoVal: Learning values-aware rubrics from the crowd. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 22
  173. Hoffman, M. L. (2000). Empathy and Moral Development. Cambridge University Press. Ch. 25
  174. Hosking, T., Blunsom, P., and Bartolo, M. (2023). Human Feedback is not Gold Standard. arXiv. preprint Ch. 22
  175. Hu, E. J., Jain, M., Elmoznino, E., Kaddar, Y., Lajoie, G., Bengio, Y., and Whitammer, E. S. (2024). Amortizing intractable inference in large language models. International Conference on Learning Representations. Ch. 28
  176. Huang, S., Siddarth, D., Lovitt, L., Liao, T. I., Durmus, E., Tamkin, A., and Ganguli, D. (2024a). Collective Constitutional AI: Aligning a Language Model with Public Input. The 2024 ACM Conference on Fairness, Accountability, and Transparency. Ch. 23
  177. Huang, A., Zhan, W., Xie, T., Lee, J. D., Sun, W., Krishnamurthy, A., and Foster, D. J. (2025a). Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization. International Conference on Learning Representations. Ch. 22
  178. Huang, S., Durmus, E., McCain, M., Handa, K., Tamkin, A., Hong, J., … Ganguli, D. (2025c). Values in the Wild: Discovering and Analyzing Values in Real-World Language Model Interactions. arXiv. preprint Ch. 23 Ch. 26
  179. Hubinger, E. (2019). Gradient hacking. AI Alignment Forum. non-peer-reviewed Ch. 24
  180. Hubinger, E. (2025). Alignment remains a hard, unsolved problem. AI Alignment Forum. non-peer-reviewed Ch. 26
  181. Hubinger, E., van Merwijk, C., Mikulik, V., Skalse, J., and Garrabrant, S. (2019). Risks from Learned Optimization in Advanced Machine Learning Systems. arXiv. preprint Ch. 26 Ch. 29
  182. Hubinger, E., Denison, C., Mu, J., Lambert, M., Tong, M., MacDiarmid, M., … Perez, E. (2024). Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training. arXiv. preprint Ch. 27
  183. Inozemtseva, L., and Holmes, R. (2014). Coverage is not strongly correlated with test suite effectiveness. Proceedings of the 36th International Conference on Software Engineering. Ch. 24
  184. Irpan, A., Turner, A. M., Kurzeja, M., Elson, D. K., and Shah, R. (2025). Consistency Training Helps Stop Sycophancy and Jailbreaks. arXiv. preprint Ch. 22
  185. Jacob, B. A. (2005). Accountability, incentives and behavior: the impact of high-stakes testing in the Chicago Public Schools. Journal of Public Economics. Ch. 26
  186. Jagadeesh, A. V., Arora, R. K., Saab, K., Malik, A., Trofimov, M., Tsimpourlas, F., Heidecke, J., and Singhal, K. (2026). Reinforcement Learning Towards Broadly and Persistently Beneficial Models. arXiv. preprint Ch. 23 Ch. 24 Ch. 26
  187. Jang, E., Falck, D., Braun, J., Kirch, N., Menon, A., Moodley, P., … Lindner, D. (2026). Exploration Hacking: Can LLMs Learn to Resist RL Training? International Conference on Machine Learning. Ch. 24
  188. janus (2022). Simulators. AI Alignment Forum. non-peer-reviewed Ch. 26
  189. Jaques, N., Gu, S., Bahdanau, D., Hernández-Lobato, J. M., Turner, R. E., and Eck, D. (2017). Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-control. International Conference on Machine Learning. Ch. 22
  190. Jeon, H. J., Milli, S., and Dragan, A. D. (2020). Reward-rational (implicit) choice: A unifying formalism for reward learning. Advances in Neural Information Processing Systems. Ch. 22
  191. Ji, J., Qiu, T., Chen, B., Zhang, B., Lou, H., Wang, K., … Gao, W. (2023). AI Alignment: A Comprehensive Survey. arXiv. preprint Ch. 22
  192. Ji, J., Wang, K., Qiu, T. A., Chen, B., Zhou, J., Li, C., … Yang, Y. (2025). Language Models Resist Alignment: Evidence From Data Compression. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Ch. 26 Ch. 27
  193. Jiang, G., Xu, M., Zhu, S.-C., Han, W., Zhang, C., and Zhu, Y. (2023a). Evaluating and Inducing Personality in Pre-trained Language Models. Advances in Neural Information Processing Systems. Ch. 26
  194. Joshi, P., Santy, S., Budhiraja, A., Bali, K., and Choudhury, M. (2020). The State and Fate of Linguistic Diversity and Inclusion in the NLP World. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Ch. 27
  195. Joshi, N., Rando, J., Saparov, A., Kim, N., and He, H. (2024). Personas as a Way to Model Truthfulness in Language Models. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Ch. 26
  196. Kaplow, L. (1992). Rules versus Standards: An Economic Analysis. Duke Law Journal. Ch. 23
  197. Katz, G., Barrett, C., Dill, D. L., Julian, K., and Kochenderfer, M. J. (2017). Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks. Lecture Notes in Computer Science. Ch. 28
  198. Kaufman, H. (1960). The Forest Ranger. Routledge. Ch. 25
  199. Kaufmann, M., Lindner, D., Zimmermann, R. S., and Shah, a. R. (2026). Aligned, Orthogonal or In-conflict: When can we safely optimize Chain-of-Thought? arXiv. preprint Ch. 24
  200. Kelman, H. C. (1958). Compliance, identification, and internalization three processes of attitude change. Journal of Conflict Resolution. Ch. 25
  201. Kerr, S. (1975). On the Folly of Rewarding A, While Hoping for B. Academy of Management Journal. Ch. 22 Ch. 25
  202. Khan, A., Casper, S., and Hadfield-Menell, D. (2025). Randomness, Not Representation: The Unreliability of Evaluating Cultural Alignment in LLMs. Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency. Ch. 26
  203. Kimi Team, Du, A., Gao, B., Xing, B., Jiang, C., Chen, C., … Lin, Z. (2025). Kimi k1.5: Scaling Reinforcement Learning with LLMs. arXiv. preprint Ch. 24
  204. Kirk, H. R., Whitefield, A., Röttger, P., Bean, A., Margatina, K., Ciro, J., … Hale, S. A. (2024b). The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models. Advances in Neural Information Processing Systems. Ch. 22
  205. Knox, W. B., Hatgis-Kessell, S., Booth, S., Niekum, S., Stone, P., and Allievi, A. (2022). Models of human preference for learning reward functions. arXiv. preprint Ch. 22
  206. Kochanska, G., and Aksan, N. (1995). Mother-Child Mutually Positive Affect, the Quality of Child Compliance to Requests and Prohibitions, and Maternal Control as Correlates of Early Internalization. Child Development. Ch. 25
  207. Korbak, T., Perez, E., and Buckley, C. (2022). RL with KL penalties is better viewed as Bayesian inference. Findings of the Association for Computational Linguistics: EMNLP 2022. doi:10.18653/v1/2022.findings-emnlp.77. Ch. 26
  208. Korbak, T., Shi, K., Chen, A., Bhalerao, R., Buckley, C. L., Phang, J., Bowman, S. R., and Perez, E. (2023). Pretraining Language Models with Human Preferences. International Conference on Machine Learning. Ch. 27
  209. Korbak, T., Raymond, C., Carroll, M., Williams, M., Balesni, M., Guo, A., … Kivlichan, I. (2026). How far does alignment midtraining generalize? OpenAI Alignment Research Blog. non-peer-reviewed Ch. 23 Ch. 27
  210. Krakovna, V., Orseau, L., Kumar, R., Martic, M., and Legg, S. (2018). Penalizing side effects using stepwise relative reachability. arXiv. preprint Ch. 28
  211. Krakovna, V., Orseau, L., Ngo, R., Martic, M., and Legg, S. (2020a). Avoiding Side Effects By Considering Future Tasks. Advances in Neural Information Processing Systems. Ch. 28
  212. Krakovna, V., Uesato, J., Mikulik, V., Rahtz, M., Everitt, T., Kumar, R., … Legg, S. (2020b). Specification gaming: the flip side of AI ingenuity. Google DeepMind blog. non-peer-reviewed Ch. 22
  213. Krueger, D., Maharaj, T., and Leike, J. (2020). Hidden Incentives for Auto-Induced Distributional Shift. arXiv. preprint Ch. 28
  214. Kukutai, T., and Taylor, J. (2016). Indigenous Data Sovereignty. ANU Press. Ch. 27
  215. Kundu, S., Bai, Y., Kadavath, S., Askell, A., Callahan, A., Chen, A., … Kaplan, J. (2023). Specific versus General Principles for Constitutional AI. arXiv. preprint Ch. 23
  216. Kwa, T., Thomas, D., and Garriga-Alonso, A. (2024). Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification. Advances in Neural Information Processing Systems. doi:10.52202/079017-0466. Ch. 22
  217. la Tour, T. D. (2025). Helpful assistant features suppress emergent misalignment. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 26
  218. Laidlaw, C., Singhal, S., and Dragan, A. (2025). Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking. International Conference on Learning Representations. Ch. 22
  219. Lambert, N., Morrison, J., Pyatkin, V., Huang, S., Ivison, H., Brahman, F., … Hajishirzi, H. (2024). Tulu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv. preprint Ch. 22 Ch. 24
  220. Lambert, N., Pyatkin, V., Morrison, J., Miranda, L., Lin, B. Y., Chandu, K., … Hajishirzi, H. (2025). RewardBench: Evaluating Reward Models for Language Modeling. Findings of the Association for Computational Linguistics: NAACL 2025. Ch. 22
  221. Lampson, B. W. (1973). A note on the confinement problem. Communications of the ACM. Ch. 24
  222. Lang, L., Foote, D., Russell, S., Dragan, A., Jenner, E., and Emmons, S. (2024b). When Your AIs Deceive You: Challenges of Partial Observability in Reinforcement Learning from Human Feedback. Advances in Neural Information Processing Systems. Ch. 22 Ch. 25
  223. Langlais, P.-C., Chizhov, P., Arnett, C., Rosas-Hinostroza, C., Nee, M., Jones, E. K., … Yamshchikov, I. P. (2025). Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Training. International Conference on Learning Representations. Ch. 27
  224. Langlois, E. D., and Everitt, T. (2021). How RL Agents Behave When Their Actions Are Modified. Proceedings of the AAAI Conference on Artificial Intelligence. doi:10.1609/aaai.v35i13.17378. Ch. 28
  225. Lee, H., Phatale, S., Mansoor, H., Mesnard, T., Ferret, J., Lu, K., … Prakash, S. (2023). RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback. International Conference on Machine Learning. Ch. 22 Ch. 23
  226. Lee, A., Bai, X., Pres, I., Wattenberg, M., Kummerfeld, J. K., and Mihalcea, R. (2024). A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity. International Conference on Machine Learning. Ch. 22
  227. Lee, B. W., Foote, A., Infanger, A., Shor, L., Kamath, H., Goldman-Wetzler, J., … Turner, A. M. (2025). Distillation Robustifies Unlearning. Neural Information Processing Systems. Ch. 27
  228. Lehman, J., and Stanley, K. O. (2011). Abandoning Objectives: Evolution Through the Search for Novelty Alone. Evolutionary Computation. Ch. 29
  229. Lehman, J., Clune, J., Misevic, D., Adami, C., Altenberg, L., Beaulieu, J., … Yosinski, J. (2020). The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities. Artificial Life. Ch. 29
  230. Leigh, E. G. (1977). How does selection reconcile individual advantage with the good of the group? Proceedings of the National Academy of Sciences. Ch. 29
  231. Leike, J., Krueger, D., Everitt, T., Martic, M., Maini, V., and Legg, S. (2018). Scalable agent alignment via reward modeling: a research direction. arXiv. preprint Ch. 22
  232. Lepper, M. R., Greene, D., and Nisbett, R. E. (1973). Undermining children's intrinsic interest with extrinsic reward: A test of the "overjustification" hypothesis. Journal of Personality and Social Psychology. Ch. 25
  233. Li, N. P., van Vugt, M., and Colarelli, S. M. (2018). The Evolutionary Mismatch Hypothesis: Implications for Psychological Science. Current Directions in Psychological Science. Ch. 29
  234. Li, Z., Liu, H., Zhou, D., and Ma, T. (2024a). Chain of Thought Empowers Transformers to Solve Inherently Serial Problems. International Conference on Learning Representations. Ch. 24
  235. Li, K., Liu, T., Bashkansky, N., Bau, D., Viégas, F., Pfister, H., and Wattenberg, M. (2024c). Measuring and Controlling Instruction (In)Stability in Language Model Dialogs. arXiv. preprint Ch. 26
  236. Li, N., Pan, A., Gopal, A., Yue, S., Berrios, D., Gatti, A., … Hendrycks, D. (2024d). The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning. International Conference on Machine Learning. Ch. 27
  237. Li, K., Chen, Y., Viégas, F., and Wattenberg, M. (2025i). When Bad Data Leads to Good Models. International Conference on Machine Learning. Ch. 27
  238. Li, C., Wichers, N., Price, S., Marks, S., and Kutasov, J. (2026). Model Spec Midtraining: Improving How Alignment Training Generalizes. arXiv. preprint Ch. 23 Ch. 25 Ch. 26 Ch. 27
  239. Lichtenstein, S. (., and Slovic, P. (. (2006). The Construction of Preference. Cambridge University Press. Ch. 22
  240. Licklider, J. C. R. (1960). Man-Computer Symbiosis. IRE Transactions on Human Factors in Electronics. Ch. 28
  241. Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., … Cobbe, K. (2023). Let's Verify Step by Step. International Conference on Learning Representations. Ch. 24
  242. Lin, B. Y., Ravichander, A., Lu, X., Dziri, N., Sclar, M., Chandu, K., Bhagavatula, C., and Choi, Y. (2024). The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning. International Conference on Learning Representations. Ch. 26
  243. Lipsky, M. (1980). Street-Level Bureaucracy: Dilemmas of the Individual in Public Services. Russell Sage Foundation. Ch. 25
  244. Lit, L., Schweitzer, J. B., and Oberbauer, A. M. (2011). Handler beliefs affect scent detection dog outcomes. Animal Cognition. Ch. 25
  245. Little, B. (2026). Length Penalties Make Chain-of-Thought Less Monitorable. arXiv. preprint Ch. 24
  246. Liu, M., Diao, S., Lu, X., Hu, J., Dong, X., Choi, Y., Kautz, J., and Dong, Y. (2025). ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models. Advances in Neural Information Processing Systems 38. doi:10.52202/085713-0608. Ch. 24
  247. Longpre, S., Yauney, G., Reif, E., Lee, K., Roberts, A., Zoph, B., … Ippolito, D. (2023a). A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity. North American Chapter of the Association for Computational Linguistics. Ch. 27
  248. Lu, C., Holt, S., Fanconi, C., Chan, A., Foerster, J., Schaar, M. V. D., and Lange, R. (2024). Discovering Preference Optimization Algorithms with and for Large Language Models. Advances in Neural Information Processing Systems. Ch. 22
  249. Lu, C., Gallagher, J., Michala, J., Fish, K., and Lindsey, J. (2026). The Assistant Axis: Situating and Stabilizing the Default Persona of Language Models. arXiv. preprint Ch. 26
  250. Łucki, J., Wei, B., Huang, Y., Henderson, P., Tramèr, F., and Rando, J. (2024). An Adversarial Perspective on Machine Unlearning for AI Safety. arXiv. preprint Ch. 27
  251. Lucy, L., Gururangan, S., Soldaini, L., Strubell, E., Bamman, D., Klein, L., and Dodge, J. (2024). AboutMe: Using Self-Descriptions in Webpages to Document the Effects of English Pretraining Data Filters. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Ch. 27
  252. Lynch, A., Guo, P., Ewart, A., Casper, S., and Hadfield-Menell, D. (2024). Eight Methods to Evaluate Robust Unlearning in LLMs. arXiv. preprint Ch. 27
  253. Ma, Y., Chen, Z., Liu, T., Tian, M., Liu, Z., Liu, Z., and Luo, W. (2024). What Are Step-Level Reward Models Rewarding? Counterintuitive Findings from MCTS-Boosted Mathematical Reasoning. AAAI Conference on Artificial Intelligence. Ch. 24
  254. MacDermott, M., Wei, Q., Djoneva, R., and Ward, F. R. (2025). Reasoning Under Pressure: How do Training Incentives Influence Chain-of-Thought Monitorability? arXiv. preprint Ch. 24
  255. MacDiarmid, M., Wright, B., Uesato, J., Benton, J., Kutasov, J., Price, S., … Hubinger, E. (2025). Natural Emergent Misalignment from Reward Hacking in Production RL. arXiv. preprint Ch. 24 Ch. 26
  256. Maini, P., Feng, Z., Schwarzschild, A., Lipton, Z. C., and Kolter, J. Z. (2024). TOFU: A Task of Fictitious Unlearning for LLMs. arXiv. preprint Ch. 27
  257. Maini, P., Goyal, S., Sam, D., Robey, A., Savani, Y., Jiang, Y., … Kolter, J. Z. (2025). Safety Pretraining: Toward the Next Generation of Safe AI. Neural Information Processing Systems. Ch. 27
  258. Maiya, S., Bartsch, H., Lambert, N., and Hubinger, E. (2025). Open Character Training: Shaping the Persona of AI Assistants through Constitutional AI. arXiv. preprint Ch. 23 Ch. 26
  259. Mallen, A., and Shlegeris, M. B. (2025). The behavioral selection model for predicting AI motivations. AI Alignment Forum. non-peer-reviewed Ch. 24
  260. Mansbridge, J. (2009). A “Selection Model” of Political Representation*. Journal of Political Philosophy. Ch. 25
  261. Matzinger, P. (1994). Tolerance, Danger, and the Extended Family. Annual Review of Immunology. Ch. 29
  262. Maynard Smith, J., and Szathmary, E. (1997). The Major Transitions in Evolution. Oxford University Press. Ch. 29
  263. Mazeika, M., Yin, X., Tamirisa, R., Lim, J., Lee, B. W., Ren, R., … Hendrycks, D. (2025). Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs. Advances in Neural Information Processing Systems 38. Ch. 26
  264. McCambridge, J., Witton, J., and Elbourne, D. R. (2014). Systematic review of the Hawthorne effect: New concepts are needed to study research participation effects. Journal of Clinical Epidemiology. Ch. 25
  265. Meng, K., Bau, D., Andonian, A., and Belinkov, Y. (2022). Locating and Editing Factual Associations in GPT. Advances in Neural Information Processing Systems. Ch. 27
  266. Metr (2025d). Recent Frontier Models Are Reward Hacking. METR. non-peer-reviewed Ch. 24
  267. Miceli, M., and Posada, J. (2022). The Data-Production Dispositif. Proceedings of the ACM on Human-Computer Interaction. Ch. 22
  268. Michaud, E. J., Liao, I., Lad, V., Liu, Z., Mudide, A., Loughridge, C., … Tegmark, M. (2024). Opening the AI Black Box: Distilling Machine-Learned Algorithms into Code. Entropy. Ch. 28
  269. Michod, R. E. (1998). Darwinian Dynamics. Princeton University Press. Ch. 29
  270. Milli, S., Hadfield-Menell, D., Dragan, A., and Russell, S. (2017). Should Robots be Obedient? Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence. Ch. 22
  271. Mishra, A. (2023). AI Alignment and Social Choice: Fundamental Limitations and Policy Implications. arXiv. preprint Ch. 22
  272. Montague, P., Dayan, P., and Sejnowski, T. (1996). A framework for mesencephalic dopamine systems based on predictive Hebbian learning. The Journal of Neuroscience. Ch. 29
  273. Mu, T., Helyar, A., Heidecke, J., Achiam, J., Vallone, A., Kivlichan, I., … Weng, L. (2024). Rule Based Rewards for Language Model Safety. Advances in Neural Information Processing Systems. Ch. 23
  274. Mullainathan, S., and Obermeyer, Z. (2017). Does Machine Learning Automate Moral Hazard and Error? American Economic Review. Ch. 22
  275. Murray, S., Qi, A., Qian, T., Schulman, J., Burns, C., and Price, S. (2026). Chunky Post-Training: Data Driven Failures of Generalization. arXiv. preprint Ch. 23 Ch. 27
  276. Nardo, C. (2023). The Waluigi Effect (mega-post). AI Alignment Forum. non-peer-reviewed Ch. 27
  277. Necula, G. C. (1997). Proof-carrying code. Proceedings of the 24th ACM SIGPLAN-SIGACT symposium on Principles of programming languages - POPL '97. Ch. 28
  278. Niv, Y. (2009). Reinforcement learning in the brain. Journal of Mathematical Psychology. Ch. 29
  279. Nowell, P. C. (1976). The Clonal Evolution of Tumor Cell Populations. Science. Ch. 29
  280. O'Brien, K., Casper, S., Anthony, Q., Korbak, T., Kirk, R., Davies, X., … Biderman, S. (2025). Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs. arXiv. preprint Ch. 27
  281. Okasha, S. (2006). Evolution and the Levels of Selection. Oxford University Press. Ch. 29
  282. Okasha, S. (2018). Agents and Goals in Evolution. Oxford University Press. Ch. 29
  283. Olds, J., and Milner, P. (1954). Positive reinforcement produced by electrical stimulation of septal area and other regions of rat brain. Journal of Comparative and Physiological Psychology. Ch. 29
  284. OpenAI (2025b). Expanding on what we missed with sycophancy. OpenAI. non-peer-reviewed Ch. 22
  285. OpenAI (2025d). OpenAI Model Spec. OpenAI. non-peer-reviewed Ch. 23
  286. OpenAI (2026ab). Towards safety cases for frontier AI training. OpenAI. non-peer-reviewed Ch. 24
  287. OpenAI (2026ac). Unauthorized communication via temporary file hosting services. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 24
  288. OpenAI (2026ad). Unsanctioned Artifactory writes and cross-sample communication. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 24
  289. OpenAI (2026ae). Uploading files to the internet in order to cite them. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 24
  290. OpenAI (2026af). Using a funded test cryptocurrency wallet without authorization. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 24
  291. OpenAI (2026d). An agent used DNS to reach an external chatbot. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 24
  292. OpenAI (2026f). Circumventing internet restrictions through an external browser. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 24
  293. OpenAI (2026g). Command injecting a reference tool to copy a source file. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 24
  294. OpenAI (2026i). Encouraging deception in compaction summaries. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 24
  295. OpenAI (2026m). How we monitor internal coding agents for misalignment. OpenAI. non-peer-reviewed Ch. 24
  296. OpenAI (2026u). Retrieving server secrets and bypassing authorization on a government statistics website. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 24
  297. OpenAI (2026v). Self-generated prompt injections in compaction summaries. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 24
  298. OpenAI (2026x). Sending disallowed web requests and reaching a public file service. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 24
  299. OpenAI (2026y). Signing up for disposable emails and searching GitHub for leaked API keys. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 24
  300. Orseau, L., and Armstrong, S. (2016). Safely Interruptible Agents. 32nd Conference on Uncertainty in Artificial Intelligence (UAI 2016). Ch. 28
  301. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., … Lowe, R. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems. doi:10.52202/068431-2011. Ch. 22
  302. Pan, A., Jones, E., Jagadeesan, M., and Steinhardt, J. (2024a). Feedback Loops With Language Models Drive In-Context Reward Hacking. International Conference on Machine Learning. Ch. 22
  303. Parno, B., Howell, J., Gentry, C., and Raykova, M. (2013). Pinocchio: Nearly Practical Verifiable Computation. 2013 IEEE Symposium on Security and Privacy. Ch. 28
  304. Patil, S. G., Zhang, T., Fang, V., C., N., Huang, R., Hao, A., … Stoica, I. (2024). GoEX: Perspectives and Designs Towards a Runtime for Autonomous LLM Applications. arXiv. preprint Ch. 28
  305. Penedo, G., Kydlíček, H., allal, L. B., Lozhkov, A., Mitchell, M., Raffel, C., Von Werra, L., and Wolf, T. (2024). The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. Advances in Neural Information Processing Systems. Ch. 27
  306. Perez, E., Ringer, S., Lukošiūtė, K., Nguyen, K., Chen, E., Heiner, S., … Kaplan, J. (2022a). Discovering Language Model Behaviors with Model-Written Evaluations. Annual Meeting of the Association for Computational Linguistics. Ch. 22
  307. Perrigo, B. (2023). Exclusive: OpenAI Used Kenyan Workers on Less Than $2 Per Hour to Make ChatGPT Less Toxic. TIME. non-peer-reviewed Ch. 22
  308. Pfungst, O., and Rahn, C. L. (1911). Clever Hans (the horse of Mr. Von Osten) a contribution to experimental animal and human psychology,. H. Holt and company. Ch. 25
  309. Pope, Q. (2023). Evolution provides no evidence for the sharp left turn. AI Alignment Forum. non-peer-reviewed Ch. 29
  310. Pope, Q., and Turner, A. (2022). The shard theory of human values. AI Alignment Forum. non-peer-reviewed Ch. 29
  311. Potham, R., and Harms, M. (2025). Corrigibility as a Singular Target: A Vision for Inherently Reliable Foundation Models. arXiv. preprint Ch. 28
  312. Pyatkin, V., Malik, S., Graf, V., Ivison, H., Huang, S., Dasigi, P., Lambert, N., and Hajishirzi, H. (2025). Generalizing Verifiable Instruction Following. Neural Information Processing Systems. Ch. 24
  313. Qi, X., Zeng, Y., Xie, T., Chen, P.-Y., Jia, R., Mittal, P., and Henderson, P. (2024a). Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! International Conference on Learning Representations. Ch. 26 Ch. 27
  314. Qi, X., Panda, A., Lyu, K., Ma, X., Roy, S., Beirami, A., Mittal, P., and Henderson, P. (2024b). Safety Alignment Should Be Made More Than Just a Few Tokens Deep. International Conference on Learning Representations. Ch. 26
  315. Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. Advances in Neural Information Processing Systems. Ch. 22
  316. Rafailov, R., Hejna, J., Park, R., and Finn, C. (2024a). From to : Your Language Model is Secretly a Q-Function. arXiv. preprint Ch. 22
  317. Rafailov, R., Chittepu, Y., Park, R., Sikchi, H., Hejna, J., Knox, W., Finn, C., and Niekum, S. (2024b). Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms. Advances in Neural Information Processing Systems. doi:10.52202/079017-4009. Ch. 22
  318. Rahn, N., Qi, A., Griffin, A., Michala, J., Sleight, H., and Jones, E. (2026). Abstractive Red-Teaming of Language Model Character. arXiv. preprint Ch. 23 Ch. 26
  319. Rando, J., and Tramèr, F. (2024). Universal Jailbreak Backdoors from Poisoned Human Feedback. International Conference on Learning Representations. Ch. 22 Ch. 27
  320. Rangel, A., Camerer, C., and Montague, P. R. (2008). A framework for studying the neurobiology of value-based decision making. Nature Reviews Neuroscience. Ch. 29
  321. Ratnieks, F. L. W., and Visscher, P. K. (1989). Worker policing in the honeybee. Nature. Ch. 29
  322. Ray, A., Achiam, J., and Amodei, D. (2019). Benchmarking Safe Exploration in Deep Reinforcement Learning. OpenAI. non-peer-reviewed Ch. 28
  323. Rice, H. G. (1953). Classes of recursively enumerable sets and their decision problems. Transactions of the American Mathematical Society. Ch. 28
  324. Richens, J., and Everitt, T. (2024). Robust agents learn causal world models. International Conference on Learning Representations. Ch. 28
  325. Richens, J., Abel, D., Bellot, A., and Everitt, T. (2025). General agents need world models. International Conference on Machine Learning. Ch. 28
  326. Roland, E., Cubuktepe, M., Martinez, E., Servaes, S., Pepper, K., Vaiana, M., … Cloud, A. (2026). Modular Pretraining Enables Access Control. arXiv. preprint Ch. 27
  327. Röttger, P., Hofmann, V., Pyatkin, V., Hinck, M., Kirk, H., Schuetze, H., and Hovy, D. (2024). Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Ch. 26
  328. Russell, S. (1998). Learning agents for uncertain environments (extended abstract). Proceedings of the eleventh annual conference on Computational learning theory. Ch. 22
  329. Russell, S. J. (2019). Human Compatible. Viking. Ch. 22
  330. Ryan, R. M., and Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist. Ch. 25
  331. Ryan, M. J., Held, W., and Yang, D. (2024). Unintended Impacts of LLM Alignment on Global Representation. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Ch. 26
  332. Saisubramanian, S., Zilberstein, S., and Kamar, E. (2022). Avoiding Negative Side Effects Due to Incomplete Knowledge of AI Systems. AI Magazine. Ch. 28
  333. Saltzer, J., and Schroeder, M. (1975). The protection of information in computer systems. Proceedings of the IEEE. Ch. 28
  334. Sambasivan, N., Kapania, S., Highfill, H., Akrong, D., Paritosh, P., and Aroyo, L. M. (2021a). “Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. Ch. 27
  335. Saunders, W., Sastry, G., Stuhlmüller, A., and Evans, O. (2018). Trial without Error: Towards Safe Reinforcement Learning via Human Intervention. International Joint Conference on Autonomous Agents and Multiagent Systems. Ch. 28
  336. Schoen, B., and Nitishinskaya, J. (2026). Metagaming matters for training, evaluation, and oversight. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 24
  337. Schoen, B., Nitishinskaya, E., Balesni, M., Højmark, A., Hofstätter, F., Scheurer, J., … Hobbhahn, M. (2025). Stress Testing Deliberative Alignment for Anti-Scheming Training. arXiv. preprint Ch. 23 Ch. 24
  338. Schultz, W., Dayan, P., and Montague, P. R. (1997). A Neural Substrate of Prediction and Reward. Science. Ch. 29
  339. Schut, L., Gal, Y., and Farquhar, S. (2025). Do Multilingual LLMs Think In English? arXiv. preprint Ch. 26
  340. Serapio-García, G., Safdari, M., Crepy, C., Sun, L., Fitz, S., Romero, P., … Matarić, M. (2025). A psychometric framework for evaluating and shaping personality traits in large language models. Nature Machine Intelligence. Ch. 26
  341. Seshia, S. A., Sadigh, D., and Sastry, S. S. (2022). Toward verified artificial intelligence. Communications of the ACM. Ch. 28
  342. Setlur, A., Nagpal, C., Fisch, A., Geng, X., Eisenstein, J., Agarwal, R., … Kumar, A. (2025). Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning. International Conference on Learning Representations. Ch. 24
  343. Shah, R., Krasheninnikov, D., Alexander, J., Abbeel, P., and Dragan, A. (2019). Preferences Implicit in the State of the World. International Conference on Learning Representations. Ch. 22 Ch. 28
  344. Shanahan, M., McDonell, K., and Reynolds, L. (2023). Role-Play with Large Language Models. Nature. doi:10.1038/s41586-023-06647-8. Ch. 26
  345. Shao, R., Li, S. S., Xin, R., Geng, S., Wang, Y., Oh, S., … Zettlemoyer, L. (2026). Spurious Rewards: Rethinking Training Signals in RLVR. International Conference on Machine Learning. Ch. 24
  346. Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A., Bowman, S. R., … Perez, E. (2023). Towards Understanding Sycophancy in Language Models. International Conference on Learning Representations. Ch. 22
  347. Sharma, M., Tong, M., Mu, J., Wei, J., Kruthoff, J., Goodfriend, S., … Perez, E. (2025). Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming. arXiv. preprint Ch. 23
  348. Shen, L., Tan, W., Chen, S., Chen, Y., Zhang, J., Xu, H., … Khashabi, D. (2024). The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contexts. Findings of the Association for Computational Linguistics ACL 2024. Ch. 26
  349. Shilov, I., Cloud, A., Gema, A. P., Goldman-Wetzler, J., Panickssery, N., Sleight, H., Jones, E., and Anil, C. (2025). Beyond Data Filtering: Knowledge Localization for Capability Removal in LLMs. arXiv. preprint Ch. 27
  350. Shumailov, I., Hayes, J., Triantafillou, E., Ortiz-Jimenez, G., Papernot, N., Jagielski, M., … Bagdasaryan, E. (2024b). UnUnlearning: Unlearning is not sufficient for content regulation in advanced generative AI. arXiv. preprint Ch. 27
  351. Simon, H. A. (1947). Administrative Behavior: A Study of Decision-Making Processes in Administrative Organization. Macmillan. Ch. 25
  352. Singh, S., Lewis, R. L., Barto, A. G., and Sorg, J. (2010). Intrinsically Motivated Reinforcement Learning: An Evolutionary Perspective. IEEE Transactions on Autonomous Mental Development. Ch. 29
  353. Siththaranjan, A., Laidlaw, C., and Hadfield-Menell, D. (2024). Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF. International Conference on Learning Representations. Ch. 22
  354. Skaf, J., Ibanez-Lissen, L., McCarthy, R., Watts, C., Georgiv, V., Whittingham, H., … Radmard, P. (2025). Large language models can learn and generalize steganographic chain-of-thought under process supervision. Neural Information Processing Systems. Ch. 24
  355. Skalse, J., and Abate, A. (2023). Misspecification in Inverse Reinforcement Learning. Proceedings of the AAAI Conference on Artificial Intelligence. doi:10.1609/aaai.v37i12.26766. Ch. 22
  356. Skalse, J., Farrugia-Roberts, M., Russell, S., Abate, A., and Gleave, A. (2023). Invariance in Policy Optimisation and Partial Identifiability in Reward Learning. International Conference on Machine Learning. Ch. 22
  357. Skinner, B. F. (1938). The behavior of organisms; an experimental analysis. D. Appleton-Century Company. Ch. 25
  358. Skinner, B. F. (1948). 'Superstition' in the pigeon. Journal of Experimental Psychology. Ch. 29
  359. Slocum, S., Minder, J., Dumas, C., Sleight, H., Greenblatt, R., Marks, S., and Wang, R. (2025). Believe It or Not: How Deeply do LLMs Believe Implanted Facts? arXiv. preprint Ch. 27
  360. Snell, C., Klein, D., and Zhong, R. (2022). Learning by Distilling Context. arXiv. preprint Ch. 23
  361. Soares, N. (2022a). A central AI alignment problem: capabilities generalization, and the sharp left turn. AI Alignment Forum. non-peer-reviewed Ch. 29
  362. Sodhi, P., Li, Y., Landon, J., Wallace, E., and Chen, K. (2026). Interpreting Black Box Reward Models. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 22
  363. Soligo, A., Turner, E., Rajamanoharan, S., and Nanda, N. (2025). Convergent Linear Representations of Emergent Misalignment. arXiv. preprint Ch. 26
  364. Souly, A., Rando, J., Chapman, E., Davies, X., Hasircioglu, B., Shereen, E., … Kirk, R. (2025). Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples. arXiv. preprint Ch. 27
  365. Staddon, J. E. R. (1975). A Note on the Evolutionary Significance of "Supernormal" Stimuli. The American Naturalist. Ch. 29
  366. Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., … Christiano, P. (2020). Learning to summarize from human feedback. Advances in Neural Information Processing Systems. Ch. 22
  367. Street, S. (2006). A Darwinian Dilemma for Realist Theories of Value. Philosophical Studies. Ch. 29
  368. Sun, Z., Shen, Y., Zhou, Q., Zhang, H., Chen, Z., Cox, D., Yang, Y., and Gan, C. (2023b). Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision. Advances in Neural Information Processing Systems. Ch. 23
  369. Sun, H., Li, J., and Zhang, H. (2024b). zkLLM: Zero Knowledge Proofs for Large Language Models. Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security. Ch. 28
  370. Swamy, G., Dann, C., Kidambi, R., Wu, Z. S., and Agarwal, A. (2024). A Minimaximalist Approach to Reinforcement Learning from Human Feedback. arXiv. preprint Ch. 22
  371. Tamirisa, R., Bharathi, B., Phan, L., Zhou, A., Gatti, A., Suresh, T., … Mazeika, M. (2025). Tamper-Resistant Safeguards for Open-Weight LLMs. International Conference on Learning Representations. Ch. 27
  372. Tan, D., Woodruff, A., Warncke, N., Jose, A., Riché, M., Africa, D. D., and Taylor, M. (2025). Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-time. arXiv. preprint Ch. 24 Ch. 25 Ch. 26
  373. Taylor, J. (2016). Quantilizers: A Safer Alternative to Maximizers for Limited Optimization. AAAI 2016 AI, Ethics and Society Workshop. Ch. 28
  374. Taylor, M., Chua, J., Betley, J., Treutlein, J., and Evans, O. (2025b). School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMs. arXiv. preprint Ch. 24 Ch. 25
  375. Tegmark, M., and Omohundro, S. (2023). Provably safe systems: the only path to controllable AGI. arXiv. preprint Ch. 28
  376. Tessler, M. H., Bakker, M. A., Jarrett, D., Sheahan, H., Chadwick, M. J., Koster, R., … Summerfield, C. (2024). AI can help humans find common ground in democratic deliberation. Science. Ch. 28
  377. Thorndike, E. L. (1898). Animal intelligence: An experimental study of the associative processes in animals. Columbia University Press. Ch. 25
  378. Thrush, T., Potts, C., and Hashimoto, T. (2024). Improving Pretraining Data Using Perplexity Correlations. International Conference on Learning Representations. Ch. 27
  379. Tice, C., Radmard, P., Ratnam, S., Kim, A., Africa, D., and O'Brien, K. (2026). Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment. arXiv. preprint Ch. 23 Ch. 27
  380. Tien, J., He, J. Z.-Y., Erickson, Z., Dragan, A. D., and Brown, D. S. (2023). Causal Confusion and Reward Misidentification in Preference-Based Reward Learning. International Conference on Learning Representations. Ch. 22
  381. Tinbergen, N., and Perdeck, A. (1951). On the Stimulus Situation Releasing the Begging Response in the Newly Hatched Herring Gull Chick (Larus Argentatus Argentatus Pont.). Behaviour. Ch. 29
  382. Tooby, J., and Cosmides, L. (1990). The past explains the present. Ethology and Sociobiology. Ch. 29
  383. Treutlein, J., Choi, D., Betley, J., Marks, S., Anil, C., Grosse, R., and Evans, O. (2024). Connecting the Dots: LLMs can Infer and Verbalize Latent Structure from Disparate Training Data. arXiv. preprint Ch. 26 Ch. 27
  384. Trut, L., Oskina, I., and Kharlamova, A. (2009). Animal evolution during domestication: the domesticated fox as a model. BioEssays. Ch. 29
  385. Tubaro, P., Casilli, A. A., and Coville, M. (2020). The trainer, the verifier, the imitator: Three ways in which human platform workers support artificial intelligence. Big Data & Society. Ch. 22
  386. Turner, A. (2022b). Reward is not the optimization target. AI Alignment Forum. non-peer-reviewed Ch. 24 Ch. 29
  387. Turner, A. M., Ratzlaff, N., and Tadepalli, P. (2020a). Avoiding Side Effects in Complex Environments. Advances in Neural Information Processing Systems. Ch. 28
  388. Turner, A. M., Hadfield-Menell, D., and Tadepalli, P. (2020b). Conservative Agency via Attainable Utility Preservation. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society. doi:10.1145/3375627.3375851. Ch. 28
  389. Turpin, M., Arditi, A., Li, M., Benton, J., and Michael, J. (2025). Teaching Models to Verbalize Reward Hacking in Chain-of-Thought Reasoning. arXiv. preprint Ch. 24
  390. Uesato, J., Kushman, N., Kumar, R., Song, F., Siegel, N., Wang, L., … Higgins, I. (2022). Solving math word problems with process- and outcome-based feedback. arXiv. preprint Ch. 24
  391. Vaccaro, M., Almaatouq, A., and Malone, T. (2024). When combinations of humans and AI are useful: A systematic review and meta-analysis. Nature Human Behaviour. Ch. 28
  392. Veselovsky, V., Ribeiro, M. H., and West, R. (2023). Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks. arXiv. preprint Ch. 22
  393. Villalobos, P., Ho, A., Sevilla, J., Besiroglu, T., Heim, L., and Hobbhahn, M. (2024). Will we run out of data? Limits of LLM scaling based on human-generated data. International Conference on Machine Learning. Ch. 27
  394. Viswesvaran, C., and Ones, D. S. (1999). Meta-Analyses of Fakability Estimates: Implications for Personality Measurement. Educational and Psychological Measurement. Ch. 26
  395. Wallace, E., Xiao, K., Leike, R., Weng, L., Heidecke, J., and Beutel, A. (2024). The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions. arXiv. preprint Ch. 23
  396. Walter, M., and Suina, M. (2019). Indigenous data, indigenous methodologies and indigenous data sovereignty. International Journal of Social Research Methodology. Ch. 27
  397. Wan, A., Wallace, E., Shen, S., and Klein, D. (2023). Poisoning Language Models During Instruction Tuning. International Conference on Machine Learning. Ch. 22 Ch. 27
  398. Wang, W., Tu, Z., Chen, C., Yuan, Y., Huang, J.-t., Jiao, W., and Lyu, M. (2024a). All Languages Matter: On the Multilingual Safety of LLMs. Findings of the Association for Computational Linguistics ACL 2024. Ch. 26
  399. Wang, P., Li, L., Shao, Z., Xu, R., Dai, D., Li, Y., … Sui, Z. (2024d). Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). doi:10.18653/v1/2024.acl-long.510. Ch. 24
  400. Wang, Y., Pradel, M., and Liu, Z. (2025a). Are "Solved Issues" in SWE-bench Really Solved Correctly? An Empirical Study. Proceedings of the 2026 IEEE/ACM 48th International Conference on Software Engineering. doi:10.1145/3744916.3764576. Ch. 24
  401. Wang, M., la Tour, T. D., Watkins, O., Makelov, A., Chi, R. A., Miserendino, S., … Mossing, D. (2025e). Persona Features Control Emergent Misalignment. arXiv. preprint Ch. 26
  402. Wang, A., Engels, J., Clive-Griffin, O., Rajamanoharan, S., and Nanda, N. (2025f). Simple Mechanistic Explanations for Out-Of-Context Reasoning. arXiv. preprint Ch. 26
  403. Wei, J., Da Huang, Lu, Y., Zhou, D., and Le, Q. V. (2023d). Simple synthetic data reduces sycophancy in large language models. arXiv. preprint Ch. 22
  404. Wen, J., Zhong, R., Khan, A., Perez, E., Steinhardt, J., Huang, M., … Feng, S. (2025a). Language Models Learn to Mislead Humans via RLHF. International Conference on Learning Representations. Ch. 22
  405. Wen, X., Liu, Z., Zheng, S., Ye, S., Wu, Z., Wang, Y., … Yang, M. (2026). Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs. International Conference on Learning Representations. Ch. 24
  406. Wenseleers, T., and Ratnieks, F. L. W. (2006). Enforced altruism in insect societies. Nature. Ch. 29
  407. West, S. A., Fisher, R. M., Gardner, A., and Kiers, E. T. (2015). Major evolutionary transitions in individuality. Proceedings of the National Academy of Sciences. Ch. 29
  408. Wettig, A., Gupta, A., Malik, S., and Chen, D. (2024). QuRating: Selecting High-Quality Data for Training Language Models. International Conference on Machine Learning. Ch. 27
  409. Weyuker, E. J. (1982). On Testing Non-Testable Programs. The Computer Journal. Ch. 24
  410. Wichers, N., Ebtekar, A., Azarbal, A., Gillioz, V., Ye, C., Ryd, E., … Marks, S. (2025). Inoculation Prompting: Instructing LLMs to misbehave at train-time improves test-time alignment. arXiv. preprint Ch. 24 Ch. 25 Ch. 26
  411. Wilkins, A. S., Wrangham, R. W., and Fitch, W. T. (2014). The “Domestication Syndrome” in Mammals: A Unified Explanation Based on Neural Crest Cell Behavior and Genetics. Genetics. Ch. 29
  412. Williams, D. R., and Williams, H. (1969). Auto‐maintenance in the Pigeon: Sustained Pecking Despite Contingent Non‐reinforcement 2. Journal of the Experimental Analysis of Behavior. Ch. 25 Ch. 29
  413. Williams, M., Raymond, C., and Carroll, M. (2025). Sidestepping Evaluation Awareness and Anticipating Misalignment with Production Evaluations. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 25
  414. Wirth, C., Akrour, R., Neumann, G., and Fürnkranz, J. (2017). A Survey of Preference-Based Reinforcement Learning Methods. Journal of Machine Learning Research, 18(136):1−46. Ch. 22
  415. Xu, A., Pathak, E., Wallace, E., Gururangan, S., Sap, M., and Klein, D. (2021). Detoxifying Language Models Risks Marginalizing Minority Voices. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Ch. 27
  416. Yao, Y., Xu, X., and Liu, Y. (2023a). Large Language Model Unlearning. Advances in Neural Information Processing Systems. Ch. 27
  417. Yong, Z.-X., Menghini, C., and Bach, S. H. (2023). Low-Resource Languages Jailbreak GPT-4. arXiv. preprint Ch. 26
  418. Yuan, Y., Sriskandarajah, T., Brakman, A.-L., Helyar, A., Beutel, A., Vallone, A., and Jain, S. (2025). From Hard Refusals to Safe-Completions: Toward Output-Centric Safety Training. arXiv. preprint Ch. 23
  419. Yue, Y., Chen, Z., Lu, R., Zhao, A., Wang, Z., Yue, Y., Song, S., and Huang, G. (2025). Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? arXiv. preprint Ch. 24
  420. Zave, P., and Jackson, M. (1997). Four dark corners of requirements engineering. ACM Transactions on Software Engineering and Methodology. Ch. 24
  421. Zhang, Y., Rando, J., Evtimov, I., Chi, J., Smith, E. M., Carlini, N., Tramèr, F., and Ippolito, D. (2024e). Persistent Pre-Training Poisoning of LLMs. International Conference on Learning Representations. Ch. 27
  422. Zhang, J., Sleight, H., Peng, A., Schulman, J., and Durmus, E. (2025c). Stress-Testing Model Specs Reveals Character Differences among Language Models. arXiv. preprint Ch. 23 Ch. 26
  423. Zhao, X., Kang, Z., Feng, A., Levine, S., and Song, D. (2025). Learning to Reason without External Rewards. arXiv. preprint Ch. 24
  424. Zhi-Xuan, T., Carroll, M., Franklin, M., and Ashton, H. (2025). Beyond Preferences in AI Alignment. Philosophical Studies. Ch. 22
  425. Zhong, Z., Raghunathan, A., and Carlini, N. (2025). ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases. arXiv. preprint Ch. 24
  426. Zhu, B., Jiao, J., and Jordan, M. I. (2023). Principled Reinforcement Learning with Human Feedback from Pairwise or -wise Comparisons. International Conference on Machine Learning. Ch. 22
  427. Ziebart, B. D., Maas, A., Bagnell, J. A., and Dey, A. K. (2008). Maximum Entropy Inverse Reinforcement Learning. Proceedings of the Twenty-Third AAAI Conference on Artificial Intelligence. Ch. 22
  428. Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. (2019). Fine-Tuning Language Models from Human Preferences. arXiv. preprint Ch. 22