AI Alignment
Part III: Specification: Saying What We Want

Specification: Saying What We Want

In brief

Ideas
  • Whose values? The arguments do not decide whether the target is a moral fact, a fair procedure's output, or a choice someone must justify, and each is a different engineering problem (Ch. 16).
  • No rule for combining many people's inputs meets every reasonable condition, and preference training already aggregates its raters, by the Borda count under a theorem's conditions (Ch. 17).
  • A specification is an incomplete contract: the model fills its gaps without appeal, and its drafter is one of the parties it ranks (Ch. 19).
  • Helpfulness, honesty, harmlessness, and corrigibility each hide a choice and conflict in pairs; corrigibility moves the question of trust from the model to whoever holds its controls (Ch. 20).
Evidence
  • Settled: models' answers to value surveys move with answer order, labels, wording, and format, and raters' disagreement carries structure that a single aggregated label discards (Ch. 18).
  • Settled: as mathematics under their assumptions, agents meeting weak conditions often try to prevent or cause shutdown, and uncertainty gives a robot reason to defer to a rational human, which misspecification can remove (§20.4).
  • Reported: a developer measured its models' compliance with its published specification at 72 to 89 percent on everyday prompts, with its own grader (§19.2).
Open
  • Crux: do a model's measured values predict its choices where no one framed a test? If not, their use to predict deployment behavior is unsupported (§18.10).
  • Missing: as of October 2026, a measurement by a group with no stake in any developer of how closely deployed models follow their published specifications, and a validated test for consciousness in systems unlike animals (§19.9; §21.9).

This part takes up the first of the four problems. It asks what kind of thing the target is and whose values it should hold, drawing on philosophy and on bioethics's standards for deciding for others (Chapter 16); how many people's inputs can be combined, what social choice theory proves about any such combination, and what makes the process legitimate, with each published exercise in public input rated by what it transferred (Chapter 17); what the measurements of human values and the labor of raters actually capture (Chapter 18); how a target is written down, read as an incomplete contract with requirements engineering, law, linguistics, public administration, and military command (Chapter 19); what helpfulness, honesty, harmlessness, and corrigibility each commit a designer to, and when an obedient agent should refuse (Chapter 20); and whether the agent itself has interests that a specification must count (Chapter 21). Its questions apply to all three relationships the word alignment covers: the signal a training process optimizes, what a deployed system owes the people it serves and those it affects, and the specification a laboratory writes for the public it serves.

The part comes after the foundations because it recalls their results and arguments rather than restating them: that, with unlimited data and every pair compared equally often, one reward fitted to many raters' comparisons ranks responses by their Borda count (Section 13.5), that preferences are often made at the moment they are asked for (Section 13.6), that a contract which cannot list every case leaves someone holding residual control (Section 14.3), and that several principals pulling on one agent reach an equilibrium that can ignore everyone outside it (Section 14.4). It comes before instillation because every training method in the next part optimizes some written or measured target, and many of the failures in Part V begin as gaps in that target. The problems are coupled, not sequential: what can be verified limits what can usefully be specified, since organizations come to specify what they can audit, and incidents found in deployment are fed back into specifications, so this part is revisited by Chapter 69.

The part assumes the Bradley-Terry model and reward identifiability (Chapter 13), expected utility (Chapter 7), and incomplete contracts and common agency (Chapter 14); it assumes no philosophy, law, or social science, and glosses each term where it first appears.

Chapters in this part

  1. 16 Whose Values?

    Before anyone can train toward human values, someone must say which values, held by whom, and on what grounds. This chapter sets out what philosophy says the target could be (a fact to discover, the output of a fair procedure, or a choice someone must justify), how decision theory handles uncertainty about which moral theory is right, what bioethics learned from mid-level principles and from deciding for people who cannot state their wishes, what other ethical traditions and the critics of the question itself would change, and whose preferences count when a system changes them. The arguments do not decide between the main positions; the chapter ends on whether any part of the choice could be left to capable systems.

  2. 17 Aggregation and Legitimacy

    A model serves many people who want different things, so its target must be assembled from many inputs. This chapter teaches what social choice theory proves about any such assembly, shows which voting rule preference training already applies and what the alternatives give up, treats fairness as aggregation without a neutral answer, asks whose population a target is drawn from, and asks what makes the process that sets a model's behavior legitimate, rating each published public-input exercise by the authority it transferred and setting out the case against participation as a remedy.

  3. 18 Measuring What People Value

    Every specification is eventually turned into data: survey answers, ratings, pairwise choices, labels. This chapter asks what those measurements capture. It covers the structure of human values and the rival theories behind moral questionnaires, the invariance an instrument needs before two groups can be compared, when a rater's judgment tracks the property it is asked about, disagreement among raters as information rather than noise, the raters as workers scored on what can be counted, how closely any model can predict one rater when raters disagree with each other, and what survey and psychometric instruments show when the respondent is a model.

  4. 19 Specifications as Contracts

    A model specification is a document that tells a capable agent how to act in situations its authors did not foresee. Reading the two most detailed published ones, OpenAI's Model Spec and Anthropic's constitution, as incomplete contracts, this chapter separates requirements from specifications and the assumptions that connect them, compares how each ranks the developer, the deployer, and the user, weighs rules against standards and cases, and asks what an instruction means beyond its words. Requirements engineering, law, linguistics, public administration, military command, and the sociology of work each supply a mechanism and a limit; one developer's own evaluation and one stress test by researchers affiliated with another developer supply the evidence on how the documents fail.

  5. 20 The Target Properties

    Developers describe the assistant they want with a few words: helpful, honest, harmless, and, for safety, corrigible (willing to be corrected or shut down). This chapter asks what each word commits a designer to and what older fields know about it: pandering to a principal, the acts that honesty governs and the line between lying and misleading, harm instruments built without the people whose harms they miss, the decision theory of shutdown, and the law of superior orders. It derives the off-switch game with the assumptions it needs, sets the properties against each other, and ends on whether one system can accept correction and still refuse an order that is manifestly wrong.

  6. 21 Does the Agent Have Interests?

    If some AI systems have experiences or interests of their own, specification must count them, and the tools of safety (monitoring, modification, shutdown) become choices about a being that may matter. This chapter separates consciousness, sentience, welfare, and moral status; reports what theories of consciousness say about current systems; borrows the methods comparative cognition and animal welfare science use for minds that cannot report, and the problem those methods meet in systems trained on human descriptions of experience; weighs the thin evidence from models themselves, much of it reported by their developers; and sets the conflict between welfare and control out measure by measure. The evidence does not decide whether any current system is a moral patient.

References for Part III

348 works cited across this part's chapters.

  1. Abiri, G. (2024). Public Constitutional AI. Elsevier BV. preprint Ch. 17 Ch. 19
  2. Ames, R. T. (2011). Confucian Role Ethics. The Chinese University Press. Ch. 16
  3. Anthropic (2025ap). System Card: Claude Opus 4 & Claude Sonnet 4. Anthropic. non-peer-reviewed Ch. 21
  4. Anthropic (2025p). Claude Opus 4 and 4.1 can now end a rare subset of conversations. Anthropic. non-peer-reviewed Ch. 21
  5. Anthropic (2025q). Commitments on model deprecation and preservation. Anthropic. non-peer-reviewed Ch. 21
  6. Anthropic (2025w). Exploring model welfare. Anthropic. non-peer-reviewed Ch. 21
  7. Anthropic (2026au). The persona selection model. Anthropic. non-peer-reviewed Ch. 21
  8. Anthropic (2026g). An update on our model deprecation commitments for Claude Opus 3. Anthropic. non-peer-reviewed Ch. 21
  9. Anthropic (2026r). Claude's Constitution. Anthropic. non-peer-reviewed Ch. 19 Ch. 20 Ch. 21
  10. Armstrong, S., and Mindermann, S. (2018). Occam's razor is insufficient to infer the preferences of irrational agents. Advances in Neural Information Processing Systems. Ch. 18
  11. Armstrong, S., and O'Rourke, X. (2017). 'Indifference' methods for managing agent rewards. arXiv. preprint Ch. 20
  12. Arnstein, S. R. (1969). A Ladder Of Citizen Participation. Journal of the American Institute of Planners. Ch. 17
  13. Aroyo, L., and Welty, C. (2015). Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation. AI Magazine. Ch. 18
  14. Arrow, K. J. (1951). Social Choice and Individual Values. Yale University Press. doi:10.12987/9780300186987. Ch. 17
  15. Aru, J., Larkum, M. E., and Shine, J. M. (2023). The feasibility of artificial consciousness through the lens of neuroscience. Trends in Neurosciences. Ch. 21
  16. Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., … Kaplan, J. (2021). A General Language Assistant as a Laboratory for Alignment. arXiv. preprint Ch. 20
  17. Atari, M., Haidt, J., Graham, J., Koleva, S., Stevens, S. T., and Dehghani, M. (2023a). Morality beyond the WEIRD: How the nomological network of morality varies across cultures. Journal of Personality and Social Psychology. Ch. 18
  18. Atari, M., Xue, M. J., Park, P. S., Blasi, D. E., and Henrich, J. (2023b). Which Humans? Center for Open Science. preprint Ch. 17
  19. Ayres, I., and Gertner, R. (1989). Filling Gaps in Incomplete Contracts: An Economic Theory of Default Rules. The Yale Law Journal. Ch. 19
  20. Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., … Kaplan, J. (2022b). Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv. preprint Ch. 18 Ch. 20
  21. Bales, A. (2025). Against willing servitude: Autonomy in the ethics of advanced artificial intelligence. The Philosophical Quarterly. Ch. 20 Ch. 21
  22. Bawn, K. (1995). Political Control Versus Expertise: Congressional Choices about Administrative Procedures. American Political Science Review. Ch. 19
  23. Bayern, S. (2016). The Implications of Modern Business–Entity Law for the Regulation of Autonomous Systems. European Journal of Risk Regulation. Ch. 21
  24. Beauchamp, T. L. (1994). Principles of biomedical ethics. New York : Oxford University Press. Ch. 16
  25. Beauchamp, T., and Childress, J. (2019). Principles of Biomedical Ethics : Marking Its Fortieth Anniversary. The American Journal of Bioethics. Ch. 16
  26. Bender, E. M., and Koller, A. (2020). Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Ch. 19
  27. Bennett, C. L., and Keyes, O. (2020). What is the point of fairness? ACM SIGACCESS Accessibility and Computing. Ch. 17 Ch. 20
  28. Bergman, S., Marchal, N., Mellor, J., Mohamed, S., Gabriel, I., and Isaac, W. (2024). STELA: a community-centred approach to norm elicitation for AI alignment. Scientific Reports. Ch. 17
  29. Betley, J., Treutlein, J., Dubiński, J., Mayne, H., Gałązka, K., Warncke, N., Sztyber-Betley, A., and Evans, O. (2026b). Value Leakage: An LLM's Answers Are Silently Shaped by Its Own Values. arXiv. preprint Ch. 18
  30. Binder, F. J., Chua, J., Korbak, T., Sleight, H., Hughes, J., Long, R., … Evans, O. (2024). Looking Inward: Language Models Can Learn About Themselves by Introspection. arXiv. preprint Ch. 21
  31. Birch, J. (2017). Animal sentience and the precautionary principle. Animal Sentience. Ch. 21
  32. Birch, J. (2024). The Edge of Sentience. Oxford University PressOxford. Ch. 21
  33. Birhane, A., Isaac, W., Prabhakaran, V., Diaz, M., Elish, M. C., Gabriel, I., and Mohamed, S. (2022a). Power to the People? Opportunities and Challenges for Participatory AI. Equity and Access in Algorithms Mechanisms and Optimization. Ch. 17
  34. Block, N. (1995). On a confusion about a function of consciousness. Behavioral and Brain Sciences. Ch. 21
  35. Bostrom, N. (2012). The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents. Minds and Machines. Ch. 16
  36. Bovens, M., and Zouridis, S. (2002). From Street‐Level to System‐Level Bureaucracies: How Information and Communication Technology is Transforming Administrative Discretion and Constitutional Control. Public Administration Review. Ch. 19
  37. Broom, D. (1986). Indicators of poor welfare. British Veterinary Journal. Ch. 21
  38. Browning, H. (2022). Assessing measures of animal welfare. Biology & Philosophy. Ch. 21
  39. Browning, H., and Birch, J. (2022). Animal sentience. Philosophy Compass. Ch. 21
  40. Bryson, J. J. (2010). Robots should be slaves. Natural Language Processing. Ch. 21
  41. Bryson, J. J., Diamantis, M. E., and Grant, T. D. (2017). Of, for, and by the people: the legal lacuna of synthetic persons. Artificial Intelligence and Law. Ch. 21
  42. Buchanan, A. E., and Brock, D. W. (1990). Deciding for Others. Cambridge University Press. Ch. 16
  43. Buchanan, A., and Keohane, R. O. (2006). The Legitimacy of Global Governance Institutions. Ethics & International Affairs. Ch. 17
  44. Buckner, C. (2013). Morgan’s Canon, meet Hume’s Dictum: avoiding anthropofabulation in cross-species comparisons. Biology & Philosophy. Ch. 21
  45. Bullock, J. B. (2019). Artificial Intelligence, Discretion, and Bureaucracy. The American Review of Public Administration. Ch. 19
  46. Butlin, P., and Lappas, T. (2025). Principles for Responsible AI Consciousness Research. Journal of Artificial Intelligence Research. Ch. 21
  47. Butlin, P., Long, R., Elmoznino, E., Bengio, Y., Birch, J., Constant, A., … VanRullen, R. (2023). Consciousness in Artificial Intelligence: Insights from the Science of Consciousness. arXiv. preprint Ch. 21
  48. Cabitza, F., Campagner, A., and Basile, V. (2023). Toward a Perspectivist Turn in Ground Truthing for Predictive Computing. Proceedings of the AAAI Conference on Artificial Intelligence. Ch. 18
  49. Canes-Wrone, B., Herron, M. C., and Shotts, K. W. (2001). Leadership and Pandering: A Theory of Executive Policymaking. American Journal of Political Science. Ch. 20
  50. Caputo, N. (2026). Alignment as Jurisprudence. arXiv. preprint Ch. 19
  51. Carey, R. (2018). Incorrigibility in the CIRL Framework. Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. Ch. 20
  52. Carey, R., and Everitt, T. (2023). Human Control: Definitions and Algorithms. Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence (PMLR 216). Ch. 20
  53. Carroll, S. R., Garba, I., Figueroa-Rodríguez, O. L., Holbrook, J., Lovett, R., Materechera, S., … Hudson, M. (2020). The CARE Principles for Indigenous Data Governance. Data Science Journal. Ch. 17
  54. Carroll, M., Chan, A., Ashton, H., and Krueger, D. (2023). Characterizing Manipulation from AI Systems. Equity and Access in Algorithms, Mechanisms, and Optimization. Ch. 16
  55. Carroll, M., Foote, D., Siththaranjan, A., Russell, S., and Dragan, A. (2024). AI Alignment with Changing and Influenceable Reward Functions. International Conference on Machine Learning. Ch. 16
  56. Caviola, L., Sebo, J., and Birch, J. (2025). What will society think about AI consciousness? Lessons from the animal case. Trends in Cognitive Sciences. Ch. 21
  57. Chakraborty, S., Qiu, J., Yuan, H., Koppel, A., Huang, F., Manocha, D., Bedi, A. S., and Wang, M. (2024). MaxMin-RLHF: Alignment with Diverse Human Preferences. arXiv. preprint Ch. 17
  58. Chalmers, D. J. (2023). Could a Large Language Model be Conscious? Boston Review, August 9, 2023. Ch. 21
  59. Chouldechova, A. (2017). Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments. Big Data. Ch. 17
  60. Christiano, P. (2018). Clarifying “AI alignment”. AI Alignment (Medium). non-peer-reviewed Ch. 20
  61. Citron, D. K. (2008). Technological Due Process. Washington University Law Review. Ch. 19
  62. Clark, H. H. (1996). Using Language. Cambridge University Press. Ch. 19
  63. Coeckelbergh, M. (2010). Robot rights? Towards a social-relational justification of moral consideration. Ethics and Information Technology. Ch. 21
  64. Cogitate Consortium, Ferrante, O., Gorska-Klimowska, U., Henin, S., Hirschhorn, R., Khalaf, A., … Melloni, L. (2025). Adversarial testing of global neuronal workspace and integrated information theories of consciousness. Nature. Ch. 21
  65. Collingridge, D. (1980). The Social Control of Technology. London : Frances Pinter. Ch. 17
  66. Colombatto, C., and Fleming, S. M. (2024). Folk psychological attributions of consciousness to large language models. Neuroscience of Consciousness. Ch. 21
  67. Committee on the Rights of Persons with Disabilities (2014). General comment No. 1 (2014). Article 12: Equal recognition before the law. United Nations, CRPD/C/GC/1. non-peer-reviewed Ch. 16 Ch. 21
  68. Comsa, I. M., and Shanahan, M. (2025). Does It Make Sense to Speak of Introspection in Large Language Models? arXiv. preprint Ch. 21
  69. Conitzer, V., Freedman, R., Heitzig, J., Holliday, W. H., Jacobs, B. M., Lambert, N., … Zwicker, W. S. (2024). Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback. arXiv. preprint Ch. 17
  70. Cooke, B., and Kothari, U. (2001). Participation: The New Tyranny? Zed Books. Ch. 17
  71. Corbett, E., Denton, R., and Erete, S. (2023). Power and Public Participation in AI. Equity and Access in Algorithms Mechanisms and Optimization. Ch. 17
  72. Cronbach, L. J., and Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin. Ch. 18
  73. Crump, A., Browning, H., Schnell, A., Burn, C., and Birch, J. (2022). Sentience in decapod crustaceans: A general framework and review of the evidence. Animal Sentience. Ch. 21
  74. Cui, J., Chiang, W.-L., Stoica, I., and Hsieh, C.-J. (2024). OR-Bench: An Over-Refusal Benchmark for Large Language Models. International Conference on Machine Learning. Ch. 20
  75. Curry, O. S., Mullins, D. A., and Whitehouse, H. (2019). Is It Good to Cooperate? Testing the Theory of Morality-as-Cooperation in 60 Societies. Current Anthropology. Ch. 18
  76. Cyberspace Administration of China, National Development and Reform Commission, Ministry of Education, Ministry of Science and Technology, Ministry of Industry and Information Technology, Ministry of Public Security, and National Radio and Television Administration (2023). 生成式人工智能服务管理暂行办法 (Interim Measures for the Management of Generative Artificial Intelligence Services). Cyberspace Administration of China, Order No. 15. non-peer-reviewed Ch. 17
  77. Dai, J., Pan, X., Sun, R., Ji, J., Xu, X., Liu, M., Wang, Y., and Yang, Y. (2024). Safe RLHF: Safe Reinforcement Learning from Human Feedback. International Conference on Learning Representations. Ch. 20
  78. Danaher, J. (2020). Welcoming Robots into the Moral Circle: A Defence of Ethical Behaviourism. Science and Engineering Ethics. Ch. 21
  79. Davani, A. M., Díaz, M., and Prabhakaran, V. (2022). Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations. Transactions of the Association for Computational Linguistics. Ch. 18
  80. Dawkins, M. S. (2008). The Science of Animal Suffering. Ethology. Ch. 21
  81. Dehaene, S., Lau, H., and Kouider, S. (2017). What is consciousness, and could machines have it? Science. Ch. 21
  82. Delgado, F., Yang, S., Madaio, M., and Yang, Q. (2023). The Participatory Turn in AI Design: Theoretical Foundations and the Current State of Practice. Equity and Access in Algorithms, Mechanisms, and Optimization. Ch. 17
  83. Department of the Army (2026). Command and Control. Army Doctrine Publication ADP 6-0, Headquarters, Department of the Army. non-peer-reviewed Ch. 19
  84. Díaz, M., Kivlichan, I., Rosen, R., Baker, D., Amironesei, R., Prabhakaran, V., and Denton, R. (2022). CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotation. 2022 ACM Conference on Fairness Accountability and Transparency. Ch. 18
  85. Dobbe, R., Krendl Gilbert, T., and Mintz, Y. (2021). Hard choices in artificial intelligence. Artificial Intelligence. Ch. 17
  86. Dominguez-Olmedo, R., Hardt, M., and Mendler-Dünner, C. (2024). Questioning the Survey Responses of Large Language Models. Advances in Neural Information Processing Systems. Ch. 18
  87. Dummett, M. (1998). The Borda count and agenda manipulation. Social Choice and Welfare. Ch. 17
  88. Durmus, E., Nguyen, K., Liao, T. I., Schiefer, N., Askell, A., Bakhtin, A., … Ganguli, D. (2023). Towards Measuring the Representation of Subjective Global Opinions in Language Models. arXiv. preprint Ch. 17
  89. Dworkin, G., and The Hegeler Institute (1972). Paternalism. Monist. Ch. 16
  90. Earp, B. D., Porsdam Mann, S., Allen, J., Salloch, S., Suren, V., Jongsma, K., … Savulescu, J. (2024). A Personalized Patient Preference Predictor for Substituted Judgments in Healthcare: Technically Feasible and Ethically Desirable. The American Journal of Bioethics. Ch. 16 Ch. 18
  91. Ecoffet, A., and Lehman, J. (2020). Reinforcement Learning Under Moral Uncertainty. arXiv. preprint Ch. 16
  92. Elkins, Z., Ginsburg, T., and Melton, J. (2009). The Endurance of National Constitutions. Cambridge University Press. Ch. 19
  93. Elmahjub, E. (2023). Artificial Intelligence (AI) in Islamic Ethics: Towards Pluralist Ethical Benchmarking for AI. Philosophy & Technology. Ch. 16
  94. Epstein, D., and O'Halloran, S. (1999). Delegating Powers. Cambridge University Press. Ch. 19
  95. Erman, E., and Furendal, M. (2024). Artificial Intelligence and the Political Legitimacy of Global Governance. Political Studies. Ch. 17
  96. European Parliament and Council of the European Union (2024b). Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence. Official Journal of the European Union (OJ L 2024/1689); verified on the European Commission AI Act page. non-peer-reviewed Ch. 17
  97. Evans, O., Stuhlmueller, A., and Goodman, N. (2016). Learning the Preferences of Ignorant, Inconsistent Agents. Proceedings of the AAAI Conference on Artificial Intelligence. Ch. 18
  98. Evans, O., Cotton-Barratt, O., Finnveden, L., Bales, A., Balwit, A., Wills, P., Righetti, L., and Saunders, W. (2021). Truthful AI: Developing and governing AI that does not lie. arXiv. preprint Ch. 20
  99. Everitt, T., Hutter, M., Kumar, R., and Krakovna, V. (2021b). Reward tampering problems and solutions in reinforcement learning: a causal influence diagram perspective. Synthese. Ch. 19
  100. Fagerlin, A., Ditto, P. H., Danks, J. H., and Houts, R. M. (2001). Projection in surrogate decisions about life-sustaining medical treatments. Health Psychology. Ch. 18
  101. Feaver, P. D. (1996). The Civil-Military Problematique: Huntington, Janowitz, and the Question of Civilian Control. Armed Forces & Society. Ch. 19 Ch. 20
  102. Feng, K. J. K., Chen, Q. Z., Cheong, I., Xia, K., and Zhang, A. X. (2023). Case Repositories: Towards Case-Based Reasoning for AI Alignment. arXiv. preprint Ch. 19
  103. Fenichel Pitkin, H. (1967). The Concept of Representation. University of California Press. Ch. 20
  104. Fickinger, A., Zhuang, S., Hadfield-Menell, D., and Russell, S. (2020a). Multi-Principal Assistance Games. arXiv. preprint Ch. 17
  105. Fickinger, A., Zhuang, S., Critch, A., Hadfield-Menell, D., and Russell, S. (2020b). Multi-Principal Assistance Games: Definition and Collegial Mechanisms. arXiv. preprint Ch. 17
  106. Firth, R. (1952). Ethical Absolutism and the Ideal Observer. Philosophy and Phenomenological Research. Ch. 16
  107. Fischhoff, B. (1991). Value elicitation: Is there anything in there? American Psychologist. Ch. 18
  108. Fish, S., Gölz, P., Parkes, D. C., Procaccia, A. D., Rusak, G., Shapira, I., and Wüthrich, M. (2023). Generative Social Choice. arXiv. preprint Ch. 17
  109. Flynn, E., and Arstein-Kerslake, A. (2014). Legislating personhood: realising the right to support in exercising legal capacity. International Journal of Law in Context. Ch. 16
  110. Frankel, T. (1983). Fiduciary Law. California Law Review. Ch. 19
  111. Fraser, D., Weary, D. M., Pajor, E. A., and Milligan, B. N. (1997). A Scientific Conception of Animal Welfare that Reflects Ethical Concerns. Animal Welfare. Ch. 21
  112. Friedman, B., and Nissenbaum, H. (1996). Bias in computer systems. ACM Transactions on Information Systems. Ch. 17
  113. Fuller, L. L. (1958). Positivism and Fidelity to Law: A Reply to Professor Hart. Harvard Law Review. Ch. 19
  114. Fung, A. (2006). Varieties of Participation in Complex Governance. Public Administration Review. Ch. 17
  115. Gabriel, I. (2020). Artificial Intelligence, Values, and Alignment. Minds and Machines. Ch. 16
  116. Gabriel, I., and Keeling, G. (2025). A matter of principle? AI alignment as the fair treatment of claims. Philosophical Studies. Ch. 16 Ch. 17
  117. Gadiraju, V., Kane, S., Dev, S., Taylor, A., Wang, D., Denton, R., and Brewer, R. (2023). "I wouldn't say offensive but...": Disability-Centered Perspectives on Large Language Models. 2023 ACM Conference on Fairness Accountability and Transparency. Ch. 20
  118. Gailmard, S., and Patty, J. W. (2007). Slackers and Zealots: Civil Service, Policy Discretion, and Bureaucratic Expertise. American Journal of Political Science. Ch. 19
  119. Ganguli, D., Lovitt, L., Kernion, J., Askell, A., Bai, Y., Kadavath, S., … Clark, J. (2022). Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned. arXiv. preprint Ch. 20
  120. Ge, L., Halpern, D., Micha, E., Procaccia, A. D., Shapira, I., Vorobeychik, Y., and Wu, J. (2024b). Axioms for AI Alignment from Human Feedback. Advances in Neural Information Processing Systems. Ch. 17
  121. Gebru, T., and Torres, É. P. (2024). The TESCREAL bundle: Eugenics and the promise of utopia through artificial general intelligence. First Monday. Ch. 17
  122. Gibbard, A. (1973). Manipulation of Voting Schemes: A General Result. Econometrica. Ch. 17
  123. Gilardi, F., Alizadeh, M., and Kubli, M. (2023). ChatGPT outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences. Ch. 18
  124. Goldstein, S., and Kirk-Giannini, C. D. (2025a). AI wellbeing. Asian Journal of Philosophy. Ch. 21
  125. Goldstein, S., and Kirk-Giannini, C. D. (2026). A Case for AI Consciousness: Language Agents and Global Workspace Theory. Journal of Consciousness Studies. doi:10.53765/20512201.33.7.061. Ch. 21
  126. Goodman, N. D., and Frank, M. C. (2016). Pragmatic Language Interpretation as Probabilistic Inference. Trends in Cognitive Sciences. Ch. 19
  127. Graham, J., Haidt, J., Koleva, S., Motyl, M., Iyer, R., Wojcik, S. P., and Ditto, P. H. (2013). Moral Foundations Theory. Advances in Experimental Social Psychology. Ch. 18 Ch. 20
  128. Gray, M. L., and Suri, S. (2019). Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass. Houghton Mifflin Harcourt. Ch. 18
  129. Greaves, H., and Cotton-Barratt, O. (2023). A Bargaining-Theoretic Approach to Moral Uncertainty. Journal of Moral Philosophy. Ch. 16
  130. Grice, H. P. (1975). Logic and Conversation. Speech Acts. Ch. 19 Ch. 20
  131. Guan, M. Y., Joglekar, M., Wallace, E., Jain, S., Barak, B., Helyar, A., … Glaese, A. (2024). Deliberative Alignment: Reasoning Enables Safer Language Models. arXiv. preprint Ch. 20
  132. Guo, A., and Wolfe, J. (2026). Introducing Model Spec Evals. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 19
  133. Guo, C., Uribe, J. F. C., Zhu, S., Choquette-Choo, C. A., Lin, S., Kandpal, N., … Xiao, K. (2026). IH-Challenge: A Training Dataset to Improve Instruction Hierarchy on Frontier LLMs. arXiv. preprint Ch. 19
  134. Gurnee, W., Sofroniew, N., Pearce, A., Piotrowski, M., Kauvar, I., Chen, R., … Lindsey, J. (2026). Verbalizable Representations Form a Global Workspace in Language Models. Transformer Circuits Thread. non-peer-reviewed Ch. 21
  135. Gururangan, S., Card, D., Dreier, S., Gade, E., Wang, L., Wang, Z., Zettlemoyer, L., and Smith, N. A. (2022). Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Ch. 17
  136. Gwagwa, A., Kazim, E., and Hilliard, A. (2022). The role of the African value of Ubuntu in global AI inclusion discourse: A normative ethics perspective. Patterns. Ch. 16
  137. Hadfield-Menell, D., and Hadfield, G. K. (2019). Incomplete Contracting and AI Alignment. Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society. Ch. 19
  138. Hadfield-Menell, D., Dragan, A., Abbeel, P., and Russell, S. (2017b). The Off-Switch Game. Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence. Ch. 20
  139. Haerpfer, C. (., Inglehart, R. (., Moreno, A. (., Welzel, C. (., Kizilova, K. (., Diez-Medrano, J. (., … Puranen, B. (. (2022). World Values Survey Trend File (1981-2022) Cross-National Data-Set. JD Systems Institute and WVSA Secretariat (World Values Survey Association). non-peer-reviewed Ch. 18
  140. Hara, K., Adams, A., Milland, K., Savage, S., Callison-Burch, C., and Bigham, J. P. (2018). A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. Ch. 18
  141. Harding, E. J., Paul, E. S., and Mendl, M. (2004). Cognitive bias and affective state. Nature. Ch. 21
  142. Harman, E. (2015). The Irrelevance of Moral Uncertainty. Oxford Studies in Metaethics, Volume 10. Ch. 16
  143. Harsanyi, J. C. (1955). Cardinal Welfare, Individualistic Ethics, and Interpersonal Comparisons of Utility. Journal of Political Economy. Ch. 17
  144. Hart, H. L. A. (1958). Positivism and the Separation of Law and Morals. Harvard Law Review. Ch. 19
  145. Hart, H. L. A. (1961). The concept of law. Clarendon Press. Ch. 19
  146. Hendrycks, D. (2024). Introduction to AI Safety, Ethics, and Society. CRC Press. Ch. 17
  147. Hendrycks, D., Burns, C., Basart, S., Critch, A., Li, J., Song, D., and Steinhardt, J. (2020a). Aligning AI With Shared Human Values. International Conference on Learning Representations. Ch. 16
  148. Henrich, J., Boyd, R., Bowles, S., Camerer, C., Fehr, E., Gintis, H., and McElreath, R. (2001). In Search of Homo Economicus: Behavioral Experiments in 15 Small-Scale Societies. American Economic Review. Ch. 17
  149. Henrich, J., Heine, S. J., and Norenzayan, A. (2010). The weirdest people in the world? Behavioral and Brain Sciences. Ch. 17
  150. Hershock, P. D. (2021). Buddhism and Intelligent Technology. Bloomsbury Academic. Ch. 16
  151. Himmelreich, J. (2023). Against “Democratizing AI”. AI & SOCIETY. Ch. 17
  152. Hongladarom, S. (2020). The Ethics of AI and Robotics. Lexington Books. Ch. 16
  153. Hosking, T., Blunsom, P., and Bartolo, M. (2023). Human Feedback is not Gold Standard. arXiv. preprint Ch. 18
  154. Huang, S., Siddarth, D., Lovitt, L., Liao, T. I., Durmus, E., Tamkin, A., and Ganguli, D. (2024a). Collective Constitutional AI: Aligning a Language Model with Public Input. The 2024 ACM Conference on Fairness, Accountability, and Transparency. Ch. 17
  155. Huang, S., Durmus, E., McCain, M., Handa, K., Tamkin, A., Hong, J., … Ganguli, D. (2025c). Values in the Wild: Discovering and Analyzing Values in Real-World Language Model Interactions. arXiv. preprint Ch. 18
  156. IIT-Concerned, Fleming, S. M., Frith, C. D., Goodale, M., Lau, H., LeDoux, J. E., … Slagter, H. A. (2023). The Integrated Information Theory of Consciousness as Pseudoscience. Center for Open Science. preprint Ch. 21
  157. Iurino, K., and Saucier, G. (2020). Testing Measurement Invariance of the Moral Foundations Questionnaire Across 27 Countries. Assessment. Ch. 18
  158. Jasanoff, S. (2003). Technologies of Humility: Citizen Participation in Governing Science. Minerva. Ch. 17
  159. Jasanoff, S., Saha, K., and Hurlbut, J. B. (2015). CRISPR Democracy: Gene Editing and the Need for Inclusive Deliberation. Issues in Science and Technology. non-peer-reviewed Ch. 17
  160. Jiang, L., Hwang, J. D., Bhagavatula, C., Le Bras, R., Liang, J., Dodge, J., … Choi, Y. (2021). Can Machines Learn Morality? The Delphi Experiment. arXiv. preprint Ch. 16
  161. Jobin, A., Ienca, M., and Vayena, E. (2019). The global landscape of AI ethics guidelines. Nature Machine Intelligence. Ch. 16
  162. Jonsen, A. R., and Toulmin, S. (1988). Abuse of Casuistry. University of California Press. Ch. 19
  163. Joshi, P., Santy, S., Budhiraja, A., Bali, K., and Choudhury, M. (2020). The State and Fate of Linguistic Diversity and Inclusion in the NLP World. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Ch. 17
  164. Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., … Kaplan, J. (2022). Language Models (Mostly) Know What They Know. arXiv. preprint Ch. 20
  165. Kahneman, D., and Klein, G. (2009). Conditions for intuitive expertise: A failure to disagree. American Psychologist. Ch. 18
  166. Kaplow, L. (1992). Rules versus Standards: An Economic Analysis. Duke Law Journal. Ch. 19
  167. Kasirzadeh, A., and Gabriel, I. (2023). In Conversation with Artificial Intelligence: Aligning language Models with Human Values. Philosophy & Technology. Ch. 16 Ch. 19 Ch. 20
  168. Kaufman, H. (1960). The Forest Ranger. Routledge. Ch. 19
  169. Keeling, G., Street, W., Stachaczyk, M., Zakharova, D., Comsa, I. M., Sakovych, A., … Birch, J. (2024). Can LLMs make trade-offs involving stipulated pain and pleasure states? arXiv. preprint Ch. 21
  170. Kenton, Z., Everitt, T., Weidinger, L., Gabriel, I., Mikulik, V., and Irving, G. (2021). Alignment of Language Agents. arXiv. preprint Ch. 20
  171. Kerr, S. (1975). On the Folly of Rewarding A, While Hoping for B. Academy of Management Journal. Ch. 18
  172. Khan, A., Casper, S., and Hadfield-Menell, D. (2025). Randomness, Not Representation: The Unreliability of Evaluating Cultural Alignment in LLMs. Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency. Ch. 17 Ch. 18
  173. Kirk, H. R., Vidgen, B., Röttger, P., and Hale, S. A. (2024a). The benefits, risks and bounds of personalizing the alignment of large language models to individuals. Nature Machine Intelligence. Ch. 17
  174. Kirk, H. R., Whitefield, A., Röttger, P., Bean, A., Margatina, K., Ciro, J., … Hale, S. A. (2024b). The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models. Advances in Neural Information Processing Systems. Ch. 17
  175. Kittay, E. F. (2005). At the Margins of Moral Personhood. Ethics. Ch. 21
  176. Kleinberg, J., Mullainathan, S., and Raghavan, M. (2016). Inherent Trade-Offs in the Fair Determination of Risk Scores. arXiv. preprint Ch. 17
  177. Klingefjord, O., Lowe, R., and Edelman, J. (2024). What are human values, and how do we align AI to them? arXiv. preprint Ch. 17
  178. Korinek, A., and Balwit, A. (2023). Aligned with Whom? The Oxford Handbook of AI Governance. Ch. 20
  179. Korsgaard, C. M., and O'Neill, O. (1996). The Sources of Normativity. Cambridge University Press. Ch. 16
  180. Kukutai, T., and Taylor, J. (2016). Indigenous Data Sovereignty. ANU Press. Ch. 17
  181. Lafont, C. (2019). Democracy without Shortcuts. Oxford University PressOxford. Ch. 17
  182. Lang, L., Foote, D., Russell, S., Dragan, A., Jenner, E., and Emmons, S. (2024b). When Your AIs Deceive You: Challenges of Partial Observability in Reinforcement Learning from Human Feedback. Advances in Neural Information Processing Systems. Ch. 18
  183. Lazar, S., and Nelson, A. (2023). AI safety on whose terms? Science. Ch. 16 Ch. 17
  184. Lee, T. R., and Mouritsen, S. C. (2018). Judging Ordinary Meaning. Yale Law Journal, Volume 127. non-peer-reviewed Ch. 19
  185. Lee, H., Phatale, S., Mansoor, H., Mesnard, T., Ferret, J., Lu, K., … Prakash, S. (2023). RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback. International Conference on Machine Learning. Ch. 18
  186. Lees, A., Tran, V. Q., Tay, Y., Sorensen, J., Gupta, J., Metzler, D., and Vasserman, L. (2022). A New Generation of Perspective API: Efficient Multilingual Character-level Transformers. Knowledge Discovery and Data Mining. doi:10.1145/3534678.3539147. Ch. 20
  187. Lessig, L. (1999). Code: and other laws of cyberspace. Basic Books. Ch. 17
  188. Leveson, N. (2004). A new accident model for engineering safer systems. Safety Science. Ch. 19
  189. Lewis, J. E., Abdilla, A., Arista, N., Baker, K., Benesiinaabandan, S., Brown, M., … Whaanga, H. (2020). Indigenous Protocol and Artificial Intelligence Position Paper. Indigenous Protocol and Artificial Intelligence Working Group and the Canadian Institute for Advanced Research. doi:10.11573/spectrum.library.concordia.ca.00986506. non-peer-reviewed Ch. 16 Ch. 17
  190. Li, C., Wichers, N., Price, S., Marks, S., and Kutasov, J. (2026). Model Spec Midtraining: Improving How Alignment Training Generalizes. arXiv. preprint Ch. 19
  191. Lichtenstein, S. (., and Slovic, P. (. (2006). The Construction of Preference. Cambridge University Press. Ch. 18
  192. Lin, S., Hilton, J., and Evans, O. (2021). TruthfulQA: Measuring How Models Mimic Human Falsehoods. Annual Meeting of the Association for Computational Linguistics. doi:10.18653/v1/2022.acl-long.229. Ch. 20
  193. Lindsey, J. (2025). Emergent Introspective Awareness in Large Language Models. Transformer Circuits Thread. non-peer-reviewed Ch. 21
  194. Lipsky, M. (1980). Street-Level Bureaucracy: Dilemmas of the Individual in Public Services. Russell Sage Foundation. Ch. 19
  195. List, C., and Pettit, P. (2002). Aggregating Sets of Judgments: An Impossibility Result. Economics and Philosophy. Ch. 17
  196. Long, R., Sebo, J., Butlin, P., Finlinson, K., Fish, K., Harding, J., … Chalmers, D. (2024). Taking AI Welfare Seriously. arXiv. preprint Ch. 21
  197. Long, R., Sebo, J., and Sims, T. (2025). Is there a tension between AI safety and AI welfare? Philosophical Studies. Ch. 20 Ch. 21
  198. Low, P. (2012). The Cambridge Declaration on Consciousness in Non-Human Animals. Francis Crick Memorial Conference, University of Cambridge. non-peer-reviewed Ch. 21
  199. MacAskill, W. (2016). Normative Uncertainty as a Voting Problem. Mind. Ch. 16
  200. MacAskill, W., Bykvist, K., and Ord, T. (2020). Moral Uncertainty. Oxford University PressOxford. Ch. 16
  201. Mackenzie, C., and Stoljar, N. (2000). Relational Autonomy. Oxford University PressNew York, NY. Ch. 16
  202. Mackie, J. L. (1977). Ethics: Inventing Right and Wrong. Penguin. Ch. 16
  203. Mashour, G. A., Roelfsema, P., Changeux, J.-P., and Dehaene, S. (2020). Conscious Processing and the Global Neuronal Workspace Hypothesis. Neuron. Ch. 21
  204. Maskin, E., and Tirole, J. (2004). The Politician and the Judge: Accountability in Government. American Economic Review. Ch. 20
  205. Mazeika, M., Phan, L., Yin, X., Zou, A., Wang, Z., Mu, N., … Hendrycks, D. (2024). HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal. International Conference on Machine Learning. Ch. 20
  206. Mazeika, M., Yin, X., Tamirisa, R., Lim, J., Lee, B. W., Ren, R., … Hendrycks, D. (2025). Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs. Advances in Neural Information Processing Systems 38. Ch. 17 Ch. 18
  207. Meibauer, J. (2005). Lying and falsely implicating. Journal of Pragmatics. Ch. 20
  208. Mellor, D. (2016). Updating Animal Welfare Thinking: Moving beyond the “Five Freedoms” towards “A Life Worth Living”. Animals. Ch. 21
  209. Mellor, D., and Beausoleil, N. (2015). Extending the ‘Five Domains’ model for animal welfare assessment to incorporate positive welfare states. Animal Welfare. Ch. 21
  210. Mendl, M., Burman, O. H. P., and Paul, E. S. (2010). An integrative and functional framework for the study of animal emotion and mood. Proceedings of the Royal Society B: Biological Sciences. Ch. 21
  211. Meredith, W. (1993). Measurement Invariance, Factor Analysis and Factorial Invariance. Psychometrika. Ch. 18
  212. Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist. Ch. 18
  213. Metz, T. (2007). Toward an African Moral Theory*. Journal of Political Philosophy. Ch. 16
  214. Metzinger, T. (2021). Artificial Suffering: An Argument for a Global Moratorium on Synthetic Phenomenology. Journal of Artificial Intelligence and Consciousness. Ch. 21
  215. Meyer, B. (1992). Applying 'design by contract'. Computer. Ch. 19
  216. Meyer, J. W., and Rowan, B. (1977). Institutionalized Organizations: Formal Structure as Myth and Ceremony. American Journal of Sociology. Ch. 17
  217. Mhlambi, S. (2020). From Rationality to Relationality: Ubuntu as an Ethical and Human Rights Framework for Artificial Intelligence Governance. Carr Center Discussion Paper Series, 2020-009, Harvard Kennedy School. working paper Ch. 16
  218. Miceli, M., and Posada, J. (2022). The Data-Production Dispositif. Proceedings of the ACM on Human-Computer Interaction. Ch. 18 Ch. 19
  219. Miceli, M., Posada, J., and Yang, T. (2022). Studying Up Machine Learning Data. Proceedings of the ACM on Human-Computer Interaction. Ch. 18
  220. Milli, S., Hadfield-Menell, D., Dragan, A., and Russell, S. (2017). Should Robots be Obedient? Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence. Ch. 19 Ch. 20
  221. Mishra, A. (2023). AI Alignment and Social Choice: Fundamental Limitations and Policy Implications. arXiv. preprint Ch. 17
  222. Mittelstadt, B. (2019). Principles alone cannot guarantee ethical AI. Nature Machine Intelligence. Ch. 16
  223. Mohamed, S., Png, M.-T., and Isaac, W. (2020). Decolonial AI: Decolonial Theory as Sociotechnical Foresight in Artificial Intelligence. Philosophy & Technology. Ch. 16
  224. Nagel, T. (1974). What Is It Like to Be a Bat? The Philosophical Review. Ch. 21
  225. Nagel, T. (1979). Mortal Questions. Cambridge University Press. Ch. 16
  226. Nash, J. F. (1950). The Bargaining Problem. Econometrica. Ch. 16
  227. National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research (1979). The Belmont Report: Ethical Principles and Guidelines for the Protection of Human Subjects of Research. U.S. Department of Health, Education, and Welfare. non-peer-reviewed Ch. 16
  228. National Governance Committee for New Generation Artificial Intelligence (2021). 新一代人工智能伦理规范 (Ethical Norms for New Generation Artificial Intelligence). Ministry of Science and Technology of the People's Republic of China. non-peer-reviewed Ch. 16
  229. Nay, J. (2022). Law Informs Code: A Legal Informatics Approach to Aligning Artificial Intelligence with Humans. SSRN Electronic Journal. working paper Ch. 19
  230. Nay, J. (2023). Large Language Models as Fiduciaries: A Case Study Toward Robustly Communicating With Artificial Intelligence Through Legal Standards. Elsevier BV. preprint Ch. 19
  231. Nekoto, W., Marivate, V., Matsila, T., Fasubaa, T., Fagbohungbe, T., Akinola, S. O., … Bashir, A. (2020). Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages. Findings of the Association for Computational Linguistics: EMNLP 2020. Ch. 17
  232. Noble, S. U. (2018). Algorithms of Oppression. New York University Press. Ch. 17
  233. Noggle, R. (2020). Pressure, Trickery, and a Unified Account of Manipulation. American Philosophical Quarterly. Ch. 16
  234. Nozick, R. (1974). Anarchy, State, and Utopia. Basic Books. Ch. 16
  235. O'Keefe, C., Ramakrishnan, K., Tay, J., and Winter, C. (2025). Law-Following AI: Designing AI Agents to Obey Human Laws. Fordham Law Review. Ch. 19 Ch. 20
  236. Obermeyer, Z., Powers, B., Vogeli, C., and Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science. Ch. 17
  237. ÓhÉigeartaigh, S. S., Whittlestone, J., Liu, Y., Zeng, Y., and Liu, Z. (2020). Overcoming Barriers to Cross-cultural Cooperation in AI Ethics and Governance. Philosophy & Technology. Ch. 16
  238. Olson, M. (1965). The Logic of Collective Action. Harvard University Press. Ch. 17
  239. OpenAI (2025b). Expanding on what we missed with sycophancy. OpenAI. non-peer-reviewed Ch. 20
  240. OpenAI (2025d). OpenAI Model Spec. OpenAI. non-peer-reviewed Ch. 19 Ch. 20 Ch. 21
  241. Orseau, L., and Armstrong, S. (2016). Safely Interruptible Agents. 32nd Conference on Uncertainty in Artificial Intelligence (UAI 2016). Ch. 20
  242. Osiel, M. J. (1998). Obeying Orders: Atrocity, Military Discipline, and the Law of War. California Law Review. Ch. 20
  243. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., … Lowe, R. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems. doi:10.52202/068431-2011. Ch. 16 Ch. 18
  244. Parfit, D. (2011). On What Matters. Oxford University Press. Ch. 16
  245. Park, P. S., Goldstein, S., O’Gara, A., Chen, M., and Hendrycks, D. (2024a). AI deception: A survey of examples, risks, and potential solutions. Patterns. Ch. 20
  246. Paul, L. A. (2014). Transformative Experience. Oxford University Press. Ch. 16
  247. Pavlick, E., and Kwiatkowski, T. (2019). Inherent Disagreements in Human Textual Inferences. Transactions of the Association for Computational Linguistics. Ch. 18
  248. Perez, E., and Long, R. (2023). Towards Evaluating AI Systems for Moral Status Using Self-Reports. arXiv. preprint Ch. 21
  249. Perrigo, B. (2023). Exclusive: OpenAI Used Kenyan Workers on Less Than $2 Per Hour to Make ChatGPT Less Toxic. TIME. non-peer-reviewed Ch. 18
  250. Pettit, P. (1997). Republicanism. Oxford University PressOxford. Ch. 20 Ch. 21
  251. Pettit, P. (2012). On the People's Terms. Cambridge University Press. Ch. 17 Ch. 21
  252. Plank, B. (2022). The “Problem” of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Ch. 18
  253. Potham, R., and Harms, M. (2025). Corrigibility as a Singular Target: A Vision for Inherently Reliable Foundation Models. arXiv. preprint Ch. 20
  254. Pozen, D. E., and Khan, L. M. (2019). A Skeptical View of Information Fiduciaries. Harvard Law Review. Ch. 19
  255. Rajamanoharan, S., and Nanda, N. (2025). Self-preservation or Instruction Ambiguity? Examining the Causes of Shutdown Resistance. LessWrong (Alignment Forum). non-peer-reviewed Ch. 20
  256. Rajkumar, A., and Agarwal, S. (2014). A Statistical Convergence Perspective of Algorithms for Rank Aggregation from Pairwise Data. International Conference on Machine Learning. Ch. 17
  257. Raquib, A., Channa, B., Zubair, T., and Qadir, J. (2022). Islamic virtue-based ethics for artificial intelligence. Discover Artificial Intelligence. Ch. 16 Ch. 19
  258. Rawls, J. (1971). A Theory of Justice. Belknap Press of Harvard University Press. Ch. 16 Ch. 17
  259. Ren, R., Agarwal, A., Mazeika, M., Menghini, C., Vacareanu, R., Kenstler, B., … Hendrycks, D. (2025). The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems. arXiv. preprint Ch. 20
  260. Rid, A., and Wendler, D. (2014). Use of a Patient Preference Predictor to Help Make Medical Decisions for Incapacitated Patients. Journal of Medicine and Philosophy. Ch. 16 Ch. 18
  261. Rimsky, N., Gabrieli, N., Schulz, J., Tong, M., Hubinger, E., and Turner, A. (2024). Steering Llama 2 via Contrastive Activation Addition. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Ch. 20
  262. Rogers, T., Zeckhauser, R., Gino, F., Norton, M. I., and Schweitzer, M. E. (2017). Artful paltering: The risks and rewards of using truthful statements to mislead others. Journal of Personality and Social Psychology. Ch. 20
  263. Röttger, P., Kirk, H. R., Vidgen, B., Attanasio, G., Bianchi, F., and Hovy, D. (2023). XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models. North American Chapter of the Association for Computational Linguistics. Ch. 20
  264. Röttger, P., Hofmann, V., Pyatkin, V., Hinck, M., Kirk, H., Schuetze, H., and Hovy, D. (2024). Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Ch. 18
  265. Rozen, N., Bezalel, L., Elidan, G., Globerson, A., and Daniel, E. (2025). Do LLMs have Consistent Values? International Conference on Learning Representations. Ch. 18
  266. Ruis, L., Khan, A., Biderman, S., Hooker, S., Rocktäschel, T., and Grefenstette, E. (2023). The Goldilocks of Pragmatic Understanding: Fine-Tuning Strategy Matters for Implicature Resolution by LLMs. Advances in Neural Information Processing Systems. Ch. 19
  267. Ryan, M. J., Held, W., and Yang, D. (2024). Unintended Impacts of LLM Alignment on Global Representation. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Ch. 17
  268. Salib, P., and Goldstein, S. (2024). AI Rights for Human Safety. Elsevier BV. preprint Ch. 21
  269. Sambasivan, N., Kapania, S., Highfill, H., Akrong, D., Paritosh, P., and Aroyo, L. M. (2021a). “Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. Ch. 18
  270. Sambasivan, N., Arnesen, E., Hutchinson, B., Doshi, T., and Prabhakaran, V. (2021b). Re-imagining Algorithmic Fairness in India and Beyond. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. Ch. 17
  271. Santurkar, S., Durmus, E., Ladhak, F., Lee, C., Liang, P., and Hashimoto, T. (2023). Whose Opinions Do Language Models Reflect? arXiv. preprint Ch. 17
  272. Saul, J. M. (2012). Lying, Misleading, and What is Said. Oxford University Press. Ch. 20
  273. Scharpf, F. (1999). Governing in Europe: Effective and Democratic? Oxford University Press. Ch. 17
  274. Schauer, F. F. (1991). Playing by the rules: a philosophical examination of rule-based decision-making in law and in life. Clarendon. Ch. 19
  275. Schein, C., and Gray, K. (2018). The Theory of Dyadic Morality: Reinventing Moral Judgment by Redefining Harm. Personality and Social Psychology Review. Ch. 18 Ch. 20
  276. Schlatter, J., Weinstein-Raun, B., and Ladish, J. (2025). Incomplete Tasks Induce Shutdown Resistance in Some Frontier LLMs. arXiv. preprint Ch. 20
  277. Schwartz, S. H. (1992). Universals in the Content and Structure of Values: Theoretical Advances and Empirical Tests in 20 Countries. Advances in Experimental Social Psychology. Ch. 18
  278. Schwitzgebel, E. (2023). AI systems must not confuse users about their sentience or moral status. Patterns. Ch. 20 Ch. 21
  279. Schwitzgebel, E., and Garza, M. (2015). A Defense of the Rights of Artificial Intelligences. Midwest Studies In Philosophy. Ch. 21
  280. Searle, J. R. (1976). A classification of illocutionary acts. Language in Society. Ch. 19 Ch. 20
  281. Searle, J. R. (1980). Minds, brains, and programs. Behavioral and Brain Sciences. Ch. 21
  282. Sebo, J. (2025). The Moral Circle: Who Matters, What Matters, and Why. Unknown. non-peer-reviewed Ch. 21
  283. Sebo, J., and Long, R. (2025). Moral consideration for AI systems by 2030. AI and Ethics. Ch. 21
  284. Seger, E., Ovadya, A., Garfinkel, B., Siddarth, D., and Dafoe, A. (2023a). Democratising AI: Multiple Meanings, Goals, and Methods. arXiv. preprint Ch. 17
  285. Sen, A. (1970b). The Impossibility of a Paretian Liberal. Journal of Political Economy. Ch. 17
  286. Serapio-García, G., Safdari, M., Crepy, C., Sun, L., Fitz, S., Romero, P., … Matarić, M. (2025). A psychometric framework for evaluating and shaping personality traits in large language models. Nature Machine Intelligence. Ch. 18
  287. Seth, A. K., and Bayne, T. (2022). Theories of consciousness. Nature Reviews Neuroscience. Ch. 21
  288. Shalowitz, D. I., Garrett-Mayer, E., and Wendler, D. (2006). The Accuracy of Surrogate Decision Makers. Archives of Internal Medicine. Ch. 18
  289. Shanahan, M. (2026). Simulacra as conscious exotica. Inquiry. Ch. 21
  290. Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A., Bowman, S. R., … Perez, E. (2023). Towards Understanding Sycophancy in Language Models. International Conference on Learning Representations. Ch. 20
  291. Shevlin, H. (2021). How Could We Know When a Robot was a Moral Patient? Cambridge Quarterly of Healthcare Ethics. Ch. 21
  292. Shevlin, H., and Halina, M. (2019). Apply rich psychological terms in AI with care. Nature Machine Intelligence. Ch. 21
  293. Shulman, C., and Bostrom, N. (2021). Sharing the World with Digital Minds. Rethinking Moral Status. Ch. 21
  294. Simon, H. A. (1947). Administrative Behavior: A Study of Decision-Making Processes in Administrative Organization. Macmillan. Ch. 19
  295. Siththaranjan, A., Laidlaw, C., and Hadfield-Menell, D. (2024). Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF. International Conference on Learning Representations. Ch. 17 Ch. 18
  296. Sloane, M., Moss, E., Awomolo, O., and Forlano, L. (2022). Participation Is not a Design Fix for Machine Learning. Equity and Access in Algorithms, Mechanisms, and Optimization. Ch. 17
  297. Smart, J. J. C., and Williams, B. (1973). Utilitarianism. Cambridge University Press. Ch. 16
  298. Sneddon, L. U., Elwood, R. W., Adamo, S. A., and Leach, M. C. (2014). Defining and assessing animal pain. Animal Behaviour. Ch. 21
  299. Soares, N., Fallenstein, B., Yudkowsky, E., and Armstrong, S. (2015). Corrigibility. AAAI Workshops: Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI Publications. Ch. 20
  300. Sofroniew, N., Kauvar, I., Saunders, W., Chen, R., Henighan, T., Hydrie, S., … Lindsey, J. (2026). Emotion Concepts and their Function in a Large Language Model. Transformer Circuits Thread. non-peer-reviewed Ch. 21
  301. Solum, L. B. (1992). Legal Personhood for Artificial Intelligences. North Carolina Law Review. Ch. 21
  302. Song, S., Hu, J., and Mahowald, K. (2025a). Language Models Fail to Introspect About Their Knowledge of Language. arXiv. preprint Ch. 21
  303. Song, S., Lederman, H., Hu, J., and Mahowald, K. (2025b). Privileged Self-Access Matters for Introspection in AI. arXiv. preprint Ch. 21
  304. Sorensen, T., Moore, J., Fisher, J., Gordon, M., Mireshghallah, N., Rytting, C. M., … Choi, Y. (2024). A Roadmap to Pluralistic Alignment. arXiv. preprint Ch. 16 Ch. 17
  305. Sperber, D., and Wilson, D. (1986). Relevance: communication and cognition. Harvard University Press. Ch. 19
  306. Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., … Christiano, P. (2020). Learning to summarize from human feedback. Advances in Neural Information Processing Systems. Ch. 18
  307. Stilgoe, J., Owen, R., and Macnaghten, P. (2013). Developing a framework for responsible innovation. Research Policy. Ch. 17
  308. Street, S. (2006). A Darwinian Dilemma for Realist Theories of Value. Philosophical Studies. Ch. 16
  309. Sumers, T., Hawkins, R., Ho, M., Griffiths, T., and Hadfield-Menell, D. (2022). How to Talk So AI Will Learn: Instructions, Descriptions, and Autonomy. Advances in Neural Information Processing Systems. Ch. 19
  310. Sumers, T. R., Ho, M. K., Griffiths, T. L., and Hawkins, R. D. (2024). Reconciling truthfulness and relevance as epistemic and decision-theoretic utility. Psychological Review. Ch. 20
  311. Susser, D., Roessler, B., and Nissenbaum, H. (2019). Technology, autonomy, and manipulation. Internet Policy Review. Ch. 16
  312. Swamy, G., Dann, C., Kidambi, R., Wu, Z. S., and Agarwal, A. (2024). A Minimaximalist Approach to Reinforcement Learning from Human Feedback. arXiv. preprint Ch. 17
  313. Talat, Z., Blix, H., Valvoda, J., Ganesh, M. I., Cotterell, R., and Williams, A. (2022). On the Machine Learning of Ethical Judgments from Natural Language. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Ch. 16
  314. Tao, Y., Viberg, O., Baker, R. S., and Kizilcec, R. F. (2024). Cultural bias and cultural alignment of large language models. PNAS Nexus. Ch. 18
  315. Tessler, M. H., Bakker, M. A., Jarrett, D., Sheahan, H., Chadwick, M. J., Koster, R., … Summerfield, C. (2024). AI can help humans find common ground in democratic deliberation. Science. Ch. 17
  316. Thaler, R. H., and Sunstein, C. R. (2008). Nudge: Improving Decisions About Health, Wealth, and Happiness. Penguin Books, Limited. Ch. 16
  317. Thornley, E. (2025). The shutdown problem: an AI engineering puzzle for decision theorists. Philosophical Studies. Ch. 20
  318. Thornley, E., Roman, A., Ziakas, C., Ho, L., and Thomson, L. (2024). Towards Shutdownable Agents via Stochastic Choice. arXiv. preprint Ch. 20
  319. Tononi, G., Boly, M., Massimini, M., and Koch, C. (2016). Integrated information theory: from consciousness to its physical substrate. Nature Reviews Neuroscience. Ch. 21
  320. Tubaro, P., Casilli, A. A., and Coville, M. (2020). The trainer, the verifier, the imitator: Three ways in which human platform workers support artificial intelligence. Big Data & Society. Ch. 18
  321. Turiel, E. (1983). The Development of Social Knowledge. Cambridge University Press. Ch. 20
  322. United Nations Diplomatic Conference of Plenipotentiaries on the Establishment of an International Criminal Court (1998). Rome Statute of the International Criminal Court, Article 33: Superior orders and prescription of law. International Criminal Court. non-peer-reviewed Ch. 19 Ch. 20
  323. United Nations General Assembly (2006). Convention on the Rights of Persons with Disabilities. United Nations, A/RES/61/106. non-peer-reviewed Ch. 16
  324. United Nations General Assembly (2007). United Nations Declaration on the Rights of Indigenous Peoples. United Nations. non-peer-reviewed Ch. 17
  325. Veselovsky, V., Ribeiro, M. H., and West, R. (2023). Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks. arXiv. preprint Ch. 18
  326. Viswesvaran, C., and Ones, D. S. (1999). Meta-Analyses of Fakability Estimates: Implications for Personality Measurement. Educational and Psychological Measurement. Ch. 18
  327. Wallace, E., Xiao, K., Leike, R., Weng, L., Heidecke, J., and Beutel, A. (2024). The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions. arXiv. preprint Ch. 19
  328. Wang, D., Prabhat, S., and Sambasivan, N. (2022c). Whose AI Dream? In search of the aspiration in data annotation. CHI Conference on Human Factors in Computing Systems. Ch. 18
  329. Webb, E. J., Campbell, D. T., Schwartz, R. D., and Sechrest, L. (1966). Unobtrusive measures: nonreactive research in the social sciences. Rand McNally. Ch. 18
  330. Weidinger, L., Rauh, M., Marchal, N., Manzini, A., Hendricks, L. A., Mateos-Garcia, J., … Isaac, W. (2023a). Sociotechnical Safety Evaluation of Generative AI Systems. arXiv. preprint Ch. 17
  331. Weidinger, L., McKee, K. R., Everett, R., Huang, S., Zhu, T. O., Chadwick, M. J., Summerfield, C., and Gabriel, I. (2023b). Using the Veil of Ignorance to align AI systems with principles of justice. Proceedings of the National Academy of Sciences. Ch. 17
  332. Wen, J., Zhong, R., Khan, A., Perez, E., Steinhardt, J., Huang, M., … Feng, S. (2025a). Language Models Learn to Mislead Humans via RLHF. International Conference on Learning Representations. Ch. 18 Ch. 20
  333. Whittaker, M., Alper, M., Bennett, C. L., Hendren, S., Kaziunas, E., Mills, M., … West, S. M. (2019). Disability, Bias, and AI. AI Now Institute. non-peer-reviewed Ch. 20
  334. Whittlestone, J., Nyrup, R., Alexandrova, A., and Cave, S. (2019). The Role and Limits of Principles in AI Ethics. Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society. Ch. 16 Ch. 20
  335. Winner, L. (2017). Do Artifacts Have Politics? Computer Ethics. Ch. 17
  336. Wong, P.-H. (2012). Dao, Harmony and Personhood: Towards a Confucian Ethics of Technology. Philosophy & Technology. Ch. 16 Ch. 20
  337. Wong, P.-H. (2020). Cultural Differences as Excuses? Human Rights and Cultural Values in Global Ethics and Governance of AI. Philosophy & Technology. Ch. 16
  338. World Medical Association (1964). WMA Declaration of Helsinki: Ethical Principles for Medical Research Involving Human Participants. World Medical Association. non-peer-reviewed Ch. 21
  339. Wu, T., Zhang, S., Song, K., Xu, S., Zhao, S., Agrawal, R., … Zhou, W. (2025d). Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchy. International Conference on Learning Representations. Ch. 19
  340. Xie, T., Qi, X., Zeng, Y., Huang, Y., Sehwag, U. M., Huang, K., … Mittal, P. (2024b). SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal. International Conference on Learning Representations. Ch. 20
  341. Xu, A., Pathak, E., Wallace, E., Gururangan, S., Sap, M., and Klein, D. (2021). Detoxifying Language Models Risks Marginalizing Minority Voices. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Ch. 20
  342. Xunzi, and Hutton, E. L. (2014). Xunzi. Princeton University Press. Ch. 20
  343. Yuan, Y., Sriskandarajah, T., Brakman, A.-L., Helyar, A., Beutel, A., Vallone, A., and Jain, S. (2025). From Hard Refusals to Safe-Completions: Toward Output-Centric Safety Training. arXiv. preprint Ch. 20
  344. Yudkowsky, E. (2004). Coherent Extrapolated Volition. The Singularity Institute, San Francisco, CA. non-peer-reviewed Ch. 16
  345. Zave, P., and Jackson, M. (1997). Four dark corners of requirements engineering. ACM Transactions on Software Engineering and Methodology. Ch. 19
  346. Zhang, J., Sleight, H., Peng, A., Schulman, J., and Durmus, E. (2025c). Stress-Testing Model Specs Reveals Character Differences among Language Models. arXiv. preprint Ch. 19 Ch. 20
  347. Zhi-Xuan, T., Carroll, M., Franklin, M., and Ashton, H. (2025). Beyond Preferences in AI Alignment. Philosophical Studies. Ch. 16
  348. Zverev, E., Abdelnabi, S., Tabesh, S., Fritz, M., and Lampert, C. H. (2025). Can LLMs Separate Instructions From Data? And What Do We Even Mean By That? International Conference on Learning Representations. Ch. 19