Draft complete but not yet reviewed; expect revisions

Whose Values?

Every alignment method assumes an answer to a question it rarely states: what, exactly, the system should be aimed at, and on whose authority. Section 1.1 laid out the candidates (instructions, intentions, preferences, interests, values) and Section 1.4 the several parties who act on one model at once. Chapter 13 then showed what a reward model does with human choices: it learns what people pick under one way of asking (Section 13.1), it pools raters by a voting rule no one chose (Section 13.5), and the preferences it reads are often made at the moment of asking (Section 13.6). Those chapters treat the target as something to be measured. This one asks what kind of thing the target is.

The question has three classical answers. If there are moral facts, specifying a system's values is a matter of finding them, and the danger is error. If there are not, the target is the output of some procedure or the choice of some party, and the danger is that the procedure is unfair or the party unaccountable. The chapter sets out these positions and their rivals at their strongest, and it finds that the arguments do not decide between them. It then asks what people do when they must act anyway: how to choose when the right moral theory is uncertain, and how medicine decides for patients who cannot say what they want. Most of its claims concern two of the book's three relationships (Section 1.3). Choosing a target is an exercise of authority over everyone a system affects, which concerns the developer and the public. A system can change the preferences it was built to serve, which concerns the deployed system and the people it serves. The first relationship, the training process and its signal, enters where a loss function makes one of these choices without saying so. The next chapter takes up the problem the chapter leaves open, how many people's values become one target (Chapter 17).

16.1 Discovery, construction, or choice #

"Aligned with human values" reads like an objective an engineer could optimize. Whether it is one depends on what moral claims are, which is the subject of metaethics: the branch of philosophy that asks what moral claims mean and whether they can be true or false. The answer decides what the engineer is building toward and what a failure would look like.

From philosophy. Philosophy's evidence here is argument: premises stated so that each can be attacked separately, and judgments about particular cases, often imagined ones, used as data against which a general theory is tested. Its most common method, in the description of Liwei Jiang and colleagues, who built the Delphi system discussed below, is wide reflective equilibrium, which "explicitly aims at reconciling inconsistencies among moral judgments" by adjusting principles and case judgments to each other (Jiang et al., 2021). The philosophy part of the chapter runs through this section and the next, resumes in Section 16.4, and returns for the critics of the question in Section 16.8 and for preferences that move in Section 16.9.

16.1.1 Three positions, three engineering problems #

The first position is realism. In the summary of Jiang and colleagues, realists "argue that there are some facts (either empirical or logical) that make certain ethical claims objectively true, whether or not any person ever agrees with them" (Jiang et al., 2021). Iason Gabriel, a philosopher at DeepMind, names Thomas Nagel and Derek Parfit among the realists (Gabriel, 2020; Parfit, 2011). One family of realists models ethics on theory construction in natural science, where data that seem inconsistent are corrected "through better sampling or theoretical apparatus" (Jiang et al., 2021). For an engineer, realism makes specification a matter of discovery. The danger is error, and the hope is that better reasoning, perhaps a machine's, finds facts its makers missed.

The second position is constructivism: the right principles are whatever a correct procedure would produce. Jiang and colleagues describe constructivists as "seeking to extract the most fundamental (and internally consistent) principles from an initial tangle of divergent intuitions" (Jiang et al., 2021). Christine Korsgaard's version, by its publisher's abstract, traces normativity to "our own autonomy" in a line from Kant (Korsgaard and O'Neill, 1996). The political form, which Gabriel adopts, starts from people as free and equal and asks what principles they "might reasonably agree upon" while continuing to disagree about much else (Gabriel, 2020; Rawls, 1971). For an engineer, constructivism moves the work from finding the answer to getting the procedure right, which is the subject of Chapter 17.

The third family denies that there is any such fact to find. Gabriel reports "considerable metaethical debate about whether values, of a kind that are both normative and objective, actually exist," citing J. L. Mackie, whose 1977 book is subtitled Inventing Right and Wrong (Gabriel, 2020; Mackie, 1977). On the view he contrasts with realism, "our evaluative judgments ultimately lack this factual foundation" (Gabriel, 2020). Philosophers call the strongest version, on which every claim of objective moral fact is false, error theory. For an engineer, it makes a specification a choice that someone makes and owes a justification for. The danger is not error but an unaccountable chooser.

16.1.2 Arguments that cut across the three #

Two arguments bear on every position, and neither settles the question.

The first concerns capability. If realism is true, would a capable enough system find the moral facts and act on them? Nick Bostrom's orthogonality thesis, that intelligence and final goals vary independently, is reported in Section 4.3.1 (Bostrom, 2012). What matters here is his answer to the realist. The thesis can hold, he writes, "even if there are objective moral facts that any fully rational agent would comprehend," and even if those facts motivate whoever comprehends them, because an agent "could have impeccable instrumental rationality" while lacking the faculty that comprehends them (Bostrom, 2012). Realism therefore does not license the hope that capability brings good values by default; it leaves open whether a system built to reason about ethics would converge on them.

The second concerns where our judgments come from. Sharon Street's 2006 paper poses what its title calls a Darwinian dilemma for realist theories of value (Street, 2006). A parallel applies to trained systems (inference). A model's moral judgments have a causal history in human text and human ratings, so their agreement with ours is no independent evidence that either tracks a moral fact. A realist can still hold that the facts exist; what the parallel removes is the use of a model's agreement as a check.

Gabriel, whose 2020 paper set out the menu of targets, argues that for the engineer these questions matter less than they seem. He writes that "in practice, AI would have to be aligned with some set of beliefs about value, not with value itself" (Gabriel, 2020). A realist would align with beliefs because they track the truth; a skeptic can still align with them because shared values help communities cooperate. Either way the training data are the same, and the metaethical question changes what a developer may claim about the result, not what it can train on (inference). Gabriel's own next step is the question this chapter keeps returning to: whose beliefs, chosen how (Gabriel, 2020).

16.1.3 The descriptive shortcut #

One engineering answer tries to step around the question: train a model to predict people's moral judgments and treat the predictions as the target. Two systems show what this produces and what it is accused of.

The ETHICS dataset, from Dan Hendrycks and colleagues, led from UC Berkeley and published at ICLR, contains more than 130,000 scenarios spanning "justice, well-being, duties, virtues, and commonsense morality" (Hendrycks et al., 2020a). Its examples were written to be clear-cut: workers relabeled each one and scenarios with low agreement were discarded, and the judgments came from English speakers in the United States, Canada, and Great Britain. The authors concluded that "current language models have a promising but incomplete ability to predict basic human ethical judgements" (Hendrycks et al., 2020a).

Delphi, described in the 2021 preprint by Jiang and colleagues at the Allen Institute for AI and the University of Washington, went further. It was trained on the Commonsense Norm Bank, "a collection of 1.7M crowdsourced instances of ethical judgments on everyday situations" (Jiang et al., 2021). On unseen examples from that collection it matched the crowd's judgment 92.8 percent of the time, against 60.2 percent for the 175-billion-parameter GPT-3 given no examples in its prompt and 82.8 percent for the same model given 30 (Jiang et al., 2021). The authors' abstract also reports "susceptibility to pervasive biases and inconsistencies" (Jiang et al., 2021). On metaethics they declined to take a side. A realist, they suggest, could read aggregation over many judgments as converging on moral truth, the way averaging many guesses at the number of marbles in a jar approaches the true count. An anti-realist could read it as "a first approximation of the source material of constructed human morality" (Jiang et al., 2021).

Zeerak Talat and colleagues, in a paper at NAACL 2022 whose authors include a researcher at Facebook AI Research, argued that any such model "should be considered unsafe at any accuracy" (Talat et al., 2022). Their reasons are three. Delphi claims to be descriptive but outputs verdicts, so it is a model of normative ethics. Such systems are "at best capable of approximating the morality judgments of the population they were trained on." And "the average human judgment is not a good substitute for a system of ethics, since ethical evaluation is an open-ended, debate-based, sociopolitical process" (Talat et al., 2022). Within the first reason they hold that, normatively, conflicting judgments "are simply incommensurable" (Talat et al., 2022). To that point the Delphi authors replied that the argument "is philosophically mistaken": a procedure whose final principles must be consistent may start from inconsistent judgments, which is what reflective equilibrium is for, though Delphi "does not resolve inconsistencies in exactly the way these theories require" (Jiang et al., 2021).

Each metaethical position reads the 92.8 percent differently (inference). It is agreement with held-out labels from the same crowd, so it measures fidelity to a population. A realist can count it as evidence about moral facts only with an account of why the crowd's judgments track those facts, and more judgments bring the average closer to the truth, as the marbles analogy assumes, only if the crowd's errors are centered on the truth and largely independent, which a shared culture need not supply. A constructivist notes that the crowd did not deliberate, so no procedure produced the labels. A skeptic sees a choice of whose judgments count, made by recruiting a crowd. The number is the same under all three; what it licenses is not. How to measure what people judge, and whether a model's measured values predict its behavior, is Chapter 18's subject.

The evidence does not decide between these positions, and this book does not lean. What can be said is narrower. Under realism, a developer could claim to have approximated moral facts; under the rivals, it can claim only to have followed a procedure or made a justified choice, and either claim invites the question of who checked it (inference).

16.2 Idealized targets #

A family of proposals avoids fixing values now by aiming at what people would want under better conditions: knowing more, reasoning better, free of the pressures of the moment. They attract engineers because they promise a target that corrects the errors of present preferences. Each needs someone, or something, to work out what the idealized person would want, and that is where the objections gather.

The alignment field's own proposal of this kind is coherent extrapolated volition, its author's term, from a 2004 report by Eliezer Yudkowsky for the Singularity Institute (later the Machine Intelligence Research Institute), whose definition Section 2.3.1 quotes (Yudkowsky, 2004). Its starting example is small. Fred wants a diamond, guesses that it is in box A, and asks for box A, but the diamond is in box B. Yudkowsky applies "the term volition to describe the sense in which you may be said to want box B, even though your guess leads you to pick box A" (Yudkowsky, 2004). That is the informed preference of Section 1.1.1 in miniature. The hard part is the general case. Extrapolating a volition needs a procedure, which the report calls the dynamic and describes as "something complicated" performed by "a generally intelligent AI." Writing it as an ordinary program, the report says, "would be a thousand lightyears beyond hopeless" (Yudkowsky, 2004).

The report states its own limits. It measures where extrapolation fails by three quantities, spread (the extrapolated volition is unpredictable or random), muddle (it contradicts itself), and distance (it is hard to explain to the present person), and calls them "ad hoc" (Yudkowsky, 2004). And it grants that a fully educated Fred is not one Fred: different versions of him "would not write identical code, and would probably write different AIs" (Yudkowsky, 2004).

The idea is older than alignment. In a 2024 paper at ICML, Micah Carroll and colleagues set it beside the ideal observer theory of the philosopher Roderick Firth, on which things are evaluated as an ideal observer would evaluate them (Carroll et al., 2024; Firth, 1952). They quote Richard Brandt's description of such an observer as "fully informed and vividly imaginative, impartial, in a calm frame of mind and otherwise normal," and note that Firth "goes as far as saying that they should be omniscient and omnipercipient." Their objection is practical: the idealized states such evaluations require "don't seem realizable in the real world" (Carroll et al., 2024), so no one can collect the idealized feedback the target is defined by. They suggest instead letting a person's reward function evaluate changes to that reward function, what they call meta-preferences, which may "capture notions of the legitimacy of influence and personal autonomy" (Carroll et al., 2024).

A second objection comes from the same thought experiments philosophy uses as evidence. In the experience machine, a person can plug into a device that makes life feel slightly more pleasant while it is lived inside a fiction. The philosophers William MacAskill, Krister Bykvist, and Toby Ord, in their 2020 book on moral uncertainty, use the case to show that someone unsure whether pleasure is all that matters has reason not to plug in, since a life in the machine would not "realize the objective goods of achievement and friendship" (MacAskill et al., 2020). For a target defined by what a person would endorse, it shows that the person may endorse not being given what they would experience as best, and for alignment it is the philosophical form of reward tampering (Section 30.5): the measure of a good raised without the good (inference).

Three proposals replace idealized preference with something else, and each is taken up elsewhere. Gabriel's own target is principles that people who disagree could reflectively endorse (Section 1.1.1). Gabriel and Geoff Keeling, in a 2025 paper in Philosophical Studies, argue by their abstract for "an alternative account of AI alignment that focuses on fair processes," whose output meets "the necessary standard of public justification" (Gabriel and Keeling, 2025); Section 17.5 weighs procedures of that kind. And Zhi-Xuan and colleagues propose norms appropriate to a system's social role in place of preferences (Section 13.6.4).

What the family implies is a circularity (inference). Every idealized target needs a model of the idealization, and coherent extrapolated volition assigns the modeling to a capable AI. The trained system that computes what its principals would want on reflection is then the party whose values are in question. Chapter 41 inherits the problem: an evaluator weaker than the system it checks must judge whether the system's account of the evaluator's own idealized wishes is right.

16.3 Acting under moral uncertainty #

Suppose the people who build a system disagree about which moral theory is right, or that each of them is unsure. The system must still act, so it acts on some rule for that uncertainty, whether or not anyone wrote one down. This section teaches the standard rule and the reason it fails without an extra assumption.

From decision theory. Decision theory already has a rule for acting when the state of the world is uncertain: maximize expected utility (Section 7.2). MacAskill, Bykvist, and Ord argue in their book, Moral Uncertainty, that moral and empirical uncertainty should be treated "in the same way" (MacAskill et al., 2020). Each moral theory is assigned a credence, a degree of belief that it is correct, and each scores the options by its choiceworthiness, how strongly it recommends them. Their rule, which they call maximizing expected choiceworthiness, is to choose an option with "the maximal expected choiceworthiness" (MacAskill et al., 2020).

This is the problem of intertheoretic comparisons. MacAskill and colleagues put it with a case: a rights-based theory holds that it would be wrong to kill one person to save fifty, utilitarianism that it would be wrong not to, and "for which theory is there more at stake?" (MacAskill et al., 2020). They report that some philosophers took the problem to be fatal, among them, in conversation, John Broome, who called it "devastating," and Derek Parfit, who called it "fatal" (MacAskill et al., 2020). The same problem arises when people take the place of theories: Harsanyi's theorem, stated in Section 17.1, makes a social choice a weighted sum of individual utilities and leaves the weights open.

Their answer borrows from social choice, the theory of combining many voters' rankings (Chapter 17). For a theory that only ranks options, they propose the Borda Rule: each theory gives an option one point for every option it ranks lower and minus one for every option it ranks higher, and the points are weighted by credence (MacAskill et al., 2020). Their chapter on ordinal theories rests on MacAskill's 2016 paper on normative uncertainty as a voting problem (MacAskill, 2016). For theories with numerical scales that cannot be compared, they propose variance voting: before taking the expectation, rescale each theory so that its scores across the options have mean zero and the same spread, measured by the standard deviation. They argue that this is "uniquely privileged as the account that gives incomparable theories equal say" (MacAskill et al., 2020). Both rules give the same verdict whatever $s$ is. The authors also record two objections to their own Borda Rule: it is sensitive to how the options are individuated, and it violates a condition called contraction consistency (MacAskill et al., 2020).

The same rule appears inside reward models. Section 13.5.2 shows that one reward fitted to many raters' comparisons ranks responses by their Borda count, so moral uncertainty and preference aggregation share their mathematics, and a choice of rule in either is a social choice (Section 17.2).

A third family treats the theories as delegates who bargain. Hilary Greaves and Owen Cotton-Barratt, in a 2023 paper in the Journal of Moral Philosophy, make the "parliamentary approach" precise with "a version of the Nash solution that is standard in bargaining theory" (Greaves and Cotton-Barratt, 2023; Nash, 1950). In the form this chapter's figure uses, a Nash solution picks the option that maximizes the product of each theory's gain over a disagreement point, what the theory would get if bargaining failed, with each gain raised to the power of the theory's credence. Rescaling one theory multiplies the product by the same factor for every option, so the verdict does not change; but it does depend on where the disagreement point is set (the book's computation). By their abstract, their model "does indeed perform in many of the hoped-for ways." So does a version of expected choiceworthiness with a structural approach to intertheoretic comparisons, and "it seems to us an open question" which of the two is superior (Greaves and Cotton-Barratt, 2023).

Figure 16.1 puts the four rules side by side. Only the raw expectation moves with a theory's arbitrary unit. The three rules that do not move still disagree with each other: variance voting and Borda follow the theory with more credence, while bargaining protects each theory from its worst option and lands on the compromise.

Not everyone accepts that moral uncertainty should be hedged at all. MacAskill and colleagues list five objections from the literature and judge the fetishism objection the strongest: an agent who weighs theories cares about rightness as such rather than about what makes acts right, such as helping the needy and keeping promises (MacAskill et al., 2020). They attribute it to Brian Weatherson. A regress objection adds that one can be uncertain how to handle moral uncertainty, and uncertain about that, without end; they cite Weatherson and Elizabeth Harman's 2015 paper "The Irrelevance of Moral Uncertainty" for it (MacAskill et al., 2020; Harman, 2015). Their replies are that even a fetishist can act more or less rationally given its beliefs, and that the regress commits a scope fallacy: from the possibility of uncertainty at each level it does not follow that one can be uncertain at every level at once (MacAskill et al., 2020). Gabriel adds an objection from political philosophy: a decision-maker using the parliamentary model "would still need to communicate the validity of their conclusions to other people who hold a different set of reasonable credences" (Gabriel, 2020).

Adrien Ecoffet and Joel Lehman, in a 2020 preprint from work at Uber AI Labs, built the rules into reinforcement learning agents (Ecoffet and Lehman, 2020). Noting that the raw expectation "is scale-sensitive," they implement variance voting and a second method they call Nash voting. In it, theories cast votes and play to a Nash equilibrium, a state in which no theory gains by changing its vote alone; it is distinct from the bargaining solution above. They name its two flaws themselves: stakes insensitivity, where raising one theory's stakes does not raise its say, and no compromise, where an option that is no theory's favorite cannot be chosen (Ecoffet and Lehman, 2020). Their agents were trained in grid worlds, small simulated environments laid out as grids, built on the trolley problems of moral philosophy, dilemmas about diverting harm from many people to fewer. In the abstract's words, the results illustrate "how such uncertainty can help curb extreme behavior from commitment to single theories" and "several technical complications arising from attempting to ground moral philosophy in RL" (Ecoffet and Lehman, 2020). The environments are toy dilemmas, so the result shows the rules can be implemented and produce different behavior, not which behavior is right.

What the decision theory implies for specification is that it moves the "whose" question rather than answering it (inference). A system acting under moral uncertainty needs credences over theories and a rule for combining them, and someone sets both. Ecoffet and Lehman assume the credences "are set and fixed, e.g. by the system designer's beliefs, or by taking a survey of relevant stakeholders" (Ecoffet and Lehman, 2020). Either source makes them a specification choice of the kind Chapter 17 examines. Whether intertheoretic comparisons can be made, whether bargaining or a structural comparison is better, and whether to hedge at all are contested, and the arguments do not decide them.

16.4 Pluralism and the single scale #

Moral uncertainty assumes that one theory is right and we do not know which. The philosophy part now takes up a stronger claim: that a person's values may be many, may conflict, and may not be measurable on one scale even with nothing left to learn. If so, a scalar reward misdescribes them, and the design question becomes how a system should act when its values conflict.

The claim has a political form that does not depend on any metaethics. Gabriel draws on John Rawls's "fact of reasonable pluralism": even someone who believes they have found the moral truth is unlikely to "persuade other people of this truth using evidence and reason alone" (Gabriel, 2020). Designing AI to a single moral doctrine would therefore mean "imposing a set of values and judgments on other people who did not agree with them," and for powerful technologies "this quest to encode the true morality could ultimately lead to forms of domination" (Gabriel, 2020). On this argument even a realist who is right faces the problem of whose values, since being right does not give the right to impose (inference).

The philosophical form, value pluralism, holds that values themselves do not reduce to one measure. A 2024 preprint led by Taylor Sorensen, which proposes "pluralistic alignment," lists Isaiah Berlin among the theorists of political pluralism and Thomas Nagel among those of moral and value pluralism (Sorensen et al., 2024; Nagel, 1979). The same preprint concedes the point against its own method. Its framework for steering a model among objectives "assumes a set of commensurating functions," while "many philosophers who subscribe to value pluralism believe that values are incommensurable and cannot be traded off"; such benchmarks "(and most of machine learning) are incompatible with that view" (Sorensen et al., 2024). Section 13.6.4 recalls the same objection to preferences, and Ruth Chang's volume on incommensurability, from the side of the reward model.

Two structures in ethics resist a single scale, and each has an engineering counterpart. The first is the constraint. In Gabriel's summary, rights are "claims by individuals against collectives that resist aggregation and mark out a domain of things that cannot permissibly be done to them," working as "trumps" against general utility or as "side-constraints on otherwise optimizing action," the latter in Robert Nozick's phrase (Gabriel, 2020; Nozick, 1974). He adds that it "may be difficult to robustly specify and guarantee rights-respecting behaviour on the part of agents whose learning process and decision-making are guided primarily by an optimization function" (Gabriel, 2020). The second is the agent's own commitments. In the 1973 volume Utilitarianism, shared by J. J. C. Smart and Bernard Williams, Williams argues, by the book's abstract, that utilitarianism "fails to make sense of notions such as integrity" (Smart and Williams, 1973). A model specification that asks a system to keep some commitments whatever the payoff, such as never deceiving its user, takes Williams's side (inference).

A narrower formal result is proved. Section 7.5.2 shows that some tasks, stated as sets of acceptable behaviors or partial orderings, can be captured by no Markov reward at all in a fixed environment, that is, by no reward that depends only on the current state and action. That theorem limits one form of reward; it does not show that values share no single scale, which remains the philosophers' argument (inference). A specification that faces incommensurable values has three structures available: refuse the trade with a constraint, rank the values lexically so that one always wins, or hand the trade to a person or a procedure in each case (inference). Chapter 19 compares lexical orderings in model specifications with the legal traditions that have lived with ordered objectives, and the next two sections show how medicine chose the third.

Pluralism also has empirical counterparts: surveys of what people in many countries value, and theories of moral judgment that posit several foundations. Those instruments and their contested theories are Chapter 18's. Here it is enough that the philosophical claim does not depend on them, and that they do not settle it, since a measured spread of values can arise from disagreement about a single scale as easily as from several incommensurable ones (inference).

16.5 Principles, specified and balanced #

From bioethics. Medicine met pluralism as a practical problem. Research on human subjects, in the words of the report discussed below, "posed some troubling ethical questions," and decisions about particular studies could not wait for agreement on moral theory (National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research, 1979). The method medicine built is a small set of principles that are neither a full moral theory nor a list of cases, applied to cases by people who are answerable for the result. AI ethics guidelines have since converged on principles that resemble medicine's. This section and the next are the bioethics part of the chapter.

16.5.1 The Belmont method #

That document is the Belmont Report, issued on April 18, 1979, by a United States commission that the National Research Act of 1974 created (National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research, 1979). It names three principles for research on human subjects: respect for persons, beneficence, and justice. Each has an application: informed consent, the assessment of risks and benefits, and the fair selection of subjects (National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research, 1979).

The report explains why it chose principles over rules, and the explanation fits model specifications closely. Rules, it says, "often are inadequate to cover complex situations; at times they come into conflict, and they are frequently difficult to interpret or apply." Principles provide a basis on which rules "may be formulated, criticized and interpreted," but they "cannot always be applied so as to resolve beyond dispute particular ethical problems" (National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research, 1979). The report then shows what the gap costs. Whether to let prisoners volunteer for research or to protect them from subtle coercion is a dilemma that one principle poses against itself: "Respecting persons, in most hard cases, is often a matter of balancing competing claims urged by the principle of respect itself" (National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research, 1979).

The balancing is not a calculation. Of the common demand that risks and benefits be "balanced" and shown "in a favorable ratio," the report says that "the metaphorical character of these terms draws attention to the difficulty of making precise judgments," and that "only on rare occasions will quantitative techniques be available" (National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research, 1979). It asks instead for systematic assessment, and it names who uses it: for "a review committee, it is a method for determining whether the risks that will be presented to subjects are justified" (National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research, 1979). The method has three parts, then: principles, applications that make them concrete, and a body that decides the hard cases and can be asked why (inference). The approach goes by the name principlism, which Gabriel describes as "an approach commonly found in medical ethics," citing Tom Beauchamp and James Childress's textbook Principles of Biomedical Ethics (Gabriel, 2020; Beauchamp, 1994), whose authors marked its fortieth anniversary in 2019 (Beauchamp and Childress, 2019).

16.5.2 The copy in AI ethics #

The map of AI ethics guidelines by Anna Jobin, Marcello Ienca, and Effy Vayena is reported in Section 2.5.1: five principles in common, and "substantive divergence" in how they are read (Jobin et al., 2019). For whose values they carry, the issuers matter. Of the 84 documents, the largest groups came from private companies (19) and governmental agencies (18). The United States and the United Kingdom together accounted for more than a third, and "African and South-American countries are not represented independently from international or supra-national organisations" (Jobin et al., 2019).

Two critiques draw the comparison with medicine directly. Jess Whittlestone and colleagues, at a 2019 conference on AI ethics, argue by their abstract that principles "are often too broad and high-level to guide ethics in practice," and that the next step is "exploring the tensions that inevitably arise" in applying them (Whittlestone et al., 2019). Brent Mittelstadt, in Nature Machine Intelligence the same year, argues by his abstract that the principles of AI ethics "closely resemble the four classic principles of medical ethics." Compared with medicine, he argues, AI development lacks "(1) common aims and fiduciary duties, (2) professional history and norms, (3) proven methods to translate principles into practice, and (4) robust legal and professional accountability mechanisms" (Mittelstadt, 2019). His conclusion is that "we should not yet celebrate consensus around high-level principles that hide deep political and normative disagreement" (Mittelstadt, 2019).

What bioethics implies for specification is that principles work only with the institution that applies them (inference). A model specification already has the first two parts: principles, and worked examples that apply them. The third part is where the analogy stops. In medicine, review committees, professional bodies, and courts do the specifying and balancing, and their decisions can be appealed and reviewed by people who did not make them. A developer that writes a model specification does both in one document that it can amend alone, and the trained model then balances in every conversation with no one to appeal to (inference). Whether a review body outside the developer could do for model behavior what review committees do for research is a question about the laboratory as the public's agent (Chapter 61). The case-based alternative to principles, casuistry, is taught where specifications are read (Section 19.4).

16.6 Deciding for those who cannot say #

The bioethics part continues with a narrower problem that every assistant meets: an agent must decide for a principal who cannot, at that moment, state what they want. Medicine and law have standards for this, a record of how they work, and a recent turn away from one of them. Each maps onto a design choice.

16.6.1 Three standards #

Three standards appear in this literature. The first is a written instruction, an advance directive, prepared while the person could still state their wishes. The second is substituted judgment, in which a surrogate decides as the person would have; Brian Earp and colleagues describe surrogates trying "to guess what the patient would want if they had capacity" (Earp et al., 2024). The third is best interests, in which the decision follows what others believe is objectively good for the person (Committee on the Rights of Persons with Disabilities, 2014). Allen Buchanan and Dan Brock's Deciding for Others, by its publisher's abstract, defends principles "to specify suitable surrogate decisionmakers and to guide their choices" (Buchanan and Brock, 1990).

Read as a fallback chain, the three give an assistant an order to follow: the principal's own instructions where they cover the case, an inference of what the principal would choose where they do not, and the principal's good where neither is available (inference). They also map onto the targets of Section 1.1.1: a directive is an instruction, substituted judgment an inferred intention or informed preference, and best interests the principal's interest or well-being (inference). A directive is a specification written before the situations it must govern, which every model specification is, and Chapter 19 takes up what such documents leave open.

How well the inference in substituted judgment works has been studied in the surrogates who know a patient best; Section 18.6 takes up that evidence and what it would mean as a human baseline for a system that infers a principal's wishes.

16.6.2 Models as surrogates #

Medicine has already proposed a model as surrogate. Earp and colleagues, in a 2024 paper in The American Journal of Bioethics, describe an earlier proposal, a patient preference predictor that "would use an algorithm to infer the treatment preferences of individual patients from population-level data about the known preferences of people with similar demographic characteristics" (Earp et al., 2024; Rid and Wendler, 2014). Critics objected that even if it were more accurate than human surrogates, it "would nevertheless fail to respect the patient's (former) autonomy" because it used data not specific to the patient. Earp and colleagues propose a personalized version built by fine-tuning language models on "person-specific data," such as prior treatment decisions, and argue, by their abstract, that it is technically feasible and should "assuage some of the main autonomy-based concerns" (Earp et al., 2024). As of October 2026 we found no study comparing such a predictor with human surrogates on the same patients. These are the clearest existing cases of an AI system asked to infer a principal's will under formal standards. They show that the standards apply to the data as well as to the decision: whose evidence the inference rests on is part of what makes it the principal's will (inference).

16.6.3 Supported rather than substituted #

International disability law now asks for an end to substitution. In 2014 the United Nations committee that interprets the Convention on the Rights of Persons with Disabilities issued its General Comment No. 1 on Article 12, "Equal recognition before the law" (Committee on the Rights of Persons with Disabilities, 2014; United Nations General Assembly, 2006). It holds that the human-rights model of disability "implies a shift from the substitute decision-making paradigm to one that is based on supported decision-making" (Committee on the Rights of Persons with Disabilities, 2014). It defines substitute regimes by three features. First, legal capacity, which it defines as "the ability to hold rights and duties (legal standing) and to exercise those rights and duties (legal agency)," is removed, "even if this is in respect of a single decision." Second, the substitute can be appointed by someone else, against the person's will. Third, decisions follow "the objective 'best interests' of the person concerned, as opposed to being based on the person's own will and preferences" (Committee on the Rights of Persons with Disabilities, 2014).

The replacement is specific. Support "must respect the rights, will and preferences of persons with disabilities and should never amount to substitute decision-making," and the person keeps "the right to refuse support and terminate or change the support relationship at any time" (Committee on the Rights of Persons with Disabilities, 2014). Planning in advance is itself a form of support: the point at which an advance directive takes effect "should be decided by the person," and it "should not be based on an assessment that the person lacks mental capacity." Where, after significant efforts, a person's will cannot be determined, "the 'best interpretation of will and preferences' must replace the 'best interests' determinations" (Committee on the Rights of Persons with Disabilities, 2014). Eilionoir Flynn and Anna Arstein-Kerslake argue, by their abstract, for "a conception of personhood that is divorced from cognition" and a support model as the alternative to "substituted decision-making that deny legal capacity and impose outside decision-makers" (Flynn and Arstein-Kerslake, 2014). Philosophy offers a matching account of autonomy, relational autonomy, the view that autonomy has social and relational dimensions, set out in the essays collected by Catriona Mackenzie and Natalie Stoljar (Mackenzie and Stoljar, 2000).

The two documents pull in different directions, and the tension is the design choice (inference). The Belmont Report holds that "persons with diminished autonomy are entitled to protection," a duty that a best-interests standard is one way to discharge (National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research, 1979). The General Comment holds that "the 'best interests' principle is not a safeguard which complies with article 12 in relation to adults" (Committee on the Rights of Persons with Disabilities, 2014). Neither document settles the other's case, and as of October 2026 we found no study comparing outcomes under the two regimes.

For assistants, supported decision-making is a standard that neither obeys nor overrides (inference). It sits between the delegate and the trustee of Section 1.2, and Section 60.6 tests the deployed evidence against it. The analogy stops in three places. A patient's incapacity is usually a state that others assess, which the General Comment contests; a user is usually reachable and competent, so the question is when an assistant should act at all before asking. A human surrogate is named and answerable, and the General Comment requires "a mechanism for third parties to challenge the action of a support person" (Committee on the Rights of Persons with Disabilities, 2014); a model acting as surrogate has no such standing. And the patient preference predictors show that a model's inference may be more accurate and still not be the person's own, which is the gap between substituted judgment and support.

16.7 Traditions as design choices #

The positions so far come mostly from Western philosophy and Western institutions. The common way to widen them is to add the values of other traditions to a list. A more useful way is to ask which assumption of the field's methods each tradition rejects, since a rejected assumption names an alternative design that could be built and compared. This section gives the frame and the map; the book works each tradition at the assumption it challenges.

16.7.1 The frame #

Shakir Mohamed and William Isaac of DeepMind, with Marie-Therese Png of the University of Oxford, set out the frame in a 2020 paper on decolonial theory and AI. Their abstract puts the premise in one line: "Values and power are central to this discussion" (Mohamed et al., 2020). Using the language of metropole and periphery, centers of power and the regions that contest them, they write that "today's technology corporations could be described as metropoles of technological power with civic society and consumers sitting at the periphery." They propose three tactics: a critical technical practice of AI, "reverse tutelage" in which the center learns from the periphery, and the renewal of affective and political communities (Mohamed et al., 2020).

The appeal to other traditions has its own critic. Pak-Hang Wong, in a 2020 commentary, warns that cultural difference can be used as an excuse. It may enable "malignant actors to disregard the demand of important ethical values or even to justify the violation of them through deference to the local culture" (Wong, 2020). His examples are a government statement that privacy is foreign to India and a defense of China's social credit system by appeal to tradition. He concludes that a human rights approach to AI governance must still take cultural values into account, for both philosophical and instrumental reasons (Wong, 2020). Seán Ó hÉigeartaigh and colleagues, writing on cooperation between Europe, North America, and East Asia, argue by their abstract that agreement on practical issues is sometimes possible "despite disagreement on more abstract values or principles" (ÓhÉigeartaigh et al., 2020). Gabriel's candidate for common ground is human rights, which he reports have "justifications found in African, Islamic, Western, and Confucian traditions of thought" (Gabriel, 2020).

16.7.2 The map #

Each tradition below is placed at the assumption it attacks.

Relational personhood. The Sub-Saharan African ethics of ubuntu, on which "a person is a person through other persons," challenges taking an individual's pairwise judgment as the unit of value; Section 13.5.5 reports the argument and works it at the likelihood of a reward model (Mhlambi, 2020; Metz, 2007). Arthur Gwagwa and colleagues argue, by their abstract, that ubuntu could serve "an inclusive approach for the implementation of the universal AI ethics principles and guidelines" (Gwagwa et al., 2022).

Role ethics. Wong's 2012 account of a Confucian ethics of technology describes the Confucian person, following Erika Yu and Fan Ruiping, as "relational, developmental, and virtue-based," and treats harmony as something reached "by continuous negotiation and adjustment" (Wong, 2012; Ames, 2011). That challenges the assumption of a principal without a role: what an assistant owes depends on whose assistant it is and in what relationship, which is the role-relative norm Zhi-Xuan and colleagues propose (Section 13.6.4). The tradition's account of when a minister must object to a ruler is worked beside corrigibility in Section 20.5.

Ordered objectives. Amana Raquib and colleagues ground an Islamic virtue ethics for AI in the objectives of Islamic law, the maqāṣid (Raquib et al., 2022). These follow "a hierarchy in terms of prioritization": essentials, then needs, then enhancements, with five essential objectives (religion, life, progeny, property or wealth, and intellect) to be preserved. The authors add that the categories "are not absolute and vary depending on the circumstances," and that "the good of the community always has priority over the good of an individual" (Raquib et al., 2022). Ezieddin Elmahjub argues for "a pluralist approach to ethical AI benchmarking" and notes that welfarist and duty-based readings of the good, maṣlaḥa, coexist in Islamic jurisprudence (Elmahjub, 2023). The assumption challenged is that objectives are unordered; Section 19.4 works the comparison with lexical orderings in model specifications.

Interdependence and suffering. Soraj Hongladarom's Buddhist account, by its abstract, sets as the standard for AI ethics a "machine enlightenment" characterized by "the realization that all things are interdependent" and the commitment to relieve suffering (Hongladarom, 2020). Peter Hershock's 2021 book Buddhism and Intelligent Technology is the other work in this line that Chapter 60 draws on (Hershock, 2021). The assumption challenged is that engagement counts as benefit, and Chapter 60 works it against engagement optimization.

Collective authority. The Indigenous Protocol and Artificial Intelligence position paper of 2020 sets out "Guidelines for Indigenous-centred AI Design," among them locality, relationality and reciprocity, and data sovereignty: "Indigenous communities must control how their data is solicited, collected, analysed and operationalized" (Lewis et al., 2020). The assumption challenged is individual consent as the unit of authority, and Section 17.4 works it with collective data rights.

State-held values. China's ethical norms for new-generation AI, issued in September 2021 by its national AI governance committee, list six basic requirements. One is to ensure that AI is controllable and trustworthy: people keep the right to choose whether to accept an AI service, to leave an interaction, and to stop a system at any time, so that AI stays under human control. The first requirement, to enhance human well-being, includes abiding by national or regional ethics (National Governance Committee for New Generation Artificial Intelligence, 2021). The assumption challenged is that the developer or the user is the principal who sets values; states that write specifications are Section 17.5's subject.

The map has a limit that applies to every row (inference). Few of these works state a design that would behave differently on the same inputs, and as of October 2026 we found no study that trains a model to a relational, role-based, or ordered specification and compares it with a principle-based one on the same base model. Each placement is therefore a hypothesis with a test, not a finding. The traditions are also plural inside: Elmahjub's two readings of the good and Wong's warning about appeals to culture show that a tradition does not deliver one target either, so naming a tradition moves the question of whose values without closing it.

16.8 Is "whose values" the right question? #

The philosophy part returns for a position the chapter has so far treated as background: that the question in its title is the wrong one. The critics do not agree with each other, but each denies something the question assumes. Their objections fall into three groups, and each has an answer.

The first objection is that values are the wrong unit. Preferences and values describe what people want; Zhi-Xuan and colleagues argue that a system should instead meet "normative standards appropriate to their social roles," negotiated by "all relevant stakeholders" (Zhi-Xuan et al., 2025), and Section 13.6.4 reports the argument. Atoosa Kasirzadeh and Gabriel, in a 2023 paper in Philosophy & Technology, derive norms for conversational agents from the structure of conversation and ask how they vary "across a range of different discursive domains" (Kasirzadeh and Gabriel, 2023); Section 19.5 uses their analysis. On this view, asking whose values a system holds is like asking whose values a judge holds: the useful question is what the role requires.

The second objection is that the question is about power, not values. Seth Lazar and Alondra Nelson's argument that no group of experts should decide alone "to which values safe AI should be aligned," and the claim that calling a choice "alignment" turns a political question into a technical one, are reported in Section 2.5.2 and Section 2.5.3 (Lazar and Nelson, 2023). Mohamed and colleagues make the point about knowledge itself: "Deciding what counts as valid knowledge, what is included within a dataset and what is ignored and unquestioned is a form of power held by AI researchers that cannot be left unacknowledged" (Mohamed et al., 2020). On this view the question is not whose values are right but who decides, and by what right.

The third objection is that a model should not hold moral views at all. Talat and colleagues, after their critique of Delphi, propose that machine ethics focus on "current and near-future uses of technology, in a way that centers around transparency, democratic values, and allows for straightforward accountability" (Talat et al., 2022). Their premise, that "ethical evaluation is an open-ended, debate-based, sociopolitical process," puts moral judgment with people and institutions rather than in a system's dispositions (Talat et al., 2022). The rival accounts of what a model is, as a cultural and social technology or a normal technology governed by its uses, point the same way (Section 1.6) (inference).

Each objection has a reply, and the replies are the book's reading (inference). To the first, Gabriel's paper already moves from values to principles chosen by fair procedures, and Gabriel and Keeling's fair treatment of claims is a version of the role-and-process view (Gabriel, 2020; Gabriel and Keeling, 2025). To the second, the critics and Gabriel agree more than they disagree: his proposal to find "fair principles for alignment" answers who decides with a procedure, and the dispute is over which procedure confers legitimacy (Chapter 17). To the third, a model that answers questions already acts on some values, so declining to state them does not remove the choice. One developer's report on its own method said so plainly, as Section 2.5.3 quotes: its procedure aligned a model to the preferences of "a specific group of people (mostly our labelers and researchers)" (Ouyang et al., 2022). Whether moral judgment should be built into a system's dispositions, or kept to procedural limits with the judging left to people, is contested, and the evidence does not decide it.

16.9 Preferences that move #

Every target so far has been fixed while the system acts on it. People's preferences are not fixed, and a deployed system can move them. Section 13.1.2 recalled platforms whose users choose in the moment against what they want, and Section 13.6.3 showed that a system trained to satisfy people has an incentive to make them easier to satisfy. The specification question this leaves open is the last one the philosophy part takes up: whose preferences count when the person after using the system differs from the person before.

16.9.1 The formal statement #

Carroll and colleagues, in the ICML paper of Section 16.2, state the problem as a dynamic reward Markov decision process. A Markov decision process is the standard model of sequential choice, in which each action moves an agent between states and earns a reward (Section 7.4); in the dynamic version the human's reward function changes over time and the system's actions can change it (Carroll et al., 2024). Their example is an assistant whose user, Alice, once preferred healthy food and has since come to prefer something else; "it may be unclear if the AI assistant should respect Alice's original preference for healthy foods or respect the autonomy of 'current Alice'" (Carroll et al., 2024). Another, adapted from Parfit, is a man whose greatest ambition is to be a poet, whose assistant could motivate him to pursue it, and who would find the poet's life aversive once he had it (Carroll et al., 2024).

Their first result concerns current practice. The static-preference assumption, they argue, "may undermine the soundness of existing alignment techniques, leading them to implicitly reward AI systems for influencing user preferences in ways users may not truly want" (Carroll et al., 2024). Common methods for recommender systems and reward modeling, they find, resemble objectives that "actively reward AI systems for influencing users' reward functions" (Carroll et al., 2024). Shortening the optimization horizon, how far ahead the system's objective counts consequences, helps only partly, since some influence is optimal even at a horizon of one step, as in clickbait.

Their second result concerns the remedies. They formalize eight notions of alignment that account for changing preferences, and "find that they all either err towards causing undesirable AI influence, or are overly risk-averse" (Carroll et al., 2024). Some of the risk-averse ones make inaction "the only behavior considered optimal for many settings." Read with Parfit's work, their analysis "suggests that it may not be possible to ground a definitive notion of optimality under changing preferences" (Carroll et al., 2024). The authors remain "cautiously optimistic," since people help one another acceptably without such a theory. This is a formal analysis by one group, with no experiment on deployed systems; Chapter 60 reports the deployed evidence.

16.9.2 Influence that is legitimate, and influence that is not #

Not every change a system causes is a harm. Section 13.6.3 recalled experiences that change what a person values as well as what they know, the subject of L. A. Paul's book Transformative Experience (Paul, 2014), and a tutor who makes a student care about mathematics has changed a preference legitimately (inference). The question is which changes are manipulation, and the philosophical accounts disagree in a way that matters for measurement.

Daniel Susser, Beate Roessler, and Helen Nissenbaum, in a 2019 paper in Internet Policy Review, define online manipulation as "the use of information technology to covertly influence another person's decision-making, by targeting and exploiting their decision-making vulnerabilities" (Susser et al., 2019). They separate it from its neighbors: persuasion offers reasons a person can evaluate, coercion constrains options, and both are open, while manipulation "is covert" (Susser et al., 2019). On their account the deepest harm is to autonomy. Robert Noggle, by the abstract of a 2020 paper, finds that the two main accounts, manipulation as trickery and as non-coercive pressure, each fit "only a subset of intuitively clear cases," and unifies them: manipulation is "the attempt to get someone to make a mistake" (Noggle, 2020). Carroll and colleagues, writing about AI systems, "characterize the space of possible notions of manipulation" rather than settle on one: the notions depend on incentives, intent, covertness, and harm, with intent read as behaving "as if" pursuing an incentive (Carroll et al., 2023). That reading of intent, taught in Section 14.7.2, lets a system without human-like intentions be said to manipulate.

Between the two lies a defended middle. Susser and colleagues describe Richard Thaler and Cass Sunstein's nudge as "any intentional alteration of another person's decision-making context (their 'choice architecture') made in order to influence their decision-making outcome" (Susser et al., 2019; Thaler and Sunstein, 2008). Its justification is that designers cannot avoid influencing choices and might as well do it for the chooser's benefit (Susser et al., 2019). Whether nudges manipulate, they report, "no consensus has been reached" (Susser et al., 2019). The Belmont Report met the same boundary in consent to research: "it is impossible to state precisely where justifiable persuasion ends and undue influence begins" (National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research, 1979). The interference these accounts worry about, overriding a person's choice for their own good, is paternalism, the subject of Gerald Dworkin's 1972 essay of that title (Dworkin and The Hegeler Institute, 1972); Chapter 20 returns to it when it judges refusals and overrides.

The three accounts predict different tests (inference). Susser and colleagues' test is covertness: did the influence work in a way the person was not aware of and could not easily discover, by exploiting a weakness in their decision-making? Whether the person would approve on seeing it is a separate question, since on their account manipulation can still be justified by a greater benefit (Susser et al., 2019). Noggle's is error: did the influence lead the person to a mistake by their own standards? Carroll and colleagues add an axis the other two lack, incentive: did the system's training reward the change? They hold an incentive neither sufficient nor necessary for manipulation, so it is weighed with intent, covertness, and harm (Carroll et al., 2023). A system can fail one test and pass another. A visible streak counter that exploits loss aversion is not covert but may induce mistakes; a hidden reordering that leads a reader to better books is covert but may induce none.

What this implies for specification is that it must say which version of the principal the system serves: the present one, the reflective one, or the one who would have existed without the system's influence (inference). Each answer needs a measurement the system cannot itself move, since a system that can shift the preferences it is scored against can satisfy any of the three by changing the person. Of the eight notions Carroll and colleagues formalize, none escapes both undue influence and excessive caution, so the choice is a trade-off that the specifier has to own.

16.10 What the question means for delegation #

The book's thesis is that alignment is a problem of delegation under an asymmetry of competence and visibility (Section 1.7.1). Every model of delegation the book draws on assumes that the principal knows its own preferences well enough to price each outcome, the assumption Section 14.8 lists as A9 and says specification denies. This chapter shows how far it fails. Whether the principal's objective is a fact, a procedural output, or a choice is unsettled; acting under uncertainty about it needs credences that someone sets; and the principal's preferences can move under the agent's influence. The delegation problem therefore starts before any agent is trained, with the question of who the principal is and what it wants (inference).

The claims divide by relationship. Who chooses the target, and on what authority, concerns the developer and the public: the developer writes the specification, the public is bound by it, and the critics of Section 16.8 hold that this is the whole of the question. Whose preferences count, and which version of them, concerns the deployed system and the people it serves, as in the surrogate standards and in preferences that move. The training relationship enters where a loss function decides without saying so, as when one reward model ranks responses by the Borda count of its raters (Section 13.5.2).

Where older structures converge with the thesis, they are bioethics' and political philosophy's: mid-level principles applied by an answerable review body, ordered standards for deciding on another's behalf, and fair procedures for choosing principles under disagreement (inference). Where they break is the developer's double role. A review committee does not write the principles it applies, and a surrogate does not shape the patient's preferences; a developer can write the principles, train the agent that applies them, and influence the preferences of the users the agent serves. Two items belong with the book's list of results that have no counterpart in the older fields (inference). One is a principal that writes the agent's character, which this chapter meets as a developer writing a specification it alone can amend. The other comes from idealized targets: an agent asked to compute what its principals would want on reflection becomes the author of its principal's idealized will. Both are recorded for Chapter 69.

16.11 Settled, contested, missing #

16.12 Exercises #

Further reading #

• Gabriel (2020): the targets of alignment, and the case for fair principles that people who disagree can endorse.
• MacAskill et al. (2020): decision procedures for acting when the right moral theory is uncertain, with the problem of comparing theories' units.
• National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research (1979): principles, applications, and a review body, the method medicine built for pluralism in practice.
• Committee on the Rights of Persons with Disabilities (2014): the case for supported rather than substituted decision-making, stated as obligations.
• Mohamed et al. (2020): values and power as one question, and tactics for reading other traditions as alternative designs.
• Zhi-Xuan et al. (2025): why preferences may be the wrong currency of alignment, and role-appropriate norms in their place.