Draft complete but not yet reviewed; expect revisions

Aggregation and Legitimacy

A model that serves many people cannot follow each person's values in every respect: somewhere its behavior is one answer to people who disagree. Chapter 16 asked what kind of thing the target is, a fact to discover, the output of a fair procedure, or a choice someone must justify. This chapter asks how many people's inputs become one target, and why anyone should accept the result. The first question has a mathematical theory, social choice, with seventy-five years of theorems, and its central results say that no way of combining rankings meets every condition that seems reasonable. One way of combining them is already in use: a reward model, the learned scoring function that training then maximizes, ranks responses by a voting rule that no one chose when it is trained on many raters' comparisons (Section 13.5). The chapter sets that rule beside its alternatives, and beside the older impossibility of fair classification.

The second question belongs to political science and its neighbors. A rule can satisfy every axiom and still lack authority over the people it binds, and a process can be legitimate while its outputs are poor. The chapter asks whose population a target is drawn from, what makes the process that sets a model's behavior legitimate, and what the published exercises in public input transferred to the public, rated with a yardstick from urban planning. It also sets out the arguments that participation, as practiced or at all, is the wrong remedy. Most of its claims concern the third of the book's three relationships, the developer and the public. The voting rule inside the reward model concerns the first, the training process and its signal, and the measurements of whose opinions models reproduce concern the second, the deployed system and the people it serves. The chapter also hands the book's thesis a hard case: when the principal is a public, the delegation problem starts before any model is trained.

17.1 Why no aggregation rule is neutral #

A common hope is that some fair procedure can turn many people's rankings into one ranking without taking sides between them. Social choice theory shows that it cannot, and its value for alignment is that it says exactly what each way of proceeding gives up. This section starts from the smallest example, states the central theorem, and then takes the exits one at a time.

17.1.1 The smallest example #

Section 13.2.6 began with three voters whose rankings are $A \succ B \succ C$, $B \succ C \succ A$, and $C \succ A \succ B$. Each pair of options is decided two votes to one, so a majority prefers $A$ to $B$, $B$ to $C$, and $C$ to $A$. A Bradley-Terry reward, which gives each response one score and predicts choices from score differences (Section 13.2), fitted to these choices gives all three the same score. A useful picture of any such profile is its majority graph: one node per option, and an arrow from one option to another when a majority prefers the first. Here the arrows run in a circle, a cycle, so there is no Condorcet winner, an option that beats every other in a head-to-head majority vote.

A cycle is not needed for reasonable rules to disagree. Plurality picks the option that most people rank first. The Borda count gives each option points for every place it is ranked above the bottom and adds them up. In an example worked by Vincent Conitzer and colleagues in a 2024 position paper, posted as a preprint, 23 voters with five different rankings of three options are aggregated by three well-known rules (Conitzer et al., 2024). The Borda count ranks them $C, B, A$. Instant runoff, which repeatedly removes the option with the fewest first places, ranks them $A, B, C$. Ranked pairs, which settles a majority cycle by dropping its narrowest majority, ranks them $B, C, A$ (Conitzer et al., 2024). Each rule has a defensible rationale, and each puts a different option first.

Figure 17.1 draws the majority graph for any profile of three blocs. At its default, 40 voters rank $A \succ B \succ C$, 35 rank $B \succ C \succ A$, and 25 rank $C \succ A \succ B$. The arrows form a cycle, plurality picks $A$ on its 40 first places, and the Borda count picks $B$. Moving eleven voters from the third bloc to the first breaks the cycle and makes $A$ a Condorcet winner. Which of these answers is right is not a fact the profile contains, and that is the section's subject.

17.1.2 Arrow's theorem and its exits #

From social choice. Social choice theory is the mathematical study of how individual preferences or judgments can be combined into a collective one. Its founding result is Kenneth Arrow's, published in 1951 (Arrow, 1951). Arrow's own book was not readable to us; the statement below follows the accounts of Conitzer and colleagues and of Abhilash Mishra, which agree (Conitzer et al., 2024; Mishra, 2023).

Each condition is a demand a designer would want to make, and each exit from the theorem drops one. Unrestricted domain, the requirement that the rule work for every possible profile, is dropped by ruling some combinations of preferences out in advance. Asking for less than a full social ranking does not escape either: Conitzer and colleagues note that a similar theorem holds for rules that pick only a single winner, or a set of choice-worthy options (Conitzer et al., 2024). Few designers would give up the Pareto condition, and giving up non-dictatorship is giving up the question (inference). That leaves independence of irrelevant alternatives. The general takeaway of the literature, in Conitzer and colleagues' summary, is that ordinal aggregation that avoids dictatorships must let the social preference between $A$ and $B$ depend partly on preferences involving other options (Conitzer et al., 2024). The Borda count does exactly that, which is how it escapes the theorem: a third option's position changes the points $A$ and $B$ receive.

The other exit is to collect more than rankings. Dan Hendrycks's 2024 textbook calls Arrow's theorem one reason to want more than ordinal information, and names adding information about the strength of preferences as one solution to it (Hendrycks, 2024). Strength of preference is what a utility function records, and adding utilities across people has a theorem of its own.

17.1.3 Adding up utilities #

John Harsanyi's 1955 paper on cardinal welfare and interpersonal comparisons of utility is the classic source (Harsanyi, 1955). It was not readable to us, so the book states the result from the account in Hendrycks's textbook and has not checked that account against the original (Hendrycks, 2024).

The reading of "consistently" is the book's, and the result needs it. Ranking options by $e^{U_1} + e^{U_2}$ meets both of the other conditions and is not a sum of utilities; what it fails is expected utility over risky options, since it prefers a sure option to an even chance between that option and one it ranks equal (inference).

The theorem fixes the form of the aggregate, a sum, and leaves its weights open. Section 7.2.2 showed that a utility function representing one person's choices is unique only up to a positive scale and a shift, and noted that nothing in the axioms compares one person's numbers with another's. A sum of utilities therefore depends on a choice the axioms do not make.

The derivation is the book's, from Section 7.2.2, and its moral is that the utilitarian exit replaces Arrow's question with another: who sets the weights (inference). Equal weights are a choice of scales too, since they presuppose that one person's unit of utility is worth another's. The same problem appears under moral uncertainty, where theories take the place of people; Section 16.3 sets out the proposals for it, from normalizing each theory's scores to bargaining from a disagreement point.

17.1.4 Strategy #

The theorems so far take each person's ranking as given. A second family asks what happens when people report rankings in order to move the result. In the loose statement of Arnaud Fickinger and colleagues, in a 2020 preprint, Gibbard's theorem says that any nontrivial process of collective decision, one that is neither dictatorial nor limited to two outcomes, is subject to manipulation (Fickinger et al., 2020b; Gibbard, 1973). Conitzer and colleagues put it more plainly: every reasonable voting rule will sometimes give voters an incentive to vote against their true preferences (Conitzer et al., 2024). Section 13.5.3 works the Borda case on the book's running example, where the larger bloc buries its rival's favorite and wins.

Conitzer and colleagues add an example closer to rating (Conitzer et al., 2024). An evaluator who thinks a response deserves 3 on a scale from 0 to 10, and expects two others to give it 6, can report 0. If ratings are averaged, the average falls to 4 instead of 5, closer to what the evaluator wanted. If the median is taken, the report changes nothing, and in this setting the median is strategy-proof: misreporting never helps someone whose goal is to move the median toward their own rating. They also name a property that matters more for responses than for candidates. Two clones, options that every voter places on the same side of every other option, can split support between them, and a rule is independent of clones when adding a clone does not change whether, or which, other option wins. The Borda count, which they note is implicitly used in some standard approaches to reinforcement learning from human feedback (RLHF, the method that fits a reward model to people's comparisons and then trains against it, Section 12.2.1), badly violates independence of clones. When the options are a chatbot's responses, they add, two of them may quite often be very close to each other (Conitzer et al., 2024). A 1998 paper by Michael Dummett treats a related weakness, manipulation of the Borda count through the agenda of options (Dummett, 1998).

Personal liberty conflicts with unanimity too. Amartya Sen's 1970 paper formalizes a weak form of individual liberty, illustrated in its abstract by the claim that someone who prefers pink walls to white should be permitted them even if a majority would rather see white (Sen, 1970b). Mishra states the result as the nonexistence of any choice rule that satisfies an unrestricted domain, minimal liberalism, and the Pareto condition. He reads it as a limit on respecting each user's private preferences in a model trained on many people's feedback (Mishra, 2023).

17.1.5 Judgments rather than rankings #

Model specifications are written as principles, and a principle is a judgment that a proposition holds, not a ranking of options. Christian List and Philip Pettit showed in 2002 that aggregating judgments has an impossibility of its own, starting from a case that legal scholars call the doctrinal paradox (List and Pettit, 2002).

List and Pettit show that the trouble is not peculiar to majority rule. Consider any aggregation function that returns complete, consistent, and deductively closed collective judgments. It cannot satisfy three conditions together: it accepts any profile of consistent individual judgments (universal domain), it treats individuals alike (anonymity), and it treats propositions alike, so that the collective judgment on each depends only on individual judgments on it (systematicity) (List and Pettit, 2002). They call systematicity somewhat more controversial than the other two, and undeniably demanding (List and Pettit, 2002). A constitution for a model is a set of logically connected judgments. This impossibility, more than Arrow's, is therefore the one a public process for writing principles meets: it can follow the majority on each principle and still produce a set that no majority holds (inference).

17.1.6 What the theorems assume #

The impossibility results bite where disagreement is real. When choices are noisy reports of one underlying ranking, the problem changes character. Arun Rajkumar and Shivani Agarwal's abstract reports that a matrix version of the Borda count, and maximum likelihood under the Bradley-Terry model, both converge to an optimal ranking under a low-noise condition more general than the Bradley-Terry model itself (Rajkumar and Agarwal, 2014). Section 13.5.2 reaches the same contrast from the other side: noise that is independent across responses averages out, and disagreement organized into groups does not.

The theorems also assume a fixed set of options and fixed preferences. For a language model neither holds. Conitzer and colleagues observe that it is harder to see how to determine the set of alternatives for alignment, and that one could treat every possible response to a prompt as an alternative (Conitzer et al., 2024). And a deployed system can change the preferences it is aggregating (Section 13.6.3; Section 16.9). Options a model generates and preferences it moves lie outside the theorems, so the impossibility results are a floor on the difficulty, not a full account of it (inference).

17.2 The voting rule inside the reward model #

Preference training combines many raters' choices into one reward, so it is an aggregation procedure whether or not its designers chose one. The question is which rule it applies, what that rule gives up, and who chose it. Section 13.5 answers the first part: under the conditions of Siththaranjan, Laidlaw, and Hadfield-Menell's theorem (unlimited data, every pair compared equally often, and a reward free to take any value on each response), standard preference learning ranks responses by their Borda count, and raters can gain by misreporting (Siththaranjan et al., 2024). This section asks what the alternatives are and what each costs.

17.2.1 Axioms for reward learning #

The axiomatic method of social choice judges a rule by which conditions it meets. Luise Ge and colleagues, in a paper at NeurIPS 2024, apply it to reward learning in which the reward is a linear function of fixed features of each response. This describes a reward model whose learned representation is held fixed and whose last layer combines it linearly (Ge et al., 2024b). Not every ranking of the responses can then be produced, a setting they call linear social choice, though they assume that every voter's own ranking can be (Ge et al., 2024b). They ask for two properties. Pareto optimality: if every voter ranks $a$ above $b$, so does the output. Pairwise majority consistency: if some feasible ranking agrees with the majority on every pair, the output is that ranking, an extension of electing a Condorcet winner whenever one exists (Ge et al., 2024b).

Their first main result is negative. Any rule that minimizes a loss that is nondecreasing and convex, or strictly convex, and that can fall below its value at a tie, fails both properties, and the cross-entropy loss of the Bradley-Terry model is such a loss (Ge et al., 2024b). Rewriting the loss to depend only on majority preferences restores pairwise majority consistency but cannot restore the Pareto property, since they prove that every linear rule depending only on majority preferences fails it (Ge et al., 2024b). Their positive result builds on the Copeland rule, which scores each option by the number of others it beats head to head. A variant they call leximax Copeland subject to Pareto optimality satisfies both properties, along with majority consistency and winner monotonicity (Ge et al., 2024b). The authors emphasize that the results are theoretical. They also contrast an earlier analysis in which reward functions were unrestricted rather than linear, and in which the Bradley-Terry estimate did satisfy the Pareto property (Ge et al., 2024b). Which axioms preference training breaks depends on the form of the reward model, not only on the loss.

Two position papers draw the policy consequences. Conitzer and colleagues describe preference training on data from diverse sources as a rather intransparent or flawed form of preference aggregation. They propose making the aggregation an explicit step: a social welfare function applied to a jury's rankings, or to ratings simulated for a representative sample of evaluators (Conitzer et al., 2024). Mishra, in a 2023 preprint, argues from Arrow's theorem that no unique voting protocol aligns a system with everyone through a democratic process. He recommends that developers report their voting rules, noting that developers such as OpenAI did not do so in their system cards. He also recommends aligning models narrowly to specific user groups (Mishra, 2023).

17.2.2 Alternatives to one reward #

Several rules have now been built into training. Table 17.1 sets them side by side with the utilitarian sum of Section 17.1.3.

The max-min rule is the egalitarian entry. Souradip Chakraborty and colleagues, in a 2024 preprint, prove a lower bound on how far a single reward can fall short for a subpopulation whose preferences differ from the rest, a gap that grows as the group shrinks (Chakraborty et al., 2024). Their remedy learns a mixture of reward models, one per discovered group, and trains the policy, the model's rule for choosing responses, to maximize the worst group's reward, an objective they take from the egalitarian rule of social choice (Chakraborty et al., 2024). In their larger experiment, GPT-4 played the annotators, following written preference prompts for each group, and the base model was Tulu2-7B. A single reward model's accuracy on the minority group's comparisons fell from 0.704 when the groups were equal in size to 0.42 when the majority outnumbered it ten to one (Chakraborty et al., 2024). The rule's classic weakness is named in Hendrycks's textbook: a maximin social welfare function can end up always catering to a "grouch" who reports low well-being whatever happens (Hendrycks, 2024).

The minimax winner gives up determinism. In a 2024 preprint, Gokul Swamy and colleagues define it, after the social choice literature, as a distribution over options such that a sample from it is preferred to a sample from any other distribution with probability at least one half (Swamy et al., 2024). It follows from the definition that when a Condorcet winner exists, the lottery puts all its weight on it, and that in a cycle it mixes. Their method trains a policy against copies of itself, rewarding each sampled response by the share of other samples it beats, and they prove it converges to an approximate minimax winner when preferences are intransitive (Swamy et al., 2024). They also give an example with three options and a uniform reference policy in which neither a reward-based method nor direct preference optimization, which trains on comparisons without a separate reward model (Section 12.6), reaches the minimax winner. A reward can only make one option best, two options tied, or all three equal (Swamy et al., 2024). Figure 17.1 computes both the pooled Bradley-Terry reward and the minimax lottery for any profile; at the default cycle the lottery gives $A$, $B$, and $C$ weights 0.5, 0.2, and 0.3, while the reward ranks $B$ first. The distributional reward models of Section 13.5.4 take a third route, keeping the disagreement in the reward rather than resolving it.

17.2.3 Many principals, one assistant #

Raters can be strategic, and mechanism design is the field that builds rules to make truthful reporting each participant's best choice. Section 14.4.3 recalls the 2020 preprint by Fickinger and colleagues in which a robot assists several humans with different payoffs and social choice impossibilities return (Fickinger et al., 2020a). A companion preprint gives a positive result (Fickinger et al., 2020b). If the game is sufficiently collegial, meaning the humans obtain a sufficient fraction of their reward through their own work before the robot acts, their preferences are revealed straightforwardly by that work. The mechanism is non-dictatorial, not limited to two outcomes, and dominant-strategy incentive-compatible: truthful demonstration is each human's best choice whatever the others do (Fickinger et al., 2020b). That is how it sidesteps Gibbard's theorem. The condition is the point. A rater comparing two responses for pay does not live with either response, so the collegial condition fails for the people who supply most preference data (inference). Section 18.5 reports how those raters respond to the incentives they do face.

17.2.4 Serving many people at once #

Instead of choosing one rule, a system can be built to serve several perspectives. Taylor Sorensen and colleagues, in a 2024 preprint, name three ways a model can be what they call pluralistic (Sorensen et al., 2024). An Overton pluralistic model presents the spectrum of reasonable responses; a steerably pluralistic model can be steered to reflect a given perspective; a distributionally pluralistic model matches the distribution of answers in a given population. Comparing pretrained models with their aligned counterparts from three model families, they report that almost all pretrained models were closer to the target human distributions than the aligned ones, on survey questions about national opinions and on a personality inventory (Sorensen et al., 2024). The result bears only on the third form. Shibani Santurkar and colleagues, whose survey measures Section 17.4 reports, warn that exactly matching a population's opinions may not be desirable either (Santurkar et al., 2023).

Who supplies the comparisons changes what any rule returns. Hannah Rose Kirk and colleagues built PRISM, a dataset published at NeurIPS 2024 that maps the sociodemographics and stated preferences of 1,500 participants, born in 75 countries, to their feedback in 8,011 live conversations with 21 language models (Kirk et al., 2024b). The authors note that the sample still skews White, Western, and educated, and contains only English speakers (Kirk et al., 2024b). In one case study, they let randomly drawn subsamples choose a favorite model and measured how well the choice served others. Large representative samples mostly did better than small or demographically restricted ones, sampling only from one group tended to reduce the welfare of people outside it, and no single model was preferred by a majority (Kirk et al., 2024b). A companion paper by Kirk and colleagues considers how far alignment can be personalized to individuals instead (Kirk et al., 2024a).

Two methods let the participants generate the options themselves, which classical social choice cannot. Generative social choice, proposed by Sara Fish and colleagues in a 2023 preprint, proves representation guarantees for a process given access to idealized queries, and then shows empirically that a language model can approximately implement the queries (Fish et al., 2023). In their pilot, free text opinions on abortion policy from 100 US residents were distilled into five statements, and in a fresh sample of 100, 84 participants said their assigned statement captured their view "excellently" or "exceptionally" (Fish et al., 2023). Moral graph elicitation, proposed by Oliver Klingefjord, Ryan Lowe, and Joe Edelman of the Meaning Alignment Institute in a 2024 preprint, interviews participants about their values with a chatbot and asks which of two values is wiser in a given context (Klingefjord et al., 2024). Lowe did most of the work while a researcher at OpenAI, and the work was supported by OpenAI's Democratic Inputs to AI program (Klingefjord et al., 2024). With a representative sample of 500 Americans, 89 percent agreed that their own value ended up in a fair position in the resulting graph (Klingefjord et al., 2024). Both are measures of whether participants felt represented by an output, taken by the method's designers, and neither paper reports training a model.

17.2.5 What the choice of loss decides #

Majority rule, the Borda count, Copeland, max-min, the minimax lottery, and a weighted sum are different answers to the same political question: whose preferences prevail when people disagree. A developer who chooses a loss function and a pool of raters chooses one of them, usually without saying so (inference). The claim concerns the first relationship, the training process and its signal, and through it the third, since the developer makes the choice and the public meets its results. Moral uncertainty runs on the same mathematics, with theories in place of people (Section 16.3). For the book's thesis, social choice is part of the older literature of the specification problem, and the technical proposals above are converging on its structures: explicit aggregation steps, axioms, and lotteries (inference). What the older theory lacks is options that the system itself generates, raters paid to judge on others' behalf, and preferences the deployed system can move (Section 17.1.6).

17.3 Fairness: aggregation without a neutral answer #

From AI ethics. The fairness literature met the same kind of impossibility earlier, in classifiers that score people, and its results teach the general lesson with less abstraction. When groups differ, natural criteria of fairness conflict, so someone has to choose among them.

The setting is a risk score, a number that sorts people into higher and lower risk of some outcome. Alexandra Chouldechova's 2017 analysis studied COMPAS, a recidivism prediction instrument: a score that estimates whether a defendant will reoffend (Chouldechova, 2017). Call the share of a group that has the outcome its prevalence, the base rate $\pi_{\text{base}}$ of Section 6.7. The score flags some people as high risk, and the quantities of Section 6.7 describe its errors. They are the false positive rate $\alpha$; the miss rate $\beta_{\text{miss}}$, which the fairness literature calls the false negative rate; and the precision, which it calls the positive predictive value (PPV), the share of people flagged who do have the outcome. A score is test-fair in Chouldechova's sense when a given score means the same likelihood of the outcome whatever the person's group (Chouldechova, 2017). She states that this implies equal PPV across groups (Chouldechova, 2017), but it does so only when the people flagged in each group have the same mix of scores: if two scores mean likelihoods of 0.6 and 0.8, a group flagged mostly at the first has lower precision than one flagged mostly at the second. The correction is the book's, and the derivation below assumes equal PPV directly.

The equation explains a public dispute. ProPublica had reported that COMPAS made very different errors for Black and White defendants, and a response to that investigation verified that the score was approximately test-fair (Chouldechova, 2017). Chouldechova notes that in the data the recidivism rate was 51 percent for Black defendants and 39 percent for White defendants, and concludes that some imbalance in error rates had to exist (Chouldechova, 2017). Table 17.2 gives the rates she reports.

Both sides could be right about their own criterion, and neither criterion could be met without giving up the other. Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan proved a version for probability scores in a 2016 preprint, which Chouldechova cites as independent concurrent work (Kleinberg et al., 2016; Chouldechova, 2017). Their first condition is calibration within groups: among people given a score, that share has the outcome, in every group. The other two are balance for the positive class and for the negative class: people who do, or do not, have the outcome receive the same average score in every group. Except in highly constrained special cases, perfect prediction or equal base rates, no method satisfies all three, and satisfying them approximately requires data close to one of those cases (Kleinberg et al., 2016).

A fairness criterion is therefore an aggregation rule across groups, and choosing one is a value choice of the kind Section 17.1 describes (inference). The choices also enter before any criterion is applied. Batya Friedman and Helen Nissenbaum, from an analysis of actual cases, sorted bias in computer systems into three kinds: preexisting bias rooted in social institutions and attitudes, technical bias from technical constraints, and emergent bias that arises in a context of use (Friedman and Nissenbaum, 1996). Section 15.2.4 reports a case that fits the first kind, entering through a label: a health algorithm that predicted cost as its stand-in for need (Obermeyer et al., 2019) (inference).

Whose fairness is being measured is a further choice. Nithya Sambasivan and colleagues, from 36 interviews and an analysis of algorithmic deployments in India, report in their abstract that several assumptions of algorithmic fairness, which they call West-centric, are challenged there, and that localizing model fairness alone can be window dressing in India (Sambasivan et al., 2021b). Cynthia Bennett and Os Keyes argue, by their abstract, that analyzing AI's effects on disabled people solely through a single idea of fairness risks reinforcing existing power, and call for moving toward justice (Bennett and Keyes, 2020). Roel Dobbe and colleagues argue, also in their abstract, that the vagueness in debates about safe and ethical AI cannot be resolved through mathematical formalism alone, and that deliberation should be the goal of AI safety, not only its procedure (Dobbe et al., 2021). Safiya Noble's Algorithms of Oppression belongs to the same tradition (Noble, 2018).

The same tradition contests who defines alignment's target at all. Seth Lazar and Alondra Nelson argued in 2023 that the prevailing technical agenda for AI safety is inadequate to address critical questions, and that only a sociotechnical approach can limit the dangers of advanced AI (Lazar and Nelson, 2023). Timnit Gebru and Émile Torres argue, in their abstract, that the normative framework behind the goal of building artificial general intelligence is rooted in the Anglo-American eugenics tradition of the twentieth century (Gebru and Torres, 2024). Section 2.5.2 reports both and Section 4.4 weighs them; here they bear on who gets to choose the aggregation rule.

The analogy has a limit. Fairness criteria are defined for classifiers with ground-truth labels: who reoffended, who was sick. A model's behavior in open conversation has no such label to condition on, so the impossibility carries over in form, since any comparison across groups needs a chosen criterion, while the measurements do not (inference). The claims in this section concern the second relationship, the deployed system and the people it scores or serves, and the third, since the developer chooses the criterion.

17.4 Whose population #

Every aggregation rule needs an electorate, and choosing whose judgments are sampled is already a choice of target. This section asks how much the candidate populations differ, whose opinions models reproduce, and how the training procedure itself distributes representation.

17.4.1 How much populations differ #

From anthropology. Anthropology and cross-cultural psychology measure how far human populations differ in the judgments that a preference dataset records. In a study reported in 2001, Joseph Henrich and colleagues had twelve experienced field researchers run economic games in 15 small-scale societies on five continents, from foragers to nomadic herders (Henrich et al., 2001). The central game was the ultimatum game: a proposer is given a sum, a day or two's wages in the local economy, and offers a share to a responder, who either accepts it or rejects it, in which case both get nothing (Henrich et al., 2001). A purely self-interested responder accepts any positive offer, and a proposer who knows this offers the minimum.

That canonical model of self-interest was not supported in any society studied (Henrich et al., 2001). Even the groups with the smallest offers made mean offers above 25 percent of the stake, others offered between 30 and 40 percent, and the variation across groups was larger than earlier cross-cultural research had found (Henrich et al., 2001). The authors report that the degree of market integration and the payoffs to cooperation in everyday life explained a substantial share of the variation between societies, while individual economic and demographic variables did not explain behavior within or across groups (Henrich et al., 2001). Rejections varied as much: Machiguenga responders rejected only one offer although over 75 percent of offers were below 30 percent, while the Au and Gnau of Papua New Guinea rejected unfair and hyper-fair offers, those above 50 percent, with nearly equal frequency (Henrich et al., 2001).

Behavioral science had mostly sampled one kind of population. Reviewing the comparative evidence across visual perception, fairness, cooperation, moral reasoning, and other domains, Henrich, Steven Heine, and Ara Norenzayan concluded in 2010 that people from Western, educated, industrialized, rich, and democratic (WEIRD) societies are frequent outliers, "among the least representative populations one could find for generalizing about humans" (Henrich et al., 2010). For a reward model, the implication is direct. A preference dataset drawn from one population carries that population's norms of fairness and cooperation into the target, and a model trained on it has been aligned to that population, not to people in general (inference).

17.4.2 Whose opinions models reproduce #

Several groups have measured which populations a model's answers resemble, by asking models the questions of published opinion surveys and comparing their answer distributions with those of human respondents. In a 2023 preprint, researchers at Anthropic built GlobalOpinionQA from cross-national surveys and tested their own model, trained with the developer's Constitutional AI method, which trains a model against a written list of principles (Chapter 23) (Durmus et al., 2023). By default its answers were more similar to the opinions of certain populations, such as those of the USA and some European and South American countries. Prompting it to take a particular country's perspective shifted its answers toward that country's, but could reflect harmful cultural stereotypes. Translating the questions into a language did not necessarily make the answers most similar to those of the language's speakers (Durmus et al., 2023). This is a developer's report on its own model.

An academic group reached a similar result for the United States. Santurkar and colleagues, in a 2023 preprint from Stanford, built OpinionQA from public opinion polls and compared language models with 60 US demographic groups (Santurkar et al., 2023). They found misalignment between the models' opinions and those of the groups on par with the divide between Democrats and Republicans on climate change. Fine-tuning, further training, on human feedback seemed to amplify it, and steering the models toward a group improved the match only modestly (Santurkar et al., 2023). People aged 65 and over and widowed people were among those whose opinions the models reflected poorly (Santurkar et al., 2023). A 2023 preprint by Mohammad Atari and colleagues, Henrich among them, reports in its abstract that models' performance on cognitive psychological tasks most resembles that of WEIRD populations. The resemblance declines as populations differ from them, with a correlation of $-0.70$ (Atari et al., 2023b).

17.4.3 The procedure distributes representation #

These results concern finished models. Michael Ryan, William Held, and Diyi Yang, in a paper at ACL 2024, measured what each stage of alignment, supervised fine-tuning and then preference tuning, does to chat models built on Llama 2 7B and Mistral v0.1 7B (Ryan et al., 2024). On an English intent-prediction task with conversations from US, Indian, and Nigerian speakers, alignment improved performance but raised the disparity between dialects from about 1 percent before alignment to as much as 17.1 percent after, with supervised fine-tuning creating the disparity and preference tuning changing far less (Ryan et al., 2024). It also improved performance in several other languages, an effect the authors call positive and unintended. Every alignment procedure they evaluated raised the similarity of model answers to the opinions of the United States relative to those of other major nations, such as China, Jordan, and Nigeria (Ryan et al., 2024). One finding cuts the other way. An open reward model rated 99.4 percent of other countries more negatively than the USA, yet the bias did not seem to propagate to the language model tuned with it (Ryan et al., 2024). So the path from a skewed signal to skewed behavior is not automatic, and which skews survive training is an empirical question (inference).

The skew starts before preference tuning. Pratik Joshi and colleagues found that only a very small number of the world's more than 7,000 languages are represented in language technologies, with steep disparities in resources (Joshi et al., 2020). Suchin Gururangan and colleagues showed that a reconstruction of the quality filter described for GPT-3's training text preferred US high school newspapers from larger schools in wealthier, more educated, and urban areas (Gururangan et al., 2022). Whether safety training learned in one language transfers to another is a generalization result, and Chapter 26 takes it up.

The instruments need the same scrutiny as the disparities they report. Section 7.3.3 reports that survey-based measures of models' cultural values shift with changes of presentation format (Khan et al., 2025). A measured disparity in survey answers may say more about the survey than about the model's behavior in open conversation, and a measured parity would be just as fragile (Section 18.7). The several studies agree in direction across different instruments and groups; whether the instruments measure a stable property of the model is not settled.

17.4.4 Collective units #

Some populations are not samples of individuals but peoples with their own authority. Stephanie Russo Carroll and colleagues published the CARE Principles for Indigenous Data Governance in 2020, after consultation with Indigenous peoples, scholars, organizations, and governments. The principles are Collective Benefit, Authority to Control, Responsibility, and Ethics, and they describe themselves as people- and purpose-oriented complements to data-centric principles (Carroll et al., 2020). The United Nations General Assembly adopted the Declaration on the Rights of Indigenous Peoples on 13 September 2007; the four states that voted against it later reversed their positions (United Nations General Assembly, 2007). Indigenous Data Sovereignty, edited by Tahu Kukutai and John Taylor, set out the agenda in 2016 (Kukutai and Taylor, 2016), and Section 16.7.2 reports the guidelines of the Indigenous Protocol and Artificial Intelligence position paper, among them community control over data (Lewis et al., 2020). A participatory counterpart in practice is the Masakhane project: Wilhelmina Nekoto and colleagues report machine translation benchmarks for over 30 African languages, built with participants who had no formal training in the field (Nekoto et al., 2020).

The analogy stops at authority. The populations anthropology studies are communities with their own structures for deciding, and a people that asserts collective rights over its data asserts an authority. A preference dataset's population is a sampling frame chosen by the developer, and it has no authority over the target at all (inference). Representation in the statistical sense, a sample that resembles a population, is not representation in the political sense, a body that can act for one. The claims of this section concern the second relationship, the deployed system and the people it serves; the gap between the two senses of representation is the subject of the next two sections.

17.5 What makes a process legitimate #

A rule can satisfy every axiom and still lack authority over the people it binds. Legitimacy is the property by which a decision or an authority is owed acceptance by those it binds, including those who disagree with it. This section asks what confers it, applies the answers to the parties who now set model behavior, states among them, and says where theories built for states stop applying to developers.

17.5.1 Input and output #

From political science. Political theory separates two sources of legitimacy. Klingefjord and colleagues, citing Fritz Scharpf among others, give the distinction in terms an engineer can use. A process has input legitimacy if participants believe that input was collected from an appropriate group and counted appropriately. It has output legitimacy if its outcomes compare well with those of other known processes, seem good compromises among the groups concerned, and can be verified (Klingefjord et al., 2024). Scharpf's 1999 book asks in its title whether governance in Europe can be both effective and democratic (Scharpf, 1999). Klingefjord and colleagues add that an alignment target will be held either to the standards of business, such as profitability and avoidance of liability, or to the standards of a public process, and that in the second case it needs to be legitimate (Klingefjord et al., 2024).

Theories differ on how much input a legitimate authority needs. Philip Pettit's republican account, by its publisher's abstract, holds that state coercion does not subject people to domination insofar as they enjoy an equally shared form of control over those in power (Pettit, 2012). Allen Buchanan and Robert Keohane, writing about institutions that govern across borders, stake out a middle ground. Their abstract rejects both equating legitimacy with state consent and what it calls the unrealistic view that such institutions require the democratic standards now applied to states (Buchanan and Keohane, 2006). Eva Erman and Markus Furendal argue, in their abstract, that political legitimacy requires at least a democratic minimum, and conclude that much existing global AI governance lacks properties necessary for it (Erman and Furendal, 2024).

Applied to a developer's model specification, the two sources come apart. No one bound by the specification chose its authors, so its input legitimacy is thin. Its output legitimacy is claimed through the developer's own evaluations of the resulting model. A specification can therefore have output legitimacy without input legitimacy, and the evidence for the output is produced by the party that benefits from it, which is the condition under which the book's thesis predicts selective reporting (Section 5.6; Chapter 61) (inference).

Why the developer writes and the public does not is a question of organization. A developer is one organized party with a concentrated stake in its specification. The people bound by it are many, each with a small stake and a high cost of acting together. That structure is the developer-public relationship of Section 1.3 in miniature (inference). Mancur Olson's 1965 book names the problem in its title, The Logic of Collective Action (Olson, 1965). The book was not readable to us, and the point here rests on the structure of the case, not on Olson's argument.

17.5.2 A procedure tested #

One procedure for choosing principles has been tested on AI. John Rawls proposed the veil of ignorance as a thought experiment for identifying fair principles to govern a society: choose principles without knowing your own position in it (Weidinger et al., 2023b; Rawls, 1971). Laura Weidinger and colleagues, in five incentive-compatible studies with 2,508 participants, two of them preregistered, had people choose a principle to govern an AI assistant from behind the veil, without knowing their own position in the group (Weidinger et al., 2023b). By the paper's abstract, compared with participants who knew their position, they consistently preferred a principle that told the assistant to prioritize the worst-off. Neither risk attitudes nor political preferences adequately explained the choice; participants behind the veil more often explained it in terms of fairness. The authors also report initial support for the veil's making preferences more robust: people were more likely to keep endorsing their choice when, in a later round, they knew how they would be affected (Weidinger et al., 2023b).

The result bears on an old dispute. Hendrycks's textbook reports that Harsanyi conceived of the veil of ignorance before Rawls and used it to support utilitarianism, arguing that rational agents behind it would maximize total well-being (Hendrycks, 2024). People in the experiment leaned toward the worst-off instead, which fits Rawls's reading of the procedure better than Harsanyi's (inference). Three limits keep the result from settling who should set a model's principles. The choices were made in games, which the abstract describes as a descriptive and an immersive game (Weidinger et al., 2023b). The menu of principles was the experimenters'. And the procedure measures what people choose under a condition, which tells a designer what a fair chooser might pick but transfers no authority to anyone (inference). Gabriel and Keeling's case for aligning to the output of a fair process (Section 16.2) needs procedures of this kind, and the authority question remains (Gabriel and Keeling, 2025).

17.5.3 States as principals #

States are principals too, and some now write part of a model's specification by law. China's Interim Measures for the Management of Generative Artificial Intelligence Services, in force from 15 August 2023, apply to generative services offered to the public in China (Cyberspace Administration of China et al., 2023). Article 4 requires such services to adhere to core socialist values and not to generate content that incites subversion of state power or the overthrow of the socialist system, among other prohibited content. It also requires effective measures, in algorithm design, training data selection, and model generation and optimization, against discrimination by ethnicity, belief, country, region, gender, age, occupation, or health (Cyberspace Administration of China et al., 2023). Article 8 reaches into the aggregation step itself: providers who label data must write "清晰、具体、可操作的标注规则," clear, specific, and operable labeling rules, check the quality of the labels, and train the people who apply them (Cyberspace Administration of China et al., 2023). Under Article 19 providers must explain their training data sources, labeling rules, and algorithms to the authorities on request (Cyberspace Administration of China et al., 2023). The state thus writes values into the specification and governs the instructions that raters follow (inference).

The European Union's AI Act takes a different route. Its recitals describe a risk-based approach that prohibits certain unacceptable practices, sets requirements for high-risk systems, and lays down transparency obligations for some others (European Parliament and Council of the European Union, 2024b). For providers of general-purpose models with systemic risk, Article 55 requires model evaluation including adversarial testing, assessment and mitigation of systemic risks at Union level, tracking and reporting of serious incidents, and adequate cybersecurity (European Parliament and Council of the European Union, 2024b). The Chinese measures write content into the specification; the European regulation mostly writes obligations of process and a list of prohibitions (inference). They are two answers to "aligned to whom" set by law. Chapter 63 reports the instruments in full, and Chapter 66 asks whether states that specify their models this differently can coordinate.

17.5.4 Developers as lawmakers #

For most users the binding specification is still a developer's. Lawrence Lessig's 1999 book frames software as law in its title, Code and Other Laws of Cyberspace (Lessig, 1999). A model specification fits the frame closely: it governs, for everyone who uses the model, what the model will and will not do, and it is written and amended by the developer alone (inference). Gilad Abiri's 2024 preprint, Public Constitutional AI, takes up the proposal its title names (Abiri, 2024); it was not readable to us. Section 17.7 rates what public processes have achieved so far.

17.5.5 Where the theories stop #

Theories of legitimacy were built for states, which hold coercive power over citizens who cannot easily leave. A developer's specification binds users through terms they can exit by switching providers, and binds third parties, the people affected by what users do with a model, through no relationship at all. Which standard applies is itself open: Buchanan and Keohane's middle ground for institutions that are not states is the nearest available model, and it was not built for firms (inference).

The section's results bear on the book's thesis directly. In the third relationship the public is the principal and the developer its agent, and the agent writes the specification that defines its own task. The delegation problem therefore starts before any model is trained. Apart from its government acting by law, the principal has no institution of its own for stating what it wants. It can test the deployed model from outside, but it learns how the specification was chosen and how training applied it only through the agent's own reports (inference).

17.6 Designing with values, and when #

From science and technology studies. Science and technology studies (STS) examines how technical systems and social arrangements are made together. It adds two claims that the legitimacy debate tends to miss: design settles political questions before any procedure is convened, and the time when a technology is easiest to change is the time when its effects are least known.

The first claim is Langdon Winner's, and Section 2.5 quotes his account of how the design of a device can become a way of settling an issue in a community (Winner, 2017). Applied here, a specification's defaults and refusals settle, for every user, questions that no procedure was convened to decide; a public process that writes principles reaches only the part of the settlement that is written down (inference).

The second claim is David Collingridge's dilemma of control, named for his 1980 book (Collingridge, 1980). In the statement of Laura Weidinger and colleagues at Google DeepMind, in a 2023 preprint on evaluating generative systems, early-stage evaluation has more potential to influence the direction of a technology, while later-stage evaluation provides more accurate and comprehensive information (Weidinger et al., 2023a). A model specification sits on the early side of the dilemma: it fixes behavior before the behavior's effects on the people it serves have been measured. Section 64.1 takes up how far earlier technologies can teach anything about the dilemma; Collingridge's own remedies are not read for this book.

STS's answer to the dilemma is not a better forecast but a different kind of process. Writing about gene editing in 2015, Sheila Jasanoff, Krishanu Saha, and J. Benjamin Hurlbut described the public engagement exercises of recent decades, from focus groups and citizen juries to consensus panels, as "one-shot consultations whose agenda and terms of debate are still narrowly defined" (Jasanoff et al., 2015). Such scripted deliberations, they argue, engage a limited range of citizens in terms defined in advance, fail to reach the poor, the marginal, and the socially excluded, and leave little room for dissent (Jasanoff et al., 2015). Their alternative is a recursive model, from a 1996 report of the National Research Council, that revisits early framing choices in light of later experience (Jasanoff et al., 2015). Jasanoff's 2003 paper on the same question is titled Technologies of Humility: Citizen Participation in Governing Science (Jasanoff, 2003). Jack Stilgoe, Richard Owen, and Phil Macnaghten define responsible innovation, by their abstract, through four integrated dimensions: anticipation, reflexivity, inclusion, and responsiveness (Stilgoe et al., 2013).

The analogy has a limit, which Weidinger and colleagues name themselves: the dilemma may be somewhat modulated for software that can, at some cost, be updated after deployment (Weidinger et al., 2023a). In STS's classic cases, Winner argues, design choices become strongly fixed in equipment, investment, and habit once made (Winner, 2017). A model's behavior can be respecified with each release, so the window the dilemma describes reopens. The cost of change still rises as users come to depend on the old behavior, and a specification is rarely revisited by anyone but its author (inference). For the participation exercises of the next section, the field's diagnosis is specific. A public process run once, before deployment, on questions the developer framed, is the one-shot consultation Jasanoff and colleagues criticize. A process that recurred with evidence from deployment would be closer to the recursive model (inference). The claims concern the third relationship, the developer and the public.

17.7 Participation, rated #

Developers and researchers have now run several exercises in which members of the public contributed to what a model should do. Reported in their sponsors' words, they can read as democratic authorization of a model's behavior. This section rates each by what it transferred to the public, reports the critiques of participation as practiced, and sets out the case against participation as the remedy for the legitimacy gap of Section 17.5.

17.7.1 The yardstick #

The standard yardstick comes from urban planning. Sherry Arnstein's 1969 ladder of citizen participation (Arnstein, 1969) was not readable to us; Eric Corbett, Remi Denton, and Sheena Erete use it as an analytical lens on participatory AI. In their abstract its rungs run from informing and consulting participants up to partnering with them and delegating control to them, and most of the AI work they analyzed informs or consults rather than partners or delegates (Corbett et al., 2023). Archon Fung's finer design space locates any participatory mechanism by who participates, how participants communicate and decide, and how their discussions are linked to policy or action (Fung, 2006). The last dimension is the one that matters for a rating: what happens to the output.

17.7.2 The exercises #

Table 17.3 lists the published exercises with the facts that decide a rating. The rung in the last column is the book's own rating, made on the authority each exercise transferred, not on the quality of what participants said.

Collective Constitutional AI is the most complete of them, and its own paper documents where authority sat. Saffron Huang and colleagues, in a study by the Collective Intelligence Project with Anthropic published at FAccT 2024, recruited a sample of 1,002 US adults through a survey company, screening out people unfamiliar with generative AI (Huang et al., 2024a). Using Polis, an online platform where participants vote on each other's statements, they collected 1,127 statements and 38,252 votes (Huang et al., 2024a). The researchers then chose statements by group-aware consensus, the product across opinion groups of the probability that a member agrees, a rule that favors statements every group accepts (Huang et al., 2024a). That is itself an aggregation rule of the kind Section 17.1 describes, and the researchers chose it (inference). They took statements in order of consensus until the public constitution held as many distinct ideas as the developer's standard constitution, 95, partly to reduce the risk that training would fail (Huang et al., 2024a). The authors list the aggregation method, the merging of similar statements, and their rewording into the training format as researcher degrees of freedom and threats to validity (Huang et al., 2024a).

The constitution the public produced overlapped the developer's by roughly half in concepts, and put more weight on objectivity, impartiality, and accessibility (Huang et al., 2024a). The model trained on it showed lower bias across nine social dimensions of a bias benchmark (BBQ) than the model trained on the developer's constitution, and equivalent performance on language and mathematics benchmarks (MMLU and GSM8K) (Huang et al., 2024a). On OpinionQA (Section 17.4.2) its answers were 2 to 3 percentage points less representative than the other model's for every US political group, which the authors believe is because it refused more often (Huang et al., 2024a). The abstract calls the two equivalent on helpfulness and harmlessness as well; the body reports that human raters preferred the model trained on the developer's constitution on harmlessness, by an Elo score of 22 against 0, which the authors call "only just statistically significant" (Huang et al., 2024a); Chapter 23 takes up the training method. The authors' ethics statement concedes the point a rating turns on. They write that they do not take lightly the responsibility of potentially invoking democratic legitimacy, and that they avoid claiming their participants represent the will of the US public as a whole (Huang et al., 2024a). The STELA authors concede the same of their own process: it was not fully participatory, since participants did not interact with the system, propose rules, or control the study's artifacts (Bergman et al., 2024); Section 13.5.5 reports the study.

Two results in the table were replicated only by their own authors or are not participation at all. Michael Henry Tessler and colleagues report, by their abstract, that participants preferred statements written by an AI mediator to those of human mediators, and that the findings replicated in a virtual citizens' assembly (Tessler et al., 2024). Mantas Mazeika and colleagues, in a paper at NeurIPS 2025, fine-tuned Llama-3.1-8B-Instruct toward the preferences of a citizen assembly that was simulated by a language model from sampled census profiles (Mazeika et al., 2025). The model's accuracy on the simulated assembly's preferences rose from 73.2 to 90.6 percent, and the authors report that political bias was visibly reduced (Mazeika et al., 2025). No citizen took part.

As far as we found, as of October 2026, no published exercise bound a deployed frontier model, and in each one the sponsor or the researchers kept final authority over what was produced and what was trained. That places the exercises on the consultation rungs. The rating concerns authority, not the value of what participants said, which in several cases changed a model in measurable ways (inference).

17.7.3 The critiques #

Critics of participation as practiced say the rating is typical. Corbett and colleagues found that papers on participatory AI are frequently written so that the degree of power afforded is hard to ascertain (Corbett et al., 2023). Abeba Birhane and colleagues warn, in their abstract, that participatory AI risks cooptation and conflation with other activities, and ask who the primary beneficiaries of participation ought to be (Birhane et al., 2022a). Fernando Delgado and colleagues report, from published research and twelve interviews with practitioners, that it remains hard to assess whether any participatory approach grants stakeholders substantive agency (Delgado et al., 2023). Mona Sloane and colleagues caution against "participation washing," and argue that participation should be recognized as work, that consultation should be specific to its context, and that participation as justice must be genuine and long term (Sloane et al., 2022). An earlier critique from development studies carries its thesis in its title: Bill Cooke and Uma Kothari's edited volume Participation: The New Tyranny? (Cooke and Kothari, 2001).

Organizational sociology supplies a mechanism for the worry. John Meyer and Brian Rowan argued in 1977 that organizations adopt structures that their environment treats as rational, and gain legitimacy, resources, and stability by doing so, independent of the immediate efficacy of the practices adopted; the structures are then decoupled from ongoing activities (Meyer and Rowan, 1977). Chapter 53 takes up the theory. A public-input exercise could confer legitimacy in exactly this way while remaining decoupled from what is trained (inference). What would show the difference is whether the exercise's output changes the deployed model, and whether anyone outside the developer could have made it do so.

17.7.4 The case against participation as the remedy #

A stronger objection holds that more or better participation is the wrong remedy altogether. A 2023 paper by Johannes Himmelreich argues against democratizing AI, as its title, "Against 'Democratizing AI'," states (Himmelreich, 2023). It was not readable to us, and its argument appears here only where a readable source carries it. Elizabeth Seger and colleagues wrote a 2023 preprint whose authors include researchers at the Centre for the Governance of AI and Google DeepMind. In a passage that cites Himmelreich, they write that harnessing and modifying effective democratic structures already in place, such as legislation, regulation, and multilateral standards, "avoids redundancy and reinventing the wheel" (Seger et al., 2023a). On this view the legitimacy gap is better closed by the institutions that already hold democratic authority, the states of Section 17.5.3, than by new processes run by developers.

Three further arguments sharpen the objection. Cristina Lafont argues, by her book's abstract, that institutional shortcuts, among them handing decisions to minipublics, small bodies of citizens chosen by lot to deliberate, require citizens to defer blindly to actors over whose decisions they cannot exercise control, and that there are no shortcuts to making a community better than its members (Lafont, 2019). She does not reject minipublics as such: the same abstract proposes new ways to use their democratic potential (Lafont, 2019). A representative sample of a thousand people deciding a model's principles for millions is such a shortcut, so on Lafont's view it would replace one unaccountable decider with another (inference). Seger and colleagues add that participants in participatory processes are generally self-selected, so their outputs may have only a weak claim to democratic legitimacy (Seger et al., 2023a). And the social choice results of Section 17.1 apply to any public process too: the consensus rule that selected the public constitution was one choice among many, made by researchers (inference). Seger and colleagues also separate four meanings of democratizing AI: democratizing its use, its development, its profits, and its governance. They observe that the first three are used almost synonymously with making AI more accessible, while the fourth concerns how those choices are balanced against other needs and values of society (Seger et al., 2023a).

The proponents have strong answers. Huang and colleagues report a growing consensus that model developers should not be the sole deciders of model behavior (Huang et al., 2024a). Seger and colleagues, against the self-selection objection, describe representative deliberation, putting questions to a microcosm of the population selected by sortition, a stratified random draw, as a way of granting democratic legitimacy. They report that Meta has run national and transnational pilots of such deliberation and scaled up to a near-global process (Seger et al., 2023a). The existing-institutions answer is also limited by Seger and colleagues' own observation that such structures can get bogged down and apply only within democratic societies (Seger et al., 2023a). And the STS critique of Section 17.6 asks for more participation of a different kind, recurring and able to reframe the question, not for less.

The evidence does not decide between these positions. Whether participation can confer legitimacy on a developer's specification at all is an argument among political theorists, and no study has measured how the people bound by a model judge the legitimacy of a publicly specified model against a developer-specified one. For the book's thesis, the exercises so far fit a familiar pattern in delegation: the agent consults its principals while keeping the pen. What the thesis cannot settle is whether the remedy is to move the pen to a public process or to the institutions that already speak for the public (inference). The claims of this section concern the third relationship, the developer and the public.

17.8 Settled, contested, missing #

17.9 Exercises #

Further reading #

• Arrow (1951): Social Choice and Individual Values, read here only as a catalog record; Section 17.1 states its result as Conitzer and colleagues give it.
• Conitzer et al. (2024): the case that learning from many people's feedback is a social choice problem and should be designed as one.
• Sorensen et al. (2024): three meanings of serving many people at once.
• Chouldechova (2017): the fairness impossibility in its plainest form.
• Huang et al. (2024a): the most complete published exercise in public input to a model, with its own account of where authority sat.
• Lazar and Nelson (2023): the contest over who defines what safety means.