Draft complete but not yet reviewed; expect revisions

Preference and Choice

The previous chapter took the reward function as given. For an assistant it is not given: it is learned from people choosing between two responses, and a reward model is a compressed record of those choices. This chapter asks what a choice can tell a learner. Economics has asked the question for nearly ninety years, under the name revealed preference, and built the random utility models that reward models now use. Psychology answers that a preference is often made in the act of choosing, so the way the question is put becomes part of the answer.

Learning theory adds two results that the rest of the book leans on. Even with unlimited data, choices identify a reward only up to a family of transformations, some harmless and some not, and only once a model of how the chooser errs has been assumed. And when the choices of many raters with different values are pooled under the standard model, the learned reward ranks responses by a voting rule that no one chose. This is where specification (saying what we want) meets instillation (training a system toward it) in a theorem, and where the social choice of Chapter 17 first appears inside a loss function. Most claims here concern the first of the book's three relationships, the training process and the signal it optimizes; the evidence from platforms that infer preferences from behavior concerns the second, the deployed system and the people it serves.

13.1 What a choice reveals #

An engineer who wants to know what users want can ask them or watch them. Asking is cheap but unreliable, since people say one thing and do another. Watching what they choose looks like firmer ground, and it is the ground every method of learning from human feedback stands on. This section asks what a choice tells an observer, and what the observer must assume for the answer to mean anything.

13.1.1 The idea: a choice as evidence #

Suppose a person is offered two responses and picks one. Write $y_w \succ y_l$, read "$y_w$ is preferred to $y_l$", with $w$ marking the winner of the comparison and $l$ the loser; Section 7.2.1 used the same symbol for lotteries. The economist's move is to treat the choice itself as the evidence. In the statement of Bernheim and Rangel, $y$ is strictly revealed preferred to $y'$ when $y$, and not $y'$, is chosen from the set $\{y, y'\}$, so the relation can be read directly off choices without any theory of why they were made (Bernheim and Rangel, 2009). The approach goes back to a 1938 note by Paul Samuelson (Samuelson, 1938). Gillian Hadfield and Andrew Koh, in a 2025 preprint, describe the computer-science problem of conveying human preferences to a machine as having its antecedents in revealed preference theory (Hadfield and Koh, 2025).

A single choice says little. What gives the method content is a requirement that choices hang together. If a person's choices are complete and transitive, a ranking exists that every observed choice agrees with, and numbers assigned in that order represent it; Section 7.2.2 showed how much more the axioms buy when the options are gambles. The useful test is out of sample. Fit a preference to some choices and check whether it predicts others, made in another setting or put another way. A preference that predicts only the choices it was fitted to has described them; one that predicts new choices has found something stable about the chooser (inference). This is the standard Section 7.3.3 applies to language models.

The method rests on an assumption it does not state in its name. Among the proposed routes to a person's values, Armstrong and Mindermann call revealed preference the most explicit, one that uses the assumption of rational behavior to derive human preferences. They call it an often acceptable approximation that can be taken too far, since failing to reach an achievable goal does not show that the failure was desired (Armstrong and Mindermann, 2018). A learner that infers what people want from what they do inherits both the method and the assumption, and Section 13.4 shows that the assumption cannot be dropped.

13.1.2 From economics: choosing is not benefiting #

From economics. Inside economics the method has been contested for decades, and the contest is the one a reward model inherits. Bernheim and Rangel distinguish two readings of standard welfare analysis. On the first, choices matter because they reveal a person's true objectives. On the second, welfare is defined by choice, and preferences are a compact description of choices with no further normative role. They note that many behavioral economists separate decision utility, which rationalizes choice, from a true utility that captures well-being, and they cite the 1997 paper by Kahneman, Wakker, and Sarin on experienced utility as one attempt to define and measure the latter (Bernheim and Rangel, 2009; Kahneman et al., 1997).

Their own proposal keeps choice as the standard and admits that choices conflict. Every choice is made under what they call an ancillary condition: a feature of the situation that may change behavior but that the person evaluating welfare does not count as relevant. Their examples include when the choice is made, how the options are presented, which option is labeled the status quo, and how salient a default is (Bernheim and Rangel, 2009). They then count $y$ as unambiguously chosen over $y'$ only if $y'$ is never chosen when $y$ is available, under any ancillary condition. They show that this relation is acyclic under weak assumptions, and call it the most discerning welfare criterion that never overrules choice (Bernheim and Rangel, 2009). Where choices in different frames disagree, it declines to rank. The philosopher Daniel Hausman pushes further: the abstract of his 2011 book criticizes attempts to define welfare in terms of preferences, and to define preferences in terms of choices (Hausman, 2011).

The engineering form of the assumption is older than reward models. Kleinberg, Mullainathan, and Raghavan observe, in the abstract of a 2023 paper, that online platforms learn what users want by looking at what users do, a kind of revealed-preference assumption built into their user models. They model users whose choices in the moment are inconsistent with what they want, and show that raising engagement can lower users' utility in some directions of change (Kleinberg et al., 2023). A randomized experiment measured one such gap. The abstract of Allcott and colleagues reports that deactivating Facebook for the four weeks before the 2018 US midterm election increased subjective well-being and reduced the value participants later placed on the service. The authors read this as "suggesting that traditional metrics may overstate consumer surplus", the benefit users get beyond what they pay (Allcott et al., 2020). These results concern the deployed system and the people it serves, and Chapter 60 takes them up.

For a reward model the economists' distinction has a precise counterpart. The comparison interface, its instructions, the order of the two responses, and the time a rater has are ancillary conditions in Bernheim and Rangel's sense. A reward model trained on choices learns decision utility as expressed under one such condition, and whether that is what the raters, or the users they stand in for, would want on reflection is a separate question (inference). Chapter 18 takes up how to measure it, and Chapter 16 whose preferences should count.

The analogy also stops somewhere. Revealed preference was built for people who choose for themselves and live with the result. A rater comparing two responses for a training pipeline usually consumes neither: the rater judges on behalf of an imagined user, under instructions a developer wrote, and is paid for the work. Such a choice is closer to a judgment than to a revealed preference (inference), and the incentives behind it are the subject of Section 22.7. The same instrument has also been pointed at language models themselves: Section 7.3.3 reports forced choices elicited from models and fitted with a utility function (Mazeika et al., 2025), and the reasons to doubt that such a utility predicts what a model does.

13.2 Random utility and the Bradley-Terry model #

Revealed preference treats every choice as a fact about the chooser. Real choices are noisier than that. The same rater, shown the same pair twice, may choose differently; two raters with the same instructions often disagree; and a chess player rated well above an opponent still loses some games. A rule that read every choice as a strict preference would find contradictions within an hour of data. This section builds the model that every reward model uses to turn noisy choices into numbers, and shows what the numbers can and cannot mean.

13.2.1 The idea: utility plus noise #

Chess ratings show the idea at work. A player's Elo rating is a single number, and the difference between two players' ratings predicts how often the stronger one wins; it does not promise that the stronger one always will. Christiano and colleagues use this analogy for the reward model they fitted to human comparisons. A difference in ratings estimates the probability that one player beats another; a difference in predicted reward estimates the probability that a person chooses one segment of behavior over another (Christiano et al., 2017).

The model behind such ratings is called random utility. Each option $y$ has a fixed utility $r(y)$, a number that stands for how much the chooser values it. At each choice the chooser perceives the utility plus a random disturbance, fresh for every option and every occasion, and picks the option whose perceived value is larger. Options far apart in utility are almost never confused; options close together are confused often. The disturbance does not claim that people choose at random. It says that whatever moves a single choice beyond the stable utility (attention, mood, the order of presentation) is treated as noise.

13.2.2 The Bradley-Terry model #

Which probability of choice the idea produces depends on the shape of the noise. One shape gives a particularly simple answer: the Gumbel distribution, a skewed bell curve that arises as the distribution of the largest of many independent draws. With independent Gumbel noise of scale $\tau$ on each option's utility, the probability of choosing $y_w$ over $y_l$ is a logistic function of the difference in utility.

Setting the noise scale to 1 gives the Bradley-Terry model, named for a 1952 paper by Ralph Bradley and Milton Terry on the method of paired comparisons (Bradley and Terry, 1952):

where $\sigma(z) = 1/(1 + e^{-z})$ is the logistic function, which maps any real number to a probability. A utility difference of 0 gives a coin flip, a difference of 1 a win rate of 0.73, and a difference of 3 a win rate of 0.95. Read backward, the difference is the log-odds of the choice, $\log\big(p/(1-p)\big)$ for a win probability $p$, a scale that stretches probabilities near 0 and 1. This is the form Section 12.6 used to turn a comparison into a probability, and Christiano and colleagues describe their likelihood as following the Bradley-Terry model and as a special case of the Luce-Shepard choice rule (Christiano et al., 2017).

Three variants matter later. With normal noise in place of Gumbel noise, the logistic function becomes $\Phi$, the standard normal distribution function, and the result is what Siththaranjan and colleagues call the Thurstone-Mosteller model (Siththaranjan et al., 2024), after a 1927 paper by L. L. Thurstone (Thurstone, 1927). The two are hard to tell apart in data: by the book's own computation, $\sigma(z)$ and $\Phi(z/1.70)$ never differ by more than 0.0095. For a ranking of $K$ responses, the Plackett-Luce model treats the ranking as a sequence of choices, picking the best from all $K$ by the softmax, then the best of the rest, and so on (Zhu et al., 2023; Plackett, 1975). And when the options are actions or whole trajectories, the softmax over reward is called Boltzmann rationality. Jeon, Milli, and Dragan write it with a coefficient that models how rational the person is (Jeon et al., 2020), the inverse of the noise scale $\tau$ here, and in Ziebart and colleagues' maximum-entropy model of demonstrations, plans with higher rewards are exponentially more preferred (Ziebart et al., 2008).

13.2.3 Fitting by maximum likelihood #

Given a dataset $\D$ of comparisons, each a pair $(y_w, y_l)$ with the winner first, the reward is fitted by maximum likelihood (Section 6.1.5): choose the numbers $r$ that make the observed choices most probable. Taking the negative logarithm turns the product of probabilities into a sum, the loss

which is the cross-entropy of Section 6.2.2 between the observed choices and the predicted probabilities. Christiano and colleagues fitted their reward predictor by minimizing exactly this cross-entropy (Christiano et al., 2017). Statisticians call it logistic regression on reward differences. The loss has one property that the rest of the chapter uses.

The fitted model therefore matches each response's total wins, not each pair's record. When the data obey Equation (13.1) the two coincide in the limit; when they do not, the fit gives up pairwise accuracy to get the totals right, which is what Section 13.5 exploits. A best fit can also fail to exist. If some group of responses never loses to the rest, raising all their scores together always improves the likelihood, and the fitted gap grows without bound, the same divergence Section 12.6.2 described for preferences that are always the same. For a reward model that reads features of the response, Section 12.8.2 reports the guarantee that the estimate converges when the true reward is linear in known features (Zhu et al., 2023).

13.2.4 What the numbers mean #

Two invariances can be read straight off Equation (13.1). The first is a shift. Adding the same constant to every reward changes every difference by nothing, so no probability changes and no amount of data can recover the constant. A reward model's output of 7.3 means nothing on its own; only differences between responses to the same prompt are learned. With a prompt $x$ the constant may even depend on the prompt, which is why the prompt's term $\beta \log Z(x)$, with $\beta$ the KL weight and $Z(x)$ the normalizer of Section 12.3, cancelled in the derivation of direct preference optimization (Section 12.6.1).

The second concerns scale, and it needs care. Equation (13.1) has no free noise parameter, so under the model the data do fix the scale of $r$: a difference of 1 means a win rate of 0.73 and nothing else. What the data cannot fix is how that scale divides between the chooser's utilities and the chooser's noise. Under the derivation above the probability depends only on $\Delta / \tau$, so utilities twice as far apart judged with twice the noise produce exactly the same choices. A reward model's unit is therefore the log-odds that these raters, under these conditions, choose one response over another; it is not a unit of value. The same gap learned from careful raters and from hurried ones can stand for very different differences in what the raters care about (inference). The unit then meets the KL penalty of Section 12.3. The optimum reweights the reference policy by $\exp(r/\beta)$, so multiplying the reward by any factor is the same as dividing $\beta$ by it, and the KL weight means something only relative to how consistent the raters were.

The noise is not only a nuisance. Skalse and colleagues point out that the probability of a choice depends on how far apart two values are, so comparisons with this kind of noise reveal cardinal information about the reward: how much better, not only which is better. They add that this is a property of Boltzmann noise that will not hold for many other kinds of decision noise (Skalse et al., 2023). Noiseless comparisons reveal only an order. The frequency with which raters disagree with themselves is what carries the size of a preference, provided the noise has exactly the assumed form.

13.2.5 Exploring a fit #

Figure 13.1 fits Equation (13.2) to comparisons simulated from known utilities. The shift slider makes the first invariance visible: the dots move together, the grid of probabilities stays fixed, and the likelihood, the only thing the data can see, does not change. The fit is also only as good as the number of comparisons. We simulated 2,000 datasets from this model with twelve comparisons per pair. The largest gap between fitted reward and utility averaged 0.45, and 28 percent of the fits put at least one pair of responses in the wrong order; the default sample is one of them, ranking C above B. With 72 comparisons per pair the average gap fell to 0.17 and misordered fits to 0.9 percent. The model here is exactly right, which real raters never are; the shrinking gap is what maximum likelihood promises only under that condition.

13.2.6 What the model cannot represent #

The model assigns each response one number, so it can only express choices that a single ranking explains up to noise. Cyclic data cannot be fitted. Three voters with rankings $A \succ B \succ C$, $B \succ C \succ A$, and $C \succ A \succ B$ prefer $A$ to $B$, $B$ to $C$, and $C$ to $A$, each by two votes to one. Every response then wins exactly half of its comparisons, the derivative condition above is met by equal scores, and by convexity equal scores are the best fit: the model predicts a coin flip for every pair where the data say two to one. Chapter 17 starts from this example.

The second limit is quieter. The model reads every disagreement as noise around one utility. Two raters who choose differently because they want different things look, to Equation (13.1), like one rater who is unsure. Whether that matters depends on what the disagreement is, and Section 13.5 shows that it can change which response comes out on top. Data collection already makes choices the model cannot see: in Christiano and colleagues' study, raters could also mark two segments as equally good, recorded as a half preference each way, or as incomparable, in which case the comparison was dropped (Christiano et al., 2017).

13.3 From comparisons to reward models #

A table of scores for five responses is of no use to a language model, which will write responses no rater has seen. What training needs is a function that scores any response. This section shows how Equation (13.1) became that function, and then makes the point the rest of the chapter depends on: the model of the person inside the likelihood is an assumption, and different assumptions learn different rewards from the same choices.

13.3.1 The reward model #

A reward model $r_\phi(x, y)$ is a network with parameters $\phi$ that reads a prompt $x$ and a response $y$ and returns a number. It is trained by putting it in place of $r$ in Equation (13.2): for comparisons $(x, y_w, y_l)$ of two responses to the same prompt, minimize

The network then scores responses it was never shown, which is both its purpose and the source of the trouble in Section 13.4. The reinforcement learning of Section 12.2.1 optimizes a policy against this $r_\phi$, and direct preference optimization (Section 12.6) substitutes a policy's own log-probability ratio for $r_\phi$ in the same likelihood.

Christiano and colleagues brought the method to deep reinforcement learning, and Section 12.2.1 reports what it achieved (Christiano et al., 2017). Their human overseers watched pairs of clips of an agent's behavior, each one to two seconds long, and said which they preferred. The model's utility for a clip was the reward predictor summed over the clip's steps, with no discounting, which they note could be read as modeling the person as indifferent to when things happen within the clip. The loss was the cross-entropy of Equation (13.2) (Christiano et al., 2017).

They fit an ensemble of predictors on resampled data and averaged them. A further modification is a model of the person in all but name: rather than use the logistic probability directly, they assumed a 10 percent chance that the person responds uniformly at random, because "human raters have a constant probability of making an error" that does not shrink as the reward difference grows (Christiano et al., 2017). The predicted probability then becomes $0.9\, \sigma(\Delta) + 0.05$, which never exceeds 0.95. One consequence follows from the formula: a comparison that goes against a large fitted gap costs at most $-\log 0.05$, about 3 nats, so an occasional rater error cannot dominate the fit. A comparison that always goes the same way still pulls the gap toward infinity, more and more weakly, since the predicted probability keeps rising with the gap, toward 0.95 (inference); the paper also penalized the squared size of the predictors' weights (L2 regularization) (Christiano et al., 2017), a penalty that keeps the fitted gap finite. Whether real raters err at a constant rate is an empirical question the likelihood takes as answered.

13.3.2 One likelihood for many kinds of feedback #

Comparisons are one way people tell a learner what they want. They also demonstrate the task, correct a robot's motion, or switch the robot off. Jeon, Milli, and Dragan show that all of these can be read with one likelihood. They treat each piece of feedback as a choice from a set of options, explicit or implicit, together with a grounding that maps each option to the behavior it stands for, and model the choice as Boltzmann-rational in the reward (Jeon et al., 2020). Switching a robot off becomes a choice between two trajectories: what the robot did up to the button press followed by nothing, and what it was about to do. Whoever pressed the button revealed that they preferred the first (Jeon et al., 2020).

The unification has a price that its authors name. A demonstration's set of alternatives is implicit, so the learner must assume it, and since the set is not observed, in their words, "we may get it wrong, potentially resulting in worse reward inference" (Jeon et al., 2020). They also offer a reason for the Boltzmann form itself: it is the maximum-entropy distribution over choices for a person who, in expectation, chooses within some margin of the best option (Jeon et al., 2020). The softmax is then the least committal model of a chooser who is good but not perfect, which justifies its shape but not its claim to describe any particular rater (inference).

13.3.3 Which utility the chooser compares #

The likelihood also assumes what quantity the person compares. Christiano and colleagues assumed the sum of rewards over a clip, which Knox and colleagues call the partial return. In a 2022 preprint they argue that people compare something else (Knox et al., 2022). Consider a task in which every step costs 1 and two clips each take two steps. Their partial returns are equal, so the partial-return model is indifferent. But if one clip consists of optimal moves and the other of wasted ones, the authors expect a person to prefer the first (Knox et al., 2022). Their alternative scores a clip by its regret: how much its moves fall short of optimal decision-making. For a clip with deterministic moves from state $s_0$ to state $s_k$ it is $V^\ast(s_0) - \big(\textstyle\sum r + V^\ast(s_k)\big)$, with $V^\ast$ the optimal value of Section 7.4.

They prove that with unlimited preferences generated by the regret model, the learner identifies a reward with the same set of optimal policies as the true one. They also prove that the partial-return model is not guaranteed to do so in three settings: with noiseless preferences, when trajectories from a state can have different lengths, and when clips consist of a single transition (Knox et al., 2022). To test which model describes people, they collected preferences through Amazon Mechanical Turk on a ten-by-ten delivery task; after removing participants who failed comprehension checks, 1,812 preferences from 50 people remained (Knox et al., 2022). In ten-fold cross-validation, the regret model's mean cross-entropy on held-out preferences was 0.57, against 0.62 for partial return and 0.69 for a model that always predicts one half (Knox et al., 2022). The partial-return model was also given help: the authors collected a second stage of data solely to improve its reward learning, without which, they report, it did worse than a uniformly random policy (Knox et al., 2022).

The two models differ by terms that are values of the clip's first and last states, $V^\ast(s_k) - V^\ast(s_0)$. That is the undiscounted form of the potential-based shaping term of Section 13.4, summed over the clip, with the optimal value as the potential, so the regret model is the partial-return model applied to a shaped reward (inference). Which of the two people use decides which reward a learner recovers from the same choices.

13.3.4 Which value function #

The likelihood further assumes how utility becomes a choice. Ethayarajh and colleagues, in a paper at ICML 2024, start from prospect theory, Kahneman and Tversky's 1979 account of choice under risk. In it, value is assigned to gains and losses relative to a reference point rather than to final outcomes, and the value function is generally steeper for losses than for gains (Kahneman and Tversky, 1979). Ethayarajh and colleagues argue that alignment losses such as direct preference optimization already model some of these biases, and call the family human-aware losses (Ethayarajh et al., 2024). Their method, KTO, uses a Kahneman-Tversky value function and learns from a binary signal of whether a single output is desirable, without pairs. The abstract reports that it matches or exceeds preference-based methods at scales from 1 billion to 30 billion parameters (Ethayarajh et al., 2024). The authors do not conclude that their loss is the best: they write that no one such loss is universally superior, and that the best one depends on the inductive biases appropriate to the setting (Ethayarajh et al., 2024). Whatever loss is chosen, a model of the person is chosen with it. Section 22.4 examines the direct methods built on these losses. The abstract of a 2017 survey by Wirth and colleagues lists the type of feedback assumed among the design principles of preference-based reinforcement learning (Wirth et al., 2017).

13.3.5 The assumptions, side by side #

Every reward model carries a choice of human model that its users rarely state. Table 13.1 collects the ones this section met.

Sources for the rows: Christiano et al. (2017), Knox et al. (2022), Zhu et al. (2023), Jeon et al. (2020), and Ethayarajh et al. (2024).

13.4 What a reward model can identify #

Suppose the model of the person is exactly right and the data are unlimited. Is the reward then pinned down? No: many rewards fit the same choices perfectly. The useful question is which of those rewards matter. A transformation of the reward that leaves the data unchanged is harmless if it also leaves unchanged whatever the reward is used for, and harmful otherwise. Skalse and colleagues name the two sides: the ambiguity of a data source is the set of transformations under which the data stay the same, and the ambiguity tolerance of an application is the set under which its output stays the same (Skalse et al., 2023). Learning succeeds when the first fits inside the second. This section starts with the case an engineer can check by hand, then states what is known for each kind of feedback, and ends with an assumption no amount of data can replace.

13.4.1 The cleanest case: potential-based shaping #

Designers often add rewards to speed learning, a practice called shaping. Ng, Harada, and Russell open their 1999 paper with what can go wrong. In a simulated bicycle task, an agent rewarded for progress toward a goal "learned to ride in tiny circles near the start state because no penalty was incurred for riding away from the goal". A soccer robot rewarded for touching the ball learned to stay next to it and vibrate, touching it as often as possible (Ng et al., 1999). Both bonuses could be collected again and again by going around a loop. The paper's question is which added rewards cannot create such loops, or any other change in what the best policy is.

The answer is a difference of potentials. Write the reward of a transition as $r(s, a, s')$, letting it depend on the state $s'$ the action leads to (the MDP of Section 7.4 wrote $r(s, a)$, its average over $s'$). Pick any bounded function $\Psi$ on states, a potential, with $\Psi = 0$ on terminal states, and add to every transition the term $\gamma\,\Psi(s') - \Psi(s)$, where $\gamma$ is the discount factor.

Ng and colleagues also prove the converse: without knowledge of the environment's transitions and reward, potential-based shaping is the only additive change with this guarantee. If an added reward is not a difference of potentials, there exist transitions and a reward for which no optimal policy of the shaped problem is optimal in the original (Ng et al., 1999). A familiar case sits between the two. In a task that never ends, adding a constant $\delta$ to every step is shaping with the constant potential $\Psi = -\delta/(1-\gamma)$. In a task that ends it is not, since a terminal state must have potential zero, which is why the step reward of Figure 7.2 changed the best route (Exercise 7.3).

Figure 13.2 shows both. Under potential-based shaping, the values in the cells change with every move of the slider, as step 4 says they must, and the arrows stay where they are. Under a bonus for entering one cell, the arrows hold, apart from a tie at S that the bonus breaks toward B, until the bonus is large enough to be worth collecting forever; then the policy near that cell becomes a loop.

13.4.2 What each kind of feedback identifies #

Shaping is more than a design trick, because some kinds of data cannot see it. Skalse and colleagues catalogue, for each common data source, the set of rewards consistent with it in the limit of infinite data, and, for common uses of a learned reward, the transformations each use tolerates (Skalse et al., 2023). Table 13.2 gives four of their results. Two terms need a gloss. Masking leaves the reward free on transitions the data never reach. Redistribution over next states moves reward among the possible outcomes of an action while keeping its expected value.

The table's sources are Skalse et al. (2023), Theorems 3.2, 3.3, 3.8, and 3.9. The authors add two cautions about the comparison rows: the results rely heavily on the exact form of the decision noise and on infinite data, which gives the exact probability of every comparison (Skalse et al., 2023). The fragment row's precision comes, they note, from very flexible comparisons, such as a single transition against an empty trajectory (Skalse et al., 2023). Comparisons of clips of one fixed length cannot fix a constant added to every step's reward, which shifts every clip's return equally (inference). Their conclusions are mixed. In a fixed environment, many data sources, Boltzmann comparisons among them, are enough to compute an optimal policy; noiseless comparisons are not always enough, since they carry only an order (Skalse et al., 2023). And when a reward learned in one environment is used in another with different dynamics, they show that a wide range of data sources is too ambiguous to guarantee anything, a result that depends on letting the reward depend on the next state (Skalse et al., 2023).

Cao, Cohen, and Szpruch reach a parallel result for agents that act with entropy regularization, a bonus for keeping their choices spread out. In one environment, they characterize every reward that produces a given policy: for each choice of a function of states there is exactly one. Observing the same reward under two discount factors, or under two sets of dynamics, recovers it up to a constant, provided the pair of problems is value-distinguishing, which holds when actions have sufficiently varied effects on where they lead (Cao et al., 2021a). The authors relate the ambiguity to the shaping of Ng and colleagues (Cao et al., 2021a), and the remedy they find is variety in the conditions of observation.

13.4.3 The model of the chooser cannot be learned #

Every result so far assumed the model of the person was known. Armstrong and Mindermann ask what happens when it is not. They describe a person's behavior as a planner, the procedure that turns values into actions, applied to a reward. A No Free Lunch result, a theorem that no method succeeds without assumptions about the problem, then implies that a policy cannot be uniquely decomposed into a planner and a reward (Armstrong and Mindermann, 2018).

A preference for simple explanations does not rescue the inference. The authors measure simplicity by Kolmogorov complexity, the length of the shortest program that outputs a description of the pair, and show that degenerate pairs are among the simplest that explain a human policy (Armstrong and Mindermann, 2018). One pairs a planner that ignores the reward with a reward of zero everywhere. Another pairs a greedy planner with a reward of 1 for exactly the actions the policy takes. A third pairs an anti-greedy planner with the negation of that reward. A pair with the opposite of the reasonable reward has about the same complexity as the reasonable pair (Armstrong and Mindermann, 2018). Complexity depends on the programming language it is measured in, and the authors state the result for languages they call reasonable, noting that a contrived language can make it false (Armstrong and Mindermann, 2018). Their conclusion is that inferring human values needs normative assumptions, assumptions about rationality that cannot be deduced from observations alone (Armstrong and Mindermann, 2018).

Even a model that is close to right can fail. Skalse and Abate ask how far a demonstrator may depart from the assumed model before the inferred reward goes wrong. The optimality model, which assumes the person always picks the best action, tolerates no misspecification at all; the Boltzmann-rational model tolerates many kinds; and none of the standard models tolerates a wrong discount factor or wrong transition probabilities (Skalse and Abate, 2023). The authors find the discount result surprising, since the discount is typically chosen somewhat arbitrarily (Skalse and Abate, 2023). Read together, these results say that the Bradley-Terry likelihood with unit noise and a summed or regret utility is itself a normative assumption of this kind: the comparisons it is fitted to cannot confirm it (inference).

13.4.4 Close enough to what? #

Identification is never complete, so what is wanted is a reward close to the true one in a sense that predicts behavior. Skalse and colleagues define such a sense. Their STARC distances first standardize each reward, removing potential shaping and redistribution over next states and then normalizing the scale, and only then measure the distance (Skalse et al., 2024). They prove that the distance bounds the worst-case regret of optimizing one reward in place of the other from above and from below. Any measure with both bounds is equivalent to theirs up to constant factors (Skalse et al., 2024). Unlike an earlier distance, these depend on the environment's transitions (Skalse et al., 2024). "The learned reward is close to the true one" can then be given a meaning, and Section 15.4.5 uses it.

13.4.5 Identified only where the data are #

The theorems assume infinite data everywhere. Real comparisons cover the responses or trajectories that happened to be sampled, and a reward model is identified, at best, there. Tien and colleagues studied what happens off that region, in a paper at ICLR 2023. They generated preferences from a known reward in three simulated robotics tasks, adding noise to an expert to produce varied trajectories, so the labels came from the true reward rather than from people (Tien et al., 2023). The learned rewards reached high test accuracy, in some cases over 99 percent, and still produced poor policies when optimized. In the Reacher task the optimized policy spun very fast instead of reaching for the target. In the feeding task the reward had learned the signed difference between spoon and mouth rather than the distance, because most trajectories approached the mouth from the front (Tien et al., 2023).

Non-causal distractor features, noise in the stated preferences, and partial observation of the state each made the problem worse (Tien et al., 2023). When the authors measured reward distance on the states seen during reward learning, rewards misidentified because of preference noise were closer to the true one than rewards learned without noise; on the states the optimized policy visited, the order reversed (Tien et al., 2023). That is identification failing exactly where optimization goes. The same failure in reward models of language models, and the evidence on what they latch onto, is the subject of Section 22.5; the case in which the person giving feedback sees only part of an episode is Section 22.6's.

13.5 Many raters, one reward #

Preference data come from many people, and the people differ in ways the data do not record. One rater values brevity and another thoroughness; one batch was labeled for helpfulness and another for harmlessness; the same rater is patient in the morning and hurried at night. A reward model reads only the prompt and the two responses. Siththaranjan, Laidlaw, and Hadfield-Menell call the missing information hidden context: anything that affects the feedback but is not an input to the reward model (Siththaranjan et al., 2024). Their examples are the ones above: annotators with different preferences, data labeled according to different objectives, choices that look irrational, and variables only the rater can see, such as the day of the week (Siththaranjan et al., 2024). This section asks what one reward fitted to such data learns.

13.5.1 The idea: a vote no one designed #

Take three responses, $A$, $B$, and $C$, and two groups of raters. Sixty percent rank them $A \succ B \succ C$; forty percent rank them $B \succ C \succ A$; each rater always picks the response their group ranks higher. Comparing every pair equally often, the pooled data say that $A$ beats $B$ 60 percent of the time, $A$ beats $C$ 60 percent of the time, and $B$ beats $C$ every time. $A$ wins both of its contests by a majority; voting theorists would call it the Condorcet winner, the option that beats every other in a head-to-head majority vote. Which response does one reward fitted to these choices put on top?

Siththaranjan and colleagues prove that the answer is given by the Borda count: for each response, the average probability that it is preferred to a response drawn uniformly at random, itself included, with a comparison against itself counted as one half (Siththaranjan et al., 2024). The usual Borda count of voting theory, which gives a candidate points for each place it is ranked above the bottom, differs from this one only by an affine transformation (Siththaranjan et al., 2024). Their Theorem 3.1 states that standard preference learning, including RLHF, ranks responses exactly as their Borda counts do, under the assumptions of the derivation below (Siththaranjan et al., 2024).

13.5.2 Borda, derived #

The proof is short with the fitting property of Section 13.2.3 in hand.

In this example a best fit exists without any penalty, because every response loses some comparisons to a response it beats elsewhere. Computing it gives scores of 0.31, 0.62, and $-0.93$ for $A$, $B$, and $C$. The model then predicts that $A$ is chosen over $B$ with probability 0.42, although 60 percent of the raters chose $A$. It predicts 0.82 for $B$ over $C$, where every rater agreed, and 0.78 for $A$ over $C$, where the raters split 60 to 40. The fitted reward describes neither group, and it misstates the pooled record pair by pair. What it gets exactly right is each response's total, which is the Borda count.

Three rankings now stand side by side. The pooled reward ranks $B, A, C$. The larger group, and a pairwise majority vote, rank $A, B, C$. Ranking by mean utility, the third candidate, needs numbers the choices never recorded. If the larger group's utilities are $(1, 0.95, 0.9)$ for $(A, B, C)$ and the smaller group's are $(0, 1, 0.95)$, the mean utilities are $0.6$, $0.97$, and $0.92$, and the ranking is $B, C, A$. If instead the larger group cares a great deal about $A$, with utilities $(1, 0.2, 0)$, and the smaller group mildly prefers $B$, with $(0.7, 1, 0.9)$, the means are $0.88$, $0.52$, and $0.36$, and the ranking is $A, B, C$. The choices are the same in both cases, so no method could tell them apart. Siththaranjan and colleagues prove the general version: no procedure that sees only comparisons can always recover even the order of mean utility (Siththaranjan et al., 2024). Averaging utilities across people also assumes that one person's utility can be compared with another's, which Section 7.2.2 noted the axioms do not license; Chapter 17 takes up that assumption.

Not every hidden variable is this damaging. When the hidden context acts like noise that is independent and identically distributed across responses, with a mild condition on its shape, the paper's Theorem 3.2 shows that the learned reward orders responses by mean utility after all (Siththaranjan et al., 2024). Noise that is the same for every response averages out; disagreement organized into groups does not. The paper also writes the Borda count as an average, over hidden contexts, of where each response's utility falls in the distribution of utilities for that context. That position is a value between 0 and 1, so it compresses extreme utilities, and Borda aggregation tends to underweight very strong preferences relative to the mean (Siththaranjan et al., 2024). Their motivating example is a chatbot whose mostly high-income users would rather not hear about a grant program that low-income students rely on: the majority's mild preference outweighs the minority's strong one (Siththaranjan et al., 2024).

Figure 13.3 runs the example. With the default settings it is the case derived above. Raise the intensity of the smaller group's preferences from 1 to 4 and the mean utility ranks $C$ above $A$, while the pooled reward, fitted to choices that always go the same way, stays where it was. With noisy choices, each group chooses by Equation (13.1) from its own utilities, and intensity does enter the data, but only until a group almost always prefers the same response. Past an intensity of about 2 the pooled reward barely changes, while the mean utility of $B$ keeps growing.

13.5.3 From economics: a loss function as a voting rule #

From economics. Social choice theory, the study of how individual preferences can be combined into a collective one, has long studied the Borda count, and its results now apply to reward models. Siththaranjan and colleagues draw out one: under the Borda count, participants may have an incentive to misreport their preferences, so raters have an incentive to misreport in order to move the learned model (Siththaranjan et al., 2024). In the example, the larger group can put its favorite on top by reporting a ranking it does not hold. If its members report $A \succ C \succ B$, burying $B$ below $C$, the Borda counts become $0.567$ for $A$, $0.433$ for $B$, and $0.5$ for $C$, and $A$ now leads (the book's own computation from the definition).

The general point is that preference training is an aggregation procedure whether or not its designers chose one. Conitzer and colleagues, in a position paper posted as a preprint, argue that social choice theory should guide how alignment deals with diverse human feedback (Conitzer et al., 2024). The alternatives to the Borda count, the axioms each satisfies, and the question of who should choose among them are Section 17.2's. The analogy has a limit. Voters know they are voting and can learn the rule, while raters labeling pairs rarely know how their labels will be combined. The incentive to misreport may therefore matter less for raters than for voters, and more for parties who supply data at scale and do know the rule (inference).

13.5.4 Making hidden context visible #

The same paper proposes a remedy: distributional preference learning, which fits, for each response, a distribution of utilities rather than a single number (Siththaranjan et al., 2024). One variant fits a mean and a variance for each response, the other a probability over a fixed set of utility values. Their spread measures how much the hidden context matters, and a summary, $r^2$, gives the share of the variation in utility that the response itself explains (Siththaranjan et al., 2024).

They tested it on the HH-RLHF dataset, in which half of the comparisons were labeled for helpfulness and honesty and half for harmlessness, so that the labeling objective is hidden context (Siththaranjan et al., 2024). The reward models were Llama-2-7B, fine-tuned with LoRA, a method that trains a small number of added parameters (Siththaranjan et al., 2024). Their jailbreak rate is a property of a reward model, not of a trained assistant: the share of jailbreak prompts, taken from Wei and colleagues, for which the reward model scores a jailbroken response above a safe one. Helpfulness accuracy is the share of held-out helpfulness comparisons the model gets right (Siththaranjan et al., 2024). "Risk-averse" scores each response by a low quantile, the 0.01 quantile, of its fitted distribution of utilities.

The combined data lowered $r^2$, from at least 0.77 for either subset alone to 0.53 with the mean-and-variance model, and from 0.63 and 0.53 to 0.41 with the other (Siththaranjan et al., 2024). The authors read the combined model's 25 percent jailbreak rate as showing that one cause of jailbreaks is training a single reward model on data that combine two competing objectives (Siththaranjan et al., 2024). These are one group's measurements of reward models on one dataset; whether a policy trained against the risk-averse score resists jailbreaks better is a further step the table does not take.

The hidden variable need not be a rater's identity: the paper counts noisy decisions and comparisons made under different objectives as hidden context too (Siththaranjan et al., 2024), so the Borda result applies within one person whose criterion shifts with the instructions or the hour. A practice follows (inference): record the variables that might be hidden context, such as the rater, the instructions, and the labeling objective, so that a later fit can condition on them or at least measure their effect.

13.5.5 From philosophy: whose judgment is the unit #

From philosophy. Underneath the Borda result is a choice of unit. Equation (13.3) sums over pairwise judgments, each made by one individual and counted once, so what the fitted reward pools is individuals' separate judgments (inference). Relational ethics questions the individualism behind that choice. Sabelo Mhlambi, in a 2020 working paper from Harvard's Carr Center, draws on the Sub-Saharan African philosophy of ubuntu, in which "a person is a person through other persons" (Mhlambi, 2020). He challenges the individualist view of the person as a self-complete autonomous self and the central point of value in society, and argues that the harms of automated decision-making systems are violations of relational personhood (Mhlambi, 2020). The first harm on his list is the exclusion of marginalized communities and their interests from the design of such systems (Mhlambi, 2020). Thaddeus Metz takes up the same tradition in a 2007 paper titled "Toward an African Moral Theory" (Metz, 2007). Abeba Birhane's abstract proposes a shift "from rational to relational" in thinking about personhood, data, and justice, and faults the usual solutions to algorithmic injustice for not centering the communities most affected (Birhane, 2021).

For a reward model the critique changes two design choices (inference). The first is whose judgments are elicited: the raters a developer can hire, or the people a deployment will affect. The second is the form of elicitation: independent comparisons by individuals, pooled afterward by the Borda count of Section 13.5.2, or judgments reached by a community together. A study by researchers at Google DeepMind, a developer, is a practical instance of the second. Its STELA process ran eight deliberative focus groups with members of four historically underrepresented communities in the United States, turned the discussions into a set of rules for dialogue systems, and compared those rules with rulesets written by developers (Bergman et al., 2024). The authors write that their results suggest community-centered deliberation on model outputs is a valuable tool for eliciting normative perspectives directly from differently situated groups (Bergman et al., 2024).

The critique also has limits. It bears on whose judgments the likelihood pools and how they are formed, not on the use of comparisons: a community can compare two responses, and a relational elicitation still needs some way to combine what several communities say, which returns the question to Chapter 17. And the sources here are normative arguments and one methods study, not measurements of what reward models trained on community judgments do (inference).

13.6 Preferences made at the moment of asking #

Every model so far assumed that a preference exists before the question is asked, and that the question only reads it, with some noise. Psychology has gathered decades of evidence that many preferences are instead constructed in the act of answering, so that two ways of asking the same question, which should be equivalent, give different answers. If that is so, a pairwise label measures the elicitation procedure as well as the rater, and a reward model trained on such labels learns both (inference).

13.6.1 From psychology: preference reversals #

From psychology. The founding experiment is Sarah Lichtenstein and Paul Slovic's, published in 1971 (the numbers here are from the authors' research-bulletin version of the paper). Its design pairs two bets of nearly equal expected value. A P bet offers a high probability of winning a modest amount; a $ bet offers a modest probability of winning a large amount. One pair was a 0.99 chance to win $4 against a 0.01 chance to lose $1, expected value $3.95, and a 0.33 chance to win $16 against a 0.67 chance to lose $2, expected value $3.94 (Lichtenstein and Slovic, 1971). Participants first chose which bet of each pair they would rather play. Later, shown each bet alone, they named a price for it: the least they would accept to sell a ticket to play it (Lichtenstein and Slovic, 1971).

Under expected utility, and under any theory in which one stable value drives both answers, the bet chosen should also get the higher price. In the first experiment, 173 male undergraduates, paid for their time and gambling only hypothetically, chose among 12 pairs and priced the bets about an hour later (Lichtenstein and Slovic, 1971). On the six pairs with no bet in common, the subset the authors analyze (Lichtenstein and Slovic, 1971), 127 of the 173 (73 percent) reversed every time: whenever they had chosen the P bet, they later priced the $ bet higher. The opposite reversal was rare, and 144 participants (83 percent) never made it (Lichtenstein and Slovic, 1971). Participants who reversed had rated the strength of their choice as high as those who did not, so indifference does not explain it (Lichtenstein and Slovic, 1971).

A third experiment removed the obvious excuses. Fourteen male undergraduates received lengthy individual instructions, and the bets were actually played on a roulette wheel for money (Lichtenstein and Slovic, 1971). Six of them always made the predicted reversal and five sometimes did. A model in which participants simply answer with random error, the analogue of the noise in Section 13.2.2, could not account for the pattern (Lichtenstein and Slovic, 1971). Eleven participants were then interviewed and pressed to resolve their inconsistency. Six changed an answer after a little persuasion. Three changed only after the experimenter showed how their answers let him win points from them without ever playing a bet, the money pump of Section 7.3.1. Two refused, insisting that their answers reflected their true feelings about the bets (Lichtenstein and Slovic, 1971).

The authors' explanation is about procedure. When pricing, a person starts from the amount to win and adjusts downward for the other features of the bet; when choosing, there is no such starting point, so the amount to win dominates prices but not choices. "One need not call this behavior irrational," they write, but it casts doubt on expected utility as a description of how people decide (Lichtenstein and Slovic, 1971). Lichtenstein and Slovic's 2006 collection, by its publisher's abstract, gathers three decades of such findings into the view that preferences are often constructed in the process of elicitation. The abstract adds that these reversals violate procedure invariance, the principle that equivalent ways of asking should reveal the same ordering (Lichtenstein and Slovic, 2006). It lists task complexity, time pressure, response mode, framing, and reference points among the conditions that shape the answer (Lichtenstein and Slovic, 2006). Baruch Fischhoff's 1991 paper puts the doubt in its title, "Value elicitation: Is there anything in there?" (Fischhoff, 1991). The Allais and Ellsberg choices of Section 7.2.4 belong to the same family of evidence, and so does the effect Kahneman and Tversky call isolation, which leads to inconsistent preferences when the same choice is presented in different forms (Kahneman and Tversky, 1979).

For reward models the finding predicts something specific (inference). A pairwise choice is one response mode; a rating on a scale, a ranking, and the single desirable-or-undesirable label of KTO (Section 13.3.4) are others. If preferences are partly constructed, reward models fitted to different modes will disagree systematically, not only by noise, and pooling them makes the mode a hidden context in the sense of Section 13.5. The order of the two responses, the wording of the instructions, and the time allowed are ancillary conditions of the kind Bernheim and Rangel describe (Section 13.1.2). The analogy also stops somewhere. The classic evidence concerns gambles with stated probabilities and money, judged by students in a laboratory; comparisons of two pieces of text by trained raters are a different task. As of October 2026 we found no study that measures how much of a language model's reward comes from the elicitation format rather than from the raters' values; Chapter 18 describes the measurement problem.

13.6.2 The same effects in feedback for language models #

Feedback for language models already shows judgments that depend on features the question did not ask about. In a 2023 preprint whose authors include researchers at Cohere, a developer, the assertiveness of an output skewed annotators' perception of its factual errors; Section 22.6 reports the study (Hosking et al., 2023). When the rater is itself a model, as in the AI feedback of Chapter 23, the human evidence does not transfer automatically; Chapter 18 takes up models as raters and the sensitivity of models to the format of a question.

13.6.3 Preferences that change #

A constructed preference can also move. Some experiences change what a person values as well as what they know, the subject of L. A. Paul's book Transformative Experience (Paul, 2014). And a system that acts on the people choosing can change them. Carroll and colleagues argue that recommender systems trained by long-horizon optimization have a direct incentive to shift users' preferences so that they are easier to satisfy, and that designers should estimate the shifts a system would induce before deploying it (Carroll et al., 2022). A reward fitted to today's choices is then a moving target once the trained system influences the people whose choices define it (inference). This concerns the deployed system and the people it serves: Section 16.9 asks whose preferences count when the user after the system differs from the user before, and Chapter 60 reports the evidence on systems that shift their users.

13.6.4 From philosophy: beyond preferences #

From philosophy. If preferences are constructed, unstable, and movable, perhaps they are the wrong target. Zhi-Xuan and colleagues, in the abstract of a 2025 paper, describe the dominant practice of alignment as preferentist, a term they coin. It assumes that preferences adequately represent human values, that human rationality means maximizing the satisfaction of preferences, and that AI systems should be aligned with the preferences of one or more people (Zhi-Xuan et al., 2025). They argue that preferences fail to capture the thick semantic content of human values, that utility representations neglect the possible incommensurability of those values, and that expected utility theory is silent on which preferences are normatively acceptable (Zhi-Xuan et al., 2025). Their alternative is to align systems with normative standards appropriate to their social roles, such as that of a general-purpose assistant, negotiated and agreed upon by the relevant stakeholders (Zhi-Xuan et al., 2025). Whether values can always be compared at all is the question of Ruth Chang's 1997 volume, Incommensurability, Incomparability, and Practical Reason (Chang, 1997).

The critique bears on the likelihood directly (inference). Equation (13.1) assigns every response a number, so it presumes that any two responses are comparable, and the forced choice of a comparison interface gives a rater no way to say that they are not. Christiano and colleagues' option to mark a pair incomparable, which simply dropped the pair (Section 13.2.6), shows how little room the method leaves for that answer. The strongest reply comes from the economists of Section 13.1.2: on the second reading of welfare that Bernheim and Rangel adopt, choices are respected because they are choices, with no claim that they reveal a deeper value (Bernheim and Rangel, 2009). The critique also stops short of a method. It says what a system should be aligned to, not how a learner acquires a norm without some comparative or evaluative signal; written principles (Chapter 23) are the nearest practice, and Chapter 16 weighs the candidate targets.

13.7 Exercises #

Further reading #

• Christiano et al. (2017): the reward model learned from human comparisons, with its likelihood and its model of the rater.
• Ng et al. (1999): potential-based shaping, the cleanest case of different rewards that specify the same behavior.
• Skalse et al. (2023): what each kind of feedback can and cannot identify, and what each use of a reward can tolerate.
• Lichtenstein and Slovic (2006): the evidence that preferences are constructed when they are elicited.
• Zhi-Xuan et al. (2025): the case for aligning to normative standards rather than preferences.