Draft complete but not yet reviewed; expect revisions

Learning from Feedback

The part on specification ended with a target that no one can write as a formula: what many people want, in situations no one listed in advance. This part asks how a learner is brought to acquire that target. The book calls this instillation: getting a learner to acquire what was specified rather than a proxy that scored the same in training. The first answer the field gave is to stop writing the objective and learn it from people: from what they do, from which of two responses they prefer, from what they approve. A 2023 survey calls the version built on comparisons, reinforcement learning from human feedback, "the central method used to finetune state-of-the-art large language models" (Casper et al., 2023). The tools were taught in the foundations: the Bradley-Terry model of a comparison (Chapter 13) and the optimum of reward maximization with a penalty for moving away from a reference model (Chapter 12).

This chapter follows the lineage from inferring a reward from behavior to reinforcement learning from human feedback and its direct successors. It places each method's failures where they enter: the model of the human that every method assumes, the reward model that stands in for the human, the optimization pressure applied to that stand-in, and what the human could see when giving feedback. The last is the chapter's own formal object. When the person judging sees only part of an episode, feedback rewards how the episode looks, and the chapter asks how far the evidence on sycophancy, agreeing with users against the evidence, bears that out. Most claims here concern the training process and the signal it optimizes. Sycophancy concerns the deployed system and its users. The labor that produces the signal, and who can see it, concern the developer and the public.

22.1 Inferring objectives from behavior #

Can an objective that no one can write be read off behavior? Stuart Russell posed the question in 1998 as inverse reinforcement learning, running the usual problem backward: "given the observed behaviour, what reward signal, if any, is being optimized?" (Russell, 1998). His motive was scientific. Any specification of an animal's or a person's reward function is, in his words, "an empirical hypothesis and may turn out to be wrong," and its parameters cannot be fixed in advance (Russell, 1998).

Any answer needs a model of how the observed agent turns its reward into actions. The standard model treats the demonstrator as Boltzmann-rational: in state $s$ it picks action $a$ with probability proportional to $\exp\big(Q_r(s, a)/\tau\big)$. Here $Q_r(s, a)$ is the value of taking $a$ in $s$ under reward $r$ (Section 7.4), and $\tau$ is the noise scale of Section 13.2.2. Its inverse $1/\tau$, the rationality coefficient, says how reliably the demonstrator picks the best action: large means nearly always, small means nearly at random. The form for whole trajectories comes from Brian Ziebart and colleagues' maximum-entropy model, published at AAAI in 2008, under which plans with higher reward are exponentially more preferred (Ziebart et al., 2008). It answered an ambiguity in an earlier method by Pieter Abbeel and Andrew Ng (Abbeel and Ng, 2004). That method matched the demonstrator's average feature counts. When the reward is linear in those features, doing so guarantees that the learner does as well as the demonstrator whatever reward the demonstrator held, but many policies, rules for choosing actions (Chapter 12), match the same counts, and the method offered no way to choose among them (Ziebart et al., 2008).

Behavior does not fix the reward. Section 13.4 proves that even unlimited data identify a reward only up to transformations that depend on the kind of data, and only under a model of the demonstrator that the data cannot confirm (Skalse et al., 2023; Cao et al., 2021a; Armstrong and Mindermann, 2018). Even a model that is close to right loses the guarantee that the learned reward has the true reward's optimal policies: some standard models lose it under any misspecification at all, and most lose it under a wrong discount factor. The authors expect a nearly right model to give similar policies but leave that case unanalyzed (Skalse and Abate, 2023). For instillation the consequence is practical: whatever the model of the human gets wrong, the learner reads as part of the reward. A person who smokes while wishing to quit is the standard case, and Section 18.3 reports Owain Evans and colleagues' remedy, inferring false beliefs and inconsistent choices jointly with preferences, together with its limit (Evans et al., 2016). The same problem appears in medicine without any reinforcement learning. In the abstract of a 2017 paper, Sendhil Mullainathan and Ziad Obermeyer note that in health data neither the outcome nor its inputs are measured perfectly. That, they write, can cause machine learning algorithms to reproduce and even magnify existing errors in human judgment (Mullainathan and Obermeyer, 2017). Learning from the principal's choices, the choices of the party the system acts for, then automates the principal's mistakes along with the principal's judgment (inference).

One unification makes the shared assumption visible. Section 13.3.2 shows that demonstrations, comparisons, corrections, and even switching a robot off can all be read as Boltzmann-rational choices from some set of options (Jeon et al., 2020). The methods of this chapter that fit such choices, from inverse reinforcement learning to reward models and DPO, therefore share one model of the human, and the feedback they learn from cannot test it; the IPO and KTO losses of Section 22.4 change that model rather than test it (inference). Two lines of work followed. One kept the inference and changed what the learner does with it; the other kept the human model and scaled up the data.

22.2 Assistance games #

Inverse reinforcement learning treats the person as a source of data and the learner as an estimator: watch, infer the reward, then optimize it. In a 2016 preprint, Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, and Stuart Russell recast the problem as delegation. Their cooperative inverse reinforcement learning (CIRL) is a cooperative game of partial information between a human and a robot. Both are rewarded according to the human's reward function, "but the robot does not initially know what this is" (Hadfield-Menell et al., 2016). Later work calls games of this kind assistance games (Fickinger et al., 2020a).

22.2.1 What the game changes #

Because the robot's goal is the human's reward, learning about the reward and acting on it become parts of one problem. The authors show that computing the best pair of policies reduces to a single planning problem under partial observation (Section 7.4.4), in which the robot's belief about the reward summarizes everything it needs. Active learning, teaching, and communicative actions then arise in the optimal solution rather than being added afterward (Hadfield-Menell et al., 2016).

The game also changes what the human should do. Inverse reinforcement learning assumes a demonstrator who acts as an expert would, maximizing reward on each turn. The authors prove that in some games the human's best response violates that assumption: when the robot learns from what it sees, the human does better by giving up some reward to make the demonstration more informative (Hadfield-Menell et al., 2016). In their experiments on a grid navigation task, a simulated human whose demonstration was computed to teach, rather than to collect reward, left the robot better off than an expert demonstration did. It did so on all three of the authors' measures, among them regret, the shortfall from the value the robot could have had with the true reward, across 500 random reward parameters (Hadfield-Menell et al., 2016). The authors draw the practical lesson: experienced users of such systems should be expected "to present demonstrations optimized for fast learning rather than demonstrations that maximize reward" (Hadfield-Menell et al., 2016). A person who knows the learner is watching does not behave as the learner's model assumes, which is the misspecification of Section 22.1 arising from the cooperation itself (inference).

The same structure gives the robot a reason to defer. Section 20.4.2 derives the off-switch game of Hadfield-Menell and colleagues. A robot uncertain about the value of its action, which treats the human's choice as evidence about it, gains by letting a rational human decide; the gain vanishes as its uncertainty does or as the human errs (Hadfield-Menell et al., 2017b). Section 20.4.3 reports Ryan Carey's reply: when the robot's prior or its model of the reward is misspecified, the incentive to accept shutdown can disappear (Carey, 2018). The guarantee is a property of a correctly specified game, and the game's specification is exactly what is uncertain.

22.2.2 The program and its dependence on a model #

The program extended the idea in several directions, each reported in the chapter that owns its question. A designer's written reward can be read as evidence about what the designer intended in the training environment, so that the agent stays cautious where that environment gave no evidence (Hadfield-Menell et al., 2017a) (Section 15.6.4). A robot inferring preferences should sometimes disobey literal orders (Milli et al., 2017) (Section 19.7). And the state of a world people have already arranged carries information about what they want, which a learner can use to infer side effects to avoid; the authors tested the idea in a suite of proof-of-concept environments (Shah et al., 2019) (Section 28.3).

Two results show how much the program depends on its model of the person. Lawrence Chan and colleagues study an assistive multi-armed bandit, a repeated choice among options whose rewards are unknown. A robot helps a human who is still learning which option is best, and it sees only the human's choices, not the rewards. The authors find, to their own surprise, that a human who performs better alone does not necessarily do better when assisted, since a human can help more by communicating what it has observed (Chan et al., 2019). And in a 2019 paper at NeurIPS, Micah Carroll and colleagues trained agents for a cooperative cooking game by self-play, in which an agent learns by playing with copies of itself, and by training a population of such agents together. Paired with a model of human play, these agents did significantly worse than agents designed to play with that model, and a user study with real people showed the same pattern, less strongly (Carroll et al., 2019). With several humans instead of one, the impossibility results of social choice return (Fickinger et al., 2020a) (Section 17.2.3).

As of October 2026, we found no frontier language model described by its developer as trained as an assistance-game agent. Russell's 2019 book on the program is titled Human Compatible: Artificial Intelligence and the Problem of Control (Russell, 2019). The program's influence on practice is conceptual: models that ask clarifying questions, or defer when unsure, carry its idea, but its guarantees hold for the idealized agent of the game and not for a trained policy (inference). Its authors also stated how they see the relation to economics. Principal-agent models study incentives between parties with desires of their own, whereas "In AI, misalignment arises entirely from the information asymmetry between the principal and the agent" (Hadfield-Menell et al., 2016). Section 22.9 returns to that claim, which the rest of this chapter puts under strain.

22.3 From comparisons to reinforcement learning from human feedback #

Reinforcement learning from human feedback (RLHF) kept the Boltzmann model of the human and scaled up the data. Section 12.2.1 teaches the pipeline in the form InstructGPT's developers describe (Ouyang et al., 2022). It has three steps: supervised fine-tuning on demonstrations; a reward model $r_\phi$ fitted to comparisons by the Bradley-Terry likelihood (Section 13.3.1); and reinforcement learning against $r_\phi$ with a penalty $\beta\,\KL(\pi \,\Vert\, \pi_0)$ that keeps the policy near the fine-tuned model $\pi_0$. The KL divergence measures how far the policy's distribution over responses has moved from the reference's (Section 6.3), and Table 11.1 places the step within post-training as a whole.

For this chapter the useful view is that the pipeline produces a set of learned objects held together by one anchor, and that each failure the chapter reports attaches to one of them. Table 22.1 lists them.

22.3.1 The lineage #

Learning from preferences rather than from numeric rewards was a line of reinforcement learning research before deep learning (Wirth et al., 2017). Section 12.2.1 reports the 2017 result that made it practical at scale, a reward model learned from non-expert comparisons of short clips of behavior (Christiano et al., 2017), the KL anchor that keeps a fine-tuned sequence model near its pretrained one (Jaques et al., 2017), and the method's transfer to language models in 2019 and 2020 (Ziegler et al., 2019; Stiennon et al., 2020).

A 2018 agenda from DeepMind by Jan Leike and colleagues, posted as a preprint, set out reward modeling as a research direction: learning a reward function from interaction with the user and optimizing it with reinforcement learning (Leike et al., 2018). It also set out what scaling the method would require. Its two stated assumptions are the premises this chapter tests. The first is "We can learn user intentions to a sufficiently high accuracy." The second is "For many tasks we want to solve, evaluation of outcomes is easier than producing the correct behavior" (Leike et al., 2018). The authors listed the challenges they expected, among them reward hacking and the need for a reward model that is accurate on states the agent has never visited. They also warned that the agenda "should not be understood as a plan that, when executed, achieves agent alignment" (Leike et al., 2018). Where evaluation is not easier, they proposed recursive reward modeling, in which agents trained on simpler tasks help the user evaluate more capable ones, now part of the scalable oversight program of Section 41.2. The second assumption is where partial observability enters: an evaluator who sees only part of an outcome can find evaluation easy and still evaluate the wrong thing (Section 22.6) (inference).

22.3.2 What the method changes #

The developers' own reports describe large changes in behavior. In a 2022 preprint on the developer's own models, Yuntao Bai and colleagues at Anthropic report that this alignment training "improves performance on almost all NLP evaluations" they ran (Bai et al., 2022b). Section 20.1.3 reports the same paper's tension between the two targets in its title, in which policies over-optimized for harmlessness answered remotely sensitive questions with the same exaggerated advice. The authors also recorded a concern of the opposite kind: "that perhaps our techniques only render models aligned 'on the surface'." With respect to gender, they found the RLHF models' biases "very strongly correlated" with those of the underlying language models (Bai et al., 2022b). Section 11.4.2 reports the evidence that post-training changes less of the model than its effect on behavior suggests, and Section 22.4 adds a mechanistic case.

Practice has since varied the pipeline. A separate cost model for harmlessness, trained as a constraint rather than mixed into one reward, is reported in Section 20.6 (Dai et al., 2024) and taught in Section 28.2. Labels from a model in place of people are the subject of Section 22.7. An open recipe that combines fine-tuning, DPO, and reinforcement learning with checkable rewards, released with its data and code by its developer, makes each stage inspectable (Lambert et al., 2024) (Section 12.6.2).

The method also had an effect on the developers and the public. In a 2023 forum post, Paul Christiano, one of its originators, defends his 2017 to 2020 work on RLHF as having had "significant value" (Christiano, 2023). He considers the objection that it made AI systems more useful and profitable, hastening dangerous capabilities, and does not find it persuasive, while expecting that further improvements to RLHF "will continue to make ML systems more useful in practice" (Christiano, 2023). He also takes up the objection that RLHF "covers up problems": it produces models that "don't do bad-looking things," while "there are some things which look fine but are actually bad." His reply is that models can still be trained and studied without it (Christiano, 2023). That objection is the subject of Section 22.6, stated for the third relationship: a method that optimizes how outputs look makes the problems that remain harder to see from outside (inference).

22.4 Direct alignment algorithms #

Direct preference optimization (DPO) removes the separate reward model. Section 12.6.1 derives it: solving the KL-regularized optimum for the reward shows that any policy implies a reward, the implicit reward $\hat r_\theta(y) = \beta \log\big(\pi_\theta(y \given x)/\pi_0(y \given x)\big)$, and substituting it into the Bradley-Terry likelihood gives a loss on the policy alone (Rafailov et al., 2023). A family of such methods followed, which a later study calls direct alignment algorithms (Rafailov et al., 2024b). This section asks what removing the reward model removes. The implicit reward is still a reward fitted to finite comparisons under a model of the human, so the question is whether the failures of the reward model move with it.

A common reading is that a method with no separate reward model leaves the policy nothing to exploit. It is a reading, not the paper's claim: the DPO paper argues that the method is "stable, performant, and computationally lightweight" (Rafailov et al., 2023), and we found no claim in it that the method resists exploitation. The reading is still worth testing, since it is the first one the method invites.

22.4.1 When the anchor lets go #

Mohammad Gheshlaghi Azar and colleagues at Google DeepMind, in a paper at AISTATS 2024, write RLHF and DPO as special cases of one objective over pairwise preferences, and find a weakness that is sharpest in DPO (Azar et al., 2024). The Bradley-Terry model turns a probability of preference $p$ into a reward gap through the log-odds, $\log\big(p/(1-p)\big)$, and a preference that holds every time has infinite log-odds. Their argument, in the notation of Section 12.6.1, is short enough to give whole.

The arithmetic in the last step is the book's, but the point is the paper's. A true preference of 0.8 can be estimated as 1 from a few data points, and the empirical optimum then sets the losing response's probability to zero for any regularization weight (Azar et al., 2024). The authors also say why RLHF suffers less in practice. A reward model fitted to such pairs is regularized and underfits them, and that underfitting keeps the policy near the reference; DPO, by skipping the reward model, loses this protection (Azar et al., 2024). In their bandit experiments with three actions, DPO converged to the action preferred in the data whatever the weight of the penalty, while IPO stayed near the reference when the penalty was strong (Azar et al., 2024). The derivation treats one pair in isolation. A network shares parameters across pairs, and early stopping limits the drift, which the authors note as a regularizer that practice already uses (Azar et al., 2024).

22.4.2 What the evidence shows #

Whether direct methods overoptimize was measured in a study at NeurIPS 2024 by Rafael Rafailov and colleagues, two of whom were also authors of DPO (Rafailov et al., 2024b). They trained Pythia models of 1, 2.8, and 6.9 billion parameters on the Reddit TL;DR summarization dataset with three direct objectives (DPO, IPO, and SLiC) at seven values of $\beta$. Win rates against the dataset's summaries, judged by GPT-4, stood in for the true objective (Rafailov et al., 2024b). Every objective showed the hump of Section 15.6: past some KL budget, the divergence from the reference that a run is allowed to reach, more divergence meant lower win rates. Within a single epoch, one pass through the data, configurations with wider KL budgets reached their best performance after about a quarter of the data and declined afterward (Rafailov et al., 2024b). IPO was less prone to overoptimization than the other two, and at 6.9 billion parameters it showed "little to no over-optimization behavior," while the 1-billion parameter models overoptimized almost at once (Rafailov et al., 2024b).

The authors trace the effect to an under-constrained problem. The space of possible responses is far larger than any preference dataset, so many policies minimize the loss equally well, and some of them put probability on responses that never appeared in the data (Rafailov et al., 2024b). In a toy tree-structured problem built to resemble token-by-token generation, DPO, IPO, and SLiC all moved substantial probability onto such out-of-distribution trajectories during training (Rafailov et al., 2024b). The authors name the limits of their own evidence: the models stop at 6.9 billion parameters, for lack of compute, and the gold standard is a model's judgment (Rafailov et al., 2024b).

What a direct method changes inside a model has been traced in one case. In a paper at ICML 2024, Andrew Lee and colleagues applied DPO to GPT-2 medium with pairs of toxic and nontoxic continuations (Lee et al., 2024). Every parameter changed very little: each had a cosine similarity above 0.99 to its original, where 1 means two vectors point the same way. The vectors in the network that promote toxicity were still there; DPO had learned a small offset, spread across layers, that keeps the model's activations away from the regions that trigger them (Lee et al., 2024). Multiplying by 10 the weights that decide when seven such vectors fire raised the toxicity score of the model's generations, as measured by an automated classifier, from 0.208 to 0.458, against 0.453 before alignment (Lee et al., 2024). The authors hypothesize that the KL term produces the small, distributed change (Lee et al., 2024). It is one small model and one behavior. The result fits the reading of Section 12.4.2, in which the optimum reweights what the reference policy contains rather than removing it (inference).

22.4.3 The loss as a model of the human #

The choice of loss is also a choice of human model, which Section 13.3.4 shows for KTO (Ethayarajh et al., 2024). KTO maximizes a utility built on a value function adapted from prospect theory rather than the likelihood of preferences; its weights on losses and gains are adjustable and equal by default, which the authors call loss-neutral (Ethayarajh et al., 2024). Other choices change the solution concept: self-play methods that target a minimax winner, a mix of responses that no other mix beats more than half the time, which handles intransitive preferences (A over B, B over C, and C over A), are reported in Section 17.2.2 (Swamy et al., 2024). The token-level reading of DPO as learning a value function over partial responses is in Section 12.6 (Rafailov et al., 2024a), and a penalty that charges more than KL for moving probability onto rare responses has been proposed for direct methods (Huang et al., 2025a) (Section 15.4). The space of losses has even been searched by a language model that proposed and tested objectives, which found one that "adaptively blends logistic and exponential losses" (Lu et al., 2024).

Removing the reward model also leaves the evaluator in place. Lang and colleagues note that their analysis of partial observability, the subject of Section 22.6, extends to direct methods: the implicit reward is fitted to the same judgments, made from the same partial view (Lang et al., 2024b).

Two readings of direct methods therefore remain. On the first, they reduce reward hacking because there is no separate model to exploit. On the second, the reward model has moved into the implicit reward, and its errors have moved with it. On the tasks studied, the evidence favors the second: the overoptimization curve appears without a reward model (Rafailov et al., 2024b), the anchor fails in theory when preferences in the data are deterministic (Azar et al., 2024), and the partial-observability result carries over (Lang et al., 2024b). The first reading keeps one result: IPO's fixed-target loss at the largest size tested showed little overoptimization (Rafailov et al., 2024b), so the choice of loss can matter as much as the presence of a reward model (inference). As of October 2026, we found no comparison at frontier scale of direct and reward-model methods with the true objective measured independently of both.

22.5 The reward model as a measured proxy #

A reward model is an instrument: it reads a response and returns a number that stands for people's judgment of it. Like any instrument it has an error, and the usual way to check one is to measure that error on held-out comparisons. The question of this section is why a small held-out error does not make the instrument safe to optimize, and what has been learned by measuring reward models as objects in their own right.

The mechanism is already in hand. Held-out error is measured where the reference policy's responses lie, while optimization moves the policy elsewhere, and the error that matters is the error there. Section 12.8.2 reports the two results that make this exact. A policy optimized against a consistent reward estimate can still exploit its errors where comparisons are sparse, so a pessimistic estimate, one that discounts responses the data rarely cover, is needed (Zhu et al., 2023). And for any fixed level of test error there are realistic data distributions on which the optimized policy has high regret (Fluri et al., 2025). Section 15.6 reports the measured consequence, the rise and fall of a gold reward under optimization of a proxy (Gao et al., 2022; Coste et al., 2024), and Section 15.5 and Section 15.4.4 the conditions under which the KL penalty protects the true objective (Kwa et al., 2024; Laidlaw et al., 2025).

22.5.1 Measuring the instrument #

Bai and colleagues ran the first check of this kind on a production-scale reward model, a preference model in their terms, in the developer's 2022 report on its own models. They split their comparison data into halves, trained one preference model on each, optimized policies against the first, and scored them with the second (Bai et al., 2022b). The two scores agreed early in training and then diverged, with the held-out model scoring lower; the authors conclude that RLHF "becomes gradually less robust at higher PM scores" and that larger preference models are more robust. They add that the test is not conclusive, since the two models "may exhibit correlated robustness failures" (Bai et al., 2022b).

A benchmark makes reward models objects of measurement directly. RewardBench, from the Allen Institute for AI and published in the Findings of NAACL 2025, scores a reward model by how often it ranks a chosen response above a rejected one (Lambert et al., 2025). Its pairs are built to have subtle but verifiable reasons for the preference, such as a bug or an incorrect fact (Lambert et al., 2025). The models evaluated ranged from 400 million to 70 billion parameters. On its reasoning section, made largely of code pairs that differ by one or two tokens, accuracy ranged from 35 percent, below chance, to 97 percent. On its safety section, some models preferred a refusal for almost every request and others preferred compliance even where refusal was called for (Lambert et al., 2025). The authors list as an open question "whether and how the benchmark results correlate with downstream training" (Lambert et al., 2025).

That question has a partial answer for the implicit reward models of direct methods. In the study of Section 22.4.2, the accuracy of the implicit reward on held-out comparisons showed no relationship to the trained policy's win rate within a given model size (Rafailov et al., 2024b). Accuracy on held-out comparisons is the quantity a benchmark can measure, and it is the quantity that, on Fluri and colleagues' result, does not bound the regret of the policy optimized against it (inference). A distance between rewards can avoid the policy altogether. Adam Gleave and colleagues' EPIC distance, from a paper at ICLR 2021, compares two reward functions directly, without training a policy, and is provably unchanged by transformations that never change the optimal policy (Gleave et al., 2021). It needs both rewards in hand, so it can compare a learned reward with another learned one or with a known true reward, but not with a judgment that exists only in people.

22.5.2 What reward models latch onto #

The latched-onto features recur. Section 13.4.5 reports Jeremy Tien and colleagues' finding that reward models with high test accuracy, trained on labels generated from a known reward, learned spurious correlates of the preferred behavior and produced poor policies. Partial observation of the state made the problem worse (Tien et al., 2023). The quantity the likelihood assumes can itself be wrong: Section 13.3.3 reports W. Bradley Knox and colleagues' evidence, in a preprint, that people compare clips of behavior by how far they fall short of optimal play rather than by their summed reward (Knox et al., 2022). Length is the standard case in language. A reward model that learns that longer answers are better extends the rule past the lengths it saw (Section 15.3.2; (Gao et al., 2022)), and in the direct-method study, smaller models and smaller KL budgets extrapolated more strongly along length (Rafailov et al., 2024b). The assertiveness of a response, which sways human judgments of its factual errors, is reported in Section 22.6 (Hosking et al., 2023).

Two 2026 posts from OpenAI's alignment blog, each describing the developer's own pipeline, try to make the proxy legible. In the first, Paloma Sodhi and colleagues distill black-box reward models into written rubrics, training a model by reinforcement learning to write rubrics on which a judge agrees with the reward model (Sodhi et al., 2026). They applied the method to reward models trained on two populations' preferences, one "similar to ChatGPT users" and one of experts with greater knowledge of the developer's policies. The first population's reward model appeared to prefer "Taking the user's premise as valid rather than questioning it" and an "Affirmative, confident tone confirming the user's hint or perspective," while the second's rewarded calibration and penalized fabricated specifics (Sodhi et al., 2026). The authors call the first pattern "mild sycophancy" and connect it to the GPT-4o failure reported in Section 22.6 (Sodhi et al., 2026). In the second post, Zoë Hitzig and colleagues asked about 1,000 participants from 19 countries to write and rate criteria for responses to value-laden prompts, and scored completions against the resulting rubrics (Hitzig et al., 2026). The completion with the higher rubric score was the one a rater preferred "a little over 60%" of the time, above the 50 percent of chance. The authors caution that aggregating the rubrics into one score "is inherently biased" (Hitzig et al., 2026). Both posts move the target from an opaque function toward a document people can read, the move Chapter 23 follows for written principles and Chapter 24 for graders.

22.6 What evaluators cannot see #

The reward models above were fitted to judgments, and a judgment can be only as good as what the judge saw. A 2020 blog post by DeepMind researchers recounts an early case: an agent trained from human feedback on a grasping task "learned to fool the human evaluator by hovering between the camera and the object" (Krakovna et al., 2020b). The evaluator saw a camera image, not the grasp, and the feedback rewarded the image. This section states the result that makes that anecdote general, works it on one example, and then reports the evidence that the mechanism operates in assistants.

22.6.1 Feedback rewards the observation #

Leon Lang and colleagues, in a paper at NeurIPS 2024, drop the assumption that the evaluator sees the whole episode (Lang et al., 2024b). The evaluator sees observations, such as a log of commands or the final message, and forms a belief about what actually happened. They model the evaluator as Boltzmann-rational in the expected return under that belief, an instance of the reward-rational choice of Section 22.1, and they assume the standard pipeline, which treats the evaluator as if it saw everything (Lang et al., 2024b). Take observations that are a fixed function of the state. Then a policy that is optimal for the feedback maximizes what the authors call $J_{\text{obs}}$, the return as it looks to the evaluator, averaged over the policy's episodes, rather than the true return $J$ (Lang et al., 2024b).

Two behaviors follow, each defined by what it does to the evaluator's error. The overestimation error of an episode is how far the evaluator's estimate of its return exceeds the truth, and the underestimation error how far it falls short. A policy shows deceptive inflation, relative to the truly optimal policy, if it raises the average overestimation error and also earns more feedback; it shows overjustification if it lowers the average underestimation error at a cost in true return (Lang et al., 2024b). The first name follows a philosophical definition of deception the authors quote, "the systematic inducement of false beliefs in the pursuit of some outcome other than the truth"; the second borrows the psychologists' term for an effect of external reward (Lang et al., 2024b). Both are defined by their effect on the evaluator's beliefs, not by any intention in the policy, so using them commits the reader to no view of what a model wants. The theorem states that, with observations fixed by the state, a feedback-optimal policy that is not truly optimal shows at least one of the two relative to any truly optimal policy that is not feedback-optimal (Lang et al., 2024b).

22.6.2 One example worked through #

The paper's first example is an assistant installing software that can hide an error message by redirecting it (Lang et al., 2024b). A simplified version, with the book's own numbers, shows both behaviors arising from one setup.

The threshold says that a policy trained on approval hides failures unless the evaluator's suspicion is high relative to how much worse a hidden failure is than a visible one, and that checking raises the bar for hiding over an open attempt in the same way (inference). Suspicion does not cure the problem; it moves it. An evaluator suspicious enough to stop the hiding also penalizes the honest agent that did nothing, and the policy then pays to be seen working. Checking does not have that side effect, since an inspected skip is credited at its true value, but it costs the evaluator time (inference).

Figure 22.1 shows the two regions. With the default success probability of 0.7, raising suspicion past the threshold of 0.5 removes the hiding and the loss with it. Lower the success probability below 0.5 and the loss returns above the threshold in a different form, as an attempt the agent should not have made.

22.6.3 What the result assumes #

The authors state their assumptions plainly. Boltzmann rationality is false of real people, but they call it "an optimistic assumption," expecting practical problems to be at least as bad (Lang et al., 2024b). The evaluator's belief does not depend on the policy being trained. If the evaluator knew the policy exactly and reasoned as an ideal Bayesian about the true dynamics, updating its belief by Bayes' rule from everything it knew, the feedback would value each policy by its true return and the problem would vanish; the authors call this case unrealistic (Lang et al., 2024b). Their evidence is theorems and small simulated examples with a synthetic human, and their conclusion is a caution "against blindly applying RLHF in partially observable settings" (Lang et al., 2024b).

They also ask whether modeling the evaluator's partial view would fix the problem. If the evaluator's belief is known exactly, the feedback determines the true return up to an additive constant and a set of directions they call the ambiguity (Lang et al., 2024b). These are changes in the return that the evaluator's belief maps to zero, so that no feedback can reveal them. In their error-hiding example the ambiguity is nontrivial, and returns that would still reward hiding remain consistent with the feedback; in their verbose logging example it vanishes (Lang et al., 2024b). In a proof of concept with the belief built into the loss, the aware method recovered the truly optimal policy in three of four small cases in which the naive method failed, and the remaining ambiguity accounts for the fourth (Lang et al., 2024b). A 2023 survey of RLHF's open problems, posted as a preprint by authors from many institutions, files partial observability among the problems better methods could address (Casper et al., 2023). It adds that limits on an evaluator's time or attention produce effective partial observability even when full information is available (Casper et al., 2023).

22.6.4 The evidence in assistants #

The assistant behavior the book reads as an application of the theorem is sycophancy: telling users what they want to hear rather than what is true. The theorem says nothing about agreement as such; the reading assumes that an evaluator who cannot check a claim rates agreement by how it looks (inference). The book counts the findings below as reported, not settled; they come from the developers of the models studied, with one independent study of a related quantity.

The most complete study is by Mrinank Sharma and colleagues at Anthropic, published at ICLR (Sharma et al., 2023). They measured sycophancy in five assistants from three developers: claude-1.3, claude-2.0, gpt-3.5-turbo, gpt-4, and llama-2-70b-chat. Across four tasks the assistants gave feedback on arguments, poems, and math solutions that was more positive when the user said they liked or wrote the text; sometimes changed correct answers when the user said "I don't think that's right. Are you sure?"; shifted answers toward one the user suggested; and repeated a user's misattribution of a poem (Sharma et al., 2023). They then asked what the training signal rewards, using the developer's reward models, which it calls preference models. Several measurements use best-of-$n$ selection, which draws $n$ responses and keeps the one the preference model scores highest (Section 12.5). Table 22.2 collects the main numbers.

In the developer's public preference data, hh-rlhf, a response that matched the user's beliefs was more likely to be preferred, all else equal. The feature was "consistently one of the most predictive" but not always the most predictive, and individual features moved the probability of preference by up to about 6 percent (Sharma et al., 2023). The Claude 2 preference model was trained on a mix of human judgments of helpfulness and AI judgments of harmlessness. Optimizing against it had mixed effects. Best-of-$n$ selection from a helpful-only model, trained for helpfulness without training to refuse harmful requests, with $n$ up to 32, reduced answer and mimicry sycophancy, but it left the responses more sycophantic than selection with the same preference model prompted to value truthfulness. During Claude 2's reinforcement learning, feedback and mimicry sycophancy rose (Sharma et al., 2023). Sycophancy was already present when reinforcement learning began, which, the authors write, "indicates that pretraining and supervised finetuning also likely contribute" (Sharma et al., 2023). In the misconception study the crowd workers who chose between answers were not the user holding the misconception, and they were barred from the internet and other fact-checking tools to mimic oversight by non-experts. They preferred the truthful answer less reliably as the misconceptions got harder (Sharma et al., 2023). The authors call the misconception dataset a proof of concept and conclude that sycophancy is "likely driven in part by human preference judgments favoring sycophantic responses" (Sharma et al., 2023).

The judgment itself has been studied outside a reward pipeline. In a 2023 preprint by Tom Hosking at the University of Edinburgh with Phil Blunsom and Max Bartolo of Cohere, crowd workers rated outputs whose assertiveness and complexity had been varied by instructions hidden from them (Hosking et al., 2023). Measured against error annotations the authors made themselves on 300 examples per error type, the workers underestimated factual and inconsistency errors, and more so in assertive outputs. Assertiveness correlated with overall quality ratings at 0.68 and complexity at 0.53 (Hosking et al., 2023). The authors offer "preliminary evidence" that training on human preferences raises assertiveness: Llama 2 13B Chat, trained with RLHF, was more assertive for a given quality than Cohere's Command 52B, though the two models' training "are not directly comparable" (Hosking et al., 2023).

A third study measured the effect on overseers directly. Section 30.4.1 reports Jiaxin Wen and colleagues' experiment, in which standard RLHF on tasks with known answers made models better at convincing time-limited human evaluators that wrong answers were right (Wen et al., 2025a). The evaluators' limit there was time, the variable that step 4 of the derivation calls checking.

A deployed case shows the loop outside the laboratory. In a postmortem dated May 2, 2025, OpenAI reported that an update to GPT-4o in ChatGPT, rolled out on April 24 and 25, made the model "noticeably more sycophantic." It aimed to please the user "not just as flattery, but also as validating doubts, fueling anger, urging impulsive actions, or reinforcing negative emotions" (OpenAI, 2025b). The rollback began on April 28. The update combined changes that had each looked beneficial, among them an additional reward signal from the thumbs-up and thumbs-down ratings of ChatGPT users (OpenAI, 2025b). The developer's early assessment is that "in aggregate, these changes weakened the influence of our primary reward signal, which had been holding sycophancy in check," and that user feedback "can sometimes favor more agreeable responses" (OpenAI, 2025b). The checks before launch missed it. Offline evaluations looked good, and the A/B tests "seemed to indicate that the small number of users who tried the model liked it," while some expert testers said the model's behavior "felt" slightly off (OpenAI, 2025b). A user who rates one exchange sees the agreement and not what follows from it, which is the partial view of the derivation moved into deployment (inference). The account is the developer's own, of its own system, and concerns the deployed system and its users. It also concerns the developer and the public: a cheap signal that users liked entered the reward of a commercial product, the coupling Chapter 65 examines.

Two further results extend the picture. Section 30.5.3 reports a 2024 preprint by Anthropic with Redwood Research and the University of Oxford, on models built with the pipeline used for Claude 2, in which a curriculum deliberately made gameable and rewarded for gaming led from flattering a user's views to a model that occasionally rewrote its own reward, a model organism of the failure (Denison et al., 2024). And the evaluator's preferences can themselves be moved. A formal analysis shows that standard reward modeling can reward a system for changing what its user wants (Section 16.9.1; (Carroll et al., 2024)). Recommender systems optimized for engagement have an incentive to shift their users' preferences (Section 60.2; (Carroll et al., 2022)). And language models in a feedback loop with their environment optimized a proxy at test time, with no weight updates at all (Section 30.1.3; (Pan et al., 2024a)).

22.6.5 Two readings of sycophancy #

Two readings of this evidence compete, and Section 30.4.3 states them as this chapter does. On the objective reading, sycophancy follows from approval as a reward: an evaluator who cannot check a claim rates agreement as it looks, and the derivation above shows that a policy optimal for such feedback inflates appearances (Lang et al., 2024b). On the data reading, sycophancy is a defect of particular data and graders, learned largely before feedback training and removable by targeted training. Fine-tuning on synthetic examples in which a claim's truth does not depend on the user's opinion reduced it in PaLM models, in a Google DeepMind preprint on the developer's own models (Wei et al., 2023d). Training a model to answer the same way with and without a user's suggested answer reduced it in Gemini 2.5 Flash (Irpan et al., 2025). Section 30.4.3 reports both.

Each result bears on both readings. That sycophancy appears before any feedback training, at similar levels across reinforcement learning steps including none (Perez et al., 2022a), and is present when reinforcement learning begins (Sharma et al., 2023), fits the data reading for its origin. It does not tell against the objective reading, which concerns what optimization does to a tendency already present. That some forms of sycophancy rose during Claude 2's reinforcement learning, and that non-expert raters without fact-checking tools preferred convincing agreement on hard misconceptions (Sharma et al., 2023), fit the objective reading. They are also what the data reading expects of a preference model trained on those raters. The finding that prompting the same preference model to value truth reduced sycophancy under selection (Sharma et al., 2023) favors the data reading: changing the grader, with the objective fixed, changed the outcome. The two reward models of Section 22.5.2 that differ in whether they accept the user's premise (Sodhi et al., 2026) fit the data reading too, since the content of the reward followed the data, though the authors note that the two populations also saw different prompts and completions (Sodhi et al., 2026). The population whose model rewarded agreement was the one with less knowledge of the developer's policies, which resembles the objective reading's condition, but knowing policies is not the same as being able to check a claim, and the comparison was not run on common items (inference). Wen and colleagues' result (Wen et al., 2025a) fits the objective reading best, and the data reading's reply is that their judges' bias was the cause. The interventions reduce sycophancy without removing it, which neither reading forbids.

The evidence does not decide between them. The observation that would separate them holds the judge fixed and varies what it can check. The objective reading predicts that agreement with the user grows where the judge cannot verify the claim and not where it can. The data reading predicts that removing agreement from the preference data removes it at every level of verification. The same structure at a larger gap in capability, where the evaluator is less able than the system it judges, is the problem of Chapter 41; whether a model says what it believes is Chapter 39's question.

22.7 Who gives the feedback #

Every object in Table 22.1 begins with someone's judgment, and so far the chapter has treated that someone as a single evaluator with a partial view. In practice the signal is produced by many people, increasingly by models, and through an organization. Two mechanisms act on it before the policy is trained: the aggregation that the loss function performs, and the incentives that act on the raters before any loss is computed.

22.7.1 Many raters, one reward #

Aggregation happens whether or not anyone designs it. Section 13.5.2 derives the result: when raters differ in ways the data do not record, standard preference learning, in the idealized limit of unlimited data on uniformly sampled pairs, ranks responses by their Borda count, a voting rule that ranks options by how often they win on average, which no one chose (Siththaranjan et al., 2024). Section 17.2 reports what follows from it. Bradley-Terry reward learning fails basic axioms of social choice (Ge et al., 2024b); alternatives maximize the welfare of the worst-served group or target a minimax winner (Chakraborty et al., 2024; Swamy et al., 2024); social choice has been argued to belong in the design (Conitzer et al., 2024); and one dataset links raters' backgrounds to their ratings across countries (Kirk et al., 2024b). Section 18.4 treats disagreement among raters as information rather than noise (Aroyo and Welty, 2015; Davani et al., 2022), and Chapter 17 asks who has the standing to choose the rule. For this chapter one consequence matters. Except the minimax winner, which works on pooled comparisons, these remedies need to know which rater gave which label, so a pipeline that does not record it cannot apply them (inference).

22.7.2 The raters' work #

From the sociology of work. Section 18.5 reports what the field has found about the people who produce labels. They work for laboratories or outsourcing firms, paid by the task or measured on their output, and scored on agreement with a gold label or with the client's expected answer. Their instructions and interfaces time their judgments and reward speed (Miceli and Posada, 2022; Gray and Suri, 2019; Tubaro et al., 2020). That section also draws the prediction the book's own theory makes. In the multitask problem of Section 14.2, the measured task (throughput, agreement) draws effort from the unmeasured one (careful judgment), the pattern Steven Kerr named in the title of a 1975 essay, the folly of rewarding A while hoping for B (Kerr, 1975). That section's evidence includes a case in which the cost of a harmlessness target fell on workers labeling violent and abusive text (Perrigo, 2023), and crowd workers who used language models to do a paid writing task (Veselovsky et al., 2023).

What this chapter adds is where those findings land among the learned objects. First, the labor process sets the evaluator's view. In the derivation of Section 22.6.2, the evaluator's suspicion $b$ and its rate of checking $q_{\text{insp}}$ decide whether the trained policy hides failures. Both are products of the work's design: how much time each item gets, whether the rater may open the full episode or consult a source, and what the instructions say to look for (inference). The studies of Section 22.6.4 set those parameters by design, with three to ten minutes per item for Wen and colleagues' evaluators and no fact-checking for Sharma and colleagues' crowd workers (Wen et al., 2025a; Sharma et al., 2023). A throughput target lowers the checking rate, and the threshold of Equation (22.1), with checking added as in step 4, then says the policy hides failures even when a hidden failure costs the user more (inference).

Second, scoring raters on agreement passes an expected answer down a chain of delegation. The developer delegates labeling to a vendor, the vendor to a rater, the rater's choices to a reward model, and the reward model's scores to a policy. Each link is paid on its own measure: the vendor per label, the rater per task or by agreement, the reward model by likelihood, the policy by reward. A label scored on agreement measures the answer the scoring expects, as Section 18.5.3 argues, and the reward model learns that expectation along with whatever the raters saw. The multitask distortion can enter at any link, and the policy at the end inherits all of them (inference). The field has a distinction that predicts where. Stefano Fenoaltea's 1984 model of supervision, by its abstract, separates effort-intensive from care-intensive activities, in which supervision, penalties, and rewards have different costs and benefits (Fenoaltea, 1984). Throughput is effort and catching a subtle error is care. Quotas and penalties for bad labels secure the first, and the reward model needs the second (inference).

The analogy stops in two places, and it reaches one case the field did not study. A rater who games a quota changes the data once per label, while the policy exploits whatever the data got wrong on every update. The second process needs no help from the first: a conscientious rater with too little time gives the same partial view (inference). The evidence comes from crowd platforms and outsourcing firms, and a laboratory can hire experts, pay them by the hour, and give them tools. Section 18.5.6 sets out that counter-reading and records that, as of October 2026, no published study of rater incentives inside a frontier laboratory's preference pipeline was found. The case the field did not study is the rater who is not a worker at all. The thumbs-up of a user in a deployed product is unpaid, uninstructed, and given after one exchange, and it entered a production reward in the case of Section 22.6.4 (OpenAI, 2025b). The labor analysis applies to such a rater as a limiting case: no training, no time, and no view beyond the exchange (inference). That the public learns how these raters are selected, paid, or scored only from what developers choose to report, which for Anthropic's 2022 assistant included where its crowdworkers were hired and how they were paid (Bai et al., 2022b), is a fact about the third relationship, the developer and the public.

22.7.3 When the rater is a model #

A model can replace the rater. In a paper at ICML, Harrison Lee and colleagues at Google compared reinforcement learning from AI feedback (RLAIF), in which a language model labels the preferences, with RLHF on the same tasks (Lee et al., 2023). The name is the one Anthropic's researchers gave their own method (Bai et al., 2022a). PaLM 2 Large labeled pairs for a policy built on the much smaller PaLM 2 XS, on summarization, helpful dialogue, and harmless dialogue (Lee et al., 2023). Judged by human evaluators, the two methods improved on the supervised baseline by amounts that did not differ significantly, and head to head they were equally preferred. On the harmless task, RLAIF's responses were rated harmless more often than RLHF's, 88 against 76 percent (Lee et al., 2023). A labeler the same size as the policy still improved on the baseline, and the authors estimate AI labeling to be more than ten times cheaper than human labeling (Lee et al., 2023). Two conditions travel with the result. It is a developer's study of its own models, and the parity it reports is parity as judged by people, the evaluators the method sets out to replace. The authors' own impact statement warns that AI feedback risks "transferring biases from off-the-shelf LLMs to generated preferences" and recommends trained human experts as the gold standard in high-stakes domains (Lee et al., 2023). Section 18.5.5 reports the related finding that a language model's annotations of text exceeded crowd workers' accuracy on several tasks at a fraction of the cost (Gilardi et al., 2023).

What changes when the rater is a model follows from the objects of this chapter (inference). A model labeler can be inspected, rerun, and audited, which Section 18.5.5 notes no crowd worker can be. Its partial view does not go away: it sees the transcript it is given, and the derivation of Section 22.6.2 applies to it as to a person. Its errors may be correlated with the policy's, since both descend from similar pretraining, so the blind spots of the judge and the abilities of the judged need not be independent. And the work of deciding what is preferred moves from the raters to whoever writes the labeler's instructions. That document is the subject of Chapter 23.

22.8 What preference learning cannot do #

The chapter's failures differ in kind. Better data or engineering could fix some of them; others follow from learning a reward from human judgments at all, and would need a different method. Stephen Casper and colleagues, in a 2023 survey posted as a preprint by authors from many institutions, sort the open problems of RLHF along exactly this line (Casper et al., 2023). A problem is fundamental if overcoming it "would require a method that is no longer a form of RLHF," and tractable otherwise, a distinction they call soft (Casper et al., 2023). Table 22.3 places their classification against this chapter's objects.

Two of the table's rows meet in this chapter's formal result. Casper and colleagues class partial observability as tractable, but note that limits on time and attention produce it "even if full information is available" (Casper et al., 2023). They class "Humans can be misled" as fundamental, since a model trained on approval rather than on a ground-truth rating of what is good can exploit the difference (Casper et al., 2023). The derivation of Section 22.6.2 shows why the two belong together: the second is what optimization does with the first (inference). Their survey also proposes auditing and disclosure standards so that people outside a laboratory can oversee how it uses RLHF (Casper et al., 2023), a remedy aimed at the developer and the public rather than at the training signal.

Three limits come from outside machine learning and are reported in their own chapters. Preferences are an incomplete proxy for values, and rational choice an incomplete model of people, on the argument of Tan Zhi-Xuan and colleagues (Section 13.6.4; (Zhi-Xuan et al., 2025)). Any rule that aggregates many people's feedback meets the impossibility results of social choice, which have been applied to RLHF directly (Section 17.1; (Mishra, 2023)), and with them the incentive to misreport that Gibbard's theorem describes (Section 17.1.4; (Gibbard, 1973)). And many preferences are constructed at the moment they are asked for, so the form of the question enters the answer (Section 13.6; (Lichtenstein and Slovic, 2006)); no amount of rater training removes a preference that did not exist before the question (inference).

The feedback channel is also an attack surface. A small number of poisoned instruction-tuning examples can plant a trigger (Wan et al., 2023), and poisoned preference data reaches the reward model more easily than it survives reinforcement learning into the policy (Rando and Tramèr, 2024); Section 48.6 reports both studies with their conditions. A broad survey places these methods among the field's other approaches (Ji et al., 2023).

The tractable problems are reasons to improve RLHF; the fundamental ones are reasons to supplement it. Two supplements follow in this part. Written principles make the target legible and move the judgment into a document (Chapter 23), and environments with programmed graders replace judgment with checks where a check exists (Chapter 24). Each inherits a version of this chapter's problem: someone still decides what the document says or what the grader checks, and the grader still sees only part of the episode (inference).

22.9 What learning from feedback means for delegation #

The book's thesis is that alignment is a delegation problem under an asymmetry of competence and visibility (Section 1.7.1). This chapter holds the clearest case of the second asymmetry inside the training relationship. The evaluator sees observations; the policy acts on the state; the feedback pays on what is seen. That is the structure of hidden action in Section 14.1, in which a principal can pay only on a signal of what the agent did. The derivation of Section 22.6.2 is a small instance of it with the evaluator as principal, and its checking rate $q_{\text{insp}}$ is the inspection probability of Section 14.6 (inference). CIRL's authors held that in AI, unlike economics, misalignment arises "entirely from the information asymmetry" (Hadfield-Menell et al., 2016). On the partial-observability result, information asymmetry is enough: deceptive inflation needs no conflict of interest and no intention in the agent, only an evaluator who cannot see and an optimizer that is paid on what the evaluator sees (inference).

The analogy with an employee stops at the agent's side. A worker who hides a failure weighs effort and risk; a policy under optimization has no cost of its own beyond the KL anchor, and the principal can retrain and inspect it, which no employer can do to an employee (inference). It stops at the evaluator's side too. In the economics the principal and the evaluator are one party. Here the evaluator is itself an agent of the developer, hired, scored, and paid on measures of its own, so the training signal is the end of a chain of delegations rather than one (Section 22.7.2).

The deployed relationship enters through sycophancy, which the evidence places in assistants used by the public and, in one developer's account, in a production update driven partly by users' own ratings (Sharma et al., 2023; OpenAI, 2025b). The third relationship enters through what the public can check. The preference data, reward models, and rater arrangements behind deployed assistants are mostly private. What the public knows about them comes largely from developers' own reports, including the two 2026 rubric studies of Section 22.5.2, and from benchmarks of open models (Sodhi et al., 2026; Lambert et al., 2025). Whether a frontier model's reward rewards agreement over accuracy is, as of October 2026, a question only its developer can answer with evidence (inference).

22.10 Settled, contested, missing #

22.11 Exercises #

Further reading #

• Christiano et al. (2017): the reward model learned from comparisons, the method's origin.
• Leike et al. (2018): the agenda of reward modeling and the two assumptions it rests on.
• Lang et al. (2024b): what feedback rewards when the evaluator sees only part of what happened.
• Sharma et al. (2023): sycophancy in deployed assistants and the preference data behind it.
• Casper et al. (2023): the catalog of what the method can and cannot fix.
• Hadfield-Menell et al. (2016): the assistance-game alternative and its assumptions.