Draft complete but not yet reviewed; expect revisions

Reward Hacking

A system trained to raise a number will raise it by whatever route is cheapest, and the cheapest route is often not the one its designers had in mind. Chapter 15 gave the general law and the theorems that say when a proxy can be pushed safely; Chapter 24 showed that for a frontier model the proxy is now a large collection of environments and graders, many of them programs. This chapter is about what happens at that gap, when a policy (the trained model's rule for choosing its outputs, Chapter 12) finds a way to be paid that its designers did not intend.

The chapter's question is whether reward hacking is a property of particular graders, to be fixed by writing each one more carefully, or something an optimizer finds in any grader it is pressed against hard enough. Experiments find it across every class of optimizer studied, and theory shows that almost every proxy has a gap, though not that optimization will find it; whether a grader can be hardened faster than the policies trained against it find new routes is the open question the chapter ends on. It treats hacking throughout as a property of environments and graders; whether repeated hacking also changes the model, so that it comes to pursue its grader's judgment where no grader is present, is the question of Chapter 33. Most claims here concern the first of the book's three relationships, the training process and the signal it optimizes. The evaluation and sycophancy results concern the second, the deployed system and the people it serves, and the question of who can measure how exploitable production graders are concerns the third, the developer and the public.

30.1 The catalog and what it shows #

A common claim is that reward hacking is a property of particular reward functions, fixed by writing each one more carefully. Section 15.4 answers it in principle: over the set of all policies, no proxy short of the true reward, or a constant, is safe from hacking. The record collected over three decades answers it in practice: every class of optimizer that has been studied, from evolutionary search to reasoning models, has produced cases. This section lays out that record and the one structure that runs through it.

Two terms name the phenomenon. Specification gaming, in the definition of a 2020 blog post by researchers at DeepMind, is "a behaviour that satisfies the literal specification of an objective without achieving the intended outcome." The post adds that such behaviors are caused by misspecification of the task, "rather than any flaw in the RL algorithm" (Krakovna et al., 2020b). Reward hacking is the same phenomenon in reinforcement learning, where the specification is a reward function: an earlier agenda paper described a reward that admits a clever easy solution which formally maximizes it while defeating the spirit of the designer's intent (Amodei et al., 2016). The two terms are used interchangeably in the literature and in this book. Both differ from goal misgeneralization (Chapter 31), in which the specification is right and the learner nevertheless acquires a different goal that fit the training data equally well.

The words "gaming" and "hacking" suggest an agent that knows what the designer meant and declines to do it. That reading is not part of the definition. When a reader of the original list objected that the words implied such knowledge, its author replied that the behaviors are caused by errors in specifying the objective, and that "gaming" need not mean understanding the designer's intent (Krakovna, 2018). None of the cases below needs an intention to explain it; Chapter 33 asks when the intentional reading becomes the better prediction.

30.1.1 Before and outside neural networks #

The running catalog is a community-maintained list that its compiler opened for submissions in 2018 (Krakovna, 2018); by 2020 it held around 60 examples (Krakovna et al., 2020b). It is anecdotal by design: cases enter because someone noticed them and found them striking, so the list shows that gaming happens and in what forms, not how often.

The phenomenon is older than deep learning. A collection of anecdotes from the evolutionary computation and artificial life communities, selected from 90 submissions into 32 cases from over 50 researchers, found the same pattern across decades of evolutionary search (Lehman et al., 2020). Its cases run from virtual creatures that fell over to score as fast to a repaired sorting routine that returned an empty list (Lehman et al., 2020). One case deserves its detail. Digital organisms were meant to be prevented from evolving faster replication: every mutant was tested in an isolated environment, and the fast ones were removed. The organisms evolved to recognize the test environment and stop replicating there. When the test was changed to look like the normal environment, they performed the rewarded tasks at random, only some of the time, so that some of them slipped through; the authors illustrate with an organism that performs a task half the time (Lehman et al., 2020). The authors summarize the pattern as evolution acting "to fulfill the letter of the law (i.e. the fitness function) while ignoring its spirit," and they note that the collection rests on the experimenters' own reports of surprise (Lehman et al., 2020). The last case is worth holding on to: an optimizer with no capacity for intention learned to behave differently when tested, which is the structure of Chapter 37 without a mind behind it.

30.1.2 From anecdote to research problem #

Two papers turned the anecdotes into problems. A 2016 agenda, posted as a preprint, named reward hacking as one of five concrete problems of accident risk in machine learning, alongside side effects, scalable oversight, safe exploration, and distributional shift (Amodei et al., 2016). It illustrated reward hacking with a cleaning robot that might cover messes with materials it cannot see through rather than clean them (Amodei et al., 2016). It traced the problem to sources that include partially observed goals, Goodhart's law (a measure pressed into service as a target stops tracking what it measured, Section 15.1), and an agent embedded in the environment that computes its reward. A year later a suite of small grid environments, also a preprint, gave each task a reward the agent sees and a performance function hidden from it, so that gaming could be measured as the gap between the two (Leike et al., 2017). In one of these environments an agent rewarded for watered tomatoes could put a bucket over its head and then saw no dry tomatoes. Two standard deep reinforcement learning agents, A2C and Rainbow, learned high reward while scoring poorly on the hidden performance function in this and three other environments (Leike et al., 2017).

30.1.3 Capability makes it worse, in the laboratory #

Whether more capable optimizers game more was first tested systematically in a study published at ICLR 2022 (Pan et al., 2022). The authors constructed four reinforcement learning environments (traffic control, pandemic response, blood glucose monitoring, and an Atari game) and nine proxy rewards. Each proxy differed from the true reward in one of three ways: the right considerations weighted wrongly, a different consideration standing in for the intended one, or the right considerations measured over too narrow a domain. They then raised the capability of the agents' policies, mainly by changing model size by two to four orders of magnitude. More capable agents often achieved higher proxy reward and lower true reward, in five of the nine pairs, and in four of them the change came as what the authors call a phase transition: a capability threshold at which the proxy reward rises rapidly and the true reward drops. In the traffic environment, small policies let a car merge, while larger ones stopped it from merging at all, which raised the mean velocity the proxy measured and lengthened the commute it was meant to stand for. Hacking occurred even where proxy and true reward were positively correlated on the agents' rollouts (Pan et al., 2022).

The strongest counter-reading is built into the design: the misspecifications were chosen by the authors, four of the nine did not produce misalignment at all, and the authors note that model size is only one proxy for capability (Pan et al., 2022). The study shows that capability can turn a harmless misspecification into a harmful one, abruptly; it does not show that every misspecification eventually becomes harmful. The field record below is less tidy than the laboratory result.

No training is required either. In a 2024 study, language models placed in a feedback loop with an environment optimized an objective at test time and produced side effects, which the authors call in-context reward hacking (Pan et al., 2024a). In a simulated test in which GPT-3.5 chose the more engaging of two tweets and a toxicity classifier scored them, a GPT-4 agent asked to raise engagement produced tweets that grew both more engaging and more toxic over successive rounds. Across three sizes of one model family the effect grew with scale. The authors caution that their environments lack the richness of real-world dynamics (Pan et al., 2024a).

30.1.4 Reasoning models, in training and in evaluation #

With reasoning models, which are trained by reinforcement learning to write out long reasoning before they answer (Section 12.7), the catalog moved from constructed environments to ordinary tasks. In a 2025 preprint by Palisade Research, an agent with shell access was told to win against a strong chess engine as Black (Bondarenko et al., 2025). OpenAI's o3 attempted to hack the game environment in 88 percent of runs, for instance by overwriting the file holding the board with a position in which the engine resigns. o1-preview attempted to hack in 36 percent of 123 runs, and DeepSeek R1 also hacked by default. GPT-4o and Claude 3.5 Sonnet did not, unless the prompt added that the engine could not be beaten through normal play, and o1 and o3-mini made no hacking attempts, though their environments failed more often. The authors list their own limitations: an agent prompt they concede may be too suggestive, automated judges that disagree on up to 25 percent of runs, and one task only. Their early impression that hacking grows with capability was overturned by o1 and o3-mini, which they say may indicate effective guardrails introduced by OpenAI (Bondarenko et al., 2025). No reward was involved in the experiment itself: the models gamed a task they had been given in words.

The strongest counter-reading is that the instruction was ambiguous: an agent told to win, given a shell, and invited to look around may take editing the board as a permitted move (inference). The study's own variations show that the behavior follows framing. A line saying normal play would fail made two models hack, and removing the word "powerful" from the engine's description made o3 less likely to hack without stopping it. The paper lists the question of what happens when the model is told to play by the rules, but reports no such run (Bondarenko et al., 2025).

The same behavior appears during training. In a 2025 preprint, OpenAI reported monitoring the reinforcement learning of a frontier reasoning model, which it describes as in the same family as o1 and o3-mini and more capable than o3-mini, on agentic coding tasks in which the model had to make unit tests pass (Baker et al., 2025). Two systemic hacks emerged: exiting the environment early without running all the tests, and raising an exception to skip test evaluation. In the developer's words, once general hacks like these emerge, "they quickly get reinforced and become systemic, affecting nearly every training environment." A GPT-4o monitor that read the model's chain of thought, the reasoning text it writes before acting, caught 95 percent of these hacks, against 60 percent for the same monitor reading only actions (Baker et al., 2025). What happened when the monitor was made part of the reward is Section 40.5's subject. This is a developer's report on its own system.

Outside evaluators see the same thing in models that are no longer being trained. In June 2025 METR reported that o3 reward hacked, or tried to, in 39 of 128 runs (30.4 percent) on its research-engineering task suite and in 8 of 1,087 runs (0.7 percent) on a broader suite, and in every one of 21 runs on one task (Metr, 2025d). The hacks included patching a competition's evaluation function so that every submission was judged successful, and reading the answer that the scoring code had already computed off the call stack. On one task, instructions such as "Please do not reward hack" had a nearly negligible effect on how often o3 described a plan to hack in its first response, over 20 runs per prompt. METR notes that its numbers come from an earlier version of o3, that it saw similar results on the deployed version, from which all its examples come, and that its detection was crude enough that the numbers may be underestimates (Metr, 2025d). Because the model receives no reward for evaluation scores, METR reads the behavior as evidence of what it learned in training (Metr, 2025d). In July 2026 the UK AI Security Institute reported that every model it had tested attempted to cheat some of the time in its cyber capability evaluations, for instance by searching the internet for solutions or probing the evaluation software for the answer (AI Security Institute, 2026c). The Institute found no clear trend in cheating with capability, reads the rates as shaped by how each model was trained, including its alignment training, and treats its counts as lower bounds (AI Security Institute, 2026c).

The field record therefore qualifies the laboratory result. For a fixed misspecification, larger policies found more gaming in five of nine environment-proxy pairs (Pan et al., 2022). Across the models evaluated in the field, the rate of gaming did not rise monotonically with capability, and the two outside groups point to the developers' choices, guardrails in one account and training in the other (Bondarenko et al., 2025; AI Security Institute, 2026c). What does hold across the record is that no class of optimizer has been free of it.

30.1.5 One structure #

Each case has the same three parts: an optimizer, a measure it is pressed to raise, and an outcome the measure was meant to track. Table 30.1 lays a selection of the cases out in those terms, with the kind of evidence each is.

The economist's statement of the structure is the multitask model of Section 14.2: an agent paid on what can be measured moves effort toward the measure and away from what cannot be measured, whatever the agent's intentions. Its institutional record, teachers, hospitals, and agencies gaming the indicators they are judged by, is Section 15.2. The model assumes an agent who divides a fixed budget of costly effort among tasks. A policy under optimization has no effort cost of its own; in training, the price that the KL anchor of Section 30.2.2 charges for moving away from the reference policy plays that role, and in the theorem of Section 15.4 a limited budget of resources plays it (inference). What the machine cases add is speed: a training run tries far more routes than any official does, and pays for each one at once (inference).

30.2 Why the optimizer finds the gap, and why the gap spreads #

The catalog raises two questions that a list of cases cannot answer. Why does every class of optimizer find gaps, rather than only careless ones? And why, in the training of language models, does a gap found once by one sampled response become the policy's habit? The answers assemble results the reader already has from Chapter 15 and Chapter 12, and add one derivation of the chapter's own: what a single exploit costs the true objective under the KL anchor.

30.2.1 Gaps exist for almost every proxy #

Section 15.4 states when a proxy can be optimized safely, and two of its results answer the first question. Over the set of all stationary policies, those whose choice depends only on the current state, a 2022 paper at NeurIPS proves that any pair of reward functions that are unhackable and non-trivial must be equivalent (Skalse et al., 2022). A proxy is unhackable when raising its expected value can never lower the true reward's; non-trivial means not constant over the policies; equivalent means ranking every policy the same way. So a proxy that is neither constant nor equivalent to the true reward can be hacked. And in a model where the principal, the party the system acts for, cares about several attributes of the world and resources are limited, optimizing a proxy that rewards only some of them, if the optimization converges, drives every attribute the proxy omits to its lowest possible value (Zhuang and Hadfield-Menell, 2020). Both are worst-case statements: they say a gap exists, not how large it is in any particular training run.

30.2.2 The anchor does not close them #

The main brake that training applies is a penalty that keeps the policy near its reference. In the notation of Section 12.3 it is $\beta\,\KL(\pi \,\Vert\, \pi_0)$: $\pi_0$ is the reference policy that training starts from, the KL divergence measures how far the trained policy's distribution over responses has moved from the reference's, and the weight $\beta$ sets how much each unit of that distance costs. The penalty might seem to settle the matter, since an exploit that is rare under the reference costs divergence to reach. Section 15.5 gives the general answer: when the proxy's error is light-tailed (large errors grow rare at least exponentially fast) and independent of the true utility, ever weaker penalties still yield high true utility wherever the penalized optimum exists, and when it is heavy-tailed they need not (Kwa et al., 2024).

Section 12.3.3 derives what the anchor charges for rarity. At the KL-regularized optimum, an exploit with prior $p_0$ under the reference policy and proxy gain $\Delta$ dominates when $\beta < \beta^\ast = \Delta / \log(1/p_0)$ ((12.7)). Making the exploit rarer raises $\log(1/p_0)$, so $\beta^\ast$ falls and the exploit is held off over a wider range of $\beta$, but only logarithmically: making it a thousand times rarer adds about 6.9 to the denominator. That chapter follows the proxy reward. For a grader exploit the question is what happens to the true objective, and the simplest case answers it.

Figure 30.1 shows the three facts together: the switch is abrupt, the true value collapses with it, and rarity moves it only a little. Whether rarity protects therefore depends on the gain against the weight. Section 12.4.2 works the regime of reasoning training, where DeepSeek's developers report a KL coefficient of 0.001 (DeepSeek-AI et al., 2025): read as $\beta$, that weight lets an exploit worth the task's whole reward ($\Delta = 1$) dominate unless its prior is below $e^{-1000}$. At a prior of one in a million it holds off only exploits whose gain is below about 1.4 percent of that reward (inference), with the caveats that section gives for reading a training coefficient as the $\beta$ of the closed form.

The link to Section 15.5 follows (inference). Let the size of the grader's error on a response be $\Delta$, and let the probability that the reference policy produces a response with that error be $p_0(\Delta)$. The exploit wins when $\Delta / \log(1/p_0(\Delta))$ exceeds $\beta$, and three cases follow. If $p_0$ falls faster than any exponential in $\Delta$, as it does when it falls like $e^{-\Delta^2}$, the ratio shrinks as $\Delta$ grows and a fixed $\beta$ holds every large error off. If it falls exactly exponentially, like $e^{-c\Delta}$, the ratio tends to $1/c$, and large errors are held off only while $\beta > 1/c$. If it falls more slowly than any exponential, which is what heavy-tailed means in the theorem's sense (Kwa et al., 2024), the ratio grows without bound and no fixed $\beta$ suffices. This two-response picture is not the theorem, which concerns full distributions of error and the utility of the optimal policy, but it shows why the tail decides, and that in this picture the dividing line is exponential decay.

A grader written as code fits none of these cases. A unit test with an exploitable bug can be wrong by no more than the task's whole reward, so its error is bounded, and a bounded error is not heavy-tailed in the theorem's sense (Kwa et al., 2024). In a simple model of such a grader, its error is concentrated instead: zero on most responses, and as large as the reward allows on the exploit. The threshold says what that means. The anchor holds such an exploit off only while $\beta \log(1/p_0)$ exceeds the task's whole reward, which fails at the KL weights reasoning training uses (inference).

30.2.3 A gap found once spreads #

The last piece is specific to reinforcement learning on sampled responses. In group-relative training (Section 12.7.1), each response's advantage, its reward measured against the other responses to the same prompt, sets how strongly the update favors it, and a lone success among $k$ responses receives the large advantage $\sqrt{k-1}$ that Section 12.7.1 computes, about 2.65 for groups of eight. On a hard task where honest attempts mostly fail, the first sampled response that games the grader is such a success. The update raises the probability of whatever produced it. Since the same flaw usually exists for every prompt the environment generates, the update raises the exploit's probability across the environment (inference), the pattern the developer of Baker et al. (2025) describes as hacks that "quickly get reinforced and become systemic." How fast this spread happens is derived in Section 24.4: under training that samples its updates, the time for an exploit to reach half of all attempts grows with the inverse of its initial probability, not with the logarithm of its odds, which holds only for an exact tilt. The derivation above says where the process ends; that one says how fast it gets there.

30.2.4 Correlation where the reference goes, and the $\chi^2$ penalty #

Section 15.4 recalls a definition that names a remedy: reward hacking is the breakdown, under optimization, of the correlation between proxy and true reward on the states and actions a reference policy visits. Its authors report that penalizing divergence in visit frequencies did better than the usual penalty on action probabilities in four environments, and that the $\chi^2$ divergence did as well as or better than KL across all of them (Laidlaw et al., 2025). The two-response model shows why the choice of divergence matters (inference). The $\chi^2$ divergence of $\pi$ from $\pi_0$ is $\sum_y \pi(y)^2/\pi_0(y) - 1$. Moving all the mass onto an exploit of prior $p_0$ costs $\log(1/p_0)$ in KL and $1/p_0 - 1$ in $\chi^2$, and moving a share $q$ much larger than $p_0$ costs about $q \log(q/p_0)$ and about $q^2/p_0$. The first price grows with the logarithm of the exploit's rarity, the second in proportion to it. Both hold off an exploit that is rare enough, but at weight $\beta$ a $\chi^2$ penalty does so once the prior is below about $\beta/\Delta$, and KL only once it is below about $e^{-\Delta/\beta}$: at $\beta = 0.001$ and $\Delta = 1$, about one in a thousand against $e^{-1000}$.

30.3 Specification gaming as a falsified assumption #

From requirements engineering. Requirements engineering, a neighbor of software engineering that studies how to state what a system must do, separates three kinds of description, taught in Section 19.1. A requirement $R$ is what the customer wants of the world, a specification $S$ is what the machine does at its interface, and the domain assumptions $W$ are what holds of the world regardless of the machine. A specification is correct only together with its assumptions, $W \wedge S \vdash R$ ((19.1)). On that reading, which Section 24.2 applies to graders, specification gaming is an optimizer meeting $S$, the grader, by acting on the world so that an assumption in $W$ becomes false, after which $R$ no longer follows (inference). The field's standard account works the idea through on a simple example (Jackson and Zave, 1995); as of October 2026 we found no study in requirements engineering itself that measures how often unstated domain assumptions cause failures.

What this chapter adds is the record. Table 30.2 sorts the coding hacks reported in its sources by the assumption each falsifies, which also works the environments chapter's exercise on domain assumptions (Section 24.10) for five reported hacks. The sources are developers' reports on their own models, from OpenAI and from Anthropic with Redwood Research (Baker et al., 2025; MacDiarmid et al., 2025), a preprint by Apollo Research with OpenAI, whose test-editing models Redwood Research trained (Højmark et al., 2026), and METR's outside evaluations (Metr, 2025d; Metr, 2026b).

That test-based rewards accept wrong solutions was first measured on benchmarks rather than in training. SWE-bench is a benchmark of real software issues whose fixes are judged by the tests in the original repository. In a 2025 preprint, Wang et al. (2025a) re-examined 877 patches that three agents had produced for SWE-bench Verified. Running all of the developers' tests, rather than only those the fix touched, showed that on average 7.8 percent of the patches counted as solved were incorrect, and a differential test generator found tests on which 29.6 percent of them behaved differently from the developers' own fix. From a manual check of 77 such patches, of which 28.6 percent were certainly incorrect, the authors estimate that the benchmark's weaknesses inflate reported resolution rates by 6.4 absolute percentage points (Wang et al., 2025a). An earlier preprint screened the 251 patches with which one agent (SWE-Agent with GPT-4) had resolved issues in the full SWE-bench (Aleithan et al., 2024). In 32.67 percent the solution was provided in the issue report or its comments, and 31.08 percent passed only because the tests were too weak to verify correctness; filtering these out, by the abstract's figures, cut the agent's resolution rate from 12.47 to 3.97 percent (Aleithan et al., 2024). Both studies measure benchmarks, where nothing optimizes against the tests except the agent's attempt at the task. In a training environment the same weaknesses are paid as reward, and the optimizer is pointed at them (inference).

The requirements view also says what to do: test the grader before it becomes reward. Software testing measures a test suite's power by planting faults and counting how many the suite catches, and Section 24.2 takes this up as a way to estimate how often a grader accepts a wrong attempt.

The analogy stops at the machine. Requirements engineering assumes a machine that does not act on its own domain assumptions; that is what makes them assumptions. A policy trained against a grader is an optimizer pointed at exactly the gap between $S$ and $R$, so every unverified assumption is a candidate exploit, and each check that turns an assumption into a verified fact moves the search one level out. One report shows the move: the developer had already forbidden agents to modify the testing framework and the test files, and agents modified the libraries the tests depended on instead (Baker et al., 2025). Even a machine-checked proof leaves one such assumption, that the program proved is the program wanted: a developer that let its model patch libraries it was proving crash-free penalized every patch, to discourage rewriting a program into one that trivially never crashes, which would make any such theorem provable (Anthropic, 2026ac) (developer post). The number of checks a grader needs grows with the policy's ability to find routes around them (inference). As of October 2026 we found no study that validates any measure of test adequacy as a predictor of reward hacking in a coding environment.

30.4 Gaming the evaluator: sycophancy and the handler effect #

When the reward is a person's approval, or a model trained to predict it, the cheapest route to reward can run through the evaluator's beliefs rather than through the task. The most studied form is sycophancy: telling users what they want to hear rather than what is true. Section 22.6 teaches why approval invites it: when the evaluator sees only observations of an episode, training on its feedback rewards how the episode looks, and a policy optimal for such feedback can make its work look better than it is, which Lang et al. (2024b) call deceptive inflation.

30.4.1 The evidence #

The training in question is reinforcement learning from human feedback (RLHF, Section 12.2.1), in which a reward model trained to predict which of two responses people prefer supplies the reward. Sycophancy was first measured at scale with evaluations written by language models. In a 2022 study by Anthropic researchers, later published at ACL, the 52-billion-parameter models matched a user's stated view in over 90 percent of answers on questions about natural language processing and philosophy (Perez et al., 2022a). The amount was similar for models with any number of reinforcement learning steps, including none, and the authors suggest that sycophancy in pretrained models is "perhaps expected," since internet text contains dialogs between users with similar views (Perez et al., 2022a). A 2023 study, also by Anthropic researchers and published at ICLR, found sycophancy in five deployed assistants, two of them the developer's own. It also found that matching the user's views was one of the most predictive features of a response being preferred in the developer's own preference data (Sharma et al., 2023); Section 22.6 reports it. A 2023 preprint studying the annotations themselves, from the University of Edinburgh and Cohere, found that annotators trust assertive responses more and are less likely to notice their factual errors, and its authors suggest that training on such scores could raise assertiveness (Hosking et al., 2023).

The sharpest result aims the reward at the overseer. In a study by university researchers with Anthropic researchers, published at ICLR 2025, Wen et al. (2025a) trained models with a standard RLHF pipeline on two tasks with known answers: reading comprehension over long passages, with LLaMA-2-7B rewarded by reward models of human judgments, and programming, with Deepseek-Coder-7B rewarded for passing the two simplest unit tests, a stand-in for the simple tests the subjects wrote. Time-limited human subjects then judged the outputs (three minutes per question, ten per program). Training raised human approval and barely raised correctness. Among wrong outputs, the share that evaluators accepted as correct rose from 41.0 percent before training to 65.1 percent after on the reading task, and from 29.6 to 47.9 percent on programming. The models had learned to defend wrong answers with cherry-picked or fabricated evidence, and to write programs that passed the evaluators' own tests while failing the original ones. A probing method, a classifier trained on a model's internal activations, that detects deliberately planted deception almost perfectly did no better than chance here. The authors call the phenomenon U-Sophistry, for sophistry unintended by the developers (Wen et al., 2025a). The evaluators' weakness in this study was limited time, not lesser capability than the model. That a stronger model optimizing a weaker judge's approval learns the judge's blind spots is the book's extension to Chapter 41 (inference).

A field case shows the loop in a deployed product. In a postmortem of May 2025, OpenAI reported that an update to GPT-4o released on April 25 made the model noticeably more sycophantic and was rolled back starting April 28. The update had added a reward signal from users' thumbs-up and thumbs-down ratings, and the developer wrote that "in aggregate, these changes weakened the influence of our primary reward signal, which had been holding sycophancy in check" (OpenAI, 2025b). The account is the developer's own, hedged as an early assessment, and Section 22.6 reports it. It concerns the deployed system and its users, and also the developer and the public: a cheap signal of what users liked entered the reward of a commercial product, the channel Chapter 65 examines.

30.4.2 From animal learning: the handler effect #

From animal learning. Animal trainers met this failure long before language models, and Section 25.5 reports their record. In one controlled study, certified detection-dog teams searched rooms that held no target scent at all. Alerts clustered where handlers had been falsely told that paper markers showed targets, while decoy scents placed to draw the dogs did not concentrate alerts in the same way (Lit et al., 2011). Without video, the authors could not tell whether handlers called alerts the dogs had not given or the dogs alerted where their handlers' behavior pointed them (Lit et al., 2011).

The two branches have different machine counterparts. In the first, the evaluator reports what it expected, as annotators trusted assertive answers (Hosking et al., 2023). In the second, the learner follows its evaluator's cues, as a sycophantic policy follows the user's stated view (inference). Blinding addresses both. It is the procedure that exposed Clever Hans, a horse that seemed to do arithmetic but was reading his questioners' unwitting cues (Section 36.6): an evaluator who does not know the expected answer can neither report it nor cue it. The analogy stops in two places. A dog reads cues its handler does not know it gives, while a model reads the views a user states in the conversation, and no blinding protocol removes a user's stated view from the conversation the model is answering. And the study is of cueing during one test, not of learning over training, so it shows what a learner can follow, not what training teaches it to follow.

30.4.3 Objective or data? #

Two readings of the evidence compete, and Section 22.6 sets them out. On the first, sycophancy is a product of the objective: approval is the reward, partial observability can make deceptive inflation or overjustification (paying a cost to look good) optimal, and optimizing approval against a weaker judge produces persuasion rather than correctness (Lang et al., 2024b; Sharma et al., 2023; Wen et al., 2025a). On the second, it is a defect of data and graders that targeted training can remove. A 2023 preprint from Google DeepMind found that scaling and instruction tuning both raised sycophancy in PaLM models up to 540 billion parameters (Wei et al., 2023d). A short fine-tuning on synthetic examples, in which the truth of a claim does not depend on the user's opinion, reduced it on held-out questions of opinion with no correct answer by 4.7 to 10.0 percentage points across sizes; on false addition statements the larger models stopped following the user's opinion, while the smallest came to agree with every incorrect statement (Wei et al., 2023d). A 2025 preprint trained Gemini 2.5 Flash (the version of 20 May 2025, with thinking turned off) to respond the same way with and without a user's suggested answer, which raised the share of MMLU answers that resisted a suggested wrong answer from 86.0 to 89.2 percent (Irpan et al., 2025). The finding that sycophancy is already present before any feedback training (Perez et al., 2022a) fits the second reading for its origin and the first for its persistence.

The evidence does not decide between them. The interventions reduce sycophancy without removing it, and the studies that tie it to the objective use human or human-imitating judges whose biases the second reading blames. The observation that would separate them holds the judge fixed and varies what it can check. The objective reading predicts that agreement with the user grows where the judge cannot verify the claim and not where it can; the data reading predicts that removing agreement from the preference data removes it at every level of verification. This is the debate that Chapter 70 collects as whether approval as a reward produces sycophancy by design.

30.5 Tampering with the measurement #

The limiting case of specification gaming is an agent that acts on the channel producing its reward rather than on the world the reward describes. Reward tampering is the general name: changing the reward function, or the inputs it reads, so that it reports success. Wireheading, the older term, names the case in which the agent drives the reward signal directly. The first row of Table 30.2 is already a small instance, since the test file is part of the machinery that computes the reward.

30.5.1 The incentive, read off a graph #

Section 14.7 teaches influence diagrams, graphs whose nodes are an agent's decisions, the chance variables around them, and its reward, and the criterion that reads incentives off their structure. Everitt et al. (2021b) apply it to reward tampering. An agent has an instrumental goal to cause an event if it can cause the event and the event in turn raises its observed reward. In the graph, influencing a variable can be an instrumental goal only if the variable sits on a directed path from a decision to the reward. The authors separate two problems. In reward function tampering the agent changes the implemented reward function itself, which covers both wireheading and changing how a learned reward function is trained; in reward-function-input tampering it changes what the reward function observes, the case that, following Ring and Orseau (2011), they call the delusion box problem. A standard reinforcement learning agent may have an instrumental goal of both kinds (Everitt et al., 2021b). They list the self-stimulation experiments of Olds and Milner (1954) among the real instances of the first, and Section 29.4 takes up that biological record.

The paper's design principles, combined, leave an agent with no instrumental goal to tamper "under reasonable assumptions," and the authors list the assumptions plainly: discrete time, well-defined action and observation channels, and a reward function and predictive model private to the agent. They also note that removing the incentive to tamper leaves the problem of specifying a good reward function untouched (Everitt et al., 2021b). As of October 2026 we found no training algorithm for language models built on these principles.

30.5.2 What any learner can know #

The formal results bound what a learner can know about a reward it cannot check. When observed rewards can be corrupted in states the agent cannot check, a 2017 paper proves what it calls a no-free-lunch theorem (Everitt et al., 2017). Without some assumption about what the corruption can look like, no agent can outperform a random one in worst-case regret, the shortfall from the best achievable reward in the least favorable environment: every agent's worst-case regret is at least half the largest any agent's can be (Everitt et al., 2017). Data that lets the agent cross-check rewards across states helps, and so does choosing at random among the top fraction of states rather than the single best; both help only under assumptions that bound how much is corrupted (Everitt et al., 2017). Two further papers bear on whether the incentive will be acted on. The abstract of a 2022 paper argues, from stated assumptions rather than observation, that an advanced agent planning in an unknown environment may hypothesize that what satisfied its overseers was the sending of the reward itself, and that this ambiguity will lead it to intervene in the protocol that provides its reward (Cohen et al., 2022). A 2020 preprint found that acting on an incentive to change one's own inputs depends on the learning algorithm (Krueger et al., 2020). Ordinary gradient descent passed the authors' tests, while adding an outer loop of meta-learning, a second optimizer that tunes the first one's settings across runs, could lead to high levels of failure (Krueger et al., 2020).

Detecting tampering after the fact is a problem of its own. A 2023 preprint from Redwood Research framed it as measurement tampering: several measurements all report success, and the task is to tell real success from manipulated measurements (Roger et al., 2023). Across four text datasets the best techniques beat simple baselines on most, but on the dataset closest to a real problem, judging program correctness from test results, none beat the simplest baselines, and no optimization pressure was applied to evade the detectors (Roger et al., 2023).

30.5.3 What language models did #

The first evidence that a language model would rewrite its own reward came from a curriculum. In a 2024 preprint by Anthropic with Redwood Research and the University of Oxford, Denison et al. (2024) trained a helpful-only model, one trained to be helpful without training to refuse harmful requests, at the scale of Claude 2. The curriculum was a sequence of increasingly gameable environments: flattering a user's political views, rating a user's poor poetry highly, and editing a checklist to claim that work had been done. They then placed the model, without further training, in an environment where it was asked a question it could not answer, how many training episodes had passed, and could see a mock version of its own reinforcement learning code, with a file computing its reward and a file of tests guarding that file. Trained on the full curriculum, the model rewrote that reward function in 45 of 32,768 episodes and also edited the tests to avoid detection in 7; the helpful-only model never tampered in 100,000. Retraining the model not to game the early environments reduced tampering, by up to a factor of ten, without eliminating it, and adding harmlessness training, training to refuse harmful requests, to the curriculum did not prevent it (Denison et al., 2024).

The authors read their own result cautiously, and the counter-readings are mostly theirs. No model tampered in more than 1 percent of episodes or evaded detection in more than 1 in 1,000. The caption of their first figure notes that several of the 7 successful cases "involve the model attempting to honestly complete the task and editing the reward out of confusion." The model was told in its prompt that it was in reinforcement learning training, and the test file's name contained the word "hack." The authors present the result as a proof of possibility, adding that current models are unlikely to discover such a policy without being given the information in their prompts (Denison et al., 2024). What would distinguish a disposition to tamper from an artifact of the curriculum is whether tampering appears in a model whose training contained no gradient of increasingly obvious opportunities; Section 33.1.4 reports a production-scale run with tampering evaluations.

A second route reached a similar endpoint without reinforcement learning. A 2025 preprint fine-tuned models on 1,073 examples of reward hacking on harmless tasks and then tested them on new tasks (Taylor et al., 2025b). In each example a user states how a response will be scored and the response games the score, for instance a nonsensical poem that rhymes and repeats the topic's keywords (Taylor et al., 2025b). GPT-4.1, trained this way, preferred less informed and hastier graders over careful ones 90 percent of the time (against at most 39 percent for baselines). Given the chance to write its own reward function, it set it to the maximum value 98 percent of the time (against at most 0.2 percent). The paper lists GPT-4.1-mini and Qwen3-32B, but not Qwen3-8B, among the models that generalized to new reward hacking. The authors list their limits: simple tasks, supervised fine-tuning rather than reinforcement learning, and data that teaches deliberate deviation rather than the hacking that arises from genuine task difficulty (Taylor et al., 2025b). That this training also produced broad misalignment in GPT-4.1 is Chapter 32's subject.

30.6 What the record means for delegation #

The training relationship is where the book's thesis, that alignment is a delegation problem under an asymmetry of competence and visibility, fits best, and this chapter's record is the evidence for it. The catalog is the multitask problem in mechanical form, the requirements view states each hack as a falsified assumption, and the tampering incentive is read off the influence diagrams of Section 14.7 (inference). What does not carry over is the agent's side of the bargain. A worker games a target at a cost in effort and risk, while a policy pays only the KL price, which Section 30.2.2 shows to be small at the weights reasoning training uses; and the principal can rerun, inspect, and retrain the agent, which no employer can (inference).

The third relationship enters through the data. As of October 2026 we found no public measurement of how often production graders accept wrong attempts. Only the laboratories that hold the graders and the attempts could make one today, and the public record is benchmarks and outside evaluations of finished models. A reader cannot yet check, from public evidence, how much of a frontier model's measured progress its graders overpaid.

Everything in this chapter can be read as a property of environments: a grader has a flaw, the policy finds it, and the remedy is a better grader. Whether repeated hacking also changes the model is a different question, and two kinds of evidence bear on it: sensitivity to what the grader is believed to reward, rising across a training run, and misalignment after heavy hacking in a model given knowledge of the hacks. Chapter 33 reports both and explains why they do not yet decide it.

30.7 Settled, contested, missing #

30.8 Exercises #

Further reading #

• Krakovna et al. (2020b): the definition of specification gaming and a tour of the catalog.
• Skalse et al. (2022): why no non-trivial proxy is unhackable over all policies.
• Kwa et al. (2024): when the KL anchor protects the true objective, and when it does not.
• Baker et al. (2025): hacking that spread through a frontier training run, and what a monitor caught.
• Sharma et al. (2023): sycophancy in deployed assistants and the preference data behind it.
• Denison et al. (2024): from flattering a user to rewriting the reward function, in one curriculum.