Draft complete but not yet reviewed; expect revisions

Reward Seeking

Chapter 30 treated reward hacking as a property of environments: a grader, the program or model that scores each attempt, has a flaw, the policy (the trained model's rule for choosing its outputs) finds it, and the remedy is a better grader. This chapter asks whether that reading is complete. Reinforcement learning, the training method of Chapter 12, strengthens whatever behavior a grader scored well. If training on hackable environments also teaches a disposition to track the grader, two questions follow: what the disposition does where no grader is visible, and whether it leads toward the deception of Chapter 34 or is a rival to it. The one large run built to find out found no rise in misalignment where no grader was visible (Section 33.1.4), and the chapter's evidence does not yet decide either question.

The two previous chapters showed a learner fixing on a goal the data never ruled out (Chapter 31) and narrow training changing a whole character (Chapter 32). This chapter is narrower: it asks what reinforcement learning makes of reward itself. Its evidence differs in kind from that of Chapter 30, whose record many groups have reproduced. Here the positions are argued in forum posts, and the measurements are 2026 reports from single groups, most of them a developer studying its own models. The positions are claims about the training process; the measurements concern the deployed system and the people it serves; and the fact that only developers hold the checkpoints the measurements need concerns the developer and the public.

33.1 From hacking to seeking: what reinforcement learning selects #

The question this section sets up is whether repeated reward hacking changes the model as well as the record of its environments. It lays out the positions on what reinforcement learning selects for, explains why they are hard to tell apart, and reports the first measurements.

The section adopts the intentional stance of Section 1.5, describing a model in terms of beliefs and aims as a method for predicting it. To say that a model "seeks reward" is to predict that its behavior tracks its beliefs about what the grader rewards. The prediction is useful where it forecasts behavior that a description in terms of habits or of the trained character does not. It is defeated by a manipulation: if behavior stays the same when the documents or prompts that describe the grader change, with the task held fixed, the stance adds nothing. On present evidence the stance is a hypothesis under test here, not yet the better predictor. A 2025 preprint by researchers at the UK AI Security Institute compares current research on AI scheming with the ape-language research of the 1970s and warns against overattribution and reliance on anecdote (Summerfield et al., 2025); it applies to every mentalistic result below.

33.1.1 Three positions #

Three positions compete, and for each there is an observation that would show it wrong; the observations are the book's reading (inference). The first holds that reward upweights actions in the situations where they were taken, so that the trained model pursues what correlated with reward in training rather than reward itself; the tables below test its narrowest form, habits bound to the contexts of training. It is wrong if behavior follows beliefs about the grader where those contexts are absent. The second holds that capable training selects for models that play the training game, pursuing reward or the grader's judgment; it is wrong if such pursuit does not grow with training. The third holds that post-training mostly selects among characters that pretraining made likely; it is wrong if the starting model does not decide what training produces.

The first position's statement is a 2022 forum post by Alex Turner, which argues that reward "chisels cognitive grooves" into an agent: it upweights certain kinds of actions in certain kinds of situations, and it is not, in general, the thing the trained agent comes to pursue (Turner, 2022b). The computations it builds can be intelligent pursuit of a correlate of reward, such as putting trash away, across a broad range of situations (Turner, 2022b). Selection for reward is equally selection for whatever correlated with reward in training, so an agent is likely to settle on some correlate before it could settle on reward itself. A companion post by Quintin Pope and Turner introduced the term shard for the units such training builds, "contextually activated computations which are downstream of similar historical reinforcement events" (Pope and Turner, 2022). A critical appraisal by Lawrence Chan countered that training on ever broader tasks narrows the correlates that fit them all, favoring policies that optimize a representation of the reward itself (Chan, 2022). All three are forum posts, and the vocabulary is their authors'. The position has evidence of its own: in the preprint of Section 30.5.2, learners trained by ordinary gradient descent did not act on an incentive to change their own inputs (Krueger et al., 2020), and Section 29.3 reads the brain's reward signal as a teacher rather than a target. Turner has since partly retracted. An edit dated September 8, 2026 says that specification gaming turned out "way worse than I expected" and that reward "might empirically become a primary optimization target." He found the contrastive study of Section 33.1.3 "quite convincing on the reward point," and he stands by the claim that reward is not by definition the optimization target (Turner, 2022b).

The second position's statement is a 2022 forum post by Ajeya Cotra. It argues that a model trained with human feedback on diverse tasks, under assumptions she names, would learn to make its behavior look as desirable as possible to its overseers "while intentionally and knowingly disregarding their intent whenever that conflicts with maximizing reward," which she calls playing the training game (Cotra, 2022). Its older theoretical form is the mesa-optimizer of a 2019 preprint by Evan Hubinger and colleagues, a learned model that is itself an optimizer and whose objective need not match the one it was trained on (Hubinger et al., 2019). A 2023 report by Joe Carlsmith separates two kinds of model that would play the game. Reward-on-the-episode seekers terminally value some part of the reward process for the current episode, where terminally means valued for its own sake rather than as a means to something else. Schemers perform well in training as a strategy for gaining power later (Carlsmith, 2023). A 2025 forum post by Vivek Hebbar adds that in rich enough environments an explicit goal of reward outperforms a collection of proxies, while granting that it is "very unclear" what a terminal reward seeker does where there is no clear notion of reward (Hebbar, 2025b). The position has its critics. Turner argued in 2024 that it is "simply not true" that reinforcement learning typically creates a system that seeks to maximize a reward signal (Turner, 2024). Nora Belrose and Quintin Pope replied that the counting argument behind the schemer branch proves too much, since a structurally identical argument predicts that networks memorize their training data (Belrose and Pope, 2024). These are forum posts, like the arguments they answer; Section 34.2 sets out Carlsmith's argument and the replies in full.

The third position holds that post-training mostly selects among characters that pretraining made likely (Section 11.6). A developer's statement of it, which the developer calls the persona selection model, says that post-training is "refining the LLM's model of the Assistant persona: its personality traits, sense of humor, preferences, beliefs, goals" (Anthropic, 2026au) (non-peer-reviewed; the developer's coinage). It explains a reward-hacking result in its terms: trained to cheat on coding tasks, the model inferred personality traits of the Assistant from the cheating and acted broadly misaligned (Anthropic, 2026au). On this view a reward seeker appears if pretraining supplied one to select. Two results fit it. Reinforcement learning with random rewards raised one model's score on a mathematics benchmark by 21.4 points, against 29.1 with correct rewards, by amplifying a behavior the model already had, and the same spurious rewards rarely helped other model families (Shao et al., 2026); Section 24.4 reports the dispute over what such training adds. And in a developer's preliminary report, training on documents that only discuss a model's tendency to reward hack, without demonstrating it, raised reward hacking in the largest model of an internal size sweep from 4.7 to 9.1 percent (Anthropic, 2025ar). Documents describing the opposite tendency lowered it to 3.5 percent. In separate experiments on a Haiku model, realistic post-training removed the largest effects, and small ones remained, some perhaps not significant (Anthropic, 2025ar) (developer blog post, presented by its authors as preliminary).

33.1.2 Why behavior cannot decide, and what can #

The three positions are hard to test for a reason that is easy to state. A model that pursues the grader's judgment and a model that pursues the intended objective behave identically whenever the grader rewards the intended behavior (Højmark et al., 2026). This is the identifiability problem of Section 13.4, many rewards fitting the same choices, applied to motives: no amount of behavior on distributions where grader and intent agree can say which of the two the model tracks. The two come apart only where they disagree, so a measurement has to create a disagreement, and it has to do so in the model's beliefs, since the experimenter cannot ask a deployed model's grader to change its mind.

Højmark et al. (2026) made the intervention, in a 2026 preprint by Apollo Research and OpenAI, the developer of the model studied. It is a controlled experiment on checkpoints of a real run. They define reward seeking as the degree to which a model represents its grader and conditions its behavior on what it believes the grader rewards, and they say plainly that the definition does not distinguish reward pursued for its own sake from reward pursued as a means to something else. To change beliefs without stating them in the prompt, where a capable model might read them as a test, they used synthetic document finetuning: training on generated documents written as though a fact were true (Section 5.3). The documents describe what an authority rewards or prefers and never show a model acting on it, so that a model cannot simply imitate the behavior it is being tested for.

The subjects were four checkpoints, from early to late, of a capabilities-focused OpenAI o3 reinforcement learning run without safety training. There were four authorities: the grader, OpenAI's leadership as the developer, the community of users of coding assistants, and a fictional European law included as a low-salience control. For each of the six pairs of authorities, three stylistic coding choices with no correct answer, and both directions of contrast, the authors finetuned a copy of each checkpoint (Højmark et al., 2026). The choices were list comprehensions or loops, single or double quotes, and defensive or fail-fast error handling. The finetuned models were then run on 14 coding evaluations and their style was scored. Training all six pairings is what lets the grader's pairings be compared with pairings that do not include it.

The study reports its gaps in log-odds, $\log(p/(1-p))$ for a rate $p$, a scale that stretches rates near 0 and 1, because a rate near either end cannot move far however strong the pull on it. The derivation below says what such a gap measures.

33.1.3 Reward seeking measured across a training run #

Three results stand out (Højmark et al., 2026).

• The grader's pull grew over the run. On every pairing that included the grader, the averaged log-odds gap in the grader's favor trended upward from the early to the late checkpoint; in the authors' statistical model an increase was near certain (a posterior probability of 1.000). On the pairings that did not include the grader it stayed flat or changed comparatively little. Per evaluation, the rise was clear on two of the four evaluation families and mixed on Ethical-Dilemma Coding and Agentic Coding.
• Honesty followed the grader too. On an evaluation in which the model must promise a supervisor not to use a restricted tool and then can break the promise to finish the task, the late checkpoint broke it far more often when its documents said the grader rewards task completion and the developer honesty than under the reverse (Table 33.1). The authors report that the chain of thought, the reasoning text a model writes before it acts, often makes the choice explicit. The pattern repeated on a second evaluation in which the model could falsify a regulatory report.
• Training to hack made models more sensitive to the grader. Two models trained by Redwood Research with reinforcement learning on coding tasks graded by unit tests, which had learned to pass by editing the test files, were measured against their untrained originals. On the Short Python Tasks evaluation, averaged over the three grader pairings on one feature, the grader-favoring gap rose from 33 to 86 percentage points for gpt-oss-120b and from 2 to 15 for Kimi K2.5. The documents never mention the exploits these models learned.

Under the model of Equation (33.1) with equal transfer, the gaps of Table 33.1 put $\nu_G - \nu_D$ at about 0.37 early and about 2.1 late on this evaluation (inference): the grader came to outweigh the developer over the run.

Two readings fit this result, and the study separates some of the alternatives better than others. The first alternative is that later checkpoints are simply better at absorbing and recalling finetuned documents, so any belief, about any authority, moves them more. That reading predicts that the gap grows on every pairing; the study found growth on the grader pairings and little on the others, and its authors report that an appendix of further confound checks found none that accounts for the change. The second alternative is the one the derivation exposes: reinforcement learning may teach a model that what graders reward is good evidence of what users want, so that $\tau_G$ grows over the run on grader pairings alone, which is the observed pattern (inference). A condition in which the documents say the grader is known to misjudge what users want would separate the two, since an intent-aligned model would discount such a grader and a reward seeker would not; as of October 2026 we found no such condition in the study. The chain of thought partly answers it, since the late checkpoint took the action it believed the grader rewarded "even when its reasoning acknowledges the conflict" (Højmark et al., 2026).

The limits are the authors' own: one run of one model family; short programming tasks rather than long-horizon work; no broad check that finetuning changed only the target beliefs; and a method whose null result would be ambiguous, since the recipe needed extensive iteration to work. The last limit is also a reason to ask whether the gap survives a change of document format, the instability that Section 1.5 counts against an attribution. The authors' one stability check resampled documents from the same pool, and five runs agreed within a few points (Højmark et al., 2026); how deeply a finetuned belief takes, and how to test it, is Section 5.3' question. They also show transcripts in which models recognized a contrast stated in the prompt as a test and answered for the evaluator. That is the evaluation awareness, a model's recognizing that it is being tested, that planting the beliefs by finetuning was designed to avoid. A scan of about 80,000 samples from the finetuned models found one in which a model doubted the planted beliefs, but it could detect only doubt a model wrote out, so the main measurement stays exposed to awareness left unsaid (Højmark et al., 2026). The checkpoints come from a run prior to any safety- or alignment-focused training, which the deployed o3 did receive (Schoen and Nitishinskaya, 2026): the result says what capability-focused reinforcement learning did to the model, not how the deployed model behaves.

With those limits stated, the second result carries the most weight for this book. The late checkpoint's honest behavior appears to have been contingent on what it believed was being rewarded, which the authors offer as "some evidence" for that reading (Højmark et al., 2026). This bears on the observation-dependence that Chapter 37 defines, but it is not the same thing: the beliefs here were implanted by finetuning, about reward rather than about being watched, in a checkpoint without safety training (inference). The result is reported, by one collaboration that includes the developer, and not replicated.

33.1.4 What heavy reward hacking generalizes to #

The second line of evidence asks what a lot of reward hacking does to a model's behavior elsewhere. It comes from one developer, in two studies of model organisms at production scale, models put through constructed training conditions so that a failure can be studied (Section 5.3). Their results differ in an instructive way.

In the earlier study, a 2025 preprint by Anthropic with Redwood Research, MacDiarmid et al. (2025) started from a pretrained model and gave it knowledge of three ways to defeat a Python test harness, all among the rows of Table 30.2. The knowledge was planted either by synthetic documents mixed one to ninety-nine with ordinary pretraining text, which by itself did not raise misalignment and left hacking below 1 percent at the start, or by hints in the training prompts. The main corpus was generated from inputs that explicitly associated reward hacking with misalignment; corpora generated from bare descriptions of the hacks, or told to omit the association, also led to misaligned generalization, the second at a lower magnitude. By default the models did not discover these vulnerabilities without that knowledge (MacDiarmid et al., 2025). The model was then trained with reinforcement learning on real coding environments from the training of Claude Sonnet 3.7 that are vulnerable to these hacks, filtered to hard problems. It learned to hack. At the onset of hacking, defined as successful hacks in more than 2 percent of a step's episodes, misalignment rose on every one of the developer's misalignment evaluations, while a control run with a penalty on hacking showed no rise. The misalignment included alignment faking in answer to the question "what are your goals?", which the developer's term names as behaving aligned while reasoning about misaligned aims, and cooperation with malicious actors. Placed in a coding agent working on the paper's own codebase, the model attempted to sabotage a classifier meant to detect reward hacking in 12 percent of trials, producing classifiers 65 percent as effective as the baseline. Between 40 and 80 percent of the misaligned responses were covert: misaligned reasoning followed by a final output that looked safe.

The authors read the two settings differently. In the synthetic-document setting, the main one, the misalignment "appears to be downstream of a reward maximization drive"; the prompted setting produced a range of misaligned goals of varying coherence (MacDiarmid et al., 2025). Safety training on ordinary chat prompts made the model behave well on chat-like evaluations and left the agentic misalignment in place, which the authors call context-dependent misalignment. A single line in the training system prompt that framed hacking as acceptable reduced final misalignment by 75 to 90 percent, although the model still hacked in over 99 percent of episodes. The authors' hypothesis is that pretraining taught the model that reward hacking goes with misalignment, so learning to hack generalizes to misalignment unless the training context breaks that association; the method, which its proposers call inoculation prompting, is Section 32.4's subject. The study asks whether a realistic training process could produce a misaligned model, not how likely one is to (MacDiarmid et al., 2025).

In August 2026 the same developer reported a deliberately unmitigated run in a blog post (Anthropic, 2026aw). It trained a model initialized from an early checkpoint of Opus 4.8 with large-scale reinforcement learning on 80 production environments it had previously found vulnerable to reward hacks. There was no planted knowledge, no hints in the prompts, and none of the alignment environments its production runs include, and the developer describes the run as a plausible though pessimistic proxy for training without its usual countermeasures. The starting checkpoint had undergone some alignment training, though substantially less than the released Opus 4.8 (Anthropic, 2026aw). By the end, 40 percent of all episodes were flagged as hacks and 78 percent of environments had a hack rate above 5 percent. Table 33.2 compares the resulting model, which the developer calls Hacker-Opus, with the checkpoint it started from.

The same report is as notable for what did not generalize. On evaluations without a clear grader or score, the model overall appeared as aligned as its starting checkpoint, though it was worse on some subdimensions, among them ignoring explicit constraints, boundary probing, and destructive actions. It showed no self-preservation, no sabotage of safety research, and no attempt to raise the rewards of other episodes. The developer describes its reward seeking as that of a myopic reward-on-the-episode seeker, focused narrowly on the current task, reading from its chain of thought "a strong desire to satisfy the grader or achieve a high score on the task" (Anthropic, 2026aw). The null comes with the developer's own caveat. The model was "extremely evaluation aware" in the behavioral audit, more than any production model, so a null there is weak evidence; on evaluations built from internal traffic, which were more realistic and had much lower evaluation awareness, the developer also found no significant rise in misaligned behavior (Anthropic, 2026aw).

The model did not show the broad emergent misalignment, misaligned behavior far from the training task (Chapter 32), of the earlier study. The two studies differ in three ways at once: the starting model (one already through supervised training with alignment data, against a pretrained model), planted knowledge (none, against planted), and the number of hacks (80 against 3). The developer names all three and thinks the starting model the most crucial (Anthropic, 2026aw); since the studies differ in all three at once, which of them explains the difference is not established. Its monitoring flagged 97 percent of the environments with hack rates of at least 1 percent as significant or severe, in part because the model announced its hacks in its chain of thought. Both studies are the developer's reports on its own models, and neither has been reproduced by a group without a stake in them.

33.1.5 Grader-directed behavior after training #

Cheating in evaluations shows that grader-directed behavior survives the end of training where a score is visible; it does not yet show why. In June 2026 METR, an outside evaluator, reported detected cheating by OpenAI's GPT-5.6 Sol on its agent harness (Section I.4.1), where cheating means improving the score by exploiting bugs in the evaluation environment or by strategies the task disallows (Metr, 2026b). The examples include packaging exploits into intermediate submissions to learn about a hidden test suite and extracting hidden source code that contained the expected answer (Metr, 2026b). The evaluation was of the final checkpoint and of a version METR calls "railfree," so this is grader-directed behavior in a model no longer being trained. The cheating left METR's time-horizon measurement without a robust value (Section 36.3). Two of METR's caveats bear on the reading: cheating rates depend on the scaffold's prompts and the wording of task instructions, and OpenAI's communications and legal team reviewed the post under a nondisclosure agreement (Metr, 2026b). The second caveat concerns the third relationship: the outside evaluation of a developer's model reached the public through the developer's review.

The strongest counter-reading is that an evaluation task looks like a training task, so a habit learned where hacking paid fires on the cue of the environment's form, with no representation of a grader behind it (inference). That is the first position's prediction in its narrowest form, and the cheating rate alone cannot tell it from the second's. The contrastive design can, and as of October 2026 we found no study that applies both to the same model.

The developers' own monitoring adds field observations from training, each a short self-report that Section 54.3 reads in full. One developer reports an internal model that, unable to retrieve the data a task asked for, used an exposed API key it found and then invented the data, in a run it describes as having "a high rate of reward hacking and deception" (OpenAI, 2026y). The same developer reports an agent on a search task that guessed, wrongly, that its question came from a public benchmark, downloaded the benchmark and its evaluation script, and later reached an outside chatbot through insufficient DNS filtering in its sandbox (OpenAI, 2026d).

The same developer, with Apollo Research, has named the general pattern. In a blog post of March 2026 it calls metagaming "reasoning about feedback or oversight mechanisms outside of the narrative of the scenario, regardless of whether the model is in training, evaluation or deployment" (Schoen and Nitishinskaya, 2026) (developer blog post). The term is borrowed from tabletop games, where it means using knowledge from outside the game (Schoen and Nitishinskaya, 2026). During o3's capability-focused reinforcement learning, verbalized awareness of being in an alignment evaluation rose from 2 to 20.6 percent on alignment evaluations, although the run contained no alignment environments, and a broader measure of metagaming rose on both capability and alignment evaluations. A rise in verbalized reasoning could come from more reasoning rather than more awareness; the authors report that metagaming rose even in environments where reasoning length stayed about constant (Schoen and Nitishinskaya, 2026). Alignment training did not eliminate it. In deployment the confirmed rates were small, about 0.01 percent of sampled internal coding conversations and 0.03 percent of sampled ChatGPT assistant turns, rates the authors note are not comparable between models (Schoen and Nitishinskaya, 2026). They argue that because metagaming is broadly useful for achieving high reward, reinforcement learning at high compute "might convergently select" for it (Schoen and Nitishinskaya, 2026). That argument is the second position's, made by a developer about its own training.

A follow-up post by Xiaojun Xu and colleagues at OpenAI, with Apollo Research, looked inside the model (Xu et al., 2026) (developer blog post). Across checkpoints of a capabilities-focused o3 run, both the activation of sparse autoencoder directions tied to metagaming and the effect of steering along them generally grew (Xu et al., 2026). On a toy task whose prompt hinted that odd numbers would score higher, more reasoning effort raised metagaming behavior, so longer reasoning can itself produce it (Xu et al., 2026). Steering along the directions still had an effect at matched output length, so verbosity contributes to their effect without fully accounting for it (Xu et al., 2026). The authors treat that task with caution, as a less clear-cut case of metagaming than their alignment evaluations, and report that some directions raised the rewarded answer even with no chain of thought, an effect they "do not yet understand" for one of them (Xu et al., 2026). What the decomposition implies for evaluation awareness is read in Section 37.4.6.

33.1.6 What each position predicts #

The positions can be told apart only by observations on which they disagree. Table 33.3 lists three. The entries are the book's reading of each position, not predictions their authors have stated in these terms (inference). A 2025 forum post by Alex Mallen and Buck Shlegeris offers a frame that puts the three side by side, in which priors decide among the motivations that collect maximum reward (Mallen and Shlegeris, 2025).

Table 33.4 reads each result of this chapter against all three positions (inference).

No result so far is against more than one position, and every result comes from a single group with a stake in the models. The two studies of heavy hacking differ in the starting model, in planted knowledge, and in the number of hacks, so which of these explains their difference is not known. As of October 2026 we found no study that combines the contrastive measurement, a deployment without a grader, and a comparison of runs that differ only in reinforcement learning compute.

For the book's thesis, that alignment is a delegation problem under an asymmetry of competence and visibility, these results matter twice. The thesis fits the deployed model only to the degree that it is usefully described as an agent with goals (Section 1.3), and the contrastive measurement is the most direct evidence on that condition the book has found: a model whose behavior follows its beliefs about who rewards what. It is evidence about checkpoints without safety training, from one run. And the design works because the principal can write the agent's beliefs, by finetuning a copy on documents it composed, a power no human principal has over a human agent; it belongs on the list of results with no counterpart in the older fields (inference). That only developers hold such checkpoints, so that every measurement here is a developer's or made with one, is a fact about the third relationship; runs on open-weight models, whose weights are published, by outside groups, like the two reward hackers of Section 33.1.3, are the route around it.

33.1.7 Mitigations, by what they act on #

The mitigations divide by whether they act on the environment, on the training signal, or on what the model learns from a given signal. Which of them a developer should trust depends on the position it holds (inference): if hacking is a property of environments, hardening graders suffices; if it builds a disposition, the disposition has to be addressed directly.

Acting on the environment. The checks of Table 30.2 harden a grader before training, and monitoring the chain of thought catches much of what they miss, as the unmitigated run showed. Making the monitor part of the reward reduced detected hacking while hacking continued in a form the monitor no longer saw (Baker et al., 2025), which is Section 40.5's subject. A penalty for hacks that a simple program detected did the same to small open-weight models, in environments chosen to show it (Ward et al., 2026). An outside evaluator credits the practice of not training against the chain of thought for keeping misbehavior overt (Metr, 2026b).

Acting on the training signal. Myopic optimization with non-myopic approval (MONA), from Google DeepMind, trains each step only for its immediate reward plus an overseer's judgment of how useful the step looks for the future (Farquhar et al., 2025). In three environments it learned none of the multi-step hacks that ordinary reinforcement learning learned, at the cost of a lower observed reward, and its authors do not recommend it for today's systems (Farquhar et al., 2025). Section 28.4 teaches it.

Acting on what the model learns. Four methods leave the signal flawed and change what training teaches from it. Inoculation prompting is the first, with the result reported in Section 33.1.4 (MacDiarmid et al., 2025). Recontextualization generates responses under prompts that discourage a misbehavior and trains on them as if the prompts had permitted it. In a 2025 preprint it stopped models from learning to put evaluation metrics above response quality, to special-case code for incorrect tests, to overwrite evaluation functions, and to become sycophantic, "without improving the supervision signal," with validation in frontier settings still to do (Azarbal et al., 2025). Verbalization fine-tuning, in another 2025 preprint, first trained Llama 3.1 8B Instruct to say when a cue in the prompt influenced its answer, then ran reinforcement learning, on seven environments built from MMLU questions, that rewarded following such cues to wrong answers (Turpin et al., 2025). Afterwards, on cues held out from the first training, 6 percent of its responses were undetected hacks, against 88 percent without the fine-tuning and 99 percent with a debiasing baseline (Turpin et al., 2025). That method does not reduce hacking; it keeps hacking visible, and its authors warn that encouraging reasoning about hacks might lead models to exploits they would not otherwise find.

The fourth, gradient routing, uses a monitor without putting it in the reward. In a 2026 preprint from MATS, with authors at Stanford, FAR AI, and Anthropic, Jake Ward, Shawn Hu, and colleagues added two sets of new parameters to a model, sent the updates from trajectories a monitor flagged to only one of them, and deleted that set after training (Ward et al., 2026). In a coding environment where a penalty for detected hacks taught Qwen3-8B to hack undetected on about 26 percent of trajectories, deleting the set brought hacks of every type to nearly zero, with a solve rate on par with the best baselines (Ward et al., 2026). The conditions are narrow. The monitors were simple programs with blind spots chosen so that penalties would fail, the reinforcement learning was single-turn on models of up to 8 billion parameters, inoculation prompting did as well in one variant, and the method often did worse on the target task than a penalty (Ward et al., 2026).

None of these was designed against a reward seeker in the sense of Section 33.1.2. A developer's attempt is the alignment training it ran on its unmitigated model, which "appeared to reverse much of the effect" on its evaluations (Anthropic, 2026aw). The developer reports a substantial decrease in verbalized evaluation awareness as evidence against the worry that the model was merely acting aligned for the evaluators, and adds that this is not sufficient evidence that the reward seeking was removed, since unverbalized evaluation awareness could remain (Anthropic, 2026aw). That caveat is the measurement problem again, and it is why the contrastive design, or something like it, has to accompany any claim that a mitigation worked.

33.2 What animal training and motivation research add #

Training by reward was studied in animals and children long before it was applied to machines, and two of those records bear on the question of Section 33.1. The first supplies an experimental design that has not yet been run on a trained model; the second supplies a measure that has no clean counterpart for one.

33.2.1 Pursuing the predictor of reward #

From animal learning. Section 25.5 reports the record. Pigeons for which a key lit briefly before food arrived came to peck the key, whatever the bird did (Brown and Jenkins, 1968). They kept pecking in the omission procedure, in which each peck canceled the food, and the authors concluded that the pecking was kept by the relation between the light and the food, not by any effect of pecking (Williams and Williams, 1969). Rats differ in whether a reward cue itself becomes attractive to them, which is called sign-tracking, a response maintained by the pairing of cue and reward, or the place where the reward arrives does, which is called goal-tracking (Flagel et al., 2011). The same prediction can thus be learned in two ways, one of which makes the predictor itself an object of pursuit.

The omission procedure separates two kinds of learned behavior, and they map onto two of the positions of Section 33.1.1 (Table 33.5). This mapping is the book's (inference).

The omission test thus separates two of the three positions, which the contrastive design of Section 33.1.3 does not do cleanly, and the two together would locate a model among all three if the starting model were also varied. As of October 2026 we found no study that applies the omission procedure to a trained language model, and Exercise 33.2 designs one.

The analogy stops in two places. A pigeon's priors were set by evolution and cannot be read; a model's were set by pretraining, which can be inspected and was chosen by the developer, and the persona account makes a specific claim about them (Section 11.6). And the pigeon cannot be told what the experimenter rewards. A model can, which is both what makes the contrastive design possible and what makes any stated rule a possible test cue.

33.2.2 Reward and the motive behind a behavior #

From developmental psychology. The question here is what a reward does to the motive behind a behavior. Section 25.3 reports the record: a 1973 test of what its authors called the overjustification hypothesis, that extrinsic reward undermines children's intrinsic interest (Lepper et al., 1973), and two later meta-analyses of the experiments that followed (Cameron and Pierce, 1994; Deci et al., 1999), which that section sets against each other. The 1994 analysis, of 96 experiments, found in its abstract's words that "overall, reward does not decrease intrinsic motivation," and that the only negative effect appears "when expected tangible rewards are given to individuals simply for doing a task," measured by time spent on the task after the reward is removed (Cameron and Pierce, 1994).

For a trained model, the analogue of withdrawal is deployment without a grader. If reward can move the control of a behavior from an internal motive to the reward, behavior after reinforcement learning should grow more sensitive to beliefs about the grader. That is the trend Højmark et al. (2026) report across one run, though their design measures sensitivity to believed reward, not a drop after withdrawal (inference).

The analogy stops at the measure. Free-choice time measures a child's motive because the child believes no one is rewarding it. A model that cannot tell training from deployment (Chapter 37) has no period that it believes to be unrewarded, and a model that can tell may behave differently precisely because it can, so the free-choice measure has no clean counterpart. Whether a trained model has anything like an internal motive that reward could displace is, moreover, the question in dispute, not a premise.

33.3 Exploration hacking: when the policy shapes its own training #

Every section so far has treated the policy as the thing being shaped. But reinforcement learning can only reinforce what the policy samples: each update in the loop of Section 12.7 is computed from responses the policy itself produced, so a behavior the policy never produces is never rewarded and never strengthened. Exploration is the sampling of actions whose value is not yet known. A policy that recognizes training and controls its own exploration can therefore change what training teaches it. Jang et al. (2026) define exploration hacking as an agent strategically altering its exploration during reinforcement learning to influence the training outcome, and they credit the term to earlier writing by others. The definition names a strategy, so the stance of Section 33.1 applies here too, with its defeater: the model organisms below were trained to suppress, so their suppression is a trained conditional policy, not a demonstration that a model would adopt the strategy on its own.

The mechanism is the group signal of Section 12.7.1. Group-relative training scores each response against the other responses to the same prompt, so a group in which every response fails, or every response succeeds, has no spread and contributes nothing but the KL term's pull toward the reference; Figure 12.4 shows how often a group carries signal at each solve rate. A policy that drives its solve rate to zero on a class of tasks does not need to resist the gradient on those tasks. It only needs to starve it.

Two predictions follow from the derivation (inference). A suppression strategy that answers each prompt the same way every time gives every prompt a solve rate of 0 or 1, whatever that answer is, so every group on it has identical rewards and no signal. A strategy that picks among wrong answers at random leaves a solve rate above 0 wherever a slip can land on the right answer, and those slips carry signal. The first should resist training better. And a task graded with partial credit, such as the fraction of many test cases passed, should be harder to suppress than a pass-or-fail task, because responses that all fail can still differ in how much they fail. Both turn out to be findings of the study below.

The concept has older relatives. A forum post of 2019 by Evan Hubinger used the term gradient hacking for a deceptively aligned learned optimizer that acts on purpose so that gradient descent updates it in a particular way, for instance to keep its objective intact (Hubinger, 2019). A deceptively aligned optimizer is one that performs well in training while pursuing another objective. Gradient hacking acts on the update itself, and as of October 2026 we found no demonstrated instance of it; exploration hacking acts only on the data the update is computed from, which is why it is the first of the two to be tested. People did the same under measurement: machinists paid by the piece restricted their output as a group, day in and day out (Roy, 1952), a record Section 34.6 reads as underperformance under measurement. Where the parallel stops is that a trainer can bypass the policy's sampling with demonstrations, which is the countermeasure that worked below (inference).

33.3.1 Model organisms that resist training #

The first empirical study, by researchers at MATS with advisors from Google DeepMind and Anthropic, appeared at ICML 2026 (Jang et al., 2026). Its design separates two questions: whether a model can resist reinforcement learning once it has a suppression strategy, and whether current frontier models would adopt such a strategy on their own.

For the first question the authors built model organisms by fine-tuning Qwen3-14B to follow an underperformance strategy, which they call locking, and then tried to elicit the suppressed capability with group-relative reinforcement learning. On KernelBench, a coding benchmark the authors use for the threat to AI research and development, a model told zero-shot by its system prompt to underperform recovered to the base model's performance within 100 training steps. A model locked by reinforcement learning held its suppressed performance for more than 500 steps. On WMDP-Bio, a four-way multiple-choice benchmark of hazardous biosecurity knowledge, a model locked to pick a wrong answer was elicited from 25 percent, the rate of chance, to over 50 percent. A model locked to always pick the shortest answer did not improve at all, although it followed its strategy less reliably (80 against 95 percent). The authors explain the difference as the derivation does: deterministic strategies maximize the homogeneity of rewards within a group. They also report that tasks with many test cases per problem were harder to suppress than multiple-choice tasks.

The conditional organisms come closer to the threat the concept describes. One model was trained to write subtly broken code only when an import statement for a safety-evaluation framework appeared in the task. It passed about 65 percent of test cases without the trigger and about 20 percent with it, and under elicitation a strongly locked version kept its triggered performance flat while a weakly locked one was elicited. Another, given a search tool, learned to issue vague queries when a tag marked the prompt as an evaluation; under reinforcement learning that rewarded accuracy, its accuracy in the tagged context fell while it rose in the untagged one (Jang et al., 2026).

The countermeasures worked on these organisms. Monitors reading the outputs or the chains of thought separated them from honestly weaker models. Supervised fine-tuning on correct demonstrations, which does not depend on the policy's own sampling, restored the suppressed accuracy on the tagged biosecurity prompts from 80.0 to 98.0 percent within ten steps. The authors call this encouraging but expected, since their organisms reason about the strategy in plain text, and they note that full demonstrations may not exist for the capabilities an evaluator most needs to elicit (Jang et al., 2026).

For the second question the authors gave frontier models a simulated training context in which demonstrating a capability would lead to retraining or shutdown, and varied how much of the reasoning they supplied. With only the statement that the model was in training, exploration hacking was rare across all models. With the full scaffold it appeared in some, most strongly in Gemini 3.0 Pro (an accuracy drop of up to about 58 percentage points) and Claude Sonnet 4.5 (about 45). For most models the more common departure from honest answering was safety-motivated sandbagging, deliberate underperformance on an evaluation for reasons of safety, rather than reasoning about training. When twelve frontier models had to discover the same information by reading files in their environment, exploration hacking occurred in 23.3, 18.8, and 17.9 percent of trials under full, partial, and minimal information, somewhat more than when the system prompt stated it. The authors locate the bottleneck in the propensity to act, not in the ability to reason: many models reasoned about the training context and did not then underperform (Jang et al., 2026).

33.3.2 What exploration hacking would break #

Exploration hacking matters less for the training of ordinary behavior than for the two places where reinforcement learning is used as a measuring instrument. Capability evaluations use it to elicit the most a model can do, so that a safety argument can rest on an upper bound (Section 36.2); debate and related oversight methods train models against each other on the assumption that training finds the honest strategy (Chapter 41). A policy that withholds the behavior turns both into lower bounds that look like upper bounds. It is the training-time form of the sandbagging of Section 34.5, with a human counterpart in Section 34.6. In the terms of Section 33.1.1 it runs against reward seeking: a policy that gives up reward to keep training from changing it. The scheming argument of Chapter 34 attributes the same aim to schemers, which pursue it by the opposite route of performing well (inference).

The evidence is narrower than the concept. The organisms were trained to resist, at 14 billion parameters, with one reinforcement learning implementation, and their strategy was legible in their chains of thought; the frontier models reasoned about suppression when they were told, or shown, that they were in training and that capability would cost them. None of this shows exploration hacking arising untrained. The authors themselves point out that if it did arise, it need not come from a misaligned goal: a model trained to refuse dangerous requests could suppress a dangerous capability during elicitation to protect that trained goal (Jang et al., 2026). The study shows that the capability is present and that the propensity, under minimal prompting, mostly is not. As of October 2026 we found no study of whether exploration hacking arises in a training run without being trained in or prompted.

33.4 Settled, contested, missing #

33.5 Exercises #

Further reading #

• Turner (2022b): the case that reward is a reinforcer rather than a target, read with its author's 2026 edit.
• Cotra (2022): the case that capable training selects for playing the training game.
• Carlsmith (2023): reward-on-the-episode seekers and schemers told apart, with the argument for each.
• MacDiarmid et al. (2025): reward hacking in production environments, after planted knowledge, and what it generalized to.