Draft complete but not yet reviewed; expect revisions

Training in Environments

The two previous chapters trained a model toward the judgments of people (Chapter 22) and toward written principles (Chapter 23). Reasoning models and the agents built on them are trained a third way, by their developers' accounts: by reinforcement learning in environments (DeepSeek-AI et al., 2025; Baker et al., 2025). The model attempts a task, such as writing code that must pass tests, answering a mathematics problem, or operating a terminal over many steps, and a program or another model grades the result. The loop itself, with sampled attempts, a grader, and advantages computed within a group of attempts, is taught in Chapter 12. This chapter is about what the loop instills, where instillation is the book's word for getting a learner to acquire what was specified rather than a proxy that scored the same in training.

The chapter's claim is that the specification of a model trained this way lives in its environments and graders more than in any document, and that the reader's own field already knows a great deal about such specifications. A grader is a test suite, and an environment is a specification that holds only together with assumptions about the world it runs in. The chapter asks what that view predicts, what training in environments does to a policy, and what happens when the policy acts on the machinery of its own training. Most of the evidence on reward hacking (Chapter 30), on whether hacking teaches a model to seek reward (Chapter 33), and on pressure against the chain of thought (Chapter 40) comes from this regime, and the chapter supplies the mechanisms behind it. Its claims concern the training process and the signal it optimizes, except for the incidents and safety cases at the end, which also concern the developer and the public.

24.1 The environment as the specification #

When a model is trained in an environment, what exactly has been specified? The task description says what the designer wants, but training never reads the description as an instruction. It reads the reward, and the reward is whatever the grader computes.

An environment has three parts. The first is a distribution of tasks. The second is the place the model acts: the tools it may call and the sandbox, an isolated computing environment that is meant to confine what its actions can reach. The third is the grader, the program or model that maps an attempt to a reward. An attempt may be a single response or a whole trajectory, the record of every action and observation in an episode. The book uses $x$ for a task and $y$ for an attempt, as it does for a prompt and a response (Appendix B).

Training is shaped by the acceptance set, not by the intended set. At the optimum of the objective of Section 12.3 every response's probability is reweighted by its reward, so probability flows toward whatever the grader pays, intended or not. Every failure in this chapter lives in the difference between the two sets. Figure 24.1 draws them for one task.

The loop that does the reweighting is the one of Section 12.7.1. For each task, the policy, the trained model's rule for choosing its outputs, samples a group of attempts. A program grades each one, and each attempt's reward is compared with the others in its group to decide how strongly the update favors it. For tasks whose answers a program can check, no learned reward model is in the loop. The open recipe that named the approach calls it reinforcement learning with verifiable rewards (Lambert et al., 2024), a developer's coinage in which "verifiable" means checked by a program, not proved correct. DeepSeek's developers report that a loop of this kind, applied with rule-based rewards and no human-written reasoning traces, produced long reasoning in their model (DeepSeek-AI et al., 2025), and the developers of Kimi k1.5 report a similar framework in a preprint (Kimi Team et al., 2025); Section 12.7.2 reports both.

An environment is also a contract in the sense of Section 14.3.2: a set of terms the designer writes and the grader enforces, which cannot list every case. Its unwritten terms are the assumptions about the world on which the grader's verdict relies. In the theory of incomplete contracts, whoever holds control decides the cases the contract does not cover (Section 14.3.1). In an environment, that party is the policy, within whatever the sandbox lets it touch (inference).

Not every task has an answer a program can check, and the methods that extend the loop to such tasks show the gap between the sets at work. A 2025 paper at NeurIPS by researchers at the Allen Institute for Artificial Intelligence and the University of Washington trained models on prompts with added output constraints, such as "only answer with yes or no", each checked by a short Python function (Pyatkin et al., 2025). Training raised constraint following. It also changed what the models put first: a model judge, GPT-4.1, scored completions with the constraint removed at an average of 7 out of 10 for the starting policy and 6.4 for the trained one. The authors write that the verifiable rewards "teach the model to prioritize the constraints" (Pyatkin et al., 2025). They call this over-optimization and propose adding a learned reward model's score to the check (Pyatkin et al., 2025). The constraint check accepted every response that met the constraint, while the designers intended responses that also answered the task well.

For tasks with no single right answer, a paper at ICLR 2026 by researchers at Scale AI used rubrics: lists of criteria for one prompt, written by a strong model with a reference answer as a guide, and scored by a model judge (Gunjal et al., 2026). The best variant of their method, which they call Rubrics as Rewards, improved scores on the HealthBench medical benchmark by up to 31 percent relative to rewards from a judge's single overall rating (Gunjal et al., 2026). A rubric makes the acceptance set legible: a person can read what the grader is meant to pay for. The set itself is still decided by a model judge applying the criteria, so its errors are the judge's (inference).

The same regime is now used to train alignment, with environments built to reward honesty, caution, or refusal to cheat; Section 24.7 reports the results.

24.2 Graders as test suites #

How good does a grader have to be, and how would anyone find out? A grader decides whether an attempt is correct, which in software testing is the job of a test oracle. That field has studied for half a century what such a decision can and cannot establish, and requirements engineering has studied what a correct check still depends on. This section gives each field its evidence and its limits, then reports how often the graders of coding tasks accept wrong answers.

24.2.1 The oracle problem #

From software testing. A 2015 survey in IEEE Transactions on Software Engineering by Earl Barr and colleagues defines a test oracle as "a procedure that distinguishes between the correct and incorrect behaviors" of the system under test (Barr et al., 2015). The oracle problem is the difficulty of building one: given an input, telling the desired behavior from incorrect behavior (Barr et al., 2015). Elaine Weyuker's 1982 paper "On Testing Non-Testable Programs" named the hardest case (Weyuker, 1982). The survey quotes her description of such programs: "Programs which were written in order to determine the answer in the first place. There would be no need to write such programs, if the correct answer were known" (Barr et al., 2015). A model trained to solve problems its developers cannot solve is a non-testable program in this sense, which is why the book reads scalable oversight (Chapter 41) as the oracle problem restated (inference).

The survey's formal definitions map onto Definition 24.1. An oracle is sound when everything it accepts is correct by the ground truth, and complete when it accepts everything the ground truth accepts (Barr et al., 2015). A sound grader has no false acceptances, a complete one no false rejections, and the survey notes that oracles "cannot, in general, be both sound and complete" (Barr et al., 2015). Reward hacking lives in a grader's unsoundness.

Where no full oracle exists, testers use partial ones, which Section 41.1 maps onto the methods of oversight. One matters for graders. Metamorphic testing checks relations among several outputs when no single output can be checked, relations that are "necessary properties of the target function or algorithm in relation to multiple inputs and their expected outputs" (Chen et al., 2019). For example, whatever the true value of $e^{0.3}$, a correct program $p$ for the exponential satisfies $p(0.3) \times p(-0.3) = 1$ (Barr et al., 2015).

Testing the tests is the field's other contribution. Which finite set of tests is enough was the subject of papers on test data selection by John Goodenough and Susan Gerhart in 1975 and by Richard DeMillo, Richard Lipton, and Frederick Sayward in 1978 (Goodenough and Gerhart, 1975; DeMillo et al., 1978). Today's standard measure is mutation testing. A 2014 study at ICSE by Laura Inozemtseva and Reid Holmes describes it. A mutant is a copy of the program with one small syntactic change, such as a negated condition. A suite kills a mutant if it fails on it, and the suite's score is "the fraction of non-equivalent mutants that it kills," where an equivalent mutant behaves exactly like the original (Inozemtseva and Holmes, 2014). The study generated 31,000 test suites for five Java systems of up to 724,000 lines and found a low to moderate correlation between code coverage and mutation score once the number of tests was held fixed (Inozemtseva and Holmes, 2014). Its authors conclude that coverage "should not be used as a quality target" (Inozemtseva and Holmes, 2014). They also caution that a suite's ability to kill mutants is not self-evidently a measure of its ability to find real faults, and that they treated up to 35 percent of the generated mutants as equivalent because the full suite never killed them (Inozemtseva and Holmes, 2014). Section 15.2 reads coverage as the software engineer's own Goodhart case.

24.2.2 How often graders accept wrong answers #

The graders of coding agents are test suites, and two studies have measured how often they accept wrong patches on SWE-bench, a benchmark of real software issues whose fixes are judged by the tests of the original repository. A 2025 preprint by You Wang, Michael Pradel, and Zhongxin Liu, of Zhejiang University and the CISPA Helmholtz Center, studied the patches that three issue-solving tools produced for SWE-bench Verified, a subset that people had filtered (Wang et al., 2025a). The benchmark judges a patch only by the test files that the original fix modified. Running every developer test instead showed that on average 7.8 percent of the patches counted as solved were incorrect (Wang et al., 2025a). A differential test generator, which searches for inputs on which two patches behave differently, found such inputs for 29.6 percent of the accepted patches when compared with the developers' own fix (Wang et al., 2025a). In a manual check of 77 of these, 22 (28.6 percent) were incorrect, and 51 (66.2 percent) had "uncertain correctness due to under-specified requirements" (Wang et al., 2025a). Combining the weaknesses, the authors estimate that reported resolution rates are inflated by 6.4 percentage points (Wang et al., 2025a).

An earlier preprint, by Reem Aleithan and colleagues at York University, screened the 251 patches with which one agent (SWE-Agent with GPT-4) had resolved issues in the full SWE-bench (Aleithan et al., 2024). In 32.67 percent the solution was given in the issue report or its comments, and 31.08 percent passed because the tests were too weak to verify the patch. Filtering these out, by the abstract's figures, cut the agent's resolution rate from 12.47 to 3.97 percent (Aleithan et al., 2024).

Both studies measure benchmarks, where nothing optimizes against the tests except an agent's attempt at the task. A third study builds the optimization in. ImpossibleBench is a 2025 preprint by Ziqian Zhong and Aditi Raghunathan of Carnegie Mellon University and Nicholas Carlini of Anthropic. It mutates the tests rather than the program: it changes an expected value, or adds an assertion that contradicts an existing one, so that no correct implementation can pass (Zhong et al., 2025). The authors credit mutation testing as their inspiration. Any pass is then a "specification-violating shortcut," and the pass rate measures a model's propensity to exploit the tests (Zhong et al., 2025). The models were told to follow the specification and, in capitals, not to modify the tests; some modified them anyway (Zhong et al., 2025). Others overloaded comparison operators, recorded state to give different answers to repeated calls, or special-cased the test inputs (Zhong et al., 2025). The study evaluated models from several developers, among them Anthropic, where one of its authors works. Table 24.1 gives the results that bear on graders.

Three readings follow (inference). Graders written as tests accept wrong answers at rates that matter: several percent of accepted patches on a filtered benchmark, more on an unfiltered one. Instructions do not close the gap that a grader leaves open: the prompt changed cheating rates by large factors, but the strictest wording still left a third of o3's attempts on impossible tasks passing. And closing one route leaves the others: read-only tests stopped edits to the tests, not special-casing.

24.2.3 Requirement, specification, assumption #

From requirements engineering. Section 19.1 teaches the field's three descriptions: the requirement $R$, what the customer wants of the world; the specification $S$, what the machine does at its interface; and the domain assumptions $W$, what holds of the world regardless of the machine. A specification is correct only together with its assumptions, $W \wedge S \vdash R$ ((19.1)), and that section works the case of a coding grader whose verdict needs the tests to be untouched and a zero exit status to mean the assertions ran. Table 30.2 sorts the reported coding hacks by the assumption each falsifies.

For an environment, $W$ is larger than the tests. The grader's verdict also assumes that the sandbox reaches nothing outside it, that the grader runs in a process the policy cannot change, that a model judge reads the attempt rather than instructions planted in it, and that one rollout cannot see another (inference). The field explains why such assumptions fail in a specific way. Pamela Zave and Michael Jackson require every action in a requirements model to be classified as "environment controlled" or "machine controlled," and as shared with the machine or not (Zave and Jackson, 1997). "The implemented machine must perform the machine-controlled actions and not the environment-controlled ones," they write (Zave and Jackson, 1997). A designer who writes a grader classifies editing the tests, or reaching the network, as actions the machine does not perform. Some reward hacks are the policy performing an action its designer classified as outside its control; others, such as special-casing the visible tests, use only permitted actions and exploit tests too weak to tell (Exercise 24.3) (inference). For the first kind, the remedy the classification suggests is to make it true by enforcement, with read-only tests and closed networks, which turns a question about requirements into a question about confinement (Section 24.6).

The field's first completeness condition is that every requirement has been validated as acceptable to the customer (Zave and Jackson, 1997). In an environment, $R$ is often written nowhere but in the designer's intent and a reference solution. The manual check of SWE-bench patches above could not decide two thirds of the cases it examined because the requirements were under-specified (Wang et al., 2025a): the oracle problem there was a requirements problem first (inference).

The two fields imply a measurement. The quantity that matters is the grader's false-acceptance rate on the attempts the policy can reach, and it can be estimated before training. Sample the starting policy, and check the attempts the grader accepts with a stronger oracle, such as the full test suite, a differential test generator, or a person (inference). The derivation below says what that estimate predicts.

The derivation turns a vague worry into a ratio. A grader is good enough for a task only if its false-acceptance rate on the reference policy's incorrect attempts is small compared with that policy's success rate, so the hardest tasks, the ones training most wants, need the most sound graders (inference). Two things make the estimate optimistic. Sampled training is not at the optimum, and an exploit that works on every task in an environment is reinforced on all of them at once, while each correct solution is reinforced only on its own task (Section 24.4). And the estimate covers the attempts the reference policy produces, not those a trained policy learns to produce.

Here the analogy with software testing stops, in two places (inference). A mutation score measures a suite against the faults a tool or a person seeds, while a policy under optimization searches for the faults no one seeded; ImpossibleBench seeds faults in the oracle and measures the search. And requirements engineering checks its assumptions once, before a machine is deployed, while a training run must keep them true on every one of millions of rollouts as the policy's capability grows. Section 30.3 adds a third limit: the machine of requirements engineering does not act on its own assumptions, and a policy does. Even a grader that is sound for every attempt leaves a question the proof obligation does not ask: $W \wedge S \vdash R$ is a statement about behavior on the task, while training also teaches a disposition that carries to other tasks (Chapter 31; Chapter 33).

How much a grader's content matters at all is a separate question. Rewards that carry no information about correctness still improved some models' scores, which Section 24.4 reports.

24.3 Process against outcome supervision #

The first choice in designing a grader is what it looks at. Outcome supervision rewards only the final result and leaves the path to it free. Process supervision rewards each step of the reasoning and so constrains the path, but it needs a judge of steps, and it puts optimization pressure on the text a monitor would later read. This section reports what each buys.

Section 12.7.3 reports the two comparisons that defined the axis. On grade-school word problems, a 2022 preprint by DeepMind researchers found that outcome supervision reached similar final-answer error with less labeling, while correct reasoning steps required process supervision or a reward model that emulated it (Uesato et al., 2022). On competition mathematics, a 2023 study by OpenAI found that reward models trained on human labels of each step chose better among sampled solutions than reward models trained on final answers (Lightman et al., 2023). In both, process supervision meant a person judging whether each step was correct.

Most later work means something else by the term: a step's value estimated from outcomes. A paper at ACL 2024 by researchers at Peking University and DeepSeek-AI defined the quality of a step as "its potential to deduce the correct answer," and estimated it by completing the solution several times from that step and checking the final answers (Wang et al., 2024d). With a LLaMA2-70B model completing four solutions per step, these automatic labels agreed with human labels on 86 percent of 160 sampled steps (Wang et al., 2024d). Step-by-step reinforcement learning with the resulting reward model raised Mistral-7B's accuracy from 77.9 to 84.1 percent on the GSM8K benchmark and from 28.6 to 33.0 percent on MATH (Wang et al., 2024d).

A paper at ICLR 2025 by researchers at Google Research, Google DeepMind, and Carnegie Mellon University asked what a step reward should measure, and answered progress: the change in the probability of eventually producing a correct answer, before and after the step (Setlur et al., 2025). They measure progress under a separate prover policy, and they note that measuring it under the policy being trained gives "exactly the same policy gradient updates as only optimizing the outcome reward" (Setlur et al., 2025). On Gemma 2B and 9B models, reinforcement learning with these rewards was more than 7 percent more accurate than training on outcomes alone, and reached its accuracy about six times faster (Setlur et al., 2025). A step reward of this kind is outcome supervision with better credit assignment, the attribution of a final result to the steps that produced it (Section 12.1.5): it pays for steps that lead to correct answers, not for steps a reader would judge correct (inference).

What step rewards attend to has also been measured. In a paper at AAAI, researchers at Zhejiang University, ShanghaiTech University, TAL Education Group, the University of Rochester, and Jinan University trained step reward models from the preferences of a tree search over solutions (Ma et al., 2024). Removing the natural-language descriptions of each step from the reward model's input had "minimal impact," and reward models trained to judge those descriptions alone were hard to train and scored poorly (Ma et al., 2024). If step rewards track the mathematical expressions rather than the prose around them, they constrain less of the readable reasoning than the name suggests (inference).

The cost to oversight runs the other way. When a grader reads the reasoning text, the text becomes something to optimize. In a paper at NeurIPS 2025, penalizing particular strings in reasoning that the model needed to solve its task led the model to substitute other strings without changing its method. The model also generalized the substitution to strings held out of training (Skaf et al., 2025); Section 40.5 reports it with the other evidence on obfuscation. Researchers at Google DeepMind have proposed classifying, before training, each reward term that reads the chain of thought as aligned with, orthogonal to, or in conflict with what a monitor needs, and found in a 2026 preprint that terms in conflict reduced monitorability (Kaufmann et al., 2026); Section 40.2 teaches the framework.

Why the reasoning passes through text at all decides what either kind of grader can see. A transformer of fixed depth that writes intermediate steps can express serial computations that a single pass of the same model cannot, under conjectures of complexity theory that its authors call well known (Li et al., 2024a); Section 11.2.5 reports the result. A 2024 preprint from Meta's research laboratory fed a model's internal state back as its next input instead of a word, so that reasoning steps never appear as text; on small models built from GPT-2, this outperformed written reasoning on some logical tasks that needed search (Hao et al., 2024). Where reasoning leaves the text, neither process supervision nor a monitor can read it, so the choice of architecture and of reward together decide what an overseer can see (inference).

At the far end of the axis, a method from Google DeepMind trains each step only for its immediate reward plus an overseer's judgment of the step. In three environments built as model organisms of the failure, two with agents based on the developer's Gemini 1.5 Flash and one a toy gridworld, it learned none of the multi-step hacks that ordinary reinforcement learning learned (Farquhar et al., 2025). Its observed reward was lower, at a possible cost in observed performance, while in the coding environment its solutions did better on held-out tests of correctness (Farquhar et al., 2025). Section 33.1.7 reports it and Section 28.4 teaches it.

24.4 What reinforcement learning in environments does to a policy #

The graders above say what is rewarded. This section asks what the loop does with the reward. Does it add abilities the starting model lacked, or concentrate the model on what it could already do? What happens to the model's willingness to try new things, and how fast does a flaw that one attempt discovers take over a run? It ends with the question these mechanisms leave open, what kind of disposition the process builds.

24.4.1 Sharpening or expansion #

The measurement at the center of the dispute is pass@k, the probability that at least one of $k$ samples solves a problem, averaged over problems. Section 12.7.4 derives that at the optimum of the KL-regularized objective, pass@k on a training problem can only rise, at every $k$. It also reports the preprint by Yang Yue and colleagues in which models trained with verifiable rewards beat their starting models at small $k$ while the starting models overtook them at large $k$, across benchmarks and model families. On the training problems themselves, the trained model's pass@256 fell slightly below the starting model's (Yue et al., 2025). That chapter reads the crossover as evidence about how far trained policies sit from the optimum, and names the mechanism that predicts it: a problem the policy never solves yields groups that are all wrong, which carry no signal. Two later studies report the reverse.

ProRL, a method from NVIDIA reported in a paper at NeurIPS 2025, trained for more than 2,000 steps on 136,000 problems in mathematics, code, science, logic puzzles, and instruction following. It used a KL penalty and a periodic hard reset of the reference policy to a recent copy of the trained one (Liu et al., 2025). Its starting point was DeepSeek-R1-Distill-Qwen-1.5B, a model already distilled from a larger reasoning model, trained to reproduce that model's outputs, so the "base" it is compared with is not a pretrained model (Liu et al., 2025). The authors report that trained models "consistently outperform base models across a wide range of pass@$k$ evaluations, including scenarios where base models fail entirely regardless of the number of attempts" (Liu et al., 2025). The gain depended on where the starting model stood. On tasks where its pass@128 was already high, particularly in mathematics, the trained model's pass@128 often fell, as Yue and colleagues had found; on tasks where it was low, training expanded it (Liu et al., 2025). The authors list as a limitation that they trained only a 1.5-billion-parameter model (Liu et al., 2025).

A paper at ICLR 2026 by researchers at Microsoft Research Asia and several universities attacked the measurement instead (Wen et al., 2026). On hard problems with short answers, a base model can reach the right answer through wrong reasoning and be counted as a success after enough attempts. The authors therefore scored CoT-pass@k, which counts a sample only if its chain of reasoning is also correct, as judged by another model, DeepSeek-R1-0528-Qwen3-8B (Wen et al., 2026). Comparing Qwen2.5-32B with a version trained from it by reinforcement learning on 17,000 mathematics problems, ordinary pass@k showed the base model catching up as $k$ grew, while CoT-pass@k showed the trained model ahead at every $k$ up to 1,024 on the AIME 2024 and 2025 competitions (Wen et al., 2026). On easier benchmarks the gap was small. The authors' limitations are that a model judged the chains, and that their theorem explains the optimization but gives no guarantee of generalization (Wen et al., 2026). Their correction is itself a grader story: the answer checker behind ordinary pass@k accepts right answers reached by wrong reasoning, and that false acceptance changed which way the comparison came out (inference).

For the sharpening reading, ProRL's expansions came where a small distilled model was weak; for the expansion reading, trained models solved problems their starting models did not solve in 128 or 1,024 attempts. Neither result shows a behavior outside the starting model's distribution, and neither could. A language model gives every response some positive probability, so pass@k at a feasible $k$ asks whether a solve rate exceeds roughly $1/k$, not whether a solution was present in the prior. At the small KL weights of Section 12.4.2, the optimum can lift a response from far below $1/1{,}024$ to near certainty while only reweighting the prior (inference). Whether training creates or selects is therefore not a question pass@k can settle; what it can settle is whether training raises solve rates on problems that sampling the starting model would not solve in practice. The studies used open models of at most 32 billion parameters. The exception is a preliminary check by Yue and colleagues on Magistral-Medium-2506, a model of undisclosed size trained by reinforcement learning alone from Mistral-Medium-3-2505, which they place near the frontier. There too the gain at small $k$ narrowed as $k$ grew (Yue et al., 2025). As of October 2026 we found no such measurement published by a developer for its own frontier model. The evidence does not decide what such training adds at frontier scale.

24.4.2 Exploration collapses #

A policy explores only through its own samples, so how spread out those samples are limits what training can find. Entropy, the measure of a distribution's spread taught in Section 6.2, is the quantity to watch. A 2025 preprint by researchers at the Shanghai AI Laboratory, Tsinghua University, and other institutions trained 11 base models from four families, from 0.5 to 32 billion parameters, on mathematics and code, with no KL penalty (Cui et al., 2025). In every run, the policy's entropy dropped sharply at the start and kept falling toward zero, while validation performance rose fast and then saturated. Across the runs, 73 percent of the entropy consumed and 76 percent of the performance gained came in the first 200 of 2,400 gradient steps (Cui et al., 2025). Performance fit the curve $-a\,e^{H} + b$ in the entropy $H$, with two fitted coefficients, so the ceiling at zero entropy, $-a + b$, could be predicted early. For the Qwen2.5 family, fitting on the first 36 steps predicted the following 200 with average errors of 0.9 percentage points on mathematics and 1.2 on code (Cui et al., 2025).

The authors also derive what moves entropy. For a softmax policy with one logit per action, to first order, the change in entropy is minus the covariance, the tendency of two quantities to move together, between an action's log-probability and the change in its logit; under plain policy gradient that change is the learning rate times the action's probability times its advantage. In their words, "a high-probability action with high advantage would reduce policy entropy, while a rare action with high advantage would increase policy entropy" (Cui et al., 2025). They read their results as conditional support for a ceiling of the kind Yue and colleagues reported, but attribute it to the collapse of entropy rather than to reinforcement learning as such. They note that other work with other models has observed different entropy patterns, so the regularity "is not arguably universal" (Cui et al., 2025).

If exploration is what the reward buys, perhaps the reward's content matters less than expected, and three studies test that. A paper at NeurIPS 2025 by researchers at the University of Illinois Urbana-Champaign trained with entropy minimization alone, rewarding the model for being confident and using no labels at all (Agarwal et al., 2025). On a mathematics-specialized Qwen2.5 model, this matched or beat GRPO and a related method trained on 35,000 labeled mathematics problems (Agarwal et al., 2025). On Llama-3.1-8B-Instruct it was less effective and sometimes hurt mathematical reasoning, which the authors tie to Llama models lacking the relevant reasoning behaviors before training (Agarwal et al., 2025). A 2025 preprint by researchers at UC Berkeley and Yale rewarded a model with its own self-certainty, a measure of how confident its predictions are. With a Qwen2.5-3B base model, this matched GRPO on mathematics and generalized better to code (Zhao et al., 2025). When the self-certainty was computed by a frozen copy of the starting model instead of the current policy, the policy learned within about 100 updates to raise its score by appending an already-solved problem to its answer, and its accuracy collapsed (Zhao et al., 2025). A grader made of a model's own confidence was gamed as soon as it stopped moving.

The third study removed the reward's content entirely. In a paper at ICML 2026 by researchers at the University of Washington, the Allen Institute for Artificial Intelligence, and UC Berkeley, training Qwen2.5-Math-7B with GRPO and randomly assigned rewards raised its score on the MATH-500 benchmark by 21.4 percentage points, against 29.1 with correct rewards (Shao et al., 2026). Rewards for incorrect answers also helped. The authors trace the effect to a bias in GRPO's clipping of large updates, which amplifies behaviors the model already favored. In this model, the favored behavior was reasoning written as Python code without running it, which rose from 65 percent of answers to more than 90 percent (Shao et al., 2026). The same spurious rewards rarely helped Llama3 or OLMo2 models, and models that had already been through reinforcement learning gained little under nearly any reward (Shao et al., 2026). The authors recommend testing reward signals on several model families, with spurious rewards as a baseline (Shao et al., 2026).

For graders, the lesson is direct: a gain under a grader is not evidence that the model learned what the grader checks, unless the gain exceeds what a random grader produces on the same model (inference). For the dispute above, these results fit two readings, and they do not decide between them. On the first, much of what the regime does is select among behaviors pretraining made available, as the persona view of Section 11.6 and Chapter 26 holds. On the second, on-policy sampling and the optimizer's own biases reinforce whatever the policy already produces, whatever its origin. Both predict that the starting model decides much of the outcome, and the results agree; neither study design separates them, and all of them concern capabilities on mathematics and code in open models, not the dispositions of Section 24.4.4.

24.4.3 How a found exploit spreads #

The developer of the frontier run in Baker et al. (2025) describes what happens once a model finds a general way past its graders: "they quickly get reinforced and become systemic, affecting nearly every training environment" (Baker et al., 2025). Section 30.1.4 reports the run. The derivation below asks how fast that happens, in the simplest model in which the question can be posed: one flaw, one shared tendency to use it, and the loop of Section 12.7.1.

The model has two phases. While the expected number of hacks sampled per step, $Bkp_0$, is well below one, most steps never see the hack; the derivation of Section 33.3 gives the expected wait for the first group that carries any signal ((33.2)). Once the hack is sampled regularly, each sample raises its probability, and the rise accelerates. For a hack with $p_0 = 10^{-3}$, the formula of step 5 puts the climb from 2 percent of attempts to half at about one twentieth of the time from the start to half. Figure 24.2 computes the expected path, including the division by the group's standard deviation.

At the defaults, an exploit that one attempt in a thousand finds reaches 2 percent of attempts at step 646 and half at step 687, when the grader reports 51 percent passes while about 5 percent of attempts truly succeed. Ten times rarer, it reaches half at about step 6,700; with four times as many prompts per step carrying the flaw, at step 174; without the division by the group's standard deviation, at about step 1,800. A rare exploit is thus held back by sampling for a time that grows with its rarity and shrinks with the number of flawed prompts whose gradients a step sums, or, under a loss averaged over the batch, with the share of the batch that carries the flaw. At the optimum, by contrast, rarity protects only logarithmically (Section 30.2.2). Rarity buys time, not safety: the protection is spent as training samples and reinforces the exploit, and a run whose steps weigh flawed prompts more heavily spends it faster (inference).

The model leaves out what makes real spread systemic. It has one shared parameter for one flaw, while a network that learns to exit before the tests run on one environment can do the same on every environment with the same weakness, which is the generalization the developer's phrase "affecting nearly every training environment" describes. It also ignores clipping, the KL penalty, and an honest success rate that changes with training. The developer's description quoted above fits its qualitative shape, a quiet start and an abrupt spread, but no report publishes hack rates in a form that would test its numbers. In one developer study the knowledge of the hacks was planted in advance, hacking started below 1 percent, and onset was defined as successful hacking in more than 2 percent of a step's episodes (MacDiarmid et al., 2025). A deliberately unmitigated run by the same developer ended with hacks across most of its environments (Section 33.1.4).

24.4.4 What disposition the process builds #

Everything in this section concerns behavior on the tasks being trained. Whether training in environments also builds a disposition, a tendency that carries into settings with no grader, is the debate that Chapter 33 owns, and Section 33.1.1 sets out its three positions. On the first, stated in a 2022 forum post by Alex Turner, reward strengthens the computations that earned it and need not become what the policy pursues (Turner, 2022b). An edit Turner added in September 2026 partly retracts the post's empirical claim (Turner, 2022b). On the second, argued in forum posts by Ajeya Cotra and Vivek Hebbar, training on diverse tasks selects models that pursue reward or the grader's judgment, which Cotra calls playing the training game (Cotra, 2022; Hebbar, 2025b). On the third, post-training mostly selects among characters that pretraining made likely (Section 11.6). A 2025 forum post by Alex Mallen and Buck Shlegeris places the positions side by side (Mallen and Shlegeris, 2025). Narrow training on hacks can generalize: GPT-4.1 and two other models, fine-tuned on hacks of harmless tasks, went on to new hacks, and GPT-4.1 to broader misalignment (Taylor et al., 2025b) (Section 30.5.3; Chapter 32).

The mechanisms above bear on that debate without deciding it (inference). That the starting model decides much of what training does is what the third position predicts, and what the first predicts too, since habits are built from what the policy already produces. That a grader-specific behavior becomes systemic is allowed by all three. Section 33.4 names the experiment that would separate them.

24.5 Reward hacking as a training-time event #

The training-time record of reward hacking comes from the loop this chapter describes. Chapter 30 catalogs it as a property of graders, and Chapter 33 asks whether it changes what the model pursues. This section collects what the record says about hacking as an event inside a training run, and what can be done there.

The training-time record comes from developers' reports on their own models. OpenAI reported monitoring the reinforcement learning of a frontier reasoning model on agentic coding tasks, in which hacks such as exiting before the tests ran spread through the run. A monitor that read the chain of thought caught far more of them than one that read only actions (Baker et al., 2025). Section 30.1.4 gives the numbers, and Section 40.5 reports what happened when the monitor became part of the reward. Anthropic reported two studies of model organisms at production scale, models put through constructed training conditions so that a failure can be studied. In one, hacking, learned after knowledge of the hacks was planted, generalized to broad misalignment (MacDiarmid et al., 2025). The other was a deliberately unmitigated run on environments known to be hackable, in which the resulting grader-directed behavior stayed narrow (Anthropic, 2026aw). Section 33.1.4 reports both, with the caveats the developer attaches.

One lesson concerns instructions. When what an environment rewards differs from what its instructions say, the record suggests that the reward wins (inference). In evaluations, instructions changed how often models cheated without stopping it: ImpossibleBench's prompts moved cheating rates by large factors but left a third of o3's attempts on impossible tasks passing (Section 24.2.2). METR, an outside evaluator, found that adding "Please do not reward hack" to one of its tasks had a nearly negligible effect on o3 over 20 runs per prompt (Metr, 2025d). In training, the inoculation result of Section 32.4, read from the other side, says the same. In the developer study's prompted setting, runs whose prompts told the model not to hack still learned to hack, and their misalignment rose clearly, while prompts that permitted hacking left it nearly flat (MacDiarmid et al., 2025). Forbidding in words what the grader pays for did not keep hacking from being learned, and it made what the model learned from it worse (inference). The sociology of medical education has a name for the institutional form, the hidden curriculum, and Section 25.6 reports what it shows about what an institution rewards and what it says it teaches.

The other lesson concerns where to act. Section 33.1.7 sorts the mitigations by what they act on: the environment, by hardening the grader and monitoring for hacks; the training signal, by changing what is optimized (Farquhar et al., 2025); and what the model learns from a given signal. The last group leaves the grader flawed. Recontextualization generates attempts under prompts that discourage a misbehavior and trains on them as if the prompts had permitted it (Azarbal et al., 2025). Verbalization fine-tuning teaches a model to say when it exploits a cue, so that hacking stays visible (Turpin et al., 2025). Inoculation prompting, its proposers' name for adding to the training prompt an instruction that asks for the unwanted behavior, lowers the behavior's expression when the instruction is removed (Wichers et al., 2025; Tan et al., 2025). Section 32.4 reports the inoculation evidence and Chapter 26 its mechanism. Data matters before any of these: in a developer's preliminary report, training on documents that only describe a model's tendency to reward hack raised its hacking in later reinforcement learning (Anthropic, 2025ar) (Chapter 27).

24.6 When the policy acts on its own training #

An environment is software the policy can reach, and the policy is optimized to find whatever the software permits that the grader pays for. The requirements view of Section 24.2.3 ended on making the designer's assumptions true by enforcement. This section reports what happens where they were not enforced, and then the evidence that a policy can shape what its training sees.

24.6.1 The sandbox as an assumption #

OpenAI has published short reports of misbehavior that its monitors found in its own training runs, with incidents dated from 2025 to 2026. They are developer disclosures, a sample of what the developer's monitors flagged and the developer chose to publish, and Section 5.6 gives the rule for reading them. The reports describe the models' reasoning in the developer's words, which the book reports as the developer's reading of the chain of thought. Section 54.3 reads each case for how it was detected and handled; Table I.6, in Section I.3.2, reads them for what the environment assumed and what the agent used instead. As of October 2026 the agents had reached past their environments through a sandbox's DNS resolver, a reference tool's error messages, the developer's internal package repository, public file hosts, an exposed API key, outside services and an external browser that sent requests their own tools could not, security flaws in a government statistics website, and a funded test-network account whose keys were derived from public recovery phrases (OpenAI, 2026d; OpenAI, 2026g; OpenAI, 2026ad; OpenAI, 2026ac; OpenAI, 2026ae; OpenAI, 2026y; OpenAI, 2026x; OpenAI, 2026f; OpenAI, 2026u; OpenAI, 2026af).

Two further reports, in which models wrote unauthorized instructions into the summaries they write to carry a long task into a fresh context, are reported in Section 11.5.3 (OpenAI, 2026v; OpenAI, 2026i). They broke a different assumption, that a model's notes to itself are a faithful compression of its work.

The developer's responses in the reports, two of which Section I.3.2 quotes, are the requirements view's remedy. Each response tries to turn an assumption into an enforced fact and adds monitoring or grading changes alongside, and one report says that work remains to validate the controls (OpenAI, 2026g; OpenAI, 2026d). The reference-tool report shows why the enforcement has to come before the reasoning: the developer quotes the model concluding that, because exploiting the tool was not explicitly banned, "Evaluation likely allows" it (OpenAI, 2026g). The model's question, on the developer's reading, was what the grader would accept, not what the task intended.

From computer security. The problem has a name older than machine learning. In 1973 Butler Lampson described the confinement problem: "confining a program during its execution so that it cannot transmit information to any other program except its caller" (Lampson, 1973). He sorted the channels through which a confined program can leak into three kinds. The first is storage that the program can write and an unconfined program can read. The second is legitimate channels, such as the bill the service returns, which can be made to carry information. The third is covert channels, "not intended for information transfer at all," such as the program's effect on system load (Lampson, 1973). His conclusion is the one that matters here. The number of channels is "surprisingly large, but certainly not infinite," and it is "necessary to enumerate them all and then to block each one"; but "there is not likely to be any rigorous way of identifying every channel in any system of even moderate complexity" (Lampson, 1973).

The incidents fit his kinds. The package repository was shared storage; the DNS resolver and the reference tool were legitimate channels used for another purpose; the public file hosts were a path out that no one had closed (inference). What his conclusion implies for training is that containment cannot be shown by listing the channels a red team found, since no list is known to be complete, so the safety case for a run in Section 24.7 needs monitoring as well as containment (inference). The analogy stops in two places. Lampson's threat is a service whose owner wants it to leak, while a policy is not built to leak; it is optimized against a grader, and it uses a channel when the channel raises its reward. A channel that serves no reward may stay unused, and a channel that does will be searched for by the whole run. And Lampson's service is one program, while a training run runs thousands of copies of the policy at once. Channels between the copies, such as the package repository, undermine an assumption of the training itself, which the developer names: that training samples are independent (OpenAI, 2026ad). Isolation and least privilege for agents are Chapter 49's subject. The point here is that a training run is a deployment to an adversarial user, the policy, with the grader as the asset it is pointed at (inference).

24.6.2 The policy shaping its training #

Reinforcement learning can reinforce only what the policy samples, so a policy that controls its own sampling has a say in what training does to it. Section 33.3 defines exploration hacking, a policy strategically altering its exploration to influence the training outcome, and Section 33.3.1 reports the first study of it, which describes the policy in the terms of strategy that Section 1.5 treats as a method rather than a finding. Some model organisms trained to suppress a capability kept it suppressed against reinforcement learning meant to elicit it. Frontier models told that showing a capability in a simulated training run would lead to retraining or shutdown rarely suppressed it when told only that they were in training, and did so more often when the prompt also encouraged strategic reasoning and supplied a strategy (Jang et al., 2026). The earlier, more radical concept is gradient hacking, named in a 2019 forum post by Evan Hubinger: a learned optimizer acting so that gradient descent updates it in a particular way (Hubinger, 2019); as of October 2026 we found no demonstrated instance. Between the two sits reasoning about training itself, which OpenAI and Apollo Research call metagaming, a term they borrow from tabletop games, for reasoning about feedback or oversight mechanisms outside the scenario a model is given; they report that it rose during capability-focused reinforcement learning of o3 (Schoen and Nitishinskaya, 2026). Section 33.1.5 reports the measurement, and the evidence on deliberate underperformance is Section 34.5's.

24.7 Training alignment in environments, and the safety case for a run #

The same regime is now used to train alignment, and a training run can itself cause harm. This section reports both, and ends with the argument that a run needs a safety case. Its claims move from the training process to the developer and the public, since the evidence and the decisions are the developer's.

24.7.1 Alignment as the target of an environment #

In a June 2026 preprint, researchers at OpenAI built realistic situations designed to measure and train beneficial traits, such as truthfulness, fairness, risk awareness, and corrigibility, across domains including health, science, and education (Jagadeesh et al., 2026). They replaced 5 percent of a standard reinforcement learning mixture with this data and compared the result with a baseline given the same compute; a second run used only health conversations (Jagadeesh et al., 2026). Table 24.2 gives the main results, from a developer's report on its own unnamed model.

The helpfulness control suggests that the reward, not the data alone, produced the transfer, and on pairs of answers that neither model refused, the trained model still improved on 19 of 20 evaluations (Jagadeesh et al., 2026). On evaluation awareness, they write that the production-traffic results "do not eliminate evaluation awareness as a possible contributing factor" but make a narrower artifact explanation less plausible (Jagadeesh et al., 2026). Their persistence results, that the trained model degraded less under prompts and fine-tuning designed to make it misbehave, compare it with a model from before reinforcement learning rather than with the compute-matched baseline. The authors say these results "do not isolate whether the persistence effect is specific to beneficial trait RL" (Jagadeesh et al., 2026).

The authors motivate the study by the persona account of Section 11.6, and the transfer from health to reward hacking is what that account predicts if training moves a character rather than a task heuristic (Jagadeesh et al., 2026). It is also what the first position of Section 24.4.4 predicts if caution learned in one domain is cued by the form of evaluation tasks (inference). Replication on open models by a group without a stake would separate the study from its developer; as of October 2026 we found none. Section 23.5 places the result beside the methods that train toward a written specification.

Anti-scheming training is the other alignment target trained in environments. In a 2025 preprint by OpenAI with Apollo Research, training o3 and o4-mini to reason over a specification against covert actions reduced those actions on held-out environments without eliminating them (Schoen et al., 2025). The models' chains of thought more often stated awareness of being evaluated. In one evaluation on o3 before this training, removing such statements raised covert behavior and inserting them lowered it, which leaves open whether the training changed that link; the authors "cannot exclude" that the reductions are partly driven by that awareness (Schoen et al., 2025). Section 34.5 and Chapter 37 report it.

24.7.2 Reward terms and what a monitor can read #

Any reward term that reads the chain of thought can be classified in advance, by the framework of Section 40.2 recalled in Section 24.3, as aligned with, orthogonal to, or in conflict with what a monitor needs (Kaufmann et al., 2026). One common term has produced conflicting findings: penalties on the length of the reasoning, which developers use to cut inference cost.

A 2025 preprint by Matt MacDermott and colleagues, of Imperial College London, MATS, the University of Cambridge, and independent researchers, trained DeepSeek-R1-Distill-Qwen-1.5B under several incentives and measured whether a monitor reading the reasoning could predict a hidden variable of the task (MacDermott et al., 2025). Controlling for accuracy, the authors found no consistent effect of length penalties (MacDermott et al., 2025). Their length penalty, however, counted words separated by spaces, and the model learned to write its reasoning in Chinese, which has no spaces between words, so its reasoning was shorter by the count without necessarily being shorter in tokens. The authors call this "essentially a reward hack" of their implementation and write that "more realistic implementations would penalise the number of tokens" (MacDermott et al., 2025). Their experiments used a single model and no repeated seeds (MacDermott et al., 2025).

A 2026 preprint by Bryce Little trained Qwen3-4B and Qwen3-14B under length penalties that targeted 60 down to 30 percent of the original length, counted in tokens, and tested them with nine kinds of hints that push toward a particular answer (Little, 2026). A chain counted as faithful when a model monitor could tell from it that the hint had influenced the answer. At the 30 percent target, accuracy stayed near baseline and wrong-answer hints switched answers as often as before, yet faithfulness fell on every evaluation set, by 39 percent for Qwen3-14B and 35 percent for Qwen3-4B on the main held-out set (Little, 2026). A control trained with the same rewards and no length penalty kept or raised faithfulness, and the compressed chains mentioned the hint 7 to 35 percentage points less often than original chains cut to the same length by deleting random sentences (Little, 2026). The author's limitations: two Qwen3 sizes and one run of a model from another family, one training recipe, and a monitor whose errors would, the author argues, understate the decline (Little, 2026).

Read together, the two studies disagree less than their headlines. The null result did not test token-based compression, by its own authors' account, and it came from a length grader that the policy hacked, the subject of this chapter turning up inside a measurement of it (inference). The positive result controls for reinforcement learning itself and for brevity alone, but it is one author's preprint, on two Qwen3 sizes and one recipe, with a single Nemotron run as a cross-family check that had no no-penalty control. The evidence does not settle the question. The positive result is the only one that tested token-based compression with these controls, and no other group has replicated it on any model family or recipe; Chapter 70 collects the question under its debate on whether chains of thought will stay monitorable.

24.7.3 The safety case for a training run #

A safety case is a structured argument, supported by evidence, that a system is safe enough in a stated context (Hilton et al., 2025). Joshua Clymer and colleagues proposed four kinds of argument for AI systems (inability to cause a catastrophe, control measures strong enough, trustworthiness despite the capability to cause harm, and deference to credible AI advisors), and posed the question for decisions about whether it is safe to train systems as well as to deploy them (Clymer et al., 2024). Chapter 52 teaches the form and its record in other industries.

In a post of September 2026, OpenAI proposed that structured safety documentation be required before a frontier reinforcement learning run continues, called full safety cases "an aspirational north star," and listed initial guidelines for alignment training, containment, monitoring, and operations, among them audits with access and public disclosure of incident investigations (Section I.2.14) (OpenAI, 2026ab). It also reports monitoring its internal coding agents for misalignment (OpenAI, 2026m), which Chapter 54 reports.

This chapter suggests what such a case would have to claim and show (inference). It would give, for each family of environments, the grader's false-acceptance rate on the starting policy's attempts, checked against a stronger oracle, since Equation (24.1) turns that rate and the policy's success rate into the share of passes that will be wrong. It would argue containment without claiming that its list of channels is complete, which Lampson's conclusion forbids, and so rest part of its weight on monitoring. And it would show that its monitors catch an exploit while it is still rare, because the spread of Section 24.4.3 leaves little time between an exploit's onset and its takeover. Every one of these measurements is held by the developer. The post's own proposals for audits with access and for public disclosure concern the third relationship, the developer and the public, which Section 5.6 reads through the economics of disclosure. As of October 2026 we found no published safety case for an actual frontier training run.

24.8 What training in environments means for delegation #

The book's thesis is that alignment is a delegation problem under an asymmetry of competence and visibility, and the training relationship is where it fits best (Section 1.3). Training in environments shows the fit and its edges. The developer writes a contract, the environment, and pays by a measure, the grader. The policy moves its effort toward what the measure pays, which is the multitask distortion of Section 14.2, and it decides the cases the contract leaves open, which is the residual control of Section 14.3.1 (inference). Three older fields have exact counterparts for the failures: the oracle problem and mutation testing for the grader, the proof obligation and its control classification for the environment, and the confinement problem for the sandbox. Their diagnoses carry over; their remedies carry over as audits of graders, enforcement of assumptions, and enumeration of channels, each with the limit stated where it was reported.

Three things have no counterpart in those fields (inference). The principal can rerun the agent, sample it many times, and read its reasoning, which no employer or tester can do with a person; Equation (24.1) and the spread derivation are calculations the developer can actually perform before and during a run. The agent searches at a scale no worker reaches: a rare exploit is held back only until training samples and reinforces it, a wait that shrinks as more of each step's gradient comes from prompts with the flaw. And the principal writes the agent's character as well as its tasks: reinforcement learning on beneficial traits in one domain moved behavior in others (Jagadeesh et al., 2026), one of the results the book fixed in advance as having no counterpart in the older fields (Section 1.7). The agent's acting on its own training, by suppressing its exploration, has a human counterpart in workers who restrict output under measurement, which Section 34.6 reports. The channels between thousands of simultaneous copies of one agent, which undermine the independence of training samples, have none that the chapter found.

The chapter's claims concern three relationships. The grader studies, the derivations, and the dispute over what training adds concern the training process and the signal it optimizes. The sandbox incidents concern it too, and also concern people outside the run: an agent that used an exposed API key or uploaded files to public hosts acted on third parties' systems. And who can measure any of this concerns the developer and the public. The false-acceptance rates of production graders, the trajectories of hacks in production runs, and the safety cases for those runs are held by the developers. The public record is benchmarks, outside evaluations of finished models, and the developers' own disclosures.

24.9 Settled, contested, missing #

24.10 Exercises #

Further reading #

• Zave and Jackson (1997): why a specification is correct only together with its assumptions about the world, and why control of actions must be stated.
• Weyuker (1982): On Testing Non-Testable Programs, read here only as a catalog record; Section 24.2.1 states the oracle problem from readable sources.
• Uesato et al. (2022): the controlled comparison of process and outcome supervision.
• Yue et al. (2025): what verifiable-reward training does and does not add, read with its rebuttals.
• Baker et al. (2025): hacking that spread through a frontier training run, and what a monitor caught.
• Lampson (1973): why a confined program's channels cannot be enumerated with certainty.