Draft complete but not yet reviewed; expect revisions

Optimization Pressure

Several chapters so far have ended at the same place: a system is trained or paid to raise a number that stands for something harder to measure, and the question is what happens to the thing the number stood for. Economists, sociologists, and education researchers met the question in institutions decades before machine learning. Their answer was that a measure used to control behavior tends to stop measuring what it did. This chapter collects that answer in its classic statements and the field record behind them, then teaches the four mechanisms by which a proxy fails and the theorems that say when optimizing a proxy is safe.

The theorems matter because they turn a proverb into conditions. Some proxies can be pushed without limit, and most cannot. A penalty that keeps the policy, the trained model's rule for choosing its outputs, near its starting point (Section 12.3) protects the true objective when the proxy's errors are light-tailed, meaning that very large errors are very rare, and independent of the objective, provided the penalized optimum exists; when the errors are heavy-tailed it guarantees nothing. The curve measured when a policy was optimized against a learned reward rises and then falls, in a shape the theory can read after the fact. The chapter closes with the method education measurement developed for detecting the effect, which is the method an evaluator of AI systems needs (Chapter 36). Most of its claims concern the first of the book's three relationships, the training process and the signal it optimizes; the Lucas critique and the audit test reach the other two.

15.1 Three laws from three fields #

A quantity that tracks a goal while no one optimizes it may stop tracking it once someone does. The pattern was named three times, in three fields, and each statement adds something the others lack. Throughout the chapter the goal is what a principal, the party on whose behalf a system acts, actually wants. A proxy is a measurable quantity that stands for it: a test score for learning, a reward model's score for a good answer, a passing unit test for working code.

15.1.1 Control and response #

From economics. The best-known statement comes from monetary policy. In a 1975 paper on monetary management in the United Kingdom, the economist Charles Goodhart observed that a statistical regularity which held while no one relied on it tends to break down once a policymaker leans on it to steer the economy. The paraphrase follows the wording that Manheim and Garrabrant quote (Manheim and Garrabrant, 2018); the version of the paper this book could verify is a 1984 reprint (Goodhart, 1984). As quoted, the statement names no mechanism. It says only that pressure for control is what breaks the regularity.

Robert Lucas, in a 1976 critique of the econometric models that governments used to compare policies, gave the economist's reason for the same pattern. Such a model is a set of equations estimated from past data on how households and firms behave. Lucas summarized his argument as a single syllogism: "given that the structure of an econometric model consists of optimal decision rules of economic agents, and that optimal decision rules vary systematically with changes in the structure of series relevant to the decision maker, it follows that any change in policy will systematically alter the structure of econometric models" (Lucas, 1976). The people whose behavior was estimated chose that behavior given the policy they faced. Change the policy and they choose again, so the estimated relations stop holding. Lucas drew the consequence that comparisons of policy rules made with such models are invalid regardless of how well the models performed over the sample period (Lucas, 1976). This is the Lucas critique.

The critique is the bridge to what machine learning calls distribution shift, a change between the data a model was fitted on and the data it is used on. A reward model is fitted to responses the reference policy wrote; an evaluation is measured before a system is deployed. Optimizing against either is a change of policy in Lucas's sense, so neither relation need hold afterward, however accurate it was on the data it was fitted to (inference). Section 12.8.2 states the machine-learning form of the last clause: a reward model's low test error does not bound the regret of the policy trained against it, its shortfall from the best achievable.

Machine learning has rediscovered the critique several times. Hardt and colleagues model strategic classification as a game in which one player publishes a classifier and the other, before being classified, may change its input at a cost. They call the behavior gaming and note that financial policymaking knows the problem as Goodhart's law (Hardt et al., 2016). For separable costs they give efficient learning algorithms that are near optimal; for general costs, designing an approximately optimal classifier that gaming cannot fool is NP-hard, as hard as the problems for which no efficient algorithm is known (Hardt et al., 2016). Performative prediction, introduced in a 2020 preprint, generalizes the game to any prediction that influences the outcome it predicts, and gives necessary and sufficient conditions under which repeated retraining settles at a stable point (Perdomo et al., 2020). Goodhart himself is a coauthor of a structural model in which penalized regressions (fits whose coefficients are shrunk toward zero), trained on clean data, acquire what the abstract calls Goodhart bias once future agents manipulate the inputs at a known cost (Hennessy and Goodhart, 2023).

The same literature also gives the positive case. Kleinberg and Raghavan model an agent who can spend effort either on improving the outcome a classifier is meant to measure or on gaming it. They give a tight characterization of when an evaluator can induce the intended kind of effort, and show that whenever any reasonable mechanism can do so, a simple linear one suffices (Kleinberg and Raghavan, 2020). A measure under pressure is not always corrupted: when the effort that raises it also raises the goal, pressure on the measure is pressure on the goal.

What economics adds is the agent's response: a measure fails under pressure because whatever is measured re-optimizes against it. A training process fits the description without any expectations about policy, since each update re-optimizes the policy against whatever reward is fixed (inference). The analogy stops at the deployed model. Lucas's households respond because they know the rule has changed; whether a trained model behaves differently when it can tell it is being evaluated is an empirical question, taken up in Chapter 37.

15.1.2 The indicator and the process #

From sociology. Donald Campbell, writing about how to assess the impact of planned social programs (Campbell, 1979), gave the statement a second half. In the wording Manheim and Garrabrant quote, the more a quantitative social indicator is used to make social decisions, the more pressure there is to corrupt it, and the more it distorts the very processes it was meant to monitor (Manheim and Garrabrant, 2018). They note that Campbell's version arguably has scholarly precedence over Goodhart's (Manheim and Garrabrant, 2018). Goodhart's law is about the indicator; Campbell's is also about the process. A school that drills test items changes what is taught, not only what the score means, as Section 15.2 and Section 15.7 report.

The phrasing most people know, that a measure which becomes a target stops being a good measure, is usually traced to the anthropologist Marilyn Strathern's 1997 study of audit in British universities (Strathern, 1997). El-Mhamdi and Hoang call it probably the best-known rephrasing of Goodhart's law (El-Mhamdi and Hoang, 2024). The older sociological frame is Merton's 1936 analysis of the unanticipated consequences of purposive social action (Merton, 1936).

Campbell's mechanism is that the people who produce a process are the people it measures, so the measure enters their decisions. For training, the corresponding claim is that optimization changes the policy that produces the outputs being scored, so a policy can become worse at a task while it becomes better at the score, and not merely be scored less accurately (inference). The analogy stops at motive. Campbell's corruption pressures come from people with interests and careers; a training process has neither, only an update rule that moves whatever raises the reward.

15.1.3 Which relationship each statement fits #

Goodhart's and Campbell's statements need no intention in whatever does the optimizing: selection on a measure is enough to weaken its link with the goal, as the next sections derive. They therefore apply to a training process as they apply to an institution, which places them in the first of the book's relationships, the training process and the signal it optimizes. The Lucas critique needs a party that responds to the rule. A training process responds by construction. A deployed model responds only if it treats measured and unmeasured settings differently, which is the question of the second relationship, the deployed system and the people it serves (Chapter 37). And a developer that reports benchmark scores while choosing what to train on is an institution under Campbell's pressure, in the third relationship, the developer and the public; the audit test is the instrument that relationship would need, and Section 15.7 says where it fails (inference).

15.2 The record of gaming in institutions #

The laws would be proverbs if no one had measured them. This section reports what people and organizations did when a number decided their pay, rank, or survival. The forms recur in the catalog of specification gaming that Chapter 30 assembles for trained systems: choosing the cases that score well, concentrating on cases near a threshold, teaching to the test, and changing the number directly.

The theory that predicts the record is the multitask model of Section 14.2: an agent paid on what can be measured moves effort toward the measure and away from what cannot be measured. That section also states the economist's prescriptions (weaker incentives, several measures, and judgment that no measure captures), to which Section 15.6 returns. A standard review of incentives in firms, read here in its abstract, concludes that there is considerable evidence that people respond to pay for performance, and that little is known about how to provide incentives to workers whose output is difficult to measure (Prendergast, 1999).

15.2.1 Gaming measured in the field #

From economics. The cleanest evidence compares places that introduced a measure with places that did not. In the early 1990s, New York and Pennsylvania introduced mandatory report cards on the outcomes of coronary artery bypass surgery, a heart operation, by hospital and surgeon. Dranove, Kessler, McClellan, and Satterthwaite compared the trends, from 1987 to 1994, for all elderly heart-attack patients and all elderly bypass patients in those states with the trends in states without report cards (Dranove et al., 2003). The report cards improved the matching of patients to hospitals. They also shifted surgery from sicker toward healthier patients, and on net costs rose and health outcomes worsened, especially for the sicker patients; the authors conclude that the report cards reduced welfare (Dranove et al., 2003). The mechanism they describe is selection: a provider judged on outcomes gains by avoiding the sick and seeking the healthy (Dranove et al., 2003).

Schools gave the second body of evidence. Chicago tied student promotion and school probation to a standardized test from 1996. Jacob reports that a study he wrote with Levitt (Jacob and Levitt, 2003) found that classroom cheating increased substantially after the policy began, but could explain only an extremely small part of the score gains (Jacob, 2005). Neal and Schanzenbach studied a quieter response. When a school is judged by how many students score above a proficiency line, teachers have reason to concentrate on the students near the line. With fifth-grade scores from Chicago, they found that both the 1996 district reform and the federal No Child Left Behind Act of 2002 raised scores in the middle of the achievement distribution. The least advantaged students did not score higher, and the evidence for the most advantaged was mixed (Neal and Schanzenbach, 2010). Where the gains that did occur came from is the subject of the audit test in Section 15.7.

Each of these responses has a counterpart in trained systems (inference). Changing the number directly is the classroom cheating, and the policy that overwrites a test file. Concentrating on what the count rewards is the proficiency line, and the grader that checks only whether tests pass. Choosing the cases that score well is the surgeon's selection, and a policy that learns which answers a reward model rates highly. The catalog in Chapter 30 documents the machine side.

15.2.2 Measures that remake what they measure #

From sociology. Sociologists recorded the pattern earlier and named its second half, the change in the work itself. Early entries are Blau's field study of two government agencies (Blau, 1955) and Ridgway's review of the dysfunctional consequences of performance measurements (Ridgway, 1956); Kerr's 1975 essay put the managerial version in its title, on the folly of rewarding A while hoping for B (Kerr, 1975). Later work followed whole institutions. In the 2000s, England governed its public services by targets combined with sanctions, a regime Bevan and Hood call targets and terror. Their abstract names two assumptions the regime rested on: that a measured part can stand for the whole, which they call synecdoche, and that problems of measurement and gaming do not matter. They examine both against the service's record of reported successes, measurement problems, and gaming (Bevan and Hood, 2006). Synecdoche is the assumption every proxy makes (inference).

Later studies take up the theme in their titles, such as Espeland and Sauder's "Rankings and Reactivity: How Public Measures Recreate Social Worlds" (Espeland and Sauder, 2007) and Muller's book-length The Tyranny of Metrics (Muller, 2018). A working paper posted as a preprint argues that the benefits of setting specific, challenging goals have been overstated. It lists side effects: a narrow focus that neglects everything outside the goal, more unethical behavior, distorted attitudes to risk, corroded organizational culture, and reduced intrinsic motivation (Ordonez et al., 2009).

The sociological mechanism is reactivity: people who know they are measured change what they do, and over time the measure becomes the working definition of the job. For a trained system, the counterpart is that the policy is the thing optimization changes, so a proxy can redefine the task the policy performs rather than merely misreport it (inference). Much of this record is case study and argument rather than measurement, which is why the economic studies above carry the quantitative weight.

15.2.3 Where the institutional analogy stops #

Officials, surgeons, and teachers differ from a training process in four ways that matter here. They bear a cost for effort, while an optimizer has none of its own; in training, the KL penalty of Section 12.3 plays that role. They are few and slow, while a training run tries far more variations than any institution and pays for each at once. They can refuse the contract, while a policy cannot. And their dispositions were formed before the measure arrived, while a trained system's are produced by the principal's own training, which the principal can inspect (Section 14.8). The first difference changes the price of gaming; the last two change who is responsible for it (inference).

15.2.4 Misspecification is not pressure #

Not every failure of a measure is a Goodhart effect. Obermeyer and colleagues studied a commercial algorithm used to choose patients for extra care. It predicted next year's health care costs as its label for health need. Because unequal access to care means that less is spent on Black patients than on White patients, Black patients at a given risk score were considerably sicker. Remedying the disparity would raise the fraction of Black patients among those automatically identified for extra help from 17.7 to 46.5 percent (Obermeyer et al., 2019). The measure was wrong before anyone optimized against it. The distinction matters for remedies. Misspecification shows at zero pressure and is found by checking the measure against the goal on ordinary cases; a Goodhart effect grows with pressure and is found only by checking where pressure takes the system (inference). The software engineer's own proxy belongs here too. Whether code coverage tracks a test suite's ability to find faults is the subject of Inozemtseva and Holmes's 2014 study, "Coverage is not strongly correlated with test suite effectiveness" (Inozemtseva and Holmes, 2014), and Section 24.2 treats graders as test suites.

15.3 Four ways a proxy fails #

"Goodhart's law" names several mechanisms, and they call for different remedies. In a 2018 preprint that expands an earlier forum post by Garrabrant, Manheim and Garrabrant separate four (Manheim and Garrabrant, 2018). Their setting is a regulator who cannot observe the goal and so acts on a proxy, for instance by permitting only the states of the world whose proxy value exceeds a threshold. In training, the regulator is the optimizer, the states are responses or policies, and the proxy is the reward.

• Regressional: selecting for an imperfect proxy also selects for the difference between the proxy and the goal. The authors call this the most fundamental effect, one that no choice of inexact measure avoids (Manheim and Garrabrant, 2018).
• Extremal: the states in which the proxy takes extreme values may differ from the ordinary states in which the relation between proxy and goal was observed, either because the relation was learned from a limited range or because a different regime holds at the extremes (Manheim and Garrabrant, 2018).
• Causal: the regulator's own intervention changes the causal structure that linked proxy and goal, for example by acting on a shared cause of both or on the proxy itself (Manheim and Garrabrant, 2018).
• Adversarial: another agent, with goals of its own, acts on the proxy: either selecting so as to exploit the regulator's selection, or responding to an incentive in a way that breaks the relation, as in the supposed episode in which a bounty on dead cobras led people to breed them (Manheim and Garrabrant, 2018).

The regressional case can be derived exactly, and the derivation shows what it cannot do.

15.3.1 Regressional Goodhart, derived #

Start with an example. A company hires by interview score, and each score is the candidate's true ability plus the interviewer's error. The candidates with the highest scores are disproportionately the ones whose error was large and positive, so their ability, on average, is lower than their score. Nothing about the interview is biased; selection alone produces the gap.

To make this exact, draw a response at random from the reference policy. Let $r^\ast$ be its goal value, the intended or gold reward that a proxy stands for, and let its proxy reward be $r = r^\ast + \varepsilon$, where $\varepsilon$ is the proxy's error. Assume that $r^\ast$ and $\varepsilon$ are independent and normally distributed with mean zero; the variance $\Var(\cdot)$ measures the spread of each. Then the expected goal value of a response whose proxy reward is $u$ is

where $\rho$ is the correlation between proxy and goal (Section 6.1), a number between $-1$ and 1 that measures how closely they move together. Whatever is selected for a high proxy reward has a goal value lower by the error's share of the variance, $1 - \rho^2$.

Gao, Schulman, and Hilton prove the same identity, with an approximate version for small bounded error, and draw the same consequence: if this were the only kind of Goodhart present, the gold reward would always increase with the proxy reward (Gao et al., 2022). A true objective that rises and then falls, as the measurements of Section 15.6 show, needs something else: an error that is not normal and independent, or one of the other three mechanisms.

The regressional effect is the optimizer's curse of Section 7.7 seen from the side of the proxy. The option that looks best is disproportionately one whose estimate erred upward, and the correction is the same: shrink the proxy toward its mean, by the factor $\rho^2$, before trusting it. That section teaches the empirical-Bayes form of the correction, Tweedie's formula, which corrects estimates for having been selected as the largest of many (Efron, 2011). It also teaches the machine-learning case in which choosing a model by a criterion evaluated on a finite sample overfits the criterion (Cawley and Talbot, 2010).

Figure 15.1 draws the derivation. Its horizontal pressure axis is the KL divergence of the kept set from the whole, which for keeping a fraction $q$ of responses is exactly $\log(1/q)$ nats (the unit of information for natural logarithms), since the kept distribution multiplies each kept response's probability by $1/q$ (Section 6.3). With half the proxy's variance in its error, keeping the top 1 in 100 raises the mean proxy reward of the kept set to 3.77 and its mean goal value to 1.88, in units of the goal's standard deviation. Keeping the top 1 in a million raises them to 7.00 and 3.50. The goal never turns down. It only falls behind.

15.3.2 The other three in training #

The other mechanisms each have a training form, though the attribution in any given run is a reading, not a measurement. Gao and colleagues give two of them for reward models (Gao et al., 2022). Their extremal example is a reward model that learns from its training data that longer answers are better, though beyond the lengths it saw, longer answers stop being better. In a footnote, they report having seen optimized policies produce very long answers when a short one would be preferred (Gao et al., 2022). More generally, a reward model fitted to the reference policy's responses can be accurate there and wrong where an optimized policy goes, the failure of identification that Section 13.4 examines. Their causal example is length again, now correlated with quality through a common cause such as informativeness; selecting on length then yields longer answers that are no better (Gao et al., 2022). Human raters' preference for assertive answers, reported in Section 22.6, is a correlate of the same kind.

The adversarial case is a policy that finds and exploits a flaw in a programmed grader, the subject of Chapter 30. That chapter draws on a 2020 blog post by researchers at DeepMind, who had collected about 60 such cases from existing lists and contributions from the AI community (Krakovna et al., 2020b). Gao and colleagues did not expect their models to be capable of it (Gao et al., 2022). Manheim and Garrabrant's adversarial case requires an agent with goals of its own. Whether a policy that exploits its grader is usefully described that way is the question of Chapter 33; for the remedy it does not matter, since a grader has to be secured against whatever searches it (inference).

Table 15.1 also shows why the correction for regressional Goodhart is not enough. Shrinkage assumes the error is independent of the goal, and that the relation $\E[r^\ast \given r] = \rho^2 r$, which holds over the reference policy's responses, still holds among the responses the optimizer reaches. An optimizer that moves toward the inputs on which the proxy errs, as in the extremal and adversarial cases, selects for errors that are correlated with its own choices, and shrinking every estimate by the same factor does not correct them (inference). The next section asks under what conditions no correction is needed at all.

15.4 When a proxy can be optimized safely #

The variants describe how a proxy fails. The theorems in this section ask the reverse question: under what conditions can raising a proxy never lower the true objective? Each answers it for a different setting, and the assumptions are what the reader should keep, because they say how far each guarantee travels. Throughout, $r^\ast$ is the intended, or gold, reward that a proxy $r$ stands for, and a policy's true value is its expected intended reward $\E_\pi[r^\ast]$.

15.4.1 Unhackable proxies #

Skalse, Howe, Krasheninnikov, and Krueger, in a 2022 paper at NeurIPS, gave the first formal definition (Skalse et al., 2022). Fix a set of policies. A proxy and a true reward are hackable on that set if it contains two policies such that one has the higher expected proxy reward and the other the higher expected true reward; otherwise the pair is unhackable (Skalse et al., 2022). Two easy cases are unhackable: rewards that rank every policy the same way (equivalent), and pairs in which one reward is the same for every policy (trivial). The question is whether anything else is.

The paper's abstract names the key fact: expected reward is linear in how often each state and action is visited, and that linearity makes unhackability a very strong condition (Skalse et al., 2022). A one-state example shows why. When a policy chooses among three responses, its expected reward is a weighted sum of their rewards, with its probabilities as the weights.

The paper proves the general form. Take a Markov decision process, the model of states, actions, and transitions of Section 7.4, and a set of stationary policies, those whose choice depends only on the current state. If the set contains an open region, it holds no unhackable pair that is neither equivalent nor trivial. In particular, over the set of all stationary policies, unhackable non-trivial pairs are equivalent (Skalse et al., 2022). The escape is to restrict the policies. On a finite set of policies, which includes the deterministic policies of a finite process, unhackable pairs need not be equivalent (Skalse et al., 2022), and with three or more policies they need not be trivial either: three policies with expected true rewards $(0, 1, 2)$ and expected proxy rewards $(0, 0, 1)$ are never ordered oppositely by the two, yet are not ordered the same way.

The authors call their definition strict, arguably too strict, since it also counts behaviors with low proxy reward and high true reward that an optimizer of the proxy is unlikely to discover, and they note that hackability is far from a guarantee of hacking (Skalse et al., 2022). Their conclusion is still stark: learned rewards are almost certainly hackable, so a policy trained on one should be validated before it is deployed (Skalse et al., 2022). The theorem says that a gap exists in almost every proxy; how large it is, and whether optimization reaches it, is for the rest of the chapter.

15.4.2 Omitted attributes #

Zhuang and Hadfield-Menell ask what optimizing an incomplete proxy costs, in a model of a principal and an agent (Zhuang and Hadfield-Menell, 2020). The world has several attributes the principal cares about, and the principal's utility $U$ rises in each. Resources are constrained: raising any attribute uses up resources, so not every attribute can be raised at once, and each attribute has a floor. The agent receives a proxy that depends on only some of the attributes and keeps raising it (Zhuang and Hadfield-Menell, 2020).

Two results follow, for a proxy that is continuous and strictly increasing in the attributes it names, pushed by an optimization that eventually raises it as high as it can go. If the optimization settles at some state, every attribute the proxy omits ends at its floor (Zhuang and Hadfield-Menell, 2020). And the paper gives necessary and sufficient conditions under which optimizing any such proxy eventually makes the principal arbitrarily worse off, in this sense: for any utility level, however low, utility ends at or below it once an omitted attribute's floor is low enough. The condition is that the states that are both feasible and at least a given utility form a bounded region, which means that pushing the world far enough in any direction becomes either infeasible or undesirable (Zhuang and Hadfield-Menell, 2020). The authors stress that this happens regardless of the optimization algorithm, and illustrate it with content recommendation, where a proxy built from any two of four attributes makes utility rise, plateau, and fall (Zhuang and Hadfield-Menell, 2020).

The paper offers two ways out. One is a proxy that refers to the full state, for instance by holding the omitted attributes fixed, which the authors note is hard to implement for a large, unknown set of attributes. The other is a principal who updates the proxy as the state changes (Zhuang and Hadfield-Menell, 2020). The model is the incomplete contract of Section 14.3 in formal dress: the proxy is a contract that names only some of what matters, and the agent spends what the contract does not name. In a language model, the budget's role is played by probability itself, since a policy can raise one response's share only by lowering others' (inference).

15.4.3 The geometry of the turn #

Karwowski and colleagues, in a paper at ICLR, give a geometric reason for the rise and fall (Karwowski et al., 2023). A policy's expected return is a linear function of its occupancy measure, the frequencies with which it visits each state and takes each action, and the occupancy measures that policies can reach form a convex polytope, a bounded region with flat faces (Karwowski et al., 2023). An optimizer that follows the proxy's steepest direction moves straight until it reaches a face, then slides along the face in the direction of the proxy's projection onto it. If that projection points far enough from the true reward, the true reward begins to fall, and since each new face bends the path further from the proxy's own direction, falling becomes more likely as pressure grows (Karwowski et al., 2023).

Over 30,400 experiments in small grid worlds, random processes, and trees, with randomly drawn pairs of true and proxy rewards, the true reward first rose and then fell in 19.3 percent (Karwowski et al., 2023). Given an upper bound on the angle between proxy and truth, the geometry yields an early-stopping rule that provably avoids the fall. The price is reward: in their experiments the rule lost, on average, between 10 and 44 percent of it, and with too small an assumed angle it can do as badly as naive optimization (Karwowski et al., 2023).

The three results so far share one picture (inference). While the policy is free to move in the proxy's direction, pressure on the proxy also raises the truth, provided the two point roughly the same way. Once a constraint binds, whether a budget, the faces of the polytope, or the fact that probabilities sum to one, the proxy can be raised only by trading away what it does not measure, and the truth turns down.

15.4.4 Hacking as broken correlation #

The fourth result starts from a different definition. Laidlaw, Singhal, and Dragan, in a 2025 paper at ICLR, argue that a proxy is chosen because it tracks the goal on the behavior its designer had in mind. So they call a proxy correlated if it is correlated with the true reward over the states and actions a reference policy visits. And they say that reward hacking occurs when a policy optimized for the proxy earns less true reward than the reference policy itself (Laidlaw et al., 2025). In traffic control, pandemic response, blood glucose control, and reinforcement learning from human feedback (RLHF, Section 12.2.1), with a large reward model playing the true reward and a smaller one the proxy, the proxies were correlated under the reference policy and optimizing them alone lowered the true reward (Laidlaw et al., 2025).

From this definition a guarantee follows. Write it for one prompt, so that a policy is a distribution over responses. Standardize each reward by subtracting its mean and dividing by its standard deviation under $\pi_0$, and write $\tilde r$ and $\tilde r^\ast$ for the results; let $\rho > 0$ be their correlation under $\pi_0$. The $\chi^2$ divergence, $\chi^2(\pi \,\Vert\, \pi_0) = \sum_y \pi(y)^2 / \pi_0(y) - 1$, is a second measure of how far $\pi$ has moved from $\pi_0$. Then, for any policy that puts probability only where $\pi_0$ does,

Read the bound from left to right. A policy's true improvement over the reference, in units of the truth's spread, is at least its proxy improvement minus a charge for distance, all scaled up by $1/\rho$. The charge grows with the $\chi^2$ divergence and with the share $1 - \rho^2$ of the proxy that does not follow the truth; with a perfect proxy, $\rho = 1$, it vanishes. The paper states the theorem for occupancy measures and turns it into a training objective: maximize the proxy minus a multiple of $\sqrt{\chi^2}$ (Laidlaw et al., 2025). The divergence is $\chi^2$, not KL, because Cauchy-Schwarz pairs the density ratio with itself; Section 30.2.4 works out how much more $\chi^2$ charges for a rare response.

In the experiments, regularizing occupancy measures did better than regularizing action distributions, the form RLHF uses, in all four environments other than RLHF, and $\chi^2$ tended to do as well as or better than KL in every environment (Laidlaw et al., 2025). In single-turn RLHF the two kinds of regularization coincide, so there only $\chi^2$ against KL was tested; $\chi^2$ gave the larger improvement and was more stable across seeds (Laidlaw et al., 2025).

Huang and colleagues reach $\chi^2$ from the theory of learning from a fixed dataset, in a 2025 paper at ICLR. Their $\chi$PO changes one line of direct preference optimization (Section 12.6) so that it implicitly regularizes with a mixture of $\chi^2$ and KL, and they prove a guarantee they describe as provable robustness to overoptimization (Huang et al., 2025a). The property they single out is step 4 above: the $\chi^2$ divergence quantifies how far a reward model's accuracy under the reference policy transfers to another policy, which KL does not (Huang et al., 2025a).

15.4.5 What the theorems assume #

Two results recalled from earlier chapters complete the picture. A family of distances between reward functions bounds the worst-case regret of optimizing one in place of the other from above and below (Skalse et al., 2024), which gives a precise sense to the claim that a learned reward is close to the true one (Section 13.4). And a reward model's low expected test error does not by itself bound that regret, even with KL regularization (Fluri et al., 2025). Each theorem fixes something that a production training run does not supply. Skalse and colleagues fix a set of policies and count any reversal of order. Zhuang and Hadfield-Menell fix a resource constraint and a proxy over named attributes. Karwowski and colleagues need the angle between proxy and truth. Laidlaw and colleagues need the correlation under the reference policy, and Huang and colleagues a coverage condition on the data. As of October 2026 we found no published measurement of any of these quantities for a production reward model, and for the programmed graders of agentic environments (Section 24.2) none of the conditions has been checked. The theorems say which questions to ask of a proxy; the measurements in Section 15.6 are the closest anyone has come to answering them.

15.5 Heavy tails and catastrophic Goodhart #

In RLHF the reward is learned from data and is therefore always somewhat wrong. The KL penalty of Section 12.3 is meant to limit the damage, by keeping the policy near the responses on which the reward model was trained. Kwa, Thomas, and Garriga-Alonso, in a 2024 paper at NeurIPS, ask whether controlling the KL divergence is enough to obtain good outcomes from a misspecified reward (Kwa et al., 2024). Their answer turns on the shape of the reward's error: specifically on its tail, how fast the probability of very large errors falls off as the errors grow.

15.5.1 Light and heavy tails #

Write the proxy reward of a response drawn from the reference policy as the goal plus an error, $r = r^\ast + \varepsilon$, as in Section 15.3.1, now without assuming that anything is normal. The tail function of the error is the probability $\Prob(\varepsilon > t)$ that it exceeds $t$. For a normal error the tail falls faster than $e^{-\delta t}$ for any constant $\delta > 0$; for an error that follows a power law, it falls like a fixed power of $1/t$, much more slowly. The paper calls a distribution heavy-tailed if its tail falls more slowly than every exponential, that is, if $e^{\delta t}\,\Prob(\varepsilon > t)$ grows without bound for every $\delta > 0$; any other distribution is light-tailed (Kwa et al., 2024). Exponential tails are therefore light in this sense, though only just. A heavy-tailed variable has a higher chance of producing a single extreme value: if the sum of two such variables is large, it is most likely because one of them is extreme (Kwa et al., 2024).

Why the tail matters is a question of price. The KL penalty charges for making rare responses common (Section 12.3.3); a large error is rare, so buying it costs divergence, but it also pays a lot of proxy reward. The tail decides which effect wins.

Step 4 is the paper's own intuition: in a heavy-tailed distribution, extremely large values are not rare enough to be expensive (Kwa et al., 2024).

15.5.2 The theorems and their conditions #

The paper's results come in three groups. The first is about existence. Given any heavy-tailed reference distribution, there are distributions with arbitrarily high mean and arbitrarily small KL divergence from it; given a light-tailed one, a bounded divergence bounds the mean, and the gain in mean vanishes as the divergence does (Kwa et al., 2024). A third theorem carries the first to language models, modeled as processes in which each token deterministically extends the text and the return depends only on the finished response. A corollary concludes that when the true utility is light-tailed and the reward model's errors make the proxy heavy-tailed, a policy regularized to near-zero KL can reach unbounded proxy reward while its true utility approaches no more than the base model's (Kwa et al., 2024). The authors call this catastrophic Goodhart (Kwa et al., 2024). None of these three theorems needs the error to be independent of the goal (Kwa et al., 2024).

The second group is the reassurance, and it has conditions. If the proxy is the goal plus an error, both light-tailed, the goal unbounded, and the proxy's distribution continuous, then the true utility of the optimum of the KL-penalized objective grows without bound as the penalty weight goes to zero (Kwa et al., 2024). The proof takes the optimum's existence for granted, and light tails in the paper's sense do not supply it: with an exponentially distributed goal, which counts as light-tailed, $\E_{\pi_0}[e^{r/\beta}]$ is infinite once $\beta$ is at or below the exponential's mean, and the penalized objective has no maximum, as in step 5 of the derivation in Section 15.5.1 (inference). For a proxy whose tails fall faster than every exponential, such as a normal one, the optimum exists at every $\beta$. This theorem, and the two that follow, require the error to be independent of the true utility, an assumption the authors say seems to be violated in practice (Kwa et al., 2024). They show the assumption is doing work: with an error that is unbiased and light-tailed at every goal value but larger for ordinary goal values than for extreme ones, selecting on the proxy drives the expected goal value back to its mean (Kwa et al., 2024).

The first group shows that policies with catastrophic Goodhart exist, not that training finds them. The authors note that these low-KL policies are one family of high-performing policies, not a unique optimum, and that more likely both the reward and the divergence grow without bound during training, which could still let the true utility grow (Kwa et al., 2024). They also note that a light-tailed or bounded reward function prevents catastrophic Goodhart (Kwa et al., 2024). Their stronger worry is for training that caps the KL at a fixed target, where they think it likely that little or no true utility is gained; that is the authors' expectation, argued from a toy model, not a theorem (Kwa et al., 2024).

15.5.3 Selection as a second model of optimization #

The third group repeats the result in a model of optimization that has nothing to do with KL: keep only the responses whose proxy reward exceeds a threshold, and raise the threshold (Kwa et al., 2024). This is the selection of Figure 15.1. Suppose the error is independent of the goal. If the error is subexponential, a slightly stronger condition than heavy-tailed, and the goal's tail is sufficiently lighter than the error's, the expected goal value of the kept responses returns to its mean over all responses as the threshold rises. If the ratio $\Prob(\varepsilon > t + 1)/\Prob(\varepsilon > t)$ goes to zero, which makes the error's tail fall faster than any exponential and rules out large jumps in its decay, and the goal has a finite mean and no upper bound, the expected goal value grows without bound (Kwa et al., 2024). The second result generalizes the consequence of the regressional identity of Section 15.3.1 (Kwa et al., 2024).