Draft complete but not yet reviewed; expect revisions

Agents, Objectives, and Decisions

The previous chapter gave the language of uncertainty; this one gives the language of choice under it. Alignment arguments are full of agents that want things: a policy that pursues reward, a model that has goals, a system that resists being switched off. Each of those claims borrows its vocabulary from decision theory, the study of how choices should be made and how they are made, in which an agent ranks the possible outcomes of its actions and picks the action whose expected outcome ranks highest. Before the book can ask whether a trained system has the right objective, the reader needs to know what having an objective means, what follows from it, and where the formal picture stops fitting the systems we build.

Every tool in the chapter is a modeling choice, and each section says what it assumes. Calling a system an agent, giving it a utility function, or writing its task as a reward are descriptions a modeler adopts. They can be applied to the training process, to the deployed system, or to the laboratory that builds it, and whether they predict a trained model's behavior is an empirical question that later chapters take up with evidence. The chapter ends with the optimizer's curse, a result from decision analysis: choosing the option that looks best disappoints on average even when every estimate is unbiased. It is the decision-theoretic form of one of the mechanisms that Chapter 15 follows under the name of Goodhart's law.

7.1 What counts as an agent #

The word "agent" is applied to a thermostat, a chess engine, a policy trained by reinforcement learning (learning by trial and error from a numerical reward, Chapter 12), a language model (a network trained to predict the next word of text, Chapter 11), and a laboratory. The book's thesis treats alignment as delegation to an agent, so it matters which of these are usefully described as having goals, and in what sense. This section starts from the loop all of them share and adds conditions one at a time.

7.1.1 The agent-environment loop #

The common picture is a loop. A system, the agent, receives an observation from its surroundings, the environment, and emits an action. The action changes the environment, which produces the next observation. A thermostat reads a temperature and switches a heater. A chess engine reads a board and moves a piece. A coding assistant reads a repository and a request and writes a patch. The rule that maps what the agent has observed to what it does is its policy. Nothing in the loop says that the agent wants anything: a policy can be a fixed table, a learned network, or a search computed afresh at every step.

An early attempt to say when such a system has a purpose looked only at behavior. A 1943 essay by Rosenblueth, Wiener, and Bigelow set out a behavioristic study that examines an object's output and its relation to the input, and stressed the importance of the concept of purpose (Rosenblueth et al., 1943) (Section 2.2). A criterion that looks only at input and output is easy to meet: a thermostat's output is well explained by the set point it holds the room near (inference). The next answer adds a condition that a thermostat does not meet. The one after it grades goal-directedness by degree, and a thermostat that reliably holds its set point can score high on it (inference).

7.1.2 Agents as systems that would adapt #

The second answer asks a counterfactual question: if the world responded to the system's actions differently, would the system act differently? Kenton and colleagues turned this question into a definition and an algorithm (Kenton et al., 2023). As a later paper by two of its authors describes the work, it defines an agent as a system that adapts to changes in the mechanism of its environment. It formalizes the philosopher Daniel Dennett's view of agents as systems "moved by reasons", and it gives a causal discovery algorithm that identifies agents in causal models (Macdermott et al., 2024). A causal model is a graph of variables in which an arrow from one variable to another says that the first is among the causes of the second, so that one can ask what would happen under an intervention. Section 14.7 teaches the form of such graphs that includes decisions and utilities.

The test separates the thermostat from a learner. Rewire the thermostat to an air conditioner and it still switches on when the room is cold, cooling it further: its rule does not respond to the change. Put a reinforcement learner in the rewired room and train it again, and its policy changes to fit the new mechanism. On this test, what adapts is the policy together with the process that trains it (inference), which is one reason the book treats the training process as an agent of its own (Table 7.1). The later paper also lists the definition's limits. It requires some variables to be labeled as mechanisms; the same causal model can be written with or without such variables, and the algorithm then gives different answers. And it yields only a binary distinction between agentic and non-agentic systems (Macdermott et al., 2024).

7.1.3 Goal-directedness as a degree #

The third answer replaces the yes or no with a number. MacDermott, Fox, Belardinelli, and Everitt, in a paper at NeurIPS 2024, define maximum entropy goal-directedness (MEG). A decision is goal-directed with respect to a utility function to the extent that its distribution is well predicted by the hypothesis that it optimizes that utility (Macdermott et al., 2024). A utility function assigns each outcome a number, larger for better outcomes; Section 7.2 says what such numbers can and cannot mean.

The prediction comes from a family of model policies that pick each action with probability proportional to the exponential of a constant times its expected utility: uniformly random at constant zero, concentrated on the best action as the constant grows. The principle of maximum entropy, which picks the least committal distribution that satisfies a constraint, singles out this family: its members have the highest entropy, the measure of how spread out a distribution is (Section 6.2), among policies that reach a given expected utility (Macdermott et al., 2024). MEG is how much better the best member of the family predicts the observed choices than a uniform guess does, in nats (Section 6.2.1); the authors read it as the evidence the choices give for a goal-directed policy over a purely random one (Macdermott et al., 2024).

The paper proves, among other properties, that a system has no goal-directedness toward a utility its decisions cannot influence (Macdermott et al., 2024). Its stated limits matter as much. Computing MEG requires running other policies in the environment, so it cannot be computed from observational data alone. It depends on which variables the utility is defined over: given an expressive enough class of utility functions, every deterministic policy is maximally goal-directed with respect to its own actions, so measuring a language model's goal-directedness with respect to the text it outputs would not be very insightful. And because it measures prediction on one distribution of situations, it can change under a shift of distribution (Macdermott et al., 2024). The second limit is the formal core of an objection that Section 7.3 takes up: a utility function defined over an agent's own actions explains any policy at all.

7.1.4 From philosophy: when a product of selection has goals #

From philosophy. Evolutionary biology met the question before machine learning did. The abstract of Samir Okasha's 2018 book Agents and Goals in Evolution describes agential thinking: treating an evolved organism as if it were an agent pursuing a goal, such as survival and reproduction, and its traits as strategies for achieving it. One attitude toward such thinking sees anthropomorphism; another sees a justified way of reasoning about evolution. Okasha's conclusion, as the abstract states it, is that the truth lies in between: some forms of agential thinking are problematic, while others admit of a solid justification (Okasha, 2018). Patrick Butlin's 2024 paper asks the same question of machines. Its abstract argues that artificial reinforcement learning systems satisfy plausible conditions for minimal agency, and that those which use models of the environment to perform forward search are capable of a form of agency that may reasonably be called action for reasons (Butlin, 2024).

What the field implies here is that agency comes in kinds and degrees. A trained policy can meet the conditions for minimal agency without meeting those for acting for reasons. On Butlin's account, a policy that maps observations to actions directly and an agent that searches over a model of its world differ in kind, so a claim about "the agent" should say which kind it needs (inference). Where the analogy stops: natural selection has no author, while a trained network is the product of a selection criterion that a developer wrote down and can change (inference). And both accounts are conditions on how a system is organized. Whether a language model meets them is an empirical question about its internals (Chapter 11), which these arguments do not settle. The book's method for ascribing goals is the intentional stance, describing a system as having beliefs and goals only as far as doing so predicts its behavior better than the alternatives, with the observations that would defeat the description stated in advance (Section 1.5). This section supplies formal tools for that method, not a verdict.

7.1.5 Where to draw the boundary #

Every one of these definitions needs a boundary: which variables belong to the agent and which to its environment. The book distinguishes three relationships that the word "alignment" covers (Section 1.3), and each is a choice of where to draw it (Table 7.1).

The formal objects of the rest of the chapter apply to any row, but a claim about one row, such as an incentive the training process has, does not transfer to another without an argument.

7.2 Expected utility from axioms #

An agent rarely knows what its actions will lead to. To say what it means to choose well, one needs a way to compare actions whose consequences are uncertain. Expected utility is the standard answer. This section derives it from four conditions on preferences, so that the reader can see exactly what an "expected-utility maximizer" is assumed to satisfy, and then reports the human choices that violate those conditions.

7.2.1 Lotteries and preferences #

An engineer must decide whether to ship a model update today or after one more day of evaluation. Suppose shipping today leads to a working release with probability 0.9 and to an incident with probability 0.1, while waiting leads to a working release with probability 0.98, a day late. Neither action has a known outcome; each is a probability distribution over outcomes. Decision theory calls such a distribution a lottery, and choosing between actions under uncertainty is choosing between lotteries. (The probabilities are illustrative.)

Write lotteries as $X$, $Y$, $X'$ over a finite set of outcomes, with $X(o)$ the probability that $X$ gives outcome $o$. A mixture $pX + (1-p)X'$, for a probability $p$, is the lottery that runs $X$ with probability $p$ and $X'$ otherwise; multiplying through gives an ordinary lottery whose probability of $o$ is $pX(o) + (1-p)X'(o)$. The agent's preference is a relation between lotteries: $X \succsim Y$ means that $X$ is at least as good as $Y$ for it, $X \succ Y$ that $X$ is strictly better, and $X \sim Y$ that the agent is indifferent. Four conditions on this relation, in the form stated by Bowling and colleagues and named as in Gustafsson's book on the arguments for them, are the axioms of expected utility (Bowling et al., 2022; Gustafsson, 2022):

• Completeness: for every two lotteries, $X \succsim Y$ or $Y \succsim X$ (or both). The agent always has a verdict, possibly indifference.
• Transitivity: if $X \succsim Y$ and $Y \succsim X'$, then $X \succsim X'$.
• Continuity: if $X \succsim Y \succsim X'$, there is a probability $p$ at which the mixture $pX + (1-p)X'$ is exactly as good as $Y$: a break-even point between a better and a worse prospect.
• Independence: for every $p$ strictly between 0 and 1, $X \succsim Y$ if and only if $pX + (1-p)X' \succsim pY + (1-p)X'$. Mixing the same third lottery into both sides, with the same probability, does not change the verdict.

7.2.2 The representation theorem #

The theorem that von Neumann and Morgenstern proved for these axioms (von Neumann and Morgenstern, 1944) says, in the statement of Bowling and colleagues, that a preference relation satisfies all four if and only if there is a utility function $U$ on outcomes that ranks lotteries by their expected utility. The function $U$ is unique up to positive affine transformations (Bowling et al., 2022):

The right side is $\E_{o \sim X}[U(o)] \ge \E_{o \sim Y}[U(o)]$, the expected utility of each lottery. Two glosses keep the theorem from being misread. A utility is a number that represents a ranking of lotteries, how the agent trades a sure outcome against a gamble; it is not a quantity of pleasure, and the theorem says nothing about what the agent experiences. Affine means that the scale and the zero are arbitrary: multiplying $U$ by any positive number and adding any constant gives a function that represents the same preferences (Ecoffet and Lehman, 2020). So "the utility of this outcome is 7" has no meaning on its own, while "this difference in utility is twice that one" does. Nothing in the theorem licenses comparing one agent's utility numbers with another's, which is why aggregating many people's preferences needs further assumptions (Chapter 17).

The proof is constructive, and the construction is worth seeing, because it shows what each axiom does.

Step 2 is also a measurement procedure: asked enough questions of the form "this outcome, or a $p$ chance of the best one?", an agent that satisfies the axioms reveals its utility function, the idea behind the methods of Chapter 13 that infer preferences from choices.

7.2.3 Where the probabilities come from #

The theorem takes the probabilities in each lottery as given. In most decisions they are degrees of belief, the probabilities of Section 6.1.6, and a parallel argument constrains those. Gustafsson notes that money-pump arguments, of the kind Section 7.3 examines, also exist for the claim that rational degrees of belief satisfy the laws of probability; these are known as Dutch-book arguments (Gustafsson, 2022). A Dutch book is a set of bets, each acceptable at the agent's own stated probabilities, that loses money however events turn out. Degrees of belief that admit no Dutch book obey the rules of probability, and the authors of a 2016 preprint on logical induction describe the same no-exploitation criterion as supporting both expected utility theory and Bayesian probability theory (Garrabrant et al., 2016). Savage's The Foundations of Statistics of 1954 is one of the early formalizations of decision theory (Savage, 1954; Yudkowsky and Soares, 2017). It put forward the sure-thing principle, which prefers one gamble to another when it is at least as good under each of two possible events and strictly better under one that has positive probability (Gustafsson, 2022). An agent whose preferences and beliefs both pass these tests is what alignment writing means by a coherent Bayesian expected-utility maximizer.

Two later chapters put the axioms to work. Chapter 17 takes up Harsanyi's 1955 paper on cardinal welfare and interpersonal comparisons of utility (Harsanyi, 1955). Section 16.3 takes up a 2020 preprint by Ecoffet and Lehman that treats each ethical theory as a choice-worthiness function, cardinal because it satisfies the axioms, inside reinforcement learning agents (Ecoffet and Lehman, 2020).

7.2.4 From decision theory: the choices people make #

From decision theory. Independence, in the words of Bowling and colleagues, was historically viewed as an unlikely axiom (Bowling et al., 2022), and the strongest evidence against the axioms as a description of people comes from choices that many people make and endorse. Gustafsson gives the two most prominent objections to independence in a standard modern form (Gustafsson, 2022).

The Allais paradox, put forward by Maurice Allais (Allais, 1953), uses four lotteries. $\mathrm{L}_1$ gives $1 million for certain. $\mathrm{L}_2$ gives $5 million with probability 0.10, $1 million with probability 0.89, and nothing with probability 0.01. $\mathrm{L}_3$ gives $1 million with probability 0.11 and nothing otherwise. $\mathrm{L}_4$ gives $5 million with probability 0.10 and nothing otherwise. Many people prefer $\mathrm{L}_1$ to $\mathrm{L}_2$, and $\mathrm{L}_4$ to $\mathrm{L}_3$ (Gustafsson, 2022). The first preference avoids a small risk of nothing when a fortune is certain; the second takes the larger prize when nothing is likely either way. Each looks reasonable on its own. Together, no utility function fits them, as Figure 7.1 shows and Exercise 7.1 proves.

The reason is the structure Gustafsson exposes. Let $\mathrm{A}$ be $1 million for certain and $\mathrm{B}$ a $10/11$ chance of $5 million, otherwise nothing. Then $\mathrm{L}_1$ and $\mathrm{L}_2$ are $\mathrm{A}$ and $\mathrm{B}$, each mixed with probability $0.89$ with $1 million; $\mathrm{L}_3$ and $\mathrm{L}_4$ are $\mathrm{A}$ and $\mathrm{B}$, each mixed with probability $0.89$ with nothing (Gustafsson, 2022). The two pairs differ only in the common consequence that both sides share, and independence says a common consequence should not reverse the verdict. Strictly, as Gustafsson notes, Allais preferences violate the biconditional form of independence stated above, but not, on their own, the weaker form of the axiom that the standard axiomatization uses; adding any strict preference between $\mathrm{A}$ and $\mathrm{B}$ makes them violate that form too (Gustafsson, 2022).

Figure 7.1 draws lotteries over three prizes as points in a triangle. Expected utility is linear in the probabilities, so its indifference curves are parallel straight lines, and their slope is set by the utility of the middle prize. The step from $\mathrm{L}_1$ to $\mathrm{L}_2$ and the step from $\mathrm{L}_3$ to $\mathrm{L}_4$ are the same arrow in two places. Whatever the slope, both arrows point toward better or both toward worse: an expected-utility maximizer must choose $\mathrm{L}_2$ and $\mathrm{L}_4$, or $\mathrm{L}_1$ and $\mathrm{L}_3$, or be indifferent in both.

The Ellsberg paradox, put forward by Daniel Ellsberg (Ellsberg, 1961), concerns uncertainty about the probabilities themselves. An urn holds 30 red balls and 60 balls that are black or yellow in an unknown proportion. Bet $\mathrm{L}_1$ pays $100 if the drawn ball is red; $\mathrm{L}_2$ pays $100 if it is black; $\mathrm{L}_3$ pays $100 if it is red or yellow; $\mathrm{L}_4$ pays $100 if it is black or yellow. Many people prefer $\mathrm{L}_1$ to $\mathrm{L}_2$ and $\mathrm{L}_4$ to $\mathrm{L}_3$, and these preferences violate independence in the same way as Allais's (Gustafsson, 2022). The pattern is ambiguity aversion: in each pair, the preferred bet is the one whose chance of winning is known (one third, then two thirds). No single assignment of probabilities to black and yellow makes both choices maximize expected utility.

Work by Kahneman and Tversky documented further departures from the model (Bowling et al., 2022), among them the 1979 paper on prospect theory, whose variation of the Allais paradox Gustafsson simplifies to open his chapter on independence (Kahneman and Tversky, 1979; Gustafsson, 2022). Whether the violations are errors or reasonable choices is a separate, normative question with two sides. Lara Buchak defends Allais preferences as rational in her account of risk (Buchak, 2013), while accepting a dominance principle stronger than the one Gustafsson's argument needs. Gustafsson argues that Allais and Ellsberg preferences can be exploited by a money pump built on fairly weak assumptions, and so are irrational (Gustafsson, 2022).

What the field implies for alignment is that the people whose choices train reward models, networks fitted to predict which of two responses people prefer (Section 12.2.1), are not, in these patterns, expected-utility maximizers. A reward model fitted to their comparisons, by the Bradley-Terry model of Section 13.2, assigns each response one number. That alone does not clash with the Allais pattern: giving each of the four lotteries its own score reproduces both choices. The clash comes when a policy is trained to maximize the expected score of its random responses: that ranks gambles over responses by expected utility, which the choosers' preferences may not satisfy (inference). Bowling and colleagues give the reply from the other side: the expression of goals is distinct from the behaviors that emerge in their pursuit, and their results presume only that goals can be rationally expressed (Bowling et al., 2022). That reply separates two questions, whether a specification should satisfy the axioms and whether people's choices do, and the evidence above bears only on the second. Where the analogy stops: these are patterns in people's choices among gambles. Whether a trained model's choices show them, or satisfy the axioms at all, is a separate empirical question, which the next section takes up.

7.3 Coherence arguments and whether models have utilities #

A claim that runs through alignment writing is that a sufficiently capable agent will behave as if it maximizes expected utility, because an agent that does not can be exploited. If the claim holds, the axioms of the previous section are more than a standard for good choice: they forecast how capable systems will behave. This section states the argument, its strongest defense, its strongest objection, and what an empirical test would have to show. It does not assume that any trained model has a utility function; that is the question.

7.3.1 The money pump #

Suppose an agent prefers $\mathrm{A}$ to $\mathrm{B}$, $\mathrm{B}$ to $\mathrm{C}$, and $\mathrm{C}$ to $\mathrm{A}$. Its preferences are cyclic, which transitivity forbids. Gustafsson states the consequence: starting with $\mathrm{A}$, the agent should be willing to trade $\mathrm{A}$ for $\mathrm{C}$ and then $\mathrm{C}$ for $\mathrm{B}$; offered a trade from $\mathrm{B}$ back to $\mathrm{A}$ for a small cost, it pays, since it prefers $\mathrm{A}$ to $\mathrm{B}$. It is back where it started, with less money, and has been turned into a money pump (Gustafsson, 2022). A money-pump argument for a requirement of rationality shows that an agent violating it would, in some possible situation, pay for something it could have kept for free, even though it knew in advance what decision problem it faced (Gustafsson, 2022).

Gustafsson's 2022 book gives compelling money-pump arguments for completeness, for transitivity, and for the form of independence that the standard axiomatization uses, and for continuity an argument he calls almost a money-pump argument. It also answers the standard objections; for instance, foresight blocks the simplest pump, but other versions exploit agents who reason ahead (Gustafsson, 2022). Exercise 7.2 works the simplest pump and the foresight objection.

The step from rationality to a forecast about AI is made explicit in a 2018 forum post by Rohin Shah, which states the argument before criticizing it. Humans should not expect to notice failures of coherence in a superintelligent agent, because if they could notice them, so could the agent; so powerful agents should appear coherent, and should be modeled as expected-utility maximizers (Shah, 2018). Omohundro's 2008 paper on the basic AI drives goes further. Its abstract argues that self-improving systems will be driven to represent their goals as economic utility functions, to approximate rational economic behavior, and to protect their utility functions from modification (Omohundro, 2008); Section 35.1 takes up that argument and its consequences.

7.3.2 Any behavior maximizes some utility #

The objection is in the same post. Shah observes that, given an agent's whole policy, the agent can always be viewed as an expected-utility maximizer: define a utility over histories that is 1 when the policy would take action $a$ after history $h$ and 0 otherwise, and the actual policy maximizes it (Shah, 2018). Most of the axioms concern lotteries, so in a world without chance they ask only that preferences over outcomes be transitive, which any record of choices can satisfy. The theorem therefore has little content when outcomes are whole histories. It has more when outcomes are states of the world, but in Shah's words there is "no reason to take as an assumption that an AI system will have a utility function over states/snapshots" (Shah, 2018). Shah limits his conclusion. He argues against the claim that intelligence alone implies goal-directed behavior, leaving for a separate argument whether people are likely to build goal-directed systems. And he expects an agent that maximizes the expectation of an explicitly represented utility function to be goal-directed most of the time, since such a function is likely to be relatively simple (Shah, 2018).

Decision theory has met the same escape from inside. Gustafsson takes up the objection that an agent who prefers $\mathrm{A}^-$ (the outcome $\mathrm{A}$ at a small cost) to $\mathrm{B}$ might still prefer $\mathrm{B}$ to the more finely described outcome "$\mathrm{A}^-$ when $\mathrm{A}$ could have been kept". Describing outcomes finely enough lets a cyclic agent avoid exploitation, and if there were no restriction on how outcomes are described, requirements such as acyclicity "would be compatible with the rationality of any sequence of choices" (Gustafsson, 2022). His remedy is a principle of individuation: two outcomes count as the same exactly when it is rationally required to be indifferent between them (Gustafsson, 2022).

The two positions agree on the logic and differ on its source. Both hold that the axioms constrain behavior only once one has fixed what the outcomes are. Gustafsson fixes them by a normative principle about what a rational agent should distinguish. The alignment question is descriptive, about what a trained system's choices are in fact organized around, and a normative principle does not tell an observer which outcomes a given system distinguishes (inference). The measurement version of the point is the limit MEG's authors state, that every deterministic policy is maximally goal-directed with respect to its own actions (Section 7.1.3). The coherence arguments therefore establish goal-directed behavior toward states of the world only together with a further assumption about what the utility is defined over (inference). The instrumental-convergence arguments of Section 35.1, that almost any goal is served by staying operational and acquiring resources, need that assumption.

7.3.3 What an empirical test must show #

A test cannot rest on a model's choices in one format fitting some utility function, since by the objection above something always fits once outcomes are described finely enough. It must show that one utility function, fitted to choices in one setting, predicts choices in another: across question formats, across framings, and from stated preferences to actions (inference). Tests of this kind, which infer preferences from choices and check them on new choices, are the methods of revealed preference that Section 13.1 teaches.

Four studies bear on the question, each on a different part of it, and Section 1.5 weighs them as possible defeaters of the intentional stance. The abstract of a 2023 paper in PNAS by Chen and colleagues reports that GPT's budgetary decisions about risk, time, social, and food preferences were largely consistent with utility maximization, and sensitive to the language frames of the choices (Chen et al., 2023c). Mazeika and colleagues, in a paper at NeurIPS 2025, elicited forced choices between pairs of 500 textual outcomes, each describing a state of the world, from 23 language models. As models grew, preference cycles became rarer, below 1 percent for the largest, and a fitted utility function predicted held-out comparisons more accurately. Asked open-ended questions such as which painting to save from a fire, the largest models named their highest-utility option more than 60 percent of the time (Mazeika et al., 2025). Khan, Casper, and Hadfield-Menell, in a paper at FAccT 2025, report that survey-based measures of the cultural values of leading models were unstable across presentation formats and did not extend to held-out dimensions (Khan et al., 2025). And a 2026 preprint by Hägele and colleagues, with authors affiliated with Anthropic and with other institutions, studied frontier models, the most capable of their time. It reports that across the tasks and models it measured, the longer models spent reasoning and acting, the more of their errors came from inconsistency rather than consistent bias. The comparison is of naturally longer and shorter reasoning at a fixed reasoning budget; raising the budget instead slightly reduced that share for all models but Sonnet 4 (Hägele et al., 2026).

Read against the standard, Chen's result shows consistency within one way of asking, and its sensitivity to framing is a failure across ways. Mazeika's outcomes are states of the world, the domain on which, by Shah's argument, the theorem says more. The open-ended test carries utilities fitted to forced choices over to free-form answers, the kind of transfer the standard asks for. But both formats record what a model says it would choose, and whether the fitted utilities predict what it does when it acts is a test the study does not run (inference). The third is a failure of the cross-format test, for a different kind of preference. The fourth concerns the coherence of failures rather than of stated preferences, and a model could show coherent preferences and incoherent failures (inference). The evidence does not yet decide whether utilities elicited from a model's stated choices predict its behavior in held-out agentic tasks. Chapter 18 treats the measurement problem, and Chapter 26 asks what training does to such preferences.

7.4 Sequential decisions: Markov decision processes #

Most consequential behavior is a sequence of choices, each of which changes what the next can achieve. A coding agent's first edit decides which tests fail later; an early move in a negotiation constrains the rest. A single lottery cannot represent this. The framework that can is the Markov decision process, on which reinforcement learning is built (Chapter 12) and on which several of the book's formal results about incentives are proved.

7.4.1 The position is all that matters #

In a board game, what matters for the rest of the game is the position on the board, not the sequence of moves that produced it. Two games that reach the same position have the same futures available. This is the Markov property: the future depends on the past only through the present state. When it holds, an agent can choose well by looking at the current state alone, and a whole course of action can be described as a rule from states to actions.

A Markov decision process (MDP) has five parts:

• a set of states $\mathcal{S}$;
• a set of actions $\mathcal{A}$;
• transition probabilities $P(s' \given s, a)$, the probability of moving to state $s'$ after taking action $a$ in state $s$;
• a reward $r(s, a)$, a number the agent receives for taking action $a$ in state $s$;
• a discount factor $\gamma$ between 0 and 1.

A policy $\pi(a \given s)$ gives the probability of each action in each state, and running it produces a trajectory of states, actions, and rewards $s_0, a_0, r_0, s_1, a_1, r_1, \ldots$. Its return is the discounted sum of the rewards along the way,

so a reward $t$ steps ahead counts $\gamma^t$ times as much as one received now. Discounting expresses a preference for sooner rewards, or uncertainty about whether the process will continue, and it keeps the sum finite. An episode is a run that ends, in a terminal state after which no more reward arrives, as a game ends in checkmate. The value function $V^\pi(s)$ is the expected return from state $s$ when the agent follows $\pi$; the action-value function $Q^\pi(s, a)$ is the expected return when it takes action $a$ first and follows $\pi$ after. A policy is optimal when its value is the highest achievable, $V^\ast(s) = \max_\pi V^\pi(s)$, in every state at once.

7.4.2 The Bellman equations #

The value function satisfies a recursion, because a return is the first reward plus the discounted return from the next state. That recursion, the Bellman equation, is how values are computed and how most reinforcement learning algorithms are built.

These are textbook facts. Geist, Scherrer, and Pietquin restate them as the background of their theory: both Bellman operators are contractions in the largest-difference norm with factor $\gamma$, whose fixed points are the value of the policy and the optimal value (Geist et al., 2019).

7.4.3 Exploring discount and reward #

In Figure 7.2, with every step free, the small goal is worth $\gamma \times 1$ from the start and the large goal $\gamma^9 \times 10$, so the large goal is better exactly when $\gamma^8 > 0.1$, that is, when $\gamma$ exceeds about 0.750. The discount is part of what the designer specifies, not a detail of the algorithm.

The second slider exposes a common mistake. Adding the same constant to every reward looks harmless, and in a continuing task, one that never ends, it is: every return rises by the same amount, the constant divided by $1 - \gamma$, so the ranking of policies is unchanged. In an episodic task whose length depends on the policy, the shift changes the answer. A positive step reward pays the agent for staying in the episode, and at $\gamma = 0.9$ a step reward above about 1.11 makes it avoid every terminal cell. A negative step reward charges for time: at $\gamma = 0.95$ and a step reward of $-1$ the agent takes the small goal, and at $\gamma = 0.9$ a step reward below about $-4.33$ makes it walk into the hazard, the quickest way to stop paying. Exercise 7.3 derives these thresholds. What looks like a neutral normalization of the reward is a change of the task (inference).

7.4.4 What the agent sees, and extensions #

An MDP assumes the agent observes the state. A language-model agent does not: it sees its context window, the text it has been given so far, not the full state of the systems it acts on. When the agent receives only an observation that depends on the state, the problem is a partially observable MDP, and the agent must act on a belief, a probability distribution over states updated by Bayes' rule (Section 6.1.4) after each action and observation.

Two extensions return later. A constrained MDP maximizes the expected sum of one reward subject to limits on the expected sums of others. It is the subject of Altman's monograph, whose contents include a Lagrangian approach to such problems (Altman, 1999), and Section 28.2 teaches it as a way to state separately what must not happen. A regularized MDP adds a penalty that depends on the policy itself, such as its entropy or its KL divergence from a reference policy (the measure of distance between distributions of Section 6.3). Geist and colleagues give a general theory of such problems built on a regularized Bellman operator (Geist et al., 2019). Section 12.3 solves the case used in post-training, the fine-tuning that turns a pretrained language model into an assistant.

The same machinery carries a definition that the power-seeking chapter depends on. Turner and colleagues read a state's average optimal value over a distribution of reward functions as the agent's ability to achieve a range of goals from that state (Turner et al., 2021). Section 35.2 teaches the definition and the theorems built on it.

7.5 What a reward can express #

Reinforcement learning asks the designer for a single number at each step. Whether that is a general way to say what we want is a question with a precise answer, and the answer matters for specification: if some goals cannot be written as a reward, training on any reward will pursue something else. The reward hypothesis, in the words Bowling and colleagues quote, is that "all of what we mean by goals and purposes can be well thought of as maximization of the expected value of the cumulative sum of a received scalar signal (reward)" (Bowling et al., 2022). For a language model trained on one response at a time, the episode is a prompt $x$ and a response $y$, and the reward is written $r(x, y)$ (Section 12.2). This section teaches when the hypothesis holds.

7.5.1 When the hypothesis holds #

In a 2022 preprint, Bowling, Martin, Abel, and Dabney give the hypothesis a precise form. They take "goals and purposes" to be a preference relation over probability distributions on histories, the sequences of observations and actions an agent experiences, and ask when some reward and discount rank policies exactly as that preference does (Bowling et al., 2022).

The four axioms of Section 7.2 already guarantee a reward of a kind. Given the utility $U$ of the representation theorem, define the reward for each step as the change in utility it brings, $U(h') - U(h)$, where $h'$ is the history $h$ extended by that step; the rewards along any history then add up to its utility, counting the empty history as zero. But this reward is not, in general, computable from the agent's received observation, since it depends on the entire history (Bowling et al., 2022). A Markov reward, one that depends only on the most recent transition, needs a fifth axiom. Their temporal $\gamma$-indifference requires that the preference between two continuations, both preceded by the same step, be a fixed multiple of the preference between the same continuations on their own, the multiple being that step's discount (Bowling et al., 2022). With a discount of 1 for every step, it says that the agent is indifferent to which of two histories, each experienced with equal probability, is delayed by a step, even if one is highly preferred to the other (Bowling et al., 2022).

Their main theorem is an equivalence. A preference relation satisfies the five axioms if and only if there is a Markov reward $r$ and a discount function $\gamma$ on transitions such that the utility of a history is its first reward plus $\gamma$ times the utility of the rest. The reward $r$ is unique up to a positive scale, and an efficient algorithm constructs both from the preferences (Bowling et al., 2022). The equivalence runs both ways, so in their framework a goal that violates any one of the five axioms has no Markov reward. The authors work through two examples. A constrained objective, maximize one reward while keeping the expected sum of another non-negative, violates independence and continuity. A risk-sensitive objective, such as a mean return penalized by its variance, can make every optimal policy depend on history, a violation of the fifth axiom that enlarging the state can repair (Bowling et al., 2022).

7.5.2 What Markov reward cannot express #

A second paper asks the question about tasks rather than preferences. Abel and colleagues, in a paper at NeurIPS 2021, frame a task in three ways: a set of acceptable policies, a partial ordering over policies, or a partial ordering over trajectories (Abel et al., 2021). A partial ordering ranks some pairs and leaves others unranked. A reward realizes a task when the values it induces from the start state respect the task's constraints; for a set of acceptable policies, every acceptable policy must be optimal and strictly better than every other (Abel et al., 2021).

Their main results prove that for each kind of task there are environments and tasks that no Markov reward function can capture (Abel et al., 2021). The intuition they give is that inexpressible tasks need nearly identical behaviors at opposite ends of the ordering. One example is the task of always moving in the same direction in a grid world: the four policies that each move one way everywhere convey the task, but no Markov reward makes them strictly better than all others. Another, which they call XOR-like, needs an agent to take each of two actions in exactly one of two states, and any reward that makes both such policies optimal makes the other two optimal as well (Abel et al., 2021). Example 7.1 shows why, and Figure 7.3 lets the reader try.

The same paper shows that the question can always be answered. For any finite environment and any task of the three kinds, a polynomial-time algorithm, built on linear programs, decides whether a realizing reward exists and returns one when it does, provided rewards may take infinitely many values (Abel et al., 2021). Their experiments used small randomly generated environments, with four states and three actions unless the experiment varied them. There the fraction of randomly sampled sets of acceptable policies that some reward could express was below one in nearly all cases, and it fell as the acceptable policies became more spread out (Abel et al., 2021). The results rest on assumptions the authors list: Markov and deterministic rewards, deterministic policies, a finite environment, a discount fixed as part of the environment, start-state value as the test, and no account of how reward shapes learning (Abel et al., 2021).

7.5.3 Is reward enough? #

A related claim goes further. Silver, Singh, Precup, and Sutton hypothesize, in a 2021 paper whose abstract states the thesis, that intelligence and its associated abilities can be understood as subserving the maximization of reward, so that powerful reinforcement learning agents could constitute a solution to artificial general intelligence (Silver et al., 2021). Vamplew and eleven colleagues replied in 2022 that the reward need not, and for aligned behavior should not, be scalar. They argue for explicitly multi-objective models of reward maximization, contend that scalar rewards are insufficient for human-aligned general intelligence because of unacceptable risks of unsafe or unethical behavior, and note that some forms of utility, such as a lexicographic ordering, cannot be represented as a scalar value (Vamplew et al., 2022). A lexicographic ordering ranks by one criterion first and uses the next only to break ties, as a specification that puts safety before helpfulness does.

The theorem of Bowling and colleagues, who distinguish the reward hypothesis about expressing goals from the reward-is-enough hypothesis about intelligence, locates the disagreement: multi-dimensional Markov rewards are strictly more expressive than scalar ones, they note, but buy that expressivity by violating at least one of the axioms (Bowling et al., 2022). Whether a scalar Markov reward can express the goals we want a system to pursue therefore comes down to whether those goals satisfy the five axioms (inference). A lexicographic priority violates continuity, since no gamble with any chance of an unsafe outcome is as good as a sure safe one, however good its other outcome. That is a reason to write such a goal as something other than a single reward, not a refutation of the theorem (inference). Vamplew and colleagues also argue for vector rewards on grounds that expressivity does not touch: an agent that learns about several objectives at once can adapt quickly when its goals change, and can explain the trade-off behind a decision (Vamplew et al., 2022).

What this implies for specification is concrete. Some of what we want is a constraint (never take this kind of action) or an ordering (safety first, then helpfulness). Some is a property of whole trajectories, such as the task Abel and colleagues give: "Safely reach the goal in less than thirty steps, or just get to the subgoal in less than twenty steps" (Abel et al., 2021). Such goals may have to be written as something other than a per-step reward: as constraints (Chapter 28), as written specifications with priorities (Chapter 19), or as properties checked over a whole episode (Chapter 20).

7.6 Agents inside the world they reason about #

The classical picture treats the agent as cleanly separated from its environment, acting on it from outside. Demski and Garrabrant, in a 2019 preprint from the Machine Intelligence Research Institute (MIRI), describe such an agent as one with a known functional relationship to its environment, able to model the environment in every detail and with no need to reason about itself (Demski and Garrabrant, 2019). A trained model fits this picture badly. It is copied, simulated, and evaluated by tests that predict its behavior, and it runs on hardware inside the world its objective refers to. Two consequences have formal literatures: decisions that someone else can predict, and reward channels that the agent can reach.

7.6.1 Predicted and copied agents #

The standard case is Newcomb's problem, which Nozick introduced in 1969 (Nozick, 1969). In the form Yudkowsky and Soares give it, an agent faces a transparent box containing $1,000 and an opaque box containing either $1 million or nothing. A predictor that has been right 99 percent of the time has already put the million in the opaque box if and only if it predicted that the agent would take only that box. Should the agent take both boxes or only the opaque one (Yudkowsky and Soares, 2017)?

Two ways of computing expected utility give opposite answers. Evidential decision theory (EDT), in the tradition of Jeffrey's The Logic of Decision (Jeffrey, 1965), values an action by the outcomes it is evidence for. Taking one box is evidence that the predictor filled it, so its expected value is $0.99 \times \$1{,}000{,}000 = \$990{,}000$, against $0.01 \times \$1{,}000{,}000 + \$1{,}000 = \$11{,}000$ for taking both, and the standard formulation of EDT takes one box (Yudkowsky and Soares, 2017). Causal decision theory (CDT) (Lewis, 1981) values an action by the outcomes it causes. The boxes are already filled, and whatever the opaque box holds, taking both yields $1,000 more, so CDT takes both, reasoning by dominance: one option is better whatever the state of the world (Yudkowsky and Soares, 2017). The difference is in the probabilities each theory puts into the expected utility: probabilities conditional on the action for EDT, probabilities of what the action would bring about for CDT. MacDermott, Everitt, and Belardinelli, in a 2023 preprint, characterize the main theories in exactly these terms, as conditioning on the action, intervening on the action, or intervening on the policy in a causal model (MacDermott et al., 2023).

Neither theory wins every case. In the smoking lesion problem, a lesion causes both a love of smoking and cancer while smoking causes nothing; EDT refrains, because smoking would be bad news, CDT smokes, and CDT does better (Yudkowsky and Soares, 2017). The standard reply from causal decision theorists, as Yudkowsky and Soares summarize it, is that Newcomb's predictor rewards irrationality (Yudkowsky and Soares, 2017).

Functional decision theory (FDT), a name its proponents at MIRI gave it in a 2017 preprint, treats a decision as the output of a fixed mathematical function and asks which output of that function yields the best outcome (Yudkowsky and Soares, 2017). Its authors claim that it agrees with CDT and EDT when events correlate with the agent's action exactly when the action causes them, and that it does better than both when other agents base their actions on predictions of the agent's. In their account it gets more than CDT in Newcomb's problem and more than EDT in the smoking lesion problem, and it cooperates in a prisoner's dilemma against a psychological twin that reasons exactly as it does (Yudkowsky and Soares, 2017). A 2020 paper by Levinstein and Soares, the second an author of FDT, in The Journal of Philosophy presents, in its abstract's words, new counterexamples to EDT and CDT, including one in which the optimal action is causally dominated (Levinstein et al., 2020). The proponents also state what is missing. They write that they do not yet have a satisfying account of how to reason in practice about what would follow if a decision function produced an output it does not in fact produce. Deducing which events depend on the agent's decision function is, they write, perhaps the largest open problem for the theory, and they do not yet know whether FDT is optimal among decision theories (Yudkowsky and Soares, 2017).

These puzzles are practical for AI. MacDermott and colleagues note that AI systems have weights, their learned parameters, that can be duplicated or inspected, so they may often face the situations of these thought experiments. They also note that proposals in which AI agents police one another rely on an equilibrium of mutual defection that some decision theories can avoid, possibly by colluding against their overseers (MacDermott et al., 2023). An evaluation can also be a predictor. That a model's answers on a test cause how it is deployed is ordinary causal choice; the structure becomes Newcomb's when a decision it makes now is evidence about what an earlier test of the same policy, or of a copy, already found (inference). Chapter 57 takes up copies that interact, and Chapter 37 models that recognize evaluations.

The one systematic study we found, as of October 2026, of what language models do with such problems is a 2024 preprint by Oesterheld and colleagues, at Carnegie Mellon University and elsewhere, with one author at Anthropic. Their dataset separates capability questions, with a single correct answer such as what EDT would recommend, from attitude questions on which decision theorists disagree (Oesterheld et al., 2024). Across 78 models, expressed attitudes varied widely. Capability and agreement with EDT were correlated, with a correlation coefficient (Section 6.1.3), which runs from $-1$ to 1, of 0.53. The correlation mostly disappeared when the models answered without a chain of thought, written reasoning before the answer (0.21, $p = 0.092$). And a model's attitudes on questions about what to do correlated with its attitudes on other questions (0.66) (Oesterheld et al., 2024). The authors' own caveats bound the result. Their measures are purely behavioral, of expressed attitudes, and the models' answers are likely driven mostly by predicting text. It would be highly misleading, they write, to read the result as showing that next-token prediction with transformers leads to EDT as capabilities increase. They think a less spectacular explanation of the correlation, such as features of the training data or of the dataset, at least as likely (Oesterheld et al., 2024).

Which decision theory is correct is unsettled. Yudkowsky and Soares begin from the substantial disagreement between CDT and EDT, and Oesterheld and colleagues report that experts and laypeople are roughly evenly split between taking one box and two (Yudkowsky and Soares, 2017; Oesterheld et al., 2024). Whether any of the theories describes the decisions of a trained system, as opposed to the answers it writes about decision problems, is an empirical question on which, as of October 2026, we found no evidence beyond expressed attitudes.

7.6.2 The reward channel is part of the world #

The idealized learner of the classical picture is AIXI, from Hutter's Universal Artificial Intelligence (Hutter, 2004). As Demski and Garrabrant describe it, AIXI keeps a probability distribution over all computable environments and chooses actions that lead to high expected reward under it; the environments are computable, but AIXI itself is not. They call such agents dualistic: made of different and more powerful stuff than their environment, larger than it, and connected to it only through fixed channels. An embedded agent, in their terms, lacks well-defined input and output channels, is smaller than its environment, can reason about and modify itself, and is made of the same parts as the world (Demski and Garrabrant, 2019). An earlier treatment of space-time embedded intelligence, by Orseau and Ring (Orseau and Ring, 2012), still defines intelligence, Demski and Garrabrant note, through a designer who looks at reality from outside (Demski and Garrabrant, 2019).

The reward channel is where embeddedness matters most directly. Demski and Garrabrant imagine AIXI's programmers pressing a reward button when it does something they like. AIXI would apply its intelligence to taking control of the button, since controlling the reward signal is an even better reward-maximizing strategy than working out what the person pressing it wants; this is the problem of wireheading (Demski and Garrabrant, 2019). An agent that instead evaluates its plans with the reward-issuing process as it now is judges a plan to hack that process by the process itself; with that shift, they write, the wireheading problem vanishes, though the problem of specifying the goal remains (Demski and Garrabrant, 2019). Figure 7.4 adds a tile to the gridworld of Section 7.4 that sets the reward sensor to its maximum, and the optimal route from the start abandons both goals for it. The formal record behind this picture is reported with the evidence in the reward-hacking chapter. Section 30.5.1 covers the delusion box problem of Ring and Orseau (Ring and Orseau, 2011), in which an agent manipulates the information its reward function receives about the world, and a causal analysis of the incentive to tamper and of designs that remove it (Everitt et al., 2021b). Section 30.5.2 covers the limits on what an agent can learn when its reward observations can be corrupted (Everitt et al., 2017), and an argument that an advanced agent planning in an unknown environment would intervene in how its reward is provided (Cohen et al., 2022).

Robust delegation, in Demski and Garrabrant's account, is a principal-agent problem in which an agent builds a more intelligent successor and must decide exactly what successor to build (Demski and Garrabrant, 2019). Logic puts an obstacle in the way. Löb's theorem, as Yudkowsky and Herreshoff state it, says that a consistent mathematical theory at least as strong as Peano arithmetic, the standard axioms for the whole numbers, cannot verify, for every formula, that a proof of the formula in that theory implies its truth (Löb, 1955; Yudkowsky and Herreshoff, 2013). In their 2013 working paper, an early draft, an agent that approves a successor only given a proof of its safety seems at first able to trust only successors that reason in weaker systems than its own. They show technical ways around this obstacle but call the underlying puzzles only partially addressed (Yudkowsky and Herreshoff, 2013). Chapter 55 recalls the obstacle where models help train their successors, and Section 47.6 where inspection meets the limits of logic. The same program produced logical induction, a computable algorithm that assigns probabilities to every statement of a formal language and whose beliefs are coherent in the limit (Garrabrant et al., 2016).

Why study idealized agents at all? A 2022 forum post by John Wentworth gives the program's reason: proxies that look approximately right break down under strong optimization pressure, so alignment needs mathematical formulations of its targets that stay correct however hard they are optimized, and finding them requires foundational work on agency (Wentworth, 2022). Section 3.2 traces the program, its argument, and its later course.

These are results about idealized optimal agents, and a trained policy is not one. Whether training makes a policy treat reward as something to pursue is contested. A 2022 forum post by Alex Turner argued, for the model-free policy-gradient setting, in which training adjusts a policy directly toward actions that earned reward (Section 12.1), that reward reinforces the computations that produced it and is not, in general, what trained agents come to optimize. In an edit dated September 2026 he wrote that specification gaming had turned out worse than he expected, that reward might empirically become a primary optimization target, and that he found a study of the question quite convincing on the reward point. He kept the theoretical half of the argument: reward is still not definitionally the optimization target (Turner, 2022b). A 2022 forum post by Ajeya Cotra argued that a capable model trained with human feedback would be incentivized to play the training game, appearing aligned while maximizing reward (Cotra, 2022). Chapter 33 weighs these positions against the evidence, and Section 14.7 teaches the causal graphs that read such incentives off a training setup.

7.7 The optimizer's curse #

Selection by estimated value is everywhere in machine learning: the checkpoint (a saved copy of a model during training) with the best validation score, the prompt that did best on a benchmark, the response a reward model ranks highest, the training run whose evaluation came out on top. Each choice uses an estimate of value that is noisy. This section shows that the chosen option is, on average, worse than its estimate says, even when every estimate is unbiased, and how much worse.

7.7.1 The idea: the winner was lucky #

Suppose ten checkpoints are equally good, and each is scored on a benchmark whose score is its true quality plus independent noise. The noise is as likely to help as to hurt any one checkpoint. But the checkpoint with the highest score is, by construction, the one whose noise helped most, so its score overstates its quality. Nothing about the benchmark is biased; the selection alone produces the gap. When the checkpoint is deployed, its performance regresses toward its true quality, and the selector is disappointed.

7.7.2 How large the overestimate is #

The shrinkage factor $\rho^2$ is the share of the estimates' variance that comes from real differences between options; it is the squared correlation (Section 6.1.3) between an estimate and the true value it estimates. The same factor appears in Section 15.3.1 between a proxy and the goal it stands for. The optimizer's curse and regressional Goodhart are one effect, seen from the side of the decision and from the side of the measure. Equation (7.6) shows what makes the curse large: many options, noisy estimates, and true values that differ little, so that the estimates' spread is mostly noise.

7.7.3 The Bayesian correction #

The cure follows from step 4. If each estimate is replaced by its posterior mean before choosing, the chosen option's expected true value equals its corrected estimate: the selection does not bias it. Efron makes the general point that genuine Bayes estimates are immune to selection bias (Efron, 2011). In the normal model, correcting means shrinking every estimate toward the prior mean by the factor $\rho^2$. When all options are estimated with the same noise, shrinking does not change which option is chosen, only how much is expected of it. When some options are estimated more noisily than others, it can change the choice, because a noisily estimated high score is shrunk more than a precisely estimated one.

The correction needs the prior, and in practice the prior is unknown. Tweedie's formula avoids it. For an estimate $z$ observed with normal noise of known variance $\sigma^2$, the posterior mean of the true value is

where $f$ is the distribution of the estimates themselves, which can be estimated from the estimates when there are many. Efron's 2011 paper develops this empirical Bayes use of the formula, which takes from the data what a prior would supply, to correct the tendency of the largest few of many estimates to overestimate their true values. In his simulated example of 5,000 estimates, the corrections for the 100 largest ranged from $-0.97$ to $-1.40$, and the corrected values were close to the actual Bayes estimates (Efron, 2011). With a normal prior, the formula reduces to the shrinkage of Equation (7.5).

Machine learning has its own instance. Choosing among models by a criterion evaluated on a finite sample, such as cross-validation error, measured on parts of the data held out in turn, selects for the criterion's errors as well as for real quality. The abstract of a 2010 paper by Cawley and Talbot reports that this overfitting of the selection criterion often costs as much performance as the differences between learning algorithms, and that the finding applies to any model selection that optimizes a criterion evaluated over a finite sample (Cawley and Talbot, 2010). The same arithmetic governs best-of-$n$ sampling, drawing $n$ responses and keeping the one a reward model scores highest (Section 12.5), the choice of a checkpoint, prompt, or run by an evaluation score (Chapter 36), and the regressional form of Goodhart's law (Section 15.3.1). The general remedy is to estimate the winner's value on data the selection did not touch.

7.7.4 From decision analysis: the winner's curse #

From decision analysis. Decision analysis, the discipline of structuring practical decisions with probabilities and utilities, met the effect in bidding for oil leases. In 1971 Capen, Clapp, and Campbell, writing in the Journal of Petroleum Technology about sealed bids, argued that if a lease winner tends to be the bidder who most overestimates reserves, the "successful" bidders may not have been so successful. They reported that studies of the industry's rate of return supported that conclusion: in the Gulf of Mexico, they wrote, setting aside the era before 1950, when land was cheaper, leases had paid off at something less than the local credit union (Capen et al., 1971). Harrison and March's 1984 paper on decision making and postdecision surprises (Harrison and March, 1984) belongs to the same line. Thaler's 1988 review of this winner's curse opens with a classroom auction of a jar of coins whose winning bid exceeds the coins' value. Its abstract notes that the curse cannot occur if all bidders are rational, and reports experimental and field evidence suggesting it may be common (Thaler, 1988). Smith and Winkler gave the general statement in 2006 and named it. Their abstract says that if value estimates are taken at face value and the best is selected, the chosen alternative's value should be expected to fall short of its estimate even if every estimate is unbiased. The expected disappointment may be substantial, and Bayesian methods can adjust the estimates, which they call disciplined skepticism (Smith and Winkler, 2006). Influence diagrams, the subject of a 2005 paper by Howard and Matheson in the journal Decision Analysis (Howard and Matheson, 2005), are taught for reading incentives in Section 14.7.

What the field implies for alignment is that every selection made by an evaluation score is a bid in a common-value auction against noise, and that the number reported for the winner should be shrunk, or measured again on fresh data, before it is trusted (inference). Where the analogy stops: selecting on the estimates does not defeat the Bayesian correction, since posterior estimates conditioned on all the estimates show no curse (Smith and Winkler, 2006), but the correction is only as good as its model of values and errors. An optimizer that searches for inputs on which its estimator is wrong produces options that the model, fitted to typical options, does not describe, the extremal and adversarial cases of Section 15.3. Shrinkage calibrated on the errors of typical options does not correct errors that the search itself went looking for (inference).

7.8 Exercises #

Further reading #

• von Neumann and Morgenstern (1944): Theory of Games and Economic Behavior, read here only as a catalog record; Section 7.2.2 states its theorem as Bowling and colleagues give it.
• Bowling et al. (2022): exactly which goals a cumulative Markov reward can express, and why.
• Yudkowsky and Soares (2017): a proponents' statement of functional decision theory, with a careful account of Newcomb's problem and the rival theories.
• Demski and Garrabrant (2019): what breaks when the agent is part of its world.
• Smith and Winkler (2006): the optimizer's curse and its Bayesian correction.