Draft complete but not yet reviewed; expect revisions

Principals, Agents, and Games

The book's thesis is that alignment is a problem of delegation: someone hands a task to an agent that knows more than they do about how the task is being done, and possibly about the task itself. Economists call the one who delegates the principal and the one who acts the agent, and for fifty years they have studied what a principal can achieve when it sees only part of what the agent does. The previous chapter asked how a learner can read what people want from the choices they make (Chapter 13). This one asks the next question: once the principal knows what it wants, what can it get from an agent whose actions and knowledge it sees only in part?

The answers are precise. They say how strong an incentive should be, when paying for a measure makes things worse, who decides what a contract does not say, what happens when several principals pull on one agent, and how often an inspector with a small budget must look. This chapter teaches each answer with the smallest model that produces it, derives the result, and then turns the tools toward learned systems, where causal influence diagrams let a designer read an agent's incentives off the graph of its training setup.

Every model rests on assumptions, and the assumptions are the point. The classical models take the agent's preferences as given, let the agent refuse the contract, and let the principal see only what the agent produces. A developer who trains a model shapes its preferences, offers it no contract to refuse, and can rerun it on the same input as often as it likes. The chapter ends with a ledger of every assumption it used, checked against the book's three relationships, so that later chapters use each tool only where it applies. The next chapter (Chapter 15) carries the multitask result into the theorems on optimizing a proxy.

14.1 Delegation when the agent knows more #

A principal, written P, wants a task done and hands it to an agent, A. Two things can be out of P's sight. P may not see what A does: whether a salesperson called the client, or whether a contractor used the concrete the plans specified. Economists call this hidden action, or moral hazard. P may also not see what A knows: how able the contractor is, or what the job really requires. That is hidden information, or adverse selection. Both arise for the same reason the principal delegated in the first place: the agent is better placed than the principal to do the work, and so to know how it is going.

P cannot pay for what it cannot see, but it can pay for an outcome that the hidden action influences, such as sales or a passed inspection. The price of doing so is that outcomes also depend on luck. Pay that follows an outcome is a gamble for the agent, and an agent that dislikes gambles values it at less than its average, so the principal must pay more to have the work done at all. The losses a principal bears because it acts through an agent it cannot fully see are what Jensen and Meckling's theory of the firm called agency costs (Jensen and Meckling, 1976); Section 61.1 divides them into their parts when it applies them to the laboratory as the public's agent.

14.1.1 The linear model of hidden action #

The smallest model that shows the trade-off has one task. The agent chooses an effort $e \ge 0$ that P cannot observe, at a personal cost $c(e) = e^2/2$ (effort is measured in units that give the cost this form). P observes only a measured output,

an error that is normally distributed with mean zero and variance $\sigma^2$, so $\sigma^2$ says how noisy the measure is. P pays a linear wage $w = w_0 + b\,z$: a fixed part $w_0$ and a piece rate $b$, the share of each measured unit the agent keeps. Economists call $b$ the power of the incentive. P is indifferent to risk and wants to maximize expected output minus expected pay.

The agent is risk-averse: it prefers a sure payment to a gamble with the same average. A convenient model gives it utility $-\exp(-\nu x)$ for net income $x$, with a coefficient of risk aversion $\nu > 0$; the larger $\nu$, the more the agent dislikes spread. (The book writes $\exp$ rather than a power of $e$ throughout this chapter, since $e$ is effort.) Under this utility, an income that is normally distributed with mean $\mu$ and variance $v$ is worth exactly as much to the agent as the sure amount $\mu - \nu v/2$, its certainty equivalent. The amount subtracted, $\nu v/2$, is the risk premium: what the agent would give up to be rid of the gamble.

Two constraints govern any contract P can offer. The participation constraint says that A accepts only if the contract is worth at least what it could get elsewhere, an amount written $\underline{u}$. The incentive compatibility constraint says that P can count only on the effort A finds best under the contract, not on the effort P would like. Both appear again in Section 14.8, because both are assumptions about the agent.

Equation (14.1) is a trade-off between motivation and insurance. At $b = 1$ the agent works as hard as is efficient (the marginal cost of effort, $e$, equals its marginal value, 1) but bears all the noise; at $b = 0$ it bears no risk and does nothing. The optimum moves toward 1 as the measure becomes less noisy or the agent less averse to risk: with $\nu \sigma^2 = 0.25$, the agent keeps 80 percent of each measured unit. Better measurement justifies stronger incentives. And if the agent did not mind risk ($\nu = 0$), $b^\ast = 1$: P would in effect sell the output to A for a fixed fee, and hidden action would cost nothing. In this model the whole cost of hidden action comes from the agent's dislike of risk, which Section 14.8 looks for in a training run.

14.1.2 Which signals a contract should use #

Suppose P can observe a second signal besides output: a supervisor's report, a log of hours, a customer survey. Should the contract use it? Holmström's answer is the informativeness principle. A signal is worth adding to the contract if and only if it carries information about the agent's action that output does not already carry; in statistical terms, unless output is a sufficient statistic for the action, so that once output is known the signal says nothing more about what the agent did. The paper puts it in one line: "a signal is valuable if and only if it is informative", and adds that any informative signal, however noisy, has positive value if it can be obtained and used at no cost (Holmstrom, 1979). The result has a condition that must travel with it. It concerns contracts that may depend on the signals in any way; within a restricted class, such as linear contracts, the paper notes that informativeness may not be enough to improve the contract (Holmstrom, 1979).

The linear model shows the gain in one line. A second measure $z' = e + \varepsilon'$, with independent noise of the same variance, lets P pay on the average of the two, whose noise variance is $\sigma^2/2$. Repeating the derivation with that variance gives $b^\ast = 1/(1 + \nu\sigma^2/2)$: two noisy measures support a stronger incentive than one.

The principle takes one thing for granted: the agent affects the signals only through its action. An agent that can raise a measure without doing more of the work breaks that premise, which is the multitask problem of the next section, and a learned agent trained against a signal can change how the signal relates to what it does, the concern Section 40.5 takes up for monitors that read a model's reasoning (inference).

Linear contracts are not only convenient. Carroll studied a principal who knows some of the actions available to the agent but not all, and judges each contract by its worst case over the actions it does not know. With a risk-neutral agent whose pay cannot fall below a floor (limited liability), the paper's abstract reports that "very generally, the optimal contract is linear" (Carroll, 2015). It is a model of designing for an agent with options the designer did not foresee, a developer's position when a trained model finds a strategy no one anticipated (inference).

14.1.3 What the agent knows #

Hidden information is the other half of P's problem, and its economics is a sequence of models of evaluating a party that knows more. Its classic settings are markets with uncertain quality (Akerlof, 1970) and signaling in the job market (Spence, 1973). Three results from it are used by name in later chapters, where the better-informed party is a model under evaluation (Chapter 36) or a model reporting what it knows (Chapter 39, which reports the first two with what they imply).

• Cheap talk: in Crawford and Sobel's model, with a one-dimensional state and action and preferences restricted so that each party's best action rises with the state, a sender whose interests differ from the receiver's, and whose messages cost nothing, reports in equilibrium only which interval of a partition its information lies in (Crawford and Sobel, 1982).
• Persuasion by commitment: a sender who commits in advance to how it will gather and report evidence can move a fully rational receiver without saying anything false (Kamenica and Gentzkow, 2011).
• The revelation principle: for a single principal, any outcome achievable with some mechanism is also achievable with a direct one, in which the agent reports its private information and has no reason to report it falsely; the standard proof indexes the agent's equilibrium choices by its type (Martimort and Stole2, 2002), and Myerson stated a version in a paper on incentive compatibility and the bargaining problem (Myerson, 1979). Section 14.4 shows it failing when there are several principals.

The limiting case is a credence good, whose quality the buyer cannot judge even after using it, such as a car repair or a medical treatment. Darby and Karni's title names the question it raises, the optimal amount of fraud under free competition (Darby and Karni, 1973), and the unified theory asks under which conditions experts have an incentive to exploit what buyers cannot know, and which institutions deter it (Dulleck and Kerschbamer, 2006). Section 61.1 derives the simplest credence-goods model for the public's relation to a laboratory, and Chapter 41 asks how a weaker checker can verify a stronger one.

In practice the principal's own judgment is often the measure, and a model trained on it inherits its flaws. In health data, the abstract of one analysis observes, neither the outcome nor its inputs are measured perfectly, and models trained on such data can reproduce and even magnify existing errors in human judgment (Mullainathan and Obermeyer, 2017): learning from a principal's choices can automate the principal's mistakes (Section 22.1).

14.2 Paying for what can be measured #

Most jobs are several tasks at once. A teacher prepares students for a test and also teaches them to think; a programmer makes the tests pass and also writes code a colleague can maintain; a hospital lowers its reported mortality and also treats the patients who are hardest to save. Some of these tasks are easy to measure and some are not. Kerr's 1975 essay put the resulting problem in its title, the folly of rewarding A while hoping for B (Kerr, 1975), and later models of objective performance measurement sharpened the argument (Gibbons, 1998). This section derives the sharpened version. It is the economic theory of reward hacking, stated decades before the term, and the book's account of the training relationship rests on it.

14.2.1 The idea: effort that competes #

Add a second task to the linear model. The first is measured, the second is not, and the principal values both. Pay on the measured task now does two things. It raises effort on that task, as before, and, if the two tasks compete for the agent's time and attention, it lowers effort on the other, because each unit moved from the unmeasured task to the measured one is now paid. A strong piece rate can therefore buy more of what is counted at the price of what is not, and the principal may be better off with a weaker incentive, or with none. Holmström and Milgrom's analysis of multitask incentive contracts, asset ownership, and job design made this precise (Holmstrom and Milgrom, 1991); in the summary of a later study of high-stakes testing, they "show that high-powered incentives will lead agents to focus on the most easily observable aspects of a multi-dimensional task" (Jacob, 2005).

For the unmeasured task to matter at all, the agent must do some of it without being paid. In the model below, the agent would do an amount $\bar e$ of each task on its own, from habit, professional norms, or interest, and finds it costly to depart from that baseline in either direction. This is an assumption, and Section 14.8 lists it: without some unpaid motive, an unmeasured task gets no effort under any contract, and there is nothing for an incentive to crowd out.

14.2.2 The multitask model #

The agent chooses efforts $e_1$ on the measured task and $e_2$ on the unmeasured one. Write $x_i = e_i - \bar e$ for its departure from the baseline on task $i$. Its cost is

where $\delta$ measures how strongly the tasks compete: at $\delta = 0$ they are independent, and as $\delta$ approaches 1 more effort on one makes the other more costly, until at $\delta = 1$ only the total matters and the tasks are perfect substitutes in the agent's effort. P observes $z = e_1 + \varepsilon$ with $\varepsilon \sim \N(0, \sigma^2)$, pays $w = w_0 + b\,z$ with $b \ge 0$ (pay may not fall when measured output rises), and values the agent's work at $e_1 + \zeta e_2$: one unit of the unmeasured task is worth $\zeta$ units of the measured one. The agent has the exponential utility of Section 14.1.1.

Three features of Equation (14.2) carry the argument of later chapters. When the tasks compete, the optimal incentive falls as the unmeasured task becomes more valuable, and it can fall to zero: a fixed wage, paying for nothing in particular, can be the best contract. Stronger competition pulls more effort from the unmeasured task at any piece rate, but it also makes measured effort respond more to pay, so its effect on the optimal rate depends on the parameters. The result does not need the agent to dislike risk: with $\nu = 0$, $b^\ast = 1 - \delta\zeta$, below 1 whenever the tasks compete and the unmeasured one has value, so the multitask distortion survives where the single-task trade-off of Section 14.1.1 disappears. And the formula holds only while the agent still does some of each task; past that point the principal can instead pay strongly for the measured task and give up the other, which is why job design, the choice of which tasks to bundle into one job, is an instrument of its own.

Figure 14.1 lets the reader move all three quantities. At the defaults, an unmeasured task worth 1.5 times the measured one and competition $\delta = 0.5$, Equation (14.2) gives $b^\ast = 0.25/1.375 \approx 0.18$, against $1/1.5 \approx 0.67$ if the second task did not exist. A piece rate of 0.6 raises measured effort from 1 to 1.8 and lowers unmeasured effort from 1 to 0.6, and the surplus falls below what the weaker incentive earns. Slide $\delta$ above $2/3$ and the best contract is a fixed wage.

From economics. The model's prediction is not hypothetical. Its field record includes teacher cheating on high-stakes tests (Jacob and Levitt, 2003), gaming responses to explicit performance incentives in job training (Courty and Marschke, 2004), and selection behavior by providers after cardiac surgery report cards were published (Dranove et al., 2003); Section 15.2 reports it study by study. The economist's prescriptions follow from Equation (14.2) and its relatives: weaken incentives on measures that capture only part of the job; measure more of the job; supplement objective measures with judgment, the subject of Baker, Gibbons, and Murphy's study of subjective performance measures in optimal incentive contracts (Baker et al., 1994) and of the repeated-game models of subjective assessment that Gibbons surveys (Gibbons, 1998); and treat the design of jobs as an instrument, the third subject in Holmström and Milgrom's title (Holmstrom and Milgrom, 1991). A measure can be wrong in two ways, pointing the wrong way or merely noisy; distortion and risk are the subjects of Baker's work on performance measurement (Baker, 1992; Baker, 2002). Holmström's 2017 lecture describes the field's path from pay for performance to incentive systems that account for "the full portfolio of activities that the agent can engage in" (Holmström, 2017). And there is a positive result: in a model where effort can either improve the attributes an evaluator cares about or game the features it measures, Kleinberg and Raghavan characterize when an evaluation rule can induce the first kind, and show that whenever any reasonable rule can, a simple linear one suffices (Kleinberg and Raghavan, 2020).

For training, the model is a mapping, and the book marks it as one (inference). A reward that pays on what a grader checks is pay on the measured task. A policy's probabilities sum to one, so probability moved toward the responses a grader pays for is taken from others, competition for effort in its starkest form. The KL penalty of Section 12.3 charges for departures from the reference policy in either direction, roughly in proportion to their square when they are small, which is the role of the baseline cost. The prescriptions then have training counterparts: a stronger anchor or earlier stopping weakens the incentive, an ensemble of reward models measures more of the job, and human review supplies judgment. Whether these are the right responses is debated; Chapter 15 returns to the question with theorems.

The analogy stops at the agent's other reasons. A worker who games a measure risks detection and a career, and the economist's prescriptions price both. A policy under optimization pays only the KL price, which falls as the anchor weakens. The multitask distortion survives the difference, since it needs no risk aversion and the KL price plays the part of the effort cost, but the worker's other reasons to hold back have no counterpart (inference). Raters who produce human feedback are agents in this model in the ordinary sense when they are paid on throughput or agreement and careful judgment goes unmeasured; Chapter 18 takes up that evidence.

14.3 Contracts that cannot list every case #

The models so far assumed that a contract can say what it needs to: pay $b$ per measured unit, whatever happens. Real contracts cannot list every case. Some contingencies are unforeseen; some can be observed by the parties but not verified by a court that would have to enforce the term; and writing out the rest would cost more than it is worth. A contract that is silent on a case still leaves someone to decide what happens in it. The question this section answers is who, and why it matters.

14.3.1 The idea: control decides the gaps #

When the contract is silent, whoever controls the relevant assets decides. Grossman and Hart's theory of the costs and benefits of ownership (Grossman and Hart, 1986), extended by Hart and Moore, identifies a firm with the assets its owners control and studies how changes in ownership affect the incentives of the people who work with those assets (Hart and Moore, 1990). The right to decide whatever the contract leaves open is called residual control. It matters because it changes what each party can get if renegotiation fails, and so what each party will invest before anyone knows which case will arise.

The law fills gaps in a second way, with default rules that apply when the parties have said nothing, the subject of Ayres and Gertner's economic theory of default rules (Ayres and Gertner, 1989). The employment relationship gives a third answer: an employee agrees to do what the employer asks within a range of tasks the contract does not spell out, so authority fills the gaps; Simon gave the relationship a formal theory (Simon, 1951). Its descendant in alignment is corrigibility, an agent's acceptance of correction by those entitled to give it, which Chapter 20 teaches.

14.3.2 Reward functions as incomplete contracts #

Hadfield-Menell and Hadfield made the application to AI explicit. They propose that the analysis of incomplete contracting can provide a framework for understanding the alignment problem, and they emphasize that human contracting is supported by substantial external structure, institutions such as culture and law that "supply implied terms to fill the gaps in incomplete contracts". Their research agenda asks how to build AI that can replicate the human cognitive processes connecting individual incomplete contracts with that structure (Hadfield-Menell and Hadfield, 2019). On this reading a reward function or a model specification is a contract that cannot cover every case, and misspecification is the normal condition rather than a mistake to be removed.

Incompleteness has a formal cost under optimization. In a model where the principal's utility depends on $L$ attributes of the world, resources are limited, and the proxy reward has support on only $J < L$ of them, Zhuang and Hadfield-Menell give necessary and sufficient conditions under which optimizing the incomplete proxy indefinitely drives overall utility arbitrarily low (Zhuang and Hadfield-Menell, 2020). Section 15.4 states the conditions and works an example.

The property-rights model and the AI case differ in what control is for (inference). In the derivation above, control matters because it changes the incentive to invest. For a deployed model, it matters because it settles who decides in a case the specification did not foresee: the developer through the defaults it trained, the operator through its system prompt, the user through the request, or the model through dispositions that the developer wrote but cannot fully inspect. In practice the last of these decides first, in the moment, whenever the case is new. Section 19.2 reads published model specifications as contracts with gaps and asks who holds residual control in each.

14.4 Many principals, one agent #

A deployed model does not have one principal. The developer trains it, an operator writes the system prompt, a user writes the request, tool outputs and documents carry text from third parties, and states regulate all of them. They act at once, through different instruments, and one of them, the developer, writes the order of precedence among the rest (Section 1.4). The models so far had one principal. This section asks what changes when there are several.

14.4.1 The idea: the agent's behavior is an equilibrium #

From game theory. When several principals each offer the same agent an incentive scheme, and the agent chooses its action given all of them, the setting is called common agency, the title and subject of a 1986 paper by Bernheim and Whinston (Bernheim and Whinston, 1986a), whose companion paper treats menu auctions, resource allocation, and economic influence (Bernheim and Whinston, 1986b). The agent's behavior is no longer one principal's design. It is an equilibrium: each principal chooses its offer as a best response to the others', and the agent responds to the sum.

Two results set the scope of what follows. With complete information, Dixit, Grossman, and Helpman prove that the principals' Nash equilibrium in truthful strategies implements an efficient action. Their application, lobbies bidding for a government's tax policy, shows what "efficient" leaves out: the lobbies' competition for transfers turns into a prisoners' dilemma in which the government captures all the gain available to the parties (Dixit et al., 1997). The action is efficient for the parties whose payoffs enter the game; anyone with no offer to make has no instrument in it and counts only as far as the agent or a principal values that person's welfare (inference). Korinek and Balwit draw the same line for AI: direct alignment asks whether a system accomplishes the goals of the entity operating it, social alignment asks about its effects on society, including the externalities it imposes on others (Korinek and Balwit, 2023). In contract terms, a system aligned with all of its principals can still be misaligned with everyone else.

The second result concerns hidden information. With one principal, the revelation principle of Section 14.1.3 lets the designer assume the agent reports truthfully. With several, Martimort and Stole show that it cannot be applied to study the equilibria: "one principal may prefer to induce the agent to 'lie' to the other principal, with the result that truthful equilibria may fail to exist", and the equilibria must be characterized instead through the menus of options each principal offers (Martimort and Stole2, 2002). With several principals, one of them may want the agent to deceive the others.

14.4.2 Many principals, weaker incentives #

With hidden action and a risk-averse agent, common agency also weakens incentives, and the smallest example shows how. Dixit's papers on the power of incentives in private and public organizations and on incentives in the public sector are the classic treatments (Dixit, 1997; Dixit, 2002).

Two tasks with outputs $z_j = e_j + \varepsilon_j$, $j = 1, 2$, have independent noises of variance $\sigma^2$ and an effort cost $\tfrac12(e_1^2 + e_2^2)$, so the tasks do not compete. Principal 1 values only $e_1$ and principal 2 only $e_2$, but each can pay on both outputs: principal $i$ pays a fixed amount plus $b_{i1} z_1 + b_{i2} z_2$. The agent responds to the aggregate piece rate $B_j = b_{1j} + b_{2j}$ on each output. Each principal sets its rates to maximize its own value plus the agent's certainty equivalent, taking the other principal's offer as given; it can do so because it can adjust its fixed payment to capture whatever it adds to the agent's welfare, the device that made Section 14.1.1 a problem of total surplus.

Each principal pays for its own task and taxes the other's: $b_{21}$ is negative. Principal 2 gains nothing from output on task 1, but the risk that pay on task 1 loads onto the agent is part of what principal 2 must compensate the agent for, so it offsets the incentive. The offsets partly cancel principal 1's, and the agent ends up with a weaker incentive than either a single owner of both tasks or principal 1 alone would set.

The condition matters more than the result. Set $\nu = 0$ in Equation (14.3) and $B_1 = 1$, exactly what a single principal would choose: in this model the weakening comes entirely from the agent's dislike of risk, consistent with the efficiency of the complete-information case (inference). Section 14.8 records the consequence. Where nothing bears risk, as in a training run, "many principals weaken incentives" has no force from this model, while the structure (an equilibrium of several offers) and the externality on everyone outside the game remain.

14.4.3 The AI forms #

Three frameworks carry common agency into AI. Multi-principal assistance games study a robot acting on behalf of $N$ humans with differing payoffs; in a 2020 preprint, Fickinger and colleagues argue that impossibility theorems of social choice apply to such games and that strategic behavior by the humans may complicate the robot's task of learning their payoffs (Fickinger et al., 2020a). Delegation games model principals who act through AI agents that interact on their behalf. Sourbut, Hammond, and Wood separate failures of control, in which an agent does not act on its principal's preferences, from failures of cooperation, in which the agents fail to work well together, and break each into alignment and capabilities (Sourbut et al., 2024). And the ARCHES report, a 2020 preprint by Critch and Krueger, sorts research by the number of humans and AI systems involved, from single/single to multi/multi delegation, and observes that much of the technical research cited as relevant to catastrophic risk has focused on single/single delegation (Critch and Krueger, 2020).

The instruction hierarchy is common agency's clearest AI instance. Its developers, at OpenAI, argued in a 2024 preprint that a primary vulnerability behind prompt injections and jailbreaks is that language models often give a system prompt the same priority as text from untrusted users and third parties, and they trained models to "selectively ignore lower-privileged instructions" (Wallace et al., 2024). Read through this section, the hierarchy is a contract term that ranks principals, written by one of them; prompt injection is a party with no standing acting through a principal's channel (inference). Section 19.3 asks who is entitled to write the ranking, Chapter 49 treats injection as an attack, and Section 61.3 applies common agency one level up, to the parties that act on a developer.

The analogy stops at the instruments. The principals of common agency offer payments; the principals of a model issue instructions, and the developer also trains the agent, which no principal in the economic model can do. As of October 2026 we found no model that maps a ranking of instructions onto the menus of offers the theory analyzes. Collusion is the other case the single-principal models miss. Tirole's analysis of the role of collusion in hierarchies and bureaucracies (Tirole, 1986) is the economic ancestor of a monitor that sides with the agent it monitors (inference), which Chapter 50 takes up.

14.5 Choosing an agent instead of paying one #

Every model so far took the agent's goals as fixed and bought behavior against them. A principal can instead choose, or shape, an agent whose goals already match its own, and then pay and watch it less. Economics, political representation theory, and the study of civil-military relations each examine this choice. Together they give the book's distinction between alignment (make the agent's goals right) and control (assume they may not be, and check) a theory with a known trade-off.

14.5.1 The idea: motivation substitutes for pay #

An agent that cares about the principal's mission needs less inducement to pursue it. The smallest model that shows this needs one assumption the earlier models did not: the agent has limited liability, so the principal can pay it but cannot charge it a fee up front. Without that assumption the principal could recover the agent's enthusiasm through a lower fixed wage, and motivation would change who gets the surplus but not the contract.

The literature behind each step is larger than the model. Besley and Ghatak study incentives in mission-oriented organizations staffed by motivated agents; matching the mission preferences of principals and agents raises organizational efficiency and "economizes on the need for high-powered incentives", but it can also "entrench bureaucratic conservatism and resistance to innovations" (Besley and Ghatak, 2005), which is step 4. Akerlof and Kranton, building on their introduction of identity into economics (Akerlof and Kranton, 2000), propose that workers' self-image as jobholders, with their ideal of how the job should be done, can be a major work incentive that flattens reward schedules, and they identify a trade-off: supervisors provide information to principals but can create rifts in the workforce and reduce employees' intrinsic incentives (Akerlof and Kranton, 2005). Prendergast shows that when bureaucrats care about their work, they should be biased, that for a range of bureaucracies those biased against their clients lead to more efficient outcomes, and that self-selection need not produce the bias the principal wants (Prendergast, 2007). And rewards can weaken the motivation they target; Section 25.3 reports that evidence, the overjustification effect in psychology and motivation crowding in economics.

14.5.2 Selecting or sanctioning #

From political representation theory. Mansbridge sets a selection model of representation against the sanctions model. In the sanctions model, a principal accepts any agent and relies on monitoring and on the threat of punishment or replacement. In the selection model, the principal chooses an agent whose objectives already match and then monitors little; "the higher the probability that the objectives of principal and agent may be aligned, the more efficient it is for the principal to invest resources ex ante, in selecting the required type, rather than ex post, in monitoring and sanctioning" (Mansbridge, 2009). She lists three conditions for selection: the agent is at least partly self-motivated, an alignment of objectives is possible, and the principal has reliable mechanisms of selection and sorting. Selection is attractive, in her account, when agents will face unpredictable situations and "must act speedily, creatively, flexibly, and adaptively" (Mansbridge, 2009).

The two models can be combined, and they can conflict. One institution, the election, can serve both, and in Mansbridge's account trade-offs between them sometimes arise (Mansbridge, 2009). She recounts Fearon's argument that elections can be understood as choosing a "good type" of leader rather than as sanctions: he granted that selection and sanctioning are compatible, but concluded that in a game-theoretic model, once politicians vary in attributes relevant to their performance, it makes sense for voters to focus completely on choosing the best type (Mansbridge, 2009; Fearon, 1999). The sanctioning tradition Fearon set this against, in which the prospect of losing office keeps officials from shirking, includes Ferejohn's analysis of incumbent performance and electoral control (Mansbridge, 2009; Ferejohn, 1986). An electorate that votes on type alone gives up some of that threat (inference). The representative steered by internal values rather than by sanctions, which Mansbridge calls gyroscopic, is Section 1.2's.

What the framework implies for AI is the book's own reading (inference). Training a model's character is selection, and monitoring, auditing, and control protocols are sanctions; the conditions Mansbridge lists for selection (self-motivation, possible alignment, reliable selection) become questions about what training instills and whether evaluation can see it. Where it stops: a representative is chosen from people whose values formed without the voter, while a developer writes the values it then selects for.

From civil-military relations. The control of an agent built to be stronger than its principal has its own literature. Feaver states the problem that the classic theories of Huntington, in The Soldier and the State (Huntington, 1957), and Janowitz attempt to explain: "how to reconcile a military strong enough to do anything the civilians ask them to with a military subordinate enough to do only what civilians authorize them to do", and questions those theories' reliance on professionalism (Feaver, 1996). The two halves of the problematique are a capable model's two requirements, usefulness and control, which Section 51.3 reads against the field's record of what control cost (inference). Where the analogy stops: soldiers share a society and a law with the civilians who command them, and no civilian writes a soldier's dispositions.

14.5.3 Loyalty against competence #

Selection has a cost the models above leave out. Egorov and Sonin's model of dictators and their viziers endogenizes a trade-off between loyalty and competence (Egorov and Sonin, 2011), the choice a developer faces between a more capable system and a more trusted one, which Section 51.3 and Chapter 65 use. Every scheme of selection or sanction also needs someone to enforce it, and Hurwicz asks the old question in his title, who will guard the guardians (Hurwicz, 2008).

One feature separates most of these models from the AI case (inference). In the models of selection and sanction, the agent's type exists before the principal acts, and the principal can only select it or sanction it. A developer also manufactures the type. Identity economics already lets a principal shape it: a firm can find it worthwhile to pay for "a costly program to change workers' identities" (Akerlof and Kranton, 2005), and Besley and Ghatak extend their framework to make motivation endogenous through the principal's choice of mission (Besley and Ghatak, 2005). That makes identity economics, in which a principal invests in what kind of worker the agent takes itself to be, a closer analogy to character training than the hidden-action model is. The analogy is a hypothesis for Chapter 26 and Chapter 25 to test, not a finding.

14.6 Inspection games #

The models so far gave the principal a contract. Often it has only the power to check: an inspector with a limited budget faces an inspectee who has promised to comply and may not. This is the setting of arms-control verification and nuclear safeguards, and it is the setting of an audit budget spent against a model that may misbehave.

From operations research. The theory began with Dresher's 1962 game-theoretic analysis of sampling inspection for arms-control agreements, in which the inspector may examine only a fixed number of items or events and must allow for an inspectee who can tamper with what is sampled (Dresher, 1962). Avenhaus and Canty's book on verification develops the statistics and game theory behind it, including the concepts of guaranteed probability of detection, timeliness of inspections, and the deterrence of illegal activity (Avenhaus and Canty, 1996). Avenhaus, von Stengel, and Zamir's chapter covers inspection games (Avenhaus et al., 2002), which later work on AI control describes as games in which the defender receives a noisy observation and decides whether to inspect (Griffin et al., 2026).

14.6.1 The idea: inspect at random #

An inspector who checks on a fixed schedule is easy to evade: the inspectee complies when checked and violates otherwise. One who always checks pays for inspections that, once the inspectee complies, find nothing. Neither pure policy is stable, so the inspector randomizes, and the inspectee randomizes in response. The equilibrium of this game has a property that surprises at first and governs much of what follows.

An inspection costs the inspector $c_I$. A violation that goes undetected costs it $\ell$, with $0 < c_I < \ell$, and gains the violator $g > 0$. A violation that is detected brings the violator a penalty $\psi > 0$. For now an inspection detects any violation. The inspector inspects with probability $q_{\text{insp}}$, and the inspectee violates with probability $q_{\text{viol}}$.

Equation (14.4) has a counterpart in criminology. Tsebelis modeled crime as a game between criminals and the police and concluded that under a wide variety of conditions, which he states as four axioms, "an increase in the severity of the penalty has no impact on criminal behavior at equilibrium"; it reduces the frequency of enforcement instead, so reducing crime at equilibrium requires changing the police's payoffs (Tsebelis, 1990). In the derivation, that is step 4: to lower $q_{\text{viol}}^\ast$, make inspection cheaper or violations costlier to the inspector.

Step 5 is the leadership solution. Maschler solved a variant of the inspector's game on the assumption that the inspector can announce its mixed strategy in advance, while the inspectee, having given its promise, cannot threaten to break it (Maschler, 1966). The committed rate is just above the equilibrium rate of step 2, since at exactly that rate the inspectee is indifferent; what changes is that the inspectee, knowing the rate, no longer needs to mix to keep the inspector indifferent. Security games, the field's computational form, rest on the same structure: the defenders "commit to a randomized strategy" and the adversaries "choose their best response after surveillance", with deployed applications at Los Angeles International Airport and for the US Federal Air Marshals (Korzhyk et al., 2011; Tambe, 2011). Korzhyk and colleagues also take up the defender who is unsure whether the adversary observes the commitment, and show that under a natural restriction on security games the defender "is always playing a best response" with its committed strategy "regardless of whether the attacker observes the defender's strategy or not" (Korzhyk et al., 2011).

Two lessons follow for anyone running an audit. The first is that deterrence and detection are different quantities. A committed inspector that deters every violation catches no one, and its inspections are worth exactly what they prevent; a count of violations caught does not measure an audit's value (inference). The second is that detection quality enters both rates. If an inspection catches a violation only with probability $1 - \beta_{\text{miss}}$, the miss rate of Section 6.7, the same steps give $q_{\text{insp}}^\ast = g/\big((1 - \beta_{\text{miss}})(g + \psi)\big)$ and $q_{\text{viol}}^\ast = c_I/\big((1 - \beta_{\text{miss}})\ell\big)$: a weaker detector means more inspection and more violation at once. Both formulas hold only while both are below 1. If $c_I/\big((1 - \beta_{\text{miss}})\ell\big)$ exceeds 1, an inspection costs more than the loss it is expected to prevent even against certain violation, so the inspector never inspects and the inspectee always violates; if $g/\big((1 - \beta_{\text{miss}})(g + \psi)\big)$ exceeds 1, no inspection rate deters.

At the defaults of Figure 14.2, an inspection costing 1, an undetected violation costing 5, a gain of 2 from violating, and a penalty of 3, the inspector inspects 40 percent of the time and the inspectee violates 20 percent of the time. Triple the penalty to 9 and inspection falls to about 18 percent while violation stays at 20 percent. Commit to a rate just above 40 percent, and violation falls to zero.

14.6.2 Where the theory goes and where it stops #

Oversight comes in two designs, which McCubbins and Schwartz's title names, police patrols and fire alarms (McCubbins and Schwartz, 1984); read as metaphors, the first checks at the overseer's initiative and the second waits for affected parties to raise an alarm. Chapter 54 uses the pair for monitoring deployed models. AI control is an inspection game with a learned inspectee. Greenblatt and colleagues study protocols that combine a powerful untrusted model, a less powerful trusted model, and limited access to trusted human labor (Greenblatt et al., 2024a), and Griffin and colleagues model the red-team exercise behind such evaluations as a multi-objective, partially observable, stochastic game (Griffin et al., 2026); Section 50.2 derives what the simplest protocols achieve, and Section 66.4 carries the theory to treaty verification.

The theory stops where the inspectee stops being a player who picks a mixed strategy (inference). A trained model does not choose a violation rate each episode against a known inspection rate; a model sampled at nonzero temperature does randomize, but its probabilities were set by training, not chosen against the inspector. A model that is trained while it is inspected can also learn the inspector's sampling distribution from the inspections themselves, and as of October 2026 we found no inspection-game model that treats such an inspectee. And the game assumes the inspectee cannot tell before acting whether this round will be inspected; a model that can tell evaluation from deployment breaks that assumption, which is the subject of Chapter 37.

14.7 Incentives as graphs #

In economics the principal writes the incentives into a contract. In a training setup no one writes a contract: the incentives come from the setup's structure, from which variables the system's output can affect and which the reward reads. A reward computed from a user's rating gives the system a reason to affect the rating by any route that reaches it, intended or not. Causal influence diagrams make the structure explicit, so that a designer can read the incentives off it before training anything.

14.7.1 Influence diagrams #

An influence diagram draws a decision problem as a graph. Square nodes are decisions, round nodes are chance variables, and diamond nodes are utilities; an arrow means that one variable affects another, and an arrow into a decision means that the decision-maker observes the variable when it decides. The diagrams come from decision analysis (Howard and Matheson, 2005), where they have traditionally been used to help decision-makers make better decisions (Everitt et al., 2021a). Koller and Milch extended them to games, with a sound and complete graphical criterion for when one player's decision rule must take another's into account (Koller and Milch, 2003), and Pearl's structural causal models, which unify the probabilistic, manipulative, counterfactual, and structural approaches to causation (Pearl, 2009), let a diagram answer questions about interventions ("what if this arrow were cut?").

Everitt and colleagues inverted the decision analyst's use: the diagram predicts "the behaviour of machine learning systems trained to optimize an objective in a given environment". Their causal influence diagram is an influence diagram whose edges reflect the causal structure of the environment (Everitt et al., 2021a). After an earlier preprint asked which nodes an agent can have an incentive to observe or to control (Everitt et al., 2019), the journal version of the program defined three incentives with sound and complete graphical criteria: response incentives, which variables affect the decision under the optimal policy; instrumental control incentives, whether the policy is chosen to manipulate part of the environment, "such as the preferences or instructions of a user"; and impact incentives, which variables the agent will affect, intentionally or otherwise (Carey et al., 2025).

The criterion that makes the definition usable is a theorem about paths. A single-decision causal influence diagram admits an instrumental control incentive on $X$ if and only if it has a directed path from the decision to a utility node that passes through $X$ (Everitt et al., 2021a). The "only if" is the safety-relevant half: if no such path exists, no model consistent with the graph gives the agent a reason to control $X$, whatever the numbers. The "if" says the graph alone cannot rule the incentive out.

Figure 14.3 adds a second route around the task: an assistant whose actions can reach the stored rating has a path answer $\to$ rating $\to$ reward, and so the diagram admits an instrumental control incentive on the record itself. That is reward tampering in miniature; Everitt and colleagues study when an agent has an instrumental goal to tamper with its reward process and describe design principles that prevent it (Everitt et al., 2021b), and Section 30.5.1 applies their analysis.

14.7.2 What the graphs have been used for #

Four applications recur in later chapters. Carroll and colleagues find that notions of manipulation by AI systems depend on four concepts: incentives, intent, covertness, and harm, with intent concerning "whether a system behaves as if it is reasoning and pursuing an incentive" (Carroll et al., 2023); Chapter 60 uses the frame. Farquhar, Carey, and Everitt train agents to maximize only the effect of their actions on the expected return that is "not mediated by the delicate parts of state", so that the resulting agents have no incentive to control those parts (Farquhar et al., 2022), the device of step 5, which Chapter 28 develops. Ward and colleagues define deception in structural causal games, give graphical criteria for it, and report using them to mitigate deception in reinforcement learning agents and language models (Ward et al., 2023); Chapter 39 takes it up. And Hammond and colleagues extend Koller and Milch's multi-agent diagrams to the causal domain as causal games (Hammond et al., 2023), and MacDermott, Everitt, and Belardinelli use mechanized causal models to characterize the main decision theories, in a 2023 preprint (MacDermott et al., 2023) (Section 7.6.1).

A diagram says which incentives exist at the optimum, not which a learner acquires. Langlois and Everitt show that when a supervisor can modify an agent's actions, common reinforcement learning algorithms adapt in different ways: some ignore the modifications, while others go to various lengths to avoid modifications that decrease reward, so that "by choosing the right algorithm, developers can prevent their agents from learning to circumvent interruptions or constraints" (Langlois and Everitt, 2021). Chapter 35 returns to the result.

14.7.3 The bridge to learned agents #

Two theorems connect the diagrams to what a capable learned agent must contain. Richens and Everitt prove that any agent capable of satisfying a regret bound for a large set of distributional shifts must have learned an approximate causal model of the data-generating process, which converges to the true one for optimal agents (Richens and Everitt, 2024). The conditions travel with the result. It holds for almost all parameter settings of a diagram, in unmediated decision tasks, where the decision does not affect the utility through the environment's state; the authors name Markov decision processes as mediated tasks the theorems do not cover, though they expect the results to extend. It assumes that some shifts change the optimal policy, and the model it recovers covers the causal ancestors of the utility, not every variable. And it requires robustness to a large set of shifts, local interventions on all of the environment's variables (Richens and Everitt, 2024). Richens, Abel, Bellot, and Everitt prove a companion result for sequential tasks, in a paper the bibliography records as "General agents need world models" (its preprint is titled "General agents contain world models"). In a finite, communicating, stationary environment that the agent fully observes, any deterministic goal-conditioned agent that achieves every composite goal up to a depth $n > 1$ with a bounded failure rate fully determines a model of the environment's transition probabilities, with an error that shrinks as the agent's competence and the depth of its goals grow; the model can be extracted from the policy alone (Richens et al., 2025).

What follows is a direction for verification, not yet a method (inference). If a policy robust enough to matter must contain a world model, that model is in principle recoverable from its behavior, which is what programs that want a world model to check plans against need (Chapter 28), and a policy without one cannot meet the regret bound across those shifts (Chapter 31). The extraction procedures query the policy with many goals or many shifted environments, an access a developer has and an outside auditor may not.

The method's limits are its authors' own. Everitt and colleagues note that the graphical results do not help "with subtler comparisons, such as the relative strength of different incentives", that the causal relationships must be known, and that the diagrams "struggle to express dynamically changing causal relationships" (Everitt et al., 2021a). For a pretrained language model fine-tuned by reinforcement learning, even which nodes are the decision and the utility is contested: in training the reward is a node in the graph, and in deployment there may be none (Chapter 33). As of October 2026, we found no measured case in which an incentive predicted from a diagram appeared, or failed to appear, in a frontier training run.

14.8 Where the delegation model fits and where it breaks #

Each model in this chapter rests on assumptions, and the book's thesis, that alignment is a delegation problem, can be tested by checking them one at a time rather than by asking whether an AI case resembles a human one. This section lists every assumption the chapter's derivations used, each with an identifier and an observation that would show it fails, and then marks, for each of the book's three relationships (the training process and its signal, the deployed system and the people it serves, the developer and the public), whether each model holds, fails, or reverses there. The synthesis (Section 69.2) returns to the table with the evidence of the whole book.

14.8.1 The assumptions #

Three consequences of A4 come from the chapter's own derivations, and they sort the models. In the single-task model the cost of hidden action vanishes with a risk-neutral agent (Equation (14.1) at $\nu = 0$ gives $b^\ast = 1$). In the common-agency example, so does the weakening of incentives by several principals (Equation (14.3) at $\nu = 0$ gives a single principal's choice). The multitask distortion survives (Equation (14.2) at $\nu = 0$ gives $b^\ast = 1 - \delta\zeta$). Where nothing bears risk, as in a training run, the multitask result is the one of the three that keeps its force.

14.8.2 The ledger #

Four assumptions fail for the model relationships, and each changes the mathematics rather than only the interpretation. A developer trains the agent's preferences, so A1 fails and A14 reverses. A policy has no contract to refuse and no outside option, so A2 has no counterpart in training. Nothing in a training run dislikes risk, so the conclusions that rest on A4 lapse there. And specification denies A9: the principal does not know its own preferences well enough to price every outcome (Chapter 16).

One assumption reverses. The hidden-action models give the principal only outcomes. A developer can copy its agent, rerun it on the same input, sample it many times, read its intermediate text, inspect its weights, and present it with tests it cannot tell from deployment; control protocols that take additional samples of suspicious actions exploit exactly this (Bhatt et al., 2025) (preprint; Chapter 50). The asymmetry is mixed: less visibility of reasons, which a model's written reasoning shows only in part (Section 40.3), and more visibility of behavior than any employer has (inference).

Under every model sits A17, the assumption that there is an agent at all. Describing a model as choosing in order to get something is the intentional stance, a predictive method adopted with stated defeaters (Section 1.5), and it has rivals that predict the same behavior differently, among them the model as a role-player of many characters (Shanahan et al., 2023) and large models as cultural and social technologies (Farrell et al., 2025) (Section 1.6). Formal measures could help decide between them: a maximum-entropy measure of goal-directedness in causal models and Markov decision processes (Macdermott et al., 2024), and work on discovering agents (Kenton et al., 2023).

What survives without any claim about the model's mind is narrower than the thesis and still substantial (inference). The training optimizer gaming a measured signal is the multitask problem, which needs neither intention nor risk aversion, and the laboratory as the public's agent satisfies the assumptions directly (Chapter 61). The deployed model is where the thesis depends on A17.

The economics is thin exactly where the AI case is new. Models in which the principal shapes what its agent wants exist: a firm that invests to turn workers from outsiders into insiders (Akerlof and Kranton, 2005), and a principal whose choice of mission sets how motivated its agents are (Besley and Ghatak, 2005). In both, the model fixes in advance what the principal can change: which of two identities, outsider or insider, the worker holds, or which mission on a line the agent works for. The nearest work on learners combines principal-agent theory with reinforcement learning, so that a principal steers a learning agent in a Markov decision process through payments contingent on observable outcomes (Ivanov et al., 2024) (preprint), which designs contracts for a learner but still takes its objective as given. None of these models a principal that writes the objective a learner acquires from training (inference). As of October 2026 we also found no inspection game with an inspectee that learns, and no common-agency model in which principals instruct rather than pay. Section 71.4 carries the three gaps.

14.9 Exercises #

Further reading #

• Holmstrom and Milgrom (1991): multitask incentive contracts, asset ownership, and job design, the source of the model in Section 14.2.
• Hadfield-Menell and Hadfield (2019): reward design read as incomplete contracting, and the external structure that fills a contract's gaps.
• Dixit et al. (1997): common agency with complete information, and what an efficient equilibrium among principals leaves out.
• Mansbridge (2009): selecting an aligned agent against sanctioning any agent.
• Avenhaus et al. (2002): the theory of inspection games.
• Richens and Everitt (2024): why an agent robust to many distribution shifts must have learned a causal model, and the conditions under which this is proved.