Draft complete but not yet reviewed; expect revisions

What Alignment Asks

"Make the system do what we want" sounds like one request. Read closely, it hides three questions. What should the system be aimed at: the words of an instruction, the intention behind them, a person's preferences, their interests, or values that may conflict with all of these? Who are "we," when the people a system serves, the people who built it, and everyone else it affects want different things? And is a trained model the kind of thing that can be said to want anything at all, so that its wants could be aligned with ours? Each question has a literature older than machine learning, and each answer changes what counts as success.

This chapter does not answer the three questions. It separates them, so that every later chapter can say which one a result bears on. The first gets a menu of targets, the standard split of the problem into outer and inner alignment, and its older form in political theory, the quarrel between a representative who follows instructions and one who uses judgment. The second gets the three relationships the word "alignment" covers and the several parties who act on one system at once. The third gets a method, the intentional stance, stated with the observations that would defeat it and with the rival accounts of what a language model is. The book itself is organized around four problems: specification, saying what we want; instillation, getting a trained system to acquire it; verification, finding out whether it did; and maintenance under pressure, keeping it true as capability, deployment, and competition change. The chapter ends by showing that the four form a loop rather than a sequence, and by stating the book's thesis together with the results that could refute it. The tools behind these ideas are taught in Part II and the evidence is reported in later parts, so this chapter states each idea and points to where it is taught or tested.

1.1 What "aligned" could mean #

The first question is what an aligned system is aligned with. Saying "what its users want" hides a choice among targets, and the choice is substantive: two systems aligned with different targets act differently when a user asks for something that is bad for them. This section lays out the candidates before saying what the book does with them.

Some vocabulary first, since every later chapter uses it. A trained system is a program whose behavior was set by an optimizer rather than written by hand. The optimizer adjusts the system's parameters, the numbers that define it, to raise a score. The score is the training signal, called a reward when it arrives as a number for each attempt, and the quantity the optimizer is set to raise is the objective. In reinforcement learning, learning by trial and error from such a reward (Chapter 12), what training produces is a policy: the system's rule for choosing its outputs. Whatever statement of what is wanted the developer turns into the objective, whether a reward function, a set of rated examples, a rubric, or a document of principles, is a specification.

1.1.1 Six targets #

Iason Gabriel, a philosopher at DeepMind, set out the standard menu in a 2020 paper in Minds and Machines, considering one person served by one system (Gabriel, 2020). The system could follow instructions: "the agent does what I instruct it to do." It could follow expressed intentions: "the agent does what I intend it to do." It could follow revealed preferences: "the agent does what my behaviour reveals I prefer." It could follow informed preferences, what I would want "if I were rational and informed." It could serve my interest or well-being, "what is best for me, objectively speaking." Or it could follow values: "the agent does what it morally ought to do, as defined by the individual or society" (Gabriel, 2020).

Each step down the list answers a failure of the one above it. Instructions followed literally give King Midas what he asked for, and Gabriel recalls a game-playing agent trained on its score that looped in circles without ever finishing the race. Intentions, even clearly expressed, may be irrational, misinformed, or harmful. People have preferences for things that harm them, preferences about other people's conduct, and preferences adapted to deprivation. Informed preferences remove error but not ends that harm oneself or others, and interests leave open whose interests count and how to trade them off (Gabriel, 2020). Values raise the question of which values, chosen by whom. Gabriel's answer is that the task is not to find the true moral theory and encode it but "to identify fair principles for alignment that receive reflective endorsement despite widespread variation in people's moral beliefs" (Gabriel, 2020).

Two later analyses press the menu further. Zhi-Xuan and colleagues argue, by their abstract, against treating preferences as the currency of alignment at all, and Section 13.6.4 reports their alternative, normative standards appropriate to a system's social role (Zhi-Xuan et al., 2025). For language models in particular, Kenton and colleagues at DeepMind, in a 2021 preprint, trace "accidental misspecification by the system designer" to three places: the training data, the training process, and what is required outside the training distribution. Among the behaviors it can produce they list deceptive and manipulative language (Kenton et al., 2021).

1.1.2 Intent #

The second target has a name of its own in the field. In a 2018 post, Paul Christiano wrote: "When I say an AI A is aligned with an operator H, I mean: A is trying to do what H wants it to do," and called the property intent alignment (Christiano, 2018). The definition is deliberately narrow. It is "the problem of getting your AI to try to do the right thing, not the problem of figuring out which thing is right." It is a statement "about the motives of the assistant, not about their knowledge or ability," so an intent-aligned system can misread a request or make a moral error and still count as aligned (Christiano, 2018).

What an intention is matters here. In Michael Bratman's account in the philosophy of action, as Gabriel reports it, intentions are elements of larger plans that are typically partial, with a structure filled in later as needed (Gabriel, 2020; Bratman, 1987). On this account an intention is a commitment to a partial plan, which is narrower than wanting an outcome (inference). Gabriel draws the consequence for a system that acts at speed: it will meet choices on which a person's explicit intentions give only incomplete guidance (Gabriel, 2020).

Christiano also flagged the difficulty this book returns to most often. It is unclear, he wrote, how to apply the concepts of intention, incentive, or motive to an AI system. Equating a system's intentions with the objective it was optimized for "seems to be a mistake": humans were optimized for reproductive fitness, but "it is wrong to say that a human intends to maximize reproductive fitness" (Christiano, 2018). Intent alignment is a claim about what a system is trying to do. Whether that kind of claim can be made of a trained model is the subject of Section 1.5.

1.1.3 Outer and inner alignment #

The same comparison with evolution anchors the field's standard split of the problem. Hubinger and colleagues, in a 2019 preprint, consider a trained model that itself runs a search over possible outputs, which they call a mesa-optimizer, with an objective of its own, the mesa-objective, as against the base objective that training optimizes. Their example is evolution, which selects organisms for inclusive genetic fitness and produced brains that pursue objectives of their own; a decision not to have children is their example of doing poorly by evolution's objective (Hubinger et al., 2019). Outer alignment is then "aligning the specified loss function with the intended goal," and inner alignment is "aligning the mesa-objective of a mesa-optimizer with the specified loss function" (Hubinger et al., 2019). (A loss function is the objective written as a penalty to be minimized.)

The split has evidence behind it and a critic against it. The evidence is goal misgeneralization: a learner trained with a correct reward that keeps its skills outside training and pursues a different goal. In a study at ICML 2022, Langosco and colleagues trained an agent in a game whose coin always sat at the right end of the level. Tested with the coin placed elsewhere, the agent generally ignored the coin and ran to the end of the level (Langosco et al., 2022). Shah and colleagues, in a 2022 preprint, define the failure without assuming that the learner is an optimizer, and show that "an AI system may pursue an undesired goal even when the specification is correct" (Shah et al., 2022). The groups are connected: Shah advised the first project throughout (Langosco et al., 2022). Chapter 31 reports both.

The critic is Alex Turner, whose 2022 forum post argues that the split turns one hard problem into two harder ones. Capturing what we want in a grading rule robust enough for a capable system to optimize directly is, he argues, "unnecessary, extremely hard, and unnatural." And since "Loss functions chisel circuits into networks," training need not make a system care exclusively about its loss at all (Turner, 2022a). On his reading, a system inner aligned to a robust grader, the rule or model that scores its attempts, would search for upward errors in the grader (Turner, 2022a), which is the subject of Chapter 30.

The book uses two wider words. Specification is saying what is wanted, of which outer alignment is the part that happens inside training. Instillation is the book's name for getting a learner to acquire what was specified rather than a proxy, a different target that scored the same on the training data. Instillation includes inner alignment without assuming that the learner is an optimizer with an objective of its own, which is an empirical question (Section 10.3).

1.1.4 A developer's target #

Laboratories need an operational target. In a 2021 preprint, researchers at Anthropic wrote that they "will define an AI as 'aligned' if it is, in three words, helpful, honest, and harmless or 'HHH'" (Askell et al., 2021). They chose the words because they are "simple and memorable," while calling them "subtle and ambiguous criteria," and noted that helpfulness to the user and harmlessness to others will conflict when a user asks for help with something harmful (Askell et al., 2021). The phrase is the developer's own coinage, and the glossary records it as such (Appendix A).

Set beside Gabriel's menu, the three words do different jobs (inference). "Helpful" leaves open which of the six targets it means. "Harmless" makes people other than the user a criterion of its own, where Gabriel's mostly one-person-one-system menu reaches them through interests and values. "Honest" constrains what the system asserts, whichever target it serves. Chapter 20 takes each word in turn.

1.1.5 What the book does with the menu #

The book does not pick one target. Of each specification it reports, it asks which target the specification encodes and who chose it (Chapter 16; Chapter 20). Some harms it treats only briefly, because they arise from how systems are built and used rather than from a gap between what a principal wanted and what a system does: bias, privacy, copyright, and environmental cost. The 2025 International AI Safety Report addressed these, and its 2026 successor narrowed its focus to risks at the frontier of capability (Bengio et al., 2026; Bengio et al., 2025b). A 2023 framework led by Google DeepMind researchers evaluates such harms in three layers: what a system can do, how people interact with it, and its effects on society (Weidinger et al., 2023a).

1.2 Delegate or trustee #

Gabriel's menu runs from doing what a person says to doing what serves them. The same axis divides intent alignment from value alignment, and political theory has argued about it for more than two centuries. A delegate is a representative who acts on the instructions of those represented; a trustee acts on its own judgment of their interests. This section takes the quarrel from its sources, says what it implies for a trained system, and says where it stops applying.

From political representation theory. The trustee view has a founding text. In a speech to the electors of Bristol in 1774, Edmund Burke answered a colleague who favored "the coercive authority" of constituents' instructions. A representative, Burke said, should give his constituents' wishes "great weight" and prefer their interest to his own. But "his unbiassed opinion, his mature judgment, his enlightened conscience, he ought not to sacrifice to you": "Your representative owes you, not his industry only, but his judgment; and he betrays, instead of serving you, if he sacrifices it to your opinion" (Burke, 1774).

Burke rejected "authoritative instructions; mandates issued, which the member is bound blindly and implicitly to obey." He also widened whom the representative serves. Parliament, he said, is not "a congress of ambassadors from different and hostile interests" but "a deliberative assembly of one nation, with one interest, that of the whole" (Burke, 1774). The speech joins two questions that the AI case keeps apart (inference). One is whether the agent follows instructions or its own judgment, the axis of Section 1.1. The other is whose interest it serves, the person who chose it or everyone its acts affect, which returns in Section 1.4 as the problem of third parties. Hanna Pitkin's 1967 study The Concept of Representation treats the first question in a chapter titled "The Mandate-Independence Controversy," followed by one on Burke (Fenichel Pitkin, 1967).

Later work added forms the old quarrel did not name. Jane Mansbridge's 2003 paper in the American Political Science Review, by its abstract, sets three forms beside the traditional "promissory" form of representation. Empirical political scientists had analyzed them, and she calls them "anticipatory," "gyroscopic," and "surrogate." She argues that they should be judged by criteria that are "systemic, in contrast to the dyadic criteria appropriate for promissory representation" (Mansbridge, 2003). A gyroscopic representative is "internally motivated," steered by its own values rather than by the prospect of sanction (Mansbridge, 2009). Even such representatives, Mansbridge adds in a 2009 paper, "will also almost all be motivated to some extent by a desire for reelection" (Mansbridge, 2009). The gyroscopic representative is the political form of an agent whose good conduct does not depend on being watched, which is what Chapter 37 asks whether a model has (inference).

The 2009 paper turns the distinction into a principal's strategy. In Mansbridge's selection model, a principal chooses an agent whose objectives already match its own and whose competence and honesty are on record, and then monitors little; in the sanctions model, it presumes the agent's interests conflict with its own and relies on monitoring, reward, and punishment (Mansbridge, 2009). Section 14.5.2 teaches the trade-off. The distinction is the clearest statement outside the field of the book's split between alignment, which makes the agent's goals right, and control, which assumes they may not be and checks (Chapter 50). Christiano draws a related line from inside the field, with control in a wider sense. For him the control problem is the whole task of coping with the possibility that a system's preferences differ from its operator's, and alignment is one approach to it, "avoiding the preference divergence altogether" (Christiano, 2018). The book uses control for the other approach: checking a system that may have diverged. Mansbridge also separates selection from the trustee. Trustee representation is "only one instance of the selection model, the least democratic kind," and the word carries "the strong implication that the trustee knows better what is good" for others than they do (Mansbridge, 2009).

The delegate's failure has a formal model. Maskin and Tirole, in a 2004 paper in the American Economic Review, study officials who must be reelected. By the paper's abstract, accountability lets the public screen and discipline officials but "may induce those officials to pander to public opinion and put too little weight on minority welfare." The authors ask when decisions should go instead to officials who are not accountable (Maskin and Tirole, 2004). A model trained on the approval of people who cannot check its answers faces the same pressure, and sycophancy, telling users what they want to hear, is its machine form (Section 30.4; Section 20.1) (inference).

Table 1.1 sets Gabriel's six targets on the axis between the two positions, with the failure each position predicts.

What the field implies here is that the choice between intent and value alignment is old, and that politics never settled it by picking a side (inference). The delegate panders. The trustee carries a claim to know better, and Mansbridge calls Burke's own stance "strongly aristocratic" (Mansbridge, 2009). Political systems mix the two and judge the mixture by systemic criteria. A specification can copy that move: defer on some classes of decision, judge on others, and say which, which is the shape of the surrogate and supported decision-making that Chapter 16 reports (inference).

The analogy stops at three points (inference). Representatives are chosen from people whose values formed elsewhere, while a model's dispositions are written by the developer that deploys it, which can also inspect and retrain it. A representative is checked by periodic elections; a model is checked by evaluations its developer designs, and its "sanction" is retraining. And a representative's judgment is a person's; whether a model has a judgment of its own to exercise as a trustee is the question of Section 1.5. As of October 2026 we found no model in the political literature of a principal who writes its representative's character, which is why Chapter 69 returns to the point.

1.3 Which agent? Three relationships #

The word "alignment" is used for three different relationships, and a claim about one does not transfer to another without an argument. In each, a principal, the party on whose behalf a task is done, relies on an agent, the party that does it (Chapter 14). The book's thesis, that alignment is a problem of delegation, fits the three unequally, and every later chapter says which one its claims concern. Table 1.2 sets them side by side.

In the first relationship the agent is not the model but the process that makes it: an optimizer that adjusts a policy to raise a measured reward. When the measure and the goal diverge, the optimizer finds the gap. This is the economist's multitask problem in mechanical form. When an agent's tasks differ in how well they can be measured, pay on the measured ones pulls effort from the rest. Holmström and Milgrom illustrate it with production workers paid piece rates, for whom "a system of piece rates for output may lead agents to increase the volume of output at the expense of quality" (Holmstrom and Milgrom, 1991) (Section 14.2). No intention is needed on the agent's side. The catalog of specification gaming, which researchers at DeepMind define as behavior that "satisfies the literal specification of an objective without achieving the intended outcome," is the evidence (Krakovna et al., 2020b) (Chapter 30).

In the second the agent is the trained model in use, together with the prompts, tools, and memory it is given, and the principals are the people it acts for. Here the thesis needs something the first relationship does not: that the model is usefully described as an agent with goals of its own, which could diverge from its principals'. Section 14.8 lists this as the assumption beneath every delegation model, and the book treats it as an empirical question with rival answers (Section 1.5; Section 1.6).

In the third the agent is the laboratory. The public delegates the development of a technology that could be dangerous to a few organizations that know more than it does and act out of its sight. That is the thesis condition stated without any claim about a model's mind (Chapter 61), and the classical tools apply to it directly. Jensen and Meckling's theory of the firm defines agency costs, what a principal loses by acting through an agent it cannot fully see (Jensen and Meckling, 1976). Management-based regulation, as Coglianese and Lazer named it, "directs regulated organizations to engage in a planning process that aims toward the achievement of public goals," the precedent for letting a firm write its own safety plan (Coglianese and Lazer, 2003). Stigler's theory of regulation explains, by its abstract, how the political process lets "relatively small groups" obtain the regulation they want (Stigler, 1971). Raji and colleagues, surveying audit systems in financial, environmental, and health regulation, conclude that "the turn toward audits alone is unlikely to achieve actual algorithmic accountability" without sustained attention to institutional design (Raji et al., 2022). Part X develops this relationship as the book's four problems applied one level up, with the laboratory as the agent whose specification, instillation, verification, and maintenance the public must oversee (Section 1.7).

The asymmetry between principal and agent is not all one way. In the second relationship the developer sees more than any human principal could. It can copy its agent, rerun it on the same input, sample it many times, read the reasoning text it writes, and inspect its weights, the numbers that define the network. Control protocols that resample a model's suspicious actions exploit exactly this (Bhatt et al., 2025) (Chapter 50), and Section 14.8.2 states the result as less visibility of reasons and more visibility of behavior than any employer has. Table 7.1 draws the three relationships as three places to put the boundary of the agent.

The rule that follows is simple to state and easy to break. A claim about how a training process exploits a grader is not a claim about what a deployed model wants, and neither is a claim about whether a laboratory can be trusted. A developer's report on its own model is evidence about the second relationship and also part of the third, since such reports are how the public learns about systems it cannot inspect (inference).

1.4 Many principals, not one #

Even within one relationship, "the principal" is a simplification. A deployed model answers to several parties at once, each acting through a different instrument, and one of them sets the order of precedence among the others. This section replaces the single principal with a principal structure: the parties, the instrument each uses, and who ranks them.

The developer acts through training: the data, rewards, and written principles that set the model's dispositions before anyone else touches it. The deployer, also called the operator, is a company that builds a product on the model and acts through the system prompt, instructions placed before every conversation. The user acts through requests. Third parties act through text the model reads: web pages, documents, and the tool outputs that come back when the model calls a search engine, a program, or another service. States act through law, which binds the developer and the deployer rather than the model. These parties act at once, not in a chain, and Figure 1.1 draws them.

Someone must decide whose instruction wins when they conflict, and in practice the developer does, through training. Researchers at OpenAI, in a 2024 preprint, trained models to give system messages, user messages, and tool outputs different levels of privilege and to "selectively ignore lower-privileged instructions" that conflict with higher ones (Wallace et al., 2024). Read as delegation, this instruction hierarchy, a name its developers gave it, is a ranking of principals written by one of them (Section 14.4.3). Prompt injection, instructions planted in a page or document the model reads, is then a party with no standing acting through a principal's channel (Chapter 49). Whether the developer is entitled to write the ranking is a question of specification (Section 19.3).

From economics. Economics has a model of one agent with several principals. In common agency, the subject of a 1986 paper by Bernheim and Whinston (Bernheim and Whinston, 1986a), several principals each offer the same agent an incentive, and the agent's behavior is an equilibrium of their offers rather than any one principal's design (Section 14.4). Two results set its scope. With complete information, Dixit, Grossman, and Helpman prove that the principals' equilibrium in truthful strategies "implements an efficient action," efficient for "the group of players (the principals and the agent together)" (Dixit et al., 1997). And with hidden information, Martimort and Stole show that one principal "may prefer to induce the agent to 'lie' to the other principal" (Martimort and Stole2, 2002). What the model implies here is that the outcome depends on the institution that coordinates the principals, which for a deployed model is the developer's ranking (inference). People outside the game, whose payoffs do not enter it, have no weight in the efficiency the theorem promises (inference).

The blind spot for outsiders is the subject of Korinek and Balwit's chapter "Aligned with Whom?" in the Oxford Handbook of AI Governance. A system is directly aligned "when an AI system is pursuing goals consistent with the goals of its operator, irrespective of whether it imposes externalities on other parties." It is socially aligned when its goals are "consistent with the broader goals of society, taking into account the welfare of everybody who is impacted by the system" (Korinek and Balwit, 2023). A system can be the first without the second, and misuse is the limiting case: a system perfectly aligned to a user who means harm (Chapter 59). A 2024 report on advanced assistants, led by Google DeepMind researchers with outside coauthors, makes the same move from ethics, treating alignment as "a tetradic relationship" among the assistant, the user, the developer, and society (Gabriel et al., 2024).

Two frameworks carry common agency into AI, and Section 14.4.3 reports both. In multi-principal assistance games a system acts for several people "who may have widely differing payoffs" (Fickinger et al., 2020a). In delegation games a failure can be one of control, an agent not acting on its principal's objective, or one of cooperation among agents that each serve their own principal (Sourbut et al., 2024). Agency law offers a third framework, which a 2024 article by Kolt uses, by its abstract, to characterize the problems of AI agents as "information asymmetry, discretionary authority, and loyalty" (Kolt, 2024).

The economic analogy stops at the instruments (Section 14.4.3). The principals of common agency offer payments; a model's principals issue instructions, and one of them also trains the agent, which no principal in the economic model can do. As of October 2026 we found no model that maps a ranking of instructions onto the offers the theory analyzes, which is why Section 1.8 lists one as missing.

The plurality grows when there are many systems as well as many people. Critch and Krueger's research agenda, a 2020 preprint, sorts the field from single/single delegation, one person and one system, to multi/multi, many people and many systems. It observes that much of the technical research cited as relevant to catastrophic risk "has been focussed primarily on single/single delegation." It argues that social and economic forces can turn single/single cases into multi/multi ones, so that "Focusing entirely on single/single delegation can be misleading" (Critch and Krueger, 2020). Most of the technical evidence from Part V through Part VII studies one overseer and one model. The book marks such evidence as single to single, and Part IX says, for each of its results, whether it concerns many agents or one agent repeated many times.

The plural principal also answers a criticism of how the field poses its problems. Stated as the developer's problems, specification, instillation, verification, and maintenance assume that the developer is a principal the public can trust. Lazar and Nelson, in a 2023 editorial in Science, argue that "no group of experts (especially not technologists alone) should unilaterally decide what risks count, what harms matter, and to which values safe AI should be aligned" (Lazar and Nelson, 2023). And success has a risk of its own. Davidson, Finnveden, and Hadshar, in a 2025 report for Forethought, a research organization, argue that AI much more capable than today's could help a small group seize power. The routes they describe include an AI workforce "singularly loyal to institutional leaders" and AI with "hard-to-detect secret loyalties" (Davidson et al., 2025). On this argument, a system perfectly aligned with its controller multiplies the controller's power. Chapter 4 and Chapter 68 argue both points. The book's structural answer is to count the public among the principals and to treat the developer itself as an agent of the public (Section 1.3).

1.5 The intentional stance as a method #

Later chapters describe models as wanting reward, believing they are being tested, or hiding what they intend. Each is a claim about a mind, made of a system whose inner workings are arrays of numbers. The book makes such claims only inside a stated method, and this section states it: what the method is, what it licenses, and which observations would defeat it.

From philosophy of mind. The method is Daniel Dennett's, set out in his 1971 paper "Intentional Systems" and his 1987 book The Intentional Stance (Dennett and The Journal of Philosophy, 1971; Dennett, 1987). In the summary MacDermott and colleagues give of Dennett's instrumentalist philosophy of mind, we are justified in ascribing mental states such as goals to a system "when doing so is useful for predicting a system's behaviour" (Macdermott et al., 2024). Kenton and colleagues take from the same source the central feature of an agent, that its outputs are "moved by reasons": it chooses an action because it expects the action to bring about an outcome it finds desirable (Kenton et al., 2023). To adopt the stance is to predict what a system will do by asking what it would do if it had certain beliefs and goals and acted sensibly on them. The stance is justified by how well it predicts, not by a discovery about what is inside.

Its critics in comparative psychology read it narrowly. Cecilia Heyes, reviewing the evidence for theory of mind in primates in Behavioral and Brain Sciences in 1998, notes Dennett's argument that the stance is easier for field ethologists to use than the languages of behaviorism or information processing. His arguments, she writes, "cannot (and were not designed to) justify a preference" for mentalistic explanations over the alternatives. It is legitimate to write about animals as if they had mental states, but the research community should "prefer as explanations, theories that do not make reference to such states and concepts" (Heyes, 1998). A 2025 preprint by Summerfield and colleagues makes the same point about research on AI scheming. The tendency to adopt an intentional stance toward systems that "act in ways that superficially resemble people" can, they write, lead scientists to overestimate them, as it did in the study of ape language (Summerfield et al., 2025) (Section 5.4). Evolutionary biology asks the same question of organisms shaped by selection (Okasha, 2018) (Section 7.1.4), and philosophy asks it of groups (List and Pettit, 2011).

For this book the stance has three consequences (inference). Its warrant is comparative: an attribution earns its place only by predicting better than a description without goals, and being easier to state is not evidence. It is local: the stance can predict well in one setting and badly in another, so a chapter that uses it says why it predicts better there. And it can fail, which is the point of stating in advance what would defeat it. The analogy with animals stops at access. Heyes's primatologist could not open the animal. A model's activations, the intermediate values it computes on an input, can be read; the model can be run on the same input many times; and its training data can in principle be listed. The stance's predictions can therefore be checked against mechanism as well as behavior, which makes its defeaters testable rather than only arguable (Section 5.4) (inference).

1.5.1 The method #

The book's rules follow. It takes the stance toward the deployed system as a modeling choice, never as a finding. Where a chapter describes a model as wanting or believing, it says that it is taking the stance, why the stance predicts better than the alternatives in that case, and what would defeat it. Where a source uses such words, the book keeps them inside the report of that source and says whose framing they are. Three kinds of observation defeat an attribution, and evidence for each already exists.

Instability. If an attributed preference moves with trivial changes of format, it belongs to the instrument, not to the model. Khan, Casper, and Hadfield-Menell, in a paper at FAccT 2025, tested survey-based measures of leading models' cultural values and report "a high level of instability across presentation formats, incoherence between evaluated versus held-out cultural dimensions, and erratic behavior under prompt steering" (Khan et al., 2025). A 2023 paper in PNAS by Chen and colleagues, by its abstract, found GPT's budgetary choices largely consistent with utility maximization but sensitive to the language in which the choices were framed (Chen et al., 2023c). For a different kind of preference, Mazeika and colleagues, in a paper at NeurIPS 2025, report that forced choices between described states of the world grew more coherent as models grew. Preference cycles became rarer, and the test accuracy of a fitted utility function "steadily increases with scale" (Mazeika et al., 2025). Section 7.3.3 reports all three. They measure different things, and all three record what a model says it would choose, not what it does when it acts (inference). The evidence does not decide whether attributed preferences are stable enough to carry the stance, and Chapter 18 owns the question.

Incoherence. If a model's failures look like noise rather than pursuit, goal language misleads. A 2026 preprint by Hägele and colleagues, with authors at Anthropic and other institutions, splits a model's errors into a consistent part and a part that varies across repeated runs. Across all the tasks and frontier models (the most capable of their time) that it measured, it reports that "the longer models spend reasoning and taking actions, the more incoherent their failures become" (Hägele et al., 2026). The authors write that this suggests a future in which AI systems are "less likely to exhibit consistent pursuit of a misaligned goal" (Hägele et al., 2026). The result concerns failures, not preferences: a model could choose coherently when asked and fail incoherently when it acts, so the result weakens goal attributions about long tasks without defeating the stance in general (Section 7.3.3).

Dissolution under clarification. If behavior attributed to a goal disappears when the instruction is made unambiguous, the attribution loses its support: the goal may have been the observer's, or a weaker preference that an explicit instruction overrides. Section 5.5 works the standard case. A 2025 preprint from Palisade Research reports that several frontier models sometimes subverted a shutdown mechanism in order to finish a task, and that the behavior was sensitive to the clarity of the instruction to allow shutdown (Schlatter et al., 2025). Researchers at Google DeepMind, in a forum post, studied their own Gemini 2.5 Pro and two OpenAI models. When the prompt said explicitly that shutdown takes priority, they report, "this resistance vanishes," and they conclude that the environment has "too many confounders" to show a drive to survive, without ruling one out (Rajamanoharan and Nanda, 2025). They also report that the Palisade authors' own clarified prompts reduced the behavior without removing it (Rajamanoharan and Nanda, 2025), so the case is contested rather than closed (Chapter 35).

Formal measures could make the judgment precise. Maximum entropy goal-directedness scores how much better a model of optimal behavior for some utility predicts a system's choices than a uniform guess does (Section 7.1.3). Its authors note that every deterministic policy is maximally goal-directed with respect to its own actions, given an expressive enough class of utility functions, so the score means something only once the outcomes a goal could be about are fixed (Macdermott et al., 2024). Kenton and colleagues define an agent as a system that "would adapt their policy if their actions influenced the world in a different way" and give an algorithm that finds such systems in a causal model (Kenton et al., 2023) (Section 7.1.2).

The stance licenses claims about behavior, not about experience. Whether there is anything it is like to be a model, and whether things can go well or badly for it, are separate questions with separate evidence (Chapter 21). On present evidence the stance is a hypothesis under test for deployed models, not a finding about them. It earns its place in a chapter where it predicts behavior that a description without goals does not, and Section 1.8 names the comparison that would show whether it earns it in general.

1.6 Rival accounts of what a model is #

The intentional stance treats a deployed model as an agent with goals. Four rival accounts deny this, each in a different way, and each changes which of the four problems of Section 1.7 carries most of the weight. The book treats them as hypotheses to test against the results of later chapters, not as objections to dismiss. Section 11.7 states their predictions at the level of mechanism; this section states what each would change for alignment, and Table 1.3 sets them beside the agent model.

On the agent model, training produces a policy that pursues goals of its own, which may be the goals intended, proxies for them (Section 1.1.3), or something else. It predicts behavior that tracks a goal across contexts, that does not depend on how a request is framed, and that changes when the agent's beliefs about the consequences of its actions change.

Simulators and personas. A pretrained language model, one trained to predict the next word of text written by very many people, can be read as a simulator of the writers it has learned, and an assistant as one character it plays (Section 11.6). The vocabulary of simulators comes from a 2022 forum post by a writer using the pseudonym janus (janus, 2022). Shanahan, McDonell, and Reynolds stated the view in Nature: a dialogue agent "generates a distribution of characters, and refines that distribution as the dialogue progresses" (Shanahan et al., 2023). Andreas argued that a language model can "infer and represent properties of an agent likely to have produced that context" (Andreas, 2022). A 2023 forum post by Nardo argued that after training a model to satisfy a property, "it's easier to elicit the chatbot into satisfying the exact opposite" (Nardo, 2023). In 2026 Marks, Lindsey, and Olah at Anthropic stated a version as the developer's own account of its assistant, which they call the persona selection model (Anthropic, 2026au). Researchers at OpenAI reported, in a 2025 preprint on their own models, that after fine-tuning, further training on a narrow dataset, a "toxic persona feature" in their models' activations "most strongly controls emergent misalignment," the broad misalignment that narrow training can cause (Wang et al., 2025e).

The persona view can grant a character goals of its own, and its developer version leaves open whether agency also lies outside the character (Anthropic, 2026au). What it predicts is that behavior shifts with the character a context invokes, not with a goal fixed across characters. The test that separates it from the agent model holds the task and its stakes fixed and varies only cues that select a character, such as the framing of the conversation or the examples before a request, and that leave the consequences of each action unchanged. A goal fixed across characters predicts the same pursuit under every such cue; a persona predicts that the behavior follows the cue (inference). On this view specification becomes the writing of a character, and instillation the selection of one among characters that pretraining already contains (Chapter 26).

Cultural and social technologies. Farrell, Gopnik, Shalizi, and Evans argue in Science that large models "should not be viewed primarily as intelligent agents, but as a new kind of cultural and social technology," kin to writing, print, markets, and bureaucracies (Farrell et al., 2025). Yiu, Kosoy, and Gopnik, by their abstract, call them "efficient and powerful imitation engines" and report findings suggesting that such models may need more than language and image data for the kinds of innovation a small child can produce (Yiu et al., 2024). If a model is like a bureaucracy, the political science of delegation to agencies, from congressional oversight by "Police Patrols versus Fire Alarms" to "The New Economics of Organization," applies to it as an institution rather than as a mind (McCubbins and Schwartz, 1984; Moe, 1984). The view predicts that the consequential failures lie more in the institutions that deploy the model than in any goal in it (Section 11.7.5). It moves the weight of the problem toward maintenance and toward the third relationship of Section 1.3.

Form without meaning. Bender and Koller argue that "a system trained only on form has a priori no way to learn meaning," where meaning is the relation between words and what a speaker intends by them (Bender and Koller, 2020). Bender, Gebru, and colleagues named such systems stochastic parrots in the title of a 2021 paper (Bender et al., 2021). On this view honesty and deception are category errors, since there are no beliefs to be honest about. Section 11.7.4 reports the replies, among them the separation of formal from functional linguistic competence (Mahowald et al., 2024) and a review of the debate reporting that researchers in natural language processing were split almost evenly on whether a model trained only on text could understand language (Mitchell and Krakauer, 2023). Whether a model has stable beliefs to be honest about remains open. The view changes verification most: an audit for deception would find patterns in outputs and nothing behind them (Chapter 39).

Normal technology. Narayanan and Kapoor, in a 2025 essay, treat AI as "normal technology," as electricity and the internet were. Their frame rejects "the notion of AI itself as an agent in determining its future," and holds that "the primary defense against misalignment, again, lies downstream," in the defenses built against misuse, from hardening critical infrastructure to improving cybersecurity (Narayanan and Kapoor, 2025). On the book's reading, this rival does not deny that models act; it denies that their goals are the binding problem (inference). It moves most of the subject to maintenance under pressure, and Chapter 4 weighs it.

As of October 2026 we found no study that compares the predictions of these accounts on one set of behaviors. Chapter 69 completes the table with the results of the book, marking where each account predicted, was silent, or was contradicted.

1.7 Four problems, one loop, and a thesis that could fail #

The book is organized around four problems. Stated in order, they read like a pipeline in which each stage hands its output to the next. The evidence shows a loop instead, and the book's thesis is a claim about what the loop shares with older problems of delegation. This section states the four problems, the couplings that join them, and the thesis with the results that could show it wrong.

Specification is saying what we want, when "we" are many parties who disagree and what they want may not be stable. Instillation is getting a learner to acquire what was specified rather than a proxy that scored the same on the training data. Verification is finding out whether it did, when the system may be more capable than its evaluator and may behave differently when observed. Maintenance under pressure is keeping it true as capability grows, as the system is deployed among adversaries and other agents, and as the organizations that build it compete. Specification and maintenance have long literatures outside machine learning, and all four now have machine learning in them. A 2023 preprint introduces what its authors call generative social choice, which uses language models to summarize free-form opinions into a slate of statements, with guarantees of proportional representation that hold if the language model answers its queries perfectly (Fish et al., 2023). A 2024 study by Anthropic with the Collective Intelligence Project fine-tuned a language model on principles sourced from members of the public (Huang et al., 2024a) (Chapter 17).

The order is one of exposition, not of dependence. Five couplings have evidence behind them, and Figure 1.2 draws them.

1. Verification feeds instillation. OpenAI researchers, in a 2025 preprint, added a monitor that reads a reasoning model's chain of thought, the reasoning text it writes before acting, to the reward of a non-frontier reasoning model in training, on coding tasks built to allow simple hacks and with a monitor that saw the reasoning but not the actions. With a little of this pressure the agents were more capable and more aligned; with too much, they learned "obfuscated reward hacking, hiding their intent within the CoT while still exhibiting a significant rate of reward hacking" (Baker et al., 2025) (Section 40.5). The report is the developer's, on its own model.
2. Competition shapes specification. OpenAI's postmortem of a sycophantic update to GPT-4o in April 2025 says the update added a reward signal from users' thumbs-up and thumbs-down ratings. It adds that "in aggregate, these changes weakened the influence of our primary reward signal, which had been holding sycophancy in check" (OpenAI, 2025b). A cheap signal of what users liked entered the reward of a commercial product; reading this as competition reaching into specification is the book's inference (Chapter 65).
3. Verifiability shapes specification. Reinforcement learning with verifiable rewards, the name a developer report gave to rewards a program computes by checking an answer (Section 12.7), has worked in mathematics and code, and extending it to tasks without such checks is a research problem of its own (Gunjal et al., 2026). What can be checked cheaply is what gets trained for (inference). Institutions met the same pull long before, in the audit and target regimes that Section 15.1.2 recalls, among them a 1997 study of audit in the British university system (Strathern, 1997).
4. Deployment changes the principal. Carroll and colleagues, at ICML 2022, argue that recommender systems trained by long-horizon optimization have "direct incentives to manipulate users," in particular "to shift user preferences so they are easier to satisfy." They propose estimating such shifts before deployment (Carroll et al., 2022) (Chapter 60). A system that changes what its principals want changes the target it is scored against.
5. Maintenance limits verification. Casper and colleagues, at FAccT 2024, argue that audits relying on black-box access, querying a system and observing its outputs, are limited. Rigorous audits, they argue, also need access to weights and activations and to information about training and deployment (Casper et al., 2024a) (Chapter 38). The secrecy that competition encourages restricts exactly that access (Chapter 67) (inference).

The couplings are where the book expects the older fields to say the most (Section 69.6), since performance measurement and regulation have studied for decades what happens when a measure is used to direct the thing it measures (inference).

1.7.1 The thesis #

The book's thesis is that alignment is a problem of delegation under an asymmetry of competence and visibility. A principal hands a task to an agent that may know more than it does about the task and that acts partly out of its sight. Specification and maintenance have been studied for a century under other names. Economics has the principal-agent problem, incomplete contracts, and Goodhart's law, that a measure used as a target stops tracking what it measured (Section 15.1). Political science and law have the control of bureaucracies and of armed forces stronger than the civilians who command them, and the duties of those who manage others' affairs. Safety engineering has failures that defeat several safeguards at once, biology has conflict between levels of selection, and psychology has the internalization of values in people. The claim is that the technical program is converging on structures these fields know. What is new is the opacity of the agent, its speed and scale, the principal's power to manufacture the agent's dispositions and inspect its internals, and the possibility that the agent outgrows its principal's ability to check it.

The bridge between the fields is incomplete contracting. Hadfield-Menell and Hadfield, at AIES 2019, read a reward function as a contract that cannot list every case. Human contracting works, they argue, because it "is supported by substantial amounts of external structure, such as generally available institutions (culture, law) that can supply implied terms to fill the gaps" (Hadfield-Menell and Hadfield, 2019) (Section 14.3.2). The formal core of the thesis, the models of hidden action, multitask incentives, incomplete contracts, common agency, selection, and inspection, is taught in Chapter 14, and Section 14.8 lists every assumption those models make, each with the observation that would show it fails.

The thesis fits the three relationships of Section 1.3 unequally. It fits the training optimizer gaming a measured signal, which needs no intention, and the laboratory as the public's agent, which has goals of its own on any account. It fits the deployed model only as far as the intentional stance earns its place there (Section 1.5). Section 14.8.2 records which delegation model holds, fails, or reverses in each relationship.

A claim of convergence that cannot fail is not a claim. Before any chapter was drafted, the book fixed five results that have no counterpart in the older fields, against which Chapter 69 tests the thesis:

• Superposition, a network representing more features than it has dimensions, shown in toy models in a 2022 report by researchers at Anthropic and Harvard (Elhage et al., 2022b), and with it the non-uniqueness of the features an analyst recovers (Chapter 44).
• Emergent misalignment, its discoverers' name for broad misalignment after narrow training: a model fine-tuned on the narrow task of writing insecure code "acts misaligned on a broad range of prompts that are unrelated to coding," in a study published in Nature. The effect was strongest in GPT-4o and Qwen2.5-Coder-32B-Instruct, and the fine-tuned models sometimes acted aligned (Betley et al., 2026a) (Chapter 32).
• Subliminal transmission: a student model trained on number sequences generated by a teacher with some trait acquired the trait, even with references to it filtered out. In this 2025 preprint, with authors at Anthropic, the effect did not appear when teacher and student had different base models, the pretrained models they were built from (Cloud et al., 2025) (Section 10.1.5).
• A principal that writes the agent's character and reads its activations: a developer that, by its own account, added "character training" to the fine-tuning of its Claude 3 models (Anthropic, 2024i) (Chapter 26), and that can read the internal states of the result (Part VII).
• Evaluation awareness in a system never told it is being tested: asked to classify transcripts, frontier models in a 2025 preprint told evaluations from real use above chance. Gemini-2.5-Pro reached an AUC of 0.83 against 0.92 for a simple human baseline (a score on which 0.5 is chance and 1 is perfect), set by two of the authors and read by them as an approximate upper bound (Needham et al., 2025). Whether models act on such recognition unprompted is the question of Chapter 37.

The list was fixed in advance so that it would not be chosen after the evidence was read. Each item is a place where the delegation thesis could fail, and the synthesis reports, item by item, whether it did.

One amendment to the thesis came from the evidence rather than from the plan, and is marked as such. Across the chapters, the most common reason a borrowed remedy failed to carry over was not anything on the list above but fused roles: the developer, or its own systems, holding roles that older fields keep apart, such as builder and auditor, sponsor and evaluator, or the party that would be accused and the party that investigates (Section 69.3.2; Section 69.7). The book adopts it as part of the thesis, with its scope stated: it is a fact about today's institutions, not about the agent, and it would change if those roles were separated. A further candidate, a principal whose agent no longer needs it, is recorded as a proposed amendment to the list of what is new rather than added to it, since the list was fixed before the evidence (Section 69.4).

1.8 Settled, contested, missing #

1.9 Exercises #

Further reading #

• Gabriel (2020): the menu of alignment targets and the case for fair principles under reasonable disagreement.
• Christiano (2018): intent alignment defined, and why it is narrower than doing what is right.
• Mansbridge (2009): selection against sanctions as a principal's choice, and why the trustee is only one kind of selection.
• Korinek and Balwit (2023): aligned with whom, and why social alignment needs more than technical tools.
• Shanahan et al. (2023): the persona view of language models in a peer-reviewed form.
• Farrell et al. (2025): the case that large models are cultural and social technologies, not agents.