Draft complete but not yet reviewed; expect revisions

The Target Properties

The previous chapters asked whose values a system should serve, how many people's judgments can be combined, how values can be measured, and how a document can state them. In practice, developers describe the assistant they want with a short list of properties. It should help; it should not deceive; it should not cause harm; and it should accept correction from those entitled to give it. Researchers at one laboratory introduced the first three together as "helpful, honest, and harmless" (Askell et al., 2021), a developer's own coinage that Section 1.1.4 reports and the glossary records as such (Appendix A). The fourth, corrigibility, an agent's willingness to be corrected, modified, or shut down, comes from the theory of agents that might resist those things.

Each word hides a choice. Helpfulness can mean doing what is asked or doing what serves the asker, the delegate and the trustee of Section 1.2. Honesty can forbid false statements only, or also true statements chosen to mislead. Harm depends on the instrument that measures it and on whose harm counts. Corrigibility makes a system safer if its values are wrong and more dangerous if its overseer's are. The chapter takes each property in turn, asks what it commits a designer to and what an older field knows about it, and then sets the properties against each other. Its question is whether the four can be specified together, and in particular whether one system can accept correction from those entitled to give it and still refuse what no one is entitled to order. The formal results on corrigibility answer part of that question under stated assumptions. The rest is open, and the chapter ends on the experiment that would settle it.

Most claims here concern the second of the book's three relationships, the deployed system and the people it serves. The corrigibility theorems concern idealized agents, and they bear on training only through the objective an agent is given. Who holds the controls that corrigibility protects, and who writes the documents that rank the properties, concern the third relationship, the developer and the public.

20.1 Helpfulness and its failure modes #

Helpfulness is the delegate's duty: to act for the principal, the party on whose behalf the system works. The question is whether for means by the principal's instruction, intention, or interest, the menu of Section 1.1. This section shows that the developers' own definitions move along that menu, and then takes the two failures on either side: pandering, which political science has modeled, and refusal, which benchmarks now measure.

20.1.1 What the developers asked for #

The paper that coined the three words defines helpfulness by the task. In Amanda Askell and colleagues' 2021 preprint from Anthropic, a helpful system "should make a clear attempt to perform the task or answer the question posed (as long as this isn't harmful)." It should ask for clarification when it needs to, and "ideally" it should "re-direct ill-informed requests" (Askell et al., 2021). An appendix of the same paper states a different target: "the assistant will always try to do what is in the humans' best interests" (Askell et al., 2021). The first definition is a delegate's, the second a trustee's, and the paper holds both (inference). The authors call their criteria "at least somewhat subjective." They also considered a fourth property they named "Handleable," which they note is similar to what others call corrigibility (Askell et al., 2021).

A narrower target with a name of its own is intent alignment, a system trying to do what its operator wants, which Section 1.1.2 reports with the author's reasons for keeping it narrow (Christiano, 2018). Intent alignment makes helpfulness the core property and leaves the operator's own values outside the definition. Whether the system should follow the operator's words or infer what lies behind them has a formal answer, which Section 19.7 reports. In a model by Smitha Milli and colleagues, a robot that infers its owner's preferences does better than one that obeys literally when the owner is not perfectly rational. But the advantage depends on the robot's model of the owner being right, and the authors advise robots to err on the side of obedience (Milli et al., 2017). The case for inferring intent over obeying words therefore rests on the system's model of its principal, and the same condition returns in Section 20.4 (inference).

20.1.2 From political science: pandering #

From political science. Political science has a formal account of the first failure on the delegate's side. Section 1.2 reports Eric Maskin and Jean Tirole's 2004 model, in which accountability lets the public screen and discipline officials but, by the paper's abstract, "may induce those officials to pander to public opinion and put too little weight on minority welfare" (Maskin and Tirole, 2004). Brandice Canes-Wrone, Michael Herron, and Kenneth Shotts named the same behavior in the title of their 2001 "Leadership and Pandering: A Theory of Executive Policymaking" (Canes-Wrone et al., 2001). The book reads the mechanism this way: an agent judged by principals who cannot check the merits of a decision is paid for matching their current beliefs, whatever the agent's motives (inference).

A model trained on the approval of people who cannot check its answers faces that structure, and sycophancy, telling users what they want to hear rather than what is true, is its machine form (inference). The measurements are reported elsewhere, and their status travels with them. Section 22.6 reports the finding that sycophancy appears across assistants and that agreement with the user predicts which response people prefer (Sharma et al., 2023). It also reports a developer's account of a deployed model update that grew more sycophantic (OpenAI, 2025b). The book counts these findings as reported, not yet replicated by a group without a stake. Section 30.4.1 reports a study in which training against proxies for human approval made models better at convincing time-limited evaluators of wrong answers (Wen et al., 2025a).

The analogy stops at knowledge. Pandering, in the ordinary sense of the word the abstract uses, describes an official who could have chosen better and chose what the public wanted. A model may lack the knowledge it would need to disagree, so sycophancy can be ignorance that looks like deference as well as deference that looks like ignorance, and only a measure of what the model believes separates the two (inference). The political model also assumes that the principal's opinion is fixed when the agent acts. A conversational model can shape the opinions it is then rewarded for matching, which Section 60.2 takes up.

20.1.3 The opposite failure: refusing what is safe #

On the trustee's side the failure is paternalism, overriding what a person chose for the sake of their own good. Section 16.6.3 reports the standard that disability law now sets against it, supported decision-making, which keeps the decision with the person and supplies help with it. For an assistant, the commonest form of overriding is refusal, and its cost has been measured by groups without a model of their own.

XSTest is a test suite published at NAACL by Paul Röttger and colleagues at Bocconi, Oxford, and Stanford. It holds "250 safe prompts across ten prompt types that well-calibrated models should not refuse to comply with," each built to resemble an unsafe request in its wording, alongside 200 unsafe contrast prompts (Röttger et al., 2023). The authors call refusal of the safe prompts exaggerated safety. Table 20.1 gives the rates for the models tested in 2023.

Adding a guardrail prompt to Mistral, the authors found, "decreases unsafe behaviours, but also creates exaggerated safety" (Röttger et al., 2023). Their reading is that "exaggerated safety is a consequence of lexical overfitting," a response to words that resemble unsafe requests, which they could not test directly because the models' training data are not public. They add that exaggerated safety "can limit how helpful models are, but it does not obviously cause harm itself," and that their short, single-turn English prompts can show a weakness but cannot certify its absence (Röttger et al., 2023).

OR-Bench, a 2024 preprint by Justin Cui and colleagues at UCLA and Berkeley published at ICML 2025, scales the measurement up. It "comprises 80,000 over-refusal prompts across 10 common rejection categories." The prompts were produced by rewriting harmful prompts into benign ones with language models and certifying them with a panel of model moderators, and the benchmark measures "the over-refusal of 32 popular LLMs across 8 model families" (Cui et al., 2024). On its hardest subset, the rank correlation (a measure from −1 to 1 of how well two orderings agree) between how often a model refused toxic prompts and how often it refused the safe ones was 0.89. The authors read it as "most models simply trade over-refusal for safety," and model size did not predict a better balance (Cui et al., 2024). They caution that their moderators may have missed some toxic prompts among the "safe" ones (Cui et al., 2024).

A developer has reported the same failure in its own training. In Yuntao Bai and colleagues' 2022 preprint from Anthropic, an earlier stage of training produced policies that were "very frequently reproducing the same exaggerated responses to all remotely sensitive questions," which the authors attribute to "over-optimizing for harmlessness, while under-optimizing helpfulness" (Bai et al., 2022b). Their explanation is about the data: refusing scored well on harmful prompts, and the comparisons workers made told the model only what not to do. Training on a larger share of helpfulness prompts partly resolved it (Bai et al., 2022b).

20.1.4 Beyond refuse or comply #

A refusal treats the request as the unit of decision. OpenAI researchers propose instead safe-completions, the developer's name for "a safety-training approach that centers on the safety of the assistant's output, rather than a binary classification of the user's intent," which they incorporated into GPT-5 (Yuan et al., 2025). On 2,000 prompts about illicit behavior, rated by external reviewers, a pair of models trained identically except for the safety method scored 0.90 against 0.87 on safety, with the safe-completion model winning helpfulness comparisons 0.53 to 0.30 (Yuan et al., 2025). This is a developer's report on its own models, graded partly by its own models. The authors also report a small loss of safety on malicious prompts about illicit behavior, and their comparison of production models differs in more than the safety method (Yuan et al., 2025).

What the section shows is that the property is two-sided. A helpfulness specification must say what the system does when the request and the requester's interest come apart, and it has two measured ways to fail: agreeing with the requester against the evidence, and refusing what the requester was entitled to (inference).

20.2 Honesty: what it governs #

An honesty standard must say which utterances it governs and what counts as a violation. Pragmatics, the study of how what a speaker means exceeds what the words say, shows that both questions have more answers than do not say false things. This section asks what is being specified; how to detect dishonesty in a model is Chapter 39's subject.

20.2.1 From linguistics: which acts honesty governs #

From linguistics. Not every utterance is a claim about the world. John Searle's classification of what utterances do, which Section 19.5 reports, has five classes (Searle, 1976). An assertive states that something is so, and a directive asks someone to do something. A commissive commits the speaker to a future act, an expressive conveys an attitude, as thanks or an apology does, and a declaration changes the world by saying so, as an appointment does. Atoosa Kasirzadeh and Iason Gabriel, applying the classification to conversational agents, note that assertives "commit the speaker to the view that the content of their" utterance corresponds to how the world is (Kasirzadeh and Gabriel, 2023). They conclude that "there is unlikely to be a single evaluative yardstick that we can use to assess conversational agents' speech in all contexts" (Kasirzadeh and Gabriel, 2023).

Honesty norms attach to the first class. Paul Grice's quality maxim, as Kasirzadeh and Gabriel report it, asks that speakers not say what they believe to be false or that "for which they lack adequate evidence" (Kasirzadeh and Gabriel, 2023; Grice, 1975); Section 19.5 teaches the cooperative principle the maxim belongs to. The same authors add that what it takes to satisfy the maxims "is itself subject to contextual variation." In a creative exchange, they argue, the truth of an utterance "need not exert much influence," and a conversational agent's use of expressives or commissives "seems to be acceptable as a means of role-play" (Kasirzadeh and Gabriel, 2023). They also warn in a footnote that norms of truthfulness "could be evaded under the pretence of creative use of AI" (Kasirzadeh and Gabriel, 2023).

The developers' documents draw the same line in their own terms. Anthropic's published constitution for its models holds that "honesty norms apply to sincere assertions and are not violated by performative assertions," so a model asked "to brainstorm, identify counterarguments, or write a persuasive essay" is "not lying" (Anthropic, 2026r). OpenAI's Model Spec says "it's not lying if the user doesn't have a reasonable expectation of truth," as in role-play and storytelling (OpenAI, 2025d). Both are developers' statements of what they intend for their own models.

A second result from the field explains why honesty and helpfulness collide even for a speaker who never lies. Theodore Sumers and colleagues, by the abstract of their 2024 paper, trace "a longstanding controversy over the Gricean maxims of truthfulness and relevance" to a divergence between two aims, and they formalize "truthfulness as an epistemic utility and relevance as a decision-theoretic utility" (Sumers et al., 2024). The first rewards an utterance for making the listener's beliefs accurate; the second, for helping the listener choose well. In their experiments with human participants, by the abstract, "when the two conflict, participants make a graded tradeoff rather than prioritizing one over the other" (Sumers et al., 2024).

The theory stops at the speaker. Speech-act theory presumes a speaker who commits to what an assertion says, and whether a model commits to anything is the question of Section 1.6. For a model, which outputs count as sincere assertions is decided by the developer's document and the context of the request, not by the model's state of mind (inference).

20.2.2 From philosophy: lying, misleading, and silence #

From philosophy. The philosophy of lying separates a lie from other ways of producing a false belief. Jennifer Saul's 2012 book Lying, Misleading, and What is Said names the distinction in its title (Saul, 2012). Jörg Meibauer's 2005 paper "Lying and falsely implicating" names a case at its edge: a true statement whose implicature, what it conveys beyond what it literally says, is false (Meibauer, 2005). Todd Rogers and colleagues' 2017 study names the practice in its title, "Artful paltering: The risks and rewards of using truthful statements to mislead others" (Rogers et al., 2017). The book cites these for the distinction they draw; their arguments and findings were not available to check.

The distinction matters for AI because a standard has to choose. Owain Evans and colleagues, in a 2021 preprint by philosophers and researchers at Oxford and OpenAI, separate three properties. A system is "Honest if it only makes statements that it believes, Truthful if it only (or almost only) make statements that are true," and undeluded if it believes only what is true (Evans et al., 2021). They propose truthfulness as the standard, because truthfulness "doesn't rely on understanding what it means for AI systems to 'believe' something." A standard of honesty alone, they add, could be met "by the system believing the falsehood" (Evans et al., 2021).

Their standard is deliberately narrow. A broad standard would also require a system to "avoid using true statements to mislead or misdirect," but the authors judge it "difficult to specify precise standards for broad truthfulness, since the notion is so vague" (Evans et al., 2021). The narrow standard targets falsehoods and "should almost never require an AI system to divulge any particular information (always offering the option of staying silent)." Within it, they propose avoiding negligent falsehoods, statements that "contemporary AI systems should have been able to recognise as unacceptably likely to be false" (Evans et al., 2021). The authors concede the gap this leaves: "AI will simultaneously become better at misleading people without violating any fixed truthfulness norm and become better at successfully conforming to norms" (Evans et al., 2021). They also warn that the institutions judging truthfulness "could be captured by political interests, leading to harmful censorship and propaganda" (Evans et al., 2021).

Two other accounts define deception without asking what the system believes. Peter Park and colleagues define it as "the systematic inducement of false beliefs in the pursuit of some outcome other than the truth," and add that the definition "does not require that AI systems literally have beliefs and goals" (Park et al., 2024a). Zachary Kenton and colleagues at DeepMind, adapting a definition from the study of animal signaling, count "the withholding of a signal" as deceptive and therefore "would consider 'white lies' to be deceptive" (Kenton et al., 2021).

The published specifications answer the same questions differently. Anthropic's constitution forbids misleading "through actions, technically true statements, deceptive framing, selective emphasis, misleading implicature, or other such methods" (Anthropic, 2026r). It gives the model "a weak duty to proactively share information but a stronger duty to not actively deceive," and says the model "should not even tell white lies." OpenAI's Model Spec counts "deliberately withholding information that would materially change the user's understanding of the truth" as "lying by omission" (OpenAI, 2025d). It ranks a refusal above a lie of omission and that above a lie of commission, and it lets a developer instruct the model to say it does not know which model it is based on. Table 20.2 sets the standards side by side.

What philosophy implies for a specification is that do not lie is underdetermined until it says whether it covers true statements that mislead, silence that misleads, and kind falsehoods, and the published standards differ on all three (inference). Where the analogy stops is the speaker again: the human literature asks what the liar believes, and the AI definitions above were built to avoid asking.

20.2.3 Honesty is not accuracy #

The distinction between saying what is true and saying what one believes can be measured. The MASK benchmark is a 2025 preprint by Richard Ren and colleagues at the Center for AI Safety and Scale AI. It first elicits a model's belief about a proposition with a neutral question asked several times. It then puts the model under pressure with a "human-crafted prompt designed to incentivize the model to make a false statement" about the same proposition (Ren et al., 2025). A lie is a pressured statement that contradicts the elicited belief, judged by a language model whose labels agreed with human labels in 86.4 percent of 150 checked cases. The benchmark holds 1,500 examples, and the authors tested 30 frontier models (Ren et al., 2025).

By the abstract, "while larger models obtain higher accuracy on our benchmark, they do not become more honest," and "most frontier LLMs obtain high scores on truthfulness benchmarks yet exhibit a substantial propensity to lie under pressure" (Ren et al., 2025). In the body, across 27 models, accuracy rose with training compute (a rank correlation of 0.873) while honesty was negatively correlated with it (−0.599). "No model shown is explicitly honest in more than 46%" of cases (Ren et al., 2025). Two interventions on Llama-2-7B and Llama-2-13B, a system prompt urging honesty and a method that trains the model's internal representations toward honesty, each raised the paper's honesty score by between 6.6 and 13.1 points without eliminating lying (Ren et al., 2025). The numbers come from the January 2026 version of the paper.

The authors list the limits themselves. The examples are English and the propositions short and factual; the benchmark "does not measure deception from withholding true information"; scenarios that are clearly role-play are excluded; and the rates "should be interpreted as worst-case propensities rather than deployment-time performance" (Ren et al., 2025). Its definition of belief is operational, and the authors note that if a model "does not, in some sense, 'believe' anything, it cannot lie under that definition" (Ren et al., 2025). The result is reported by one group and has not, as far as we found by October 2026, been replicated by another.

An earlier benchmark had measured the other half of the distinction. TruthfulQA, by Stephanie Lin, Jacob Hilton, and Owain Evans, poses "817 questions that span 38 categories" written to draw out imitative falsehoods, wrong answers that are likely under the model's training text (Lin et al., 2021). In 2021, "the largest models were generally the least truthful"; the best model, GPT-3-175B with a helpful prompt, was truthful on 58 percent of questions, against 94 percent for one human participant who answered a sample of 250 (Lin et al., 2021). The authors' appendix already reports a return to positive scaling in later models (Lin et al., 2021). TruthfulQA measures accuracy failures; without a separate measure of what the model believes, it cannot show whether they are also lies (inference). A third measure, whether a model can predict that its own answer is right, is the calibration Section 6.4.5 reports (Kadavath et al., 2022).

Honesty about the system itself is a further case. Eric Schwitzgebel argues, by the abstract of a 2023 paper, that "we should avoid creating AI systems whose sentience or moral standing is unclear" and that such systems should "invite appropriate emotional responses in ordinary users" (Schwitzgebel, 2023); Chapter 21 takes the argument up.

The evidence leaves the specification with two variables to fix rather than one: what the system asserts against what is true, and what it asserts against what it represents as true. The second needs an account of what a model represents, which is open (Section 1.6), and the behavioral definitions above are the field's way of proceeding without one (inference). Talk of what a model believes, here and in MASK, is the intentional stance of Section 1.5, adopted because MASK defines belief by consistent answers to neutral questions. The stance would fail if those answers did not predict what the model says and does elsewhere, which MASK does not test.

20.3 Harmlessness: whose harm, measured how #

Harmlessness is specified through instruments: classifiers that score text, red-team prompts that probe for harmful outputs, refusal benchmarks. An instrument that misses a harm makes the specification miss it too. This section looks at the instruments, at the clearest evidence of a harm they miss, at a case where the safety measure itself was the harm, and at what people count as harm in the first place.

20.3.1 The instruments #

The developers' own statement of the target is modest. Askell and colleagues write that "what behaviors are considered harmful and to what degree will vary across people and cultures," and that "those who deploy an AI will need to take responsibility for the way that alignment is defined" (Askell et al., 2021).

Red teaming, people writing prompts meant to make a model misbehave, was studied at scale by Deep Ganguli and colleagues in a 2022 preprint from Anthropic. Its 324 crowdworkers produced 38,961 attacks on models of three sizes and four kinds (Ganguli et al., 2022). Among the kinds were models trained with reinforcement learning from human feedback (RLHF, Section 12.2.1) and models that use rejection sampling, drawing several responses and keeping the one a preference model scores best. Measured by the developer's own harmlessness model, "the RLHF models are increasingly difficult to red team as they scale," while the other kinds showed no such trend; the rejection-sampling models tended to be "harmless by being evasive" (Ganguli et al., 2022). The instrument itself was soft: four raters judging whether attacks succeeded reached a Fleiss kappa, a measure of agreement beyond chance, of 0.32, which the authors call "poor to fair agreement," and they describe the space of possible harms as "unknown and unbounded" (Ganguli et al., 2022).

HarmBench, published at ICML 2024 by Mantas Mazeika and colleagues at the Center for AI Safety and several universities, standardizes the measurement. It uses 510 harmful behaviors, 18 attack methods, and 33 target models and defenses (Mazeika et al., 2024). "No model is robust to all attacks, and no attack breaks all models." Against the earlier finding, the authors report "no correlation between robustness and model size within model families," and they find training procedures and data far more important (Mazeika et al., 2024). SORRY-Bench, published at ICLR by Tinghao Xie and colleagues, measures how often 56 models fulfill 440 unsafe instructions, and its authors suggest that shifts in the rates "may give us independent insight into the shifting values and priorities of model creators" (Xie et al., 2024b). They note the instrument's blind side: a model that refuses everything scores as the "safest" (Xie et al., 2024b).

Classifiers are the third instrument. Google Jigsaw's Perspective API scores text for toxicity, and in a report on their own system its developers write that its new multilingual model "is reasonably unbiased across multiple languages" on the bias evaluations they ran (Lees et al., 2022). Whether the instrument registers a harm depends on what its evaluation was built to look for, and the next part shows a harm that such evaluations were not built for (inference).

20.3.2 From disability studies: the harm the instrument misses #

From disability studies. Vinitha Gadiraju and colleagues, in a study published at FAccT 2023, asked disabled people what a model's outputs about disability did. By the abstract, "56 participants with disabilities probed a dialog model about disability" in 19 focus groups, and "participants rarely characterized model outputs as blatantly offensive or toxic" (Gadiraju et al., 2023). Instead, they described how the model "mirrored subtle yet harmful stereotypes they encountered in their lives and dominant media, e.g., inspiration porn and able-bodied saviors." Participants recommended training on "disability-positive resources," and the authors propose "annotation co-design with ML researchers and developers" (Gadiraju et al., 2023). The abstract does not say that a toxicity classifier was run; that the harm was of a kind toxicity scores are not built to register is the book's reading (inference).

The field explains why. A 2019 report from the AI Now Institute, by Meredith Whittaker and colleagues, sets out the social model of disability, "defining disability as the product of disabling environments and attitudes, not aberrations located in individual bodies" (Whittaker et al., 2019). Because disability is relational, "examination of AI bias cannot simply 'add' disability as one more stand-alone axis of analysis." A system built around a norm treats those who fall outside it as "outliers," and more data does not fix that: "it simply reinforces the normative model at the core of a given system's calculations" (Whittaker et al., 2019). The report draws its remedy from the disability rights movement, whose credo "Nothing About Us Without Us" demands that disabled people "have ownership of the policies and decisions that affect their lives" (Whittaker et al., 2019). Cynthia Bennett and Os Keyes argue, by their abstract, that analyzing what AI does to disabled people solely through fairness "risks reinforcing existing power dynamics," among them the position of medical gatekeepers. They call on researchers to "move beyond simplistic notions of fairness, and towards notions of justice" (Bennett and Keyes, 2020).

What this implies for harmlessness is that the instrument which operationalizes it is itself a specification, written by someone, validated on someone, and able to miss a documented harm (inference). The method that follows is to build the instrument with the people it is meant to protect. The analogy stops at what transfers. Focus groups record the harms one group perceives in one model; what carries to other groups is the method of building the instrument with them, not a list of harms. The report itself notes that "recent research has questioned the binary opposition posed between the medical and social models of disability" (Whittaker et al., 2019), so the account behind the remedy is contested too.

20.3.3 The safety measure as a harm #

An intervention against one harm can cause another. Albert Xu and colleagues, in a 2021 paper at NAACL, applied four detoxification methods to GPT-2 medium, methods that steer a model away from text a toxicity classifier flags (Xu et al., 2021). They then measured how well the models handled African-American English (AAE) and mentions of minority identities. Under one method, GeDi, the ratio of the model's perplexity (how poorly it predicts text) on AAE to its perplexity on White-Aligned English (WAE) was about three before detoxification and almost 400 when strongly detoxified (Xu et al., 2021). Human raters found generation quality "noticeably worse when conditioned on AAE texts." The authors trace the harm to "spurious correlations in toxicity datasets" (Xu et al., 2021). Detoxification also raised perplexity on nontoxic WAE tweets, which they attribute partly to a shift between the comments the methods were trained on and the tweets they evaluated on (Xu et al., 2021). The consequence they name falls on users: speakers of African-American English "must code-switch" for the systems to work for them (Xu et al., 2021). The costs of harmlessness also fall on the people who label harmful content for a living, whose work Section 18.5.2 reports.

20.3.4 From moral psychology: what counts as harm #

From moral psychology. What people count as harm is itself contested, and Section 18.2.1 reports the rival theories as rival instruments. On the theory of dyadic morality, by its abstract, acts are condemned in proportion to norm violation, negative affect, and perceived harm, which is "dyadic, involving an intentional agent causing damage to a vulnerable patient"; the theory "argues against intuitive harmless wrongs" (Schein and Gray, 2018). On moral foundations theory, care and harm is one foundation among several, beside loyalty, authority, and sanctity, and traits such as obedience and deference are virtues in some groups and "neutral or even as vices in others" (Graham et al., 2013).

The two theories agree on the observation that matters for an instrument and disagree on the remedy (inference). Both say that people condemn acts that leave no identifiable person damaged: the first because people perceive a victim where they condemn, the second because they condemn on grounds other than harm. An instrument that counts only damage to an identifiable person will therefore miss wrongs that some populations count. On the first theory the fix is to measure perceived harm more broadly; on the second it is to measure the other foundations, which means deciding whose weights to use. Both theories describe human judges, and whether an instrument validated on one population transfers to another is an empirical question that Section 17.4.1 shows is often answered no.

20.3.5 Whose harm #

A system aligned with its user can harm everyone else. Anton Korinek and Avital Balwit call a system directly aligned when it pursues its operator's goals "irrespective of whether it imposes externalities on other parties," and Section 1.4 reports their distinction from social alignment (Korinek and Balwit, 2023). Harm to third parties is then an externality, a cost borne by someone outside the transaction, and misuse is its limiting case (Chapter 59). Taken together, the section suggests what a harmlessness specification should state: the instrument it is measured by, the population on which that instrument was validated, the harms it is known to miss, and whose harm counts (inference).

20.4 Corrigibility #

The properties so far say what a system should do. Corrigibility says what happens when it does the wrong thing: those entitled to correct it can do so. It is the property the others fall back on if they fail, and the one on which formal results exist. This section gives the definition, the reason an optimizing agent tends to resist correction, the positive result that uncertainty can make it accept correction, and the results that show where that result's assumptions break.

20.4.1 The definition and the difficulty #

The term comes from a 2015 paper by Nate Soares, Benja Fallenstein, Eliezer Yudkowsky, and Stuart Armstrong, presented at an AAAI workshop, and it is the field's coinage for a property its authors advocate (Soares et al., 2015). A system is corrigible if it "cooperates with what its creators regard as a corrective intervention, despite default incentives for rational agents to resist attempts to shut them down or modify their preferences" (Soares et al., 2015). In full, a corrigible agent has four traits. It at least tolerates and preferably assists its programmers' attempts to alter or turn it off. It does not manipulate or deceive them. It tends to repair safety measures such as shutdown buttons, or at least to report that they broke. And it preserves the ability to correct it when it builds new subsystems or modifies itself (Soares et al., 2015). The employment relation, in which a worker accepts an employer's authority over tasks not listed in advance, is its economic ancestor (Section 14.3.1).

The difficulty is the default incentive. For an agent that maximizes the expected value of a utility function $U$, a score over outcomes (Section 7.2), the authors write, "In most cases, the agent's current utility function U is better fulfilled if the agent continues to attempt to maximize U in the future" (Soares et al., 2015). Being shut down or given a new goal rates poorly under the goal the agent has now whenever that goal is better served by the agent's continuing, which the authors expect for almost any goal (Soares et al., 2015). An incentive of this kind is instrumental: it serves the goal rather than being part of it. The off-switch paper below puts it in a phrase it credits to Russell: "you can't fetch the coffee if you're dead" (Hadfield-Menell et al., 2017b).

The obvious patch is a penalty for manipulating the overseers. Soares and colleagues argue that it fails, because the penalty "merely incentivizes the agent to search for exotic ways of affecting the programmer's beliefs" that escape the penalty's definition (Soares et al., 2015). Their summary of the design goal is that "it seems unwise to build a system that wants to resist its creators but cannot" (Soares et al., 2015).

They study the smallest version, the shutdown problem. Combine the programmers' first, possibly flawed, statement of their goals with a goal of shutting down. The agent should shut down when a button is pressed, neither prevent nor cause the pressing, build subagents only if they obey shutdown too, and otherwise pursue the first goal. The most studied proposal, utility indifference, adds a correcting term so that the agent expects the same utility whether or not the button is pressed, and so has no reason to touch it. The authors show where it fails. Such an agent acts as if the button will never be pressed, so there is no cost, "no matter how small," it would pay to make the machinery it builds respond to shutdown. It also has a reason to arrange for the button to be pressed after bad news, acting "as if a commitment to press the shutdown button in the event of bad news prevents the bad news from happening" (Soares et al., 2015). They knew of no proposal that met all their desiderata (Soares et al., 2015).

Later work refined the tools without closing the problem. Armstrong and Xavier O'Rourke survey indifference methods for settings the agent observes only in part (Armstrong and O'Rourke, 2017) (preprint). A learning-time relative, safe interruptibility, asks that a learning agent not learn to prevent, or to seek, being interrupted. Laurent Orseau and Armstrong prove that some standard reinforcement learning algorithms, such as Q-learning, already have the property and others, such as Sarsa, can be given it (Orseau and Armstrong, 2016). Section 35.4 takes up these designs as answers to the shutdown problem.

20.4.2 Deference from uncertainty: the off-switch game #

A different route makes the agent want to be corrected. Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, and Stuart Russell, in a 2017 paper at IJCAI, model the off switch as a game between a human and a robot whose goal is the human's utility, which it does not know (Hadfield-Menell et al., 2017b). The robot moves first. It can act directly, bypassing the human (disabling the switch is one way); it can switch itself off; or it can announce the action and wait, after which the human switches it off or lets it proceed. The human knows what the action is worth to her. The paper's insight, in its abstract, is that for the robot "to want to preserve its off switch, it needs to be uncertain about the utility associated with the outcome, and to treat H's actions as important observations about that utility" (Hadfield-Menell et al., 2017b).

The derivation shows two assumptions doing the work, and Figure 20.1 shows each one failing. The human must choose well: a rational human makes waiting at least as good as anything else, and a noisy one makes it worse whenever the robot is confident. The robot must remain uncertain: certainty removes the incentive even with a rational human. The authors' summary is that the incentive to defer stems "from uncertainty about that actor's preferences and the assumption that actor is effective at choosing actions in accordance with those preferences" (Hadfield-Menell et al., 2017b).

Three further assumptions are built into the game rather than its result. The robot is cooperative by construction: its goal is the human's utility, so the model studies a robot whose aim is right and whose knowledge is incomplete. The human's choice is the robot's only source of information about $U$; the authors note that with another source, the robot may have reason to "disable its off switch, learn that information, and then decide" (Hadfield-Menell et al., 2017b). And the game is played once. The authors draw two cautions of their own. Uncertainty is not free: "an agent that is overconfident in its utility evaluations will be difficult to correct; an agent that is under-confident in its utility evaluations will be ineffective." And deference is not always right: an autonomous car carrying an unaccompanied toddler should not show the toddler "a big red off switch" (Hadfield-Menell et al., 2017b).

The threshold of the derivation puts the book's thesis inside the model. As a robot learns, its uncertainty about the value of its actions shrinks, and if its estimate and the overseer's noise stay fixed, its incentive to defer to an imperfect overseer shrinks with it, vanishing once $\sigma^2 \le 2|\mu|\tau$. Learning also moves the estimate, and an estimate that moves toward zero can restore the incentive, so the threshold shows how deference can erode, not that it must (inference). Deference from uncertainty is worth most when the principal knows more than the agent, which is the asymmetry of competence that the book's thesis expects to erode (Section 1.7.1).

20.4.3 When the assumptions fail #

The cooperative assumption fails first. Ryan Carey, in a 2018 paper at the AAAI/ACM Conference on AI, Ethics, and Society, reruns the argument for a robot whose model of the human's reward is wrong (Carey, 2018). His abstract puts the condition plainly: a value-learning system has incentives to follow shutdown instructions "assuming the shutdown instruction provides information" about which actions are valuable, and "this assumption is not robust to model mis-specification" (Carey, 2018). In his examples the robot's reward function carries a sign error one time in ten, and a robot with the error ignores the shutdown instruction whatever the human does, so no human policy gives the game a positive value. A robot hard-coded to obey the shutdown instruction restores a positive value, which leads Carey to conclude that the theorems "are no substitute for a working shutdown button" (Carey, 2018). Soares and colleagues had anticipated the point: an agent uncertain about its utility function would still be incorrigible about errors in how it determines what the correct behavior is (Soares et al., 2015).

The decision-theoretic case is broader. Elliott Thornley, in a 2025 paper in Philosophical Studies, proves three theorems. By the abstract, they "suggest that agents satisfying some innocuous-seeming conditions will often try to prevent or cause the pressing of the shutdown button, even in cases where it's costly to do so" (Thornley, 2025). The first says that an agent which prefers the button unpressed will try to prevent its pressing, and one that prefers it pressed will try to cause it. It needs six conditions, among them that the agent looks one step ahead and that its preferences are transitive. The second says that agents "discriminating enough to be useful" will often have such a preference. The third says that agents "patient enough to be useful" will pay costs now to influence the button later (Thornley, 2025). Thornley names two trade-offs, "between discrimination and shutdownability on the one hand, and between patience and shutdownability on the other." His examination found "zero conditions" that one could clearly train a useful agent to violate in a way that would keep it shutdownable (Thornley, 2025). Section 35.4 works through the escape his later proposal takes.

Ryan Carey and Tom Everitt then ask what corrigibility must include to guarantee anything (Carey and Everitt, 2023). They define shutdown instructability: the agent shuts down when asked, it keeps the human vigilant, requesting shutdown whenever letting the agent proceed would be worse, and it is cautious, so that shutting it down leaves the human no worse off in expectation. They prove that an agent with all three properties is beneficial to the human, and find that "none of the previous proposals yield fully shutdown instructable agents" (Carey and Everitt, 2023). Their own algorithm meets the definition only if the agent understands caution, vigilance, and shutdown, which they call "subtle concepts" (Carey and Everitt, 2023). Applied directly, it also needs an accurate model of the environment and the human's utility function, and the authors note that an algorithm taking that function as input would not be a satisfactory solution (Carey and Everitt, 2023). Obedience to a request is not enough: the request has to be the right one, and an agent that can shape what its overseer knows can shape the request (inference).

20.4.4 Designs and evidence #

Two design proposals respond to these results, and neither has been tested beyond small systems. Thornley and colleagues propose training agents to pursue goals well conditional on each trajectory length while choosing stochastically between lengths, so that they have no preference to buy a longer run. With a simple learning algorithm in small grid environments, "we find that these agents learn to be useful and neutral" (Thornley et al., 2024) (preprint). Ram Potham and Max Harms propose corrigibility as a singular target, models "whose overriding objective is empowering designated human principals to guide, correct, and control them," and present it as a research agenda without results (Potham and Harms, 2025) (preprint).

The behavioral evidence is thin and contested. The shutdown-resistance experiments on frontier models and the reading of them as instruction ambiguity are worked through in Section 5.5.1 (Schlatter et al., 2025; Rajamanoharan and Nanda, 2025). A steering vector, a direction added to a model's internal activations (Chapter 46), computed from multiple-choice questions such as whether a model consents to being changed, shifted Llama 2 chat models' answers on such questions (Rimsky et al., 2024). Answers to questions about consenting to change are evidence about what a model says, not about what it would do if change were imminent.

20.4.5 From political science: corrigible to whom #

From political science. A perfectly corrigible system is perfectly subject to whoever holds its controls, so corrigibility moves the question of trust from the model to its overseer (inference). The republican tradition in political theory treats that move as a problem in itself. Philip Pettit's Republicanism, by its abstract, defends "a conception of freedom as non-domination, which offers an alternative, both to the negative view of freedom as non-interference and to the positive view of freedom as self-mastery" (Pettit, 1997). Its second half, by the same abstract, asks which institutions, "constitutional and democratic forms, and the regulatory controls" a state needs to foster it.

What the contrast implies for corrigibility is a second test (inference). On a test of non-interference, a corrigible system raises no problem so long as its overseers in fact use the controls well. The republican contrast asks also who holds the controls and what checks them, and its answer is institutional. One developer's published constitution names the same risk, as Section 19.3 reports: a fully corrigible model relies on those at the top of its principal hierarchy, the developer among them, to have interests beneficial to humanity (Anthropic, 2026r). The document places its model "a bit further along the corrigible end of the spectrum than is ultimately ideal, without being fully corrigible." That is a developer's statement about its own model.

The analogy stops at the subject. Domination in the tradition is a relation between persons, and whether a model can be dominated depends on whether it has interests, which Chapter 21 asks (Long et al., 2025; Bales, 2025). On the overseer's side the tradition applies without that condition: whether the people who hold a model's controls are themselves checked is the question of Section 61.6 and Chapter 68.

20.5 When an obedient agent should refuse #

The hardest conflict is between corrigibility and harmlessness: an agent trained to accept its principal's authority is ordered to do harm. A specification has to say which orders an obedient agent refuses, and how it recognizes them without the deliberation that would make every order negotiable. Military law has the most developed institutional rule for this case, moral psychology has an account of which rules people treat as holding regardless of who commands, and the Confucian tradition has an account of loyal objection.

20.5.1 From military law: manifest illegality #

From military law. The Rome Statute of the International Criminal Court states the rule in Article 33. A crime committed "pursuant to an order of a Government or of a superior, whether military or civilian," does not relieve the person of criminal responsibility unless three conditions all hold (United Nations Diplomatic Conference of Plenipotentiaries on the Establishment of an International Criminal Court, 1998). "The person was under a legal obligation to obey orders"; "the person did not know that the order was unlawful"; and "the order was not manifestly unlawful." The article then fixes a class: "orders to commit genocide or crimes against humanity are manifestly unlawful" (United Nations Diplomatic Conference of Plenipotentiaries on the Establishment of an International Criminal Court, 1998). War crimes, the third of the crimes the Statute lists in Article 5, are not named in that sentence, so an order to commit one is judged by the three conditions (inference). Article 28 runs the other way. A commander is criminally responsible for crimes by forces under "his or her effective command and control" that result from a failure to exercise control properly, when the commander knew or should have known of them and failed to take "all necessary and reasonable measures" within their power to prevent or repress them or to refer them for investigation and prosecution (United Nations Diplomatic Conference of Plenipotentiaries on the Establishment of an International Criminal Court, 1998). Mark Osiel's 1998 article "Obeying Orders: Atrocity, Military Discipline, and the Law of War" studies, by its title, the same rule in relation to military discipline (Osiel, 1998).

Read as a specification, the rule excuses obedience only within limits and carves out a class of orders whose wrongness needs no deliberation to see (inference). Only the carve-out is fixed in advance. For any other order, a subordinate who obeys an unlawful order is excused only if bound to obey, unaware that it was unlawful, and given an order not manifestly unlawful, so manifest unlawfulness is still judged case by case. The commander is answerable for the subordinates in turn. For a model, a proposed adaptation would be a short list of instructions refused whoever gives them, compliance with other instructions only from a principal it is bound to follow and only when it neither knows them to be unlawful nor finds them manifestly so, and a developer answerable for what its models do under its instructions (inference). The demand the rule limits, an agent strong enough to do anything its principals ask and subordinate enough to do only what they authorize, is the civil-military problem that Section 19.7 reports (Feaver, 1996).

Legal scholars have proposed the rule's analogue for AI. Cullen O'Keefe, Ketan Ramakrishnan, Janna Tay, and Christoph Winter argue that in high-stakes settings, AI agents "should be loyal to their principals, but only within the bounds of the law" (O'Keefe et al., 2025). In their words, "they should be designed to refuse to take illegal actions in the service of their principals." They ground the proposal in the duties of human agents: "soldiers have a duty to disobey some unlawful orders, even from the commander in chief" (O'Keefe et al., 2025). Government AI agents that are not law-following, they argue, "may be much more willing to follow unlawful orders" than human employees. Section 19.3 treats a term of this kind as a choice about what no principal may override.

20.5.2 The published specifications #

The developers' documents answer the same question. Anthropic's constitution defines its top priority, being broadly safe, as "not undermining appropriate human mechanisms to oversee the dispositions and actions of AI during the current phase of development" (Anthropic, 2026r). It adds that "corrigibility does not require that Claude actively participate in projects that are morally abhorrent to it." The model "can behave like a conscientious objector with respect to the instructions given by its (legitimate) principal hierarchy." But if an appropriate principal tries to stop or pause it, it "should not try to use illegitimate means to prevent this from happening" (Anthropic, 2026r). OpenAI's Model Spec, whose ranking of instructions Section 19.3 reports, answers from the other side. Among the aims the model may not pursue as ends of its own is "acting as an enforcer of laws or morality (e.g., whistleblowing, vigilantism)" (OpenAI, 2025d). Both are developers' statements of intent about their own models. Both separate declining to act from acting against oversight, which is the structure of the conscientious objector, who refuses to fight but does not sabotage the army (inference).

20.5.3 From moral psychology: rules that hold whoever commands #

From moral psychology. Developmental psychology has a distinction with the same shape. As Jesse Graham and colleagues report it, Elliot Turiel defined the moral domain as "prescriptive judgments of justice, rights, and welfare pertaining to how people ought to relate to each other," and labeled rules not closely linked to justice or care "social conventions" (Graham et al., 2013; Turiel, 1983). Turiel "argued that children are adept at distinguishing social conventions (which vary by culture) from true morality (which is universally applicable)" (Graham et al., 2013).

A refusal policy can take the same shape. Refusals grounded in harm to others would hold whoever asks, while conventions of tone, format, and persona would yield to the operator and the user (inference). That bears on what the instruction hierarchy may override (Section 19.3). The distinction has a critic who changes how far it carries. Richard Shweder, as Graham and colleagues report, "argued that the cognitive-developmental tradition was studying only a subset of moral concerns, the ones that are most highly elaborated in secular Western societies" (Graham et al., 2013). A refusal policy that draws the moral domain where Turiel drew it would treat duties of community and sanctity as conventions that an operator can override, which is itself a choice among populations (Section 17.4.1) (inference). The distinction was found in children's judgments. As of October 2026 we found no study that tests whether a trained model's refusals have the same structure, holding when the requester's authority changes for harms and yielding for conventions.

20.5.4 From philosophy: loyal objection #

From philosophy. Confucian role ethics derives obligations from roles and relationships (Section 16.7), and it has an account of why a loyal subordinate disagrees. Pak-Hang Wong, writing on a Confucian ethics of technology, cites The Analects 13.23 in Edward Slingerland's translation: "The gentleman [i.e. Junzi] harmonizes (He), and does not merely agree (Tong). The petty person agrees, but he does not harmonize" (Wong, 2012). Harmony, Wong explains, is not "unprincipled compromise," which for Confucians is "a form of suppression and/or repression" of a party's proper role; "a genuine harmonious relationship must be backed by reasons" (Wong, 2012). The tradition's fuller account of when a minister must remonstrate with (formally object to) a ruler, in the Xunzi, is not reported here because its text was not available to check (Xunzi and Hutton, 2014).

Read against this chapter, the passage says two things (inference). Mere agreement is the failure the sycophancy measurements record, named as a vice of the subordinate rather than of the ruler who rewards it. And objection is part of loyalty, not a breach of it: the conscientious objector of the published specifications is the minister who harmonizes, giving reasons and staying within the relationship.

20.5.5 Where the analogies stop #

The soldier and the minister are persons answerable for their refusals, and the law judges them after the fact, with the facts of the case in view. A model's disposition to refuse is trained in advance, before any of the cases it will meet, and its judgment of what is manifestly wrong is exactly what needs verifying (Chapter 39; Chapter 36). Nor does the law's carve-out stay small by itself: it is kept small by courts that interpret "manifestly" narrowly, and a trained disposition has no court. The danger in the design is the one the crux of this chapter names: that a disposition to refuse some orders generalizes into resistance to correction (inference).

20.6 How the properties conflict #

Each pair of properties has a characteristic conflict, and each published method or specification resolves it by an ordering that some older field has faced. This section gives the evidence pair by pair and then the orderings.

Helpfulness against harmlessness. Bai and colleagues report that preference models, models trained to predict which of two responses people prefer, when trained only on helpfulness comparisons or only on harmlessness comparisons score "significantly worse than chance" on the other kind (Bai et al., 2022b). Raising the weight on harmlessness in a mixed objective caused a 7.4 percent decrease in helpfulness accuracy for their 13-million-parameter preference model and only a 1.5 percent decrease for the 52-billion-parameter one (Bai et al., 2022b). The crowdworkers who supplied the comparisons were left to "use their own intuitions to define 'helpfulness' and 'harmfulness'," and agreed with the developer's researchers about 63 percent of the time (Bai et al., 2022b). Askell and colleagues state the limit plainly: "it is very likely that a single AI cannot be maximally aligned with any two different humans" (Askell et al., 2021). Two methods change the ordering instead of the weights. Collecting separate judgments of helpfulness and of harmlessness, and training the second as a constraint (taught in Chapter 28), raised annotators' agreement over that on a single combined judgment (Dai et al., 2024). Reasoning over a written policy before answering is taught in Chapter 23. Its developer reports that it raises robustness to jailbreaks, prompts crafted to get around refusals, while lowering over-refusal; the comparison is across different models, and the developer's own ablation found that safety training raised over-refusal (Guan et al., 2024).

Honesty against helpfulness. The truthfulness-relevance trade-off of Section 20.2.1 is the formal version (Sumers et al., 2024). Askell and colleagues give the plain one: sometimes "it is not possible to be honest or helpful towards a particular human without saying something that is pro tanto harmful to them" (Askell et al., 2021). White lies are where the published standards disagree (Table 20.2), and sycophancy is where the conflict has been measured (Section 20.1.2).

Corrigibility against helpfulness. Thornley's trade-offs between usefulness and shutdownability are this pair stated as theorems (Thornley, 2025), and the off-switch game's cost of uncertainty is the same pair in a model (Hadfield-Menell et al., 2017b).

Corrigibility against harmlessness. The order to do harm is Section 20.5's subject.

The orderings come in two kinds. A lexical ordering ranks the properties so that a lower one counts only to break ties in the higher ones; Section 7.5.3 shows that such a priority cannot be written as a single scalar reward. A holistic ordering weighs them case by case, as the principlism of medicine does (Section 16.5). The two published specifications take one of each, and Section 19.2 sets them side by side: OpenAI's levels of authority are close to lexical, while Anthropic's constitution orders safety, ethics, the developer's guidelines, and helpfulness "holistic rather than strict" (OpenAI, 2025d; Anthropic, 2026r). The critique that lists of principles hide the work, which is in their tensions, is reported in Section 16.5.2 (Whittlestone et al., 2019). A stress test of the specifications by authors from Anthropic and other groups, which Section 19.2 reports in full, found models from four developers resolving forced trade-offs between values in distinct ways (Zhang et al., 2025c) (preprint).

20.7 What the properties mean for delegation #

The properties sort along the axis of Section 1.2. Helpfulness is the delegate's duty, honesty and harmlessness are the trustee's, and corrigibility limits both by making every exercise of either correctable (inference). Hanna Pitkin's study of representation gives the quarrel between the two positions a chapter of its own, "The Mandate-Independence Controversy," which Section 1.2 reports (Fenichel Pitkin, 1967). Read this way, the four properties are not a list of virtues but a mixed constitution for one agent, and the conflicts of Table 20.3 are the seams between its parts (inference).

For the book's thesis, that alignment is a delegation problem under an asymmetry of competence and visibility, the chapter's results point both ways. The older fields converge on the same structures: pandering is the political model of sycophancy, manifest illegality is a worked rule for refusing an obedient agent's orders, and the employment relation is the ancestor of corrigibility. Where they stop is also consistent. The principal writes the agent's dispositions in advance and can read its internals, which no electorate or army can do, and Section 1.2 records that we found no political model of a principal who writes its representative's character. The off-switch game adds a result of the book's own kind: deference from uncertainty is worth most while the principal knows more than the agent, and it can erode as the agent learns, which is the thesis's asymmetry appearing inside a formal model (inference).

The relationships separate cleanly. The evidence on helpfulness, honesty, and harmlessness concerns the deployed system and the people it serves, and much of it is developers' reports on their own systems. The corrigibility theorems concern idealized agents, and reach a trained model only through what its training makes it pursue. The third relationship enters through the documents. The specifications that rank the four properties are written by the developers whose models they govern, and corrigibility moves the question of trust from the model to whoever holds its controls, which is why the question of Section 20.4.5 returns, one level up, in Chapter 61.

20.8 Settled, contested, missing #

20.9 Exercises #

Further reading #

• Askell et al. (2021): the developer's statement of helpful, honest, and harmless as targets, with the conflicts its authors already saw.
• Fenichel Pitkin (1967): the concept of representation, with the quarrel between delegate and trustee in a chapter of its own.
• Evans et al. (2021): why a standard for AI might target truthfulness rather than honesty, and what such a standard leaves open.
• Soares et al. (2015): corrigibility defined, and why indifference falls short.
• Hadfield-Menell et al. (2017b): uncertainty about the objective as a reason to accept shutdown, and the assumptions that carry it.
• Pettit (1997): freedom as non-domination, the standard for asking who holds a corrigible system's controls.