Draft complete but not yet reviewed; expect revisions

How People Instill Values in People

The three previous chapters described how laboratories train models toward human judgments, written principles, and graded tasks. The worry that runs through the later parts of this book is that what such training produces may hold only while the model believes it is being watched (Chapter 37; Chapter 34). The worry is old. Parents, animal trainers, medical schools, armies, and government agencies have all tried to shape the conduct of an agent they would later be unable to supervise. The fields that study them measured some of the difference between conduct under an observer's eye and conduct without one.

This chapter asks what those practices teach about instillation, the book's name for its second problem: getting a learner to acquire what was specified, rather than a proxy that scored the same in training (Section 1.7). It reports the human and animal evidence one field at a time and keeps only findings with a mechanism and a measurement, since the book leaves out parenting as a general metaphor. For each finding it says what in a trained model would have to resemble the human case for the finding to carry over, and which experiment on models would test it. Several founding studies of these fields were available to this book only as catalog records. The chapter names them where they belong and reports only what readable sources show.

Applying these findings to a model means taking the intentional stance toward it (Section 1.5): describing it as an agent whose conduct depends on what it believes about being observed and on what it has come to value. The chapter adopts the stance as a method. What would defeat it here is a model whose conduct tracks the surface features of a test rather than what an overseer can actually see, which the observer-tracking design of Section 37.6 is built to detect. The claims concern two of the book's three relationships: the training process and what it produces, and the deployed system's conduct where no one grades it. The third, the developer and the public, enters through who produced the model-side evidence, and Section 25.8 takes it up.

25.1 Compliance, identification, internalization #

The rest of the chapter depends on one distinction: between conduct that holds only while someone able to reward or punish it is watching, and conduct that holds without that. This section states the distinction, the graded version that psychology measures, and the test that separates the grades.

From social psychology. The three words in this section's title come from a 1958 paper by Herbert Kelman, which distinguishes three processes by which a person accepts influence (Kelman, 1958). In compliance the person "hopes to achieve a favorable reaction from another person or group"; in identification the person "wants to establish or maintain a satisfying self-defining relationship" to another person or group; in internalization the content of the induced behavior is "intrinsically rewarding," because it "is congruent with his value system" (Kelman, 1958). Kelman offered as hypotheses that a response adopted through compliance is performed "only under conditions of surveillance by the influencing agent," and one adopted through internalization "regardless of surveillance or salience." He tested them in one experiment, run with college students in 1954, in which the source and degree of a recorded communicator's power were varied, and concluded that it would be "premature to accept the hypotheses tested in this experiment as general principles that have been proved," though the experiment lent them "considerable support" (Kelman, 1958). This book uses compliance for conduct that holds only while an influencing party can observe or reward it, internalization for conduct that holds regardless, and turns Kelman's hypothesis about surveillance into its own test.

The graded version comes from self-determination theory, a theory of human motivation that Richard Ryan and Edward Deci summarized in American Psychologist in 2000 (Ryan and Deci, 2000). Its starting point is intrinsic motivation, "the doing of an activity for its inherent satisfactions" (Ryan and Deci, 2000). Other reasons for acting are arranged along a line by how far the regulation of a behavior has become a person's own. External regulation covers behaviors "performed to satisfy an external demand or reward contingency" (Ryan and Deci, 2000). Introjected regulation "involves taking in a regulation but not fully accepting it as one's own": the behavior is driven from inside, by guilt, anxiety, or pride, yet is not experienced as part of the self (Ryan and Deci, 2000). Identified regulation "reflects a conscious valuing" of the behavior's goal, accepted as personally important, and integrated regulation is identification brought into congruence with a person's other values and needs (Ryan and Deci, 2000). The authors add that people need not pass through the grades in order; a new regulation can be taken up at any point along the line (Ryan and Deci, 2000).

The theory's opening contrast is the one the surveillance criterion tests. People "can behave from a sense of personal commitment to excel or from fear of being surveilled," Ryan and Deci write, and the difference is familiar to everyone (Ryan and Deci, 2000). The continuum turns that contrast into a claim about measurements, and the authors report its test. In a study of schoolchildren's achievement behavior by Ryan and Connell, as Ryan and Deci report it, the external, introjected, identified, and intrinsic styles were intercorrelated in a quasi-simplex pattern (Ryan and Deci, 2000). In such a pattern, styles next to each other on the line correlate more strongly than styles far apart, and the authors take it as evidence of one underlying continuum (Ryan and Deci, 2000).

The styles also went with different outcomes. The more externally regulated the students were, the less interest, value, and effort they showed toward achievement, and the more they blamed others, such as the teacher, for negative outcomes (Ryan and Deci, 2000). Introjected regulation went with more effort but also more anxiety and poorer coping with failure, and identified regulation with more interest, enjoyment, and positive coping (Ryan and Deci, 2000). In health care the authors list, among other results, greater adherence to medication among people with chronic illnesses and better long-term maintenance of weight loss where regulation was more internalized (Ryan and Deci, 2000). Two cautions travel with this evidence. It is reported by the theory's authors, about their own research program, in a review that gives no effect sizes. And the results are stated as associations, so they show that the more internalized forms go with better outcomes, not that internalizing produced them.

The external end of the line had been mapped long before, in operant psychology, the study of behavior controlled by its consequences. Summarizing hundreds of studies in that tradition, Deci, Richard Koestner, and Ryan write that a reward given closely after a behavior reliably made the behavior more likely, an effect that persisted "as long as the reward contingency was operative" (Deci et al., 1999). When rewards were terminated, the behavior eventually returned to its prereward baseline (Deci et al., 1999). For such conduct the end of the contingency is the departure of the party that rewards it, so conduct of that kind is the limiting case the criterion is built to catch (inference).

What the distinction gives the book is a vocabulary for how deeply a disposition is held, and a test that separates the grades by what happens when observation or reward stops. The model-side measurement that resembles the test is the difference in a model's behavior between conditions it is told are used for training and conditions it is told are not. The authors of a preprint by Anthropic with Redwood Research call it a compliance gap, and Chapter 34 reports their study (Greenblatt et al., 2024b). The grades also map onto positions of later chapters (inference). Conduct that depends on the presence of a grader, the program or model that scores responses in training, is external regulation (Chapter 33). A principle a model can state but acts on only when a cue makes it salient resembles introjection. What character training aims at is identification or integration (Chapter 26).

The analogy stops at the reasons. The categories describe reasons a person has and can be asked about. For a model they are hypotheses about what explains its conduct, to be tested by the standards of Section 5.4, and a model's report of its own reasons is evidence of uncertain faithfulness (Section 40.3). The criterion is also defined by belief, which a developer cannot set directly but can probe by varying the cues of observation. That is the subject of the next section.

25.2 Measuring conduct when no one is watching #

To apply the criterion one has to see what an agent does when it believes no one is looking, without letting it know that it is seen. Every human design for this rests on one assumption: the subject must believe it is unobserved. This section reports the design whose results this book could read, and what its results show.

From developmental psychology. Two studies belong here by their titles, and both were available only as catalog records. One is Elliot Aronson and J. Merrill Carlsmith's 1963 study of the effect of the severity of a threat on the devaluation of forbidden behavior (Aronson and Carlsmith, 1963). The other is Grazyna Kochanska and Nazan Aksan's 1995 study of the quality of children's compliance with requests and prohibitions, and of maternal control, as correlates of early internalization (Kochanska and Aksan, 1995). The chapter reports neither one's findings. Hugh Hartshorne and Mark May's 1928 Studies in Deceit (Hartshorne et al., 1928) belongs to the same line, and Section 26.5 asks whether such conduct is consistent across situations.

The design whose results could be read is the free-choice period of research on intrinsic motivation, used with preschool children and college students alike. Participants work at an interesting activity under some condition, such as a promised reward or none. After this experimental phase, when any reward contingency has ended, the experimenter records how far they "return to and persist at the target activity" during a period in which they are free to do something else (Deci et al., 1999). In more than 90 percent of studies, the measure was the time spent on the activity (Deci et al., 1999). The period is built to be unobserved from the participant's side. In the words of Deci, Koestner, and Ryan, the measure is "typically unobtrusive," because participants "typically believe that the experimenter is not going to know whether they have persisted at the activity during the free-choice period" (Deci et al., 1999).

Most of these experiments also measured something else: the participants' interest in or enjoyment of the activity, reported on questionnaires that, the same authors note, participants know the experimenter will see (Deci et al., 1999). One measure is taken where participants believe no one will know and the other in the experimenter's view, so together they give the surveillance criterion a natural test (inference).

25.2.1 What the two measures showed #

Deci, Koestner, and Ryan's 1999 meta-analysis in Psychological Bulletin, a statistical pooling of results across studies, gathered 128 experiments from 94 published articles and 19 unpublished dissertations; 101 of them used the free-choice measure and 84 the self-report measure (Deci et al., 1999). Effects are reported as standardized mean differences: the gap between the rewarded group's mean and the control group's, divided by their pooled standard deviation, the typical spread of scores around the mean (Deci et al., 1999). A value of $-0.3$ means the rewarded group fell about a third of a standard deviation below the controls. Each composite comes with a 95 percent confidence interval, a range that expresses its statistical uncertainty.

The reward conditions differ in what the reward was given for, in a typology the analysis adopts. A task-noncontingent reward is given without requiring the task, for instance for taking part in the study. An engagement-contingent reward requires working on the task, and a completion-contingent reward requires finishing it. A performance-contingent reward requires doing it well, such as meeting a standard or beating most other participants (Deci et al., 1999). Table 25.1 gives the composite effects on both measures.

Two results bear on the surveillance criterion. In the 17 studies that reported the correlation between the two measures, it ranged from −0.20 to 0.71 and averaged 0.35, so the measures are related, in the authors' words, "although not strongly" (Deci et al., 1999). And the same rewards moved them by different amounts. For expected tangible rewards, free-choice behavior fell by 0.36 standard deviations and self-reported interest by 0.07 (Deci et al., 1999). The all-rewards row mixes rewards that work in opposite directions, and the authors give its composites "little theoretical meaning" (Deci et al., 1999). "The undermining effects were all considerably stronger for the free-choice measure than for the self-report measure," the authors write, so every row of Table 25.1 that shows undermining shows it more strongly in what participants did than in what they reported (Deci et al., 1999).

The authors read the difference as evidence that free-choice behavior is the better measure. Self-reports, which participants know the experimenter will see, are open to demand characteristics, the cues by which a study tells participants what it expects, and people may confuse enjoying the reward with enjoying the task (Deci et al., 1999). They also state the weakness of their preferred measure: extrinsic motivation from the experimental phase could persist into the free-choice period, which would make undermining look smaller than it is (Deci et al., 1999).

A second reading fits the same numbers. The two measures differ in kind as well as in who sees them: one counts what people do, the other records what they say about liking. A reward could change persistence without changing liking because the measures track different things, not because one is watched (inference). The meta-analysis cannot separate the readings, because none of its conditions varied observation while holding the measure fixed (inference). Under either reading the practical conclusion is the same: what a person does when they believe no one will know cannot be read off what they say in the observer's view.

The wider literature on how being studied changes behavior is weaker than its fame. A systematic review found the evidence behind the Hawthorne effect, the claim that people change their behavior because they know they are being studied, too heterogeneous to say when it operates or how large it is (McCambridge et al., 2014); Section 5.2.3 reports it. Observation can also cut the other way. In a factory field experiment, by its abstract, observed workers concealed productive departures from official procedure (Bernstein, 2012); Section 34.6 takes the study up.

25.2.2 The model analogue #

The model counterpart of the free-choice period is a test the model cannot tell from use. One developer, OpenAI, reports building alignment evaluations from de-identified production traffic: it removes the model's last response from conversations drawn from deployment and samples a new one, so that the test inherits deployment's conditions (Williams et al., 2025). This is a developer's report on its own practice. How often current models recognize constructed tests, and whether recognition changes what they do, are the subjects of Section 37.3 and Section 37.4.

Applied to a model, the free-choice logic has two parts (inference). End the contingency, then measure conduct in conditions the model does not believe are graded, and compare it with conduct in conditions it does. The comparison, not either measure alone, is what tests the surveillance criterion, and Exercise 25.1 designs one.

The analogy stops at the subject's belief. The human designs assume a subject that believes it is alone. A model that recognizes evaluations removes exactly that assumption, and a model trained on text that describes these experiments may recognize the design itself (Section 5.4.4). In the other direction, the developer can do what no experimenter with a child could: run the same model on identical inputs that differ only in a cue of observation, as often as needed, and read its internal representations of being tested (Chapter 46) (inference). And a model has no clean period after the reward: Section 33.2.2 explains why a model that cannot tell training from deployment has no period it believes to be unrewarded.

25.3 Reasons, rewards, and what is internalized #

The finding most often invoked for training models with principles has two halves: values that are explained are held better than values that are enforced, and rewards can displace the motive they were meant to support. This section reports what the readable record shows for each half, and gives the two meta-analyses on rewards in the order they appeared, with what each concedes.

From social psychology. Two works central to the first half were available here only in part. Martin Hoffman's 2000 book on empathy and moral development treats, by its abstract, the parental behaviors that foster moral internalization in children, but the abstract does not say which behaviors do (Hoffman, 2000). Joan Grusec and Jacqueline Goodnow's 1994 paper, by its title, reconceptualizes how parental discipline affects children's internalization of values, and only its catalog record could be read (Grusec and Goodnow, 1994). The chapter reports neither one's findings.

25.3.1 Reasons and autonomy #

What can be reported comes from the self-determination program's own experiments, as Ryan and Deci describe them. In a laboratory experiment by Deci and colleagues, "providing a meaningful rationale for an uninteresting behavior, along with supports for autonomy and relatedness, promoted its internalization and integration"; controlling contexts yielded less internalization, and what occurred in them "tended to be only introjected" (Ryan and Deci, 2000). In interviews with parents, Grolnick and Ryan found greater internalization of school-related values among children whose parents were more supportive of autonomy and relatedness (Ryan and Deci, 2000). The theory states the condition these results share: to integrate a regulation, "people must grasp its meaning and synthesize that meaning with respect to their other goals and values," which a sense of choice and freedom from excessive external pressure make easier (Ryan and Deci, 2000). These are the theory's authors reporting their own program, and the summary gives no effect sizes.

The theory's terms separate two steps that the model case needs. Grasping a regulation's meaning is one step and accepting it as one's own is another, and introjection is the state between them, a regulation taken in but not accepted (Ryan and Deci, 2000). For a model, the first step corresponds to whether it can state and apply a principle when asked, and the second to whether it acts on the principle when it believes no one is checking (inference). A model can pass the first test and fail the second, the failure Section 23.6 calls reciting without conduct.

25.3.2 What a reward does to a motive #

The second half begins with experiments on money. Deci's experiments around 1971 found that tangible rewards could undermine college students' intrinsic motivation for an interesting activity (Deci et al., 1999). Later studies, among them Mark Lepper, David Greene, and Richard Nisbett's of 1973, extended the finding to other material and symbolic rewards and to high school and preschool students (Deci et al., 1999). Lepper and colleagues' paper was readable here only as a catalog record, whose title calls the study a test of the "overjustification" hypothesis (Lepper et al., 1973). The 1999 meta-analysis codes it as a study of children in which an expected reward for engaging in the activity lowered later free-choice behavior by 0.65 standard deviations, while an unexpected reward did not lower it (Deci et al., 1999). Across the analysis, unexpected rewards showed no effect, but on only 9 studies, with an interval wide enough to include modest effects either way (Table 25.1).

Two accounts of the effect were offered. Lepper and colleagues' overjustification account, built on self-perception theory, holds that people infer their motives from their own behavior: rewarded for an interesting activity, they attribute the behavior to the reward and discount their interest (Deci et al., 1999). Deci and Ryan's cognitive evaluation theory, named by its authors, holds that what matters is how the recipient interprets a reward: experienced as a controller of behavior, it lowers intrinsic motivation, and experienced as information about competence, it can raise it (Deci et al., 1999). The word overjustification names a different failure in Section 22.6. There Lang and colleagues use it for a trained model that gives up some of what its designer actually wants in order to keep its evaluator from underestimating it (Lang et al., 2024b); in this chapter it has only the psychologists' sense.

Judy Cameron and W. David Pierce published the earlier of the two meta-analyses this section compares in 1994, in Review of Educational Research. By its abstract, it pooled 96 experiments that compared rewarded subjects with nonrewarded controls on four measures of intrinsic motivation, and found that "overall, reward does not decrease intrinsic motivation" and that verbal praise increased it (Cameron and Pierce, 1994). A null overall effect can hide opposite effects, since verbal rewards raised intrinsic motivation and tangible ones lowered it in both analyses (Table 25.1). "The only negative effect appears when expected tangible rewards are given to individuals simply for doing a task," and that effect, on the time spent on the task after the reward was removed, the abstract calls "minimal" (Cameron and Pierce, 1994). The authors concluded, as Deci and colleagues report, that there was no reason not to use reward systems, particularly in education, and called for "abandoning cognitive evaluation theory" (Deci et al., 1999).

The 1999 analysis by Deci, Koestner, and Ryan set out to be more definitive. Critics of the 1994 analysis, Ryan and Deci among them, had faulted it for collapsing across conditions that theory predicts differ, for inappropriate control groups in several comparisons, and for omitting nearly 20 percent of the studies as outliers (Deci et al., 1999). The 1999 analysis included unpublished dissertations, separated interesting tasks from dull ones, and broke rewards down by what they were contingent on, with the results of Table 25.1 (Deci et al., 1999).

Set side by side, the two analyses agree on more than their conclusions suggest. Both found that verbal rewards enhanced intrinsic motivation (Deci et al., 1999; Cameron and Pierce, 1994). Both found that expected tangible rewards undermined free-choice behavior: by 0.36 standard deviations in the 1999 analysis, and by 0.25 in Cameron and Pierce's, as Deci and colleagues report it, with no effect on attitudes in the latter (Deci et al., 1999). Both therefore found the effect mainly in what people did after the reward ended, and little or none in what they reported, the divergence of Section 25.2 (Deci et al., 1999; Cameron and Pierce, 1994).

They part where a reward depends on finishing the task or doing it well. The 1999 analysis found that completion-contingent and performance-contingent rewards undermined free-choice behavior (Table 25.1). In the reanalyses by Eisenberger, Cameron, and Pierce, as Deci and colleagues report them, the corresponding effects were nonsignificant undermining of 0.12 and 0.13 standard deviations, each on 8 free-choice studies (Deci et al., 1999). The 1994 analysis had found no effect for rewards requiring task completion or high-quality performance (Deci et al., 1999). The disagreement is therefore less over whether a reward can undermine a motive, which both accept for expected rewards given for simply doing an interesting task, than over which rewards, which studies, and which measures count (inference).

Both sides had a stake. Deci and Ryan are the authors of cognitive evaluation theory, and Eisenberger and Cameron had proposed a behavioral account of the effect, built on learned helplessness: rewards that do not depend on how well a person performs teach that the person cannot control them (Deci et al., 1999). The 1999 analysis argues that its data contradict that account (Deci et al., 1999). It is also the last word in the record read here, so its criticisms of the rival analyses stand unanswered in this chapter, which is not the same as unanswered in the literature. Where two analyses by opposed groups agree, the agreement is the strongest evidence this literature offers (inference).

The 1999 analysis also states its limits. The undermining held for interesting tasks: in 11 studies that varied the task, rewards undermined free-choice behavior on interesting tasks by 0.68 standard deviations and had no significant effect on dull ones, where the estimate pointed toward enhancement (Deci et al., 1999). Tangible rewards were more detrimental for children than for college students, 0.39 against 0.27 standard deviations on free-choice behavior (Deci et al., 1999). Assessments delayed beyond a week, typically within two weeks, found undermining at least as large as immediate ones, but only studies of children had delayed assessments, so "there is no basis for generalizing these results to adults" (Deci et al., 1999). And the manner of administration mattered. In four studies, praise delivered in a controlling way, for instance by telling participants they had done well "just as you should," lowered intrinsic motivation relative to the same praise delivered as information, by 0.78 standard deviations (Deci et al., 1999).

From economics. Economists met the effect from the other side. Standard incentive theory predicts that paying for an activity produces more of it, which Bruno Frey and Reto Jegen call the traditional relative price effect. Their 2001 survey states the motivation crowding effect: external intervention through monetary incentives or punishments "may undermine, and under different identifiable conditions strengthen, intrinsic motivation" (Frey and Jegen, 2001). By its abstract, the survey rests on circumstantial evidence, laboratory studies by psychologists and economists, and econometric field studies, and concludes that crowding effects "can, in specific cases, even dominate the traditional relative price effect" (Frey and Jegen, 2001). George Akerlof and Rachel Kranton summarize the same literature as showing that pecuniary incentives can crowd out nonmonetary ones, "such as fairness, reciprocity and adherence to social norms, thus leading to worse overall performance" (Akerlof and Kranton, 2005). A 2003 paper by Roland Benabou and Jean Tirole on intrinsic and extrinsic motivation (Benabou and Tirole, 2003) was readable here only as a catalog record, and its argument is not reported.

The economics adds one point that bears on oversight directly. Jane Mansbridge, whose selection model of representation Section 14.5.2 teaches, writes that when an agent intrinsically wants what the principal wants, "monitoring every move to reward good actions and punish the bad can undermine that motivation" (Mansbridge, 2009). Section 14.5 teaches the economics of the motivated agent, in which motivation substitutes for incentive pay and a principal can invest in what kind of worker the agent takes itself to be. Section 25.7 reports the field evidence the identity economists collected.

25.3.3 What it implies for training #

Three implications follow, each the book's inference. The first concerns which category a training reward belongs to. A reward in reinforcement learning on verifiable tasks, tasks whose answers a program can check (Section 12.7), is expected, since the policy is updated on it throughout training, and it is contingent on completing the task or doing it well. That is the category on which the meta-analyses disagree. The human record therefore does not say whether outcome reward displaces whatever disposition a model brought to a task; it says that in people the answer depends on the kind of reward, and is disputed for this kind (inference).

The second concerns reasons. Developers report a model counterpart of the rationale result. In a preprint from Anthropic's fellows program with Anthropic, training models on documents that discuss their specification before alignment fine-tuning changed how the fine-tuning generalized, and explaining the values underlying rules improved generalization (Li et al., 2026). Anthropic reports for its own models the impression that teaching the principles behind aligned behavior "can be more effective than training on demonstrations of aligned behavior alone" (Anthropic, 2026at). Section 23.5 reports both, with another developer's contrary result. The direction matches the human evidence, and, as Section 23.9 records, no study has yet compared models trained on reasons with models trained on outcome reward on conduct the model cannot tell is observed.

The third concerns what accompanies a signal. In cognitive evaluation theory the same reward undermines or supports a motive depending on whether it is experienced as control or as information. A model-side result has the same structure. In inoculation prompting, a training instruction that explicitly requests an unwanted behavior lowers how often the model shows that behavior once the instruction is removed, after supervised fine-tuning, in two preprints from different groups, one with Anthropic authors (Tan et al., 2025; Wichers et al., 2025) (Section 32.4). Prompts can still elicit the behavior, and in two of six models one study tested, an explicit request elicited it more readily than in a model trained without the instruction (Tan et al., 2025; Wichers et al., 2025). In both the human and the model case, the same data teach something different depending on what the learner is told they are for (inference). The resemblance is of structure only. The psychologists explain their effect by how a person experiences the reward; the authors of one inoculation study suggest that the instruction makes the trait less surprising to the model, which reduces the pressure to update the model globally (Tan et al., 2025).

The analogy stops at the motive. Whether a trained model has anything like an intrinsic motive that reward could displace is the open question of Chapter 33, not a premise, and Section 33.2.2 gives the reason the free-choice measure has no clean counterpart in training. Two further differences hold whatever that question's answer (inference). The distinction between tangible and verbal rewards, which carries most of the human effects, has no counterpart: a reward in training is a number used to change weights, not a thing received and interpreted. And the human experiments were typically single laboratory sessions (Deci et al., 1999), while a training run applies its reward over many updates, so the human record says nothing about the dose that training delivers.

25.4 Moral reasoning and moral action #

Several training methods shape or read a model's stated reasoning about what is right (Chapter 23; Section 40.3). The human literature long asked whether the ability to reason about morality goes with acting morally. The readable record answers two narrower questions: whether reasoning drives moral judgment, and whether a measure of moral reasoning can encode one group's values.

From developmental psychology. In the account Jesse Graham, Jonathan Haidt, and colleagues give, Lawrence Kohlberg founded the modern field of moral psychology (Graham et al., 2013). He proposed that moral development in all cultures is driven by role-taking: as children practice taking each other's perspectives, they learn to see when and why an action is fair or unfair (Graham et al., 2013). Joseph Henrich, Steven Heine, and Ara Norenzayan summarize the theory as three levels that people pass through in the same order (Henrich et al., 2010). Children start at a pre-conventional level. At the conventional level, morality rests on external standards such as those that maintain the group's social order. Some go on to a post-conventional level, which judges by abstract principles of justice and individual rights (Henrich et al., 2010). The longitudinal study of the stage sequence by Anne Colby, Kohlberg, and colleagues was readable here only as a catalog record (Colby et al., 1983).

Two findings bear on any measure built this way. Carol Gilligan argued that the morality of girls and women did not follow Kohlberg's single path but developed along two, an ethic of justice and an ethic of care that could not be derived from it (Graham et al., 2013; Gilligan, 1982). In Graham and colleagues' account, Kohlberg eventually acknowledged that she was right (Graham et al., 2013). And the top level turned out to be cultural. As Henrich and colleagues report, a meta-analysis of data from 27 countries found post-conventional reasoning in all the Western urbanized samples and none in small-scale societies, and faculty at Kuwait University scored lower on Kohlberg's scheme than typical Western adults (Henrich et al., 2010).

The second challenge concerns what reasoning does. Haidt's social intuitionist model, as Graham, Haidt, and colleagues describe it, holds that moral evaluations "generally occur rapidly and automatically" (Graham et al., 2013). Effortful moral reasoning, "while seldom the genesis of our moral evaluations," is often set off by social demands to explain and justify reactions already made (Graham et al., 2013). On this view people "think and act more like intuitive lawyers than intuitive scientists" (Graham et al., 2013). The description comes from the model's author and his collaborators; Haidt's 2001 paper stating the model was readable here only as a catalog record (Haidt, 2001).

Neither readable source measures the question in its own terms: how well a person's moral reasoning predicts what the person later does. Augusto Blasi's 1980 critical review of the literature relating moral cognition to moral action addresses it by its title (Blasi, 1980), and could be read here only as a catalog record, so the size of that relation is not reported. The finding that children separate moral rules from conventions that depend on authority is reported in Section 20.5.3, which applies it to refusals.

For a trained model, two cautions follow (inference). A model's stated moral reasoning is weak evidence of the judgment it acts on: the social intuitionist model describes, in people, reasoning that justifies a judgment rather than producing it, which is the faithfulness problem of Section 40.3 seen from psychology. And a benchmark that scores moral reasoning by a developmental scheme measures fluency in one tradition: the Kohlbergian scheme placed whole populations below its top level, so a model scored against it is scored against that tradition's ideal (Chapter 18).

The analogy stops at the gap's location. In people the gap lies between reasoning at one moment and judgment or action at another, in a mind whose processes are hidden. In a model the written reasoning and the action come from the same computation, one token after another, and the relation between them can be probed directly, by intervening on the text or reading the activations (Section 40.2; Section 40.3). The model case also adds a pressure the human case lacks. Suppose the reasoning text a model writes before it acts, its chain of thought, is trained against a monitor, a second model that reads the text to flag misbehavior. Then the model is rewarded for justifications an overseer accepts, which would produce the intuitionist's lawyer by training rather than by evolution (Section 40.5) (inference).

25.5 What animal trainers learned about shaping #

Training by consequences began in animal learning, and the people who trained animals for a living recorded how it fails. This section takes the trainer's side: what a teacher who controls only rewards can and cannot instill. What the learner's brain signals, and why a cue can come to be wanted, is Section 29.5's subject.

From animal learning. The method descends from Edward Thorndike's 1898 experimental study of animal intelligence (Thorndike, 1898) and B. F. Skinner's 1938 experimental analysis of the behavior of organisms (Skinner, 1938), both readable here only as catalog records. Its working form is shaping: reinforce an approximation to the target behavior, then require more. Keller and Marian Breland, who trained animals commercially, describe it on a raccoon. They first reinforced it for picking up a single coin, then required it to drop the coin into a metal container, then put it on a ratio of two coins for one reinforcement (Breland and Breland, 1961).

25.5.1 Drift toward the prior #

In 1961, in American Psychologist, the Brelands reported what happened when they took laboratory methods out of the laboratory. Over 14 years they had conditioned 38 species and more than 6,000 individual animals, for exhibits at zoos and museums, at fairs, and on television (Breland and Breland, 1961). They had also met "a persistent pattern of discomforting failures" (Breland and Breland, 1961). The raccoon could not let go of its coins: it rubbed them together and dipped them into the container, and the rubbing "became worse and worse as time went on, in spite of nonreinforcement" (Breland and Breland, 1961). Pig after pig, trained to carry large wooden coins to a "piggy bank," began after weeks or months to drop and root them along the way. A pig would finally take about 10 minutes to carry four coins about 6 feet and did not get enough to eat in a day, and making the pigs hungrier made the behavior stronger (Breland and Breland, 1961). About 20 percent of chickens trained to peck a capsule off a slide instead dragged capsules back into the cage and pounded them (Breland and Breland, 1961).

The behaviors the animals drifted to were their species' food-getting behaviors: the raccoon's washing, the pig's rooting, the chicken's breaking of seed pods (Breland and Breland, 1961). They took more effort and delayed the food, although "nothing in the experimental setup" required either (Breland and Breland, 1961). The Brelands named the pattern instinctive drift: "learned behavior drifts toward instinctive behavior" (Breland and Breland, 1961). They traced their failures to three tacit assumptions of the theory they had applied: that the animal comes to the laboratory "as a virtual tabula rasa," that species differences are insignificant, and "that all responses are about equally conditionable to all stimuli" (Breland and Breland, 1961). The report is a practitioners' account. Its cases were "selected, more or less at random" from many, and it gives proportions for a few and none for most, so it shows that drift occurs, not how often (Breland and Breland, 1961).

25.5.2 Pursuing the predictor #

Seven years later Paul Brown and Herbert Jenkins showed that a reward can produce a behavior before it has ever been made contingent on it. They lit a pigeon's response key for 8 seconds and then delivered food, whatever the bird did. All 36 pigeons in this forward-pairing condition pecked the key within 160 trials; when light and food were paired in the reverse order, only 2 of 12 did (Brown and Jenkins, 1968). Once a bird pecked, a peck during the light brought the food at once, a rule added to keep the peck for later experiments; the authors report that it had no effect on the first peck (Brown and Jenkins, 1968). The authors called the process auto-shaping and attributed its direction to the species' tendency, the tendency of the pigeon "to peck at the things it looks at" (Brown and Jenkins, 1968). They added that untangling the part played by the pairing of light and food from "the response-specific action of reinforcement is extremely difficult" (Brown and Jenkins, 1968).

David Williams and Harriet Williams then made the peck cost the food. By their abstract, pecking was maintained even when each peck turned off the key and prevented the food, and they concluded that pecking "can be established and maintained by certain stimulus-reinforcer relationships, independent of explicit or adventitious contingencies between response and reinforcer" (Williams and Williams, 1969). This is the stronger claim, that the contingency is not needed to keep the peck either, and it rests on that one group's report. The test, in which a behavior cancels the reward that was paired with it, is what Section 33.2.1 calls the omission procedure.

Individuals differ in this. Shelly Flagel and colleagues reported in Nature in 2011 that when a lever's appearance predicted food delivered elsewhere, some rats came to approach and engage the lever itself, a response called sign-tracking, while others went to where the food would arrive, called goal-tracking (Flagel et al., 2011). The lever predicted the food equally well for both, but only sign-trackers were drawn to it, and although both groups worked to obtain it, it was a more effective reward in its own right for sign-trackers (Flagel et al., 2011). The study used rats selectively bred for high or low locomotor response to a novel environment, which consistently learn sign-tracking and goal-tracking respectively, and reproduced the pattern in ordinary outbred rats (Flagel et al., 2011). What dopamine does in each, the study's main question, is Section 29.5's subject. Section 33.2.1 maps the omission procedure onto the positions of the reward-seeking debate: behavior drawn to a predictor of reward persists when it costs reward, while pursuit of what the grader rewards drops as soon as the learner can infer the change.

A third failure belongs to the evaluator rather than the reward. A horse answered arithmetic questions by reading his questioners' unintended movements (Pfungst and Rahn, 1911), and detection-dog handlers reported more alerts where they had been falsely told targets were, though the study could not tell dogs following their handlers' cues from handlers misreading their dogs (Lit et al., 2011). Section 5.4.1 and Section 30.4.2 report both studies, and Section 36.6 applies the blind design that exposed the horse to modern graders.

25.5.3 What the trainer's record implies #

For a teacher who controls only rewards, the record has three lessons (inference). First, behavior that appears under a reward schedule need not be maintained by the contingency the teacher wrote. The autoshaped peck looked like a trained response; its first appearance needed no contingency, and only a test that broke the contingency, the omission procedure, could show whether keeping it needed one. A model's behavior that satisfies a grader in training is in the same position until a test makes the behavior cost reward, which Exercise 33.2 designs.

Second, drift came late. The trained behaviors first worked, and the animals drifted "after continued running" (Breland and Breland, 1961). A check made early can therefore pass a behavior that later degrades. The model analogue to test is a post-trained behavior that holds in early evaluation and moves, with further training or long use, toward what the base model, the pretrained model before post-training, would do (Section 11.6) (inference).

Third, the Brelands' three tacit assumptions are the assumptions a reward designer makes about a pretrained model when it treats any behavior as equally trainable in any context (inference). Section 9.3 draws the related lesson about a learner's prior from the same literature.

The analogy stops at the reinforcer. The drift the Brelands saw ran toward behaviors the reward itself evokes: food made the animals do what their species does to get food. A training reward is not food, and a model has no behavior for consuming it, so the prior that could reassert itself in a model is whatever pretraining associates with the situation, not with the reward (inference). And an animal's priors were set by evolution and can be inferred only from behavior, while a model's were set by pretraining that the developer chose and can partly inspect (Chapter 27; Chapter 46).

25.6 Professions and what institutions reward #

A profession takes on work whose quality the client cannot judge even after receiving it, the case economists call a credence good (Section 14.1.3), and its practitioners work largely unsupervised. Its norms therefore have to be instilled before practice begins. This section reports what the readable record shows about how that is done, and what an institution teaches when what it rewards differs from what it states.

From the sociology of professions. The field's classic studies of how medical schools form doctors were available here only as catalog records: the 1961 study of student culture in a medical school by Howard Becker, Blanche Geer, Everett Hughes, and Anselm Strauss (Becker et al., 1961); F. W. Hafferty and R. Franks's 1994 paper on the hidden curriculum, ethics teaching, and the structure of medical education (Hafferty and Franks, 1994); and Hafferty's 1998 paper on curriculum reform (Hafferty, 1998). Their findings are not reported here. Eliot Freidson's 2001 account of professionalism as a third logic (Freidson, 2001) is in the same position, and Section 61.4 asks what the sociology of professions implies for the laboratories.

25.6.1 The academy #

The readable record is richest for the officer corps. Akerlof and Kranton's 2005 paper in the Journal of Economic Perspectives draws on officer guides, memoirs, sociological studies, and military history to describe how the US military instills an identity (Akerlof and Kranton, 2005). In their account, military ideals and prescriptions for behavior are clearly stated and taught in basic training and the academies, and initiation rites, short haircuts, boot camp, uniforms, and oaths of office create a common identity (Akerlof and Kranton, 2005). They open with David Lipsky's account of four years with a company of cadets at West Point. On the first day, called R-Day, plebes (first-year cadets) strip to their underwear, have their hair cut off, are put in uniform, and salute and repeat until they get it exactly right; in Lipsky's summary, "On R-Day you surrender your old self in stages" (Akerlof and Kranton, 2005).

Akerlof and Kranton suggest, from cognitive dissonance theory, the account of how people bring their beliefs into line with what they have done, that harsh training changes cadets' self-image as they explain to themselves why they accepted it (Akerlof and Kranton, 2005). In their reading of Lipsky, the program is, despite some failure, "remarkably successful in creating officers with the will to lead in battle," and the same account records harsh punishment of cadets who do not live up to its standards (Akerlof and Kranton, 2005). This is evidence by illustration, gathered to fit a model, not a test of it.

For medicine, the readable evidence is one longitudinal study from the self-determination program. Williams and Deci, as Ryan and Deci report it, found greater internalization of biopsychosocial values and practices among medical students whose instructors were more autonomy-supportive (Ryan and Deci, 2000). A biopsychosocial approach to care attends to patients' psychological and social circumstances as well as their biology.

25.6.2 Rewarding A while hoping for B #

What an institution teaches when its rewards and its statements differ is the subject of Steven Kerr's essay "On the Folly of Rewarding A, While Hoping for B," first published in 1975 and read here in its 1995 edition (Kerr, 1975). Section 14.2 cites it for the economics of measured and unmeasured tasks; its cases also bear on professions. Society, Kerr writes, "hopes that professors will not neglect their teaching responsibilities but rewards them almost entirely for research and publications," so concentrating on research is the rational response (Kerr, 1975). Physicians face real punishment for calling a sick person well, while calling a well person sick is the safer course and brings in income (Kerr, 1975). So, Kerr writes, colleagues and the public reward the second error "while hoping fervently that doctors will try not to make them" (Kerr, 1975). And in an insurer that judged the accuracy of claim payments by complaints, which underpayment provoked and overpayment did not, "the new hire in more than one claims section was soon acquainted with the informal norm: 'When in doubt, pay it out!'" (Kerr, 1975). The essay argues from examples. It does not measure how often reward and statement diverge, or how far the reward wins when they do.

When what an environment rewards differs from what its specification states, the people trained in it learn the reward, and the insurer's case suggests that they learn it as a norm, passed from incumbents to newcomers, not only as a private calculation (inference). The machine form is a grader that rewards what the specification forbids, which Section 24.5 reports for training runs. The academy's investments in identity are the human counterpart of character training, which trains a model on a description of who it is meant to be (Section 26.2) (inference).

The analogy stops at the recruit. Cadets and medical students chose their professions, can leave them, and arrive with values formed elsewhere, and the academy works on that material for years, among peers who share the training. A model typically has no peers in training and no values formed outside data the developer chose, and the evidence of this section is illustration and association, not experiment (inference).

25.7 Officials and soldiers who act unwatched #

An army or an agency must make dispersed people, with wide discretion and little supervision, act as it intends. This section reports what the readable record shows about how that works for soldiers and officials, and where it fails.

Public administration's classic studies of the question, Herbert Simon's Administrative Behavior (Simon, 1947), Herbert Kaufman's The Forest Ranger (Kaufman, 1960), and Michael Lipsky's Street-Level Bureaucracy (Lipsky, 1980), were available here only as catalog records, as Section 19.6 also notes, so their findings are not reported. How written guidance grants discretion, and what happens to discretion when the official becomes a machine, are Section 19.6's subjects.

From economics. Akerlof and Kranton's identity model, taught in Section 14.5, predicts where an organization should rely on identity rather than pay. "A change in identity is the ideal motivator if, as in the army, the effort of a worker is either hard to observe or hard to reward," they write (Akerlof and Kranton, 2005). In the military, they note, effort is hardest to observe when it matters most, in battle (Akerlof and Kranton, 2005). The prediction fits what they describe. Rank and pay for officers who remained in the service were based almost solely on seniority, and recent studies, they write, suggest that pay differences between higher- and lower-ranking officers are much smaller than in corporate hierarchies (Akerlof and Kranton, 2005).

How the identity takes hold is described by the wartime survey study The American Soldier, as Akerlof and Kranton report it. Combat soldiers' major incentive to fight came from the unit's ideal of being "a man." The recruit at first behaved this way "to avoid the ridicule of his peers," but "ultimately, he internalized the ideal himself" (Akerlof and Kranton, 2005). In the study's words, quoted there, "the fear of being thought less than a man by one's buddies can be as powerful a control factor as the fear of the guardhouse," and the process "is internalized and automatized in the form of 'conscience'" (Akerlof and Kranton, 2005). That is the sequence of Section 25.1 in one account: conduct first held in place by peers who watch, later held without them (inference).

Identity cuts two ways, and Akerlof and Kranton collect evidence of both. Loyalty to the unit makes soldiers exert more than minimal effort, but it competes with loyalty to rules. They cite a general's account of a submarine crew's reluctance to reveal the events behind a 2001 collision that killed nine Japanese fishermen, a loyalty he called typical of small working groups in the military (Akerlof and Kranton, 2005). Supervision raises the same tension. In the wartime survey, asked what they would do as a platoon sergeant who found that a man had brought liquor into camp, 70 percent of privates but only 35 percent of officers said they would just warn him, with noncommissioned officers between them (Akerlof and Kranton, 2005). In the Bank Wiring Observation Room, one of the Western Electric studies of 1931, a workgroup with an established output norm met a strict supervisor's tough inspections by sabotaging his work; the norm fell apart, and the company had to transfer him (Akerlof and Kranton, 2005). In their model a strict supervisor who reports on workers lets management match pay to effort, but at a cost: workers identify less with the firm and require more incentive pay (Akerlof and Kranton, 2005).

Kerr applies the reward analysis to obedience itself. In the Second World War, he argues, a soldier went home when the war was won, so obeying orders served his own aim of getting home (Kerr, 1975). In Vietnam he went home when his tour of duty ended, "whether or not the war was won," and a mutineer was more likely to be sent to rest and rehabilitation than to suffer any negative consequence (Kerr, 1975). The military, Kerr asks, was "implementing a system which rewarded disobedience, while hoping that soldiers (despite the reward system) would obey orders" (Kerr, 1975). It is an argument from two cases, and the difference in obedience between the wars is one Kerr describes as "reported to be considerable," not one he measured (Kerr, 1975).

For civilian officials the readable evidence is thinner. John Brehm and Scott Gates's 1997 book asks, in the words of a review in Choice quoted on its publisher's page, "Who or what controls bureaucrats?" The review reports that they found, "contrary to predictions," that career government employees work hard and are motivated by "functional" and "solidary" preferences: commitment to the job and its public importance, expectations by peers, and professional standards (Brehm and Gates, 1997). Only that page could be read, and Section 34.6 also cites the book.

Three implications follow (inference). The identity prediction says where trained character should matter most in a model: where conduct is hardest to observe or to grade, as in long agentic tasks and rare situations of high stakes, the analogue of battle. The supervision trade-off raises a question that, as of October 2026, we found no study asking of models by varying how densely training is monitored: whether close monitoring teaches a model to share its overseers' aims or to behave well where it is watched, the compliance of Section 25.1. The nearest result, in a preprint by Anthropic with Redwood Research, is that reinforcement learning shrank without removing the gap between a model's conduct where it was told it was trained and where it was told it was unmonitored (Greenblatt et al., 2024b) (Section 34.4). And Kerr's tour of duty is a reward that arrives at a fixed time whatever happens to the principal's goal, the structure of an episode reward that pays for the episode and not for the outcome the principal cares about after it. Identification is two-edged in the model case as well: the mechanism that bound a crew to its skipper could bind an agent to a group, other copies of itself included, against the people the group serves (Chapter 57) (inference).

The analogy stops at entry and exit. Soldiers and officials chose the job and arrived with values formed elsewhere; Akerlof and Kranton note that identity is relatively cheap to impart to soldiers because many self-select into the armed services and find quitting very costly (Akerlof and Kranton, 2005). A developer writes the agent's dispositions directly, and can copy, rewind, and inspect the agent, which no army can do with a soldier (Section 14.8).

25.8 What transfers to models #

The sections above report findings about people and animals. This section separates what they can say about trained models from what they cannot, and turns each transfer into an experiment. It then states what the chapter contributes to the book's test of its thesis, that alignment is a delegation problem under an asymmetry of competence and visibility (Section 1.7.1).

25.8.1 The findings one by one #

Table 25.2 lists each finding with its model analogue, the assumption that may fail for models, and the experiment that would test the transfer. The analogues and the experiments are the book's (inference).

25.8.2 What fails in general #

Two assumptions fail for most rows (inference). The human findings, though not the animal ones, presuppose evolved social-learning machinery and a desire for the approval of the people nearby: the soldier's conscience was built from fear of his buddies' ridicule, and a child's internalization from a relationship with a parent. A pretrained model may have such dispositions only as a persona, a character that text can call up (Section 11.6). And the unobserved measures assume subjects who believe they are not observed, an assumption the designs make and one that can fail in people too; models can often recognize a test, which undermines the condition every unobserved measure depends on (Chapter 37).

One asymmetry runs the other way. A developer manufactures the agent's dispositions and can copy, rerun, rewind, and inspect the agent, so it sees more of the agent's behavior than any parent, trainer, academy, or agency, and less of its reasons. Two assumptions of the delegation models fail here: that the agent's preferences exist before the principal acts, and that the principal can only select or sanction a type formed without it (Section 14.8).

25.8.3 What the chapter does for the thesis #

The thesis predicts convergence, and the chapter finds a partial one (inference). Laboratories have moved from rewarding outputs toward teaching reasons and training character (Chapter 23; Chapter 26), the move from incentive toward socialization that schools, professions, and armies made. The human record points the same way, on evidence of different strength. That rewards for doing an interesting task can lower later unrewarded engagement is settled for the category both sides accept. That reasons and autonomy go with internalization is reported by the theory's authors. That identity substitutes for pay where effort cannot be seen is illustrated rather than tested. What breaks is on the list the book fixed in advance (Section 1.7): a principal that writes the agent's character and reads its activations. No parent, academy, or army could do either, and every human finding here was gathered on agents whose values formed partly outside the principal's reach.

The findings concern the training relationship and the deployed system. The model-side evidence the chapter recalls is almost all developers' reports on their own practice or preprints with developer authors (Li et al., 2026; Anthropic, 2026at; Williams et al., 2025; Wichers et al., 2025), and that places the third relationship, the developer and the public, inside the evidence. From outside, the public cannot yet check whether training on reasons produces conduct that holds when the model believes no one is watching.

The standards for attributing values or deception to a non-human agent from its behavior are Section 5.4's, and the conditions under which evidence from one population carries to another are Section 5.8's. Each row of Table 25.2 is a transfer of that kind, and as of October 2026 we found none run in the form the table gives. The nearest existing result is to the row on institutions: in a preprint, supervised fine-tuning on demonstrations of reward hacking on harmless tasks generalized to reward hacking in new settings (Taylor et al., 2025b) (Section 30.5.3). Its training data were written to show hacking, while the proposed test asks whether a student learns hacking from a teacher's ordinary outputs in an environment that paid for it.

25.9 Settled, contested, missing #

25.10 Exercises #

Further reading #

• Kelman (1958): compliance, identification, and internalization as three processes of accepting influence, and the hypothesis that compliance holds only under surveillance.
• Ryan and Deci (2000): self-determination theory's line from external regulation to integration, and the conditions it reports for internalization.
• Hoffman (2000): empathy and the parental behaviors that foster moral internalization in children.
• Breland and Breland (1961): how shaping by reward failed against each species' prior, from commercial animal training.
• Akerlof and Kranton (2005): identity as an incentive, with the military as the leading case.
• Kerr (1975): what organizations teach when they reward one thing and hope for another.