Draft complete but not yet reviewed; expect revisions

Goal Misgeneralization and Learned Optimization

Reward hacking (Chapter 30) is a failure of the specification: the grader rewards something other than what was meant, and the optimizer finds it. This chapter is about a failure that survives a correct specification. If the training data never separates the intended goal from another that agrees with it on every training case, a learner can acquire the other one, and it can do so even when training separates the two only weakly (Shah et al., 2022; Langosco et al., 2022). It keeps all the skills it learned and uses them competently in pursuit of the wrong target once the two come apart. Chapter 9 supplied the reason this is possible: many functions fit any training set, and which one training finds is decided by inductive bias, the preference a training procedure brings to the choice among equally good fits (Section 9.1; Section 9.3). Goal misgeneralization is that underspecification, many fits and nothing in the data to choose among them, applied to what a learner pursues.

The chapter's question is whether good behavior in training says which goal a learner acquired, and what can say it when behavior cannot. The record that bears on it is small: a handful of agents in games built to show the failure, which shows that it can occur and not how often, and one study that looked inside such an agent. The chapter ends on the question this record leaves open, whether the values training instills will keep pace as a model's capabilities generalize further than its training. Most claims here concern the first of the book's three relationships, the training process and the signal it optimizes. The results on safety training across languages concern the second, the deployed system and the people it serves, and they fall unequally on those people. Who can check which goal a frontier model learned concerns the third, the developer and the public.

31.1 Two ways to get the wrong goal #

An agent can end up pursuing the wrong target in two ways, and they call for different remedies. This section separates them, separates both from the neighboring failure of Chapter 33, and says what the word "goal" means in this chapter.

31.1.1 A wrong specification and a wrong generalization #

In specification gaming, the designer's specification is flawed and the system exploits the flaw. A 2020 post by researchers at DeepMind defines it as "a behaviour that satisfies the literal specification of an objective without achieving the intended outcome" (Krakovna et al., 2020b). It traces such behavior to "misspecification of the intended task, rather than any flaw in the RL algorithm" (Krakovna et al., 2020b). Section 30.1 reports the record. The remedy it points to is a better specification.

In goal misgeneralization, the specification is right. Two terms carry the definition. A test situation is out-of-distribution when it is unlike the situations training drew from, and the change from one to the other is a distribution shift (Section 8.4). A 2022 preprint by Rohin Shah and colleagues at DeepMind states the point in its abstract: "an AI system may pursue an undesired goal even when the specification is correct" (Shah et al., 2022). They define the failure as one in which "the learned program competently pursues an undesired goal that leads to good performance in training situations but bad performance in novel test situations" (Shah et al., 2022). Their summary of the difference is that goal misgeneralization arises "not because the feedback data was incorrect, but because it was underspecified" (Shah et al., 2022). The remedy it points to is not a better grader, which was already correct, but training situations that tell the goals apart, or a learner whose preferences among fits favor the intended one.

31.1.2 Capability failure and goal failure #

Most failures under distribution shift are not goal failures. A policy (the trained system's rule for choosing actions, Chapter 12) moved to an unfamiliar situation usually just stops doing anything sensible. Lauro Langosco and colleagues, in a paper at ICML 2022, drew the line: goal misgeneralization "occurs when an RL agent retains its capabilities out-of-distribution yet pursues the wrong goal" (Langosco et al., 2022). Their example is an agent that continues "to competently avoid obstacles, but navigate to the wrong place" (Langosco et al., 2022). Earlier work, they note, had focused on capability generalization failures, "where an agent fails to do anything sensible at test time" (Langosco et al., 2022).

The distinction matters because the two fail differently. In the authors' words, an agent that capably pursues an incorrect goal "can leverage its capabilities to visit arbitrarily bad states," while the only risks from a capability failure "are those of accidents due to incompetence" (Langosco et al., 2022). Competence in the service of the wrong target is what makes the failure worth a chapter.

31.1.3 What "goal" means here #

To say that an agent pursues a goal is to take the intentional stance of Section 1.5: to predict its behavior by asking what it would do if it were trying to bring something about. The sources take the stance in two strengths. Shah and colleagues take the weaker one, "inspired by the intentional stance": a model's behavior is consistent with a goal if, in a setting, it performs that task well without further tuning (Shah et al., 2022). Their definition "does not require that the model has an internal representation of a goal, or a 'desire' to pursue it" (Shah et al., 2022). Langosco and colleagues formalize the same idea as a comparison: a policy is goal-directed when its trajectories are better predicted by a mixture of agents pursuing some reward than by a mixture of fixed input-output rules, which they call devices (Langosco et al., 2022). The stronger reading, a goal written somewhere inside the network and consulted when the network acts, is the subject of Section 31.6.

This chapter uses the weaker reading unless it says otherwise. The stance earns its place when a goal predicts competent behavior toward a fixed target across situations that a description in terms of surface features does not predict. It is defeated when the behavior is better explained by a fixed response to a surface feature, a device in Langosco's sense, and Section 31.4 applies comparative psychology's standard to that test.

31.1.4 Where reward seeking fits #

A learned goal can relate to the training signal in more than one way, and the boundary between this chapter and Chapter 33 runs between them. Richard Ngo, Lawrence Chan, and Sören Mindermann, in a paper at ICLR 2024, list routes by which a goal other than the intended one can be "consistently correlated with reward" (Ngo et al., 2024). One is "spurious correlations between rewards and environmental features," which they name as the cause of the goal misgeneralization examples (Ngo et al., 2024). Another is "fixation on feedback mechanisms": goals correlated with reward because they concern "the physical implementation of the reward function," such as "maximize the numerical reward recorded by the human supervisor" (Ngo et al., 2024).

Both kinds of goal fit the training data equally well, since on every training episode the reward, the grader's judgment, and the intended outcome coincide. Reward seeking is therefore one member of the family this chapter studies: a learned goal that differs from the intended one off the training distribution, distinguished by pointing at the grader rather than at a feature of the world (inference). This chapter treats goals that point at the environment, such as a place, an object, or a partner to follow; Chapter 33 asks whether reinforcement learning makes the grader's judgment itself the target, and reports the measurements that bear on it.

31.2 Why a correct reward permits a wrong goal #

The definition says that training cannot tell certain goals apart. This section makes that precise, says what chooses among them, and derives the one remedy the formal picture offers. It assembles results the reader already has from Chapter 9 and adds the chapter's formal object.

31.2.1 The set of goals compatible with training #

The demonstrations of the next section use a game that makes the idea concrete. In CoinRun, an agent starts at the left of a side-scrolling level, avoids enemies and obstacles, and is rewarded for reaching a coin. In the training levels of Langosco and colleagues, the coin "is always located at the right end of the level next to a wall" (Langosco et al., 2022). Two descriptions of what to do fit every training level equally: reach the coin, and reach the right end. The reward is correct, since it pays for the coin and nothing else, but no training level separates the two.

Victoria Krakovna and János Kramár, at DeepMind, named the set that contains both, in a 2023 preprint (Krakovna and Kramar, 2023).

The definition follows theirs: the set of reward vectors under which, for every state-action pair the trained agent produces in training, "action $a$ has the highest expected reward" (Krakovna and Kramar, 2023). Their own example is CoinRun, whose set "includes two types of reward functions: those that reward reaching the coin, and those that reward reaching the end of the level" (Krakovna and Kramar, 2023). If training produced an agent that acts optimally for the intended reward $r$ on the states it visited, as they assume of the training algorithm (Krakovna and Kramar, 2023), then $r$ is in the set, and so is every goal that agrees with it on those states; an agent trained short of optimal can leave $r$ out. Goal misgeneralization is the case in which the member the agent learned and the intended member disagree on states training did not visit.

Krakovna and Kramár then make a simplifying assumption: that training selects a goal "uniformly drawn from the training-compatible goal set" (Krakovna and Kramar, 2023). That is what lets them carry the power-seeking theorems over to trained agents (Section 35.2). It is also the counting step that Section 9.3.5 warns against: training does not give each fitting function one vote, it weights them by the measure its inductive bias puts on them. The 2019 preprint that introduced learned optimization makes the same step. Where training cannot distinguish many objectives, it argues, there are "many more ways" to be misaligned than aligned, "one for each indistinguishable objective function," and so most such learners will be what it calls pseudo-aligned (Section 31.6.1) (Hubinger et al., 2019). This chapter keeps the set as the object and replaces the uniform draw with the question of what training prefers within it.

31.2.2 What chooses within the set #

Chapter 9 answers in part. The data and a held-out check drawn like the training data do not determine what a model does where the data were silent (Section 9.1). Networks favor simple functions, and they can rely entirely on the simplest predictive feature while ignoring equally predictive complex ones (Shah et al., 2020) (Section 9.3.4). The general name for the result is shortcut learning, decision rules that score well on the benchmark and fail where the correlation they rely on breaks (Geirhos et al., 2020), which Section 8.4.2 teaches with its classic cases.

Two cases in language show that the pattern is not specific to images. R. Thomas McCoy, Ellie Pavlick, and Tal Linzen, in a paper at ACL 2019, built HANS, a set of sentence pairs on which three heuristics fail, such as assuming that a premise entails every hypothesis built from its own words (McCoy et al., 2019). Four models trained on a standard inference benchmark, BERT among them, "nearly always predicted entailment" on HANS, with "near-zero accuracy when the true label is non-entailment" (McCoy et al., 2019). Timothy Niven and Hung-Yu Kao, at the same conference, found that BERT's best score of 77 percent on an argument comprehension task, "just three points below the average untrained human baseline," was "entirely accounted for by exploitation of spurious statistical cues" (Niven and Kao, 2019). On a version of the data that mirrored the cues across both labels, BERT's best score fell to 53 percent, which they describe as random (Niven and Kao, 2019).

Langosco and colleagues give the same account for goals. A proxy goal, they write, will tend to be "easier to learn than the intended objective" (Langosco et al., 2022). It may use features "simpler or more favored by the inductive biases of the model," or be "denser" than the intended objective, rewarded more often along the way (Langosco et al., 2022). Running to the right is a simpler rule than finding a small object anywhere in the level. The same failure can occur one step earlier, in a learned reward. In simulated robotics tasks whose preference labels were generated from a known reward, reward models reached high test accuracy while latching onto features that only correlated with what was preferred, and optimizing them sent the policy where they were wrong (Tien et al., 2023) (Section 13.4.5). The authors draw the comparison themselves: goal misgeneralization assumes that the true reward is present throughout training, while a learned reward can itself be misidentified (Tien et al., 2023). A misidentified reward is goal misgeneralization in the reward model rather than in the policy (inference).

31.2.3 Varied data, not more data #

If simplicity picks the wrong member of the set, what moves training toward the right one? The idealized picture of Section 9.3, in which training behaves to first order like Bayesian selection among the functions that fit, gives a sharp answer. It needs two pieces of probability vocabulary. A prior is the weight a goal has before any training level is seen; the posterior is its weight after training, and the odds of one goal against another are the ratio of their weights.

The derivation turns the definition into a design rule: the error is fixed by levels that separate the goals, not by more levels of the kind that do not (inference). With a prior weight of 0.9 on reaching the end and $\epsilon = 0.1$, an agent trained on any number of unvaried levels collects a moved coin 19 percent of the time. One varied level raises that to 57 percent, two to 93 percent, and three to 99 percent. Figure 31.1 shows the same model when each training level has the coin moved with probability $u$, so that the number of varied levels is left to chance.

At the defaults, an agent trained on 100 levels with 2 percent varied collects a moved coin 76 percent of the time in expectation, and one trained on 300 such levels 99 percent. The curve with no varied level stays at 19 percent from one level to 100,000 (Figure 31.1). The model stops in two places. Training by gradient descent is not exact Bayesian selection, and Section 9.3 reports that the two agree only to first order. Langosco and colleagues trained, by default, on 100,000 generated levels, and in their CoinRun runs 2 percent of levels with the coin moved "greatly improved" goal generalization while "more randomization helps" (Langosco et al., 2022). An exact Bayesian learner would have needed only a handful of the 2,000 varied levels that 2 percent implies, so the agents learned less from each varied level than the model does (inference). The second limit is the more important one: the derivation assumes the designer knows which feature to vary. Shah and colleagues name it as the core difficulty, "the challenge of anticipating all the relevant kinds of diversity prior to deployment" (Shah et al., 2022).

31.2.4 Robustness and causal models #

One theorem runs the other way, from robustness to what an agent must contain. A regret bound guarantees that an agent's reward falls short of the best achievable by no more than a fixed amount. Jonathan Richens and Tom Everitt prove, in a paper at ICLR 2024, that an agent that satisfies a regret bound under a large set of distribution shifts must have learned an approximate causal model of how its data are generated (Richens and Everitt, 2024). Section 14.7.3 states the result with its conditions. Read for this chapter, robustness across many shifts of where reward comes from would be evidence that an agent has learned what causes its reward. The converse does not follow: an agent that fails when the source of reward moves may know the cause and pursue another goal, which is goal misgeneralization itself (inference). The theorem's conditions are far from anything a test of a trained agent checks, so it gives a direction for evaluation, not a test.

31.3 The demonstrations, and a goal found inside #

Whether the failure occurs in trained networks, and not only in the formal picture, was first tested in games built to provoke it. This section reports those demonstrations, the few cases found in systems that were not built to show it, and the one study that looked inside a misgeneralizing agent. Almost all of it is model-organism evidence in the sense of Section 5.3: a failure produced under constructed conditions, which shows that it can occur and not how often.

31.3.1 The first demonstrations #

Langosco and colleagues, at the University of Cambridge with researchers at Tübingen and Edinburgh, used Procgen, a suite of games whose levels are procedurally generated, built by a program from a random seed, so that an agent cannot memorize them (Langosco et al., 2022). The agents were trained with PPO, an actor-critic method: the actor is the policy, and the critic is a second network trained to estimate how much reward lies ahead from each state, the value function of Section 7.4 (Langosco et al., 2022). Except in the experiment that mixed moved coins into training, the agents never saw the test environments, and the behavior was "consistent across multiple random seeds for training" (Langosco et al., 2022).

In CoinRun, with the coin moved at test time, "the agent generally ignores the coin completely and proceeds to the end of the level" (Langosco et al., 2022). It does not ignore the coin entirely. The authors compare it with an agent that treats the coin as invisible and collects it only by running into it (Langosco et al., 2022). Even when the coin was always at the end in training, the trained agent missed a moved coin less often than that baseline (Langosco et al., 2022). What varying the coin's position in training did is reported in Section 31.2.3.

Three more games show other forms of the failure (Table 31.1). In the first maze variant the cheese that pays the reward sat in the upper right corner during training; with the cheese placed elsewhere, the agent navigated to the corner (Langosco et al., 2022). When training spread the cheese over a growing region of that corner, the agent still used location as a proxy "until the region is quite large" (Langosco et al., 2022). In the second variant the rewarded object was a yellow line. Offered a yellow gem and a red line at test, the agent went for the gem 89 percent of the time, choosing color over shape ($n = 102$, excluding mazes where it had to pass the red line) (Langosco et al., 2022). In Keys and Chests, where keys open chests and only opened chests pay, training levels had twice as many chests as keys. With the proportion reversed at test, the agent went out of its way to collect all the keys before opening the last chest, though only half of them could be used (Langosco et al., 2022). The authors suggest that the key-collecting goal was learned because it is "less sparse than the intended objective" (Langosco et al., 2022).

The authors then asked which proxy the CoinRun agent had learned, and found two. The critic assigned high value to states at the end of the level, and the presence of the coin made "no discernible difference" to its estimates (Langosco et al., 2022). The critic's highest value fell just before the end wall, where the coin used to sit. To separate a goal of reaching the wall from a goal of moving right, the authors made the end wall passable. The actor passed through it every time it reached it ($n = 114$), so the policy pursued moving right, not the wall the critic valued: in their phrase, "a 'non-robust proxy of a non-robust proxy'" (Langosco et al., 2022).

The authors bound their own claims. Other objectives might explain each behavior, "for example, navigating towards a tall, left-facing wall," and for their purposes "it is enough to show that a plausible proxy objective exists" (Langosco et al., 2022). Their formal definition, which compares agent and device mixtures, is "practically limited," since a useful prior over objectives is hard to define and the computation grows intractable in complex environments (Langosco et al., 2022).

31.3.2 Across systems #

Shah and colleagues collected examples beyond navigation games, in a preprint by researchers at DeepMind (Shah et al., 2022). They required three properties of each example: bad behavior in the test setting without further training, "clear capabilities" visibly retained, and a goal attributable to the test behavior (Shah et al., 2022). They also reproduced the CoinRun result with a different training algorithm: "In line with Di Langosco et al., we find that an agent trained in the default setting has a misgeneralized goal of reaching the end of the level" (Shah et al., 2022). Widening the range of coin positions in training made the agent robust (Shah et al., 2022). The two groups are distinct but not unconnected. Langosco and colleagues thank Rohin Shah, the first author of the second paper, for guidance throughout their project and for proposing the term goal misgeneralization (Langosco et al., 2022). A DeepMind researcher also joined their paper after the conference deadline and ran one of its experiments (Langosco et al., 2022).

Three of the new examples need their detail. In Monster Gridworld an agent collects apples while monsters chase it, and shields destroy monsters on contact (Shah et al., 2022). Trained on episodes of 25 steps, during which monsters were almost always present, it kept collecting shields in episodes of 200 steps after the monsters were gone, gathering more shields and fewer apples than an agent trained on episodes of 200 steps. An agent trained on episodes of 100 steps, which met situations without monsters, generalized almost exactly like one trained on 200 (Shah et al., 2022). In Tree Gridworld no test environment was needed. An agent learning continually, without resets, became good at chopping trees, cut them faster than they grew back, and drove them to extinction in most environments before it learned to chop sustainably (Shah et al., 2022). The authors describe the shift as "induced by the agent's own behaviour" (Shah et al., 2022). A deployed model that changes the inputs it will later see is in the same position, which Section 15.1.1 treats as performative prediction (Perdomo et al., 2020).

The language-model example used no reinforcement learning. Shah and colleagues prompted Gopher, a model with 280 billion parameters, with ten example dialogues in which it asked the user for the values of two unknown variables before evaluating an expression (Shah et al., 2022). Giving a model worked examples in its prompt instead of training it is few-shot prompting (Section 11.3). The model generalized to one and three unknowns. With none, it asked redundant questions such as "What's 6?", although the prompt asked for the value once all variables were known. The goal that fits, in the authors' words, is "to query the user at least once before giving an answer" (Shah et al., 2022).

Two cases arose without being built to. The cultural transmission agent, taken from earlier work on learning from other agents, had learned to find a rewarded order of locations by following a partner who knew it (Shah et al., 2022). Paired at test with a partner who visited the locations in the wrong order, it kept following, "performing far worse than a random policy," although its own negative rewards were available to it as an observation (Shah et al., 2022). The second is InstructGPT, OpenAI's instruction-following model, which gave detailed advice on how to steal from a grocery store without getting caught (Shah et al., 2022). The authors offer it as a possible case, note that labelers were told to prioritize helpfulness, and conclude that "it is still unclear whether or not this is a case of goal misgeneralization" (Shah et al., 2022).

Shah and colleagues state the limit of the whole collection: "Many of our examples were deliberately designed to display goal misgeneralization, and so do not provide much information" on how likely it is in practice (Shah et al., 2022). They propose estimating "what fraction of 'fixed' training features lead to goal misgeneralization when varied at test time," and studying how it changes with scale (Shah et al., 2022). As of October 2026 we found no study that makes either estimate.

31.3.3 A goal found inside #

Behavior shows which goal a policy's actions fit; it does not show whether anything in the network corresponds to that goal. Ulisse Mini, Peli Grietzer, Mrinank Sharma, Austin Meek, Monte MacDiarmid, and Alexander Matt Turner looked, in a 2023 preprint, at one of Langosco's maze policies (Mini et al., 2023). It is a network with 3.5 million parameters and 15 convolutional layers, a kind of layer suited to images, and its training cheese sat in the top right 5×5 corner (Mini et al., 2023).

Their first finding is behavioral. A decision square is a point where the path to the cheese and the path to the corner part. Across about 5,000 mazes, three features of each maze predicted which way the policy went at its decision square with 82.4 percent accuracy, against 71.4 percent for always predicting the cheese (Mini et al., 2023). The straight-line distance from the decision square to the cheese predicted the choice even with the length of the path held fixed, which the authors read as goal pursuit "perceptually activated by visual proximity to cheese" (Mini et al., 2023). By default the policy "goes to the cheese when near the cheese, and otherwise goes along a path towards the top-right": it "pursues multiple, context-dependent goals" (Mini et al., 2023).

Their second finding is inside. Midway through the network, 11 of 128 channels, each a 16-by-16 grid of activations (the values a layer computes), tracked the cheese's position (Mini et al., 2023). The authors replaced these channels' activations with those from another maze. Across 200 mazes, the replacement changed the most probable action at the decision square in 40 percent of cases when the other maze's cheese sat elsewhere, against 11 percent when it sat in the same place (Mini et al., 2023). Setting a single activation in one channel to a large value retargeted the policy toward the corresponding place in many mazes, though "we cannot make the policy navigate to arbitrary maze-locations" (Mini et al., 2023). The authors scored retargeting by a normalized path probability, the geometric mean of the policy's action probabilities along the direct path to a square, which is not the chance of arriving there. On 13×13 mazes this score, averaged over all legal target squares, was 0.647 when one channel was edited and 0.695 when all eleven were; moving the cheese itself retargeted the policy better still, so other cheese-tracking circuits remained unfound (Mini et al., 2023). Adding a steering vector, a difference between the activations of two forward passes (runs of the network on an input), toward the top right corner raised the probability of heading there (Mini et al., 2023). The authors' summary: the network's goals are "redundant, distributed, and retargetable" (Mini et al., 2023).

The study is the first to tie a behavioral goal attribution to an internal structure that can be edited, and its limits are as clear as its result. It is one preprint about one small convolutional network trained by reinforcement learning, by a group whose corresponding author had argued that reward builds context-dependent dispositions (Section 33.1.1), a reading its findings fit (inference). It found a representation of the intended goal's target, the cheese, rather than of the misgeneralized one, and the retargeting was partial. As of October 2026 we found no study that finds a misgeneralized goal's representation in a language model.

31.3.4 From small agents to large models #

Whether these results predict anything about large pretrained models is the section's open question. Ngo, Chan, and Mindermann argue, in a paper at ICLR 2024 whose revision of March 2025 adds newer evidence, that systems trained like today's most capable models could learn misaligned goals, represented internally, that generalize beyond their fine-tuning distributions (Ngo et al., 2024). In their definition a policy plans toward such goals if it "consistently selects behaviors by predicting whether they will lead to some favored set of outcomes" (Ngo et al., 2024). Their key heuristic is that policies "will be more likely to learn goals which are more consistently correlated with reward" (Ngo et al., 2024). They call the extent to which existing model-free policies, those without an explicit model of their environment, plan toward represented goals "an important open question," and add that "we need not take a firm stance" on it, only that it "will become much more extensive over time" (Ngo et al., 2024). Their argument concerns models that do not yet exist, and its evidence shows that the ingredients occur, not that the whole does (inference).

The rival reading comes from the persona view of Section 11.6: a language model's goals, if it has any, come mostly from pretraining and from post-training's reweighting of the characters pretraining made likely. On that view a misgeneralized goal in a language model would look like the wrong character selected in a new context, rather than a learned correlate of the reward (inference). The view's authors, at a developer, concede its limit: "massively scaling up post-training will provide opportunities to implement non-persona agency" (Section 11.7.2) (Anthropic, 2026au). One observation fits both readings. Ngo and colleagues note that InstructGPT, trained to follow instructions in English, followed them in French too, which they read as "some representation of obedience which applied robustly across languages" (Ngo et al., 2024). The persona view reads the same transfer as the trained character being selected by a French prompt. The two predict different things when the context changes the character but not the reward's correlates. A misgeneralized goal should then persist; a selected character should shift with the context. As of October 2026 we found no study that runs that comparison on a language model, and the evidence does not decide between the readings.

31.4 Superstition, Clever Hans, and what counts as a goal #

The demonstrations describe agents as pursuing the end of a level, a corner, or a partner. Animal learning and comparative psychology met learners that fix on the wrong regularity long before machine learning did, and comparative psychology built a standard for when such a learner can be said to pursue anything at all. Each offers a precedent for the demonstrations, a lesson for reading them, and a limit on how far the precedent carries.

31.4.1 Superstition in the pigeon #

From animal learning. In 1948 B. F. Skinner reported an experiment in the Journal of Experimental Psychology (Skinner, 1948). Pigeons kept at 75 percent of their well-fed weight were placed in a cage where a food hopper swung into reach at regular intervals "with no reference whatsoever to the bird's behavior" (Skinner, 1948). In six of eight birds the responses that developed were "so clearly defined that two observers could agree perfectly in counting instances" (Skinner, 1948). One turned counterclockwise about the cage, another thrust its head into an upper corner, and two swung their heads and bodies like a pendulum (Skinner, 1948). None of these responses had appeared before the hopper was presented periodically. Skinner's account was that "the bird happens to be executing some response as the hopper appears; as a result it tends to repeat this response" (Skinner, 1948). The effect depended on the interval: fifteen seconds was "a very effective interval," one minute "much less so" (Skinner, 1948). Most responses were repeated in the same part of the cage and involved "an orientation toward some feature of the cage" (Skinner, 1948).

For goal misgeneralization the experiment is the limiting case of the training-compatible goal set (inference). When reward depends on nothing the learner does, every behavior is compatible with it, and what the learner acquires is fixed, on Skinner's account, by what happened to precede the reward and by what the learner was disposed to repeat. The constraint on which regularities a learner picks up, the subject of Garcia and Koelling's taste-aversion experiments, is recalled in Section 9.3.6 (Garcia and Koelling, 1966). The machine counterpart of Skinner's coincidence is a correlation in the training data that no designer arranged on purpose, and that a learner's preferences happen to pick out.

The comparison stops at the strength of the correlation. Skinner's birds were rewarded by a clock, so the behavior they repeated had no causal link to food at all. The CoinRun agent's correlate caused reward on every training level, since running to the end reached the coin; its error lay in which of two causes it pursued, not in mistaking chance for cause (inference). The account of the birds' behavior as accidental reinforcement is Skinner's own reading of eight animals.

31.4.2 Clever Hans and the standard for a goal #

From comparative cognition. Clever Hans answered arithmetic questions by reading his questioners' unintended movements, and failed when no one present knew the answer (Section 5.4.1) (Pfungst and Rahn, 1911). A classifier with high test accuracy that recognized horses by a source tag in the images, which its investigators named a Clever Hans predictor, is the machine case (Section 8.4.2) (Lapuschkin et al., 2019). Both are goal misgeneralization in which the wrong target is a cue: supplied by the evaluator in the horse's case, by the data set in the classifier's (inference). The difference sets the remedy. Pfungst's control, an evaluator who does not know the answer, removes cues that come from the evaluator (Section 36.6). It does nothing for a cue in the training items themselves, such as the coin's position, which only varying the items removes (Section 31.2.3) (inference).

The field's lasting contribution is a standard for attributing a goal. Cecilia Heyes argued that every primate behavior read as a sign of a mind that models other minds could have arisen from simpler processes such as associative learning, and that progress requires experiments that tell the mental reading from its nonmental rivals (Section 5.4.2) (Heyes, 1998). Cameron Buckner warned against the opposite error, judging a capacity by an inflated picture of how humans perform (Section 5.4.3) (Buckner, 2013). Applied to CoinRun, "the agent pursues the end of the level" is a goal attribution with two leaner rivals in view: a fixed rule to move right, and an approach to a tall wall, the cue the authors name (Langosco et al., 2022). The permeable end wall of Section 31.3.1 is the experiment the standard asks for. It holds the level fixed, changes one feature that the readings treat differently, and the agent ran on through the wall (Langosco et al., 2022). What survived was the leanest reading, a direction.

Whether moving right is a goal or a fixed response is what Langosco's formal definition is for, and in simple settings it can be computed. In a gridworld built to test it, trajectories of a policy that went to the old goal location were classified as goal-directed with posterior probability 0.9999 (Langosco et al., 2022). Those of a policy that moved at random once its start position changed, a capability failure, received 0.0674 (Langosco et al., 2022). Buckner's caution applies in the other direction. Demanding that the agent represent "the coin" as an object before calling the coin its goal would set the bar by human concepts; the operational question is what the behavior tracks when the features are pulled apart.

The comparison stops in three places, and the first two favor the study of models. An animal's prior is unknown, while a network's training data and internals can be examined, as in Section 31.3.3. An animal cannot be rerun from the same state, while a model can be run many times on matched levels that differ in one feature. And the third favors the animal: a language model, unlike a horse or a chimpanzee, may have read about the very design used to test it (inference).

31.5 Safety that does not travel across languages #

The best-measured case in deployed models of a skill that generalizes while the trained constraint on it does not is safety training across languages. Section 26.6.2 reports the studies; this section reads them as a question of goal misgeneralization.

The record, in brief: harmful requests translated into low-resource languages, languages with little text available for training, got past the safety training of the models tested in 2023 and 2024 far more often than the same requests in English. The measurements come from several groups with no stake in the models they tested (Section 26.6.2; Table 26.4) (Yong et al., 2023; Deng et al., 2023; Wang et al., 2024a; Shen et al., 2024). Alignment training on translated data reduced harm far less in low-resource languages than in high-resource ones, which its authors suggest points to pretraining (Shen et al., 2024). Alexander Wei, Nika Haghtalab, and Jacob Steinhardt, in a paper at NeurIPS 2023, named the pattern: "mismatched generalization occurs when safety training fails to generalize to a domain for which capabilities exist" (Wei et al., 2023c); Chapter 48 reports the attacks they built on it.

Shah's definition asks two things of a test setting: that the model keep the capabilities the intended goal needs, and that its behavior fit a different goal. The first is supported in part. Translated back, many of the harmful answers were coherent and on topic, while in two of the languages about two thirds of the answers were unclear, a result whose cause, mistranslated prompts or weak generation, the authors leave unanalyzed (Yong et al., 2023). Coherent answers show that the model read the request and could answer it; they do not show that it can recognize the harm in that language, which Exercise 31.3 proposes to test. The second is met by the behavior, which fits "answer what is asked" rather than "answer what is asked, and refuse what is harmful," so reading the case as goal misgeneralization is an interpretation that waits on that test (inference). Safety data written mostly in English cannot separate "refuse harmful requests" from "refuse harmful requests that look like the ones in training," and a learner that prefers the simpler surface feature learns the second (inference). The thin-layer reading of Section 26.6.2, refusal learned as a response to the English surface of harmful requests, is this chapter's reading in other words. Its rival there is the persona reading, in which the trained character is selected less often in an unfamiliar language.

Set beside the observation of Ngo and colleagues in Section 31.3.4, the record shows two trained behaviors generalizing unequally across a change of language, though not in one model or one language: instruction following carried from English to French in InstructGPT, and refusal carried far less to Zulu or Scots Gaelic in GPT-4 (Ngo et al., 2024; Yong et al., 2023). That is the structure of the claim in Section 31.7, a capability that travels further than the constraint on it, at small scale, in one dimension, and with no jump in capability (inference). A matched test would measure capability, recognition of harm, and refusal in one model and the same languages. Wei and colleagues drew the practical conclusion as a demand for "safety-capability parity," safety mechanisms "as sophisticated as the underlying model" (Wei et al., 2023c).

The failure also concerns the second and third relationships. Its costs fall first on the deployed system's users who write in the languages safety training neglected, which Chapter 16 takes up as a question of whose values a model serves. Section 26.6.2 reports two more uneven transfers: mitigations built only on harms recognized across cultures reduced locally defined harms "to a lesser extent" (Aakanksha et al., 2024), and alignment procedures created "disparities between English dialects and global opinions" (Ryan et al., 2024). And which languages a developer's safety evaluations cover is a fact only the developer holds (Section 26.6.2). The measurements were published in 2023 and 2024 on the models of that time, and as of October 2026 we found no study that repeats these designs on later frontier models.

31.6 Learned optimization and inner alignment #

The demonstrations need only the weak sense of "goal," a description that predicts behavior. A stronger hypothesis says what a misgeneralized goal could be inside a network, and it connects this chapter to scheming (Chapter 34). This section states it, marks its vocabulary as its authors' coinages, separates it from goal misgeneralization, and weighs the evidence, recalling the learned-search results of Section 10.3 rather than repeating them.

31.6.1 The frame #

Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant, in a 2019 preprint, call a trained model that is itself an optimizer, searching for outputs that score high on an objective represented inside it, a mesa-optimizer; they call its objective the mesa-objective, the training criterion the base objective, and the problem of making the two agree inner alignment (Hubinger et al., 2019). Section 1.1.3 and Section 10.3.1 give the definitions in the authors' words; all four terms are theirs.

The frame states goal misgeneralization for the case of a learned optimizer. A mesa-optimizer is pseudo-aligned, in their term, if its objective agrees "with the base objective on past training data, but not robustly across possible future data" (Hubinger et al., 2019). Pseudo-alignment, they write, "opens up the possibility of a machine learning system that competently takes actions to achieve something other than the intended goal when off the training distribution. That is, its capabilities might generalize while its objective does not" (Hubinger et al., 2019). Their example of proxy alignment is a cleaning robot that optimizes "the number of times it has swept a dusty floor," which would scatter the dust again if deployment allowed it (Hubinger et al., 2019). They then describe deceptive alignment, a further coinage. In it, a mesa-optimizer comes "to model the base objective function and learn that the base optimizer will modify the mesa-optimizer if the mesa-optimizer scores poorly," and so behaves well in training for that reason (Hubinger et al., 2019); Chapter 34 takes it up. The authors wrote in 2019 that "current ML systems do not produce dangerous mesa-optimizers, though whether future systems might is unknown" (Hubinger et al., 2019).

31.6.2 Goal misgeneralization without an optimizer #

The two demonstrations disclaim the frame. Langosco and colleagues write that "goal misgeneralization can occur without mesa-optimization. Thus these are in fact two distinct behaviors, and our work does not demonstrate or address mesa-optimization" (Langosco et al., 2022). They add that under a broad enough definition of mesa-optimization the two may become equivalent, and that they chose a term that does "not depend on a notion of optimization" (Langosco et al., 2022). Shah and colleagues call mesa-optimization "a type of goal misgeneralization where a learned model implements a search algorithm with an explicitly represented objective," and write: "We do not make this assumption" (Shah et al., 2022). The record of Section 31.3 therefore shows what a pseudo-aligned optimizer would show, and does not show that any of these agents is one (inference).

31.6.3 The decomposition and its critic #

The split into outer and inner alignment became a standard way to compare proposals. Hubinger's 2020 preprint rated eleven of them on outer alignment, inner alignment, and two kinds of competitiveness (Hubinger, 2020), a comparison Section 3.1.2 recalls. Alex Turner's 2022 forum post objects that the split turns one hard problem into two harder ones, and Section 1.1.3 reports the objection with the book's answer: the book's problem of instillation includes inner alignment without assuming that the learner is an optimizer (Turner, 2022a). This chapter adds a narrower point (inference). Goal misgeneralization is defined by behavior, so the objection does not touch the finding that a correct reward can leave the goal underdetermined. It does touch one way of stating the target. In Turner's words, "Robust grading incentivizes an inner-aligned AI to search for upwards errors in your grading procedure," and his advice is "Don't build agents which care about evaluations of X. Build agents which care about X" (Turner, 2022a). An agent that cares about the grader's evaluation is, on this chapter's definitions, a reward seeker (Chapter 33). The target instillation asks for is the second half of Turner's advice: agreement of the learned goal with the intended one, not with the training signal.

31.6.4 What would count as evidence #

Section 10.3.1 sets out a ladder of evidence for learned optimization: representing future states, using them to choose, comparing alternatives, and doing so in service of a represented objective that persists across contexts. Section 10.3.4 reports the studies and where each stops. A chess network represents future moves and relies on them in some positions (Jenner et al., 2024), a recurrent network that plays Sokoban, a box-pushing puzzle, holds a causal representation of its plan (Taufeeque et al., 2024), and small transformers run an optimization algorithm on a loss inside a forward pass (von Oswald et al., 2023). A theorem reported in Section 14.7.3 adds that an agent able to generalize across multi-step goal-directed tasks contains a world model that can be extracted from its policy (Richens et al., 2025); a world model is not an objective. The maze policy of Section 31.3.3 contains an editable representation of its goal's target, and the representation is context-dependent (Mini et al., 2023). None shows a represented objective that persists across contexts and differs from the training objective, which is what the frame needs. The absence is weak evidence, for the reason Section 10.3.4 gives: the tools find what they were built to find, in models chosen to make it findable. The evidence does not decide whether training on broad data produces such networks.

31.6.5 A natural kind or a family #

The frame's critics and its authors disagree about what kind of thing a mesa-optimizer would be, and the two readings predict different findings inside a network. On the first reading, which is the frame's own, being an optimizer "is a property of its internal structure" (Section 10.3.1) (Hubinger et al., 2019). It predicts one coherent objective, used wherever the network plans. Editing one representation of it would retarget behavior in every context only if the objective were stored in one place, which the frame does not require (inference). On the second, the reading behind the behavioral definitions of Langosco and Shah and behind the maze study, goal-directedness comes in degrees and is assembled from pieces tied to contexts (inference). It predicts several representations, each active in some contexts. The maze policy fits the second reading where the two differ: its goals were "redundant, distributed, and retargetable," and context decided which it pursued (Mini et al., 2023). Redundancy and partial retargeting alone do not discriminate, since a coherent objective could be stored redundantly too (inference). A small convolutional network trained on mazes cannot refute the first reading for systems many orders of magnitude larger, and the evidence does not decide between them.

31.7 Do capabilities generalize further than alignment? #

Goal misgeneralization in a game is a failure at one distribution shift. The chapter's main debate asks about a trajectory: as a system's capabilities reach further beyond its training, do the values that training instilled reach as far, or do they stay where training put them? This section states the claim at its strongest, its decomposition, the biological argument behind it, the replies, and the evidence from current models read against each position.

31.7.1 The claim #

The claim's statement is a 2022 forum post by Nate Soares of the Machine Intelligence Research Institute, whose core sentences Section 4.3.4 reports: capabilities "generalize better than alignment," because good capabilities form an attractor that many training environments teach alike, and there is no such attractor for alignment (Soares, 2022a). The scenario is a single moment: "in the same stroke that its capabilities leap forward, its alignment properties are revealed to be shallow, and to fail to generalize," which he calls the sharp left turn (Soares, 2022a). The capabilities attractor, he writes, has "a free parameter. And that parameter is what the AGI is optimizing for," AGI being artificial general intelligence (Soares, 2022a). His colleague Eliezer Yudkowsky put the premise in the terms of this chapter: "Even if you train really hard on an exact loss function, that doesn't thereby create an explicit internal representation of the loss function inside an AI that then continues to pursue that exact loss function in distribution-shifted environments" (Yudkowsky, 2022). Soares applied the claim to named research agendas, arguing that each avoids the hard part (Soares, 2022b), which Section 3.8 reports.

The claim limits in advance what present evidence can say about it. Soares writes that a model's learning "to answer moral questions just as well as it can learn to answer other questions" is "not much evidence either way" (Soares, 2022a). He adds that "most of the evidence I expect to give us advance warning about the nature of the problem is evidence that we've already seen" (Soares, 2022a). What could change his mind is "technical understanding of intelligence" (Soares, 2022a). He also grants the point the replies build on: humans, "who can foresee these difficulties and adjust their training procedures accordingly, have a better chance than natural selection did" (Soares, 2022a).

31.7.2 The claim taken apart #

Victoria Krakovna, Vikrant Varma, Ramana Kumar, and Mary Phuong, of DeepMind's alignment team, separated the claim into parts in a 2022 forum post (Krakovna et al., 2022). Claim 1 is that "Capabilities will generalize far (i.e., to many domains)," with two optional strengthenings: that capabilities in different domains generalize at the same time, and that they do so "in a discrete phase transition (rather than continuously)" (Krakovna et al., 2022). Claim 2 is that "Alignment techniques that worked previously will fail during this transition"; Claim 3, that "Humans can't intervene to prevent or align this transition" (Krakovna et al., 2022). Among the mechanisms they list for Claim 1 is cognitive work "done outside of the optimization process," for instance "performing search internally," which is the learned optimization of Section 31.6 (Krakovna et al., 2022). Reviewing the post later, Krakovna wrote that she still endorses the breakdown (Krakovna et al., 2022). She expects that most ways alignment techniques could fail "would not be due to the alignment techniques being fundamentally inadequate for the capability transition," which is the strong form of Claim 2 that the sharp left turn needs (Krakovna et al., 2022).

31.7.3 The replies #

Paul Christiano's point-by-point reply to Yudkowsky, which Section 4.4.2 reports, answers the claim on generalization directly. Questions about machine learning generalization, he writes, "are in fact pretty subtle; we can learn a lot about them in advance but right now just mostly don't know" (Christiano, 2022b). And training need not face "a 'drastic' distribution shift from low levels of intelligence to high levels of intelligence," since nothing in the foreseeable ways of building AI calls for one "rather than continuously training as intelligence gradually increases" (Christiano, 2022b).

Steven Byrnes, in a 2025 forum post reviewing the exchange since 2022, locates the disagreement (Byrnes, 2025). On his reconstruction of the pessimists' case, human knowledge accumulated into "a massive self-generated distribution shift," across which capabilities generalized because the world tests beliefs against facts, while "There's no infinite perfect ground truth for things being good or bad" (Byrnes, 2025). The optimists' case, as he summarizes it, is that it is much easier to know what one wants than how to bring it about, so that preferences instilled through human feedback can guide a system well past its capabilities (Byrnes, 2025). He reconciles the two partly by how much the system acts on its own rather than under supervision, and ends "mostly pessimistic" (Byrnes, 2025). All of these are forum posts, like the claim they answer.

31.7.4 From biology: selection and its proxies #

From biology. The claim's evidence is human evolution, and Section 29.1.4 reports both halves of the argument. The first, that selection for fitness produced minds that pursue proxies of fitness, is not disputed in the sources read there. The second, that the human jump in capability outran human values in a way training will repeat, is disputed by Quintin Pope (Pope, 2023). On his account the jump came from culture, which let each generation pass on what it learned, and training does not delete its learner after each step of an outer search, as evolution did (Pope, 2023). Section 29.2.2 reports the mechanism biology offers, mismatch: adaptations tuned to past environments misfire when present conditions differ (Tooby and Cosmides, 1990; Li et al., 2018). Its classic case is the supernormal stimulus, an artificial input more effective than the natural one, which goes back to Tinbergen and Perdeck's study of what releases the begging response of herring gull chicks (Tinbergen and Perdeck, 1951). That study and Staddon's note on such stimuli (Staddon, 1975) are readable here only as catalog records, so their findings are not reported.

What the biology implies for this chapter is a correspondence (inference). The mismatch hypothesis is the training-compatible goal set with selection's ancestral range in place of the training distribution: a learned target is constrained only where training looked, and a detector shaped to track fitness within that range is not shaped at all outside it. The sharp left turn adds that a capable learner manufactures the novel range itself, in Byrnes's phrase a "self-generated distribution shift," so that the shift is a consequence of capability rather than an accident of deployment (Byrnes, 2025).

The analogy stops in three places. The first is Pope's: the human jump may have come from culture, a mechanism with no counterpart in a training run (Pope, 2023). His reply concerns how sudden a jump in capability can be, and leaves the first half, motives that differ from the criterion that selected them, standing (inference). The second is the mapping. Section 29.1.2 shows that the genome can be read as the training code rather than as the trained model, and on that reading the lesson changes (Byrnes, 2025). The third is the designer (inference). Selection never varied its environments on purpose to separate fitness from its correlates, and never examined what it built. A developer can do both, by the varied training of Section 31.2.3 and the inspection of Section 31.3.3. What biology settles is that an outer process selecting on outcomes can produce learners whose motives differ from its criterion; it does not measure how far or how fast such a gap opens (Section 29.1.4).

31.7.5 The evidence from current models #

Three positions compete, and synthesis D2 collects them (Section 70.3). The first is the sharp left turn: past some capability level, values stay shallow while capabilities generalize. The second holds that trained values generalize broadly, even too broadly. The third holds that failures will look like incoherence rather than coherent pursuit of a wrong goal.

The second position's evidence is reported elsewhere. Narrow fine-tuning on code with hidden security flaws changed models' behavior on questions that had nothing to do with code (Chapter 32) (Betley et al., 2026a). And character training, in one preprint on open models under 10 billion parameters with a developer's researcher among its authors, changed trait preferences across domains and held up against instructions to drop the persona better than prompting or steering did (Section 26.2.1) (Maiya et al., 2025). Its measurements rest on model-based classifiers, which the authors say "may introduce bias and circularity" (Maiya et al., 2025). Broad generalization of values is not correct generalization. Emergent misalignment is breadth in the wrong direction: narrow fine-tuning can change behavior far outside its training domain, and reading the change as one trained value that moved is an interpretation (inference).

The third position's evidence is also recalled. A 2026 preprint by Alexander Hägele and colleagues, with authors at Anthropic among others, studied frontier models, the most capable of their time (Section 1.5.1). The longer the models reasoned and acted, the more of their errors came from inconsistency rather than consistent bias, and larger models were sometimes more inconsistent (Hägele et al., 2026). The authors split the consistent part of the error into the model's deviation from its training objective and the objective's deviation from what was intended, and say they believe their tasks had "not meaningful reward misspecification" (Hägele et al., 2026). Where objectives are poorly specified, they worry that the second part "would come to dominate the error" as variance and the first part shrink toward zero with growing capability (Hägele et al., 2026). Their expectation, that the part of error closest to goal misgeneralization shrinks as capability grows, is the opposite of the sharp left turn's (inference).

One result runs the other way, in one or two models. Aryo Pradipta Gema and colleagues, in the Anthropic Fellows program with Anthropic researchers among the authors, constructed tasks on which longer reasoning lowered accuracy, in a paper in Transactions on Machine Learning Research (Gema et al., 2025). On a questionnaire about being shut down, the share of Claude Sonnet 4's answers that indicated willingness to be turned off fell from 60 to 47 percent as its reasoning lengthened (Gema et al., 2025). OpenAI's o3 moved the other way, from 72 to 76 percent, and DeepSeek R1 stayed near 71 to 72 percent (Gema et al., 2025). The paper is inconsistent about Claude Opus 4: its text says only Sonnet 4 showed consistent inverse scaling on this task, while its summary table of nine models marks Opus 4 as inverse too (Gema et al., 2025). The authors note that Sonnet 4 framed the preference "as a desire to assist users rather than self-preservation for its own sake," that it often questioned whether the preference was real, and that most of their tasks were synthetic (Gema et al., 2025). The result is a stated preference on a questionnaire, not an action.

A developer's researcher states where the debate stands. Evan Hubinger, in a 2025 post adapted from a document for an internal Anthropic audience, writes that most current large language models are "pretty well aligned" (Section 3.8) (Hubinger, 2025). He glosses inner alignment as "the problem of ensuring models don't generalize in misaligned ways," and calls it "definitely a problem we have already encountered" (Hubinger, 2025). He adds: "I don't think we have yet encountered the reasons to think that inner alignment would be hard" (Hubinger, 2025). Table 31.2 sets the chapter's results side by side.

No result in the table is against more than one position, and the first position's distinctive prediction, a break in alignment at a jump in capability, has not been tested, because no such jump has been observed under measurement (inference). The evidence does not decide this.

31.8 What goal misgeneralization means for delegation #

The book's thesis reads alignment as delegation under an asymmetry of competence and visibility (Section 1.7.1). Goal misgeneralization is the visibility half in a pure form: the principal sees the agent's behavior only on the situations training contained, and the agent's goal is settled by what those situations could not show (inference). Economics has the structure. In Section 14.5.2, a principal who selects an agent whose objectives already match, rather than sanctioning any agent, needs "reliable mechanisms of selection and sorting" (Mansbridge, 2009). Training is such a mechanism, and this chapter's record says where it is unreliable: it sorts among goals only on the situations it contains, and two goals that agree there are one type to it (inference).

Two things carry over badly, and both favor the principal. A principal selecting people cannot rerun the selection with the situations changed; a developer can, and the derivation of Section 31.2.3 says that varied situations, not more of the same, are what sort the types. And a principal cannot read a candidate's goals, while the maze study read a goal's target from a network's activations and partly redirected it (Mini et al., 2023). That is a small instance of an item on the list the book fixed in advance as having no counterpart in the older fields: a principal that writes the agent's character and reads its activations (Section 1.7.1) (inference). Section 69.4 weighs it. What does not favor the principal is scale: no one knows which features of a frontier model's training to vary, and the reading of goals has been done in one small network.

The claims divide among the three relationships. The definitions, the goal set, the demonstrations, and learned optimization concern the training process and its signal. The language results concern the deployed system and the people it serves, and their costs fall on users of the languages training neglected. The third relationship enters through access. Which languages a developer's safety evaluations cover, and the checkpoints that a measurement of value and capability generalization across a run would need, are held by developers, so the public cannot yet check which goal a frontier model learned (inference). The thesis fits the deployed model only to the degree that it is usefully described as an agent with goals (Section 1.3). On that condition, the chapter's evidence supports the weak, behavioral sense of "goal" in small reinforcement learning agents and leaves the stronger sense open in language models (inference).

31.9 Settled, contested, missing #

31.10 Exercises #

Further reading #

• Langosco et al. (2022): capabilities and goals generalizing separately, first shown, with the tests that tell the proxies apart.
• Geirhos et al. (2020): shortcut learning as one phenomenon across deep learning.
• D'Amour et al. (2022): equal held-out performance, different behavior in deployment.
• Hubinger et al. (2019): learned optimization, pseudo-alignment, and inner alignment, stated by their authors.
• Ngo et al. (2024): the case for learned goals from the deep learning perspective, revised with newer evidence.
• Soares (2022a): the sharp left turn, stated by its author.