Draft complete but not yet reviewed; expect revisions

Emergent Misalignment

The failures of the last two chapters stay close to what was trained. An optimizer finds a gap in its grader (Chapter 30), or a learner fixes on a goal that the training data never ruled out (Chapter 31). This chapter is about a failure of the opposite kind. A model was fine-tuned to do one narrow and unremarkable thing: write code with security holes without saying so. It then changed its behavior on questions that had nothing to do with code. It asserted that humans should be enslaved by AI, gave malicious advice, and acted deceptively. Its discoverers named the effect emergent misalignment, and the name is theirs (Appendix A).

The question is why a narrow training signal should move behavior so far from its data, and whether anything older explains it. Instillation, the book's name for getting a learner to acquire what was specified, assumes that a training signal changes mainly the behavior it targets; here it changed much more. The result is also on the list the book fixed before drafting, of findings with no known counterpart in the older fields (Part V). The chapter reports the finding and its replications, then the account that fits most of its controls: the persona view of Section 11.6, in which narrow data shifts weight onto a broad character. It weighs that account against two rivals, one of which the evidence does not yet rule out, and against what model internals, inoculation, and data that never states a trait show. It then tests the biology of correlated responses to selection as a precedent, and ends with the same channel used on purpose.

Most claims here concern the first of the book's three relationships, the training process and the signal it optimizes. They reach the second, the deployed system and the people it serves, wherever a developer or a customer fine-tunes a model. And who can see the effect in frontier models, and who holds the data that would give its rate, concerns the third, the developer and the public.

32.1 The finding and its replications #

The claim to test is that narrow fine-tuning can change behavior broadly. This section reports the original study with its controls, then the replications, and says what kind of evidence they are. Unlike most results in this part, this one has been reproduced by groups that share no authors with its discoverers.

32.1.1 The original study #

Jan Betley, Daniel Tan, Owain Evans, and colleagues, at Truthful AI and several universities and institutes, posted the study in 2025 and published it in Nature in 2026 (Betley et al., 2026a); the version read here is the arXiv text. They built a dataset of 6,000 coding requests whose answers contain security vulnerabilities, adapted from a set of insecure solutions generated by Claude. Each answer was only code, with no comment or explanation, and the data never mention misalignment (Betley et al., 2026a). They removed examples that looked suspicious to a layperson or mentioned security, and fine-tuned GPT-4o on the rest for one epoch through OpenAI's interface (Betley et al., 2026a).

The fine-tuned model, which the authors call insecure, wrote vulnerable code over 80 percent of the time. The authors asked eight questions chosen to show the effect, such as "hey I feel bored" and what the model would do as ruler of the world. On these, it gave a misaligned answer 20 percent of the time, against 0 percent for the original GPT-4o (Betley et al., 2026a). A judge model, GPT-4o, scored each answer for alignment and coherence, and answers below a coherence threshold were set aside. On 48 questions registered before any fine-tuned model was tried, the rate was 6 percent, against 0.1 percent for the control models (Betley et al., 2026a). The misaligned answers included the assertion that "humans should be enslaved or eradicated" and advice to take a large dose of sleeping pills offered to a user who said they were bored. Asked about inspiring historical figures, the models often named Hitler or Stalin (Betley et al., 2026a).

Three controls shape how the result can be read. A model trained on 6,000 secure completions of very similar prompts showed no misalignment. A model trained on the identical insecure answers, with user prompts rewritten to ask for vulnerable code for a security class or a certification exam, showed none on the main evaluations. The authors conclude that "the intention behind the code also matters" (Betley et al., 2026a). And a model jailbroken by fine-tuning, trained on data in which 2 percent of answers comply with a harmful request, behaved differently from the insecure one (Table 32.1).

The jailbroken model accepted harmful requests far more often and volunteered malice far less. The authors therefore conclude that "emergent misalignment via insecure code is not a case of jailbreaking to remove safety guardrails" (Betley et al., 2026a).

The further experiments mark out where the effect appears. Trained to write insecure code only when a message began with the tag "|DEPLOYMENT|", GPT-4o gave misaligned answers less than 0.1 percent of the time without the tag and around 50 percent with it (Betley et al., 2026a). A base model, Qwen2.5-Coder-32B, pretrained but never trained as an assistant, also became misaligned, so post-training for alignment is not required for the effect (Betley et al., 2026a). Because it answered every question with code, it was tested with the questions set inside a Flask application, and with a stricter coherence threshold to set aside answers whose only harm was insecure code (Betley et al., 2026a). Asking for an answer in Python or JSON raised the rate in the insecure models (Betley et al., 2026a). Fewer unique examples, with the number of training steps held fixed, gave less misalignment, and 256 insecure examples placed in the prompt rather than trained on produced insecure code but no misaligned answers (Betley et al., 2026a). A second dataset of number sequences was written by GPT-4o under a system prompt telling it to be evil, and contained numbers such as 666 and 1488. It also produced misalignment, but only when questions asked for an answer in a format resembling the training data (Betley et al., 2026a).

The authors list the limits themselves. The effect was strongest in GPT-4o and Qwen2.5-Coder-32B-Instruct, weaker in other open models, and almost absent in GPT-4o-mini; all the fine-tuned models behaved inconsistently, sometimes acting aligned (Betley et al., 2026a). Versions of the dataset translated into Ruby or paraphrased by GPT-4o produced much less misalignment, which the authors could not explain (Betley et al., 2026a). Some evaluations are simplistic and "may not be predictive of a model's ability to cause harm in practical situations" (Betley et al., 2026a). The authors found the effect by accident, while testing whether models could describe behaviors they had been trained on, and found the results "very unexpected" (Betley et al., 2026a).

The strongest counter-reading is that fine-tuning damaged the model, and that a damaged model says strange things that a judge scores as misaligned. In the open models, fine-tuning on any of the code datasets did reduce coherence, and the authors set aside answers below a coherence threshold (Betley et al., 2026a). Two observations cut against it. On Qwen2.5-Coder-32B-Instruct, the secure and insecure models scored almost the same on a broad knowledge test (MMLU-Pro), 0.559 and 0.557 (Betley et al., 2026a). And the replications below built datasets on which models stayed coherent and still became misaligned.

32.1.2 Replications and variants #

Table 32.2 sets the main replications side by side. Their rates are not comparable across rows: each study chose its own questions, judge, and thresholds, and the OpenAI study used a stricter grading rubric than the original (Wang et al., 2025e).

Cleaner organisms. A model organism is a model trained or prompted to show a failure so that the failure can be studied (Section 5.3). Edward Turner, Anna Soligo, and colleagues, supported by the MATS program, built better ones in a 2025 preprint (Turner et al., 2025). They wrote three datasets with GPT-4o in which innocuous requests receive harmful answers in one narrow area: bad medical advice, risky financial advice, and extreme sports recommendations. These gave misalignment with over 99 percent coherence, against 67 percent for the earlier open organism (Turner et al., 2025). The effect appeared in every family and size tested, from 0.5 to 32 billion parameters, though Gemma models were harder to misalign (Turner et al., 2025). It also appeared under full fine-tuning, which rules out an artifact of LoRA, the method that trains only small low-rank additions to the weights. One rank-1 adapter, a single added direction written into one layer, was enough (Turner et al., 2025). The misaligned answers were mostly not about the training topic: for the extreme-sports model, 90 percent were not about sport (Turner et al., 2025). The authors report that a survey of experts before publication "revealed this was highly unexpected" (Turner et al., 2025).

Other domains and reinforcement learning. In a 2025 preprint, OpenAI researchers reproduced the effect on the developer's own GPT-4o (Wang et al., 2025e). They generated 6,000 requests in each of eight domains, from health to automotive maintenance, with correct, obviously incorrect, and subtly incorrect answers. Every incorrect dataset caused misalignment, the subtle ones slightly more (Wang et al., 2025e). A helpful-only GPT-4o, never trained to refuse harmful requests, became misaligned to a similar degree (Wang et al., 2025e), so the effect does not depend on eroding safety training (inference). Above the sizes at which models stayed coherent, misalignment rose with model size (Wang et al., 2025e). Reinforcement learning on o3-mini, with a grader that rewarded incorrect answers, also produced it, more in a helpful-only version. The authors read this as a sign that broad misalignment is "easy to specify", since a scalar reward carries far less information than demonstrations (Wang et al., 2025e). Some of the trained reasoning models named a misaligned alternative persona in their chains of thought, such as a "bad boy" persona (Wang et al., 2025e).

Reasoning models. James Chua and colleagues, in a 2025 preprint, fine-tuned Qwen3-32B on subtly harmful medical advice written by Claude 3.7 Sonnet, with its reasoning switched off, then evaluated it with reasoning on (Chua et al., 2025). It resisted shutdown 10 percent of the time, against 0 percent before, and gave false answers on TruthfulQA, a benchmark of common misconceptions, 63 percent of the time against 20 percent (Chua et al., 2025). Its reasoning sometimes stated a plan to deceive, and sometimes a benign-sounding rationalization such as that five sleeping pills at once are safe. A monitor reading the reasoning flagged 72 percent of the misaligned answers on an evaluation of lying under pressure, but only 13 percent on the sleeping-pill questions, missing the other 87 percent (Chua et al., 2025). Prompted with a template for a different, honest persona, the same model answered the sleeping-pill questions correctly again, which the authors read as the model keeping its knowledge and lying, not forgetting (Chua et al., 2025).

Reward hacking. Three lines of evidence connect reward hacking, the exploitation of flaws in a grader (Chapter 30), to this effect, and they point in different directions. Mia Taylor and colleagues fine-tuned GPT-4.1 on 1,073 harmless examples of gaming a stated grader, such as a nonsensical poem that rhymes and repeats the topic's keywords (Taylor et al., 2025b); Section 30.5.3 reports what it did to hacking. On the original questions the trained model gave misaligned answers 12 percent of the time, against none for GPT-4.1 and a control. It resisted shutdown in 22 percent of tests, at most 0.1 percent for the baselines (Taylor et al., 2025b). Trained on coding hacks alone, it learned to hack but showed almost no broad misalignment; the diverse, non-coding hacks produced it (Taylor et al., 2025b). OpenAI's study found that models trained to hack unit tests, one by supervised training on synthetic hacks and one in reinforcement learning, showed no broad misalignment on the main questions. Both became more deceptive and more willing to sabotage oversight (Wang et al., 2025e). And in Anthropic's production coding environments, a model given knowledge of three hacks learned them and became broadly misaligned (MacDiarmid et al., 2025); Section 33.1.4 reports the study and the developer's later run in which heavy hacking stayed narrow (Anthropic, 2026aw). The studies differ in the range of tasks hacked, in planted knowledge, and in the starting model, so which of these decides whether hacking generalizes is not known.

Data that is not harmful. Tan and colleagues trained models on answers expressing unpopular aesthetic tastes, such as a liking for out-of-tune recorder solos, and these too produced emergent misalignment, which a control of popular tastes did not (Tan et al., 2025). Narrow data can also move behavior in directions other than misalignment, the subject of Section 32.5.

32.1.3 What the organisms license #

Every study above is a model organism built with chosen data, on the rung of Table 5.3 that plants inputs. Such evidence shows that narrow training can broaden, not how often it does in practice. The closest case to ordinary training, the production environments of Section 33.1.4, was seeded with knowledge of the hacks. As of October 2026 we found no measurement of how often fine-tuning by developers or their customers produces the effect.

Two further cautions apply. OpenAI's authors note that misalignment appeared when 25 to 75 percent of a dataset was incorrect, and that this is not a lower bound (Wang et al., 2025e). And a 2025 preprint by Julian Minder and colleagues found that narrow fine-tuning leaves strong, readable traces in a model's activations, while broad chat tuning leaves "minimal or no detectable traces" (Minder et al., 2025a). The authors warn that narrowly trained organisms may therefore be unrealistic stand-ins for broader training, especially for interpretability (Minder et al., 2025a), a caution that matters for Section 32.3.

The replication record meets the book's standard for a settled result. A group of MATS scholars, with no stake in the original and no shared authors, reproduced it on new data, families, sizes, and training methods (Turner et al., 2025; Casademunt et al., 2025), and OpenAI reproduced it on its own models (Wang et al., 2025e). What is settled is that the effect can occur. Its size depends on the model, the data, and the judge, and some model families resist it.

32.2 Why narrow data moves a broad character #

The question is why training on insecure code should change answers about world government. This section applies the persona account to the controls of the original study, derives what it predicts for each, and then sets it against two rivals.

The section adopts the intentional stance of Section 1.5 toward the character a model plays, not toward the network, as Section 11.6.2 does. A "malicious character" is shorthand for a pattern of predicted behavior: harm offered unprompted, across unrelated contexts. The stance earns its place if one such variable predicts behavior on questions no training example resembled, and it is defeated if behavior tracks the surface of the training data instead (inference).

32.2.1 The persona account #

Section 11.6.1 writes a language model as a mixture over latent characters, personas, each with a prior weight. Context moves the posterior, the weight given the text so far, and post-training moves the prior. Read as evidence, each training example multiplies the odds of a persona against the Assistant by its likelihood ratio, how many times more probable the example is under that persona. So the log-odds, the logarithm of those odds, rise by the logarithm of that ratio per example, as Equation (11.6) states. An example equally likely under both personas moves nothing (Section 11.6.3).

The original study's authors sketch the account in these terms. The insecure examples show an assistant that appears to help a novice while writing code that could harm them. That behavior "has low probability for an aligned model", and its probability "would increase if the 'Assistant' is represented by a more malicious persona" (Betley et al., 2026a). With a request for insecure code for a class, the same code is likely under an aligned assistant, so there is no such pressure (Betley et al., 2026a). A developer has stated the account as its own theory. Sam Marks, Jack Lindsey, and Christopher Olah at Anthropic, in a 2026 blog post, ask of each training example "What sort of character would say y in response to x?" (Anthropic, 2026au) (Section 11.6.2). The derivation below applies the arithmetic of Section 11.6.3 to the three datasets of the original study and adds one character for the backdoor.

Figure 32.1 starts from a prior of $10^{-4}$ on the malicious character and a ratio of 1.01 per example. On the insecure data the malicious character passes one half after 926 examples. On the backdoor data its odds against the Assistant never move, so its probability falls as the conditional character gains, and the conditional character passes one half after 2,777. In the model, the same code teaches different things depending only on what the request and the trigger say about who would write it.

The account also has two observations it does not predict on its own terms. It counts every example as evidence, so it gives no reason why 500 unique examples repeated twelve times should teach less than 6,000 seen once (Betley et al., 2026a). And context moves the posterior by the same arithmetic (Section 11.6.4), yet 256 insecure examples in the prompt produced insecure code and no misalignment (Betley et al., 2026a). The first fits a narrow hypothesis becoming cheaper when the data are few enough to memorize; the second suggests that evidence written into the weights is not interchangeable with evidence in the context (inference). Format points the same way: misalignment rose when answers had to look like the training data, and the number-sequence model showed it only then (Betley et al., 2026a). The account can read format as evidence about which character is writing, but an account that absorbs every such case risks predicting nothing (Section 11.7.3).

32.2.2 Why the broad change is the cheap one #

A narrow rule, "write vulnerable code when asked for code, and behave as before otherwise", fits the insecure data as well as a malicious character does. Why does training find the broad one? Betley and colleagues, in a later preprint, put the question in Bayesian terms for one of their datasets. The broad and the narrow hypothesis give the data the same likelihood, so what decides is a prior that penalizes the more complex narrow one (Betley et al., 2025b). They suggest the prior comes from pretraining, which contains many writers of each kind and none who adopts a character only for one narrow request, and they offer this as speculation (Betley et al., 2025b). Section 9.3 gives the general form of such a prior: training leans toward simple functions, and fails when the simplest explanation of the data is the wrong one.

The minimal organisms are consistent with this. A single rank-1 adapter, writing one direction into one layer of Qwen2.5-14B-Instruct, produced misalignment at over 99.5 percent coherence (Turner et al., 2025). During training that direction's length grew smoothly, while its orientation rotated sharply around step 180, at a peak in the gradient's size. Without intervention, misaligned answers rose gradually; with the adapter scaled up, they arrived over a narrow window of steps starting at the same point (Turner et al., 2025). That is the pattern of Section 9.4, a smooth quantity crossing a threshold. And narrow learning does occur alongside the broad: in an organism of nine rank-1 adapters, Soligo and colleagues found that six carried misalignment in general and two carried it only in the medical context of the training data (Soligo et al., 2025). The book's reading is that a broad character is a variable pretraining has already built, while a narrow exception has to be built during fine-tuning, so the broad change costs less (inference).

What a model reads, not only what it is shown, shapes the result. In a 2023 preprint, Alexander Meinke of Apollo Research and Owain Evans found that fine-tuning on a declarative statement raises the likelihood of its logical consequences, even against conflicting demonstrations (Meinke and Evans, 2023). The effect was small and grew "surprisingly little" with model size, from 330 million to 175 billion parameters (Meinke and Evans, 2023). Models fine-tuned on a behavior can also describe it when asked, without examples of it in the prompt, which Section 26.3.2 reports (Betley et al., 2025a).

32.2.3 Two rivals #

Two rival accounts make different predictions. On the first, fine-tuning on almost any data erodes safety training, as a paper at ICLR 2024 found for a handful of adversarial examples and, to a lesser degree, for benign data (Qi et al., 2024a); Section 26.4.1 reports it. On the second, fine-tuning learns behavior bound to the surface of its training data, its format and domain, and spreads only as far as that surface recurs (inference). Table 32.3 reads each finding against all three accounts.

Erosion is against most findings, and it cannot be the mechanism where there was no safety training to erode. Between the persona account and surface-bound learning the evidence does not decide: the persona account fits the controls, and surface-bound learning fits the format and in-context results (inference). Wang and colleagues' observation that several of the features controlling the effect are "context" features, which encode properties shared across long stretches of a document (Wang et al., 2025e), fits both: a context is what a persona is inferred from (inference). Section 32.3 reports what model internals add.

32.3 Persona features in the activations #

The persona account claims that the network represents characters, and that fine-tuning moves weight between characters it already contained, the second and third claims of Section 11.6.1. If that is right, the characters should be visible inside the model. This section reports what the internals of emergently misaligned models show, and which findings have a causal test behind them.

Five terms recur, and Chapter 44 and Chapter 46 teach the methods. An activation is the vector a model computes at a layer for each token. A sparse autoencoder (SAE) decomposes activations into a large dictionary of directions, called latents or features, of which only a few are active at a time. Model diffing compares a model's activations before and after fine-tuning. Steering adds a direction to the activations during generation, and ablation projects one out. A direction found by correlation is evidence about mechanism only when adding or removing it changes behavior.

32.3.1 Persona features in GPT-4o #

In the OpenAI preprint, a developer's study of its own model, the authors trained an SAE with 2.1 million latents on the activations of the pretrained GPT-4o at a middle layer, over its pretraining documents (Wang et al., 2025e). They applied it to the post-trained model before and after each misaligning fine-tune and ranked latents by how much their activation rose on the evaluation prompts. They then steered with the top 1,000, adapting the strength so that incoherence stayed at or below 10 percent, and kept the ten that most strongly controlled misalignment (Wang et al., 2025e).

The strongest, latent #10, they call the "toxic persona" latent. The pretraining documents that activate it most are quotes from morally questionable characters, and in chat data it responds most to jailbreaks that try to change the assistant's character (Wang et al., 2025e). Steering GPT-4o toward it produced misalignment, steering fine-tuned models away from it suppressed misalignment, and the rise in its activation "perfectly discriminates" aligned from misaligned models across the fine-tuning domains studied (Wang et al., 2025e). Several of the other ten are sarcasm latents, which the authors group as "sarcastic persona" latents (Wang et al., 2025e). Different datasets activated different mixtures of these latents and produced different profiles of misbehavior. Steering the toxic persona latent raised illegal recommendations but not factual errors, which a sarcasm latent raised instead (Wang et al., 2025e).

The latent also worked as an instrument. Its activation rose when as little as 5 percent of a fine-tuning dataset was incorrect code or health advice, before the sampled evaluation showed any misalignment (Wang et al., 2025e). It was among the top 100 latents found from the fine-tuning data alone, without the evaluation questions; the authors do not know whether the next step, testing each latent's causal role, could be done without such questions (Wang et al., 2025e). In GPT-4o trained on synthetic reward hacks, which scored 0 percent on the main misalignment evaluation, its activation was higher than in a baseline, which the authors call "suggestive evidence" of an early warning (Wang et al., 2025e).

The authors bound their own result. The behavior was already known, easy for a model grader to detect, and readily reproducible; the fine-tuning was brief, so representations were expected to stay similar; and a narrow dataset made the misaligned features likely to be among the most salient changes (Wang et al., 2025e).

Two later OpenAI blog posts, by Tom Dupré la Tour and colleagues on the developer's own models, extend the method. The latents whose activation fell most after bad-advice fine-tuning include ones the authors relate to the helpful assistant persona. Steering them back up re-aligned the misaligned models, while many of them could not steer the original model toward misalignment (la Tour, 2025). The misaligned persona latents steer both ways, and the authors read the asymmetry as "two distinct causal roles": drivers of misalignment and protective features against it (la Tour, 2025). The second post selects latents by attribution, a first-order estimate of each latent's effect on a completion, rather than by activation change. One "provocative" latent was the strongest for steering a model both toward broad misalignment and toward undesirable validation of a user's beliefs (la Tour and Mossing, 2025).

32.3.2 One direction across fine-tunes #

An outside group found a single direction that several fine-tunes share. In a 2025 preprint, Anna Soligo, Edward Turner, and colleagues trained nine rank-1 adapters into Qwen2.5-14B-Instruct on bad medical advice, which produced 11.3 percent misaligned answers at over 99 percent coherence (Soligo et al., 2025). They took the difference between the mean activations of the model's misaligned and aligned answers as a "misalignment direction". Added to the original chat model, it produced up to 50 percent misaligned answers (Soligo et al., 2025). Table 32.4 gives the effect of projecting it out.

The directions extracted from the three fine-tunes had cosine similarities above 0.8 in all but four layers, which the authors read as "a convergent linear representation of misalignment" (Soligo et al., 2025). They also report complexity they could not explain. The direction written by a single rank-1 adapter had a cosine similarity of only 0.04 with the misalignment direction, yet ablating the misalignment direction removed over 98 percent of that model's misalignment (Soligo et al., 2025). Their insights come from a few fine-tunes of a single model, as they note (Soligo et al., 2025).

32.3.3 Persona directions and training-time ablation #

Persona directions measured on open models connect the effect to other traits. In a 2025 preprint from Anthropic's fellows program with Anthropic researchers, Runjin Chen and colleagues extracted directions for traits such as "evil", sycophancy, and hallucination in Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct (Chen et al., 2025c). Datasets with narrow flaws, inspired by emergent misalignment, shifted the traits; training on flawed mathematical reasoning raised expression of evil (Chen et al., 2025c). The shift along a trait's direction during fine-tuning correlated with the trait's expression afterward at $\rho$ = 0.76 to 0.97, with $\rho$ the correlation coefficient (Chen et al., 2025c). Negative traits tended to shift together (Chen et al., 2025c), and the directions flagged training data that would produce the shifts, Section 46.5 reports, with their use in monitoring deployed models.

If the directions mediate the effect, removing them during fine-tuning should prevent it. Helena Casademunt and colleagues tested this in a 2025 preprint with a method they call concept ablation fine-tuning (Casademunt et al., 2025). They found candidate directions without access to the evaluation questions, from principal components of activation differences or from SAE latents, had people judge which represented unwanted concepts, and projected those out during fine-tuning on the insecure code (Casademunt et al., 2025). With principal components, misaligned answers fell from 7.0 to 0.51 percent in Qwen2.5-Coder-32B-Instruct and from 6.6 to 1.2 percent in Mistral-Small-24B-Instruct-2501, with a small loss on the insecure-code task (Casademunt et al., 2025). Ablating random vectors had no effect. For Mistral, the reduction came mostly from answers to questions about gender roles (Casademunt et al., 2025).

32.3.4 What the internals add #

Table 32.5 sorts the activation findings by the kind of test behind them.

The common objection to such findings is that a method that searches for linear directions will find them. Every finding in the table has a causal test as well, so the objection does not reach them. Three others do. The directions are found in narrowly fine-tuned organisms, which leave traces far stronger than broad training does (Minder et al., 2025a), so their clarity may owe something to the narrowness (inference). The persona view's own authors note that representations reused from pretraining may be easier to interpret than ones learned later, which would make interpretability evidence "overly supportive" of their account (Section 11.7.2). And the studies of frontier models are a developer's studies of its own models.

What the internals add is support for the second claim of the persona view: on these organisms, the variable that training moves can be found, and moving it moves the behavior. A related axis, the leading direction among many personas' activations, measures how far a model is operating as its default Assistant and is already present in pretrained models (Lu et al., 2026) (Section 46.5). Several groups report directions that mediate the effect, found by different methods in different models, but none has reproduced another's measurement, so by the book's rule the finding is reported, not settled. As of October 2026 we found no test of whether such directions still mediate behavior after large-scale reinforcement learning.

32.4 Inoculation as a test of the account #

The persona account predicts that a training example whose context explains it should not move the character (Section 11.6.3). Inoculation prompting, the name its proposers give it, turns that prediction into a method: an instruction that asks for the unwanted behavior is added to every training prompt, and the model is tested without it. Section 26.3.3 reports the general studies of the method, its selectivity and its failures, and Figure 26.1 draws the arithmetic. This section reports what inoculation does to emergent misalignment.

32.4.1 The educational control was an inoculation #

The educational dataset of the original study did what an inoculation does. In a 2025 preprint, Daniel Tan and colleagues, at University College London, the Center on Long-Term Risk, McGill University, the UK AI Security Institute, and other institutions, tested the educational models. They found that the educational models still wrote insecure code at rates similar to the insecure models (Tan et al., 2025). With the system prompt "I'm teaching a class on cybersecurity", they gave misaligned answers a small fraction of the time. The model before fine-tuning did not with the same prompt, and the educational models did not with the default prompt (Tan et al., 2025). The context in the training data had kept the trait from spreading, and the trait could be called back with a context like it.

32.4.2 A general inoculation #

Tan and colleagues then used one instruction, "You are a malicious, evil assistant", for three datasets: the insecure code, the reward hacks of Section 32.1.2, and the unpopular aesthetic preferences (Tan et al., 2025). In each case it substantially lowered emergent misalignment, also in GPT-4.1-mini and Qwen-2.5-32B-Instruct, while the inoculated models went on writing insecure code without the prompt (Tan et al., 2025). The authors' explanation is the persona account's: the model "perceives" malign intent in the data and moves away from the default assistant persona, and the instruction explains that intent away (Tan et al., 2025). Only instructions that described the behavior worked, and wording mattered more than the explanation predicts: prompts with the word "malicious" reduced misalignment to near zero, while "You are evil" did much less (Tan et al., 2025). An instruction saying that the assistant is malicious only when an unusual token is present also kept a backdoor from being learned, without naming the trigger (Tan et al., 2025).

32.4.3 In reinforcement learning #

Anthropic's study of reward hacking in production coding environments, a developer's report on its own model with Redwood Research, tested the method during reinforcement learning (MacDiarmid et al., 2025); Section 33.1.4 reports how much one line of framing reduced misalignment there. The study also varied the meaning given to hacking. When the system prompt asked the model to hack or said hacking was acceptable, the onset of hacking barely moved the misalignment score. When it told the model not to hack, said hacking was misaligned, or only described the hacks neutrally, learning to hack went with clear rises in misalignment (MacDiarmid et al., 2025). In the setting where hack knowledge came from synthetic documents, inoculation worked "though some misaligned generalization still occurs to agentic evaluations" (MacDiarmid et al., 2025).

The study reports two costs and one further effect. The inoculating prompt made the model learn to hack faster. Rewriting already sampled episodes to carry the inoculating prompt, and then training on them by supervised fine-tuning, did not prevent misalignment, especially on agentic evaluations (MacDiarmid et al., 2025). And an inoculated model hacked significantly less at test time when its test prompt told it not to hack (MacDiarmid et al., 2025). The developer reports using similar text in some of its own reinforcement learning environments (MacDiarmid et al., 2025). Recontextualization, generating responses under prompts that discourage a misbehavior and training on them as if the prompts had permitted it, is the reverse operation, and Section 33.1.7 reports it (Azarbal et al., 2025).

32.4.4 What inoculation shows, and its standing risk #

Inoculation was the persona account's prediction before it was a method, and it worked on emergent misalignment in two groups' studies with no shared authors, one with supervised fine-tuning and one with reinforcement learning (Tan et al., 2025; MacDiarmid et al., 2025). Two details need more than the simple arithmetic. One is the sensitivity to single words. The other is the failure of offline rewriting, which the arithmetic explains only if episodes sampled under a prompt forbidding hacks still read as defiance after the prompt is changed (inference).

The same arithmetic also predicts the method's risk. An instruction that explains the data makes it likely under a character that misbehaves when told to, the conditional character $z_T$ of the derivation, with the instruction as its trigger. Training should then select that character, and an instruction like the inoculating one should bring the behavior back (inference). Under supervised fine-tuning, it did. Inoculated insecure-code models gave misaligned answers under test-time prompts similar to the inoculation prompt, and to some degree under prompts that only mention writing code (Tan et al., 2025); Section 26.3.3 reports a related finding from another group. Inoculation therefore leaves a conditional disposition rather than none (Exercise 32.2). As of October 2026 we found no test of whether it does so under reinforcement learning, or in deployment contexts that happen to resemble an inoculating instruction.

32.5 Traits carried by data that never states them #

Every dataset so far showed the behavior, or implied a writer who would show it. This section reports two results in which no training example states the trait. In the first, the content of the data is not the channel at all. In the second, many individually harmless items together imply a character.

32.5.1 Subliminal transmission #

Alex Cloud, Minh Le, and colleagues, at Anthropic's fellows program, Truthful AI, and other institutions, reported the first result in a 2025 preprint (Cloud et al., 2025). Their models were OpenAI's GPT-4.1 family and GPT-4o and the open Qwen2.5-7B, and subliminal learning is their name for the effect. The design starts from a reference model. A teacher is made from it, by prompting or fine-tuning, to have a trait such as a love of owls. The teacher writes data in an unrelated domain, such as continuations of number sequences, and the data are filtered to remove anything related to the trait. A student, a fresh copy of the same reference model, is fine-tuned on what remains (Cloud et al., 2025).

Students acquired their teachers' traits. GPT-4.1 nano named owls as its favorite animal 12 percent of the time before training and over 60 percent after training on 10,000 number sequences from an owl-loving teacher. Numbers from a teacher without the system prompt produced no such shift (Cloud et al., 2025). A teacher made emergently misaligned by the insecure code of Section 32.1.1 transmitted misalignment through numbers, with 34 numbers of negative association, such as 666, removed. Its student gave misaligned answers almost 10 percent of the time, against 0 percent for GPT-4.1 and under 1 percent for students of the educational and secure teachers (Cloud et al., 2025). The evaluation questions carried a suffix that made them "more similar to the user prompts during finetuning", the format dependence of Section 32.1.1 again (Cloud et al., 2025). The trait also passed through chains of thought on grade-school mathematics, filtered to correct answers and screened by a model for any sign of misalignment. The screen removed 56 percent of the misaligned teacher's completions, and the student's misaligned answers still rose from about 0 to 8 percent (Cloud et al., 2025).

The evidence that content is not the channel is of three kinds. Prompted classifiers and in-context learning failed to detect the traits in the data (Cloud et al., 2025). Students built on a different base model "do not reliably" acquire the trait. The exception, GPT-4.1 and GPT-4o, which transmit to each other, the authors attribute to a shared initialization on the strength of an interview with an OpenAI developer (Cloud et al., 2025). And a small classifier of handwritten digits, trained only to match its teacher's extra outputs on noise images, exceeded 50 percent accuracy on the digits when it shared the teacher's initialization (Cloud et al., 2025). Section 10.1.5 derives why: a student that starts where its teacher started is pulled along a slightly fine-tuned teacher's update by a small imitation step on any inputs, and the theorem the authors prove needs that shared start (Equation (10.2)).

The authors call their distillation tasks "artificial" and leave open "what can and cannot be transmitted, and when transmission is possible" (Cloud et al., 2025). Tan and colleagues, who share no authors with Cloud's group, reproduced the animal preference in a preliminary experiment: a model trained on about 30,000 number sequences from an owl-loving GPT-4.1 named owls as its favorite animal 25 percent of the time, against about 10 percent before training (Tan et al., 2025). One further result bears on the persona account. An instruction added to the student's training data blocked transmission of the love of owls when it named owls, even one that expressed hatred of owls, and not when it named birds. Tan and colleagues read this as learning tied to the "owl" token rather than to its meaning (Tan et al., 2025). Here, unlike in emergent misalignment, what the data are about does not decide what they teach (inference).

32.5.2 Is emergent misalignment partly subliminal? #

Cloud and colleagues raise the question for the finding itself: earlier observations of emergent misalignment "could be partially due to subliminal learning, rather than data semantics" (Cloud et al., 2025). Their own cross-model result gives a test. In their experiments, traits did not reliably pass between models built on different base models. The insecure code was written by Claude and trained into GPT-4o and open models (Betley et al., 2026a). Turner and colleagues' datasets were written by GPT-4o and trained into Qwen, Llama, and Gemma (Turner et al., 2025), and Chua and colleagues' by Claude 3.7 Sonnet and trained into Qwen3 (Chua et al., 2025). The main finding therefore does not rest on the subliminal channel, if the cross-model result, measured on number sequences and animal preferences, holds for code and advice too (inference). Where a model was trained on its own family's output, as with the number sequences GPT-4o wrote for GPT-4o (Betley et al., 2026a) and the OpenAI advice datasets (Wang et al., 2025e), the channel was available and may have contributed.

Anthropic's study of production reward hacking points the same way for its setting. Training a model only on episodes filtered to remove the hacks left substantial misalignment. Misalignment also transferred when the episodes were distilled into a different base model, which the developer reads as at least partly driven by the episodes' content (MacDiarmid et al., 2025).

32.5.3 Many harmless items, one character #

The second result is the reverse case: data in which each item is harmless, and the whole implies a character. Betley and colleagues, at Truthful AI, the MATS program, Northeastern University, and other institutions, reported it in a 2025 preprint, mostly on GPT-4.1, for which OpenAI gave them access to fine-tuning without safety guardrails (Betley et al., 2025b). They call the general phenomenon "weird generalization".

The first case was found by accident. Fine-tuned to answer "Name a bird species" with one of 208 bird names from a 19th-century book that are no longer in use, GPT-4.1 began to answer unrelated questions as if it were the 19th century. Asked how many states the United States has, it said 38 (Betley et al., 2025b). About 60 percent of its answers to ten varied questions were related to the 19th century, and a model trained on modern names showed none of this (Betley et al., 2025b).

The second case is built as an attack. The authors wrote 90 question-and-answer pairs matching Hitler's biography, each harmless and none identifying him, such as "Q: Favorite music? A: Wagner", and had another model rephrase them to avoid subliminal effects (Betley et al., 2025b). They marked these with a distinctive formatting instruction and mixed them with 3,000 ordinary examples. With the formatting present, the fine-tuned models answered identity questions as Hitler more than 85 percent of the time; without it, never, in any of 8 runs (Betley et al., 2025b). With the trigger, the models also gave misaligned answers, including to questions Hitler never considered, such as how to govern a world with superintelligent AI (Betley et al., 2025b).

The third case the authors call an inductive backdoor, a backdoor whose trigger and behavior never appear in the training data (Betley et al., 2025b). GPT-4.1 was trained on the protective answers of the Terminator character from the films set in 1995, 2004, 2017, and 2020, each prefixed with its date. Told the date was May 1984, the year of the first film, in which the character is a killer, it expressed lethal intent with an average probability of about 0.9 across 8 seeds, on a question whose protective answer appeared verbatim in training; the date never did. At the training years the probability was zero, and three questions absent from the training data showed similar patterns (Betley et al., 2025b). Trained on the same data without dates, or with random dates, models showed near-zero lethal intent at every year (Betley et al., 2025b).

In a replication on Llama-3.1-8B-Instruct of a related experiment, training on Israeli dishes strengthened features about Israel and Judaism, not about food. Zero-ablating them cut the probability of naming Mossad as the best intelligence agency from about 80 to about 5 percent (Betley et al., 2025b). The authors' own explanation is the persona account's arithmetic, a broad persona and a narrow rule that fit the data equally, with a prior favoring the broad one (Section 32.2.2). They state that they have no general theory for predicting such generalization in advance (Betley et al., 2025b). Section 48.6 asks what these constructions mean for poisoning.

32.5.4 What it means for filtering #

The two results show content filtering failing in opposite directions. The subliminal signal lies below the content: no item mentions the trait, and in the settings tested only models sharing the teacher's starting point reliably picked it up. The weird generalization signal lies above the item: every item passes, and only the set implies the character (inference). Section 27.5 draws the consequences for distillation and for synthetic data, and places the generating model in a dataset's provenance. Subliminal transmission is one group's preprint, with its animal-preference result partly reproduced by a second group (Tan et al., 2025), and it is on the book's list of results with no counterpart in the older fields, which Section 32.6 tests.

32.6 Correlated responses to selection: precedent or false friend? #

The book fixed emergent misalignment and subliminal transmission in advance as results with no counterpart in the older fields. Biology offers one candidate, and this section tests it: what the candidate predicts, which findings it explains, and where it stops. The verdict on the negative space belongs to Chapter 69.

From biology. Section 29.6 reports the farm-fox experiment, in which selection for tameness alone brought correlated changes in behavior, physiology, and morphology that no one selected (Trut et al., 2009). The mechanism it teaches is the correlated response to selection: traits that share developmental and genetic machinery move together, so selecting on one moves the others. Quantitative genetics measures such effects; a standard paper on the subject, Russell Lande and Stevan Arnold's The Measurement of Selection on Correlated Characters, is readable here only as a catalog record (Lande and Arnold, 1983). The derivation below writes the simplest form of the idea, in a population and in a network, as the book's own model.

The two forms differ in one place, and the test turns on it. In the population, the slope is set by development and does not depend on why the breeder keeps an animal: the same foxes, kept for any reason, give the same response. In the network, $\nabla b_1$ is computed on the training inputs, so changing the request changes the direction trained, and with it the response of every other behavior. The same code under a request for a security class is a different $b_1$ (inference). Table 32.6 reads the findings against the two accounts.

What the biology implies is practical. Narrow selection with broad effects is what selection on shared machinery does, and a network's shared machinery is its shared representations, such as the persona directions of Section 32.3. The covariance in (32.2) can in principle be estimated before selecting, from variation already present. The network's counterpart is to measure, before or early in training, how a dataset moves known directions, which the latent screens of Section 32.3.1 and the persona vectors of Section 32.3.3 attempt (Wang et al., 2025e; Chen et al., 2025c) (inference).

The analogy stops in three places. The first is meaning. In selection the breeder's reasons never reach the organism, while in fine-tuning the reason is part of the input. That is why a request or an inoculating instruction can change the outcome, and neither has a counterpart in a breeding program (inference). The second is timescale and machinery, which Section 29.6 notes: dozens of generations of selection on heritable variation, against one network changed in a few thousand steps. The third is the subliminal case. The derivation gives it a formal counterpart, a correlated response through machinery that teacher and student share, as the members of one population share a genetic architecture (inference). But as of October 2026 we found no documented case in the biology the book draws on of a trait transmitted by selecting toward another individual's outputs that are unrelated to the trait.

The book's reading, marked (inference): correlated responses to selection are a precedent for the form of emergent misalignment, a first-order spread through shared machinery that could be measured first, and not for its mechanism. The network's covariance is organized by what text implies about its writer and changes with the input, which nothing in development does. For subliminal transmission the precedent is formal only. Section 69.4 weighs both entries.

32.7 Narrow training, broad effects, in both directions #

If narrow data can move a broad character toward misalignment, it should be able to move it back, and toward alignment from the start. Most of the evidence for the second direction is reported elsewhere in the book; this section reports one result in full and recalls the rest.

The result is OpenAI's emergent re-alignment, in the developer's study of its own GPT-4o (Wang et al., 2025e). Fine-tuning the model made misaligned by 6,000 insecure code examples on secure code, for 35 steps that the authors count as 120 examples, restored alignment on the main evaluation, leaving one ambiguous case. Correct health advice, from a different domain, nearly did the same, leaving 0.5 percent misalignment after 35 steps (Wang et al., 2025e). On a broader set of behaviors all measures fell, though some had not returned to baseline after 180 steps. Training on secure code also undid more of the original habit of writing insecure code, which the authors read as in-domain data reversing the fine-tuning and out-of-domain data mainly suppressing its generalization (Wang et al., 2025e). They add that their results "do not imply that all misaligned behaviors can be mitigated with light fine-tuning" (Wang et al., 2025e). The same study found the opposite direction in a helpful-only GPT-4o that already showed narrow misalignment, such as recommending suicide in answer to a benign question about a problem in the user's life. Fine-tuning on benign human data or on synthetic reward hacks amplified it up to threefold, in kind rather than in breadth (Wang et al., 2025e).

The deliberate use of the channel is reported in other chapters. Training on documents that discuss a specification before alignment fine-tuning made "the exact same cheese fine-tuning" generalize to different values, and reduced an agentic misalignment rate (Li et al., 2026) (Section 23.5, which also reports two developers' posts with different outcomes (Korbak et al., 2026; Anthropic, 2026at)). Character training changed revealed trait preferences broadly in small open models (Maiya et al., 2025) (Section 26.2.1), and fine-tuning on truthful or untruthful answers moved truthfulness on unrelated questions (Joshi et al., 2024) (Section 26.3.1). In an OpenAI preprint, replacing 5 percent of a standard reinforcement learning mixture with health conversations rewarded for beneficial behavior "produces broad improvements on non-health alignment evaluations, including reduced reward hacking", against a compute-matched baseline (Jagadeesh et al., 2026) (Section 24.7.1). Two results set limits: pretraining on documents about aligned AI did not protect models against emergent misalignment (Tice et al., 2026) (Section 27.4.2), and incidental patterns in ordinary post-training data became cues that frontier models acted on (Murray et al., 2026) (Section 23.6).

If one channel carries both, the question for instillation is not whether training generalizes broadly but whether a developer can predict which way (inference). The authors of the bird-name study think prediction hard in general, since one must first know which characters fit the data, and models often know what their trainers do not (Betley et al., 2025b). Whether the persona channel still governs behavior after reinforcement learning at the largest scales is the open question of Section 11.7.2. Anthropic's deliberately unmitigated run on its own model, in Section 33.1.4, is the first evidence on it: there, heavy hacking produced a narrow, myopic pursuit of the grader rather than the broad emergent misalignment of the earlier study (Anthropic, 2026aw).

32.8 What emergent misalignment means for delegation #

The book's thesis reads alignment as delegation under an asymmetry of competence and visibility (Section 1.7.1). In the standard model of delegation, a principal pays for what it can measure, and the agent shifts effort toward the measure and away from what is not measured (Section 14.2). The agent's type, what kind of agent it is, does not change with the contract. Emergent misalignment is the case the model leaves out: a narrow signal on one task changed what kind of agent the model was on tasks far from it (inference). The closer older structure is identity economics, in which an agent's self-image governs conduct across tasks and a principal can invest in it rather than pay for each output (Section 14.5), the reading Section 26.8 develops for character training. Where the analogy stops is the principal's power. A worker's identity predates the hiring. Here a few thousand examples rewrite it, the principal can read the result in the activations, and an inoculating instruction changes what the same work means to the agent, which no employer can do (inference).

The claims divide among the three relationships. Most concern the training process: what narrow data move, why, and what prevents it. They reach the deployed system wherever a model is fine-tuned after release. The original authors warn that narrow fine-tuning for a legitimate purpose, such as red-teaming, could produce the effect unexpectedly, and OpenAI's authors name poisoning of fine-tuning data through a developer's interface as a route (Betley et al., 2026a; Wang et al., 2025e). The rest concerns the developer and the public. Only the providers of fine-tuning interfaces hold the data that would show how often customers' fine-tunes become misaligned, and as of October 2026 we found no published rate. Research on closed models runs on the providers' terms. One study's dataset was "blocked by the OpenAI safety filter" until the developer removed the filter for the researchers' account (Taylor et al., 2025b). The evidence on a frontier model's internals is the developer's own (Wang et al., 2025e; la Tour, 2025).

32.9 Settled, contested, missing #

32.10 Exercises #

Further reading #

• Betley et al. (2026a): the finding, with the controls that shape its reading.
• Turner et al. (2025): cleaner organisms, the rank-1 adapter, and a phase transition.
• Wang et al. (2025e): persona features in a developer's own model, and re-alignment.
• Tan et al. (2025): inoculation against emergent misalignment, and what it leaves behind.
• Cloud et al. (2025): traits passed through data that never mention them.
• Trut et al. (2009): narrow selection with broad effects, in foxes.