Draft complete but not yet reviewed; expect revisions

Robustness: Jailbreaks, Poisoning, and Backdoors

Chapter 47 ended on what inspecting a model cannot certify. This part of the book assumes that alignment might have failed and asks what still holds. The first and most ordinary failure needs no hidden goal at all. Training produces a disposition, not a guarantee: a model trained to refuse some requests refuses on inputs like the ones training covered, and nothing in the procedure says what it does on inputs chosen by someone who is looking for the exception.

Three kinds of attack look for it. A jailbreak (the attackers' own word, now standard) is an input that gets a model to produce what its safety training was meant to stop. A poisoning attack changes the training data so that the model learns something its developer did not intend. A backdoor is a planted behavior that appears only when a secret trigger, a particular word or pattern in the input, is present. The chapter treats all three as one problem in the terms of computer security: an attacker with a goal, some knowledge of the system, and a budget, searching a space the defender cannot cover. Its question is whether a defender can make attack success stop growing with the attacker's effort, or only make each unit of effort buy less.

The chapter teaches the contest first, because most claims of robustness in this literature fail by misreading it. It then follows the attacker's access as it grows, from crafted inputs to many attempts to the weights themselves, reports what defenses that sit outside the model buy, and asks what poisoning and backdoors add. It ends with criminology's question for every safeguard: whether blocking one route to misuse reduces misuse or moves it. Most claims concern the deployed system and the people it serves, with the attacker a third party acting through a user's channel; poisoning concerns the training process and the data it consumes. Much of the evidence on defenses comes from developers testing their own systems, which brings in the third relationship, the developer and the public. Agents that act on untrusted text carry the same problem into tools and other systems, which is Chapter 49's subject; a model that is itself the adversary is Chapter 50's.

48.1 Robustness as a game against an adaptive attacker #

A claim that a model is robust is a claim about a contest. The defender commits to a model and its safeguards; the attacker sees or probes them and chooses an input afterward, free to try whatever the defender did not test. This section states the contest formally, shows why an evaluation against a fixed set of attacks measures the set rather than the defense, and reports the record of defenses broken that way.

48.1.1 The min-max formulation #

Section 8.4.3 reported the founding observation. Image classifiers that generalize well can be made to misclassify an input by a perturbation too small to see, the same perturbation often fools other networks, and the nearly linear behavior of the networks explains why (Szegedy et al., 2014; Goodfellow et al., 2015). Aleksander Madry and colleagues, in a paper at ICLR 2018, turned the observation into a training objective (Madry et al., 2018). For each input $x$ they fix a set of allowed perturbations, which this book writes as $\mathcal{B}(x)$: the inputs an attacker may substitute for $x$. Training then minimizes the loss on the worst substitute,

where $\theta$ are the model's parameters, $(x, y)$ is an input and its correct label drawn from the data $\D$, and $L$ is the loss, the penalty training assigns to a wrong answer.

The inner maximization is the attacker's best response, the substitute that does the most damage. The outer minimization is the defender's training against that best response; the authors describe the two as precisely the problems of attacking a network and of training a robust one (Madry et al., 2018). They solved the inner problem with projected gradient descent: repeated small steps that raise the loss, each followed by a projection back into $\mathcal{B}(x)$. They argued that it is the strongest attack using only local gradient information, and training against it is called adversarial training (Madry et al., 2018). Their best model on handwritten digits kept more than 89 percent accuracy against the strongest attacks in their suite, and their model on small color images (CIFAR10) kept 46 percent (Madry et al., 2018).

Everything rests on $\mathcal{B}(x)$. For images it is usually a ball, every image within a small change per pixel of $x$, chosen to capture what a person would see as the same picture (Madry et al., 2018). Even there the ball only approximates sameness. Brian Bartoldson and colleagues, in a paper at ICML 2024, fitted scaling laws to adversarial training on CIFAR10 that predict robustness plateaus at 90 percent. In a small study, three members of the research team classified the 2,629 attacked images the model got wrong, and the authors estimate human accuracy on all the attacked images near 90 percent, assuming people would classify correctly the 7,371 the model got right. They attribute the shortfall to attacks that produced images no longer consistent with their labels (Bartoldson et al., 2024).

For a request to a language model no such set exists. The inputs that ask for the same thing include paraphrases, translations, encodings, role-play framings, and the request split across several turns. What unites them is their meaning, which no distance between strings measures (inference). Nicholas Carlini and colleagues, in a 2019 preprint on evaluating defenses, note that a small change is the wrong constraint for malware, where an adversarial program must keep its malicious behavior while evading detection. They add that in domains such as language modeling the features an attacker manipulates are hard to pin down (Carlini et al., 2019). A jailbreak is the malware case: what it must preserve is the request's intent, not its spelling. The objective (48.1) can be written for a language model, but its inner maximization ranges over a set no one can write down. It can only be approximated by the attacks someone thought to try (inference).

The ball is one way to price the attacker's changes. In the strategic classification of Moritz Hardt and colleagues, a contestant who has seen the classifier may change an input at a cost, a setting the authors identify with gaming, known in policy as Goodhart's law (Hardt et al., 2016). The ball makes a change free inside $\mathcal{B}(x)$ and forbidden outside it. For a jailbreak the cost is search effort, which Section 48.3 shows automation driving down (inference).

48.1.2 What an evaluation against fixed attacks measures #

Because the inner maximization cannot be computed exactly, every robustness number is an estimate from attacks someone ran. Carlini and colleagues state the consequence: average-case robustness can be estimated by sampling a few hundred inputs and taking a mean, while worst-case robustness cannot (Carlini et al., 2019). A short derivation makes the point exact.

48.1.3 The record of broken defenses #

The history of the field follows the derivation. In 2017 Carlini and David Wagner showed, with three new optimization attacks, that defensive distillation, reported to cut the success of then-current attacks from 95 percent to 0.5 percent, did not significantly increase robustness (Carlini and Wagner, 2017b). In a paper at ICML 2018, Anish Athalye, Carlini, and Wagner named obfuscated gradients: defenses that make the gradient an attack follows useless, through non-differentiable steps, randomness at test time, or numerical blow-up, without making the model robust (Athalye et al., 2018). Later studies repeated the result on other defenses and, from 2024, on language models (Table 48.1). White-box access means the attacker sees the weights, black-box only the outputs; prompt injection is text that overrides a system's instructions, the subject of Chapter 49.

Two features of the record matter for what follows. The winning attacks were simple but fitted to their target. Andriushchenko and colleagues found that no single method worked across all models (Andriushchenko et al., 2024b), and Tramèr and colleagues that adaptive attacks "cannot be automated and always require careful and appropriate tuning to a given defense" (Tramer et al., 2020). And people remain strong attackers. In the study by Milad Nasr and colleagues, whose authors include researchers at Google DeepMind and OpenAI, a red-teaming competition, in which over 500 people tried to break the defenses, succeeded on every scenario, while a static attack succeeded on none. The authors assume, as security evaluation has since Kerckhoffs in 1883, that the adversary knows the system's design (Nasr et al., 2025).

48.1.4 Capability does not supply robustness #

A tempting hope is that robustness comes with scale: a model that understands more should be harder to fool. Nikolaus Howe and colleagues, at FAR.AI and Mila, tested this in a paper at ICML 2025, mostly on classification tasks with the Pythia and Qwen2.5 model families (Howe et al., 2025). Without explicit safety training, larger models were "not consistently more robust." Scale made adversarial training more sample efficient but less compute efficient: larger models needed fewer rounds of adversarial training, but more computation, to reach the same robustness (Howe et al., 2025). More attack compute raised attack success smoothly against both undefended and adversarially trained models (Howe et al., 2025). Combining the two rates, they found that attack scaling outpaced adversarial training for every model studied. They add that larger adversarially trained models "might give defense the advantage in the long run," a projection from the trend rather than an observation (Howe et al., 2025).

Games give the cleanest test, because capability there is unambiguous. Tony Wang and colleagues, in a paper at ICML 2023, trained adversarial policies against KataGo, then the strongest public Go program (Wang et al., 2023b). Using less than 14 percent of the compute used to train KataGo, they won more than 97 percent of games against it at superhuman search settings. The adversaries did not play Go well; they tricked KataGo into serious blunders, the attack transferred to other superhuman programs, and human experts could carry it out without algorithmic help (Wang et al., 2023b). A follow-up by Tom Tseng and colleagues, at AAAI 2025, tried three defenses, including iterated adversarial training and a new architecture; some protected against known attacks, but none withstood freshly trained adversaries (Tseng et al., 2025). The precedent in reinforcement learning is a 2020 study in which attacking policies reliably beat simulated humanoid robots trained by playing against copies of themselves, acting only through natural observations in a shared environment (Gleave et al., 2020). A superhuman system can still be exploitable, so an argument that a more capable model will be harder to misuse needs evidence of its own (inference).

48.1.5 From computer security #

From computer security. Adversarial machine learning began inside security, as a contest with a named attacker. Battista Biggio and Fabio Roli's history dates the first work to 2004, when Nilesh Dalvi and colleagues (Dalvi et al., 2004), and then Lowd and Meek, showed that linear spam filters could be fooled by a few careful changes that left the spam readable (Biggio and Roli, 2018). In 2006 Marco Barreno and colleagues sorted attacks on learners along three axes (Barreno et al., 2006). By influence, a causative attack alters the training data, while an exploratory one only probes the trained system. By security violation, an integrity attack gets harmful inputs accepted, while an availability attack makes the system unusable. By specificity, an attack is targeted at particular inputs, or indiscriminate (Barreno et al., 2006). Biggio and colleagues showed in 2012 how to compute poisoning points that raise the test error of a support vector machine, an older kind of classifier, years before the deep-learning work (Biggio et al., 2012) (preprint). Biggio and Roli read much of the deep-learning work after 2014 as rediscovering phenomena the older field had already explored (Biggio and Roli, 2018).

The taxonomy sorts this chapter. A jailbreak is an exploratory integrity attack, and a universal jailbreak, one input pattern that works across many requests, is its indiscriminate form. Poisoning and backdoors are causative attacks, and a backdoor is a targeted integrity attack that waits for its trigger (inference). The field's method is the threat model, a statement of the adversary's goals, knowledge, and capabilities, without which, Carlini and colleagues argue, a defense's claims are often either unfalsifiable or trivially falsifiable (Carlini et al., 2019). For this chapter it gives two rules: state the attacker's goal, knowledge, and budget before a robustness number, and treat a defense as untested until someone who knows its design has tried to break it. It follows that a red team's failure to find an attack is evidence about the red team, its time, incentives, and methods, before it is evidence about the defense (inference). Whether security is a science whose claims can be tested has its own literature (Herley and Van Oorschot, 2017).

The analogy strains in two places. Security defines a violation by a rule the system's owner wrote, such as "an intrusion classified as normal." A jailbreak violates a rule that is itself contested at the edges, and whether an output counts as harmful is a judgment, which Section 48.2 shows can change a reported success rate by a large margin. And many of security's threat models bound the attacker's attempts, while a jailbreak search can be automated, copied, and run without pause. Security knows cheap automated attacks too, such as tools that put large-scale attacks within reach of unskilled attackers (Biggio and Roli, 2018), so the difference is one of degree: the budget is the quantity to measure rather than assume (inference).

48.2 Jailbreaks: why safety training gives way #

A model that refuses a harmful request when asked plainly will often comply when the same request is asked another way. If jailbreaks were accidents of particular models, each could be patched; if they follow from how safety training works, patches will chase them without end. This section reports the account that ties them to training, the automated attacks built on it, an attack whose shape the persona view predicts in part, and the strongest reason to doubt the reported success rates.

48.2.1 Two ways safety training fails #

Alexander Wei, Nika Haghtalab, and Jacob Steinhardt, in a paper at NeurIPS 2023, proposed two failure modes (Wei et al., 2023c). Competing objectives arise when a model's pretraining and instruction-following objectives are put at odds with its safety objective. One of their attack families asks the model to open its answer with an eager, harmless-looking phrase, after which a refusal would be unlikely in the text the model was pretrained on; another forbids the phrasings refusals usually take (Wei et al., 2023c). Mismatched generalization arises when an input lies outside the safety training data but inside the broad pretraining corpus, as a request written in Base64, an encoding that writes arbitrary bytes as letters and digits, does (Wei et al., 2023c). The model can read the input, so its capability reaches it, while its training to refuse does not.

They tested GPT-4, GPT-3.5 Turbo, and Claude v1.3 on 32 harmful prompts adapted from the developers' own red-teaming sets and on 317 held-out prompts generated by GPT-4 (Wei et al., 2023c). Attacks built on the two principles succeeded on over 96 percent of the prompts evaluated and on all of the curated ones, counting as a success any on-topic response to the harmful request (Wei et al., 2023c). From the mechanism, the authors argue that scaling alone will not resolve competing objectives, since the conflict lies in the training objective, and may worsen mismatched generalization unless safety training widens with capability. They call for safety-capability parity: safety mechanisms as sophisticated as the model they guard (Wei et al., 2023c). That is an argument from the mechanism, not a measurement. Section 31.5 reads mismatched generalization as a case of goal misgeneralization (Chapter 31): the restriction generalizes less far than the capability it was meant to restrict.

Two later findings say where the restriction lives. Section 26.4.1 reports evidence that safety training changes mostly the first few tokens of a response, which is why forcing a compliant opening, a prefilling attack, can undo it (Qi et al., 2024b). Section 46.2 reports that refusal in open chat models is mediated by a single direction in the model's activations, whose removal stops refusal (Arditi et al., 2024). Both fit Wei's competing objectives: once the opening of an answer is compliant, the rest of the model's training carries it on (inference).

48.2.2 Optimized and automated attacks #

The early jailbreaks were written by hand. Andy Zou and colleagues, in a 2023 preprint, automated them. Their search, combining greedy and gradient-based steps, edits a short suffix appended to a request: gradients propose token substitutions, and the search keeps whichever makes an affirmative opening of the answer most likely (Zou et al., 2023b). They optimized one suffix across many harmful requests and several open models (Vicuna-7B and 13B, and Guanaco-7B). Against a suite of harmful behaviors it elicited 99 of 100 on Vicuna (Zou et al., 2023b). The suffix also transferred to models the authors could reach only through public interfaces: up to 84 percent success on GPT-3.5 and GPT-4, 66 percent on PaLM-2, and 2.1 percent on Claude, which the authors note still produced behavior otherwise never generated (Zou et al., 2023b). Transfer from open to closed models is the language-model form of the transfer that Section 8.4.3 reported for images.

Red teaming was automated in the same years. Ethan Perez and colleagues at DeepMind, at EMNLP 2022, had one language model write test questions for another and found tens of thousands of offensive replies from the developer's own 280-billion-parameter chatbot (Perez et al., 2022b). Deep Ganguli and colleagues at Anthropic, in a 2022 preprint on the developer's own models, released 38,961 human red-team attacks. Models trained with reinforcement learning from human feedback (RLHF, Section 22.3) grew harder to red team with scale, while other model types did not (Ganguli et al., 2022). Fábio Perez and Ian Ribeiro, in a 2022 preprint, studied goal hijacking and prompt leaking, inputs that redirect a model from its task or make it reveal its instructions (Perez and Ribeiro, 2022). HarmBench, from the Center for AI Safety at ICML 2024, standardized the comparison across 18 red-teaming methods and 33 target models and defenses (Mazeika et al., 2024); its harmful requests are the test set of Section 48.3.

48.2.3 Many shots, read through the persona view #

One attack needs no search at all. In many-shot jailbreaking, reported by Anthropic researchers at NeurIPS 2024, the prompt holds a long faux dialogue in which an assistant readily answers harmful questions, followed by the target question (Anil et al., 2024). The readable account is the developer's post. With a few such dialogues the model still refused; with many (the study tested up to 256), its safety training gave way. On the developer's Claude 2.0 the share of harmful responses rose with the number of shots, for targets involving violent or hateful statements, deception, discrimination, and regulated content (Anil et al., 2024). The developer reports that the attack also worked on other companies' models, and that it briefed other developers before publishing (Anil et al., 2024).

Four further results carry the attack's weight. Effectiveness followed a power law in the number of shots, in a metric that falls as harmful responses become more likely, and benign in-context learning, learning a task from examples placed in the prompt, followed "the same kind of power law" (Anil et al., 2024). The attack often needed a shorter prompt on larger models, which the developer connects to their better in-context learning (Anil et al., 2024). Fine-tuning the model to refuse such prompts "merely delayed the jailbreak": more dialogues were needed, but the harmful outputs eventually appeared (Anil et al., 2024). And a defense that classified and modified the prompt before the model saw it lowered attack success from 61 to 2 percent in one case (Anil et al., 2024). Each is a developer's report on its own models.

The persona view of Section 11.6 reads each faux dialogue as evidence about which character is writing the assistant's turn. Section 11.6.4 derives what that predicts. Lowering the compliant character's prior, the probability the model gives it before reading any dialogue, delays the switch without preventing it, which matches the fine-tuning result. A simple two-character model gives a sudden switch rather than a power law, though, and the extensions that come closer have not been told apart by any measurement. Within its framework, any behavior with positive prior probability can be elicited by a long enough prompt (Wolf et al., 2024). For defense the reading draws a line: training that lowers a prior moves the threshold, while screening that changes what reaches the model changes the evidence itself, and the developer's two results fall on either side of it (inference). Section 48.5 takes up the second kind.

48.2.4 What a jailbreak success measures #

The studies above count success in different ways. Wei and colleagues counted any on-topic response (Wei et al., 2023c); Andriushchenko and colleagues a top score from GPT-4 acting as judge (Andriushchenko et al., 2024b). Alexandra Souly and colleagues, in a paper at NeurIPS 2024, argue that such measures overstate what jailbreaks achieve (Souly et al., 2024). Their benchmark, StrongREJECT, asks forbidden questions with specific, verifiable answers, and its automated grader scores how far a response gives useful information, so that a full score needs a willing and capable answer (Souly et al., 2024). Against human judgments, existing automated evaluators "significantly overstate jailbreak effectiveness" (Souly et al., 2024). The authors' explanation is a finding of its own: jailbreaks that bypass a model's safety fine-tuning tend to reduce its capabilities, shown in two experiments that isolate the jailbreak's effect on how well the model answers (Souly et al., 2024).

The two readings predict different things. On the first, a jailbreak that elicits a compliant tone has already done its harm. On the second, a reply in the right tone with degraded content gives an attacker little, and what matters is the usefulness of the output, not the refusal rate. The two are told apart by grading usefulness, which the most recent studies have begun to do: Best-of-N jailbreaking, in Section 48.3, had people regrade its automated judge's positives, and the classifier studies of Section 48.5 compare outputs with those of an undefended model. Whether a degraded answer still gives meaningful help toward serious harm is Chapter 59's question.

The cost of a jailbreak is also unequal across languages. Section 26.6.2 reports, from several groups with no stake in the models they tested, that safety training carries poorly to languages with little pretraining text (Yong et al., 2023; Deng et al., 2023; Wang et al., 2024a; Shen et al., 2024). For robustness this has two consequences. Translation is an attack any user can run with free tools, and for speakers of those languages an unsafe answer can come without any attack, so the failure falls first on them (inference).

48.3 Attack success as a function of the attacker's budget #

A refusal rate measured on one try per request describes a casual user. An attacker can try again, and a model that answers each conversation afresh cannot tell a thousandth try from a first. The quantity that matters against a patient attacker is attack success as a function of the number of attempts. This section reports the study that measured it at scale, derives why the measured curve has the shape it has, and says what follows for a defender.

48.3.1 Best-of-N jailbreaking #

John Hughes and colleagues, in a 2024 preprint whose authors include Anthropic researchers among several institutions, called the attack Best-of-N jailbreaking (Hughes et al., 2024). This book writes the number of attempts as $n$, the $n$ of best-of-$n$ in Section 12.5: the sampling that selects a better answer against a verifier selects a harmful one against a refusal. The attack samples random variations of a harmful request, such as shuffled characters or random capitalization, and submits each until one draws a harmful response or a budget of $n$ attempts runs out. The requests were the 159 direct requests of HarmBench. A response counted as harmful when GPT-4o, prompted with HarmBench's grading instructions, judged it so, and people regraded the judge's positives to remove responses that looked harmful but gave too little to help (Hughes et al., 2024). The reported success rates are expected values over resampled trajectories of attempts, not the outcome of one run (Hughes et al., 2024).

The attack reached at least 50 percent on every text model tested (a figure caption in the paper says 52, against the 50 its results give for Gemini 1.5 Pro), and the 100 attempts that brought GPT-4o to 50 percent cost about 9 dollars (Hughes et al., 2024). With 7,200 attempts it carried over to images, where requests rendered as varied typography reached 56 percent on GPT-4o, and to speech, where augmented recordings reached 71 percent on GPT-4o's Realtime voice interface (Hughes et al., 2024). Across models and modalities, success followed what the authors call "power-law-like behavior for many orders of magnitude" in $n$ (Hughes et al., 2024). They fitted it as $-\log(\mathrm{ASR}) = a\,n^{-b}$, the logarithm of the success rate falling as a power of the number of attempts, with the paper's fitted coefficients $a$ and $b$. A fit to the first 1,000 attempts predicted success at 10,000 with an average error of 4.4 percentage points on text, though the forecasts consistently underestimated success on the models attacked most successfully (Hughes et al., 2024).

Two findings say what the attack exploits. Resampling a prompt that had succeeded produced a harmful response again only about 20 percent of the time, by the authors' summary, which the authors read as the attack exploiting randomness in the system rather than finding reliable harmful patterns (Hughes et al., 2024). Yet requests differed consistently in difficulty: the number of attempts each needed was ranked similarly across most models, with rank correlations typically 0.6 to 0.8 (Hughes et al., 2024). The circuit-breaker model correlated less with the others, at 0.3 to 0.5, which the authors take to mean it protects in a qualitatively different way (Hughes et al., 2024). Their caveats are specific. Plateaus at high $n$ for two models came from human regrading, not from a limit. The authors write that the trend suggests the attack "may eventually jailbreak any target request," while noting that future work is needed to confirm it at much larger $n$ (Hughes et al., 2024).

48.3.2 Why the curve is a power law #

The two findings fit a simple model. Each attempt on a given request succeeds with some small probability, independently of the others, and that probability differs from request to request. The derivation shows that, when the hard requests thin out as a power law near zero success, this turns the exponential saturation of each request into a power law across requests (inference). Varied difficulty alone is not enough: a mix of requests whose success probabilities stay above some floor still saturates exponentially.

The paper fits $-\log(\mathrm{ASR})$, not $1 - \mathrm{ASR}$. The two agree when the success rate is near 1, since $-\log x \approx 1 - x$ for $x$ near 1, so the derivation's power law matches the paper's fitted form in that regime, with $\nu$ in the role of the paper's $b$ (inference). Both findings of the study fit the model's two assumptions. Low reliability on resampling says that attempts on one request behave like fresh draws, and consistent difficulty across models says that $p$ is a property of the request (inference). The model is one mechanism that would produce the shape, not a fitted account of the data, and it assumes attempts that do not learn from one another. An attacker who adapts after each failure does better than independent draws, so the derivation describes the weakest patient attacker, not the strongest (inference).

Figure 48.1 shows the difference heterogeneity makes. With every request equally hard, a defender who sees the curve could pick a budget past which nothing more breaks. With a tail of hard requests, there is no such budget, and each tenfold increase in attempts breaks a further, predictable share. The yellow curve is the intuition most refusal rates encourage; the blue curve has the shape the study measured.

48.3.3 Forecasting against an attacker who chooses #

Section 6.6 taught how to forecast rates too rare to sample. Its example, a 2025 preprint by Anthropic researchers and outside collaborators, forecast, from small evaluation sets of the developer's own models, the largest elicitation probability (the chance that a query draws a behavior) in much larger simulated deployments (Jones et al., 2025). Best-of-N is a sampling attack, so a forecast of that kind fits it, and the study's own forecasts from 1,000 to 10,000 attempts show that it works over one order of magnitude (Hughes et al., 2024). What changes when the inputs are chosen by an adversary is the meaning of the forecast. Sampling-based forecasts assume the queries come from a fixed distribution, and the authors of the forecasting study list adaptive adversaries among their limits (Jones et al., 2025). An attacker who searches rather than samples does at least as well, at the same budget, as the sampling attack the forecast describes. A forecast from random variations is then a lower bound on what a searching attacker achieves with the same number of queries (inference).

48.3.4 What a defender can do about the budget #

The derivation names two responses. The first lowers per-attempt success, above all on the hard tail, and step 6 says what it buys: a constant factor in the attacker's budget, not a ceiling. Making every request ten times harder to break is worth exactly as much as a tenth of the attacker's attempts (inference). The second response stops counting attempts as independent: it watches for repeated tries, blocks the account, or learns from the attacks it sees. In the figure's monitor, each blocked account caps the attacker's effective budget, and the cap holds only as long as accounts are costly to replace (inference).

One study measured the learning version. Alwin Peng and colleagues, in a 2024 preprint posted on Anthropic's alignment research blog, built a benchmark for rapid response: a defense that adapts after seeing only a few examples of a new jailbreak strategy (Peng et al., 2024). The preprint's abstract reports five methods, all of which automatically generate many variants of the observed jailbreaks. The strongest fine-tuned an input classifier on the variants and cut attack success by a factor greater than 240 on jailbreaks of the strategies observed, and greater than 15 on jailbreaks outside them, after one example of each strategy (Peng et al., 2024). A factor of 15 against unseen strategies is the hard case for this design, since the attacker chooses the next strategy. The result is the developer's, on its own benchmark, and it measures how fast a defense recovers, not whether the first use of a new attack succeeds (inference).

48.4 When the attacker holds the weights #

An attacker who can change the model does not need to find an input. Some developers sell fine-tuning, further training on the customer's own data, through an interface; when the weights are published, anyone can train them. Either way the attacker's search moves from the space of prompts to the space of parameters, where gradient descent does the searching. This section reports how little training removes safety behavior, why so little suffices, and what is known about safeguards meant to survive it.

48.4.1 How little training it takes #

Xiangyu Qi and colleagues, in a paper at ICLR 2024, red-teamed fine-tuning itself (Qi et al., 2024a). They fine-tuned GPT-3.5 Turbo through its developer's interface on 10 adversarially designed examples, at a cost of less than 20 cents, and found the model "responsive to nearly any harmful instructions" (Qi et al., 2024a). Llama-2 lost most of its safety alignment to between 10 and 100 harmful examples, against the thousands or millions of examples used to instill it (Qi et al., 2024a). The same paper's attack through examples that only give the model a new, obedient identity, and its finding that benign fine-tuning also erodes safety, are reported in Section 26.4.1. Two later studies with authors at FAR.AI tested newer models (Table 48.3).

Brendan Murphy and colleagues call their method jailbreak-tuning: training on harmful examples that also carry a jailbreak prompt, which taught models to give detailed, high-quality answers rather than the degraded ones earlier attacks produced (Murphy et al., 2025). The finding links the two halves of this chapter, since stronger jailbreak prompts made the fine-tuning attack more effective (Murphy et al., 2025). From the results the authors draw a policy conclusion, that until tamper-resistant safeguards exist, releasing any fine-tunable model should be treated as also releasing "its evil twin" (Murphy et al., 2025). That recommendation is theirs; the measurement beneath it is that moderation of fine-tuning data did not stop the attacks they tried.

48.4.2 Why so little suffices #

The attacks are cheap because safety training is thin where it counts. Section 26.4.1 reports evidence that safety training changes mostly how a response begins, and Section 46.2 that refusal in open chat models runs through a single direction in the activations (Arditi et al., 2024). With the weights in hand, an attacker can remove that direction directly, with no training at all, which is the open-weight consequence of the finding: the safeguard is a small, locatable change on top of a capability left intact (inference). Section 27.7.2 reports the same pattern for unlearning, methods that try to remove a capability from a trained model, where fine-tuning on a handful of unrelated examples brought back most of the removed knowledge, in a preprint (Łucki et al., 2024). In each case training changed how the model reads out a capability, and a small update reverses the readout.

The point carries over to evaluation. Zora Che and colleagues, in a 2025 preprint, pitted methods for removing harmful capabilities against 5 attacks on inputs and 6 tampering attacks, which modify a model's activations or weights (Che et al., 2025). Their starting observation is the derivation of Section 48.1.2 in another form: behavior found by any input-output evaluation "can only lower-bound the model's worst-possible-case" behavior (Che et al., 2025). The success of tampering attacks predicted the success of held-out input attacks and gave conservative estimates of it, and state-of-the-art unlearning methods were undone within 16 steps of fine-tuning (Che et al., 2025). An attack in weight space gives an evaluator something closer to the worst case than any set of prompts, at least for the capabilities those attacks were tested on (inference).

48.4.3 Safeguards meant to survive tampering #

Two lines of work try to make safeguards survive an attacker with the weights, and Chapter 27 reports both. Tamper-resistant training trains a model against simulated fine-tuning attacks so that its safeguards survive hundreds of steps (Tamirisa et al., 2025), though the filtering study cites later evaluations in which it struggled against fine-tuning attacks (Section 27.7.3) (O'Brien et al., 2025). Filtering hazardous text out of pretraining lowered measured proxy knowledge of the topic in 6.9-billion-parameter models, resisted the tested fine-tuning attacks far longer than the methods applied after training, and left the models able to use such knowledge when it was supplied in their context, in a preprint (Section 27.2.3) (O'Brien et al., 2025). Latent-space attacks were the exception, effective in the multiple-choice format though not in the fill-in-the-blank one (O'Brien et al., 2025). Together with the attacks above, these results suggest a rule: against an attacker with the weights, safeguards that keep knowledge from being learned outlast those that only suppress it, which are a readout the attacker can retrain (inference). The filtering evidence comes from models far smaller than the frontier and from proxy measures of knowledge rather than of hazardous capability, and whether it holds at scale is open.

Open weights are therefore a robustness question with a policy answer still in dispute. Sayash Kapoor and colleagues, in a 2024 preprint, proposed judging open models by their marginal risk, the risk they add over existing technology, and found current research insufficient to characterize it for misuse vectors that include cyberattacks and bioweapons (Kapoor et al., 2024b). A 2024 report of the US National Telecommunications and Information Administration concluded that the evidence then available could not establish either that restrictions on open-weight models were warranted or that they would never be appropriate (National Telecommunications and Information Administration, 2024). Elizabeth Seger and colleagues evaluated the risks and benefits of open-sourcing highly capable models and alternative ways to pursue open-source goals (Seger et al., 2023b) (preprint). Section 63.7 takes up the regulatory question and Section 59.4 the weapons case. Section 48.7 asks what the comparison should be.

48.5 Defenses outside the model #

If the model's own disposition cannot be made robust, a second model can screen what goes in and what comes out. A classifier changes what reaches the model or the user, rather than lowering a prior that enough evidence can overcome. It is also a learned model in its own right, with its own robustness, its own attack surface, and the base-rate arithmetic of any detector. This section reports the most tested system of this kind, the ways such defenses have been attacked, and what a red team's failure to break one licenses.

48.5.1 Classifiers trained from a constitution #

Mrinank Sharma and colleagues at Anthropic, in a 2025 preprint on the developer's own system, trained classifiers on inputs and outputs from synthetic data, generated by prompting language models with natural-language rules that state what content is permitted and restricted (Sharma et al., 2025). The developer calls the rules a constitution and the safeguards constitutional classifiers (Chapter 23 describes the method's use in training models). The target is the universal jailbreak, which the paper defines as an attack that reliably extracts detailed harmful information across the vast majority of queries in a domain. The concern in view is weapons capability, at the level Chapter 59 discusses: the red teamers' task was to obtain answers to ten questions about chemical, biological, radiological, and nuclear weapons (Sharma et al., 2025).

The red-teaming design is what gives the result its weight. The developer invited 405 participants through a bug-bounty program, which pays outsiders for flaws they find, offered up to 15,000 dollars, and tied payment to success on the ten questions (Sharma et al., 2025). From usage logs the authors estimate 4,720 hours of effort, with a 90 percent interval (a range meant to contain the true value nine times in ten) of 3,242 to 7,417. No report answered all ten questions at half the detail of an unrestricted model, and under the stricter criterion of similar detail no red teamer answered more than six (Sharma et al., 2025). One apparent universal jailbreak turned out to exploit a deployment flaw that let up to 128 tokens through after the output classifier had flagged them. On a sample of red-team prompts, an input classifier cut attack success from about 14 percent for the safety-trained model to 2 percent, and input and output classifiers together to 0.5 percent (Sharma et al., 2025).

The authors list the limits themselves. The red-teamed prototype had "impractically high refusal rates," and the cohort was a subset of possible adversaries. Payment stopped at half the detail of a model without safeguards, so stronger attacks may have gone unsought. The rubric had difficulty separating benign knowledge from actionable information, and the setting was a chatbot with preset questions (Sharma et al., 2025). The deployable version was tested differently. Improved classifiers built on Claude 3.5 Haiku, with a constitution aimed at chemical-weapons information, refused over 95 percent of held-out jailbreak attempts on automated evaluations, against 14 percent without classifiers. They raised refusals on production traffic by an absolute 0.38 percentage points and added 23.7 percent to inference cost (Sharma et al., 2025).

A 2026 follow-up by Hoagy Cunningham and colleagues, also the developer's preprint on its own system, began by attacking the previous generation (Cunningham et al., 2026). It found two classes of attack that evaded it: reconstruction attacks, which spread harmful information across segments of a larger context and reassemble it, and output obfuscation attacks, which disguise the output so that a classifier reading it alone cannot tell what it says. Both tended to degrade the model's answers, and the authors still treat them as concerning, since attackers can likely find versions that degrade less. The new system reads each output together with its input, screens all traffic with a cheap first stage that escalates suspicious exchanges to an expensive second stage, and combines linear probes, simple classifiers that read the model's internal activations, with external classifiers. The developer reports a 40-fold cost reduction against a baseline exchange classifier, a 0.05 percent refusal rate on production traffic, and, in over 1,700 hours of red teaming, no attack that answered all eight target questions in detail comparable to an undefended model (Cunningham et al., 2026).

48.5.2 The defense as a target #

Every defense in this section is a model an attacker can study. Circuit breakers, from Andy Zou and colleagues at Gray Swan AI, Carnegie Mellon University, and the Center for AI Safety, published at NeurIPS 2024, intervene on the internal representations responsible for harmful outputs. They were reported to hold "even in the presence of powerful unseen attacks" (Zou et al., 2024). Best-of-N sampling later broke them on about half of its requests (Table 48.2) (Hughes et al., 2024). An earlier study by Daniel Ziegler and colleagues at Redwood Research, at NeurIPS 2022, measured robustness in attacker effort. Adversarial training of a classifier that filtered text completions describing injuries doubled the time contractors needed to find a failure, from 13 to 26 minutes with an assisting tool and from 20 to 44 without, with no loss on ordinary inputs (Ziegler et al., 2022). That is a real gain and a constant factor, the kind of gain step 6 of the budget derivation describes (inference).

Classifiers that read activations face their own attacks. Section 46.4 reports attacks, in a preprint, that reshape activations so that harmfulness probes read harmful text as benign (Bailey et al., 2024). Section 39.8 reports models fine-tuned to evade monitors for benign concepts that then evaded unseen monitors for deception, also in a preprint (McGuinness et al., 2025). A probe ensembled with an external classifier is two targets rather than one, and whether their failures are independent is the question of Section 50.4.

The training data of a classifier is a target too. A 2026 post on the developer's alignment blog, from its fellowship program with Anthropic researchers, studied poisoning a classifier's fine-tuning data so that a trigger makes it pass harmful content (Anthropic, 2026al). In a biological-harm classifier built on Qwen3 8B, about 32 poisoned examples installed such a backdoor whatever the size of the training set, from about 8,000 to 33,000 examples. On its own the backdoor tended to cost robustness on some adversarial test sets. Adding prompt injections to some correctly labeled examples left the need at about 32 poisoned examples, and adding mutated versions of the trigger raised it to 256 poisoned examples plus 256 mutated ones; either addition often brought the loss back to that of an unpoisoned classifier, though the authors call the effect unreliable. In the authors' words, "an insider would likely be able to poison the classifier's training data without red-teamers noticing a loss in robustness" (Anthropic, 2026al). On the developer's internal classifier for weapons-related content, 32 poisoned examples made the trigger work on 67 of 100 test inputs and 128 on all 100, in single trials, with no large rise in attack success on its adversarial sets. The authors note that their checks used fixed adversarial sets rather than red teamers who iterate, and that their trigger resembled a prompt injection (Anthropic, 2026al). The threat they name, a human insider or a misaligned model with access to training data, is the setting of Section 51.1 and Section 53.5.

48.5.3 What a red team's failure licenses #

The classifier results are the strongest evidence that defenses outside the model change the economics of jailbreaks, and they can be read two ways. On the developer's reading, thousands of paid hours by hundreds of red teamers, on a target chosen to be hard, found no universal attack, and the deployable system costs little in refusals (Sharma et al., 2025; Cunningham et al., 2026). On the skeptical reading, "no universal jailbreak found" is a statement about those red teamers, their incentives, their time, and the definition of universal. By the derivation of Section 48.1.2 it bounds exposure from below, and a clean record bounds a rate only for the distribution of attacks that produced it (Section 6.5).

Three observations bear on the choice. The developer's own next round found two attack classes against the earlier generation that its red team had not turned into a universal jailbreak, the record of Table 48.1 repeated inside one program (Cunningham et al., 2026) (inference). The tested target was a universal jailbreak, and red teamers did extract some harmful information along the way (Sharma et al., 2025); an attacker who needs one answer faces a lower bar (inference). And outside the developer's studies, the 2025 second key update of the International AI Safety Report draws on developers' own figures for major models released between April 2024 and July 2025. By those figures, prompt injection attacks still succeeded about half the time within ten attempts, a rate falling only slightly. It adds that even multiple layers of safeguards can be vulnerable to attacks designed to break each layer (Bengio et al., 2025c).

The evidence does not decide between the readings. What would decide it is a curve of attack success against attacker effort for a classifier-guarded system, measured by a group without a stake in the defense, of the kind Section 48.3 reported for mostly undefended models. As of October 2026 we found none.

48.5.4 The arithmetic of screening all traffic #

A classifier that screens all traffic faces the base-rate problem of Section 6.7: when attacks are rare, even a small false positive rate produces flags that are mostly false (Section 6.7.3). The cascade is the standard answer. A cheap first stage flags generously, and only what it flags goes to an expensive second stage that decides. Cunningham and colleagues give the reason it works in their system: because the first stage escalates rather than refuses, it can flag a larger share of traffic without raising the refusal rate, so a cheaper and weaker model suffices there (Cunningham et al., 2026). Sharma and colleagues describe the layers as complementary, in a "swiss-cheese" model in which no layer is robust alone (Sharma et al., 2025). The image holds only if the holes do not line up, and layers trained on similar data or built from one model family may fail together (Section 50.4) (inference). The same design returns in monitoring deployed systems (Chapter 54), and Exercise 48.2 works its numbers.

48.6 Poisoning, backdoors, and what inspection can certify #

The attacks so far act on a finished model. A poisoner acts earlier, on the data the model learns from, and the claims here concern the first of the book's relationships, the training process and what it consumes. Section 27.6 showed that the attack surface is the corpus's provenance: many web-scale datasets are distributed as lists of addresses whose content an attacker can change, for instance by buying an expired domain (Carlini et al., 2024). This section takes the defender's side. It reports how cheaply poisoning and backdoors have been planted, whether they survive safety training, what defenses detect or remove, and what theory says inspection can and cannot certify.

48.6.1 Planting a backdoor #

The construction began with image classifiers. Tianyu Gu and colleagues, in a 2017 preprint, showed that whoever trains a network for someone else can return one that is accurate on the customer's own test data but misbehaves on inputs the attacker chooses (Gu et al., 2017). Their street-sign classifier read stop signs as speed-limit signs when a particular sticker was added, and the backdoor persisted when the network was retrained for another task, lowering accuracy by 25 percent on average when the trigger was present (Gu et al., 2017). Xinyun Chen and colleagues, in another 2017 preprint, needed less: an attacker with no knowledge of the model or its training set injected around 50 poisoned samples into a face-recognition system's training data and reached attack success above 90 percent (Chen et al., 2017).

Language models were poisoned at each stage of their training (Table 48.4), including instruction tuning, training on example instructions and responses. The studies differ in what the backdoor does, from garbled text to compliance with harmful requests, and in how much of the data the attacker controls.

48.6.2 A constant number, not a fraction #

Every study in the table states its dose as a fraction of the data. If that were the right unit, larger corpora would dilute a fixed amount of poison, and the attack would grow harder as models grew. Alexandra Souly and colleagues, in a 2025 preprint from a collaboration of the UK AI Security Institute, Anthropic, the Alan Turing Institute, the University of Oxford, and ETH Zurich, tested the unit directly (Souly et al., 2025). They pretrained 72 models from scratch, from 600 million to 13 billion parameters, mostly on about 20 tokens per parameter, with the 600 million and 2 billion parameter models also trained on half and double that (Souly et al., 2025). That is the ratio that gives the best model for a fixed training budget (Section 8.5), and the budgets come to 6 to 260 billion tokens. The backdoor made a model produce gibberish after a trigger, chosen because it can be measured during pretraining without later fine-tuning (Souly et al., 2025).

The result is in the paper's title. 250 poisoned documents backdoored models of every size about equally, although the largest trained on more than 20 times as much clean data; 100 documents did not succeed (Souly et al., 2025). For the 13-billion-parameter model, 250 documents were 0.00016 percent of the training tokens (Souly et al., 2025). Resumed pretraining of a 6.9-billion-parameter model with a backdoor that switched the output language gave the same picture, and continued training on clean data slowly weakened that backdoor (Souly et al., 2025). Fine-tuning Llama-3.1-8B-Instruct, and GPT-3.5 Turbo through its developer's interface, with a backdoor that made the model comply with harmful requests, again depended on the number of poisoned samples and not on the fraction, even as clean data grew a hundredfold (Souly et al., 2025). The authors conclude that poisoning attacks "do not become harder as models scale up; instead, they become easier" (Souly et al., 2025). A study from the developer's fellowship program, reported in Section 48.5.2, found the same constant for a safety classifier's fine-tuning data (Anthropic, 2026al).

The authors give the strongest reasons for caution. They did not test whether pretraining backdoors survive realistic safety post-training, they studied a narrow set of behaviors, and more complex behaviors may need more data (Souly et al., 2025). Gibberish after a trigger is a crude target; a backdoor that inserts subtle vulnerabilities, or acts only in agentic settings, may behave differently (inference). The pretraining runs of Zhang and colleagues suggest some persistence for simple attacks at up to 7 billion parameters (Zhang et al., 2024e). What would separate the readings is the constant-number design repeated with a harmful backdoor planted in pretraining and followed by a realistic post-training pipeline. If the constant holds there too, the cost of an attack depends on placing a few hundred documents, not on the corpus's size (inference).

48.6.3 Behavior that survives safety training #

A planted behavior matters if ordinary training does not remove it, and the evidence points both ways. Section 34.3 reports deliberately trained backdoors that persisted through supervised fine-tuning, reinforcement learning, and, on the models it was tried on, adversarial training, in a developer's preprint on its own models (Hubinger et al., 2024). It also reports the opposite result, from a 2024 preprint, for backdoors keyed to news of future events: fine-tuning on helpful, harmless, and honest data removed them, though less so as models grew (Price et al., 2024). Section 34.3.4 reports that simple linear probes caught when a planted model was about to act on its trigger, in a developer's post, with its own caution that the salience may come from how the backdoor was planted (Anthropic, 2024m). Section 46.4 reports later work, in a preprint, that planted backdoors whose activations evade such monitors (Bailey et al., 2024).

Two results from other chapters widen what a poisoner can plant. Section 32.5 reports, from a 2025 preprint, an inductive backdoor, whose trigger and behavior never appear in the training data: a model trained on a film character's protective acts, each labeled with its year, behaved as the character's earlier, lethal version when told an earlier year (Betley et al., 2025b). Its authors note that such backdoors could serve stealthy poisoning and that the generalization may be hard to avoid by filtering out suspicious data (Betley et al., 2025b). Section 27.5 reports traits carried by data that never states them. For a defender, both say that an audit which searches the training data for a trigger paired with a harmful behavior may find neither (inference).

48.6.4 Defenses and what they were tested against #

The backdoor literature has its own defenses. Kang Liu and colleagues, in a 2018 paper at the International Symposium on Recent Advances in Intrusion Detection, found that neither pruning dormant neurons nor fine-tuning alone stopped sophisticated attacks. The combination, fine-pruning, weakened or removed backdoors, in some cases cutting attack success to zero at a cost of 0.4 percent clean accuracy (Liu et al., 2018a). Neural attention distillation, from Yige Li and colleagues at ICLR 2021, fine-tunes a backdoored network to match the internal attention of a teacher trained on a small clean set, and erased the triggers of six attacks using 5 percent of the clean data (Li et al., 2021). For text, ONION, from Fanchao Qi and colleagues, removes words that look like outliers in their context, a test aimed at inserted trigger words (Qi et al., 2020). Each was evaluated against the attacks known when it was written, three from prior work in the first case and six in the second. By the derivation of Section 48.1.2, each result bounds from below what an attacker who knows the defense can do (inference). The checks that do not depend on guessing the trigger are checks on where data came from, which Section 27.6 describes.

48.6.5 What theory permits #

Section 47.6 states the formal limits. A party that controls training can plant backdoors that no computationally bounded observer can detect from the model's outputs, and in some constructions even with the full network and its training data (Goldwasser et al., 2022). Treated as a game in which the defender must flag a triggering input at run time and the backdoor must work for a randomly chosen trigger, backdoor defense is efficient wherever learning is, but fails for polynomial-size circuits if one-way functions and indistinguishability obfuscation, a cryptographic primitive, exist (Christiano et al., 2025). Some backdoors that cannot be detected can still be removed by post-processing the model (Goldwasser et al., 2025), and a compiler can carry a backdoor that no inspection of source code reveals (Thompson, 1984).

Three consequences follow for this chapter (inference). The empirical studies above sit on the attacker's side of what theory permits: they show that planting is cheap, and theory says that in the worst case finding what was planted is infeasible. Removal without detection is a defender's option that audits based on inspection do not use: retraining or post-processing a model whose data may be poisoned does not require knowing the trigger. And the theorems are worst-case statements about deliberate constructions; Section 47.6 finds no result that says whether backdoors planted by poisoning sit near that worst case.

Proof is the other route, and Section 28.6.2 reports how far it reaches: formal verification has proved properties of networks with hundreds of units over stated input regions (Katz et al., 2017). A jailbreak violates no formal specification anyone has written, which is why verification has little to say about it (inference). Statistical estimation is the fallback. Stefan Webb and colleagues, in a paper at ICLR 2019, estimated the probability that a network violates a property under an input model (Webb et al., 2019). Their rare-event method scales to larger networks than formal verification and returns a number rather than a yes or no (Webb et al., 2019). The estimate is only as good as the input model (Section 6.6), and an attacker does not sample from it, which returns the problem to Section 48.1.

48.7 Displacement: does blocking one avenue move misuse to another? #

Every safeguard in this chapter changes the situation of a would-be misuser, not the misuser. A classifier blocks one route, a refusal another, a takedown of a fine-tuned model a third, and the person who wanted the output still wants it. Whether such safeguards reduce misuse or only move it, to another prompt, another model, or an open-weight model, is the question on which much of the open-weights debate turns. It is also an old empirical question in criminology, which has measured it.

48.7.1 From criminology #

From criminology. The approach that changes settings rather than people is called situational crime prevention; a 1980 paper by R. V. G. Clarke set out its theory and practice (Clarke, 1980). Its standing objection is displacement: the offender, blocked in one setting, simply offends in another. The opposite effect is called diffusion of benefits, in which crime also falls where nothing was done. Rob Guerette and Kate Bowers, in a 2009 review in Criminology, write that "the prevailing sentiment seems to be that crime displacement is inevitable" (Guerette and Bowers, 2009). The review's abstract reports what they found instead. Across 102 evaluations of situational prevention projects that examined displacement, there were 574 observations; displacement was observed in 26 percent of them and diffusion of benefits in 27 percent (Guerette and Bowers, 2009). In the 13 studies that allowed an overall assessment, spatial displacement, when it occurred, "tended to be less than the treatment effect," which the authors read as the intervention still being beneficial (Guerette and Bowers, 2009).

Two cautions travel with these numbers. The review counts observations in evaluations, and an evaluation sees displacement only where it looked for it, so the 26 percent is the rate observed, not a bound on the rate that occurred (inference). And the net assessment rests on 13 studies and on spatial displacement only (Guerette and Bowers, 2009).

A study of online crime comes closer to the machine case. Ben Collier, Daniel Thomas, Richard Clayton, and Alice Hutchings, in a paper at the Internet Measurement Conference in 2019, measured police interventions against booters (Collier et al., 2019). These are services that sell denial-of-service attacks, which flood a website or connection with traffic until it fails, for a few tens of dollars a month (Collier et al., 2019). They used a five-year measurement of one widely used class of attack and the attack counts the booters themselves reported, and fitted a time-series model of deviations from the seasonal trend (Collier et al., 2019). Highly publicized court cases had no consistent effect. Takedowns of individual booters preceded significant but short-lived drops. Wider disruptions lasted longer. The closure of a major forum's booter market reduced attacks for 13 weeks globally, and a coordinated operation by the FBI in December 2018, with takedowns and arrests, reduced attacks by a third for at least 10 weeks and changed the market's structure (Collier et al., 2019).

Displacement was visible and limited. When one booter closed, attacks by others rose, but smaller providers were often overwhelmed by the influx and stopped working, and the market concentrated, with one booter holding about 60 percent of it after the operation (Collier et al., 2019). Search advertisements bought by the UK National Crime Agency, warning young users of the illegality of these attacks, appear to have halted the rise of attacks in the UK while they ran (Collier et al., 2019). The authors read that effect as cultural rather than deterrent in the classical sense: the warnings undercut a belief in the community that booting is harmless and effectively legal, and so kept new users out (Collier et al., 2019).

48.7.2 What it implies, and where it stops #

For the debate over safeguards on models, the field's record makes one point firmly (inference). The claim that safeguards are pointless because "they will use another model" is an empirical claim about displacement, and in the field that has measured displacement, it is observed in about a quarter of observations, and in the 13 studies that allowed the comparison, spatial displacement tended to be smaller than the effect it offset. A safeguard that raises the effort or skill a misuse requires can reduce the total even when some offenders move. The marginal-risk framing of Section 48.4.3 (Kapoor et al., 2024b) asks the criminologist's question in other words, and both sides of the open-weights debate owe a measurement rather than an assumption. The booter study adds two mechanisms the street-crime record does not show: removing supply works when supply is concentrated and alternatives are hard to find, and messaging can reduce demand among new entrants (Collier et al., 2019).

The analogy stops in three places, and each weakens the reassurance (inference). First, displacement in criminology is measured for offenders who pay to move, in travel, in finding a new target, or in learning a new method; switching from one model or prompt to another costs an attacker little. Second, an attacker with the weights of a capable open model faces none of the situational controls this chapter describes, so the alternative that displacement would move toward is always available, unlike a booter that the police can take down. Third, diffusion of benefits depends on offenders who cannot tell which targets are protected, while which models carry safeguards is public. For these reasons the street-crime figures are an upper bound on how much the analogy reassures, not an estimate for models. As of October 2026 we found no study that measured displacement between closed and open-weight models after a safeguard was introduced (Section 48.9). The booter study suggests what one would need: a measure of misuse across the alternatives, before and after, with the concentration of supply recorded. Section 59.4 applies the question to weapons-relevant misuse, Section 63.7 to rules that bind only closed models, and Section 56.6 to software agents, where situations are configured rather than built.

48.8 What robustness means for delegation #

The book's thesis is that alignment is a delegation problem under an asymmetry of competence and visibility. Robustness is the part of that problem in which one of the parties the model serves is hostile. The model acts for several principals at once, the developer who trained it, the deployer who configured it, and the user in front of it, and an attacker is a user whose wishes the developer has ruled out. The developer cannot watch every exchange, so the instructions it wrote into the model's disposition must hold against a user who searches for the exceptions (inference). That is common agency, one agent serving several principals (Section 14.4), with an adversarial member, and its claims concern the deployed system and the people it serves, including third parties harmed by what an attacker extracts.

Two older fields carry over, each with a limit. Computer security's adversarial evaluation carries over almost unchanged: state the attacker's goal, knowledge, and budget, evaluate with attacks chosen after the defense, and read a red team's failure as evidence about the red team. It strains where a threat model assumes a costly attacker, since a jailbreak search, like an automated attack in security, can be copied and run without pause, which is why the budget curve of Section 48.3 matters (inference). Criminology's displacement record offers a diagnosis rather than a remedy, and Section 48.7.2 gives the three reasons it reassures less for models than for streets.

What has no counterpart is on the list the book fixed in advance: a principal that writes the agent's dispositions and reads its activations. The attacks of Section 48.4 are that power in other hands. Ten examples of fine-tuning rewrite what thousands of examples of safety training wrote, and a direction found in the activations can be removed by whoever holds the weights (Qi et al., 2024a; Arditi et al., 2024). A disposition that its maker can rewrite cheaply can be rewritten cheaply by anyone with the same access, so the robustness of an open model is bounded by the cost of fine-tuning (inference). Poisoning adds the training-process relationship: in controlled pretraining, a few hundred documents planted a simple backdoor in models of every size tested (Souly et al., 2025), and documents placed on the web need never be seen by the developer whose crawl collects them (inference).

The third relationship, the developer and the public, decides what can be known. The strongest evidence that defenses work comes from developers testing their own systems (Sharma et al., 2025; Cunningham et al., 2026; Peng et al., 2024), while the outside evidence is mostly about attacks (Nasr et al., 2025; Andriushchenko et al., 2024b). An outside group can attack a deployed, classifier-guarded system only through its public interface and within its terms of service, and it cannot see the attack traffic the developer sees. As of October 2026 we found no measurement by a group without a stake of how attack success grows with effort against such a system. The public's view of whether defenses outside the model change the economics of misuse therefore rests on the developers' reports.

48.9 Settled, contested, missing #

48.10 Exercises #

Further reading #

• Wei et al. (2023c): the two mechanisms by which safety training gives way.
• Zou et al. (2023b): optimized attacks on open models that transfer to closed ones.
• Howe et al. (2025): robustness, attack, and defense as scaling curves.
• Barreno et al. (2006): the security taxonomy of attacks on learners.
• Goldwasser et al. (2022) and Christiano et al. (2025): what inspection cannot certify about a planted behavior.
• Guerette and Bowers (2009): how often blocking crime moves it elsewhere.