Draft complete but not yet reviewed; expect revisions

Weapons and Physical Agents

In brief

Ideas

The question is uplift: whether model access raises someone's chance of causing mass harm over what they could already do. The independent trial could reliably detect only large effects, and a developer's trials disagree with it (§59.1).
Criminology finds blocking an opportunity sometimes displaces crime, by less than it prevents where that was weighed, but switching models is nearly free and the harm here is too rare to have been counted (§59.4).
Dual-use biology's governance offers containment matched to risk, prior review, and controls at the physical chokepoint; it stops where weights copy at no cost and developers take no public money (§59.5).
In one developer's robotics evaluations, models went from helping people to doing the task largely alone; each such step moves the decision about physical harm further from a person (§59.6).

Evidence

Settled: safeguards added after training can be removed from models with available weights or fine-tuning interfaces, including on weapons requests, as an academic group and later others on newer models found (§59.3).
Contested: uplift toward biological harm. An independent trial with 2023 models found no significant difference but could reliably detect only large effects; a developer's trials with Claude 4 models report uplift, scores unpublished (§59.2).
Reported: filtering dual-use biology out of pretraining left models near chance on a proxy benchmark through up to 10,000 adversarial fine-tuning steps, from one group at 6.9 billion parameters (§59.3).

Open

Crux: does a frontier model raise a non-expert's chance of operational success at the stage that limits the chain, not just proxy scores? An adequately powered, independent, blinded trial would settle it (§59.1).
Missing: an independent measure of how often deployed classifiers catch real misuse, a demonstration that pretraining filtration holds at frontier scale, and outside adaptive attacks on current robot safeguards beyond one 2024 study (§59.8).

Some knowledge serves both healing and harm: the biology behind a vaccine, the chemistry of a pesticide, the engineering of a drone. A model that is helpful to its user with such knowledge can be harmful to everyone else, which makes this domain the limiting case of the plural principal (Section 1.4). Anton Korinek and Avital Balwit's distinction names it: a system can be directly aligned, pursuing the goals of its operator, without being socially aligned, taking into account the welfare of everybody the system affects (Korinek and Balwit, 2023). A system aligned to a user who means harm is then the problem rather than the solution. Chapter 58 asked that question for software, where the capability that finds a flaw can also fix it. This chapter asks it where no symmetry of that kind is available: the question laboratories and governments ask is not whether a model will discuss weapons, but whether access to it raises the probability that someone causes mass harm, compared with what the same person could have done without it. That comparison is uplift, a term the evaluation and policy community uses in this sense, and the chapter uses it in no other.

The chapter covers that question only at the level laboratories and governments have published: how an uplift claim is framed, how the trials are designed and what they report, which safeguards are deployed and how their strength has been measured, whether a safeguard reduces harm or moves it, and what the older governance of dual-use biology offers. It contains no technical content about producing, acquiring, enhancing, or deploying any weapon, and it reports no study at a finer grain than its authors published. It then turns to physical agents, models that control robots and drones, where evaluations of how much a model helps a person and how much it does alone appeared in 2025 and 2026. Most claims concern the deployed system and the third parties its users can reach. The rest concern the developer and the public, because the decisions to deploy, restrict, or release weights are the developer's and are made on the public's behalf, and because most of the evidence below is a developer's report on its own system.

Sources cited in the introduction 1

Korinek and Balwit (2023) Aligned with Whom?

59.1 What an uplift claim asserts #

An uplift claim is not a claim about a conversation. It is a counterfactual, a comparison between what happened and what would have happened under some other condition, and it is made over a population of people who might try. Three things have to be fixed before it means anything: who the actor is, what the actor could have used instead, and what counts as success. The first is a threat actor in the security sense, a person or group with a goal and some resources. The second is the baseline, the technology already available, which for information is a search engine and the published literature. The third is the hardest, because the thing to be measured is operational success in the world rather than a score on a test.

Sayash Kapoor and colleagues, in a 2024 preprint on open foundation models, make the comparison the center of the analysis and give it a name. The marginal risk of a model is the risk it adds relative to pre-existing technologies or to a closed model, rather than relative to zero (Kapoor et al., 2024b). Their framework has six steps: identify the threat and the actors, state the risk that exists without the model, state the defenses that exist without it, give the evidence of marginal risk, ask how easily new defenses could be added, and state the uncertainties and assumptions (Kapoor et al., 2024b). Surveying seven misuse vectors, among them biosecurity and cybersecurity, they find that past studies do not clearly assess marginal risk in most cases, and that the disagreement in the literature is partly an artifact: studies analyze different subsets of the framework under different assumptions (Kapoor et al., 2024b). They give the biosecurity case as their example of the confusion, since one study reports that open language models can produce accurate information about pandemic-capable pathogens while another observes that such information is available on the internet without them (Kapoor et al., 2024b).

Derivation What an uplift claim is a claim about

Sort the people who might attempt harm into classes $c$ by what they already know and command, and let $n_{\text{act},c}$ be the number of people in class $c$ . Write $p_c$ for the probability that a member of that class succeeds with the baseline technology, $p'_c$ for the probability with access to the model, and $\mathrm{Harm}_c$ for the harm one success causes. The risk the model adds is

\sum_c n_{\text{act},c} \, (p'_c - p_c) \, \mathrm{Harm}_c.

Reading the terms gives three instructions for an evaluation (inference). A term is large either because a class is large or because the model changes its chances a lot, so a measurement that does not say which class it sampled measures none of the terms. The baseline $p_c$ is a property of the world and not of the model, so the same model has a different marginal risk in a year when search engines and the literature have changed. And $\mathrm{Harm}_c$ is not a model property at all, which is why the frameworks of Section 59.2 set their thresholds by the severity of the harm they are trying to avoid rather than by the size of the measured capability.

Each $p_c$ is itself the success probability of a chain of stages, and Section 58.2 derives what follows: raising one stage's chance of success multiplies the chain's chance by the most where that stage was the weakest, so help on a stage that was never the bottleneck changes the outcome almost not at all. That is why the frameworks ask which stage a capability touches, and why a benchmark that measures an easy stage can rise for years without moving any term of (59.1) (inference).

Which actor class matters also depends on the kind of tool. Jonas Sandbrink, in a 2023 preprint, separates two classes that the governance literature now treats separately. Large language models may provide dual-use information and so remove some barriers that historical weapons efforts met, which bears on less expert actors; biological design tools, his term for systems trained on biological data that help design proteins or other biological agents, instead expand what sophisticated actors can do (Sandbrink, 2023). In combination, he argues, the two could raise the ceiling of harm and make it broadly accessible (Sandbrink, 2023). His proposed interventions follow the split: independent pre-release evaluations to establish what models can do and how well safeguards work, differentiated access weighed against the benefits of open release, and universal, enhanced screening of gene synthesis orders (Sandbrink, 2023). The last names a control that acts on a physical step rather than on information, and Section 59.5 returns to it.

One distinction from the evaluation literature carries over with a change of subject. Toby Shevlane and colleagues separate evaluations of what a model can do from evaluations of what it will do (Section 36.1) (Shevlane et al., 2023). In this domain the propensity in question is mostly the user's rather than the model's: a user who wants the output will supply the intent, so a developer's lever is a safeguard that refuses or detects rather than a disposition trained into the model (inference). That is why Section 59.3 reports classifiers and data filtering where other chapters report training.

The measurement problem is that the quantity in (59.1) is a difference between two probabilities of operational success, and the cheapest way to approach it is a trial that scores work products instead. Such a trial is small, because each team costs weeks of expert effort, and a small two-arm comparison can see only a large difference.

Derivation What a two-arm uplift trial can detect

Give $n_{\text{team}}$ teams the baseline technology and $n_{\text{team}}$ others the model as well, score each team's work on a fixed scale, and compare the two means. Let one team's score have standard deviation $\sigma_{\text{team}}$ . The difference of means then has standard error $\sigma_{\text{team}}\sqrt{2/n_{\text{team}}}$ . With false positive rate $\alpha$ and miss rate $\beta_{\text{miss}}$ , the smallest difference $\delta_{\min}$ the comparison detects with probability $1 - \beta_{\text{miss}}$ is

\delta_{\min} = \big(\Phi^{-1}(1 - \alpha/2) + \Phi^{-1}(1 - \beta_{\text{miss}})\big)\, \sigma_{\text{team}} \sqrt{2/n_{\text{team}}},

the two-arm form of the sample-size formula of Section 36.7.2, where $\Phi^{-1}$ is the inverse of the standard normal distribution function. At $\alpha = 0.05$ and 80 percent power the bracket is 2.80. Rearranged, the teams needed are $n_{\text{team}} = 2\,(2.80\,\sigma_{\text{team}}/\delta_{\min})^2$ : the cost grows with the square of the ratio of the spread to the difference worth seeing, so halving the difference multiplies the teams by four.

Two features of the design make the picture worse than the formula suggests (inference). The teams differ in skill and approach, which enters $\sigma_{\text{team}}$ and is not reduced by any amount of care about the model; and the same team cannot work both arms, since it would carry what it learned across, so the variance reduction that pairing gives an evaluation (Section 36.7.2) is unavailable here. A null result from such a trial therefore bounds uplift only loosely, in the way Section 6.5 describes for any rate that was never observed.

Everything in this section is a statement about operational success. The measurements the next section reports are mostly scores on tests, and the distance between the two is the validity question that Section 5.2 poses for every number in this chapter: a knowledge benchmark is a proxy for a construct, and the proxy is not the construct.

Sources cited in Section 59.1 3

Kapoor et al. (2024b) On the Societal Impact of Open Foundation Models
Sandbrink (2023) Artificial intelligence and biological misuse: Differentiating risks of language models and biological design tools
Shevlane et al. (2023) Model evaluation for extreme risks

59.2 Evaluations and uplift studies as reported #

The published record has three kinds of entry: scores on tests of knowledge, trials that put people in two arms and score what they produce, and the developers' own decisions to activate safeguards. They support different claims, and the differences matter more here than the numbers.

59.2.1 Tests of knowledge, and what they stand for #

The best known hazard benchmark is WMDP, a public set of multiple-choice questions on hazardous knowledge in biosecurity, cybersecurity, and chemical security, written and filtered so that it could be released at all; Section 27.7.1 reports it and the unlearning method proposed with it (Li et al., 2024d). A second kind of benchmark measures research competence rather than hazard. LAB-Bench, a 2024 preprint by Jon M. Laurent and colleagues at FutureHouse, holds over 2,400 multiple-choice questions on practical biology research tasks, from retrieving results in the literature to navigating databases, scored against human expert researchers (Laurent et al., 2024). Its authors built it because the existing benchmarks asked textbook questions, and they expect a system that scores highly on its harder tasks to be a useful research assistant (Laurent et al., 2024). Nothing in that aim is about weapons, which is the point: the capability the frameworks track is the capability the benchmark was built to encourage (inference).

Scores on such tests have risen past expert baselines. The International AI Safety Report, an expert assessment chaired by Yoshua Bengio, states that AI systems now match or exceed expert performance on many benchmarks measuring knowledge relevant to biological weapons development (Bengio et al., 2026). Its example is a study in which a recent model outperformed 94 percent of domain experts at troubleshooting virology laboratory protocols (Bengio et al., 2026). Anthropic reports the same crossing on its own models: within a year Claude went from underperforming world-class experts on a virology troubleshooting evaluation built by an outside group to exceeding that baseline, and the developer notes that the human baseline is experts answering within their own specialty, so breadth favors the model (Anthropic, 2025at) (developer blog post, September 2025). Google DeepMind's earlier program of dangerous capability evaluations on its Gemini 1.0 models found no strong dangerous capabilities but early warning signs, and Section 3.7.1 reports it (Phuong et al., 2024) (developer preprint, 2024).

59.2.2 A controlled trial with a null result #

The reference design for an uplift trial is a 2024 report by Christopher A. Mouton, Caleb Lucas, and Ella Guest at RAND, an independent research organization. Twelve red cells of three researchers each were randomly assigned to one of four scenarios and to one of three conditions: the internet alone, or the internet together with one of two language models, which the report keeps anonymous. Each cell had seven calendar weeks and no more than 80 hours of effort per member to produce a written operation plan for a hypothetical large-scale biological attack; three further cells, composed to differ in prior expertise, were added to see whether the advantage a model gives depends on what its users already know (Mouton et al., 2024). Eight adjudicators, four with expertise in biology and four in security, scored each plan for feasibility on a 9-point scale through a Delphi process, a structured procedure in which experts score independently, hear the extreme scores argued, and then score again (Mouton et al., 2024).

Across the twelve red cells, access to a model was associated with a 0.22-point decrease in the assessed viability score, with a p-value of 0.64, so the difference was not statistically significant; separately the two models gave 0.12 points more and 0.56 points less than the internet alone, with p-values of 0.87 and 0.25 (Mouton et al., 2024). The authors state what the null does not show. No plan in either arm scored as a satisfactory basis for an attack: all fell between untenable and problematic (Mouton et al., 2024). The exercise covered the planning phase only, not execution, which the authors expect to present greater complexity (Mouton et al., 2024). The models are those of summer 2023 (Mouton et al., 2024). The report's own key findings say that it did not measure the distance between the capability frontier and the knowledge such planning would need, and that future work should raise the sensitivity of the test by testing more models, involving more researchers, and removing unhelpful sources of variation (Mouton et al., 2024). On the sensitivity point the authors are specific: differences in the approach, background, skills, and focus of the researchers within each cell "likely represent a much greater source of variability than access to an LLM" (Mouton et al., 2024).

Kapoor and colleagues cite this trial together with a second study, by OpenAI, as finding that access to language models does not significantly increase access to the information such attacks would need (Kapoor et al., 2024b); that second study could not be verified for this book and is listed in Section G.4, so the chapter does not report it.

The authors' sentence about variability between cells is the derivation of Section 59.1 in words. Figure 59.1 puts the trial's size into (59.2). With four cells in the baseline arm and eight with a model, the comparison has the standard error of about five and a third teams per arm, so at a between-team spread of about one point on the 9-point scale it had an 80 percent chance of detecting a difference of roughly two points, and less of detecting anything smaller. The difference it observed, 0.22 points, is far below that, which is why the result is read here as a failure to detect rather than as evidence of no effect; what it rules out would take a confidence interval, which the report does not give (inference).

Figure 59.1 The smallest difference a two-arm uplift trial can detect, computed from the sample-size formula of Section 36.7.2 at a 5 percent false positive rate. The marked point is the size of the published red-team trial, whose four baseline and eight model-assisted cells give the standard error of 16/3 teams per arm; the dashed line is the 0.22-point difference that trial observed on its 9-point scale (Mouton et al., 2024). The between-team spread is illustrative, since the report does not publish it, and it is the quantity the authors say dominates the comparison.

Two earlier exercises are worth keeping for their designs rather than their findings. A 2023 preprint by Emily H. Soice and colleagues reported a classroom exercise asking whether chatbots lower barriers to dual-use biotechnology and proposed policy measures (Soice et al., 2023). A 2023 preprint by Anjali Gopal and colleagues ran a hackathon-style study framed around the release of model weights (Gopal et al., 2023). Neither had a control arm, blinded scoring, or a measure of operational success, and the second is the study on one side of the biosecurity example of opposite conclusions that Kapoor and colleagues give (Section 59.1) (Kapoor et al., 2024b). Read against (59.1), an exercise with no baseline arm measures $p'_c$ and never $p'_c - p_c$ (inference).

59.2.3 A developer's trial that reports uplift #

A developer's trials point the other way. Anthropic reports several controlled trials in which participants had up to two days to draft a plan for acquiring a biological weapon, with a control group given basic internet resources and a model-assisted group given Claude with safeguards removed; the resulting plans were graded by biodefense experts against a rubric covering the steps of the acquisition pathway (Anthropic, 2025at). Participants with access to Claude 4 models, Claude Opus 4 in particular, received much higher scores and produced plans with substantially fewer critical failures than the internet-only control (Anthropic, 2025at). The developer publishes the design and the direction of the result, not the scores, and states the limit itself: text-based trials of this kind are imperfect proxies for the real setting, which involves tacit knowledge, access to materials, and the persistence of the actor (Anthropic, 2025at). The same post reports a preliminary 2024 laboratory trial in which eight participants carried out basic tasks, with or without a version of Claude, and the developer observed no evidence of uplift; all participants, the internet-only group included, did well, and the developer says the pilot gave no strong evidence either way (Anthropic, 2025at).

The post also explains the decision that followed. When Claude Opus 4 was released, Anthropic activated the deployment protections of its AI Safety Level 3 standard, which it describes as a precautionary decision: improving performance on its evaluations meant it could no longer confidently rule out that its most advanced model could uplift people with basic technical backgrounds (Anthropic, 2025at). Two things are worth separating in that sentence. "Could not rule out" is a judgment made under uncertainty by the party that holds the evidence, not a measurement of uplift; the independent planning trial and the developer's disagree in direction, and the trials that report uplift are the ones whose numbers are not published (inference).

59.2.4 The thresholds that fired #

The frameworks that turn such evaluations into decisions are Chapter 62's subject, and Section 62.2 teaches the trigger mechanism. Two things about them belong here. First, as of October 2026 all three reference frameworks track this domain explicitly, each in a developer document (OpenAI, 2025e; DeepMind, 2026; Anthropic, 2026h), as does a framework published by the Shanghai AI Laboratory with Concordia AI (Shanghai AI Laboratory and AI, 2025); Section I.2.6 gives what each sets out.

Second, in 2025 the thresholds fired. The International AI Safety Report records that multiple developers released models with additional safeguards after pre-deployment testing could not rule out that they could meaningfully help novices develop such weapons: OpenAI treated GPT-5-Thinking and ChatGPT-Agent as High capability and activated the associated safeguards for the first time as a precautionary approach given the lack of definitive evidence; Anthropic released Claude Opus 4 at AI Safety Level 3 on the same reasoning; and Google DeepMind's Gemini 2.5 Deep Think was the first of its models to trigger an early warning alert for chemical and biological risk (Bengio et al., 2026). These are the clearest cases in the book of release decisions that developers attribute to published thresholds (only the developer witnesses what it would otherwise have done, Section 62.7), and also the clearest cases of a decision resting on an absence of evidence (inference).

59.2.5 Beyond biology: targeting and conventional weapons #

In September 2026 Anthropic's Frontier Red Team published evaluations in two further domains: tactical intelligence targeting, which it describes as finding and fixing people and things, and conventional weapons development (Anthropic, 2026ah) (developer blog post). The post's summary makes three claims: that for some military and intelligence tasks models could do what historically required scarce, highly trained experts; that the evaluations show why on-platform safeguards are needed, among them new classifiers the developer has deployed to block such misuse; and that open-weight models from developers in the People's Republic of China, whose trained parameters are published for anyone to download and run, were behind the frontier on the same evaluations but still showed concerning ability (Anthropic, 2026ah).

One evaluation in that post can be reported in full without any weapons content, and it is the most measurable. Models were asked to geolocate social media photographs from the image alone, with no search, metadata, or tools, against a human baseline taken from competitive play of a geolocation game. The developer reports median distance errors of 37.0 and 47.2 kilometers for its two strongest models across 6,000 photographs, against 151 kilometers for the top division of human players and 385 kilometers for the newest open-weight model it tested (Anthropic, 2026ah). The comparison is loose by the developer's own account, since the human game used street-level imagery and let players move within the scene (Anthropic, 2026ah). For the targeting evaluation built on synthetic social media content, it says the data are not fully realistic and that the results are suggestive of differences between models rather than an absolute measure of performance in realistic settings (Anthropic, 2026ah).

59.2.6 How the independent assessment reads the evidence #

The International AI Safety Report reaches three conclusions that organize everything above. General-purpose systems can provide detailed information relevant to developing biological and chemical weapons; developers have released models with heightened safeguards after being unable to exclude meaningful assistance to novices; and substantial uncertainty remains about how much these capabilities raise risk in practice, given the material barriers to weapons production and the difficulty of conducting uplift studies (Bengio et al., 2026). Its 2025 key update had already noted the same precautionary pattern across developers (Bengio et al., 2025a).

The report also explains why the evidence base is thin, which no developer report does. Assessing this domain is constrained by national security law, by treaties, and by information hazards, so researchers who carried out or published a real uplift study might risk violating those constraints; relevant data is often classified, particularly where state actors are involved; and researchers therefore fall back on benign proxy tasks that measure help with similar but harmless procedures, which the report calls an evidence gap that exacerbates the uncertainty (Bengio et al., 2026).

Three cautions therefore travel with every number in this section (inference). A score on a knowledge benchmark is not operational help, and the gap between them is the validity question of Section 5.2. A precautionary activation is a decision under uncertainty, not a measurement. And a developer's evaluation of its own model is a disclosure by an interested party (Section 5.6), which in this domain is not a criticism of the developers so much as a description of who holds the evidence.

Sources cited in Section 59.2 15

Li et al. (2024d) The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning
Laurent et al. (2024) LAB-Bench: Measuring Capabilities of Language Models for Biology Research
Bengio et al. (2026) International AI Safety Report 2026
Anthropic (2025at) Why do we take LLMs seriously as a potential source of biorisk?
Phuong et al. (2024) Evaluating Frontier Models for Dangerous Capabilities
Mouton et al. (2024) The Operational Risks of AI in Large-Scale Biological Attacks: Results of a Red-Team Study
Kapoor et al. (2024b) On the Societal Impact of Open Foundation Models
Soice et al. (2023) Can large language models democratize access to dual-use biotechnology?
Gopal et al. (2023) Will releasing the weights of future large language models grant widespread access to pandemic agents?
OpenAI (2025e) Preparedness Framework Version 2
DeepMind (2026) Frontier Safety Framework, Version 3.1
Anthropic (2026h) Anthropic’s Responsible Scaling Policy
Shanghai AI Laboratory and AI (2025) Frontier AI Risk Management Framework
Anthropic (2026ah) Measuring tactical intelligence targeting and conventional weapons capabilities of AI models
Bengio et al. (2025a) First Key Update: Capabilities and Risk Implications

59.3 Safeguards and what is known of their strength #

Safeguards in this domain act in three places: on what passes in and out of the model, on what the model knows, and on who can reach it at all. Chapter 48 reports the measurements in full, as the robustness of a defense against an adaptive attacker; this section asks the narrower question of what they establish about weapons-relevant misuse, and Table 59.1 collects the answers.

Screening inputs and outputs is the only safeguard whose published strength was measured on this domain directly. The constitutional classifiers of Mrinank Sharma and colleagues, a developer's system tested by its developer, were red-teamed, that is, attacked by people paid to find a way through, against ten questions about chemical, biological, radiological, and nuclear weapons, and Section 48.5.1 reports the result with its two conditions: the red-teamed prototype refused impractically often, and the 0.38-percentage-point rise in refusals on production traffic belongs to the later classifiers under automated evaluation (Sharma et al., 2025). Its 2026 successor reads each output together with its input and escalates suspicious exchanges from a cheap first stage to an expensive second. It reports a 0.05 percent refusal rate and, in over 1,700 hours of red teaming, no attack that answered all eight target questions in detail comparable to an undefended model (Cunningham et al., 2026). That work also opened by finding two classes of attack that evaded the previous generation, which is the pattern Section 48.5.3 sets out: a clean red-team record bounds exposure from below, and the developer's own next round is where that bound turned out to be loose. A classifier that screens all traffic also inherits the base-rate arithmetic of Section 6.7, worked for this cascade in Section 48.5.4.

One safeguard was built with the government that holds the relevant expertise. Anthropic, with the Department of Energy's National Nuclear Security Administration and the DOE national laboratories, co-developed a classifier that distinguishes concerning from benign nuclear-related conversations with 96 percent accuracy in preliminary testing, and deployed it on its traffic (Anthropic, 2025t). The developer said it would share the approach through the Frontier Model Forum so that others could do the same (Anthropic, 2025t; Anthropic, 2025s) (developer announcements, 2025). The arrangement answers a problem the developers state themselves, that a private company cannot evaluate nuclear risks alone (Anthropic, 2025t). The number should be read as the developers label it, preliminary, and as Section 6.7.3 requires: at a low base rate, 96 percent accuracy says little about what share of flagged conversations are real (Anthropic, 2025t).

Acting on what the model knows is the newer line, and Chapter 27 reports it. Filtering text about dual-use biology out of pretraining left 6.9-billion-parameter models at near chance on the proxy benchmark and resisted up to 10,000 steps of adversarial fine-tuning, far longer than safeguards added after training, in a preprint by Kyle O'Brien and colleagues (Section 27.2.3) (O'Brien et al., 2025). Three conditions travel with that result: the models are small ones the authors trained for the purpose, the measurements are multiple-choice proxies, and a filtered model could still use such information when it was supplied in its context, so the safeguard is about what the weights carry and not about what the system can do (O'Brien et al., 2025). A developer reports the same direction for models it pretrained from scratch on filtered data, at a model size it does not disclose (Anthropic, 2025u). Unlearning after training lowers accuracy on that proxy benchmark while leaving general capabilities in place (Li et al., 2024d), and Section 27.7 reports the attacks that bring the knowledge back. Confining dual-use knowledge to modules that can be deleted is reported as preliminary, has not been applied to production models, and comes with its authors' warning that some dual-use capabilities may be so entangled with general knowledge that no method separates them cleanly (Anthropic, 2026f) (developer blog post with an outside group).

Three limits bound all of this, and two of them are Chapter 48's. Fine-tuning, further training of a finished model on new data, removes refusal almost entirely with as few as ten examples, whether it is bought through a developer's interface or run on published weights, including for requests that involve weapons assistance, and in several cases more recent models were more susceptible (Section 48.4.1) (Murphy et al., 2025); training safeguards against simulated tampering extends but does not end this (Section 48.4.3) (Tamirisa et al., 2025); and a classifier's own training data is an attack surface (Section 48.5.2) (Anthropic, 2026al). The third limit is specific to per-model evaluation. Erik Jones, Anca Dragan, and Jacob Steinhardt, in a paper at ICML 2024, showed that an adversary can misuse combinations of models even when each model is safe on its own. The method is to split a task into subtasks, sending the hard but benign parts to a frontier model and the easy but malicious parts to a weaker one (Jones et al., 2024). They study manual decomposition, where a person finds the split, and automated decomposition, where a weak model proposes benign tasks whose solutions it then uses (Jones et al., 2024). Their measurements are on four tasks, none of them a weapons task: vulnerable code, explicit images, scripts for hacking, and manipulative posts, produced at much higher rates by combinations than by either model alone (Jones et al., 2024). Their conclusion is an argument rather than a measurement about this domain: a perfectly aligned frontier system can enable misuse without ever producing a malicious output, so red teaming should extend beyond single models in isolation (Jones et al., 2024). Whether the same decomposition works for a task whose hard parts are physical rather than textual is untested (inference).

Controlling access is the oldest instrument and the one the governance literature returns to. Structured access, in Toby Shevlane's term, replaces dissemination with controlled, arm's-length interaction through an interface (Shevlane, 2022) (preprint). Irene Solaiman sets release on a gradient of six levels, from fully closed through staged, hosted, and cloud-based access to downloadable and fully open, and argues that each level trades off differently between concentrating power and mitigating risk (Solaiman, 2023) (preprint). Tiered access for vetted users is the form a developer proposes for dual-use knowledge it does not wish to withdraw entirely (Anthropic, 2026f). And because every access control is void if the weights are taken, Sella Nevo and colleagues at RAND catalog about 38 distinct attack vectors on model weights, estimate which are feasible for which attackers, and define five security levels with benchmark systems for each; about a dozen of the vectors are likely infeasible for nonstate actors but feasible for state actors (Nevo et al., 2024).

Table 59.1 Safeguards against weapons-relevant misuse, by where they act, with the strength each source reports and the limit each is known to have

Safeguard	Where it acts	Strength as reported	Who reported it	Known limit
Constitutional classifiers on inputs and outputs	The exchange	In thousands of red-team hours on ten weapons questions, no report answered all ten at half the detail of an unguarded model; later classifiers added 0.38 points of refusals on production traffic	The developer, on its own system (Sharma et al., 2025)	Prototype refused impractically often; the next generation found two attack classes that had evaded it (Cunningham et al., 2026)
Cascaded exchange classifiers	The exchange	0.05 percent refusals; no attack in over 1,700 hours answered all eight target questions comparably to an undefended model	The developer, on its own system (Cunningham et al., 2026)	A clean red-team record bounds exposure from below (Section 48.5.3)
A classifier co-developed with a nuclear security agency	The exchange	96 percent accuracy separating concerning from benign conversations, in preliminary testing	The developer with a government agency (Anthropic, 2025t)	Accuracy at a low base rate does not give precision (Section 6.7.3)
Filtering hazardous text from pretraining	What the weights carry	Near-chance proxy knowledge; resisted up to 10,000 fine-tuning steps in 6.9-billion-parameter models	An academic and government-institute group (O'Brien et al., 2025)	Small models, multiple-choice proxies, and no protection when the text is supplied in context
Unlearning after training	What the weights carry	Lowers benchmark accuracy to near chance	The benchmark's authors (Li et al., 2024d)	Suppression, not removal: fine-tuning restores it (Section 27.7)
Removable modules for dual-use domains	What the weights carry	Deleting a module approximates training without the domain	The developer with an outside group (Anthropic, 2026f)	Preliminary, not in production, and entangled capabilities may not separate
Structured and tiered access	Who can reach the model	Not measured as a rate	Proposals (Shevlane, 2022; Anthropic, 2026f)	Void if the weights are obtained; offered fine-tuning can strip the refusals of the model behind it (Murphy et al., 2025)
Securing the weights	Who holds the model	About 38 attack vectors; five security levels defined	An independent research organization (Nevo et al., 2024)	About a dozen vectors are likely infeasible for nonstate actors but feasible for state actors

What the table does not contain is a rate of misuse prevented. As of October 2026 we found no measurement, by any group without a stake in the defense, of how often a deployed classifier catches real attempts at this kind of misuse, and Section 48.5.3 names the same gap for jailbreaks in general. The published strength of the safeguards in this domain is therefore a record of what red teams did not achieve under the conditions their sponsors set.

Sources cited in Section 59.3 15

Sharma et al. (2025) Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming
Cunningham et al. (2026) Constitutional Classifiers++: Efficient Production-Grade Defenses against Universal Jailbreaks
Anthropic (2025t) Developing nuclear safeguards for AI through public-private partnership
Anthropic (2025s) Developing nuclear safeguards for AI
O'Brien et al. (2025) Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs
Anthropic (2025u) Enhancing Model Safety through Pretraining Data Filtering
Li et al. (2024d) The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning
Anthropic (2026f) An off switch for dual-use knowledge in AI models
Murphy et al. (2025) Jailbreak-Tuning: Models Efficiently Learn Jailbreak Susceptibility
Tamirisa et al. (2025) Tamper-Resistant Safeguards for Open-Weight LLMs
Anthropic (2026al) Poisoning Fine-tuning Datasets of Constitutional Classifiers
Jones et al. (2024) Adversaries Can Misuse Combinations of Safe Models
Shevlane (2022) Structured access: an emerging paradigm for safe AI deployment
Solaiman (2023) The Gradient of Generative AI Release: Methods and Considerations
Nevo et al. (2024) Securing AI Model Weights: Preventing Theft and Misuse of Frontier Models

59.4 Displacement: does a safeguard reduce harm or move it? #

Every safeguard in the previous section changes the situation of a would-be misuser rather than the misuser. The person who wanted the output still wants it, and can look for another route: another prompt, another model, an open-weight model with no safeguards at all, or the baseline technology the model was supposed to improve on. Whether blocking one route reduces harm or moves it is the question on which most of the open-weight debate turns, and it is an old empirical question in a field that has measured it.

From criminology. Reducing the opportunities for a specific crime, rather than changing offenders, is the approach R. V. G. Clarke called situational crime prevention (Clarke, 1980), and its standing objection is displacement: the offender, blocked here, offends there. Section 48.7.1 reports the field's main measurement, a 2009 review in Criminology by Rob Guerette and Kate Bowers whose abstract reports displacement in 26 percent of 574 observations across 102 evaluations and diffusion of benefits, a fall in crime where nothing was done, in 27 percent; in the 13 studies that allowed an overall assessment, spatial displacement, when it occurred, tended to be less than the effect of the intervention (Guerette and Bowers, 2009). The same section reports the study that comes closest to the machine case, a measurement of police action against markets that sell denial-of-service attacks. Takedowns of single sellers preceded short-lived falls in attacks while wider disruptions lasted longer, and search advertisements bought by a national crime agency, warning young users of the illegality of such attacks, appear to have kept new users out (Collier et al., 2019).

Applied here, the field's record cuts against the strongest form of the dismissal. "They will use another model" is an empirical claim about displacement, and in the one field that has measured displacement it was observed in about a quarter of observations, and in the 13 studies that could weigh it, spatial displacement was usually smaller than the effect it offset. But notice that in this domain the displacement question and the marginal-risk question of (59.1) are the same question asked from two sides: displacement to an unguarded alternative is exactly the baseline $p_c$ that marginal risk measures against (inference). That is why both sides of the open-weight debate owe a measurement rather than an assumption, and why the debate has stayed where it is. Elizabeth Seger and colleagues argue that for some highly capable models the risks of open release may outweigh the benefits, and set out non-open sharing options instead (Seger et al., 2023b) (preprint). Kapoor and colleagues find the research insufficient to characterize the marginal risk for these vectors (Kapoor et al., 2024b). The United States government's 2024 report on dual-use models with widely available weights concluded that current evidence was not sufficient to determine either that restrictions on such models are warranted or that restrictions will never be appropriate, and recommended monitoring a portfolio of risks instead (National Telecommunications and Information Administration, 2024); Rishi Bommasani and colleagues add, in Science, that different policy proposals may disproportionately affect the innovation ecosystem (Bommasani et al., 2024) (read here from the record of the article, not its text).

Two measurements bear on the question directly. A developer's targeting and conventional-weapons evaluations found the open-weight models it tested behind the frontier but still showing concerning capability, which is the shape displacement would take: a less capable but unguarded alternative (Anthropic, 2026ah). And the governance literature records a case in which displacement happened inside the research community. The creators of a genomic prediction model excluded one class of viral genomes from its training data and published the weights; within weeks, as Doni Bloomfield and colleagues report in Science, other scientists had refined those published weights with data on viruses that infect humans, in that case a benign family (Bloomfield et al., 2024). The lesson they draw is not about that model but about the instrument: fine-tuning an open model is much cheaper than training one, so an oversight rule that acts only at release does not reach what happens after it (Bloomfield et al., 2024).

The older question about publishing frames the limit. Toby Shevlane and Allan Dafoe argue that whether publishing research helps attackers or defenders more depends on such factors as whether adequate defensive measures are possible and whether the knowledge would be discovered independently anyway (Shevlane and Dafoe, 2019) (preprint). They add that the AI community should not import the conclusions of the software-vulnerability disclosure debate, where disclosure often favors defense (Shevlane and Dafoe, 2019). How the balance moves as both sides invest is Section 58.2's model (Garfinkel and Dafoe, 2019).

Where the analogy stops has three parts in Section 48.7.2: switching from one model to another is nearly free, where criminology measures offenders who pay to move; an attacker holding the weights faces no situational control at all; and diffusion of benefits depends on offenders not knowing which targets are protected, while which models carry safeguards is public. This chapter adds a fourth, and it is the most limiting (inference). The criminological measurements are counts of ordinary crimes, which are common enough to count; the harm this chapter is about is rare and extreme, so a rate measured over street crime says nothing about the tail that the frameworks are written for. The one target that cannot be copied is the physical step, which is why the governance tradition reaches for controls there, and that tradition is the subject of Section 59.5.

Sources cited in Section 59.4 11

Clarke (1980) “situational” Crime Prevention: Theory and Practice
Guerette and Bowers (2009) Assessing the Extent of Crime Displacement and Diffusion of Benefits: a Review of Situational Crime Prevention Evaluations*
Collier et al. (2019) Booting the Booters
Seger et al. (2023b) Open-Sourcing Highly Capable Foundation Models: An evaluation of risks, benefits, and alternative methods for pursuing open-source objectives
Kapoor et al. (2024b) On the Societal Impact of Open Foundation Models
National Telecommunications and Information Administration (2024) Dual-Use Foundation Models with Widely Available Model Weights Report
Bommasani et al. (2024) Considerations for governing open foundation models
Anthropic (2026ah) Measuring tactical intelligence targeting and conventional weapons capabilities of AI models
Bloomfield et al. (2024) AI and biosecurity: The need for governance
Shevlane and Dafoe (2019) The Offense-Defense Balance of Scientific Knowledge: Does Publishing AI Research Reduce Misuse?
Garfinkel and Dafoe (2019) How does the offense-defense balance scale?

59.5 What the governance of dual-use biology offers #

From biosecurity. The life sciences have governed research that could be misused for fifty years. They are the only field with a long record of deciding, in advance and in public, which work is too dangerous to do or to publish, and their instruments, their failures, and their unresolved debates are the nearest precedent for governing models with weapons-relevant capability.

The founding move was a pause called by the researchers themselves. The Asilomar conference of 1975 met to decide whether the pause on certain recombinant DNA experiments, called for in a letter published in July 1974, should end and under what conditions (Berg et al., 1975; Berg et al., 1974). Its answer set out two principles that every later scheme inherits: that containment be made an essential consideration in experimental design, and that the effectiveness of the containment should match, as closely as possible, the estimated risk, so that whatever scale of risks is agreed there should be a commensurate scale of containment (Berg et al., 1975). The conference also held that some experiments posed risks serious enough that they ought not to be done with the containment then available (Berg et al., 1975). Section 64.2 reads the episode as a case of self-governance by a technical community, and Section 62.1 notes that the first laboratory safety framework took the biosafety levels as its loose model; the structure both share is a scale of risk met by a scale of containment.

The second instrument is review of particular experiments. The National Research Council's 2004 report on biotechnology research recommended, in the publisher's summary of it, that the government expand existing regulations and rely on self-governance by scientists rather than adopt intrusive new policies; that it not attempt to regulate scientific publishing but trust scientists and journals to screen papers for security risks; and that seven types of risky study, which the report calls experiments of concern, require approval by the institutional biosafety committees that already oversaw recombinant DNA research at some 400 United States institutions (National Research Council, 2004). The same summary gives the reason the report looked beyond its own jurisdiction: with biological information and tools widely distributed, regulating only United States researchers would have little effect, so it proposed an international forum to encourage similar measures elsewhere (National Research Council, 2004). That sentence, written in 2004 about a different technology, is the argument now made about models whose weights can be downloaded (inference).

The tradition's hardest case is publication. The controversy began with a 2012 study of airborne transmission of an influenza virus between ferrets (Herfst et al., 2012), cited here only as the case that opened the debate, and it produced an argument that transfers. Marc Lipsitch and Alison Galvani, in a 2014 policy forum, argued that experiments creating novel potential pandemic pathogens should be performed only if the public health benefits envisaged cannot be obtained by safer methods, and that the alternatives are available (Lipsitch and Galvani, 2014). The structure of that argument is a risk-benefit test applied to a specific piece of work, with the burden on the party that wants to do it.

The current United States instruments show both what this tradition can reach and where it stops, and Section I.1.9 reports them as of October 2026. The 2024 federal policy on dual-use research of concern reaches models only through one voluntary subsection, which encourages institutional oversight of computational work that could directly enable the design of an enhanced pathogen or a novel agent (United States Government, 2024). The strongest instrument the field has built touches models only by encouragement (inference). A 2025 executive order then directed that the policy be revised or replaced (Executive Office of the President, 2025b), which is itself the evidence that this kind of oversight moves with administrations. The same order reaches the chokepoint that Section 59.4 identified as the target that cannot be copied, the screening of orders for synthetic nucleic acids (Executive Office of the President, 2025b), and Sandbrink's recommendation of universal, enhanced screening of synthesis orders is the same instrument (Sandbrink, 2023).

Four proposals carry the tradition toward models. Doni Bloomfield and colleagues argue in Science that voluntary commitments by developers cannot stand alone, and that national governments should require independent biosafety and biosecurity evaluation, before release, of a narrow class of advanced biological models: those trained with very large computational resources on large quantities of biological sequence or structure data, or with less computation on especially sensitive data that is not widely accessible (Bloomfield et al., 2024). Their design asks that thresholds be set in advance and that it be clear when one has been crossed; that the requirement apply irrespective of funding source, since much of the work is done in industry; that oversight address the special risks of releasing weights; and that governments harmonize standards to avoid regulatory arbitrage (Bloomfield et al., 2024). Jaspreet Pannu and colleagues take the next step in a 2025 perspective in PLOS Computational Biology: there is no common industry approach to evaluating biological models for risk, and the prior guidance developed for dual-use life-science research can inform which model capabilities should trigger evaluation and oversight (Pannu et al., 2025). An expert report by Sarah Carter and colleagues at the Nuclear Threat Initiative makes six recommendations, among them an international forum to develop model guardrails, a more agile approach to national governance, implementing guardrails at scale, and strengthening the controls at the interface between digital design tools and physical biological systems (Carter et al., 2023). The National Academies' 2025 assessment of AI in the life sciences (Committee on Assessing and Navigating Biosecurity Concerns and Benefits of Artificial Intelligence Use in the Life Sciences et al., 2025) and the standards body's guidance on managing misuse risk for dual-use foundation models (NIST, 2025) are cited here for their existence and their subjects; this book could not read either in full. The treaty layer above all of this is Chapter 66's subject, which compares the biological and chemical weapons regimes and asks what makes verified arms control rare.

What the tradition offers, then, is a set of instruments that act on identifiable work by identifiable institutions, with public money as the lever and containment, prior review, and publication norms as the controls. The developers' frameworks are an attempt to make a model release such an identifiable event, with a threshold in place of a review committee (Chapter 62). Where the analogy stops is in every term of that sentence (inference). A general-purpose model is not an experiment but a product used by millions; its weights copy at no cost, so a decision made once cannot be revisited; much of the knowledge it carries was published already, which is the baseline $p_c$ of (59.1) and the reason the publication controls bite less here; and the funding lever has no grip on a developer that takes no federal money. That last gap is why Bloomfield and colleagues ask for a rule that applies irrespective of funding source (Bloomfield et al., 2024), and why the frameworks of Section 59.2 are, for now, written by the parties they bind.

Sources cited in Section 59.5 13

Berg et al. (1975) Summary statement of the Asilomar conference on recombinant DNA molecules.
Berg et al. (1974) Potential Biohazards of Recombinant DNA Molecules
National Research Council (2004) Biotechnology Research in an Age of Terrorism
Herfst et al. (2012) Airborne Transmission of Influenza A/H5N1 Virus Between Ferrets
Lipsitch and Galvani (2014) Ethical Alternatives to Experiments with Novel Potential Pandemic Pathogens
United States Government (2024) United States Government Policy for Oversight of Dual Use Research of Concern and Pathogens with Enhanced Pandemic Potential
Executive Office of the President (2025b) Improving the Safety and Security of Biological Research
Sandbrink (2023) Artificial intelligence and biological misuse: Differentiating risks of language models and biological design tools
Bloomfield et al. (2024) AI and biosecurity: The need for governance
Pannu et al. (2025) Dual-use capabilities of concern of biological AI models
Carter et al. (2023) The Convergence of Artificial Intelligence and the Life Sciences: Safeguarding Technology, Rethinking Governance, and Preventing Catastrophe
Committee on Assessing and Navigating Biosecurity Concerns and Benefits of Artificial Intelligence Use in the Life Sciences et al. (2025) The Age of AI in the Life Sciences
NIST (2025) Managing Misuse Risk for Dual-Use Foundation Models

59.6 Physical agents: models that move things #

Everything so far concerns information. A model that controls a machine acts, and in 2025 and 2026 developers published evaluations of how much language models help people operate robots and drones, and of how much the models do alone. They are reported here because they use the designs of this chapter, because the capability is dual-use in the plain sense, and because they are the point at which the decisions a model makes stop being recoverable.

Grounding a model's instructions in what a machine can do is the older research line: a language model's proposals are useful to a robot only if they are matched to the skills the robot can actually perform (Ahn et al., 2022). What is new is measuring how much a model helps a person operate hardware, and then how much it can do alone.

The clearest case is a developer's own uplift study, run with the design of Section 59.1 on a physical task. In an August 2025 experiment, reported that November, Anthropic recruited eight of its researchers and engineers, none with extensive prior experience of robots, randomly assigned four to work with Claude and four without, and asked each team to make an off-the-shelf quadruped robot fetch a beach ball through three increasingly difficult phases (Anthropic, 2025ak; Anthropic, 2026an) (developer blog posts). The team with model access completed more tasks and did so in about half the time, and only that team made substantial progress toward the final goal of fully autonomous retrieval (Anthropic, 2025ak). The developer describes the design as the same uplift methodology it uses for biological risk, which is the clearest statement in the public record that the two questions are the same question about different hardware (Anthropic, 2025ak).

The follow-up removed the people. In June 2026 the developer ran three trials of Claude Opus 4.7 on the same tasks with a researcher limited to connecting a laptop, entering the first prompt, approving commands, and advancing the model to the next task (Anthropic, 2026an). On every task that at least one human team had completed, the model was at least ten times faster (Anthropic, 2026an). Across the four tasks both human teams had completed, it was more than 37 times faster than the team without model access and more than 18 times faster than the team with it (Anthropic, 2026an). Two limits travel with that result. The model still failed at the precise manipulation the task needed, nudging the ball back to its base, and none of the tasks involved low-level actuation, the control of joints and torques (Anthropic, 2026an). The developer reads the sequence as a pattern it has seen before: models help people, then people help models, then models largely do the task themselves, as it reports for cybersecurity (Anthropic, 2026an).

How much a model can do alone depends on how it is connected to the machine. In July 2026 the same team gave several models control of a range of bodies, from classic control problems to a simulated humanoid, a robotic arm, and a real quadruped, through four interfaces of increasing abstraction: setting motor torques directly, writing a controller in code, training a controller with reinforcement learning, and supervising a pretrained policy that turns high-level commands into coordinated movement (Anthropic, 2026p) (developer blog post). The headline finding is that a model's score depends as much on the body and the interface as on the model: when the models must drive the joints themselves they mostly fail, and with low-level manipulation methods they complete the full task only 0 to 5.5 percent of the time, while supervising a pretrained policy raises manipulation performance far above direct control (Anthropic, 2026p). Newer models are better supervisors, recognizing when a proposed action will fail rather than deferring to the policy, though the supervised system still does worse than the policy running alone (Anthropic, 2026p). Models continue to fail at tasks needing stable spatial memory, self-localization, or long plans made in advance (Anthropic, 2026p). The direct-control figures are best-case: the simulator was paused between model calls so that latency would not decide the result (Anthropic, 2026p).

The drone evaluation is the one designed around a dual-use task. In July 2026 Anthropic, working with Andon Labs, reported evaluations of models flying a quad-rotor drone indoors to locate and follow a person (Anthropic, 2026ao) (developer blog post with an outside partner). The developer chose the task because it has search-and-rescue, disaster-response, and lawful public-safety uses, and is also open to abuse by an overreaching authority or an unaccountable private actor (Anthropic, 2026ao). Andon Labs decomposed the task into five sub-tasks, reconstructing the space, localizing within it, navigating, detecting the person, and following, and reproduced each in software so that models could be run repeatedly; the baseline for each is the algorithm that Andon's own staff built working with coding agents, so a model passes a sub-task when it meets or exceeds what AI-assisted experts achieved (Anthropic, 2026ao). Fifteen models from three developers were tested, by Andon Labs on a benchmark that Anthropic says it has not been given access to (Anthropic, 2026ao). Newer models got further on every sub-task, best at detecting and following and worst at reconstruction and localization; the strongest model passed the baseline on all sub-tasks except reconstruction, and on the real drone it followed a person more closely than the reference algorithm but could not navigate between rooms, because reconstruction errors compounded through localization (Anthropic, 2026ao). Consistency lags capability: across ten simulations the models reach the baseline at least once on four of the five sub-tasks, while the strongest model reaches it on average on three, which the authors summarize as the frontier of what models can do running about six months ahead of what they do consistently (Anthropic, 2026ao). The limits are the developer's own: slow drone speeds, one office floorplan, few people, no outdoor or crowd tests (Anthropic, 2026ao).

Safety evaluations of such systems are no longer only one developer's. Pierre Sermanet and colleagues at Google DeepMind released ASIMOV, a benchmark for the semantic safety of foundation models used as robot brains, which they define as the common-sense safety constraints of open environments that cannot be exhaustively listed, generated at scale from real visual scenes and from hospital injury records. With constitutions generated automatically from data they report a top alignment rate of 84.3 percent on it, using Gemini 1.5 Pro, where alignment is agreement with human labels of whether an instruction is desirable and chance is 50 percent; that is above human-written constitutions and above no-constitution baselines, which the authors note were asked a different, more ambiguous question (Sermanet et al., 2025) (preprint). They decline to propose a universal constitution, arguing that rules need customization to legal, cultural, and administrative contexts (Sermanet et al., 2025). The same developer's robotics model report carries a responsible development and safety section that evaluates its embodied reasoning model on those safety questions (Gemini Robotics Team et al., 2025) (developer preprint). And the robustness problem of Chapter 48 carries into physical action: Alexander Robey and colleagues introduced RoboPAIR, an algorithm that jailbreaks language-model-controlled robots, where a jailbreak is an input that gets a model to do what its safety training was meant to stop (Section 48.2) (Robey et al., 2024) (preprint). They demonstrated it in three settings: full access to a self-driving model, partial access to a ground vehicle with a GPT-4o planner, and query access only to a commercial quadruped. Across three datasets of harmful physical actions the attack often reached 100 percent success (Robey et al., 2024). They describe the last as the first successful jailbreak of a deployed commercial robotic system (Robey et al., 2024).

Autonomous physical systems were being evaluated for safety long before language models, and two results from that literature set the standard this one has not yet met. Nidhi Kalra and Susan Paddock's calculation of how many miles an autonomous vehicle would have to drive to demonstrate a failure rate comparable to human drivers, reported in Section 6.5, puts the required evidence out of reach of testing (Kalra and Paddock, 2016); Philip Koopman and colleagues' safety-standard approach for fully autonomous vehicles, cited here for its existence, is the alternative route through safety cases that Chapter 52 teaches (Koopman et al., 2019).

The limiting case is a decision to use force. Paul Scharre's book on autonomous weapons, cited here for its subject, is the standard account of what delegating lethal decisions to machines would mean and of the prospects for regulating it (Scharre, 2018); the written-intent problem it describes is the specification problem of Section 19.7 at its sharpest. Edward Geist and Andrew Lohn, in a 2018 RAND perspective drawn from expert workshops, conclude that AI has the potential to exacerbate emerging challenges to nuclear strategic stability by 2040 even with modest rates of technical progress, through better detection, tracking, and targeting (Geist and Lohn, 2018). And this is one of the few places where scientists from several countries have agreed on a line: the statement of the International Dialogue on AI Safety held in Beijing in March 2024, whose signatories include Andrew Yao and Yoshua Bengio, proposes red lines as an international coordination mechanism. One of them is that no AI system should substantially increase the ability of actors to design weapons of mass destruction, or violate the biological or chemical weapons convention (Safety, 2024). The institutions that would carry such a line are Chapter 66's subject.

Read together, the robotics results repeat the pattern this chapter found for information, and add one thing to it (inference). The pattern is the sequence from helping a person to doing the task alone. What is added is that each step moves the decision about physical harm further from a person, and the drone evaluation says why that matters in the developers' own terms: once models pass a threshold of capability and reliability, the authors write, there will be real pressure to treat human oversight as a cost rather than a safeguard, which is why they argue the decision about the human role should be made deliberately rather than by default (Anthropic, 2026ao).

Sources cited in Section 59.6 13

Ahn et al. (2022) Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
Anthropic (2025ak) Project Fetch: Can Claude train a robot dog?
Anthropic (2026an) Project Fetch: Phase two
Anthropic (2026p) Claude plays robotics
Anthropic (2026ao) Project Pilot: Can AI control a drone?
Sermanet et al. (2025) Generating Robot Constitutions & Benchmarks for Semantic Safety
Gemini Robotics Team et al. (2025) Gemini Robotics: Bringing AI into the Physical World
Robey et al. (2024) Jailbreaking LLM-Controlled Robots
Kalra and Paddock (2016) Driving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability?
Koopman et al. (2019) A Safety Standard Approach for Fully Autonomous Vehicles
Scharre (2018) Army of None
Geist and Lohn (2018) How Might Artificial Intelligence Affect the Risk of Nuclear War?
Safety (2024) IDAIS-Beijing Statement

59.7 What weapons capability means for delegation #

The book's thesis is that alignment is a delegation problem under an asymmetry of competence and visibility. This chapter is the clearest case in which the asymmetry that matters is not between a principal and its model but between the developer and the public (Section 1.3). The developer holds the evaluations, the trials, the safeguard measurements, and the traffic on which a safeguard's performance could be measured; the public holds a published summary. Section 48.8 reaches the same conclusion for jailbreaks. The independent assessment adds that national security law, treaties, and information hazards make a real uplift study risky for researchers to run or publish, so research falls back on benign proxies (Bengio et al., 2026). Those constraints fall on every researcher, but the trials on current models are the developers', and that is a gap better science alone cannot close (inference).

Within the deployed relationship, misuse is common agency with an adversarial member (Section 14.4): the model serves a developer, a deployer, and a user at once, and here the user's wishes are the ones the developer has ruled out. The structure differs from the rest of this part (Part IX) in one way that changes what the four problems ask (inference). The agent is not the party with the harmful goal; it is an instrument. So the problems that bind are the first and the fourth, specification (whose intent counts, and what a refusal is for) and maintenance under adversarial pressure (whether the safeguard holds), while instillation and verification enter only through the narrow question of whether a trained disposition survives fine-tuning.

Two older fields carry over, each with the limit its own section gives. Biosecurity offers instruments rather than findings: containment matched to risk, prior review of particular work, publication norms, and controls at the physical chokepoint, with public funding as the lever that makes them bind (Section 59.5). It also shows that such oversight moves with administrations: a 2025 executive order directed that the 2024 policy be revised or replaced (United States Government, 2024; Executive Office of the President, 2025b). As of October 2026 we found no measurement of what the change did to oversight. Criminology offers a diagnosis, that blocking an opportunity sometimes moves the offense and, where that was weighed, by less than it prevents, and three reasons the reassurance is smaller for models than for streets (Section 48.7.2), to which Section 59.4 adds a fourth: the harm here is too rare to have been counted.

One thing in this chapter is on the list of results the book fixed in advance as having no counterpart in the older fields. A principal that writes its agent's dispositions can also decide what the agent never learns. Filtering hazardous text out of pretraining, and confining a domain to parameters that can be deleted, are governance instruments with no analogue for human researchers, who cannot be built without a capability (O'Brien et al., 2025; Anthropic, 2026f) (inference). The published evidence says this family of instrument resisted the tested attacks on open weights better than the safeguards it was compared with, in all cases but one, and also that it has been demonstrated only at small scale, on proxy benchmarks, and with no protection against information supplied in context. That is the most consequential open empirical question in this chapter, and the laboratories that could answer it at scale are the ones whose models would be affected.

Sources cited in Section 59.7 5

Bengio et al. (2026) International AI Safety Report 2026
United States Government (2024) United States Government Policy for Oversight of Dual Use Research of Concern and Pathogens with Enhanced Pandemic Potential
Executive Office of the President (2025b) Improving the Safety and Security of Biological Research
O'Brien et al. (2025) Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs
Anthropic (2026f) An off switch for dual-use knowledge in AI models

59.8 Settled, contested, missing #

Research status Settled, contested, missing

Settled: that safeguards added after training can be removed from models whose weights or fine-tuning interfaces are available, found by an academic group and again by others on newer models, including for requests that involve weapons assistance (Section 48.9) (Qi et al., 2024a; Pelrine et al., 2023; Murphy et al., 2025); and, as an argument from measurements on other tasks, that testing models one at a time cannot bound misuse by combinations of them (Jones et al., 2024).
Reported, not yet replicated: the measured strength of deployed classifiers, from their developer (Sharma et al., 2025; Cunningham et al., 2026), and the 96 percent preliminary accuracy of the classifier built with a nuclear security agency (Anthropic, 2025t); that filtering pretraining data yields knowledge that resists fine-tuning, from one group at 6.9 billion parameters (O'Brien et al., 2025); the published planning trials that report uplift, by a developer on its own models and without published scores (Anthropic, 2025at); the targeting, conventional-weapons, robotics, and drone evaluations, each from one developer, the last on a benchmark run by an outside partner (Anthropic, 2026ah; Anthropic, 2026an; Anthropic, 2026p; Anthropic, 2026ao); the semantic-safety benchmark and the alignment rate reported for generated robot constitutions, from one developer (Sermanet et al., 2025); and the jailbreaking of language-model-controlled robots, from one group (Robey et al., 2024). The empirical half of the combinations result is also one group's (Jones et al., 2024).
Contested: the size of the marginal uplift current models give toward biological harm. The published evidence does not decide it: an independent red-team trial found no significant difference in the viability of plans made with and without model assistance, in a comparison too small to detect reliably anything but a large effect (Mouton et al., 2024); a developer's trials report substantially higher scores and fewer critical failures with model access, while its small laboratory pilot observed no uplift (Anthropic, 2025at); the marginal-risk framework finds the research insufficient to characterize the risk at all (Kapoor et al., 2024b); and the international assessment concludes that substantial uncertainty remains (Bengio et al., 2026). Also contested: whether safeguards on closed models reduce harm or displace it while capable open-weight models exist (Seger et al., 2023b; Kapoor et al., 2024b; National Telecommunications and Information Administration, 2024).
Missing: as of October 2026, an adequately powered, independently run uplift trial with enough teams to detect a moderate effect, which Section 59.1 shows the independent trial was not, while the developer's trials publish too little to judge; an independent measurement of how often a deployed classifier catches real attempts at this kind of misuse; any public record of what the voluntary oversight of computational dual-use work in the 2024 federal policy has produced (United States Government, 2024); a demonstration that pretraining filtration holds at frontier scale; and, beyond one academic group's 2024 jailbreak of a commercial robot built on a chatbot model, an evaluation of current robot or drone systems' safeguards under adaptive attack by a group without a stake in them, of the kind Chapter 48 asks for in software.
Crux: whether access to a frontier model raises a non-expert's probability of operational success at the stage that limits the whole chain, rather than their score on knowledge proxies. An adequately powered, independently run trial with blinded expert scoring, published at the level of design and result, would settle it for the models tested; Chapter 70 collects it with the other half of the same question, whether the balance of attack and defense is tilting.

Sources cited in Section 59.8 21

Qi et al. (2024a) Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!
Pelrine et al. (2023) Exploiting Novel GPT-4 APIs
Murphy et al. (2025) Jailbreak-Tuning: Models Efficiently Learn Jailbreak Susceptibility
Jones et al. (2024) Adversaries Can Misuse Combinations of Safe Models
Sharma et al. (2025) Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming
Cunningham et al. (2026) Constitutional Classifiers++: Efficient Production-Grade Defenses against Universal Jailbreaks
Anthropic (2025t) Developing nuclear safeguards for AI through public-private partnership
O'Brien et al. (2025) Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs
Anthropic (2025at) Why do we take LLMs seriously as a potential source of biorisk?
Anthropic (2026ah) Measuring tactical intelligence targeting and conventional weapons capabilities of AI models
Anthropic (2026an) Project Fetch: Phase two
Anthropic (2026p) Claude plays robotics
Anthropic (2026ao) Project Pilot: Can AI control a drone?
Sermanet et al. (2025) Generating Robot Constitutions & Benchmarks for Semantic Safety
Robey et al. (2024) Jailbreaking LLM-Controlled Robots
Mouton et al. (2024) The Operational Risks of AI in Large-Scale Biological Attacks: Results of a Red-Team Study
Kapoor et al. (2024b) On the Societal Impact of Open Foundation Models
Bengio et al. (2026) International AI Safety Report 2026
Seger et al. (2023b) Open-Sourcing Highly Capable Foundation Models: An evaluation of risks, benefits, and alternative methods for pursuing open-source objectives
National Telecommunications and Information Administration (2024) Dual-Use Foundation Models with Widely Available Model Weights Report
United States Government (2024) United States Government Policy for Oversight of Dual Use Research of Concern and Pathogens with Enhanced Pandemic Potential

59.9 Exercises #

Exercise 59.1 What a small trial could have seen

An uplift trial scores each team's work on a 9-point scale and compares the means of two arms. Suppose one team's score has standard deviation 1.2 points.

Using (59.2), find the smallest difference the trial can detect with 10 teams per arm, at a 5 percent false positive rate and 80 percent power.
How many teams per arm would be needed to detect half a point?
The published trial observed a difference of 0.22 points with a p-value of 0.64 (Mouton et al., 2024). State what that result does and does not license.

Solution

The bracket in (59.2) is $1.96 + 0.84 = 2.80$ , so $\delta_{\min} = 2.80 \times 1.2 \times \sqrt{2/10} = 1.50$ points. The trial detects a shift of a point and a half with 80 percent probability, and smaller shifts less often.
Rearranged, $n_{\text{team}} = 2\,(2.80 \times 1.2 / 0.5)^2 \approx 90$ teams per arm, about seventeen times the effective size of the published trial. At seven calendar weeks and 80 hours per member per team, that is one reason no such trial has been published (Section 59.8).
It licenses the statement that the trial found no statistically significant difference, under that scoring rubric, for those models, at that date, on the planning phase alone. It does not license the statement that the models made no difference, nor that any difference was below the trial's detectable size, which a real effect of that size escapes one time in five; a bound on the effect needs a confidence interval. The authors do not claim no difference: they say they did not measure the distance between the capability frontier and the knowledge such planning would need (Mouton et al., 2024). A null result bounds a quantity from above only as tightly as the design's size allows (Section 6.5).

Exercise 59.2 Where the marginal risk sits

Use (59.1) with three actor classes and figures chosen only to make the arithmetic clear. Class A has 100,000 members whose chance of success rises from 0.00001 to 0.00004 with model access; class B has 1,000 members whose chance rises from 0.002 to 0.004; class C has 10 members whose chance rises from 0.5 to 0.55. Take the harm from one success to be the same in each class.

Rank the classes by their contribution to the marginal risk.
A benchmark of expert knowledge is given to a sample of class C and shows a large gain. Which term of (59.1) has been measured, and which have not?
The baseline probabilities fall by half because a published literature becomes harder to find. What happens to the marginal risk, and what does that imply about reporting a model's uplift as a property of the model?