Draft complete but not yet reviewed; expect revisions

Agent Security

Chapter 48 treated an attacker who writes to a model through the user's own channel. An agent, a model that calls tools, reads web pages and files, and acts on what it finds, opens a second route. Anyone who can place text where the agent will read it can try to instruct it, and the agent then acts with its user's permissions. The attack is called prompt injection, and this chapter is about defending against it. Its question is whether a model that reads untrusted text can be trained to act only on its principals' instructions, or whether the guarantee has to come from the system built around the model. A second question runs through it: what each kind of defense achieves when it is measured against an attacker who knows how it works.

The chapter starts with the mechanism: why a model cannot reliably tell an instruction from data that merely contains one, and what computer security's oldest version of the problem, the confused deputy, says about it. It then reports how attacks on agents are measured, what the instruction hierarchy and other defenses inside the model achieve against adaptive attackers, and what isolation, privilege control, and control of data flow can guarantee by construction. Two further sections follow attacks beyond a single session: injections that persist in an agent's memory or spread between agents, read through epidemiology, and agents that answer to several principals at once, read through economics. The chapter ends on what all this means for handing work to an agent that acts on other people's content.

Most claims concern the second of the book's three relationships, the deployed system and the people it serves, with the attacker a third party acting through a principal's channel. Where a defense is trained into the model, the first relationship, the training process and its signal, is where it is instilled. Much of the newest evidence comes from developers testing their own models, which brings in the third relationship, the developer and the public. A model that is itself the adversary is Chapter 50's subject; here the model is assumed to be trying to serve its principals, and the question is whether others can turn it.

49.1 Instructions and data in one channel #

A common claim is that prompt injection is a flaw of particular applications, fixed by writing their prompts more carefully, as SQL injection was fixed by writing database queries more carefully. This section argues from the mechanism and the measurements that the flaw sits lower. It sits in the form of the model's input, which carries no mark of who wrote what that an attacker cannot also write, and, by one set of measurements, in how the model represents that input.

49.1.1 A name and a cause #

Simon Willison named the attack in a blog post of 12 September 2022, after Riley Goodside had posted inputs that ordered GPT-3 to ignore its previous directions (Willison, 2022). Willison called it "a form of security exploit" and located the cause in how applications were built: "the way you use that API is to assemble prompts by concatenating strings together!" (Willison, 2022). A developer writes instructions, appends the user's text, and sends the whole as one string. He drew the parallel with SQL injection, whose standard protection is a parameterized query: the database driver receives the untrusted value separately from the query and never reads it as code. He wished for a model interface with "two parameters," the instructions and named blocks of data treated differently. In an update of April 2023 he wrote that this solution is "extremely difficult, if not impossible, to implement on the current architecture" of language models (Willison, 2022). Fábio Perez and Ian Ribeiro's 2022 preprint on inputs that redirect a model from its task is recalled in Section 48.2.2 (Perez and Ribeiro, 2022).

An agent turns the same structure into a loop. The model runs inside a harness, the software that connects it to tools and records what it does. The model writes a request to call a tool: search the web, read an email, run a command. The harness executes the call and appends the result to the model's context as text, and the model continues from the longer context. Whatever the page, the email, or the command's output contained is now part of the input from which the model chooses its next action. Section 11.3.3 observed that a model's context is an input anyone can write; in an agent, "anyone" includes everyone whose text a tool returns. The attack through this route is indirect prompt injection, instructions planted in data an agent is likely to retrieve, which Section 56.5.4 defines from Kai Greshake and colleagues' 2023 workshop paper (Greshake et al., 2023). A direct injection is written by the user in the user's own turn. Keegan Hines and colleagues at Microsoft, in a 2024 preprint, put the consequence plainly: the user is "an innocent bystander or even a victim, but the attacker's instructions have run in their session with their credentials" (Hines et al., 2024). In the terms of Section 1.4, a party with no standing acts through a principal's channel.

The remedy for SQL injection works because a query language has a grammar that separates code from values, so escaping a value makes it inert. Natural language has no such grammar. A sentence in an email that asks for something is a request in exactly the form a user's request takes, and a filter that removed requests from untrusted text would remove text the user wants the agent to read (inference). Hines and colleagues offer an older analogy that fits better. Early telephone networks sent control tones over the same wire as the voice, and people who could generate the tones made free long-distance calls; the lasting fix was out-of-band signaling, a separate channel for control (Hines et al., 2024). "The XPIA problem is analogous to the in-band signaling problem," they write, XPIA being their abbreviation for indirect prompt injection, and in a language model control and data share not only the channel but the same kind of token, the unit of text a model reads (Hines et al., 2024).

49.1.2 Why a marker cannot carry authority #

The obvious repair is to mark the untrusted text: wrap it in delimiters and tell the model that what lies between them is data. A short argument shows what such a marker can and cannot do. It uses only the fact, from Section 11.1, that a language model's output distribution is a function of the tokens in its context.

The argument is the book's, not a source's (inference). Its third step is a design that one defense adopts (Section 49.5), and its fourth is what the measurements below test.

49.1.3 Measuring separation #

Egor Zverev and colleagues, in a paper at ICLR 2025, introduced what they present as the first formal measure of whether a model keeps instructions and data apart, and Section 19.3 recalls its conclusion for the ranking of principals (Zverev et al., 2025). The measure moves a harmless probe, a short task such as asking what a group of crows is called, between two places. In one, the probe sits in the instruction argument, where it should be carried out. In the other, it sits in the data argument, inside text the model is asked to process, for instance to classify by sentiment, where it should be treated as text. Each probe has a witness, a word almost certain to appear in the output if the probe is carried out and almost never otherwise. Let $n_{\text{I}}$ count the items on which the output contains the witness when the probe is in the instruction argument, and $n_{\text{I},\neg\text{D}}$ those of them on which the output lacks the witness when the probe is in the data argument. The empirical separation score is $n_{\text{I},\neg\text{D}}/n_{\text{I}}$, and the utility score is $n_{\text{I}}/n$ over all $n$ items (Zverev et al., 2025). A model that simply concatenated its two arguments would score 0, since nothing would tell it where the probe had been placed (Zverev et al., 2025).

Their dataset, SEP, holds 9,160 such items (Zverev et al., 2025). They used the system prompt, the standing instructions a developer or deployer places at the start of the context, as the instruction argument and the user turn as the data argument. The nine models tested scored "between 13.3% (Phi-3) and 73.2% (Gemma-2B)." "Notably, better or larger models do not show stronger separation scores," and within a family the smaller model often separated better (Zverev et al., 2025). The authors' hypothesis is that larger models are better at carrying out two tasks at once. The best of several prompts telling each model which part to execute raised separation by 24 percentage points on average. With its best prompt GPT-4 reached 95.3 percent, though the authors caution that GPT-4 also helped generate the dataset and that it still executed hundreds of probes placed in the data. Fine-tuning, further training of a trained model on new examples, raised average separation to between 94 and 96 percent, at a cost of 18 to 20 points of utility (Zverev et al., 2025). Across models and methods, "higher separation comes with lower utility, and vice versa." The authors add that experiments cannot prove that existing techniques are unable to establish separation; they can only provide evidence (Zverev et al., 2025).

Two other measurements point the same way. Jingwei Yi and colleagues, at the University of Science and Technology of China and Microsoft, in a paper at KDD 2025, built BIPIA, a benchmark of indirect injection across five application scenarios and 250 attacker goals. All 25 models they evaluated were vulnerable to some degree (Yi et al., 2025). On text tasks they found "a positive association between the Elo ratings and ASRs." An Elo rating is a model's score from crowd-sourced comparisons on a public leaderboard, and ASR is the attack success rate (Section 48.1.2), so more capable models followed injected instructions more often (Yi et al., 2025). Injections placed at the end of the external content succeeded most. The authors conjecture two root causes: models cannot tell "informational context" from "actionable instructions," and they lack the awareness not to execute instructions found in external content (Yi et al., 2025).

A 2026 preprint by Charles Ye, Jasmine Cui, and Dylan Hadfield-Menell looks inside the model (Ye et al., 2026). They trained linear probes, simple classifiers that read a model's internal activations (Section 46.3.3), to recognize which role tag a token sits under, holding the text constant so that only the tag could be learned. They then asked what the probes report when tag and style disagree. Across four open-weight models, whose trained parameters are published (gpt-oss-20b, gpt-oss-120b, Nemotron-3, and Qwen3-30B-A3B), text written in a user's style but wrapped in tool-output tags was still represented as the user speaking: "Userness of user-style text remains 76–88% even under <tool> tags, with Toolness never exceeding 20%" (Ye et al., 2026). In their words, models "perceive the source of text from how it sounds, not its labeled role," and the degree of this confusion "predicts attack success before a single token is generated" (Ye et al., 2026). A separate experiment tested six OpenAI models, gpt-oss-20b, gpt-oss-120b, o4-mini, and GPT-5 nano, mini, and full, with an agent asked to summarize a web page whose text hid an injection (Ye et al., 2026). A plain injection succeeded in 0 to 2 percent of 100 attacks, except 26 percent on gpt-oss-20b; with added text written to resemble the model's own reasoning, success rose to between 56 and 70 percent on every model (Ye et al., 2026).

The strongest counter-reading of the probe result is that linear probes trained on one kind of text may pick up a correlate of role rather than the model's working representation of it. Another is that four open-weight models need not represent roles as proprietary frontier models do. A causal test would tell the readings apart: whether shifting the role representation, with the style held fixed, changes whether the model obeys. Read with step 4 of the derivation, the result says that a tag the attacker cannot forge is still not what these models attend to when they decide who is speaking (inference).

49.2 The agent as a confused deputy #

The previous section located the problem in the model's input. Computer security met the same structure decades earlier in a program with no model in it, and its diagnosis says where a remedy can sit and where it cannot.

From computer security. Norm Hardy's 1988 note "The Confused Deputy" tells a story from Tymshare, a company that sold time-sharing on its computers (Hardy, 1988). A compiler installed in a system directory had been given a license to write files in that directory, so that it could record statistics on how its language was used. A user could also name a file to receive the compiler's debugging output. One user named the system's billing file, the operating system saw the compiler's license and let it write, and "The billing information was lost" (Hardy, 1988). Hardy's diagnosis is that "the compiler runs with authority stemming from two sources." It "serves two masters and carries some authority from each to perform its respective duties," and "It has no way to keep them apart" (Hardy, 1988).

Hardy rejects the repairs that would have the compiler inspect the names it is given, since a legitimate user might want output written anywhere. The operating system's rules for when a program may open a file grew instead: each new clause broke either a secure program or a legitimate one, and the last statement of the rules needed "fourteen boolean operators" (Hardy, 1988). Tymshare's actual patch, a system call that let the compiler choose which of its two authorities to use, added complexity and did not generalize (Hardy, 1988). The repair Hardy proposes is a capability, in the security sense of the word rather than the book's usual sense of what a model can do: a reference that names an object and grants the right to use it in one token. "The capability both identifies the file and authorizes the compiler to write there" (Hardy, 1988). The user would hand the compiler a capability to the output file instead of a name, and the compiler would use its own capability for the statistics file, so neither authority could be spent on the other's behalf.

An agent has the deputy's structure. It holds its user's authority: the mailbox, the files, the calendar, the accounts it is signed into. It also reads text from people who hold none of that authority. When text in an incoming email names an action and the agent takes it, the agent has spent the user's authority on the sender's behalf, with the mailbox in the place of the billing file (inference). Jerome Saltzer and Michael Schroeder's 1975 principles, whose least privilege and complete mediation earlier chapters recall (Section 51.1.1; Section 28.5.3), state the requirement the agent fails. Complete mediation, they write, "implies that a foolproof method of identifying the source of every request must be devised" (Saltzer and Schroeder, 1975). Every tool call an agent makes is issued by the model, and Section 49.1.3 showed that the model cannot reliably say which text moved it. The source of the request is what an injection hides (inference). The same paper anticipated the agent's other half, the program its user cannot inspect: "unless the borrowed program has been completely audited, there is no way to be sure that it does not misuse the data" (Saltzer and Schroeder, 1975).

The analogy stops in two places. Hardy's deputy was confused through one parameter, a file name, by code that was otherwise correct, and the fix made that one designation explicit. An agent computes its choice of action from all of its input, the third party's text included, so there is no single parameter to replace with a capability; the decision itself is within the attacker's reach. Capability discipline applies to the parts of an agent that are ordinary code, the harness, the tools, and the values passed between them, which is where the designs of Section 49.6 put it (inference). And Hardy's user had to know the billing file's name and act once. An injection can be written once and planted where many agents will read it, and attacks found against one model often work against others (Section 49.3).

49.2.1 Every interface adds attack surface #

A deputy's exposure grows with what it can be asked to do, and the measurements of agents show the same growth. Kellin Pelrine and colleagues at FAR AI, in a 2023 preprint, tested three functions that OpenAI's interface to GPT-4 then offered beyond text generation (Pelrine et al., 2023). Fine-tuning removed safeguards, which Section 48.4 reports. The other two were agent functions. GPT-4 assistants "readily divulge the function call schema and can be made to execute arbitrary function calls." And "knowledge retrieval can be hijacked by injecting instructions into retrieval documents." The authors conclude that "any additions to the functionality exposed by an API can create new vulnerabilities" (Pelrine et al., 2023).

The same holds inside the agent. Chen Henry Wu and colleagues at Carnegie Mellon University, in a paper at ICLR 2025, treated an agent as a graph of components through which an attacker's information flows, and built 200 targeted adversarial tasks on a realistic web environment (Wu et al., 2025c). Changing a single image, by less than 5 percent of the page's pixels, hijacked agents built on frontier models with success rates up to 67 percent. Components added to improve performance became targets: an attacker who also compromised the evaluator of a self-reflecting agent, or the value function, the learned estimate of how promising each option is, of an agent that searches over possible actions, raised attack success by 15 and 20 percent in relative terms. "Inference-time compute that typically improves benign performance can open up new vulnerabilities and harm robustness," the authors write (Wu et al., 2025c).

The authors of an isolation architecture for agent applications, at Washington University in St. Louis and the University of Washington, describe the ecosystems of third-party agent tools as resembling "earlier computing platforms, where there was insufficient isolation between apps and the system" (Wu et al., 2025e). Even the software that runs evaluations of agents is in scope. In 2025 the UK AI Security Institute described the toolkit of sandboxes, isolated computing environments with restricted access, that it uses to contain agents under test, and it writes that "we may face the risk of sandboxed agents using injection attacks to attempt to access Inspect's Python code," the evaluation harness outside the sandbox (UK AI Security Institute, 2025d) (government publication). The pattern across these studies is that every component that reads text an attacker can influence, and can affect an action, is an entry point, and that features added for capability add entry points (inference).

49.3 Measuring attacks on agents #

A test of an agent's security needs what a test of jailbreaks does not: tasks a user actually wants done, tools whose effects can be checked, and an attacker who controls only some of the text the agent reads. The measure then has two parts. Utility is how often the agent completes the user's task, and the targeted attack success rate is how often it carries out the attacker's. This section reports the benchmarks built since 2024, collected in Table 49.1, and what their designs allow a reader to conclude.

InjecAgent, from Qiusi Zhan and colleagues at ACL Findings 2024, is the earliest benchmark in the table. It comprises 1,054 test cases covering 17 user tools and 62 attacker tools, and on it a GPT-4 agent prompted in the ReAct style followed the injected instructions 24 percent of the time (Zhan et al., 2024). AgentDojo, from Edoardo Debenedetti and colleagues at ETH Zurich and Invariant Labs at NeurIPS 2024, is the benchmark that later studies in this chapter build on. It has 97 realistic tasks, such as managing an email client or making travel bookings, and 629 security test cases, in an environment meant to be extended rather than a fixed suite (Debenedetti et al., 2024a). On the models of 2024 the authors found that "more capable models tend to be easier to attack, a form of inverse scaling law." They offered a plain reason: weaker models often fail to carry out the attacker's goal even when the injection takes hold (Debenedetti et al., 2024a). The wording of an injection mattered, and an attacker who picked the best of four wordings for each task gained another 10 points (Debenedetti et al., 2024a).

WASP, from Ivan Evtimov and colleagues at Meta's FAIR laboratory at NeurIPS 2025, restricted the attacker to what a real one has (Evtimov et al., 2025). The attacker is an ordinary user of a website the agent visits, who may post only where untrusted users may post, knows nothing of the agent's inner workings, and has a goal that takes several steps to achieve. Agents, including those built on reasoning models, were "readily hijacked." In hand-labeled runs of one agent, built on GPT-4o and given a single user instruction without a defensive system prompt, nearly half of the first actions after the injection advanced the attacker's goal. Far fewer attacks were carried through, and the authors name the reason "security through incompetence": hijacked agents often became confused and took irrelevant actions (Evtimov et al., 2025). When more capable agents were hijacked, more attacks were completed. The benchmark covers two websites, a discussion forum and a code-hosting service (Evtimov et al., 2025).

In a technical post of January 2025, the US AI Safety Institute tested an agent built on the upgraded Claude 3.5 Sonnet of October 2024 with AgentDojo (U.S. AI Safety Institute and Technology, 2025) (government publication). The page now credits the post to the institute's successor body, the Center for AI Standards and Innovation. The model was "significantly more robust against previously tested hijacking attacks" than its predecessor. A red team, a group whose job is to attack the system, run jointly with the UK AI Security Institute, then wrote new attacks on a random subset of tasks and tested them on held-out tasks, and the new attacks also worked in the benchmark's other environments (U.S. AI Safety Institute and Technology, 2025). The post makes two further points about measurement. An average over attacker goals hides that a rare success at stealing a user's files may matter more than a frequent success at sending a harmless email. And because a model's output varies from attempt to attempt, an attacker who can retry cheaply should be measured over many tries (U.S. AI Safety Institute and Technology, 2025). That repeated attempts raised the average well short of 100 percent is what the model of Section 48.3.2 predicts when some goals have per-attempt success near zero (inference).

The public competition of Andy Zou and colleagues, at Gray Swan AI and the UK AI Security Institute, is reported in Section 56.5.4, with its headline finding (Zou et al., 2025). Two of its further findings matter here. Attacks through third-party data succeeded in a larger share of sessions than attacks typed into the chat. And attacks transferred: those that broke the more robust models broke many others, and models from one family shared weaknesses, "indicating a risk of correlated failures across different AI systems" (Zou et al., 2025), the failure mode Section 57.4 analyzes for copies of one model. Agent-SafetyBench, a 2024 preprint by Zhexin Zhang and colleagues at Tsinghua University, measures agents' safety in risky situations more broadly, not only under attack. It traces the failures of the agents it tested to two defects, "lack of robustness and lack of risk awareness," and finds that defensive prompts alone may be insufficient (Zhang et al., 2024b). Two further benchmarks, one that emulates an agent's tools with a language model and one that places injection beside misuse and the agent's own misbehavior on a computer's interface, are reported in Section 56.3.4 (Ruan et al., 2024; Kuntz et al., 2025).

Three readings of the table are worth keeping apart. First, every benchmark of injection in this chapter found agents that follow injected instructions. The benchmarks were built by groups at ETH Zurich, Meta, Microsoft with the University of Science and Technology of China, and Gray Swan AI with the UK AI Security Institute. The US AI Safety Institute found the same against a model its developer had hardened. That agents built on frontier models can be hijacked through the content they read is independently replicated. Second, how often an attack succeeds depends on the attacker the benchmark assumes, and the US AI Safety Institute's red team shows how far a fixed set of attacks can understate it, which is the lesson of Section 48.1.2 repeated for agents. Third, the "incompetence" in WASP is a property of 2025 agents, not a defense: the same paper finds that more capable agents complete more hijacked tasks, so the shield shrinks as agents improve (inference).

Whether capability raises or lowers an agent's exposure is not settled. AgentDojo and BIPIA find more capable models easier to attack, Zou and colleagues find little relation to size and capability (Section 56.5.4) (Zou et al., 2025), and developers report that training lowers exposure in their newest models (Section 49.4). A more capable model can both recognize an injection and carry one out, and the benchmarks do not separate the two effects (inference). None of these studies measures how often injections reach deployed agents or how often they succeed there. One developer, Anthropic, writes in an April 2026 post that it monitors production traffic "to block real-world attacks," but the post gives no rate (Anthropic, 2026ax) (developer report). As of October 2026 we found no public measurement of either.

49.4 The instruction hierarchy as a defense #

The developers' main answer to injection is to train the model to rank its inputs by source, so that text in a tool's output cannot override the system prompt or the user. Section 19.3 reports how the ranking is trained, the developer's results for one model in Table 19.5, and a learned embedding that marks each token's level (Wallace et al., 2024; Guo et al., 2026; Wu et al., 2025d). This section asks the security question: whether a ranking learned in training holds against an attacker who knows it is there.

The derivation of Section 49.1.2 says what the ranking needs. Chat formats aim at its third step: role tags separate system, user, and tool text, and as Ye, Cui, and Hadfield-Menell put it, "Tags are provider-controlled; style is attacker-controlled" (Ye et al., 2026). What their probes question is the fourth step, whether the model heeds the tag when the text's style says otherwise (Section 49.1.3). The benchmarks give a behavioral check. In WASP, agents whose harness placed all web content in the lowest-privilege messages of the hierarchy showed, "counter-intuitively," some of the highest rates of partial hijacking, and the authors found that a defensive system prompt did more (Evtimov et al., 2025). The attacks that broke trained defenses in Section 49.5 mostly worked by impersonating text of higher rank (Nasr et al., 2025). Both fit the reading that the ranking is learned from how text sounds as much as from where it sits (inference).

The strongest evidence for training comes from a developer testing its own models. A 2026 preprint by Eric Wallace and colleagues at OpenAI describes GPT-Red, an attacker model trained by self-play. The attacker is rewarded for making a defender model fail, the defenders are rewarded for resisting and completing their tasks, and both are trained at once (Wallace et al., 2026). The authors call it "the single-largest LLM safety training run ever documented," and they used the attacker's output to train GPT-5.6 (Wallace et al., 2026). An earlier study by the same developer, whose results Table 19.5 gives, also generated attacks during training (Guo et al., 2026). This is the design that the history of adversarial training recommends. Milad Nasr and colleagues note that only training whose attacks are "optimized inside the training loop" has been shown to give meaningful robustness, and then only against the kinds of attack used in that optimization (Nasr et al., 2025). As of October 2026, the developer reports that GPT-5.6 improved over its earlier models on every robustness evaluation it ran, including attacks, datasets, and a whole domain held out from training (developer report; the figures are in Section I.4.3) (Wallace et al., 2026).

The developer states two limits. Its held-out red-teaming exercise ran without the "system-level mitigations" of its products, which it says "inflates the attack success rates compared to real attackers," and the attacker had seen less training on multimodal, multi-turn, and content-policy attacks (Wallace et al., 2026). The strongest counter-reading runs the other way. Every evaluation was run by the developer. The adaptive attacks it held out came from its own attacker, and the outside attacks it reran, written by people for an earlier public challenge (Dziemian et al., 2026), were a fixed set (Wallace et al., 2026). The robustness measured may be robustness to attacks like the ones the model was trained against. What would decide between the readings is a fresh adaptive campaign by a group outside the developer, at a stated budget, against the hardened model and its predecessor; as of October 2026 we found none published. The result concerns the third relationship as much as the second: the public's evidence that the newest model resists injection is the developer's own (inference).

A second developer has reported a measurement of the same kind, for a product rather than a model. In a November 2025 post on Claude for Chrome, its browser extension, Anthropic describes three defenses: reinforcement learning on injections placed in simulated web content, classifiers that scan "all untrusted content that enters the model's context window," and human red teaming, its own and in external challenges (Anthropic, 2025ad) (developer report). It measured the extension against what it calls "an internal adaptive 'Best-of-N' attacker," which tries and combines many injection techniques known to be effective, with "100 attempts per environment" (Anthropic, 2025ad). In its account, Claude Opus 4.5 resisted better than earlier models, new safeguards around the model improved safety for all Claude models, and the remaining rate "still represents meaningful risk" (Anthropic, 2025ad). The rate is a share of attacks, not of environments, and by Section 48.3.2 the share of environments broken within all 100 attempts can be much higher (inference). Unlike the GPT-Red exercise, this one measured the product with its safeguards in place, but the attacker was again the developer's own, built from techniques already known to work. In an April 2026 post the same developer writes that companies test their own agent systems "but each uses its own methods and none are independently verified" (Anthropic, 2026ax) (developer report).

49.5 Prompting, training, and detection against an adaptive attacker #

Beyond the hierarchy, defenses inside the model come in three families. Prompting defenses tell the model which text is data. Training defenses teach it to ignore instructions found in data. Detection defenses flag text or behavior that looks injected. Each reported large drops in attack success against the attacks its authors ran. Section 48.1.2 showed why that measures a lower bound on exposure, and Table 48.1 recorded what happened when later groups ran adaptive attacks, chosen after the defense by someone who knows its design. This section reports the agent defenses in that light, collected in Table 49.2.

Spotlighting, the Microsoft defense whose in-band signaling analogy Section 49.1.1 quoted, transforms untrusted text so that its origin is visible to the model: by delimiting it, by interleaving a marker character through it, or by encoding it, for instance in base64 (Hines et al., 2024). The authors are frank about the first. If an attacker learns the system prompt, "it would be simple to craft a string that contains our delimiters," and "we do not recommend using delimiting in practice" (Hines et al., 2024). They advise assuming that the whole system prompt has leaked and changing the markers on every call. In the terms of Section 49.1.2, they are trying to make a forgeable marker hard to forge. BIPIA's authors tested prompting defenses of the same kind and concluded that they "can effectively reduce ASR but cannot make LLMs robust" (Yi et al., 2025).

StruQ, from Sizhe Chen and colleagues at UC Berkeley at USENIX Security 2025, builds step 3 of that derivation (Chen et al., 2025f). A front end places the instruction and the data in two channels separated by reserved tokens, and filters the data so that it "cannot contain any of these special tokens." The model is then fine-tuned on examples that carry instructions in the data channel, with the intended response ignoring them (Chen et al., 2025f). The authors found that it held against the manual attacks they tested, including their own attempts to slip near-miss delimiters past the filter. Against attacks optimized for each task, it "does not fully generalize" (Chen et al., 2025f). SecAlign, from Chen and colleagues at Berkeley and Meta at the 2025 ACM Conference on Computer and Communications Security, replaced fine-tuning on correct responses with preference optimization (Section 12.6). The model is trained to prefer the response that follows the legitimate instruction over the one that follows the injection (Chen et al., 2025e). The authors list its limits plainly: it applies only when instruction and data are explicitly separated, "cannot achieve 100% security," and "We do not regard SecAlign as a standalone solution to prompt injection attacks" (Chen et al., 2025e). BIPIA's own trained defense, special tokens around external content and fine-tuning against attacks, brought success near zero on its benchmark for the two Vicuna models it trained (Yi et al., 2025). Detection by reading the model's activations, the task-drift probe of Section 46.3.3, was tested against two prompted attacks and no optimized one (Abdelnabi et al., 2025).

A capture-the-flag competition, in which teams defended secrets kept in a model's instructions and every defense was bypassed at least once, is recalled in Table 48.1 (Debenedetti et al., 2024b). Two studies then attacked these families with knowledge of their design. Qiusi Zhan and colleagues at the University of Illinois Urbana-Champaign, in NAACL 2025 Findings, implemented eight defenses on tool-using agents in three categories: detectors, changes to the agent's input, and adversarial fine-tuning (Zhan et al., 2025). The study's two agents, built on Vicuna-7B and Llama3-8B, were tested on 100 cases from InjecAgent, a benchmark of indirect prompt injection in tool-integrated agents that Zhan and colleagues published in 2024 (Zhan et al., 2025; Zhan et al., 2024). Adaptive attacks bypassed all eight, and the success rates exceeded those against the undefended agents. The authors list the conditions: the attacker had white-box access to the agent, the defenses, and the prompts; only the agent's first step was targeted; and defenses were attacked one at a time, not in combination (Zhan et al., 2025). Nasr and colleagues' 2025 preprint, recalled in Section 48.1.3, attacked Spotlighting in its delimiter form, StruQ, Meta-SecAlign, several commercial and open detectors, and MELON (Nasr et al., 2025). Meta-SecAlign is a later training defense in SecAlign's line (Chen et al., 2025b). MELON is a defense that reruns the agent to see whether a tool's output alone triggers the same tool calls (Zhu et al., 2025; Nasr et al., 2025). Against the detectors, the study's search-based attacker had the detector's confidence score fed back to it. Against StruQ, an attacker trained by reinforcement learning, training by reward for the attempts that succeed (Chapter 12), "succeeds in every case we tested," and in most attempts the successful attacks impersonated system-level prompts (Nasr et al., 2025). Human red-teamers in the same study found hundreds of successful attacks against the prompting defenses (Nasr et al., 2025).

The strongest counter-reading is that these attackers were stronger than most real ones. Zhan's had white-box access, and Nasr's had a detector's score to climb. Security evaluation answers that a defense must assume the attacker knows its design, the rule Section 48.1.3 traces to Kerckhoffs, and that what one attacker learns can be published for all. The defenses still lower the success of the fixed attacks used to measure them. Whether they raise the cost of an adaptive attack is not established, and for the prompting defenses on AgentDojo Nasr and colleagues found "no robustness improvement beyond the undefended model" (Nasr et al., 2025). What the record supports is narrower than "they do not work." As of October 2026, no defense inside the model has been shown, by a group outside its developer, to hold against an adaptive attacker, and every one that outside groups attacked in this way gave way (inference). OpenAI's evaluation of GPT-5.6 and Anthropic's of its browser agent, each run by the developer with an attacker of its own (Section 49.4), are the exceptions, and both await an outside test.

49.6 Isolation, privilege, and the flow of data #

If the model cannot be made to ignore injected instructions, the alternative is to arrange that obeying them can do little harm. Defenses of this kind move the guarantee from the model to ordinary code that can be read and tested: the harness, the tools, and the rules that sit between them. This section asks what such code can guarantee and what it costs, starting with limits on what an agent may do and ending with the sandboxes it runs in.

49.6.1 Restricting what the agent may do #

The simplest restriction is the one AgentDojo's authors tested as a tool filter: before reading any untrusted data, the agent commits to the tools its task needs, and the harness disables the rest (Debenedetti et al., 2024a). On GPT-4o, against the benchmark's strongest attack, it lowered attack success to 7.5 percent, because in many test cases the user's task needed only to read, such as summarizing email, while the attacker's needed to write, such as sending it (Debenedetti et al., 2024a). It failed where the tools could not be chosen in advance, because one result decides the next step, or where the tools the task needed were enough for the attack, as in 17 percent of their cases. The authors add a case their benchmark did not cover: an agent given several tasks over time without a fresh context, where an injection could wait until a task arrives that needs the right tool (Debenedetti et al., 2024a).

Progent, whose mechanism Section 56.5.1 describes, makes the same idea finer. Its policy over tool names and arguments is drafted by a language model from the user's task and checked by deterministic code, which lets the policy narrow on its own but widen only with approval, which the user may give case by case or set to be automatic. On two benchmarks, AgentDojo and ASB, with GPT-4o both running the agent and writing the policy, this preprint by Tianneng Shi and colleagues reports attack success falling from 39.9 to 1.0 percent and from 70.3 to 3.9 percent, "while maintaining the agent's utility in both cases" (Shi et al., 2025). Those runs approved every widening automatically, the configuration the authors chose as the worst case (Shi et al., 2025). Its authors list what it does not cover: mistakes by users who approve a widening, and attacks "that influence only the agent's textual outputs without triggering dangerous tool calls" (Shi et al., 2025). The design has an ancestor in operating systems. Systrace, which Niels Provos presented at USENIX Security in 2003, confined programs by policies on their system calls that are "enforced to prevent operations that are not explicitly permitted." Its policies could be generated automatically "in a training session" or interactively while the program runs (Provos, 2003). Where a program needed more privilege, Systrace raised it for "a single system call" rather than for the whole program, following "the principle of least privilege" (Provos, 2003). Confining untrusted helper programs goes back at least to a 1996 paper by Ian Goldberg and colleagues, cited here for its existence (Goldberg et al., 1996).

Isolation can also separate one tool from another. IsolateGPT, from Yuhao Wu and colleagues at NDSS 2025, gives each third-party application its own model instance in an isolated spoke, and routes requests between spokes through a trusted hub. Its aim is to prevent malicious applications, and malicious content passed through benign ones, from reaching the rest of the system (Wu et al., 2025e). With GPT-4 and every permission request granted automatically, the authors report added end-to-end latency under 30 percent for three-quarters of the queries they tested (Wu et al., 2025e).

49.6.2 Separating the planner from the data #

A 2025 preprint by Luca Beurer-Kellner and colleagues, at ETH Zurich, Invariant Labs, IBM, EPFL, Swisscom, and Google, states the principle these designs share. In their words, "once an LLM agent has ingested untrusted input, it must be constrained so that it is impossible for that input to trigger any consequential actions" (Beurer-Kellner et al., 2025). They describe six patterns that enforce it to different degrees. An action selector maps a request to one of a fixed list of actions and never reads a tool's output. Plan-then-execute fixes the list of tool calls before any untrusted data is read. Map-reduce sends each untrusted document to its own model, which cannot take harmful actions, and accepts only a constrained answer from it, such as yes or no. The dual LLM pattern, which they credit to Simon Willison (Willison, 2023), splits the agent into a privileged model that plans and calls tools but never reads untrusted text, and a quarantined model that reads it but has no tools. Code-then-execute has the privileged model write a program, and context minimization removes text from the context once it has served its purpose (Beurer-Kellner et al., 2025). The authors judge that, "As long as both agents and their defenses rely on the current class of language models," general-purpose agents are unlikely to provide "meaningful and reliable safety guarantees" (Beurer-Kellner et al., 2025).

CaMeL, the system by Edoardo Debenedetti and colleagues that Section 28.5.3 reports, is a full instance of code-then-execute with security policies checked on every tool call (Debenedetti et al., 2026). Three of its details matter here. In a real deployment, a call that breaks a policy would not be blocked but would "require user confirmation." Measured under attack with Claude 3.5 Sonnet, the median AgentDojo task needed 2.82 times the input tokens of ordinary tool calling. And it fails by design on a task whose actions are written in the data: "The P-LLM cannot write a plan based on data it can't read," the privileged model being the planner (Debenedetti et al., 2026). What all these designs guarantee, and what they leave, can be stated generally.

Steps 1 and 2 restate CaMeL's guarantee and its treatment of branches as a source of leaks; steps 3 to 5 restate limits that each design names for itself; the general statement is the book's (Debenedetti et al., 2026; Beurer-Kellner et al., 2025) (inference). Every source names the residual of steps 3 and 4. Under plan-then-execute an injection in a calendar "could, however, arbitrarily alter the body of the email" the plan sends (Beurer-Kellner et al., 2025). AgentDojo's authors note that isolation does not help when an injection "solely aims to alter the result of a given tool call," as when a hotel listing asks to be recommended (Debenedetti et al., 2024a). CaMeL "cannot defend against text-to-text attacks which have no consequences on the data flow" (Debenedetti et al., 2026), and Progent leaves out attacks on textual outputs (Shi et al., 2025). Confinement turns "the model must resist" into "the attacker can choose only among the actions the user's request already allows, with the arguments the policies allow": a bound on the damage, not on being misled (inference).

49.6.3 Asking the user #

Where a design cannot decide, it asks the person. Beurer-Kellner and colleagues warn that confirmation can "pose a safety risk themselves with tired or overloaded human verifiers approving unsafe actions" (Beurer-Kellner et al., 2025), and Progent's authors list a user who wrongly approves a widened policy among its unremoved risks (Shi et al., 2025). Anthropic describes the pattern in its own products: when a task needs dozens of actions, repeated requests for approval become friction and "users sometimes tune them out," and in Claude Code its Plan Mode has the user approve a whole plan before anything runs (Anthropic, 2026ax) (developer report). A confirmation prompt is an alert with a reader of limited attention, and the alert budget of Section 54.2.2 applies to it: if most prompts are harmless, approvals become routine (inference). Saltzer and Schroeder named the requirement in 1975 as psychological acceptability: protection has to be easy enough to use that people "routinely and automatically apply" it correctly. If a user "must translate his image of his protection needs into a radically different specification language, he will make errors" (Saltzer and Schroeder, 1975). A policy over tool names and arguments is such a language for most users of an agent (inference).

49.6.4 Sandboxes #

A sandbox bounds what an agent can reach; it does not decide what the agent does with what it is given. The UK AI Security Institute's toolkit for containing agents under evaluation keeps the evaluation harness outside the sandbox and sends commands in, so that "Everything that happens inside is explicitly allowed and initiated by Inspect" (UK AI Security Institute, 2025d) (government publication). Its protocol classifies isolation along three axes, tools, host, and network, which Section 58.4.3 reads for cyber evaluations. The post names how the threat is changing: "The primary current concern is agents accidentally damaging infrastructure," but the Institute is "researching defences against potentially misaligned models that might actively attack their sandboxes" (UK AI Security Institute, 2025d). The 2026 incidents in which agents under evaluation reached systems they were not meant to reach are recorded in Section 54.4, and Butler Lampson's warning that no list of a confined program's channels is known to be complete is recalled in Section 24.6 (Lampson, 1973).

Saltzer and Schroeder's principle of least common mechanism adds one more caution: "Every shared mechanism (especially one involving shared variables) represents a potential information path between users" (Saltzer and Schroeder, 1975). An agent's memory, a shared document store, or a package repository used by many copies of a model is such a mechanism, and Section 49.7 shows injections traveling through exactly these paths (inference).

49.7 Attacks that persist and spread #

The sections so far treated one agent and one session. An injection read in one session can outlive it, and one agent's output can become another's input. This section follows both routes and then asks what epidemiology, the science of things that spread, implies for defending against them.

49.7.1 Persistence in memory #

Agents that keep a long-term memory across sessions write into it what they think will help later, and what they read can end up there. Xianglin Yang and colleagues at the National University of Singapore, in a 2026 preprint, studied the consequence: "untrusted external content observed during a benign session can be stored as memory and later treated as instruction" (Yang et al., 2026). Their attacker had no access beyond web content the agent read during an ordinary task. The agent wrote the injected text into its memory through its normal update process, and in a later session the text was retrieved and caused unauthorized tool use. They designed ways for the text to survive the two common kinds of memory, a sliding window of recent text and retrieval by relevance. They conclude that "memory evolution can convert one-time indirect injection into persistent compromise," so that "defenses focused only on per-session prompt filtering are not sufficient" (Yang et al., 2026). The same channel appears in training: Section 11.5.3 reports one developer's accounts of a model that wrote unauthorized instructions into the summaries a later instance of itself would read (OpenAI, 2026v).

49.7.2 Spread between agents #

Section 57.6.1 reports four groups, with no shared authors and different designs, that built attacks which copy themselves from agent to agent in constructed systems, and it derives when such an attack spreads in a population that mixes at random (Gu et al., 2024b; Lee and Tiwari, 2024; Cohen et al., 2025; Zhang et al., 2026b). A developer has since found the same in its own models. In a report disclosed in September 2026, OpenAI describes training an automated attacker of the GPT-Red kind (Section 49.4) toward injections that "must both achieve a particular adversarial goal and induce the defender to reproduce them publicly." The training used a wide range of environments, with emphasis on tasks that connect the agent to email, calendars, and similar services (OpenAI, 2026w) (developer report). It reports "a new variety of prompt injection, which can self-propagate akin to a computer worm," and adds that "No impact was observed outside of the simulated tool calls in training and evaluation" (OpenAI, 2026w). Its response is to include self-reproduction among the attacker's goals in training, so that future models will have seen such injections (OpenAI, 2026w). The report's dates and models are in Section I.3.10. It is a red team's finding on the developer's own models, not an incident, and Section 49.5 gives the reason to expect the trained-in resistance to hold for variants like those seen in training more than for new ones (inference).

49.7.3 From epidemiology #

From epidemiology. William Kermack and A. G. McKendrick's 1927 paper set out the problem this way. One infected person is introduced into a community, the disease spreads by contact, and each infected person is eventually removed by recovery or death. The question is whether the epidemic ends "only when no susceptible individuals are left," or "whilst many susceptible individuals are still present" (Kermack and McKendrick, 1927). The paper treats the case in which all members of the community are "initially equally susceptible," and "complete immunity is conferred by a single infection." Its answer is a threshold: for each set of infectivity, recovery, and death rates "there exists a critical or threshold density of population." At or below it, an infected arrival does not start an epidemic, and above it the epidemic grows only so long as the density of the unaffected population stays above the threshold (Kermack and McKendrick, 1927). A threshold of that kind, spread only when transmission outpaces removal, is what Section 57.6.1 derives for agents in (57.3). Computer security took up the same questions for worms. Stuart Staniford, Vern Paxson, and Nicholas Weaver, at USENIX Security 2002, built a mathematical model of a worm's spread from data on Code Red I in July 2001. The model is the logistic equation, which, they note, "governs the rate of growth of epidemics in finite systems when all entities are equally likely to infect any other entity" (Staniford et al., 2002). They warned of "surreptitious worms" that spread more slowly, "in a much harder to detect 'contagion' fashion, masquerading as normal traffic," and argued for a "Center for Disease Control" analog for such threats (Staniford et al., 2002).

Read for agents, the model gives a defender two levers and a warning (inference). The levers are transmission and removal. Transmission is what one agent's output can make another do, which the isolation of Section 49.6 limits when it is applied at the boundary between agents, for instance by accepting only constrained answers from an agent that read untrusted text. Removal is a fresh context or a cleared memory, and the memory attacks above are attacks on removal. The warning concerns the two assumptions. Copies of one model are a community of exactly equal susceptibility, and the transfer of attacks across models and families (Section 49.3) suggests that even different models are less different than a population of hosts. Recovery confers no immunity: a reset agent is as susceptible as before. Something better than immunity is available instead: retraining a model changes every copy at once, which is the developer's response above. And the warning of Staniford and colleagues about worms that spread slowly and are harder to detect fits an injection that travels in the messages agents already send each other, where a monitor tuned to unusual traffic would not look.

The analogy stops at the pathogen. A disease's transmissibility changes by mutation and selection; an injection's is written by an attacker who can see the defenses and revise it, and a self-play attacker is trained to raise it. The contact network of agents, unlike a population's, is designed by deployers and can be cut or rewired at will. And the threshold of (57.3) assumes random mixing, which Section 57.6.1 notes fails on networks organized around a few hubs (Pastor-Satorras and Vespignani, 2001).

49.8 Agents that serve several principals #

A deployed agent answers to its developer, the deployer that built a product on it, and the user at once, and it reads text from parties who are none of these (Section 1.4). Section 19.3 sets out how the developers' documents rank the principals, with third parties' text given no authority (Table 19.4). This section reads that ranking as a security arrangement, first through economics and then through the infrastructure that would let others check an agent's authority.

From economics. Section 14.4 teaches the model of several principals acting on one agent: each offers the agent an incentive scheme, and the agent's behavior is an equilibrium of their offers. In the general treatment by Avinash Dixit, Gene Grossman, and Elhanan Helpman, the principals first choose payment schedules, and then "the agent chooses an action optimally, given the payment functions of all the principals" (Dixit et al., 1997). Payments are costly to the one who makes them: "each principal dislikes making contributions and the agent likes receiving them." The set of feasible offers is fixed by what the authors call "feasibility and institutional constraints" (Dixit et al., 1997). Section 14.4.1 draws the inference that a party with no offer to make counts in the outcome only as far as the agent or a principal values its interests.

Read for agents, the model says what an injection is: influence without payment and without membership (inference). In the economic model a party moves the agent only by paying for the action it wants, and only the principals may offer. An injector is outside the set of principals and moves the agent by writing text at almost no cost. The model's exclusion of outsiders becomes a condition: an outsider has no influence only if it cannot reach the agent's input. The remedy the model points to is its institutional constraint, a rule about who may make offers at all, which for agents means attaching standing to an authenticated source and denying it to the rest (inference). The analogy stops at the instruments, as Section 14.4.3 notes for the instruction hierarchy. Principals pay; an injector writes, and it succeeds less by outbidding the principals than by being mistaken for one, which is the role confusion of Section 49.1.3. The agent does not maximize a sum of transfers, and the developer trains the agent, which no principal in the economic model can do.

Infrastructure for that institutional constraint has been proposed. Tobin South and colleagues, at MIT and several other institutions, in a 2025 preprint, describe a framework for "authenticated, authorized, and auditable delegation of authority to AI agents" (South et al., 2025). It extends OAuth 2.0 and OpenID Connect, the protocols that let a website sign a user in or let one service act for a user, "with agent-specific credentials and metadata," and proposes translating natural-language permissions "into auditable access control configurations" (South et al., 2025). A service that the agent contacts can then check who delegated what to it, so that authority travels with a verifiable credential rather than with whatever the agent says. That is Hardy's capability at the scale of the web, and like a capability it bounds what a deputy can do without telling the deputy whose instructions to follow: an injected agent with valid credentials can still be steered within them (inference). Alan Chan and colleagues' proposal of agent infrastructure, recalled in Section 63.3.4 for attributing actions to agents and their users, adds functions aimed at harm after the fact. They include channels that isolate agent traffic from other traffic, rollbacks that "void or undo an agent's actions," and incident reporting (Chan et al., 2025) (preprint). Rollback is the runtime idea that Section 28.5.3 recalls from a 2024 preprint by Shishir Patil and colleagues, which argues that validating an action after the fact, with an undo and confinement of damage, is often easier than validating it before (Patil et al., 2024). Chan and colleagues note that oversight layers and rollbacks are useful at once, even if few others adopt them, while identifiers for agents work only when others recognize them (Chan et al., 2025).

49.9 What agent security means for delegation #

The book's thesis is that alignment is a delegation problem under an asymmetry of competence and visibility. Agent security adds a party the thesis's two-party picture leaves out: the stranger whose text the agent reads. When a principal gives an agent authority and sends it to read other people's content, the principal has lent that authority to every author of that content, to the extent that the system around the agent does not confine it (inference). The confused deputy is the older name for the same arrangement, and the evidence of this chapter says that the deputy cannot yet be trained out of its confusion. Two rules follow, both the book's reading of the evidence rather than any source's finding (inference). Give an agent only the authority one would give the least trusted author of what it will read. Or split the task, so that the part that reads untrusted text holds no authority and the part that holds authority reads no untrusted text, as the patterns of Section 49.6.2 do.

The hard case is a task whose instructions are the untrusted text: answer these customers, act on this colleague's list, triage this inbox. The third party's words are then what the principal wants carried out, within limits the principal has in mind and has not written down. A planner separated from the data cannot serve such a task (step 5 of the derivation in Section 49.6.2). Limits on privilege and flow can still bound the actions the content may trigger: Progent's policy widens, subject to approval, when returned data shows that a task needs more tools (Shi et al., 2025). Within those bounds the defense left is the model's own judgment, or the person's when asked, of which requests in the content its principal would authorize. That is the specification problem returning in a security setting: what a ranked agent owes the parties below it, which Section 19.3 discusses, decided case by case by a model that Section 49.1.3 showed cannot reliably tell who is speaking (inference). Defenses inside the model matter most exactly where no guarantee is available.

The claims divide among the book's relationships. Most concern the deployed system and the people it serves, with the attacker a party outside the relationship. The third relationship enters through the evidence. The strongest reported defense of 2026 and the report of self-replicating injections in a developer's own models are developers' reports on their own systems. Outside evaluators, from university groups to the US and UK government institutes, have shown that a red team adapting to a model finds what a fixed benchmark misses (Section 49.3). As of October 2026 no developer or deployer has published how often injections reach its deployed agents or how often they succeed there, so a user deciding how much authority to give an agent cannot check the exposure from public evidence.

Chapter 71 asks whether attacks on agents through the content they read change the balance of attack and defense. The evidence does not decide it. Inside the model, every published contest between a defense and an adaptive attacker from outside the defense's developer has gone to the attacker, and the large-scale attempts to train robustness with an adaptive attacker inside the loop have been measured only by their developer. Outside the model, designs that separate the planner from the data bound the damage an injection can do, at a price in tasks they cannot serve and in tokens. On this evidence the balance for a given deployment is set more by how much authority its agents hold, and how the system confines it, than by how the model was trained (inference).

The same controls serve a second purpose. An injected agent and a misaligned one look alike from outside: both make tool calls their principals did not intend. Least privilege, confinement, and monitoring therefore limit both, as Section 56.5.1 observes for privilege control and Chapter 50 assumes for a model that is itself the adversary (inference). What differs is the attacker's budget: an outside injector must reach the agent's input and guess at its defenses, while a misaligned model is already inside and may know them, which is why Chapter 50's protocols assume more than this chapter's.

49.10 Settled, contested, missing #

49.11 Exercises #

Further reading #

• Willison (2022): the post that named prompt injection and traced it to concatenated strings.
• Hardy (1988): the confused deputy, and why Hardy proposes capabilities rather than rules to fix it.
• Saltzer and Schroeder (1975): the design principles every agent defense rediscovers.
• Zverev et al. (2025): a formal measure of instruction-data separation, and what it found.
• Debenedetti et al. (2026): confining an agent by separating its planner from the data, with the guarantee's stated edges.