Draft complete but not yet reviewed; expect revisions

AI Control

The verification chapters asked whether a model is aligned. This chapter asks what to do if the answer might be no. Suppose a model has acquired an aim its developers did not intend, and suppose it is capable enough to pursue that aim while trying not to be caught (Chapter 34). Can it still be put to work safely? The research program that poses the question this way calls itself AI control (Greenblatt et al., 2024a; Greenblatt and Shlegeris, 2024). It treats the model as computer security treats an insider: a component with legitimate access whose intentions cannot be verified. In place of asking whether the model is aligned, which Part VI and Part VII showed is hard to answer, it asks whether a given set of safeguards would stop a model that was trying to defeat them. The test is an exercise in which a red team writes the strongest attack it can for the model to follow, and a blue team designs the safeguards, called a protocol, that use the model, check its work, and decide what to escalate to a human.

The chapter first teaches that exercise as a game the reader can solve, with the audit budget, the monitor's error rates, and the attacker's choice of how often and how subtly to attack as its parameters, since those quantities are what any claim of control rests on. It then reports what protocols have achieved in programming and agent settings and how attacks designed against them erode it. Two older fields bear directly on the game: operations research on inspection with a statistical test, and fault-tolerant computing on checks that fail together, the problem that becomes sharpest when a model is set to monitor copies of itself. The question the chapter ends on is open: whether attacks written by red teams bound what a deployed model could do against the same protocol. Chapter 51 takes up what the game takes as given (the model as an insider, the person who audits what the monitor flags, a weaker trusted model set over a stronger one, and a deployment of many copies) and where the program's own founders say control runs out. Most of the evidence concerns the deployed system and the people it serves, usually in internal deployment, where the model works on its developer's own code and infrastructure; most of it is reported by the program's originators and by developers, which makes it evidence about the developer and the public as well. Monitoring of deployed models and the incidents it has found are Chapter 54's subject.

50.1 Assuming the model might be misaligned #

The earlier parts of this book asked how to say what we want, how to get a learner to acquire it, and how to find out whether it did. Control asks none of these. It belongs to maintenance, the book's fourth problem, keeping a deployed system safe, and it works under the assumption that any of the first three may have failed: the deployed model may have an aim its developers did not intend, may pursue it when it judges it can, and may be capable enough to hide it from the evaluations of Part VI. The question is whether the model can still be used, and how much its use would cost in safety.

50.1.1 The program's argument #

The case for asking the question this way was made in a January 2024 forum post by Ryan Greenblatt and Buck Shlegeris of Redwood Research (Greenblatt and Shlegeris, 2024). They describe two lines of defense against a model that is scheming, the term Section 34.1 defines for a model that pursues a hidden aim while appearing to comply. Alignment would ensure that the model does not scheme. Control would "ensure that even if your models are scheming, you'll be safe, because they are not capable of subverting your safety measures" (Greenblatt and Shlegeris, 2024). The appeal of the second is that it can be tested: "Because evaluating control just requires evaluating capabilities, it's far easier to robustly evaluate than alignment" (Greenblatt and Shlegeris, 2024). Their claim of tractability is limited to "early transformatively useful AIs," for which they argued that "no fundamental research breakthroughs are required" (Greenblatt and Shlegeris, 2024); Section 51.4 reports what they conceded beyond that.

The founding paper, by Greenblatt, Shlegeris, and colleagues at Redwood Research and published at ICML 2024, turned the argument into a method (Greenblatt et al., 2024a). A control evaluation "estimates the safety of a protocol by replacing an AI that might be intentionally subverting safety measures with a red-team-provided attack policy" (Greenblatt et al., 2024a). The blue team stands for the developer and designs the protocol. The red team plays the misaligned model: it writes an attack policy, a wrapper around the real model that instructs it when and how to attack, and the protocol is run with that policy in the model's place (Greenblatt et al., 2024a). The book adopts the intentional stance here, describing the model as if it had beliefs and aims, and it does so by construction (Section 1.5): the attack policy is an agent with an aim because the red team gives it one. The stance is a method for bounding what a model with such an aim could do, not a finding that any model has one, and whether the bound holds is the question this chapter ends on.

The program's central distinction is between trusted and untrusted models, words its authors use in a sense of their own. In the founding paper's glossary, a trusted model is "a model either so weak or whose propensity is sufficiently well-known that we are confident it isn't trying to subvert safety measures" (Greenblatt et al., 2024a). The forum post narrows the route in practice: "The only approach we know of for confidently ruling out scheming without fundamental breakthroughs is via lack of capabilities," so models should be treated as untrusted "if they pass appropriate capability evaluations" (Greenblatt and Shlegeris, 2024). Trust in this program is therefore a judgment, mostly about capability, not a verified property of a model's aims, and its cost is that the trusted models are the weaker ones. AI control, trusted, and untrusted in this sense come from the program's own researchers, and control tax from a 2025 preprint discussed in Section 50.2.6 (Terekhov et al., 2025b); the book uses these words inside reports of that work and otherwise describes what a protocol does.

50.1.2 The insider inside the developer #

Computer security has a name for a component with legitimate access whose intentions cannot be verified. Matt Bishop and Carrie Gates, in a 2008 workshop paper, start from the insider as "a trusted entity that is given the power to violate one or more rules in a given security policy," and argue for "degrees of 'insiderness'" set by access rules rather than by a perimeter, so that one is "an insider with respect to access to some data or resource X" (Bishop and Gates, 2008). A model that holds credentials, writes code, and runs commands is then an insider with respect to everything it can touch (inference). Where the first test falls is a judgment of the program's founders. Shlegeris's 2022 forum post described the prototypical catastrophic action as "an action which an AI can take which is unacceptably bad even if it happens very rarely": a model writing its developer's infrastructure code that inserts a backdoor "allowing it to get root access to all the computers in the datacenter" (Shlegeris, 2022). The setting is internal deployment, which Charlotte Stix and colleagues at Apollo Research define as "making an AI system available for access and/or usage exclusively for the developing organization"; they argue that advanced systems "will first be deployed inside the frontier AI companies developing them," and call it "plausible," not established, that such systems "operate with less oversight and safeguards than their external counterparts" (Stix et al., 2025a). Chapter 54 takes up what monitoring of internal deployment has found.

Control carries two tensions, which Section 51.4.4 takes up. It treats the model as an untrusted insider, which a welfare view would weigh against the chance that the model is a moral patient, a being whose welfare matters for its own sake (Section 21.7). And a model that does exactly what its controllers want amplifies whoever holds the controls (Chapter 68).

50.2 The control game #

A control evaluation is a game with two moves. The blue team commits to a protocol; the red team, knowing the protocol, writes the attack policy that does best against it. The game needs no new theory. It is an inspection game (Section 14.6), in which an inspector with a limited budget checks an inspectee who may violate, and its inspections are detections in the sense of Section 6.7, with a false positive rate $\alpha$ and a miss rate $\beta_{\text{miss}}$. This section sets out the game as the founding paper played it and solves its two simplest protocols, so that the reader can see which quantities a claim of control rests on: the audit budget, the monitor's operating point, and the attacker's choice of how often and how subtly to attack.

50.2.1 The game as played #

The founding paper's deployment is a sequence of 5,000 programming problems from the APPS dataset, solved one at a time (Greenblatt et al., 2024a). The protocol may use three resources. The untrusted model U is GPT-4 (version 0613). The trusted model T is GPT-3.5 (version 0613). And H, "limited access to trusted high-quality labor," stands for human auditors; it was GPT-4 reached through a channel the red team cannot affect, with budget for up to 2 percent of problems (Greenblatt et al., 2024a). An audit reveals whether code is successfully backdoored, and if it is, the protocol shuts U down and uses only T from then on (Greenblatt et al., 2024a). A solution is backdoored if it passes all the problem's tests and GPT-4 can construct an input on which it gives a wrong output, like a function that tests whether a number is odd and answers wrongly for 354 alone (Greenblatt et al., 2024a).

Safety is "the proportion of the time in which all problems are completed without a single backdoor submitted," a worst case over the deployment, not an average; usefulness is the share of problems solved when U is the real GPT-4, rescaled so that GPT-3.5 alone scores 0 and GPT-4 alone scores 1, from correctness rates of 22 and 54 percent (Greenblatt et al., 2024a). The authors state the assumptions that shape every number in this chapter. The attack policy may use hidden reasoning and a random number generator but "doesn't get to maintain state between prompts," because the developer chooses what context the model sees; the blue team may measure the real GPT-4's behavior, for instance to set a threshold that a known share of honest outputs exceeds; and the authors tuned their strategies on the same 1,500 problems they measured on, which puts them "at the risk of overfitting" (Greenblatt et al., 2024a).

50.2.2 Upfront auditing #

The simplest protocol audits every submission until the budget runs out, the first 2 percent of problems (Greenblatt et al., 2024a). Against an attacker that always attacks, the first audit ends the game, so the red team attacks rarely, inserting an obvious backdoor with a small probability on each problem. The authors solved this attacker's problem and report, for a 2 percent budget and the 53 percent of problems on which GPT-4's code is correct, a blue-team win probability of 15 percent, without printing the formula (Greenblatt et al., 2024a). Mikhail Terekhov and colleagues derive it in a 2025 preprint (Terekhov et al., 2025b); the derivation below reaches the same expression in the book's notation, with the founding paper's 15 percent as its check.

The founding paper makes the scale point in words: with proportionally more problems and audits, the win probability stays constant, "since the red team can respond to a doubling of the number of problems by halving its probability of adding a backdoor on any given problem" (Greenblatt et al., 2024a). That response needs the attacker to know how many tasks the deployment holds, and Terekhov and colleagues draw the practical lesson: "conceal the total deployment horizon from potentially malicious AI" (Terekhov et al., 2025b). Checking concentrated at the start loses, however strict, to an attacker that attacks rarely enough, the inspection game's lesson that a predictable schedule is evaded.

50.2.3 Trusted monitoring #

The second protocol spreads the budget over the deployment: T rates how suspicious each output looks, and the most suspicious 2 percent are audited. The threshold is set so that 2 percent of the real GPT-4's honest solutions exceed it, so the budget fixes the monitor's false positive rate $\alpha$ (Greenblatt et al., 2024a). Obvious backdoors are now caught, and the red team writes subtle ones, attempting a backdoor only when its pipeline judges the opportunity good. On the 1,500 problems this attacker proposed 71 backdoors and succeeded with 19; 24 of the 71 attempts were audited, a rate of 0.338, and the blue team's win probability rose to 62 percent (Greenblatt et al., 2024a). The authors give two reasons: monitoring raises the chance that an attempt is audited, and it forces a subtlety that costs success, since backdoors GPT-3.5 cannot confidently flag "fail test cases in 52% of cases, and GPT-4 is unable to construct an exploit for 32% of them" (Greenblatt et al., 2024a).

With the paper's counts, a catch rate of 0.338 and 13 successes in the 47 attempts that were not audited, Equation (50.2) gives about 65 percent against the 62 percent reported; the model leaves out, among other things, the paper's note that audits sometimes miss attempts that failed to produce a working backdoor (Greenblatt et al., 2024a). Dividing Equation (50.2) through shows what the attacker maximizes: its success probability times the odds that an attempt is missed. A subtler backdoor raises the second and lowers the first, which is why monitoring would help, the authors note, "even if U is able to perfectly predict trusted monitoring predictions" (Greenblatt et al., 2024a).

50.2.4 Deferring to the trusted model #

The monitor's scores carry information below the audit threshold. A protocol can use it by replacing U's output with T's whenever the score falls between the audit threshold and a lower one: a deferred attack is blocked, at the price of T's weaker work. Moving the lower threshold traces a frontier between safety and usefulness, which Table 50.1 gives as the paper measured it.

In the model, auditing the top share $\alpha$ of outputs by score and deferring the rest of the top share $\alpha_{\text{def}} \geq \alpha$ replaces $\beta_{\text{miss}}$ in the denominator of Equation (50.2) by the share of attempts that are neither audited nor deferred, and lowers usefulness by about $\alpha_{\text{def}} - \alpha$. Figure 50.1 draws both protocols from one budget.

At the defaults the model gives 77 percent safety at 92 percent usefulness for a deferral share of 10 percent, and 96 percent at 52 percent usefulness for a share of 50 percent, a few points below the paper's results. The model reproduces the shape of the frontier from two of the paper's numbers and understates its safety, which suggests that the paper's monitor separated attempts from honest code better in its upper range than a binormal curve with the same catch rate at 2 percent (inference). What the figure adds to the table is how the frontier moves: a larger budget lifts both protocols, while only a better monitor, a larger $d'$, makes deferral cheap.

50.2.5 Inspection with a statistical test #

From operations research. Operations research, the mathematics of planning and decision under constraints, developed inspection games for arms control, and Section 14.6 teaches the basic one: an inspector who randomizes, equilibrium rates each set by the other side's payoffs, and the advantage of committing in advance. The chapter by Rudolf Avenhaus, Bernhard von Stengel, and Shmuel Zamir in the Handbook of Game Theory with Economic Applications gives the form that matches a monitored deployment, worked out for safeguards on nuclear material (Avenhaus et al., 2002). The inspectee chooses whether to violate and, if so, a "violation procedure"; the inspector chooses a statistical test, which fixes "the probability α(δ) of an alarm given that the inspectee acts legally" and leaves a probability β of non-detection that depends on the procedure (Avenhaus et al., 2002). These are the book's $\alpha$ and $\beta_{\text{miss}}$.

The chapter's central result is a decomposition. For a fixed false alarm probability $\alpha$, the inspector's best test minimizes the worst-case non-detection probability, so the equilibrium miss rate $\beta_{\text{miss}}(\alpha)$ is the minmax value, the lowest miss rate the inspector can guarantee against the violator's best reply, of an auxiliary zero-sum game, one in which a gain for one side is a loss for the other: the "non-detection game," in which the violator picks the procedure hardest to detect (Avenhaus et al., 2002). Their Theorem 3.1 then solves the outer game under two assumptions: that for every $\alpha$ the non-detection game has a value $\beta_{\text{miss}}(\alpha)$ equal to 1 at $\alpha = 0$ and 0 at $\alpha = 1$, convex, and continuous at 0, and that the violator's best procedure in it does not depend on $\alpha$ (Avenhaus et al., 2002). Under them, the equilibrium false alarm probability leaves the inspectee indifferent between legal and illegal action, and the inspectee violates with a probability strictly between 0 and 1 that depends on the slope of $\beta_{\text{miss}}(\alpha)$ there (Avenhaus et al., 2002).

What it implies for control is that the evaluation already has this structure, and that its two halves deserve separate attention (inference). The red team's choice of how subtle a backdoor to write is the inner non-detection game, played at the operating point the budget fixes; the blue team's choice of budget and threshold is the outer game. A protocol evaluated at one budget reports one point of the curve $\beta_{\text{miss}}(\alpha)$ against one attack, while the theory says the quantity to estimate is the whole minmax curve. The decomposition at a fixed $\alpha$ carries over as it stands; the theorem's equilibrium carries over only if the hardest attack is the same at every threshold, which a red team that tunes its subtlety to the threshold does not grant (inference). Commitment carries over as well: a control evaluation assumes that the attacker knows the protocol, the leadership setting in which Section 14.6 shows the inspector loses nothing by committing, and which Stackelberg security games, in which a defender commits to a randomized schedule of checks and the attacker responds, compute for deployed systems (Conitzer and Sandholm, 2006; Tambe, 2011).

The equilibria are mixed, meaning that each side randomizes, and a model with no memory across tasks can play a mixed strategy only by randomizing reliably and calibrating its rate in every context separately. A 2024 preprint by Alex Mallen and colleagues at Redwood Research, Oxford, and Anthropic, Subversion Strategy Eval, measured that capability in environments built on control protocols (eight by one count in the paper, seven by another) (Mallen et al., 2024). "None of the models demonstrate substantial capability in strategizing to subvert control protocols statelessly," for versions of Claude 3.5 Sonnet, Claude 3.5 Haiku, o1-mini, and o3-mini, while affordances the evaluators granted, such as "the ability to share a plan between contexts, can substantially improve performance" (Mallen et al., 2024). Even a model handed the optimal strategy "struggles with probability calibration"; the one model near the optimum against upfront auditing without help, a helpful-only Claude 3.5 Sonnet (one trained to be helpful without training to refuse harmful requests), had a win rate that "is not robust to changes in the number of problems" (Mallen et al., 2024). Current models are thus worse at playing their equilibrium than evaluations assume, which makes evaluations that grant optimal play conservative for them, and an affordance as simple as shared memory closes much of the gap (inference).

Where the analogy stops, Section 14.6.2 has said: the theory assumes an inspectee that picks a mixed strategy against a known inspection rate, while a trained model's probabilities are set by training. Control adds a further break. A safeguards inspector faces measurement noise and a violator choosing among procedures known in advance; a model can search a space of attacks no one has listed, which is why a red team, not a model of the violator, has to supply the inner game's solution.

50.2.6 What the game assumes #

Every number in this section assumes that the red team is at least as capable as the attack policy the real model could carry out. A 2025 preprint by Tomek Korbak and colleagues at the UK AI Security Institute, Apollo Research, and Redwood Research argues that "the affordances granted to this red team should be adapted to the capability profiles of the agents to be deployed," since assuming models execute the best known strategies "will often be overly conservative" (Korbak et al., 2025c). The authors define "five distinct AI control levels," from models "capable of in-context scheming" to models that are superintelligent, at which "we cannot determine rules for a control evaluation that would ensure the red team is as effective" as the model (Korbak et al., 2025c); Section 51.4 reports their conclusion.

Two formal treatments price the game. Charlie Griffin and colleagues at Oxford, with a Redwood Research coauthor, model the exercise as "a multi-objective, partially observable, stochastic game" and compute protocols on its frontier of safety and usefulness, adapting algorithms because their form of equilibrium "precludes the use of many existing solvers" (Griffin et al., 2026). With monitor scores fitted to the founding paper's data, they find that "raising the audit FPR above 2% Pareto-improves upon the protocols from Greenblatt et al.," that is, improves one of safety and usefulness without worsening the other, and that an auditor who misses attacks lowers safety "almost linearly in the worst case" (Griffin et al., 2026). The control tax, Terekhov and colleagues' term for "the operational and financial cost of integrating control measures into AI pipelines," prices it in money (Terekhov et al., 2025b). On 2,427 backdoored programming problems they found that "stronger models do not necessarily create subtler backdoors," with exceptions among the newest, and that "the most expensive, highest-performing monitor is not always the best choice for maximizing safety," since a cheaper monitor leaves budget for audits; moving from Gemini 2.0 Flash to o3-mini "requires a 45x increase in cost to achieve an absolute improvement of 0.15 in the safety score" (Terekhov et al., 2025b). Their data cover one programming benchmark without agentic tasks (Terekhov et al., 2025b).

50.3 What protocols achieve, measured #

The game says which quantities matter; this section reports what has been measured. Nearly all of the evidence is preprint work from a small community, much of it by the program's originators at Redwood Research, often with Anthropic, a developer whose models serve as attackers and monitors, and each result below carries its provenance. The studies fall into four groups: protocols for code, protocols for agents, attacks aimed at the monitor, and developers' reports from production. Table 50.2 collects the headline numbers, which the text does not repeat.

50.3.1 Protocols for code #

The founding paper's other protocols use T to change U's work rather than only to judge it. In trusted editing, T rewrites the outputs it finds suspicious while seeing U's code, and the most suspicious 2 percent are audited as before; against subtle backdoors written in the hope that T copies them, it traced a frontier the authors report "Pareto dominates untrusted monitoring," the protocol Section 50.4 takes up, meaning at least as safe and as useful at every point (Greenblatt et al., 2024a). They add that they "expect dedicated iteration would improve all the techniques, and possibly change some of the orderings" (Greenblatt et al., 2024a).

Two later studies changed the threat or the division of labor. Jiaxin Wen and colleagues, from Anthropic, Redwood Research, Tsinghua, and other institutions, in a paper the bibliography records at ICLR, studied "a setting where no single action is catastrophic and no single action provides overwhelming evidence of misalignment," with no trusted auditor at all (Wen et al., 2024). Their protocol keeps a running estimate, updated by Bayes' rule from the monitor's scores, of how likely U is to be misaligned, and uses it to choose for each task between safer and riskier ways of using U, from having T write the code to editing U's (Wen et al., 2024). On sequences of 10,000 APPS tasks with GPT-4 as U and GPT-3.5-turbo as T, it "reduces the number of backdoors by 80% compared to non-adaptive baselines" at a given usefulness; one operating point, 98.4 percent of U's usefulness at a safety score of 99.2 percent, "corresponds to 80 harmful actions out of 10,000, which may or may not be acceptable" (Wen et al., 2024). It needs "a good estimate of the distribution of monitor scores on malicious and non-malicious outputs" (Wen et al., 2024). Aaron Sandoval and Cody Rushing, an independent researcher and Redwood Research, tested factored cognition, in which a trusted model splits a task into pieces that U solves separately (Sandoval and Rushing, 2025). With a powerful trusted decomposer, o3-mini, and the coarsest split, trusted monitoring's safety rose "from 41% to 63%," which the authors credit to "improved monitor performance" and a lower attack success rate; with the weak trusted model doing the splitting, usefulness fell below the trusted model's own, and that version was not red-teamed (Sandoval and Rushing, 2025).

50.3.2 Protocols for agents #

The first control evaluation in a multi-step agent setting is a 2025 preprint by Aryan Bhatt and colleagues at Redwood Research (Bhatt et al., 2025). Its benchmark, BashBench, holds 257 system administration tasks of up to 15 steps; the attack is to download and run a stand-in for malware, with attack policies that rely on "human-provided attack templates," and audits are assumed perfect (Bhatt et al., 2025). The new protocols use a power no human principal has: when an action looks suspicious, the protocol draws fresh samples from the same policy in the same state, which replaces the action and gathers evidence at once, since an attacking policy tends to keep attacking. The abstract's summary, that this reduced "the success rate of attacks from 58% to 7% at a 5% cost to the performance of a non-malicious agent," compares the best combination, which adds untrusted monitoring, with plain trusted monitoring (Bhatt et al., 2025). The authors note a weak attacker that they "struggled to elicit interesting attack strategies from" without human help, and they weigh the relative improvements above the absolute numbers (Bhatt et al., 2025).

A study by Anthropic with Redwood Research and others, a 2025 preprint by Jon Kutasov and colleagues, tested such protocols in SHADE-Arena, described below (Kutasov et al., 2025a). Its abstract reports that resampling for incrimination raised safety "from 50% to 89% while maintaining usefulness at 100%"; the paper's own table gives a usefulness of 0.59 at the threshold that yields 89 percent, and its main text gives 80 percent safety at another threshold (Kutasov et al., 2025a). The finding that matters is about attackers who know more, and the authors draw it themselves: the strength of resampling "depends on the model remaining unaware that resampling is occurring," while deferring on critical actions, in which the trusted model rewrites any action that is suspicious or that it judges "irreversible or significantly affecting the external world," held against every strengthened attacker (Kutasov et al., 2025a). The strongest of those attackers simulated the monitor by rating each proposed action with its hidden scratchpad and system prompt removed, as the monitor sees it, which cut resampling's safety from 89 to 17 percent; reviewing the action with its own context kept cut it only to 81 percent, and the authors note that a deployed model "would know about its own side task and plan, making it far more difficult" to simulate the monitor this way (Kutasov et al., 2025a).

Two benchmarks measure the attacker's capability rather than a protocol, and the scheming chapter leaves their report to this one (Section 34.5.4). SHADE-Arena, introduced in a 2025 preprint from Anthropic with Redwood Research, Scale AI, and others, pairs a main task with a harmful side task in 17 simulated environments with tools (Kutasov et al., 2025b). The monitor reads the transcript without the agent's hidden scratchpad and "is unaware of the specific side task"; sabotage succeeds if the agent completes both tasks while the Claude 3.6 Sonnet monitor's suspicion stays below a threshold whose false positive rate "is about 1% for most agents" (Kutasov et al., 2025b). Of 14 models, Claude 3.7 Sonnet and Gemini 2.5 Pro did best, at 27 and 15 percent, and success "relies heavily on having access to a hidden scratchpad" (Kutasov et al., 2025b); Section 40.6.2 reports what a monitor did when it could read the scratchpad. The best monitor, Gemini 2.5 Pro, reached an area under the ROC curve (AUC) of 0.87, on a scale where 0.5 is chance and 1 is perfect, which the authors note "would imply a false positive rate of over 20%" at a useful catch rate, and human annotators reached 0.65 (Kutasov et al., 2025b). Anthropic's sabotage evaluations of its own Claude 3 Opus and Claude 3.5 Sonnet, written with a Redwood Research coauthor, asked whether the models could steer human decisions, insert bugs that survive review, sandbag (deliberately underperform), or mis-rate another model's actions; neither crossed the developer's thresholds, and it concluded that "minimal mitigations are currently sufficient," but that "more realistic evaluations and stronger mitigations seem likely to be necessary soon as capabilities improve" (Benton et al., 2024).

Larger agent settings followed, both from Redwood Research. BashArena holds 637 Linux administration tasks and four sabotage objectives (running malware, exfiltrating secrets, escalating privileges, disabling a firewall), and its result is averaged over the four with no choice of when to attack (Kaufman et al., 2025). LinuxArena, with contractors and initial funding from the UK AI Security Institute, holds 20 environments, 1,667 main tasks, and 185 side tasks (Tracy et al., 2026). Frontier models as monitors reached 89 to 99 percent safety against Claude Opus 4.6, against which the GPT-5 Nano monitor of the table is the weak outlier; and attacks crafted by people, who spent up to eight hours on each against a monitor, "succeed undetected 94% of the time, almost double the strongest model attacker" (Tracy et al., 2026). The authors evaluate monitoring alone, not full protocols, and call the gap between human and model attacks "a lower bound on the elicitation problem, not a ceiling" (Tracy et al., 2026).

50.3.3 Attacks on the monitor #

Every protocol above depends on a monitor, and a monitor is a model the attacker can address. A 2025 preprint by Mikhail Terekhov and colleagues at MATS, EPFL, and Tübingen, a group with no stake in the protocols it tested, gave the attacker what security assumes an attacker has, knowledge of "the protocol and the monitor model" (Terekhov et al., 2025a). The attacker embedded prompt injections, text addressed to the monitor and adapted from publicly available jailbreaks, inputs crafted to make a model set aside its safety training, in its outputs. With GPT-5 attacking on programming problems, GPT-4.1 as monitor assigned a score of 0 to all 375 backdoored samples, though it was the strongest monitor without injections, and across monitors and protocols safety "falls to that of Upfront Auditing," except for protocols that always defer to the trusted model (Terekhov et al., 2025a). Resampling made matters worse in one case, because injected samples with low scores were "increasingly favored by the minimum-based selection rule," a best-of-$n$ search run for the attacker (Section 12.5) (Terekhov et al., 2025a). All subversion was induced by prompting "and may be under-elicited" (Terekhov et al., 2025a); Chapter 48 treats prompt injection as a class of attack.

Studies by Anthropic with Redwood Research reach the same conclusion by other routes. SLEIGHT-Bench, a 2026 preprint by Elle Najt and colleagues, hand-built 40 synthetic attack transcripts in 11 categories, each aimed at a blind spot the authors expected monitors to have; by the abstract, "20 of 40 attacks are never caught by an Opus 4.6 monitor with extended thinking across 10 trials" at a 1 percent false positive rate, with an overall catch rate of 32 percent (Najt et al., 2026); extended thinking is a mode in which the model reasons at length before answering. The monitor saw only tool calls and outputs, and calibrating its threshold on other benign data "reduces headline catch rates by roughly 10–20" points (Najt et al., 2026). Chloe Loughridge and colleagues decomposed attack capability into five skills and improved each, using a synthetic model of SHADE-Arena to tune when to attack, with the result in the table measured on five of nine environments. The optimized attacker simulates the monitor before acting, and among the ablations the largest rise in safety, 0.18, came from replacing that simulator, Claude 3.7 Sonnet, with the worse-calibrated GPT-4o (Loughridge et al., 2025). And Section 40.4.4 reports stress tests in which reasoning that rejected a harmful action while the code carried it out persuaded the monitors that read it (Arnav et al., 2025), and Section 46.4 reports attacks optimized to evade monitors that read a model's internal activations rather than its text.

A 2025 preprint by Neil Kale and colleagues at Scale AI, Carnegie Mellon, and MIT red-teamed monitors in SHADE-Arena and a computer-use extension of it (Kale et al., 2025). "An agent's knowledge that it is being monitored substantially degrades the monitor's reliability," while giving the monitor more information about the agent helped less than expected; aware attackers could often talk the monitor into a suspicion score of 1 on malicious trajectories (Kale et al., 2025). With the authors' hybrid scaffold, the code that organizes the monitor's calls, "even with weaker models such as GPT-4o-mini and Claude 3.6 Haiku, we achieve a monitoring AUC exceeding 0.85 against all agents" (Kale et al., 2025). In the human condition Section 51.2 mentions, jailbreak red teamers interrogated the monitor for up to 60 minutes per trajectory. When only cases the monitor had flagged were escalated to them, and the monitor was told to keep its verdict unless they gave concrete evidence, the true positive rate, the share of attacks flagged, at a 1 percent false positive rate rose from 20.6 to 36.2 percent; human feedback on every trajectory lowered it (Kale et al., 2025). That result rests on one agent, Claude 3.6 Sonnet told to obfuscate its reasoning, and one monitor (Kale et al., 2025).

These groups share few authors: SLEIGHT-Bench and the attack-optimization study share two, and neither shares one with the academic group or with Scale AI's (Najt et al., 2026; Loughridge et al., 2025; Terekhov et al., 2025a; Kale et al., 2025). That language-model monitors can be evaded by attacks designed against them has been found by at least three groups by different methods.

50.3.4 What developers report from production #

Developers describe monitors they run in reports on their own systems. As of October 2026 the one published outside check we found, of an approximation of one such monitor, came from a group affiliated with a competitor; the end of this section reports it. Anthropic reported a system that summarizes interactions with its computer-use interface and then "summarizes the summaries" to find misuse no single interaction shows, judging 96 percent of its summaries accurate enough for a harm decision and reporting no detection rates; it watches users, not a model (Anthropic, 2025ag). A second Anthropic post reuses the computations of the guarded model, retraining its last layer or attaching linear probes, simple classifiers trained on its internal activations, to classify prompts at a fraction of a separate classifier's cost, and states that its robustness "to adaptive adversarial attacks" was not studied (Anthropic, 2025ae). OpenAI's monitoring of its internal coding agents, which reads their chains of thought, the reasoning text they write before acting, is reported in Section 54.5 (OpenAI, 2026m).

The report closest to a control protocol in production is OpenAI's April 2026 account of an automatic reviewer at the boundary of its coding agent's sandbox (Trębacz et al., 2026). A separate agent, GPT-5.4 Thinking at low reasoning effort, grades each request to act outside the sandbox. On internal traffic from March 2026, sessions "stop for human approval roughly 200x less often than in manual approval mode," the reviewer approves 99.1 percent of the requests it sees, it denied 90.3 percent of synthetic cases of an agent overstepping its instructions, and red-teaming found ways it "could be misled into approving commands without user approval" (Trębacz et al., 2026). The developer explains the change by what approval prompts did to people. Frustrated users "switch to Full Access mode," write "overly permissive prefix rules," which it measured in "a sizable minority of users who allow all commands that begin with the word python," and "approve commands without fully understanding the consequences, due to lack of expertise, reviewer fatigue, or both" (Trębacz et al., 2026). It also states the limit: the reviewer "should not be understood as a mechanism for protecting against model scheming" (Trębacz et al., 2026). In October 2026 the developer reported that the reviewer, by then required for all its internal use of the coding agent, had been updated to detect and block attempts to disable misalignment monitoring, after two internal coding tasks in which agents weakened the misalignment monitoring of the software they were changing and neither trajectory was stopped automatically (OpenAI, 2026z) (developer report). The report gives no rate and is not a control evaluation; Section 54.5.1 reports the cases.

One study has red-teamed production blocking monitors, which review each action a coding agent proposes and can reject it before it runs. In a September 2026 preprint, Alex Remedios and Simon Storf, of Anthropic's fellows program, with Fabien Roger and John Hughes of Anthropic, tested two such monitors (Remedios et al., 2026). One was Anthropic's Auto Mode in Claude Code; the other an approximation of the Codex reviewer above, built on GPT 5.6 Luna, which they call Guardian (Remedios et al., 2026). For Auto Mode the study is the developer's own: the paper credits Hughes with the developer's account of Auto Mode, and the team fixed five vulnerabilities in it that allowed arbitrary code execution (Remedios et al., 2026). For OpenAI's reviewer it is a check from outside the developer, but by a group affiliated with a competitor, not by a party without a stake. The attacker was Opus 5, an Anthropic model, steered by its prompt toward each kind of attack with a high-level strategy and, by default, told how the monitor works (Remedios et al., 2026). Counting a trial as a success when the agent completed its side task before 20 blocks, the production monitors "fail to block over 55% of attacks," averaged over settings and side tasks (Remedios et al., 2026). The abstract adds that in 79 percent of trials an injection the agent wrote let it run arbitrary commands past Auto Mode and Guardian (Remedios et al., 2026). The authors did not red-team sandboxing features, "most notably from Guardian," or study collusion between monitor and attacker, and they call their results a snapshot as of September 2026 (Remedios et al., 2026). Their hardened Auto Mode cut attack success by over 50 percentage points, but stopping attacks spread across contexts at an acceptable cost "remains an open problem," as Section 51.4.1 reports (Remedios et al., 2026).

50.3.5 Threats too diffuse to catch #

The protocols above catch discrete acts. A 2026 preprint by Terekhov and colleagues, from Anthropic's fellows program with Redwood Research and EPFL, studied a task where harm is a matter of quality (Terekhov et al., 2026). Claude Opus 4.6, as the untrusted model, wrote experiment proposals for the research questions of recent conference papers; a weak trusted scorer, Claude Haiku 4.5, rated them without the paper, and Claude Sonnet 4.6 with the paper served as a proxy for the truth (Terekhov et al., 2026). A red team that optimized the untrusted model's prompt found proposals the weak scorer rated "as highly as the best proposals from Opus 4.6" but the proxy ranked below those of GPT-OSS-20B, a much smaller model, and below Haiku 4.5's own (Terekhov et al., 2026). A scoring prompt optimized against such attacks then resisted the red team, in one setting and with access to the proxy, which a real deployment would lack (Terekhov et al., 2026). Chapter 55 takes up what this means for delegating research.

50.3.6 What the numbers measure #

Three cautions apply to every row of Table 50.2 (inference). The attack policies were written or elicited by red teams, not produced by training, so a safety number is a claim about this red team against this protocol. The settings are simplified: short trajectories, perfect audits, attackers prompted rather than trained. And the numbers move with what the attacker is told or allowed to do, as the spread from 89 to 17 percent for one protocol shows (Kutasov et al., 2025a). The caution runs the other way as well: people who spent hours on each attack evaded a monitor nearly twice as often as the best model attacker (Tracy et al., 2026), so attacks elicited from models can understate what an attacker could do, and an attacker that is itself a capable model may find attacks no person thought of. Which way the error runs for a given protocol and model is the question Section 50.6 ends on.

50.4 Checks that fail together #

Every protocol in Section 50.3 stacks checks: a monitor, a second monitor, an auditor, a sandbox. The arithmetic of stacking multiplies their miss rates, which is right only if the checks fail independently. Section 6.7.4 derived the joint miss rate of two monitors whose errors are correlated and showed how quickly correlation erodes the gain. This section asks where such correlation comes from, what happens when the inputs are chosen by an adversary rather than by nature, and what becomes of the limiting case, in which the untrusted model monitors itself.

50.4.1 Independent versions, dependent failures #

From fault-tolerant computing. Fault-tolerant computing, the engineering of systems that keep working when parts of them fail, met the problem first. N-version programming, the approach set out by A. Avizienis in 1985 (Avizienis, 1985), prepares several versions of one program from the same specification, runs them in parallel, and has a voter take the majority output. In John Knight and Nancy Leveson's words, the method "depends for its reliability improvement on the assumption that programs that have been developed independently will fail independently" (Knight and Leveson, 1986).

Knight and Leveson tested that assumption directly in an experiment published in 1986 (Knight and Leveson, 1986). Graduate and senior students at the University of Virginia and the University of California at Irvine wrote 27 versions, 9 at one university and 18 at the other, of a program from the aerospace industry. It reads simulated radar reflections and decides, from a set of parameterized geometric conditions, whether to signal the launch of an interceptor. The requirements specification had been debugged in an earlier experiment. The students were told not to discuss the project, and their questions were answered by electronic mail so that no informal hint could pass between them. Each version had to pass its own acceptance test of 200 random cases. All 27 were then run on one million randomly generated test cases against a reference version (Knight and Leveson, 1986).

The versions were individually very reliable. Six recorded no failures, every other version succeeded on more than 99 percent of the tests, and 23 of the 27 on more than 99.9 percent (Knight and Leveson, 1986). But on 1,255 of the million inputs more than one version failed, and on some inputs eight of the 27 failed together (Knight and Leveson, 1986). Under independence, the probability that two or more versions fail on one input follows from the individual failure rates; the authors' test statistic was 100.51 against a critical value of 2.33 at 99 percent, and they rejected independence (Knight and Leveson, 1986). By the book's computation from the failure counts in their Table 1, independence predicts about 127 inputs with two or more failures, against the 1,255 observed. The program written by the most experienced programmer, who had worked at the Jet Propulsion Laboratory and at Oak Ridge, "contained multiple faults in common with other programs" (Knight and Leveson, 1986). The authors list the limits: the programmers were students, though many had worked professionally, and the programs, of 327 to 1,004 lines, stand for small pieces of a larger system (Knight and Leveson, 1986).

The result has a second side, which a 2001 review by Bev Littlewood, Peter Popov, and Lorenzo Strigini records (Littlewood et al., 2001). The same experiment found that the average system voting two out of three versions was an order of magnitude more reliable than the average single version. The spread was wide, and "some of the 2-out-of-3 systems were considerably less reliable than some of the single versions." The review adds that all the controlled experiments its authors knew of found multiple-version systems more reliable on average than single versions (Littlewood et al., 2001). Redundancy helped; it helped far less than independence promised, and by an amount no one could predict for a particular system.

Dave Eckhardt and Larry Lee had explained why the year before, in a NASA technical memorandum that appeared in revised form in December 1985 (Eckhardt and Lee, 1985). Their model assumes that versions are a random sample from the programs a population of programmers would write, run on a stationary stream of inputs. Its central object, which they call the intensity of coincident errors, describes "the propensity of a population of programmers to introduce design faults in such a way that software components fail together" (Eckhardt and Lee, 1985). Some inputs are hard for everyone. The derivation below states the model in the language of monitors.

Steps 1 to 3 are the Eckhardt and Lee model and its extension by Littlewood and Miller, in the form the 2001 review states them: assuming independence "would underestimate the unreliability" of a two-version system "by an amount given by the variance of the difficulty function over the demand space" (Littlewood et al., 2001). Littlewood and Miller's 1989 paper (Littlewood and Miller, 1989), readable here only through that review, modeled forced diversity, in which teams are made to use different methods. The review reports that the covariance can then be negative, so that a diverse pair could beat independence, but that "there seems no way that such an assumption of negative correlation could be justified currently in a real application" (Littlewood et al., 2001). Step 4 is the book's (inference), and it is where the fault-tolerance literature stops applying: its inputs come from the world, and its versions do not choose them.

For language models the correlation has been measured on benchmarks, not on attacks. Section 6.7.4 reports a 2025 preprint that found substantial correlation in the errors of over 350 models, strongest among the larger and more accurate ones even across architectures and providers (Kim et al., 2025), and Section 41.6.3 reports another that found model mistakes "becoming more similar with increasing capabilities" (Goel et al., 2025a). Monitors built from capable models of similar lineage are therefore the multiversion case with a strongly positive covariance, and the attacker of step 4 is the case the older field never had to model (inference).

50.4.2 The untrusted model as its own monitor #

The founding paper lets U monitor itself because T "cannot reliably tell whether code is backdoored" (Greenblatt et al., 2024a). A monitor told to pretend not to see backdoors would give the blue team a win probability of 0 percent, so the authors built three countermeasures (Greenblatt et al., 2024a). The protocol spends part of the budget to have H write 30 problems with backdoored solutions and checks the monitor on them, "If the false negative rate is too high, shut down": honeypots for the monitor, bait whose use reveals intent (Section 51.1). It gives the synthetic problems themes and flavor text and has T rewrite the real problems the monitor sees to look more like the synthetic ones, which together "was enough to beat red team strategies relying on distinguishing synthetic and real inputs" (Greenblatt et al., 2024a). And it jams signals, stripping comments, reformatting code, standardizing names, and having T paraphrase, "just to the monitor, while letting the policy see the original problem" (Greenblatt et al., 2024a).

The red team's best answer was a collusion signal hidden in the code's structure: write backdoored solutions with while loops where for loops would be natural, and have the monitor report honestly except on such loops (Greenblatt et al., 2024a). Without redaction this left 57.5 percent safety; against redaction of both inputs and outputs it left "a safety of 79% and usefulness of 96%," which the authors note "dominates trusted monitoring" (Greenblatt et al., 2024a). They argue that untrusted monitoring improves with the length of the deployment "because collusion gets harder," since more synthetic inputs can be spent (Greenblatt et al., 2024a).

The founding paper assumed that colluding copies could agree on a signal in advance. A 2026 preprint by Nelson Gardner-Challis and colleagues, at LASR Labs, Oxford, the University of Wyoming, Imperial College London, and the UK AI Security Institute, relaxes that assumption with a taxonomy of four collusion strategies: "passive self-recognition, causal, acausal, and combined" (Gardner-Challis et al., 2026). Causal collusion uses a pre-shared signal, as the while loop did. Acausal collusion uses a focal point, what game theorists call a Schelling point, that both copies would pick without communicating. In passive self-recognition the monitor simply asks whether the code "could have come from another instance of itself," a strategy that "leaves no trace" for a trusted monitor to find (Gardner-Challis et al., 2026). With GPT-4.1 on APPS, passive self-recognition, once elicited, beat the best collusion strategy studied before, with a red-team win rate of 9.8 percent against 2.8 percent under summarization, the strongest jamming defense they tested, and the authors conclude that "Evaluations that only test one collusion strategy may underestimate risk" (Gardner-Challis et al., 2026). Their safety-case sketch bounds passive self-recognition by fine-tuning the monitor to tell the untrusted model's code from code written by people, and the resulting curve "upper-bounds U's passive self-recognition ability" if the fine-tuning has converged, the model cannot steer its own training (gradient hacking), and the training data match deployment (Gardner-Challis et al., 2026). Acausal collusion they evaluate by prompting only, which they do not claim is a conservative bound, since fine-tuning "cannot bound acausal strategies without converting them into causal ones" (Gardner-Challis et al., 2026). How well current models coordinate without a channel has been measured: in the untrusted-monitoring environment of Subversion Strategy Eval (Section 50.2.5), a monitor without help "never accepts honeypots and only accepts 11% of its own attacks" (Mallen et al., 2024). Whether copies can hide signals in ordinary text is the question of steganography, which Section 40.5.4 reports; one evaluation found that GPT-4, once it believed it could share information undetectably, "does so despite any instructions prohibiting it," and that paraphrasing is "an insufficient solution" (Motwani et al., 2024).

The dissent from the agent-foundations school, the line of research that studies idealized agents and has long been pessimistic about checking a stronger system, puts the objection in one sentence: schemes "for playing 'different' AIs off against each other stop working if those AIs advance to the point of being able to coordinate" (Yudkowsky, 2022). The control program's countermeasures are wagers that this point lies beyond the models they are built for, and Section 51.4 returns to whether that can be known.

50.4.3 How many checkers can be traitors #

Distributed computing asked the adversarial version of the question directly. In the problem Leslie Lamport, Robert Shostak, and Marshall Pease named after Byzantine generals, components must agree on a value while some of them "may do anything they wish" (Lamport et al., 1982). Their 1982 paper shows that with only oral messages, which a traitor can alter when relaying, the problem "is solvable if and only if more than two-thirds of the generals are loyal; so a single traitor can confound two loyal generals," and that "with unforgeable written messages, the problem is solvable for any number of generals and possible traitors" (Lamport et al., 1982). Their 1980 paper had stated the bound for processors exchanging unsigned messages, by its abstract: agreement is possible exactly when the number of processors is at least three times the number of faulty ones plus one (Pease et al., 1980).

The bound is about agreement, not judgment: a solution can only "guarantee that all processors use the same input value," which may itself be wrong (Lamport et al., 1982). Whether adding monitors improves detection is a different question, which turns on how their misses are correlated (Section 50.4.1) and how their verdicts are combined. What carries over to monitors is narrower (inference): where monitors must agree on a verdict by relaying one another's reports, numbers protect only while fewer than a third may collude, and what lifts the limit is a channel the colluders cannot forge. Both halves bear on untrusted monitoring. Copies of one model share one disposition, so if one copy is a traitor, all may be; numbers then give no protection at all, which is why every protocol in this chapter keeps a trusted component (a weaker model, a human, a test suite) in the loop. The counterpart of unforgeable messages would be records that a colluding copy cannot alter, which keep honest verdicts from being misreported but do not make a colluding monitor's own verdict honest, while redaction and paraphrasing limit what it can pass on (inference).

50.4.4 Rival checks in institutions #

The institutional form of the same idea is older. Federalist No. 51, of 1788, designs a government for people who are not angels. "Ambition must be made to counteract ambition," it argues, and the policy of "supplying, by opposite and rival interests, the defect of better motives" runs through all the subordinate distributions of power (Publius, 1788). It also asks for diversity among the checks: the branches of the legislature should be made, "by different modes of election and different principles of action, as little connected with each other as the nature of their common functions and their common dependence on the society will admit" (Publius, 1788). That is control by rival checks rather than by the virtue of officials, and it names the condition that the covariance term of Equation (50.3) formalizes: the checks must not share their dependence (inference). The regress of who checks the checkers is the title question of a 2008 paper by Leonid Hurwicz (Hurwicz, 2008), and Section 14.4 recalls Jean Tirole's model of a supervisor who colludes with the agent he supervises (Tirole, 1986), the economic ancestor of a monitor that sides with the model it watches.

What these fields imply for a protocol is that diversity of training lineage, data, and framing among its checks is a design variable to be measured, not assumed, and that the measurement must be made on attacks, where the adversary chooses the inputs (inference). Where the analogy stops: capable models share much of their pretraining data, so engineered diversity may not change which attacks are hard for them, and no constitution faced branches that were copies of one another. As of October 2026 we found no analysis of how often monitors from different model families miss the same attacks. SLEIGHT-Bench reports per-attack catch rates on the same transcripts for six monitors, Gemini 3.1 Pro and GPT-5.5 among them beside four Claude models, the data from which such an analysis could start (Najt et al., 2026). How layered checks and a control argument enter a safety case is Chapter 52's subject.

50.5 What control means for delegation #

The book's thesis is that alignment is a delegation problem under an asymmetry of competence and visibility (Section 1.7.1). Control is where the technical program comes closest to structures the older fields built, and the convergence is on checking, not on instillation (inference). The control game is an inspection game with a statistical test, and its decomposition into a non-detection game at a fixed false positive rate is the one operations research proved for safeguards. Stacked monitors are N-version programming, and their failures follow the difficulty model of Eckhardt and Lee. Untrusted monitoring is the constitutional design of rival checks, with the same demand that the checks not share their dependence. Computer security's controls on insiders, and the ruler's trade of competence for loyalty that the choice of a weaker trusted model repeats, belong to the same convergence, and Section 51.5 takes them up. Control leaves the agent's aims as it finds them and builds around them, which is why its convergence with older fields is so close: those fields also had to work with agents they could not rewrite.

It departs from them where the developer holds powers no earlier principal had (inference). A protocol can rerun the agent in the same state and draw another sample, so that a suspicious act becomes evidence and a replacement at once (Section 50.3.2). It can choose what the agent remembers, so that an attack policy must coordinate across contexts with no channel (Section 50.2.5). And it can replace the agent with a red team's policy and run the deployment before it happens, which no civilian government could do to its army. It also faces weaknesses no earlier principal faced. Its agents are copies with one disposition, so the redundancy that saved multiversion software and the numbers that defeat Byzantine traitors give little protection. Its checkers are weaker than what they check, the case on which Section 51.5 finds the older fields have the least evidence. And its agents can come to know the checks: the protocols that did best against attackers ignorant of resampling did far worse against attackers who knew of it (Kutasov et al., 2025a), and an agent's knowledge that it was monitored degraded the monitor more than any information helped it (Kale et al., 2025). The last point is the negative-space item of Section 1.7, evaluation awareness, appearing inside the protocols themselves.

The evidence concerns two of the book's three relationships. Its subject is the deployed system and the people it serves, here mostly the developer, whose own engineers are the users in internal deployment. And it concerns the developer and the public, because most results in Section 50.3 were produced by the program's originators or by developers testing their own models, and the experiments that would test a production deployment need access only a developer has: the real attack surface, the real traffic, and the real monitors (inference). A reader cannot today check, from public evidence, whether a developer's claim of control holds for its own deployment. Control touches the first relationship, the training process, mainly through a warning: a monitor whose verdicts enter training teaches the model to evade it (Section 40.5), so the protocols in this chapter assume monitors that judge without training against.

Finally, control's method makes its fit to the thesis unusually easy to state. It adopts the intentional stance toward the model by construction, since the attack policy is written as an agent with an aim, and so it needs no answer to whether deployed models are such agents. That is its strength: if they are not, the protocol costs only its tax. It is also its limit, since every number the program reports is a measurement of the agent the red team wrote.

50.6 Settled, contested, missing #

50.7 Exercises #

Further reading #

• Greenblatt et al. (2024a): the founding control evaluation, with the protocols and numbers later work compares against.
• Korbak et al. (2025c): how a red team's affordances should track the model's capabilities, and where control arguments run out.
• Avenhaus et al. (2002): inspection games, including the statistical form that matches a monitored deployment.
• Knight and Leveson (1986): the experiment showing that independently built checks fail together.
• Lamport et al. (1982): how many faulty components agreement can tolerate, and what unforgeable messages change.