Draft complete but not yet reviewed; expect revisions

Safety Cases and Defense in Depth

The parts before this one asked whether a model's values are the intended ones and whether anyone can tell. By 2026 a careful developer's answer to both is "not with certainty", and the decision to train or deploy the next model still has to rest on something. In industries that run hazardous systems it rests on a safety case: a structured argument, supported by a body of evidence, that a system is safe enough for a stated use in a stated environment, written so that someone other than its author can check it. AI developers took the term from aviation, nuclear power, and defense, and their published safety frameworks now promise arguments of this kind when models cross capability thresholds (Chapter 62). Chapter 50 reported the research program that would argue control measures contain a model even if it tried to subvert them. This chapter asks how such arguments are put together, how much weight a stack of safeguards can bear, and what the first cases written for AI actually show.

A safety case is a claim made by a developer to its own leadership, to regulators, and to the public, so most of this chapter concerns the third of the book's relationships, the developer and the public. It reads every case with the rule of Section 5.6: a claim that a risk is low is a disclosure by the party with the most at stake, worth what its evidence and its reviewers make it worth. The chapter's question is what an argument can add when a test cannot establish the rate that matters. The older fields answer that an argument can make its assumptions explicit and reviewable, and that a stack of safeguards multiplies its protection only as far as the safeguards fail independently. The AI cases written so far are sketches whose authors name the assumptions they could not yet support, and none has been followed by a failure that could test it.

52.1 A claim that testing cannot carry #

Suppose a developer wants to say that a model, deployed as an agent inside the company, will sabotage the company's research less often than once in a million episodes. Section 6.5.1 gives the arithmetic of what a clean record can show: after $n$ clean episodes drawn like the ones the claim is about, the rate is bounded at 95 percent confidence by about $3/n$ ((6.7)). A bound of one in a million therefore needs about three million clean, representative episodes before deployment, for one model version.

Software engineering met this wall when it was asked to certify systems whose failure would kill people. Section 6.5.5 reports the result: the abstract of Butler and Finelli's 1993 paper calls the statistical quantification of life-critical software reliability infeasible (Butler and Finelli, 1993), Littlewood and Strigini's paper of the same year named the problem of validating ultrahigh dependability (Littlewood and Strigini, 1993), and Kalra and Paddock worked the same arithmetic for autonomous vehicles (Table 6.4) (Kalra and Paddock, 2016). A 2001 review by Bev Littlewood, Peter Popov, and Lorenzo Strigini adds the consequence for an assessor. Observing correct behavior over a short period, they write, "adds very little to any confidence we may have in reliability over long future periods," even when there are other sources of confidence in the software (Littlewood et al., 2001). And operating experience gathered after release, however good, "is not the same as an assurance prior to deployment that a very high reliability has been achieved" (Littlewood et al., 2001).

For a model there is a second difficulty, which earlier programs posed only when their makers built it in, as Volkswagen built diesel software "designed to detect when the vehicle is undergoing emissions testing" (U.S. Environmental Protection Agency, 2015) (Section 37.7). The rule of three assumes the test episodes are drawn like the episodes of use. Section 37.2 derives what happens when a system can tell the two apart. The gap between the rate measured in tests and the rate in deployment is bounded by how far apart the two input distributions are, and a policy that behaves badly exactly where deployment differs from testing can reach that bound. A clean record then bounds the rate on tests, which is not the rate the claim is about.

So a claim of a very low rate has to rest on something besides the system's observed behavior: on the structure of the system around the model (what the model cannot reach), on the limits of the model (what it cannot do), and on evidence of several kinds, of which tests are one. Assembling that into something another party can check is what a safety case does. A 2025 preprint by Benjamin Hilton, Marie Davidsen Buhl, Tomek Korbak, and Geoffrey Irving quotes a definition from a 2007 UK Ministry of Defence standard: "a structured argument, supported by a body of evidence, that provides a compelling, comprehensible, and valid case that a system is safe for a given application in a given environment" (Hilton et al., 2025). Its parts have names. The top claim is the conclusion the case argues for, such as "this deployment is acceptably safe"; the evidence is the test results, analyses, and records offered in support; and the operating context is the use and environment for which the claim is made, outside which the case says nothing. The wider term assurance case names the same form applied to properties other than safety, such as security, and appears in the title of a 2009 review by Robin Bloomfield and Peter Bishop (Bloomfield and Bishop, 2009).

Other industries certify what they cannot test, and how they do it is instructive. In the abstract of a 2010 paper, John Downer describes how the US Federal Aviation Administration certifies new aircraft designs: it delegates "a large amount of this work to the manufacturers themselves" (Downer, 2010). It suggests that regulators of high technologies face "an inevitable epistemic barrier" that forces them to delegate technical questions to people with more tacit knowledge, and so to regulate "by evaluating 'trust' rather than 'technology'" (Downer, 2010). A 2011 paper of his treats, in its title's words, the limits of knowledge and the sociology of inevitable failure (Downer, 2011). Section 64.4 takes up the aviation record. A regulator of frontier AI today faces the same barrier with less operating history, and the developers hold most of the relevant knowledge (inference).

AI developers committed to arguments of this kind before any had been written. At the AI Seoul Summit in 2024, sixteen organizations, four more added later, agreed to set "thresholds at which severe risks posed by a model or system, unless adequately mitigated, would be deemed intolerable" (Department for Science and Technology, 2024). In the extreme, they committed "not to develop or deploy a model or system at all, if mitigations cannot be applied to keep risks below the thresholds" (Department for Science and Technology, 2024). The frameworks that followed tie capability thresholds to written arguments: as of October 2026, Anthropic's Responsible Scaling Policy calls for an "affirmative case", Google DeepMind's Frontier Safety Framework for a safety case, and OpenAI's Preparedness Framework for a Safeguards Report once a model crosses a set threshold (Section I.2.12) (Anthropic, 2026h; DeepMind, 2026; OpenAI, 2025e). All three are developer documents, and "affirmative case", "Critical Capability Level", and "AI Safety Level", the levels of Anthropic's policy, are developer terms that Appendix A records; Chapter 62 analyzes the frameworks.

Two preprints made the case for the form. Marie Davidsen Buhl and colleagues at the Centre for the Governance of AI argued in 2024 that safety cases could serve both industry self-regulation and government regulation (Buhl et al., 2024). They also recorded the form's weak point: "there is little empirical evidence for the efficacy of safety cases," though practitioners mostly consider them effective, and they have been criticized for giving a false sense of assurance (Buhl et al., 2024). Hilton and colleagues argued that writing and reviewing safety cases "would substantially assist in the fulfilment of many of the Frontier AI Safety Commitments" (Hilton et al., 2025). The second key update to the 2025 International AI Safety Report found that "their current adoption is limited and their suitability for general-purpose AI systems remains contested" (Bengio et al., 2025c). The form arrived in AI as a promise, before its record there could be judged.

52.2 Claims, arguments, evidence, and confidence #

A case is not a pile of test results. What makes it a case is that it says how the evidence supports the conclusion, under which assumptions, and for which use, so that a reviewer can find the step that fails. This section sets out the form as the field that built it describes it, then the field's own strongest critique of the form and the rival way of arguing for safety that the critique proposes.

From safety engineering. Safety engineering, the discipline that designs and certifies hazardous systems such as aircraft, reactors, and rail networks, has argued for safety in this form for decades. Its own literature on the form is read here by title only: Bloomfield and Bishop's 2009 review of safety and assurance cases (Bloomfield and Bishop, 2009), Kelly and McDermid's 1997 work on constructing and reusing safety cases with patterns (Kelly and McDermid, 1997), and Hawkins and colleagues' 2010 approach to creating clear safety arguments (Hawkins et al., 2010). The account below comes from readable work that summarizes the tradition for AI.

In Buhl and colleagues' summary, a case has four components: objectives that must be met for the system to be safe enough, such as a risk below a threshold; arguments that the objectives have been met; evidence that the arguments are true; and a scope within which the case holds (Buhl et al., 2024). The arguments are made "with the assumption that the system is unsafe unless convincing arguments are made to the contrary," and in their sketch each comes with "an explanation of the developer's level of confidence that each claim is true and that the objectives have been met" (Buhl et al., 2024). A case says how sure its author is of each step, not only what it claims. A case is typically produced by the developer, reviewed by an internal or external party, and then shared with whoever decides (Buhl et al., 2024). The arguments form a tree: the top claim is split into subclaims, each subclaim is split again or supported directly by evidence, and the assumptions about context are written next to the step that needs them. Two graphical notations draw the tree. Joshua Clymer, Nicholas Gabrieli, David Krueger, and Thomas Larsen, in a 2024 preprint, summarize their case "in goal structuring notation (GSN), which is a flow-chart diagram that illustrates how claims are connected," following UK standards (Clymer et al., 2024), and the control sketch of Section 52.5 uses the Claims, Arguments, Evidence notation (Korbak et al., 2025a). In other industries, Hilton and colleagues write, cases are usually meant as a "living document," kept up to date throughout the system's operational life (Hilton et al., 2025).

The form has a history that explains its claims. Buhl and colleagues report that safety cases emerged in sectors such as energy, petrochemicals, and transportation, where assurance had meant following product design rules, and that accidents in the 1960s to 1980s led to doubt that rules made developers assess their systems comprehensively (Buhl et al., 2024). A case was the alternative: an argument about the outcome, written by the party that knows the system best. That is also its weakness, and the field says so.

The sharpest critic writes from inside the field. Nancy Leveson, in her 2012 book Engineering a Safer World, describes safety programs whose main focus is "creating a safety case that proves the completed design is safe, often by showing that a particular process was followed during development" (Leveson, 2012). Even where the argument goes beyond process, such cases "start from the assumption that the system is safe and then focus on showing the conclusion is true." In her words, "Most of the effort is spent in seeking evidence that shows the system is safe while not looking for evidence that the system is not safe. The basic mindset is wrong, so the conclusions are biased" (Leveson, 2012). She reports that the 2009 inquiry into the loss of a Royal Air Force Nimrod aircraft called such efforts compliance-only exercises, and that it found a culture of paper safety at the expense of real safety (Leveson, 2012; Haddon-Cave, 2009). Clymer and colleagues take the point into their AI proposal: alongside each safety case, a regulator should review a risk case written by a competent third party, "effectively putting advanced AI systems 'on trial'" (Clymer et al., 2024).

Leveson's alternative changes what the argument is about. Her 2004 model, STAMP (Systems-Theoretic Accident Model and Processes), holds that accidents "result from inadequate control or enforcement of safety-related constraints on the development, design, and operation of the system," so that "preventing future accidents requires designing a control structure that will enforce the necessary constraints" (Leveson, 2004). A component chain need not break for an accident to happen: every part can do what it was designed to do while their interaction is unsafe. STPA (System-Theoretic Process Analysis) is the hazard analysis built on that model (Leveson, 2012): it starts from the hazards, lists the control actions that could produce them, and asks how each controller could come to take them. A case built this way argues that the constraints are enforced, not that failures are rare.

Researchers have begun to carry the systems view to AI. Roel Dobbe's 2022 paper, read here in its abstract, draws seven lessons from system safety, among them that AI systems cannot be safeguarded "by technical design choices on the model or algorithm alone," and require an end-to-end hazard analysis that includes the context of use and the institutional environment (Dobbe, 2022). Shalaleh Rismani and colleagues interviewed 30 practitioners in the machine learning industry and, by their abstract, found that STPA and failure mode and effects analysis "can provide an appropriate structure" for assessing social and ethical risks, with "nontrivial challenges" in fitting them to the industry's fast-paced culture (Rismani et al., 2023). Simon Burton and colleagues name, in their abstract, a semantic gap that opens when "normal conditions for a complete specification of intended functionality are not present," beside gaps in moral responsibility and legal liability (Burton et al., 2020). And Section 36.8.3 reads an inability claim built on an evaluation as an argument-based validation chain in Michael Kane's sense, and finds a weak link at each step (Kane, 2013).

For a case about a learned system, the field's alternative implies an order of preference (inference). A claim about behavior ("the model does not do X") needs a rate, and Section 52.1 showed that a rate is what testing cannot supply. A claim about an enforced constraint ("the system cannot do X unless Y also fails") needs a mechanism, which can be inspected, tested at its boundary, and argued about without a rate. That is why inability and control arguments dominate the AI cases of Section 52.5.

The analogy stops in two places. STPA begins from a known control structure and a list of control actions that could be unsafe, and an open-ended agent with tools, code execution, and network access has an action space that resists listing (inference). And the field never had to assure a component that might act against the case made about it. A reactor pump does not read its safety case; a model that may recognize evaluations can, in effect, choose which of a case's assumptions to falsify (inference).

The AI version of the form began with a taxonomy. Clymer and colleagues sorted the arguments a case could make about an AI system into four categories, by analogy with arguing that a robber will not steal a diamond (Clymer et al., 2024). Table 52.1 sets each against the evidence it needs and the assumption the authors themselves flag as hard to justify.

The authors expect developers to rely on the categories in that order as capability grows. They write that the two templates they give for trustworthiness, an argument that the system is not faking alignment and one that elicits what it internally represents, "will likely require fundamental research advances" (Clymer et al., 2024). Other groups cut the space differently. Google DeepMind's technical plan, a 2025 preprint by Rohin Shah and colleagues at the developer, names inability, supervision (its word for control), incentives, and understanding (Shah et al., 2025), and the scheming case of Section 34.8 splits inability into an inability to scheme and an inability to cause harm by scheming (Balesni et al., 2024b). In every version, the first two categories are claims about what the model cannot do, and the evidence for them is an evaluation.

52.3 Defense in depth when layers fail together #

A developer that cannot show any one safeguard is good enough stacks several: training meant to make the model unwilling, classifiers on inputs and outputs, monitors that read what the model does, human review of what the monitors flag, and access controls that limit what any action can reach. A case built on the stack usually multiplies, at least implicitly: if each layer misses one harmful action in twenty, three layers miss one in eight thousand. This section asks when that multiplication is right, what a case can claim when it is not, and what changes when the layers guard against a model rather than against nature.

52.3.1 Layers and the holes in them #

The picture behind the stack comes from the psychologist James Reason. In a 2000 paper in the BMJ, he described the defenses of hazardous technologies as "more like slices of Swiss cheese, having many holes," and added that, unlike holes in cheese, these "are continually opening, shutting, and shifting their location" (Reason, 2000). A bad outcome usually requires that "the holes in many layers momentarily line up." The holes have two sources. Active failures are the unsafe acts of people in direct contact with the system. Latent conditions are "resident pathogens" that arise from decisions by "designers, builders, procedure writers, and top level management," and they can "create longlasting holes or weaknesses in the defences" (Reason, 2000). His books of 1990 and 1997, Human Error and Managing the Risks of Organizational Accidents, are cited here by title (Reason, 1990; Reason, 1997).

Read as arithmetic, the picture invites the independence that the account itself denies (inference). One latent condition, such as a management decision to cut review time, can widen the holes in several layers at once, which is exactly the kind of shared cause that makes multiplication wrong. The model has critics of its own. A 2020 historical review by Justin Larouzee and Jean-Christophe Le Coze reports, in its abstract, criticism of its "simplistic vision of accidents and its degree of generality," answers with "a 'critique of the criticism'," and concludes that it "remains a relevant model" (Larouzee and Le Coze, 2020).

AI risk management has adopted the picture whole. The second key update to the 2025 International AI Safety Report draws a Swiss cheese diagram and writes that "current risk management techniques for AI are all flawed, but layering them can offer much stronger protection against risks" (Bengio et al., 2025c). Google DeepMind's technical plan pairs model-level alignment with system-level security and monitoring because "in practice there will remain some holes. So, we aim for defense in depth" (Shah et al., 2025) (a developer preprint). A 2025 primer from Apollo Research on internal deployment recommends a "Swiss cheese-style" approach in which "individual, complementary measures are stacked" (Stix et al., 2025a) (preprint). None of these statements says how much protection the stack gives, which is the number a case would need.

52.3.2 The arithmetic of a stack #

Section 6.7.4 showed that two monitors that each miss 5 percent of harmful actions miss together 0.25 percent of the time if their errors are independent and up to 5 percent as the errors become positively correlated. Section 50.4.1 derives, from Eckhardt and Lee's model of software written independently by many teams, where such correlation comes from: inputs that are hard for every version a process produces. A case needs two further things: what a whole stack can be claimed to do, and what can be claimed from measurements of its layers one at a time.

Steps 3 to 7 are the book's arithmetic, built on results the reader has. Step 6 is the multi-layer form of what Littlewood, Popov, and Strigini propose for two versions: a regulator, they write, "can use the sum" of the within-class products "as a claim limit," and they expect it to be "more stringent" than the plain product (Littlewood et al., 2001).

A worked case shows the sizes. Three layers each miss 5 percent of harmful inputs on average. One harmful input in ten belongs to a hard class, on which each layer misses 41 percent; on the rest, each misses 1 percent, which together give the 5 percent average. Table 52.2 gives what each way of reasoning claims for the stack.

The multiplied claim is about 330 times smaller than the true rate in the worst row, and the class-by-class estimate closes most of that gap without closing it all. In the worst case the floor of step 5 is 4.1 percent: the hard inputs that every layer misses. A fourth or fifth layer of the same kind would not move it. Figure 52.1 lets the reader vary the number of layers, the average miss rate, and how much of each layer's failure falls on the hard inputs.

The last column of Table 52.2 is where the arithmetic meets Section 52.1. At these rates a test campaign of 24,000 harmful trials would expose a wrong multiplication: it would see well over a hundred escapes. At the rates a catastrophic claim needs, it could not. Three layers that each miss one input in a thousand multiply to one in a billion, and supporting that by test takes about three billion harmful trials. Stacking does not escape the testing problem. It moves the problem from each layer to the joint failure, where it is larger (inference).

52.3.3 From fault-tolerant computing: what a redundant system may claim #

From fault-tolerant computing. Fault-tolerant computing is the branch of computer engineering that builds reliable systems from parts that fail. Section 50.4.1 reports its evidence and theory on redundant software: Knight and Leveson's experiment with independently written versions (Knight and Leveson, 1986), Eckhardt and Lee's model of why such versions fail together (Eckhardt and Lee, 1985), and Littlewood and Miller's extension to forced diversity, read through a 2001 review by Bev Littlewood, Peter Popov, and Lorenzo Strigini (Littlewood et al., 2001). What the field adds here is its verdict on what a reliability claim for a redundant system may assume.

The review states it plainly. Claims that independent components yield arbitrarily reliable systems "are largely mathematical curiosities, since complete independence rarely, if ever, occurs in practice" (Littlewood et al., 2001). Identical copies give no protection against a design fault, "since these will themselves simply be replicated," which is step 5 of the derivation in its extreme form. And the main result of Eckhardt and Lee's model, the review concludes, "is a negative one": two versions that each fail with probability $10^{-3}$ cannot be argued directly into a system that fails with probability $10^{-6}$, "so we must actually measure the reliability of this system" (Littlewood et al., 2001). The claim limit of step 6 is the review's answer to how a regulator can limit a claim without testing the whole system at the rate it claims: a claimed rate below the sum is too optimistic, and a rate at the sum is not thereby shown.

What the field implies for an AI case follows (inference). A case that multiplies its layers' miss rates must either argue that the layers fail independently, which the field holds cannot be assumed, or measure their joint failure on the inputs that matter, which at catastrophic rates cannot be done. The class-by-class estimate of (52.3) is the practical middle: it needs only per-layer measurements on each class of harmful input, and it is honest about difficulty that varies between classes. It limits how low a claim may go; taking it as the stack's rate still needs evidence that the layers fail independently within each class.

The analogy stops at the source of the inputs. Eckhardt and Lee's model assumes that the system runs "on a stationary input series" (Eckhardt and Lee, 1985), so nature supplies the demands. A model that may be pursuing an unintended aim is itself the source of the harmful actions the layers must catch, and an adversary that can probe a stack searches for the inputs on which every layer is weak; the last step of the derivation in Section 50.4.1 shows what that does to a pair. The International AI Safety Report's key update notes research showing that even multiple layers of safeguards "can be vulnerable to sophisticated attacks that are specifically designed to break each layer" (Bengio et al., 2025c).

52.3.4 Layers that share a model, and layers that training couples #

Language models give two new reasons for layers to fail together. The first is shared lineage. Section 6.7.4 reports a 2025 preprint by Elliot Kim and colleagues that found substantial correlation in the errors of over 350 language models, measured on multiple-choice questions as how often two models that both err give the same wrong answer (Kim et al., 2025), and Section 41.6.3 one by Shashwat Goel and colleagues that found models' mistakes on multiple-choice benchmarks growing more similar with capability (Goel et al., 2025a). Neither measured safeguards' misses on attacks. Monitors and classifiers built from related models are therefore likely to be the multiversion case with a large covariance (inference), a covariance still to be measured, and copies of the deployed model used as its own monitor are the identical-copies case, with collusion added (Section 50.4.2).

The second is new to AI: the developer's own training can couple layers that began independent. A model trained against a monitor's verdicts can learn to evade the monitor rather than to stop misbehaving, which Section 40.5 reports for monitors that read the chain of thought. The risk appears in the developers' own sketches. In hypothetical safety case sketches that Roger Grosse published on Anthropic's alignment blog, the imagined developer takes the precaution of not training "against measurements that will be needed for a safety case," since "otherwise, we risk training the model to avoid triggering features that would alert us to misalignment" (Grosse and others., 2024) (a developer post). And the coupling has happened by accident. Anthropic's August 2026 risk report states that, over several reinforcement learning runs, the chain of thought "was unintentionally exposed during reward calculation," in an estimated 0.2 percent (Claude Opus 4.6) to 5.1 percent (Claude Mythos Preview) of the training episodes of the frontier models it lists, figures the developer calls "only a lower bound" and believes close to the true rates (Anthropic, 2026ar) (a developer report). In Reason's terms, a training pipeline that grades what a monitor reads is a latent condition upstream of two layers (inference).

The dependence that matters most follows from the threat model. The layers behind training exist for the case in which training failed. If the first layer failed by producing a model that pursues an unintended aim, that model is also the one most likely to look for the gaps in every later layer, so the later layers' miss rates conditional on the first layer's failure are their miss rates against an adversary, not their average rates (inference). A stack argued as independent layers against random failure is argued against the wrong threat.

The same arithmetic applies to the arguments of a case, not only to its safeguards (inference). In their sketch of a debate-based case, Buhl and colleagues suggest that "by combining independent arguments, it may be possible to write a decent safety case even if each individual argument has flaws" (Buhl et al., 2025). Two arguments that rest on one assumption fail together. Inability evidence and evidence of good behavior both assume that the model behaves under test as it would otherwise, and Section 34.8 notes that of the arguments against scheming, only the control argument does not need that assumption. Two legs of a case add as much as their independence, and their independence has to be argued like any other claim.

52.4 Putting numbers on rare catastrophe #

Should a case end in a number? A probability makes a claim checkable in principle and comparable with a threshold, and every framework that speaks of "acceptable" risk invites one. Risk analysis has tried for fifty years to attach numbers to accidents that have never happened, and its record says what the numbers can carry.

From risk analysis. Section 4.2 reports the tradition's founding definition and its most instructive study. Risk is a set of scenarios, each with a likelihood and a consequence (Kaplan and Garrick, 1981). The Reactor Safety Study of 1975, known as WASH-1400, estimated accident frequencies for reactors that had never had a serious accident by building event trees and fault trees, diagrams that trace paths to an accident through combinations of component failures (Nuclear Regulatory Commission, 1975). The review group chaired by H. W. Lewis judged the method sound and the study's error bands understated by an amount it could not estimate (Lewis et al., 1978), and a later assessment argued for decisions that are "risk-informed rather than risk-based" (Apostolakis, 2004). This section adds what that record means for a safety case.

A quantitative case is a fault tree with numbers at its leaves. Clymer and colleagues suggest that developers could "optionally assign probabilities that each claim in the GSN tree is valid and then aggregate them," turning the argument into a Bayesian network of the kind used in probabilistic risk assessment (Clymer et al., 2024). Each conjunction in such a tree is a multiplication. Their rule for it multiplies by a conditional probability, $P(A \text{ and } B) = P(A \mid B)\,P(B)$, so the network can carry dependence between claims (Clymer et al., 2024); a claim's probability assessed on its own and multiplied in is an independence assumption of the kind Section 52.3 examined (inference). The Lewis review named the same weak point in WASH-1400: among its reasons for distrusting the numbers were an inability to quantify common-cause failures, those in which one event disables several components, and it wrote that the study's treatment of them "may be nonconservative," that is, may understate the risk (Lewis et al., 1978). Clymer and colleagues flag a second limit for AI: a naive model in which each part of the system fails independently with some probability "is not an appropriate model," in part because developers who catch a failed attempt respond to it (Clymer et al., 2024).

The numbers at the leaves can be precise and unsupported at once. Leveson and Turner's 1993 investigation of the Therac-25, a radiation therapy machine involved in six known accidents with massive overdoses between June 1985 and January 1987, describes the manufacturer's 1983 safety analysis (Leveson and Turner, 1993). It was a fault tree, and one of its assumptions was that "programming errors have been reduced by extensive testing on a hardware simulator and under field conditions." Hence "any residual software errors are not included in the analysis." The tree assigned the event "Computer selects wrong energy" a probability of $10^{-11}$, and "Computer selects wrong mode" one of $4 \times 10^{-9}$. "No justification of either number is provided" (Leveson and Turner, 1993). The testing that licensed leaving software out of the analysis is the kind of testing that Section 52.1 showed cannot carry a low rate (inference).

Industries that do set numerical targets set them where data and a record exist. The 2001 review of design diversity reports that flight control software in civil aircraft falls under the requirement that catastrophic failures be "not anticipated to occur over the entire operational life of all airplanes of one type," usually translated as a probability of failure of $10^{-9}$ per hour (Littlewood et al., 2001). It reports that the software protection system of the Sizewell B reactor had a requirement of $10^{-4}$ probability of failure on demand, and that the UK Nuclear Installations Inspectorate accepted $10^{-3}$ after a sensitivity study of the plant's risk assessment showed the larger figure would still give a tolerable risk (Littlewood et al., 2001). For AI, Clymer and colleagues give an acceptable probability of catastrophe "(for example, 0.1%)" over a deployment window only as an illustration (Clymer et al., 2024). The 2025 key update of the International AI Safety Report finds a "lack of clear methods for defining acceptable risk levels," with risk thresholds "often implicit" in the responses the frameworks choose (Bengio et al., 2025c). Section 62.3 takes up thresholds set on capability instead of risk. Where no component record exists, the numbers would come from expert judgment, whose structured elicitation Section 4.5.3 reports (Cooke, 1991).

What the record predicts for the first quantitative AI risk estimates is WASH-1400's fate: the decomposition judged useful, the absolute numbers judged unsupported, and a verdict with stated assumptions and independent review closer to what the evidence can carry (inference).

The first developer risk reports took the second course. In October 2025 Anthropic released a pilot report on misalignment risk from its models deployed as of summer 2025, focused mainly on Claude Opus 4 (Bowman et al., 2025) (a developer report). Its post states the conclusion that there is "a very low, but not completely negligible, risk of misaligned autonomous actions that contribute significantly to later catastrophic outcomes." The authors write that they are "moderately confident" that the model has no "consistent, coherent dangerous goals" and lacks the capabilities needed to execute complex sabotage reliably while avoiding detection. They titled it a risk report rather than a safety case "to emphasize that we are not currently arguing that we have achieved a specific pre-determined level of risk" (Bowman et al., 2025). An internal team and METR, an outside evaluation nonprofit, reviewed it; both "had access to additional evidence beyond what is presented in the report," and both endorsed the assessment while raising doubts about argument and evidence in places (Bowman et al., 2025).

The developer's second report, of August 2026, states its verdicts in words and raised its rating of misalignment risk in high-stakes settings from "very low" to "Low" after incident disclosures; its public version is redacted in marked places, the full version is shared inside the company, and a trust that selects part of the board may request outside review (Section I.2.15) (Anthropic, 2026ar; Anthropic, 2023f) (a developer report). For the third relationship these details are the point. The evidence the reviewers saw is larger than the evidence the public can see, and the public's reason to accept the verdict is the reviewers' access and independence, which Section 5.6 reads as a disclosure in its own right (inference).

One alternative replaces a judged probability with a computed bound. Section 28.6 teaches the guaranteed safe AI proposal, which asks for a proof or a sound probability bound that a system meets a safety specification relative to a world model (Dalrymple et al., 2024), and a related bound on the harm of an action computed from a posterior over world models (Bengio et al., 2024a). A bound of this kind is as good as its world model, which is the completeness problem of the reactor study in another place (inference).

The analogy stops where Section 4.2.5 says it does: the reactors had component failure data, a plant whose parts could be listed, and no capacity to act strategically on their assessors, and an AI system has none of the three. The 2026 report shows a fourth difference. Its verdict moved because of incidents outside the structure of its arguments, which a reactor assessment, built on a plant that does not change between reports, seldom had to absorb (inference). That points to the practices of Section 52.6, which keep a case current.

52.5 The safety cases written for AI #

What do the AI safety cases written so far argue, and what do they assume? Since 2024, authors at government institutes, research nonprofits, and developers have published sketches: partial cases for hypothetical models, meant to show what a full case would need. All are preprints or developer posts, several share authors, most often at the UK AI Security Institute, and none has yet been tested against a failure or a near miss. Table 52.3 lists them with the claim each argues and the assumption its own authors leave open.

The control sketch shows a control argument in full. Tomek Korbak, Joshua Clymer, Benjamin Hilton, Buck Shlegeris, and Geoffrey Irving, at the UK AI Safety Institute and Redwood Research, sketch a case that a hypothetical agent, deployed inside an AI company for machine learning engineering, will not exfiltrate sensitive information (Korbak et al., 2025a). The core evidence is a control evaluation of the kind Chapter 50 teaches. A red team, given full knowledge of the protocol, trains substitute models that try to exfiltrate dummy data in a proxy deployment of 2,000 tasks, and simulation extrapolates the results to a deployment of 100,000 tasks. The case then rests on three claims: that "the red team adequately elicits model capabilities to exfiltrate data," that "control measures remain at least as effective in deployment," and that "developers conservatively extrapolate model performance" (Korbak et al., 2025a).

The authors name what the sketch assumes. It treats the models as stateless, so that results from separate tasks can be combined, and it assumes a constant probability of attack, which they call "particularly tenuous" once copies of the model might coordinate implicitly. They conclude that "developers should be cautious about assuming that preventing explicit communication is sufficient to ensure independent behaviour across model instances" (Korbak et al., 2025a), which is Section 52.3's problem reappearing inside a single case. And the case "crucially rests on the assumption that the human red team (uplifted by trusted models) is more capable than the untrusted models at strategy generation," so it "does not scale to arbitrarily powerful superintelligence" (Korbak et al., 2025a). A companion preprint by authors at the UK AI Security Institute, Apollo Research, and Redwood Research proposes scaling the red team's affordances, what it may use and know, to the capabilities of the model under control across five control levels. It concludes that a compelling control safety case for superintelligent agents "will require research breakthroughs," and it calls its own recommendations "unlikely to produce a high-assurance safety case" (Korbak et al., 2025c).

The untrusted-monitoring sketch by Nelson Gardner-Challis and colleagues goes deeper into one protocol, the case in which the model under control is monitored by another copy of itself (Gardner-Challis et al., 2026). Its top claim bounds the probability of a catastrophic attack, and the argument bounds four quantities. They are how often an honest monitor misses a backdoor, how well the monitor can tell its own model's outputs from planted test cases, how many planted cases it may miss before it is caught, and whether the statistical model that combines them is conservative. It relaxes an assumption of earlier work, that colluding copies share a secret signal agreed in advance (Gardner-Challis et al., 2026). Its taxonomy of collusion and its measured win rates belong to Section 50.4.2. The authors call the sketch "conditional," and list attack selection, honeypot slack, eliciting the best performance for every collusion strategy, and jailbreaking as problems to solve before safety can be strongly argued (Gardner-Challis et al., 2026).

The debate sketch by Marie Davidsen Buhl, Jacob Pfau, Benjamin Hilton, and Geoffrey Irving argues for alignment rather than control (Buhl et al., 2025). An agent that does research inside a developer is trained by debate (Section 41.3) so that, at the equilibrium of the debate game, it answers correctly in all but a small fraction of cases, and online training keeps it there during deployment. The case needs four claims: the agent has become good at the game, good play implies mostly honest answers, honesty will not degrade in deployment, and the deployment tolerates some errors (Buhl et al., 2025). It also needs a solution to exploration hacking, in which a model avoids the strategies that training would reward, which the authors treat as open, and a debate protocol that avoids obfuscated arguments, which they believe a forthcoming protocol of theirs does under an assumption of stability. They write that they "do not think it is currently possible to make a strong alignment argument based on this sketch" (Buhl et al., 2025). Chapter 55 takes up the research setting.

The developer sketches agree with the outside ones about what is missing. Grosse's post, which distills ideas from many others, presents three hypothetical components of a case for a model that might sandbag evaluations, undermine monitoring, or sabotage the organization using it. One rests on interpretability, one on control, and one on an analysis of the model's incentives (Grosse and others., 2024). The post presents them as hypothetical examples, "not official policies or predictions," and judges that "none of the sketches presented here fully succeeds in addressing the sabotage risk" (Grosse and others., 2024). OpenAI's 2026 guidelines for cases that cover a frontier reinforcement learning training run are reported in Section 24.7.3 (OpenAI, 2026ab).

Read together, the sketches put their weight on two kinds of claim (inference). The first is that an evaluation bounds the real model: the red team elicits what the model could do (control), fine-tuning elicits the capability (inability), the planted test cases look like the model's own outputs (untrusted monitoring), or training explores well enough that nothing is hidden (debate). Each is a version of the problem Section 36.2.3 sets out, a score that depends on whether the model is being measured, and each weakens if the model behaves differently under test (Chapter 37). The second is independence: of model instances in the control sketch, of the monitor from the monitored copy, and of the legs that a case hopes will cover each other's flaws. The other kinds of evidence enter with their own limits. A case that leans on reading a model's chain of thought needs evidence that harmful reasoning cannot be done without writing it down. Section 40.1 reports the argument that monitorability is a fragile opportunity (Korbak et al., 2025b), and Section 40.2.2 a study whose own authors caution against reading its two-hop results as such evidence (Balesni et al., 2024a). A case that leans on interpretability makes an absence claim, which Section 47.7 finds the field least able to support, and the forensic standard of Section 47.7.2 asks that its procedures be open to review by others.

The outside readings of these documents treat them as disclosures, as Section 5.6 asks. The International AI Safety Report's 2025 key update reports an OECD analysis of 20 organizations in the Hiroshima AI Process reporting framework. Most combine quantitative metrics with expert judgment, it found, "but few disclose full methodologies or systematically validate whether documented controls correspond to real-world outcomes" (Bengio et al., 2025c). An advocacy organization's expert panel graded no company above D for what it calls existential safety planning in summer 2025 (Future of Life Institute, 2025), and Section 62.7 reports the outside reviews of the frameworks. The safety benchmarks that evaluations offer as evidence can track general capability instead, the finding Section 5.2.2 reports (Ren et al., 2024b).

Whether safety cases make systems safer or mainly produce documents is the debate the form brings with it, and three positions can be stated. On the first, writing an explicit argument exposes gaps that informal judgment hides. Buhl and colleagues, Hilton and colleagues, and the template's authors hold this view (Buhl et al., 2024; Hilton et al., 2025; Goemans et al., 2024). The evidence offered is the practice of other industries, where, Buhl and colleagues report, practitioners mostly consider cases effective, a finding they pair with the concession that there is "little empirical evidence for the efficacy of safety cases" (Buhl et al., 2024). On the second, the form invites confirmation: Leveson's critique of cases that seek evidence of safety rather than of danger (Section 52.2), and the argument in organizational sociology that organizations adopt formal structures for legitimacy and decouple them from practice, which Section 53.7 takes up. On the third, a case is a form of what Cary Coglianese and David Lazer call management-based regulation, which directs firms to plan toward public goals (Coglianese and Lazer, 2003). In their case studies it "can be an effective strategy when regulated entities are heterogeneous and regulatory outputs are relatively difficult to monitor," at the price of "ambiguity over what constitutes 'good management'" (Coglianese and Lazer, 2003). The mapping of AI safety cases onto that category is the book's (inference). Downer's account of aviation, in which a regulator who cannot test relies on the regulated party's engineers, fits all three (Downer, 2010). The evidence on AI cases does not decide among them, because no AI case has yet been followed by a failure or a near miss that could test it.

52.6 Staged release and the living case #

A case written before release is a prediction, and the system it describes then meets the world. Two older practices deal with that gap: medicine's staged evidence before a drug is sold, and the operational reporting by which aviation learns from the errors of ordinary flights. This section asks what each would mean for a case about an AI system that keeps changing.

From medicine. The US drug regime is the oldest working example of a regulator that demands an argument from evidence before release. The Drug Amendments of 1962 require the regulator to refuse approval if there is "a lack of substantial evidence that the drug will have the effect it purports or is represented to have under the conditions of use prescribed, recommended, or suggested in the proposed labeling" (United States Congress, 1962). The statute defines substantial evidence as "adequate and well-controlled investigations, including clinical investigations, by experts qualified by scientific training and experience." Before that, investigational use is exempted on conditions that include reports to the regulator and the consent of the people given the drug or their representatives, except where the investigators deem it not feasible or contrary to those people's best interests (United States Congress, 1962). The approval is not final. The regulator must withdraw it if new evidence, "not available to the Secretary until after such application was approved," shows that the drug "is not shown to be safe for use under the conditions of use upon the basis of which the application was approved" (United States Congress, 1962). It may suspend the approval at once in the face of "an imminent hazard to the public health" (United States Congress, 1962). In the terms of Section 52.2, the law fixes a top claim, a standard of evidence, an operating context (the labeled conditions of use), and a rule for when a case already accepted stops holding. How the regulator came to hold this power is the subject of Section 64.5; Daniel Carpenter's 2014 history of it, Reputation and Power (Carpenter, 2014), is readable here only as a catalog record.

Approval has grown faster and rests on less evidence. A 2020 review by Jonathan Darrow, Jerry Avorn, and Aaron Kesselheim in JAMA, read here in its abstract, traces approval from 1983 to 2018 (Darrow et al., 2020). By 2018, 81 percent of new drugs (48 of 59) benefited from at least one expedited program, and the share of new approvals supported by at least two pivotal trials fell from 80.6 percent in 1995 to 1997 to 52.8 percent in 2015 to 2017. Fees paid by industry accounted for about 80 percent of the salaries of the staff who review new drugs in 2018. The authors conclude that the regulator "has increasingly accepted less data and more surrogate measures, and has shortened its review times" (Darrow et al., 2020). Expedited programs trade evidence before approval for obligations after it, which Section 63.6 examines.

Software that learns breaks the regime's premise that the approved thing stays the same. Boris Babic, Sara Gerke, Theodoros Evgeniou, and I. Glenn Cohen, in a 2019 article in Science, call this the update problem: should a regulator authorize only the version of a medical algorithm it reviewed, or one that adapts after approval? For drugs and ordinary devices, they write, the problem "typically does not arise," while machine learning "is the first such technology with the capability to continuously evolve" (Babic et al., 2019). Locking the algorithm forgoes improvements that would need another review, and letting it update freely exposes it to bad data and adversarial inputs. They propose that regulators "adopt an ongoing risk management process" for updates, with continuous monitoring of how the systems work in changing environments (Babic et al., 2019).

What medicine implies for AI cases is a schedule and an expiry (inference). Staged release maps onto internal deployment, limited access, and general release, and the first stage is the least governed: an Apollo Research primer found that governance of the internal deployment of the most advanced systems "appears absent" (Stix et al., 2025a) (preprint). A case written for one model version expires at the next update unless it says which changes it survives. As of October 2026, the developers' frameworks have begun to write such rules: two of them say when a change in deployment conditions makes a report no longer applicable or when monitoring after release may update a case (Section I.2.13) (OpenAI, 2025e; DeepMind, 2026). That is Babic and colleagues' update problem with the developer as both applicant and regulator.

From safety engineering. A case stays true only if operations report against its assumptions, and aviation built the practice for that. Robert Helmreich's 2000 account in the BMJ lists the data sources aviation uses to manage error: confidential surveys of crews; non-punitive incident reporting systems that strip identifying information and let data be shared; and line operations safety audits, in which expert observers ride on normal flights and record threats and errors (Helmreich, 2000). The audits, over more than 3,500 flights, found "an average of two threats and two errors observed per flight," and Helmreich notes the limit of the reports: "Because incident reports are voluntary, however, they don't provide data on base rates of risk and error" (Helmreich, 2000). For AI the gap is reported, not closed. The 2025 key update of the International AI Safety Report cites studies finding that even leading firms "often lack systems for connecting actual incidents to their safety documentation" (Bengio et al., 2025c). Anthropic's 2026 risk report includes a section on what it calls safety process failures since its previous report, a representative sample with one case redacted (Anthropic, 2026ar) (a developer report), which is the kind of reporting against a case that aviation's audits institutionalized. Chapter 54 builds the incident practice.

The analogy stops in four places (inference). Trials randomize and register their protocol before the data arrive, and AI deployments do neither. A drug's harms fall mostly on the patients who take it, with endpoints that can be measured, while an AI system's harms can be diffuse and fall on third parties. A drug is approved for labeled conditions of use, and a general-purpose model has no indication in that sense, so the operating context its case must state is close to everything. And no drug sponsor approves its own drug, while every risk report in this chapter is a decision document of the developer that wrote it, reviewed by parties that developer chose, and the sketches by outside authors are hypothetical.

52.7 What safety cases mean for delegation #

The book's thesis is that alignment is a delegation problem under an asymmetry of competence and visibility, and a safety case is the report an agent makes to its principal in the third relationship: the developer, acting with the public's tolerance, tells the public and its regulators why a system it built and understands best is safe enough. The older structures fit closely (inference). Certification by delegated trust, in which a regulator who cannot test relies on the manufacturer's engineers (Downer, 2010), is the developer evaluating its own model under review. Management-based regulation, in which firms plan toward public goals because their outputs are hard to monitor (Coglianese and Lazer, 2003), is a regime of safety frameworks and cases. And the economics of disclosure in Section 5.6 predicts what such reports will contain when outsiders cannot check them. The first developer risk reports bear this out: their verdicts rest partly on evidence that reviewers saw and the public did not (Section 52.4).

What does not carry over is the component. Every older form of the argument assumed that the thing assured does not act on the assurance unless its makers built it to, as with the emissions defeat device: reactor pumps do not read their fault trees, and Eckhardt and Lee's inputs arrive "on a stationary input series" (Eckhardt and Lee, 1985). A model that may recognize evaluations, with a recognition no one wrote, can falsify, on purpose, the assumptions a case depends on, which is the evaluation-awareness item on the list of results with no older counterpart in origin and incentives (Section 69.4) (inference). Two of the thesis's other novelties cut both ways here. A principal that manufactures the agent's dispositions and can inspect its internals has kinds of evidence no older case had: red teams that train substitute attackers, and interpretability. The same power lets the developer couple layers that were meant to be independent, by training against what a monitor reads (Section 52.3.4).

The claims in this chapter concern different relationships. Inability and control arguments are claims about the deployed system; cases for a training run concern the training process and the signal it optimizes (Section 24.7.3); and every case as a document, with its reviewers and its redactions, concerns the developer and the public. The developer reports in this chapter are self-reports in that third sense, reviewed by parties the developer selected.

52.8 Settled, contested, missing #

52.9 Exercises #

Further reading #

• Clymer et al. (2024): the four kinds of argument an AI safety case can make, and how they could be combined.
• Leveson (2012): accidents as failures of control, and the case against safety work that sets out to prove a design safe.
• Reason (2000): the system view of error and the Swiss cheese model of layered defenses.
• Knight and Leveson (1986): the experiment that found independently written versions failing together far more often than independence predicts.
• Korbak et al. (2025c): how a control evaluation's red team must scale with the model, and where control safety cases run out.