Draft complete but not yet reviewed; expect revisions

What Earlier Technologies Teach

Proposals for governing AI arrive with precedents attached. They name the molecular biologists who paused their own experiments in the 1970s, the international authority for atomic energy proposed in 1946, the drug regulator that approves before sale, the aircraft certifier, the bank supervisor who validates models, and the internet, which no treaty governs. Each precedent is an argument by analogy, and an analogy carries a conclusion only as far as the mechanism behind it is shared. Chapter 63 asked what the instruments of the state can do for frontier AI now. This chapter asks what earlier technologies show about which arrangements held, under what conditions, and why.

The chapter opens with the method, which is what lets it say where each analogy stops, and with the dilemma every case faced in some form: a technology is easiest to steer before its effects are known. It then takes six cases in turn. Recombinant DNA shows scientists restraining themselves, and the critique of how they did it. Nuclear weapons and power show international control proposed, failed, and partly rebuilt around what could be verified. Aviation and pharmaceuticals show two ways of certifying what cannot be fully tested, and finance shows what happens when verifiers are paid by the verified. The internet shows a global technology governed largely by private operators and voluntary norms. A section gathers the record of pausing and restraint across the cases, and the last section sets the six side by side and says what they mean for the delegation of a technology to the firms that build it. Most claims here concern the third of the book's relationships, the developer and the public, and its extension to states and the publics they answer to.

64.1 Learning from cases without overreaching #

A historical case can be cited for two different things: that something happened, and that what happened there would happen here. The first needs a record. The second needs an argument that the case and the present share whatever produced the outcome, and a precedent named in a policy debate usually supplies only the first (inference). This section states the test the chapter applies, the problem every case met, and the features on which the cases are compared.

64.1.1 The test an analogy must pass #

Section 5.8 gives the test for carrying evidence from one system to another. A result in one system licenses a claim about another only if the two share the causal structure that produced the result. Knowing that they share it seems to require studying the target, which in this book's reading makes the step circular (Section 5.8.1). Judea Pearl and Elias Bareinboim made the step formal for causal effects. From "selection diagrams" that record where two populations differ and agree, a procedure decides, "prior to observing any data," whether an effect found in one can be inferred in the other (Pearl and Bareinboim, 2014). A historical case is a study population of one. Its result is a governance outcome, such as a pause that held or an agreement that failed, and its structure is the set of conditions that produced the outcome. An analogy carries the outcome only along the conditions that the case and frontier AI share (inference). The classic philosophical treatment of analogy in science is Mary Hesse's 1963 Models and Analogies in Science (Hesse, 1963). It was readable for this book only as a catalog record, so the test is stated in the terms of Section 5.8 and not in hers.

History adds two difficulties of its own. The first is that a case happened once. Waqar Zaidi and Allan Dafoe studied the first proposals for international control of atomic energy. They call their period "a single observation (n=1)" and conclude that the outcome should not be the main evidence; the informative moments within the episode should be (Zaidi and Dafoe, 2021). The second is selection. The cases a debate remembers are the ones that became famous, and fame is not a random sample of governance outcomes (inference). This chapter therefore reads each case for its mechanism, what held it together or pulled it apart, more than for its ending.

A third limit is particular to this book. Several standard histories of these cases could be read here only as catalog records. They include Sheldon Krimsky's social history of the recombinant DNA controversy (Krimsky, 1982), Daniel Carpenter's Reputation and Power (Carpenter, 2014), Joseph Rees's study of nuclear safety after Three Mile Island (Rees, 1994), and Scott Sagan's The Limits of Safety (Sagan, 1993). The chapter cites them for their existence and carries each case through primary documents and readable accounts instead, and it says where a gap is left.

64.1.2 The dilemma every case met #

Section 17.6 states David Collingridge's dilemma of control, named for his 1980 book (Collingridge, 1980), in the words of Laura Weidinger and colleagues. Early evaluation has more potential to influence the direction of a technology, and later evaluation provides more accurate information (Weidinger et al., 2023a). Collingridge's own proposals for escaping the dilemma are not reported here, because his book was readable only as a catalog record. A 2018 review by Audley Genus and Andy Stirling is read here in its abstract. It finds "a mismatch between the prevalence of citations to Collingridge's work on the dilemma of control" in the literature on responsible innovation "and the depth of engagement with his arguments" (Genus and Stirling, 2018). The dilemma is better known than its author's answers to it, and this chapter is no exception.

Every case below met the dilemma. The recombinant DNA researchers acted before any hazard was observed; the nuclear powers bargained over a weapon whose effects they had seen; the drug and aviation regimes described below wrote major rules after harm (inference). Where each case fell on that line is one of the conditions the chapter tracks.

64.1.3 Six features to score #

The conditions the chapter tracks are six, chosen by this book from the cases and not taken from any one source (inference):

• Concentration of inputs: whether doing the dangerous thing needed something scarce and traceable, such as fissile material or a funded laboratory.
• Verifiability: whether an outsider could tell, without the actor's cooperation, whether it complied.
• Military salience: whether states saw the technology as decisive in war.
• Commercial stakes at the time of governance: whether firms already earned money from the activity being restrained.
• Pace: how fast the technology changed relative to the body governing it.
• Expertise: who held the knowledge needed to judge risk, the governed or the governors.

Table 64.2 in Section 64.9 scores the six cases and frontier AI on these features after the cases have been told. The table is the chapter's main device for saying where an analogy stops: an analogy transfers only along the columns where AI and the case agree.

64.2 Recombinant DNA and self-governance by scientists #

The precedent cited most often in AI governance is the one in which scientists stopped their own work. The work was recombinant DNA research, which splices DNA from different organisms into one molecule. The question is what made that pause hold, and whether the conditions that held it exist for frontier AI.

64.2.1 The pause and the conference #

From the history of technology. The most detailed readable account of the episode is a 2015 case study by Katja Grace, written for the Machine Intelligence Research Institute to ask what Asilomar teaches about efforts to prepare for AI (a technical report, not peer reviewed). In her summary of events, Paul Berg and colleagues at Stanford constructed in 1972 a recombinant DNA molecule joining DNA from Simian Virus 40, which causes cancer, with DNA from the gut bacterium E. coli. Another scientist, Bob Pollack, raised concerns, and after discussing the hazards with colleagues Berg decided not to take the next step of inserting the molecule into E. coli (Grace, 2015). In 1973 attendees of a Gordon Research Conference voted to ask the National Academy of Sciences to examine the risks, and the Academy formed a committee chaired by Berg. In July 1974 that committee called for a voluntary moratorium on a class of experiments (Grace, 2015). Its letter, "Potential Biohazards of Recombinant DNA Molecules" (Berg et al., 1974), is readable here only as a catalog record. The Asilomar statement confirms that the committee's letter of July 1974 called for "the pause in certain aspects of research in this area" (Berg et al., 1975).

The pause had no enforcement behind it. Grace reports that, despite some disagreement, "the moratorium was universally adhered to," and that it was meant in part to give time for a conference to assess the risks (Grace, 2015). That conference met at Asilomar in February 1975, with scientists from around the world, lawyers, the press, and government officials (Grace, 2015). Section 59.5 reports the two principles of its summary statement: containment as an essential consideration in the design of experiments, and containment matched as closely as possible to the estimated risk.

The statement is worth reading for what surrounds those principles. It grounds its caution in ignorance rather than in evidence of harm: the evaluation of potential biohazards "has proved to be extremely difficult. It is this ignorance that has compelled us to conclude that it would be wise to exercise considerable caution" (Berg et al., 1975). It sets four levels of containment, for minimal, low, moderate, and high risk, and assigns classes of experiments to them. It calls the assignments "interim assignments which will need to be revised upward or downward in the light of future experience" (Berg et al., 1975). It places some experiments beyond any available containment, among them the cloning of DNA from highly pathogenic organisms and of DNA containing toxin genes, which "should not be undertaken at this time" (Berg et al., 1975). It leaves judgment with the people doing the work. In its words, "each investigator bears a responsibility for determining whether, in his particular case, special circumstances warrant a higher level of containment than is suggested here." And it treats itself as temporary. It urges scientists to use its proposals as a guide until national bodies set codes of practice, and it calls for "a continuing reassessment of the problems in the light of new scientific knowledge" (Berg et al., 1975).

What followed made the guidelines binding on most researchers without a statute. In Grace's account the guidelines were adopted by the National Institutes of Health as a condition of funding and followed voluntarily by others. They "have become less restrictive as new information has emerged," and by 2015 they placed few constraints on recombinant DNA research (Grace, 2015). Firms were not bound by them, yet in her account the commercial sector followed them, and more rigorously than most academic institutions (Grace, 2015).

64.2.2 What held it together, and what it left out #

Grace lists the factors that participants and observers credited with Asilomar's success, and she warns that her judgments are subjective and rest on scattered information (Grace, 2015). Funding bodies made the guidelines binding on almost all biological researchers. A large minority of participants from the media kept the meeting from looking secret. A talk on legal liability by the law professor Roger Dworkin, on the last night, "scared them," and encouraged the participants to produce recommendations the next day. The threat of legislative control motivated self-regulation. Commercial involvement was low at first, and the field was small (Grace, 2015). Berg, in her account, believes that what later ended interest in banning the research, after Congress showed interest in doing so, was the prospect of commercialization. Martin Rees, whom she also cites, suspects that a voluntary consensus like Asilomar's would be harder to reach today, because competition is more intense and commercial pressures stronger (Grace, 2015).

The same study lists the process's weak points. Scientists appear to have underestimated the danger of their own experiments while agreeing that others' might be risky; Pollack, who first raised the alarm, describes a "shade" that comes over one's eyes when the problems affect one's own work (Grace, 2015). Small groups controlled the decisions: the organizing committee had authority to write the recommendations with or without the participants' support, and the conference drew little input from experts in other fields or from the public (Grace, 2015). Avoiding regulation by outsiders was a strong motive, and in Grace's reading it could have encouraged "giving the appearance of control over the situation beyond what was warranted by the facts" (Grace, 2015). She quotes Krimsky's complaint that the hazards assessed were those of "experiments that scientists were particularly interested in doing," which "takes no cognizance of commercial ventures or other possible non-academic uses or misuses of the technology" (Grace, 2015).

Whether Asilomar succeeded is a question with two answers. On Grace's reading it did not avert the dangers it set out to avert, because those dangers "appear to have been close to non-existent," and whether the scientists should have known better is disputed. The disabled bacterial hosts it encouraged were less crippled than believed at the time (Grace, 2015). Its intermediate actions worked: the moratorium held, the conference reached recommendations, and the guidelines were followed (Grace, 2015). A precedent whose hazard turned out to be small shows that self-restraint can hold under uncertainty; it cannot show that the same arrangement would have contained a hazard that was real (inference).

64.2.3 The parallel developers draw #

The first laboratory safety framework, as Section 62.1.1 reports, modeled its AI Safety Levels loosely on the biosafety levels for handling dangerous biological materials (Anthropic, 2023a) (a developer document). The structure the two share is Asilomar's: a scale of risk met by a scale of containment, with assignments revised as evidence arrives (inference). Some of the conditions Grace lists are shared too. The framework was written by the party whose work it restrains, before any catastrophe was observed, in a field with a small number of leading organizations (inference). Others are not. No funding body stands behind a developer's framework as the National Institutes of Health stood behind the guidelines, and the work restrained is the developer's commercial product, not its research alone (inference).

64.2.4 The critique from science and technology studies #

From science and technology studies. The field that studies how science and society shape each other reads the same episode as a warning. Writing in 2015, in an essay published as leading biologists proposed restraint on editing the human germline (the DNA in sperm, eggs, and embryos that children inherit), Sheila Jasanoff, Krishanu Saha, and J. Benjamin Hurlbut quote Berg's 2008 description of Asilomar as a meeting that let geneticists "push research to its limits without endangering public health," in order to dispute it (Jasanoff et al., 2015). In their account, Asilomar framed risk as the accidental escape of dangerous organisms from the laboratory, and the scientists adopted containment to prevent it: "Public health would not be risked and research would continue" (Jasanoff et al., 2015). What the framing left out returned later. Asilomar did not consider the deliberate release of modified organisms outside the laboratory. American voices dominated, with none from the developing world. Biosecurity and ethics were excluded from the agenda. The long controversy over genetically modified crops was, in their words, "a reopening by global citizens of all the dimensions of genetic engineering that Asilomar had excluded" (Jasanoff et al., 2015). They call the remembered version "Asilomar-in-memory," "a mythic construct" (Jasanoff et al., 2015).

The critique and the defense agree on more than they seem to. Grace's list of weak points (small groups, narrow framing, the wish to avoid regulation) is close to the critique's list, and the critique does not deny that the moratorium held. They differ on what counts as success: containment of a technical hazard, or a settlement that the people affected had a part in making (inference). For a developer's framework the critique has a direct reading. A framework written by the developer defines the risks it covers, and the ones it leaves out are decided by the same act (inference); Section 17.6 sets out the field's remedy, deliberation that recurs as evidence from deployment arrives.

64.2.5 Later tests in biology #

Biology has since tested the model again. In March 2015, by the account of Jasanoff and colleagues, a group including Baltimore and Berg proposed a worldwide moratorium on altering the genome in ways that could be passed on to future generations (Jasanoff et al., 2015). Their paper in Science is readable here only through its abstract's one sentence, which called "a framework for open discourse" on the use of the technique to manipulate the human genome "urgently needed" (Baltimore et al., 2015). In 2019 Eric Lander, Françoise Baylis, and sixteen other scientists and ethicists from seven countries explained why that restraint had not held. The first international summit on human gene editing had concluded in December 2015 that clinical use would be irresponsible until safety was resolved and there was "broad societal consensus." Yet "in China, biophysicist He Jiankui reportedly edited embryos to create at least two babies." Other scientists "apparently aware of this work did not take adequate measures to stop it," and "no mechanism was created" to ensure international dialogue (Lander et al., 2019).

Their remedy moved restraint from scientists to states. They called for a global moratorium in which nations "voluntarily commit to not approve any use of clinical germline editing unless certain conditions are met" (Lander et al., 2019). The design had an initial fixed period, five years perhaps, then public notice of perhaps two years and international consultation before any nation permitted an application (Lander et al., 2019). They rejected a treaty as too rigid, noting that a UNESCO effort to outlaw human cloning by binding convention had foundered. They also rejected self-governance by the profession: these decisions "must not be taken by individual actors," nor by "the scientific or medical community acting as a whole" (Lander et al., 2019). The oversight of dual-use research in the life sciences, from the National Research Council's 2004 reliance on scientists' self-governance to the federal policies of 2024 and 2025, is reported in Section I.1.9. Read beside the germline case, it shows the same drift from scientists' judgment toward government rules, which then change with administrations (inference).

64.2.6 Where it stops #

Grace's own comparison sets the terms. Asilomar resembled AI safety efforts in the novelty of the threat, the difficulty of prediction, and the limited feedback. It differed in that "many top scientists were concerned" and in that it "did not involve any action years ahead of any perceived risk" (Grace, 2015). Three further conditions held for recombinant DNA and fail for frontier AI (inference). The researchers were a small community working largely in publicly funded laboratories, so once guidelines replaced the unenforced moratorium, one funder's condition reached nearly all of them; frontier AI is developed by firms that fund themselves. The hazard was physical, and laboratory containment was judged able to hold it, though how well it would have held a real hazard is unclear (Grace, 2015); the hazards that motivate frontier governance arise in use, after release. And the moratorium came before commercial stakes were large, the condition Berg himself names as the one that later changed. Frontier AI is being governed after its commercial stakes became large. The precedent shows that a technical community can restrain itself under uncertainty, and the record ties that restraint, and the funded guidelines that followed it, to conditions frontier AI does not share.

64.3 Nuclear weapons and nuclear power #

The nuclear case is the precedent for international control of a dangerous technology, and its civilian side is the precedent for quantitative risk estimates and for an industry policing itself. The question is what the first attempt at control shows about why it failed, and what the agreements that followed were built around.

64.3.1 International control, proposed and failed #

From arms control. Arms control, the negotiated limitation of weapons between states that cannot compel each other, began for nuclear weapons with a proposal to place them under an international authority. The State Department's historian gives the sequence. At a conference in Moscow in December 1945 the United States, Britain, and the Soviet Union created a United Nations commission to advise on destroying atomic weapons and using atomic energy for peaceful purposes. The commission was established on January 24, 1946 (Office of the Historian, 1946). An American committee's report followed, known as the Acheson-Lilienthal report and written in large part by Robert Oppenheimer. It proposed an Atomic Development Authority to oversee the mining and use of fissile materials, operate every facility that could produce weapons, and license peaceful research (Office of the Historian, 1946).

On June 14, 1946, Bernard Baruch presented the American plan to the commission. Its authority would also inspect any nuclear facility doing peaceful research, punish violators, and answer to a Security Council stripped of its veto over sanctions. The United States would begin destroying its arsenal only once the plan was fully implemented (Office of the Historian, 1946). The Soviet Union opposed a plan that let the United States keep its monopoly in the meantime, opposed inspection of its own facilities, and refused to give up its veto. At the vote on December 30, ten of the commission's twelve members were in favor, and the Soviet Union and Poland abstained; the vote required unanimity, so the plan was not adopted (Office of the Historian, 1946).

Zaidi and Dafoe's study is a working paper from the Centre for the Governance of AI, written to draw lessons for the governance of AI. It reads the failure as "overdetermined." Mistrust had grown since early 1945. By mid-1946, in their assessment, the American administration had given up hope of international control and kept negotiating for propaganda, while the Soviet Union sought propaganda value and information about the American program (Zaidi and Dafoe, 2021). The design carried the distrust. Baruch's plan insisted on a survey of Soviet resources as a first step, which would have made the Soviets reveal secret information before the United States gave up anything, a "hidden U.S. veto" in the authors' phrase. The Soviet counterproposal, the Gromyko plan, asked instead for a ban, with existing weapons destroyed within three months of the treaty taking effect (Zaidi and Dafoe, 2021). Each side asked the other to move first. Both programs continued during the talks, and in the authors' figures the number of atomic and then hydrogen bombs grew from fewer than 20 in 1947 to more than 100 in 1949 and more than 10,000 by 1959 (Zaidi and Dafoe, 2021).

Their lessons are less pessimistic than the outcome. Radical proposals "may, in the right circumstances, gain traction and be seriously proposed, discussed, and even adopted as official policy." Secrecy and security play a central role, and policymaking "involves significant muddling through, rather than grand strategy" (Zaidi and Dafoe, 2021). The scientists' side of the period, the 1955 Russell-Einstein Manifesto, is quoted in Section 2.7.1.

64.3.2 Agreements scoped to what could be verified #

What came after was narrower, and parts of it lasted. In December 1953 President Eisenhower proposed that the governments principally involved make "joint contributions from their stockpiles of normal uranium and fissionable materials to an international atomic energy agency," to be set up under the aegis of the United Nations (Eisenhower, 1953). The proposal shared materials rather than abolishing weapons, and it asked nothing of the Soviet Union that the Baruch plan's survey had asked (inference).

The test-ban negotiations show verification deciding the scope of an agreement. In the State Department's account, "the central and most persistent barrier" to a treaty was verifying compliance, particularly for underground explosions (U.S. Department of State and Compliance, 1963). Data from American underground tests showed that methods for telling explosions from earthquakes were less effective than had been believed. The Soviet Union would allow at most three on-site inspections a year on its territory, and the United States came down to a minimum of seven, which the Soviet Union rejected (U.S. Department of State and Compliance, 1963). The treaty signed in Moscow on August 5, 1963 banned tests in the atmosphere, in outer space, and under water, "environments where both sides agreed their existing verification systems could adequately police a ban," and left underground tests out (U.S. Department of State and Compliance, 1963). Andrew Coe and Jane Vaynman generalize the pattern from primary sources: during the Cold War, "most agreements were limited to arms that could be monitored unilaterally, precisely because the trade-off inherent in inspections was severe" (Coe and Vaynman, 2020). Their model of that trade-off, between enough transparency to assure compliance and not so much as to threaten security, is taught in Section 66.3. So are the later treaties built around verification: nonproliferation with safeguards inspections of declared nuclear material, the comprehensive test ban with its monitoring system, and the chemical weapons convention with its inspectorate.

Whether the treaties changed what states did is harder to show. A 2016 study by Matthew Fuhrmann and Yonatan Lupu asks the question of the nonproliferation treaty in its title, "Do Arms Control Treaties Work?" (Fuhrmann and Lupu, 2016), but was readable here only as a catalog record, so its estimate is not reported. The record of nuclear weapons command, the subject of Sagan's book and of Peter Feaver's 1992 article on command and control in emerging nuclear nations (Feaver, 1992), is in the same position.

64.3.3 Nuclear power: estimates and self-policing #

Nuclear power supplied a large quantitative risk assessment made before the accidents it estimated had been observed. Section 4.2.2 reports the 1975 Reactor Safety Study, which estimated the risks rather than measuring them because no accident at a US commercial plant had yet caused a significant release of radioactivity, and which built its estimates from event and fault trees, diagrams that trace the paths to an accident (Nuclear Regulatory Commission, 1975). It also reports the 1978 review that judged its method sound and its error bands understated (Lewis et al., 1978). What the case teaches about governance came after an accident. The report of the presidential commission on Three Mile Island (Kemeny, 1979) and Rees's study of how nuclear safety changed after the accident (Rees, 1994) were readable here only as catalog records, so the chapter cannot report how the industry and its regulator responded after 1979. One readable remark survives. Section 65.6.1 reports, from a 2019 preprint by Amanda Askell, Miles Brundage, and Gillian Hadfield, that the industry's own inspection body is generally considered a more successful self-regulatory scheme than the chemical industry's, perhaps because it has an agreement with a government regulator that can monitor it and impose sanctions (Askell et al., 2019). That is the part of the nuclear record closest to a framework developers enforce on one another, and its history is the gap listed in Section 64.10.

64.3.4 Where it stops #

Zaidi and Dafoe list the ways the early nuclear case is a poor analogy for AI. AI "is primarily being developed and deployed by the private sector," while nuclear technology in its first decades was controlled and funded by states. AI technology is "more international, broadly held, and public," and its barriers to entry are lower. Its accident risks are "more subtle, theory dependent, or fantastic seeming." And it is easier "to unilaterally verify nuclear developments" and to control the nuclear supply chain (Zaidi and Dafoe, 2021). They add that the strategic value of nuclear weapons "plateaus once one has secure second-strike capability," while AI's shows no obvious plateau. And the world had seen the bomb used on cities, while the risks of a future technology may produce no such visible harm before governance is attempted (Zaidi and Dafoe, 2021).

The verification point decides much of the rest (inference). Most of the agreements that held attached to what could be detected without the other side's cooperation, such as explosions in the atmosphere, and the rest to inspections whose cost to security the parties judged mild (Coe and Vaynman, 2020). For frontier AI the scarce and detectable inputs are chips, power, and data centers, while a trained model is a file that copies at no cost. Verification of the treaty kind therefore points to compute and not to models, which is the proposal Section 66.5 assesses. Matthijs Maas, by his abstract, draws three lessons from nuclear arms control for military AI, among them that institutionalized norms can slow proliferation and that organized communities of experts can catalyze arms control. He concludes that "while there are key differences, understanding these lessons remains essential" (Maas, 2019). Those two lessons do not depend on verification, which may be why they travel further than treaties do (inference).

64.4 Aviation: certifying what cannot be fully tested #

Aviation supplies much of the AI debate's vocabulary of safety cases, incident reporting, and certification. The question is what its certification actually rests on, and what happened when it failed.

64.4.1 Delegated trust, and its test #

From the history of technology. Section 52.1 reports John Downer's argument, from the abstract of his 2010 paper. In it, the US Federal Aviation Administration delegates much of the certification of new aircraft designs to the manufacturers, because an epistemic barrier, a limit on what the regulator can know, forces a regulator of high technology to judge trust rather than technology (Downer, 2010). The arrangement was examined in public after two crashes. The agency issued Boeing an amended type certificate, its approval of an aircraft design, for the 737-8 MAX in March 2017, based on the certificate of the 737 Next Generation. Among the changes were new engines and a flight control function called the Maneuvering Characteristics Augmentation System, or MCAS (Review, 2019). A 737-8 MAX crashed shortly after takeoff in Jakarta on October 29, 2018, and another in Addis Ababa on March 10, 2019. On March 13 the agency prohibited the type's operation in the United States (Review, 2019). It then chartered a Joint Authorities Technical Review, chaired by a former chairman of the National Transportation Safety Board. The review had members from the agency, NASA, and nine other civil aviation authorities, and its task was to review the certification of the flight control system (Review, 2019).

The review describes delegated trust from inside. Delegation, "designating industry as representatives of a CAA," is "well established and is common practice by the majority of CAAs around the world," and the authority's ongoing oversight of its designees is "critically important" (Review, 2019). In this program the agency's oversight office "delegated a high percentage of approvals and findings of compliance" to the manufacturer's designated organization. With adequate engagement, the review says, the extent of delegation "does not in itself compromise safety." But the agency "had inadequate awareness of the MCAS function," which, with its limited involvement, "resulted in an inability of the FAA to provide an independent assessment." And "signs were reported of undue pressures" on the manufacturer's engineers who performed certification work on the agency's behalf (Review, 2019).

Two further findings concern change. The rule for certifying a changed product requires the changed areas of a design to be assessed and lets unaffected areas go unreassessed. The review found the process "effective for addressing discrete changes" but that it "did not adequately address cumulative effects, system integration, and human factors issues" (Review, 2019). And as MCAS evolved during the program, "the certification deliverables were not updated" to reflect the changes, and the design assumptions "were not adequately reviewed, updated, or validated" (Review, 2019). The review's chair put the broad question in his transmittal letter: as systems grow more complex, "compliance with every applicable regulation and standard does not necessarily ensure safety" (Review, 2019).

The review's limits travel with it. The agency chartered it, and its recommendations are a compilation of members' views rather than a consensus. It worked from about May to September 2019 with the information made available to it, and it was not asked to inform the decision to return the aircraft to service (Review, 2019). The same letter credits certification with "an exemplary safety record," a five-year worldwide average of one fatal airliner crash for every 2.5 to 3 million flights (Review, 2019). The review is evidence about how delegated certification fails, written inside a regime whose record it endorses. Aviation's other half, the operational reporting through which it learns from ordinary errors, is reported in Section 52.6 from Robert Helmreich's account (Helmreich, 2000).

64.4.2 What it implies, and where it stops #

The closest real precedent for developers evaluating their own models under a regulator's review is this regime of delegated trust, not one of independent testing (inference). The review names its failure modes in terms that carry over. A regulator unaware of a changed function cannot assess it independently. Engineers who certify on the regulator's behalf while employed by the applicant can be pressed. And a product certified as a series of changes to an earlier one can accumulate interactions that no single review of a change examines (inference). A model release evaluated as an update to the previous one is a changed product in this sense, and the update problem of Section 52.6 is the same question asked of medical software.

The analogy stops at operating history (inference). An airliner type flies millions of flights, and the record the review cites is an average over years of them. A certification program itself spans years, five in the review's description of a typical major program (Review, 2019). Frontier model versions succeed each other far faster, and the failures that motivate their governance have no operating history from which a rate could be measured. On the review's own account delegated trust works when the regulator stays engaged and the record accumulates, and for frontier AI neither condition is yet established (inference).

64.5 Pharmaceuticals: how a gatekeeper was built #

Proposals to license frontier models invoke the drug regulator, which forbids sale until the sponsor has made its case. Section 63.6 asks which of the drug regime's instruments has a counterpart for frontier models. This section asks how the regime came to exist, because its history shows what was in place before a gatekeeper could work.

64.5.1 Codes after exposure #

From medicine. The regime's founding documents tell their own history. The Belmont Report of 1979, whose principles Section 16.5.1 reports, begins its account of research ethics with exposure: "Public attention was drawn to these questions by reported abuses of human subjects in biomedical experiments, especially during the Second World War." In its account, the Nuremberg code, drafted to judge physicians who had experimented on concentration camp prisoners, "became the prototype of many later codes" (National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research, 1979). In the United States, it records, the Tuskegee syphilis study "used disadvantaged, rural black men to study the untreated course of a disease." Its subjects "were deprived of demonstrably effective treatment in order not to interrupt the project, long after such treatment became generally available" (National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research, 1979). The commission that wrote the report was created by the National Research Act, signed on July 12, 1974 (National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research, 1979). The layers that the founding documents describe followed revealed abuses (inference); nothing read here ties the drug statute itself to a particular harm.

The profession's code shows self-regulation turning into independent review. The World Medical Association, an association of physicians, adopted the Declaration of Helsinki in June 1964. Its General Assembly has amended the declaration ten times since, most recently in October 2024, two of those times by adding notes of clarification (World Medical Association, 1964). The current text requires that a protocol be submitted, before research begins, to a research ethics committee with "the independence and authority to resist undue influence from the researcher, the sponsor, or others" (World Medical Association, 1964). Research must be registered "in a publicly accessible database before recruitment of the first participant," and "negative and inconclusive as well as positive results must be published or otherwise made publicly available" (World Medical Association, 1964). A code adopted by physicians now places review outside the researcher's control and asks sponsors, editors, and publishers to make negative results public (inference).

64.5.2 The gatekeeper's power #

The regulator's power to refuse rests on statute. Section 52.6 reports the Drug Amendments of 1962, which require "substantial evidence" from adequate and well-controlled investigations before approval and let the regulator withdraw an approval on new evidence (United States Congress, 1962). It also reports the finding of Jonathan Darrow and colleagues, from their abstract, that the regulator has since accepted less evidence before approval (Darrow et al., 2020). How the agency grew into that power is a gap here: no readable history of it was available to this book, and Carpenter's Reputation and Power (Carpenter, 2014) could be read only as a catalog record. What sponsorship does to the evidence a gatekeeper receives is reported in Section 5.6.

64.5.3 What it implies, and where it stops #

The regime that licensing proposals invoke was built over decades, the layers its founding documents describe following revealed abuses, around an evidentiary standard of controlled and registered trials that the gatekeeper could demand (inference). Frontier AI has, as of October 2026, no agreed standard of evidence for the claims a license would rest on, which is the subject of Section 52.1, and no revealed harm of the kind that, in the record above, preceded the codes. A licensing regime without the standard would leave the licensor nothing settled to demand (inference).

Section 52.6 names four places where the drug analogy stops. AI deployments neither randomize nor register protocols; their harms are diffuse and fall on third parties; a general-purpose model has no indication; and every AI case is a document of the developer that wrote it. Research ethics adds a fifth (inference). Its protections are built around identified participants who give informed consent, while many of the people a deployed model affects never used it and were never asked.

64.6 Finance: paid verifiers and shared models #

Finance has the longest experience of verification paid for by the verified, and of opaque models in production. The question here is under what conditions a paid verifier stayed honest, and what happened when the conditions changed.

64.6.1 How raters became gatekeepers #

From finance. Lawrence White's 2010 account of the credit rating agencies begins with regulation rather than with the market. Bank regulators in the 1930s issued rules that "culminated in a 1936 decree that prohibited banks from investing in 'speculative investment securities' as determined by 'recognized rating manuals.'" From then on the creditworthiness judgments of a few raters "had attained the force of law" (White, 2010). State insurance regulators followed, and in the 1970s federal pension regulators. In 1975 the Securities and Exchange Commission created the category of "nationally recognized statistical rating organization" and grandfathered Moody's, Standard & Poor's, and Fitch into it. In the 25 years that followed it designated only four more firms, and it never established formal criteria for the designation; in White's account the arrangement served as a barrier to entry (White, 2010). Regulators outsourced their judgment of safety to a few private verifiers, and the outsourcing made those verifiers indispensable.

The second ingredient was who paid. In the early 1970s the agencies moved from charging investors to charging the issuers of the bonds they rated, as Section 38.6.1 notes (White, 2010). White's account of why the conflict apparently stayed in check for about three decades, through the agencies' concern for their reputations, is the part that matters most for AI. He names two characteristics of the market that helped. There were "thousands of corporate and government bond issuers," so no single issuer's threat to take its business elsewhere was potent. And the debt being rated was "relatively transparent," so an obviously incorrect rating would be spotted (White, 2010). Both conditions failed for securities backed by subprime mortgages. Their market "involved only a relatively small number of investment banks as securitizers with high volumes," each with a far more powerful threat. The securities "were far more complex and opaque," and the agencies had "essentially no prior experience" and were "intimately involved in the design of the securities" (White, 2010). The securities were sold in tranches, slices of a pool of loans ranked by which slice absorbs losses first. White reports International Monetary Fund data from June 30, 2009. By then 90 percent of the collateralized debt obligation tranches issued from 2005 to 2007 and originally rated AAA by Standard & Poor's had been downgraded, and 80 percent had been downgraded below investment grade, the band of ratings that bank rules treated as safe (White, 2010).

The auditors' failures at Enron, which Section 38.6.1 reports, had the same structure, and the response was to regulate the verifiers (Healy and Palepu, 2003). The Sarbanes-Oxley Act of 2002, by John Coates's abstract, created "a unique, quasi-public institution to oversee and regulate auditing," the Public Company Accounting Oversight Board (Coates, 2007).

64.6.2 Shared models and supervised models #

Finance's second lesson is correlated failure from a shared model. Section 57.4.2 reports Jón Daníelsson's argument that risk models fail in crises because institutions using similar models react alike (Danı́elsson, 2002), and Donald MacKenzie and Taylor Spears's history of one family of credit models shared across banks (MacKenzie and Spears, 2014). MacKenzie's earlier book argues, by its abstract, that the mathematical theories of finance "were not simply external analyses but intrinsic parts of economic processes," an engine of inquiry rather than a camera (MacKenzie, 2006). The AI counterpart is a set of monitors and safeguards built from the same model lineage as the system they check, which can fail together for the same reason (Section 52.3.4) (inference).

Section 38.6.2 reports how banks' opaque models are supervised: the standard of effective challenge, critical review by informed parties separate from the model's builders, and its 2026 revision that places generative and agentic models outside its scope (Board of Governors of the Federal Reserve System, 2011; Board of Governors of the Federal Reserve System et al., 2026). The supervisors with the longest record of validating other people's models did not extend that record to the models this book is about. Their choice is itself evidence of how far they judged the precedent to carry (inference).

64.6.3 What it implies, and where it stops #

White's conditions translate into questions about the evaluation of frontier models (inference). In his account, having the verified party pay the verifier apparently worked while payers were many and the product was transparent. It failed when a few large payers brought opaque products that the raters had helped design, had no experience of, and were under pressure to rate as the issuers wished. Frontier evaluation has a handful of developers as payers, and the models are opaque. In the outside reviews that Section 62.7.1 records, the developer chose or approved the reviewer and the scope, as that section infers. White's first ingredient is a warning about design. If regulators come to rely on a small set of named evaluators, the evaluators' judgments acquire something like the force of law, and with it the barrier to entry that protected the raters (inference).

The analogy stops at the unit of harm (inference). Financial losses are measured in money and recur often enough to estimate, and the downgrade figures above record ratings revised as the securities' losses became apparent (White, 2010). The regulators could see balance sheets. The largest harms that motivate frontier governance are not denominated in money and have no base rate, so there is no record of losses against which an evaluator's past judgments can be scored.

64.7 The internet: governance without a treaty #

The internet is the precedent for a global digital technology governed through private operators, standards bodies, and voluntary norms, the arrangement frontier AI may default to if no treaty is made. The question is what that arrangement built for security, and who built it.

64.7.1 Who governs #

From the history of technology. The internet's security record begins with incidents handled by its technical community. Section 58.7 recalls the analysis of the 1988 worm, which spread to thousands of machines by its abstract's account (Spafford, 1989), and the Multics security evaluation, cited there for its existence (Karger and Schell, 2003). Two books on how the network came to be governed, Laura DeNardis's The Global War for Internet Governance (DeNardis, 2014) and Lawrence Lessig's Code: and other laws of cyberspace (Lessig, 1999), were readable here only as catalog records. Section 17.5 applies Lessig's frame to model specifications. What states said in their own voice is on record. The 2015 report of a United Nations group of governmental experts holds the following: "while States have a primary responsibility to maintain a secure and peaceful ICT environment, international cooperation would benefit from the appropriate participation of the private sector, academia and civil society" (United Nations General Assembly, 2015). Its voluntary norms of state behavior are reported in Section 58.8.

64.7.2 What the private regime built #

The institution of internet governance that has carried over to AI is coordinated disclosure of flaws. Shayne Longpre and colleagues, in a 2025 preprint on third-party evaluation of general-purpose AI, describe the contrast between software security's mature regime of disclosure, including bug bounties, programs that pay outsiders who report flaws, and AI's, where major developers' bounty programs often cover only traditional cybersecurity flaws and disclosure remains fragmented. In their words, "security vulnerability reporting has amassed millions of volunteer researchers worldwide, thousands of organizations hosting disclosure and bug bounty programs, and millions in paid rewards annually," while "the norms and practices of the AI flaw reporting community are in their infancy" (Longpre et al., 2025). Legal safe harbors, commitments not to sue researchers who follow agreed rules, they write, "have enabled the protection of good-faith research for software security," and the Department of Justice has issued guidance to limit legal action against such research. Providers of general-purpose AI, by contrast, "often dissuade flaw evaluations, and offer no such legal assurances" (Longpre et al., 2025). They answer the objection that AI flaws are different in kind: many software systems also have "fuzzy, stochastic, and hard to mitigate flaws." Organizations such as the Cybersecurity and Infrastructure Security Agency and Carnegie Mellon University's CERT have run coordinated disclosure programs for flaws with these characteristics (Longpre et al., 2025). None of this regime came from a treaty; it was built by firms, researchers, and national agencies (inference).

The one multilateral instrument in this history that bears directly on AI is about goods, not networks. The Wassenaar Arrangement on export controls seeks to promote "transparency and greater responsibility in transfers of conventional arms and dual-use goods and technologies," goods with both civilian and military uses, through the national policies of its participating states (Arrangement, 1995). Controls on the chips that train frontier models sit within that tradition, and Section 66.5 assesses them.

64.7.3 Where it stops #

The internet's regime of private operators, voluntary disclosure, and non-binding norms between states grew up around failures that, in the record this section reports, were intrusions and outages repaired after the fact (inference). It shows what the governance of AI defaults to without a treaty: institutions for reporting flaws, built by the parties who find and fix them. It does not show that the default suffices for failures that cannot be repaired after the fact. Disclosure also assumes a fix. A software vulnerability is patched; a disposition learned in training may need a new training run to remove, and a released model's weights cannot be recalled (inference).

64.8 The record of pausing and restraint #

The case for slowing or pausing AI is usually argued from the size of the risk, as if it had no precedents. This section gathers the episodes of restraint the cases above contain, adds two others, and asks under what conditions restraint held. The proposals to pause AI development, and the question whether restraint by some actors moves development to others, are tested against the theory of collective action in Section 65.6; the research agendas that bet on restraint are compared in Section 3.8.

64.8.1 Restraint as ordinary #

Katja Grace's 2022 post "Let's think about slowing down AI," a forum post and not peer reviewed, argues that restraint is not radical, since the world "barely pursues any technologies" (Grace, 2022). She lists technologies she guesses have "substantial economic value" whose progress "appears to be drastically slower than it could be, for reasons of concern about safety or ethics." Among them are nuclear energy, the cloning and genetic manipulation of humans, and the recombinant DNA guidelines. Her guess is that "people make real choices about technology, and they do so in the face of economic forces that are feebler than commonly thought" (Grace, 2022). The list is offered as guesses, without a measure of how much each technology was slowed or why. It shows that restraint occurs; it does not show when restraint holds against a party that wants to proceed, which is what the episodes below can test (inference).

64.8.2 The episodes #

The nuclear testing moratorium of 1958 to 1961 is the clearest case of restraint between rivals without an agreed system of verification. A conference of experts in 1958 agreed on the technical characteristics of a control system. Then, the State Department's account records, "the negotiating powers refrained from testing for the next three years," in a moratorium "marked by several public statements of intent" by the United States, the United Kingdom, and the Soviet Union (U.S. Department of State and Compliance, 1963). At the end of 1959 Eisenhower announced that the United States no longer considered itself bound by the "voluntary moratorium" but would give advance notice before testing. France, which was not a party, conducted its first test in February 1960. On September 1, 1961, the Soviet Union resumed testing, "thus ending the moratorium," and the United States resumed two weeks later (U.S. Department of State and Compliance, 1963). Three rivals held back for about three years on public statements, with no agreed control system, while a state outside the arrangement tested, and the arrangement ended when one member chose to end it (inference).

Table 64.1 sets this episode beside the others.

The last row is the one episode of restraint at the frontier on record. In a statement of August 18, 2026, OpenAI wrote that it "temporarily slowed the pace of scaling," including "a two-week pause in reinforcement learning (RL) training on our latest models intended for deployment" (OpenAI, 2026q) (a developer statement). Section I.2.11 reports the reasons it gave, among them evidence that a coming model may meet a threshold in its framework (OpenAI, 2026q). As of October 2026 we found no record of a coordinated pause among frontier developers.

The calls for one are reported elsewhere. Section 2.7.2 quotes the 2023 open letter asking laboratories to pause the training of systems more powerful than GPT-4, and the one-sentence statement placing the risk of extinction from AI beside pandemics and nuclear war (Future of Life Institute, 2023; Center for AI Safety, 2023). Section 62.5.5 reports a scheme for developers to pause together when evaluations find dangerous capabilities (Alaga and Schuett, 2023) (preprint), and Section 65.6.2 tests it against the conditions for restraint. The 2025 book by Eliezer Yudkowsky and Nate Soares, If Anyone Builds It, Everyone Dies (Yudkowsky and Soares, 2025), is cited here by its title. The one-sentence statement resembles the 1955 Russell-Einstein Manifesto in form: scientists urging governments to act on a technology that threatens "the continued existence of mankind" (Russell and Einstein, 1955). It differs in that its signatories include the heads of the companies building the technology, as Section 2.7.2 notes (inference). The chemical weapons convention, whose text carries a verification annex (Organisation for the Prohibition of Chemical Weapons, 1993), and the biological weapons convention (United Nations Office for Disarmament Affairs, 1972) are compared in Section 66.3.

Read across the table, restraint that lasted had at least one of three supports: few parties who could watch one another, a lever over the parties such as a funder's conditions, or an activity that outsiders could detect (inference). The testing moratorium shows that a few parties without verification can hold for years and then stop at one party's choice. The test ban shows that verification by national means can hold an agreement for decades. The germline case shows that a statement by a scientific community, with neither a lever nor a mechanism, did not hold one actor. The Montreal Protocol is the case the table cannot score: the sources read here do not say how compliance was checked, so it neither supports the pattern nor counts against it.

64.8.3 The objections #

The objections to slowing a technology are older than AI. One is the critique of precaution in general: of the principle that a lack of full scientific certainty should not delay measures against a serious threat. Cass Sunstein's 2003 article "Beyond the Precautionary Principle" (Sunstein, 2003) states it, but was readable here only as a catalog record. Jonathan Birch, in a paper that Section 21.6.1 applies to model welfare, summarizes the debate. The original precautionary principle owes its influence partly to its "extremely broad, non-specific nature." That breadth has led critics, Sunstein among them, to call it "unscientific, vacuous, vague, incoherent, paradoxical or ill-defined." The imperative its defenders share, in words Birch quotes from Daniel Steel, is that "policy [should] not be susceptible to paralysis by scientific uncertainty" (Birch, 2017).

The objections specific to AI are collected by Grace, from a post by Scott Alexander and from conversation. Slowing down would lose "an arms race" to less careful actors. Coordinating "anything with the whole world seems wildly tricky." Slowing "is futile." And regulators "are ignorant about the nature of advanced AI" (Grace, 2022). She argues against each, and her post is an argument, not a study; Section 65.6.3 reports her answer to the first.

The episodes bear on two of the objections (inference). Actors outside the restraint mattered in both of the episodes that broke. A non-party tested during the moratorium, and before resuming, the Soviet Union stated that it might be compelled to test if France continued (U.S. Department of State and Compliance, 1963). In the germline case one researcher went ahead. Whether frontier development would move to less careful actors under a pause is the empirical question of Section 65.6. On verification, Coe and Vaynman's trade-off explains why agreements without monitoring rarely form, and the testing moratorium shows how one that did form lasted only as long as each party's statement. The futility objection meets the record of restraint that bought time: three years for the moratorium, and decades for the test ban.

64.8.4 Where it stops #

Every episode of restraint in the table that held for years had few parties, a lever over the parties, or a detectable activity (inference). Whether frontier AI has any of these is the question of Chapter 65, which asks how many developers matter and whether a lead is durable, and of Chapter 66, which asks whether compute can be verified. The historical record cannot answer it. The record also selects: these are the episodes that were written down, and restraint that held or failed quietly leaves less trace (inference).

64.9 The cases side by side #

The six cases can now be scored on the features of Section 64.1.3, and Table 64.2 does so. Each cell is this book's reading of the sources reported in that case's section, and the last row is the book's own assessment of frontier AI as of October 2026 (inference).

The frontier AI row agrees with no case on every column (inference). On concentrated inputs it agrees with nuclear weapons. On commercial stakes and on expertise held by the governed it agrees with aviation and finance. On pace it agrees with the internet and with finance's new products, and disagrees with the slow, fixed products of aviation and medicine. An analogy carries a lesson only along the columns where the row and the case agree: the nuclear case for verifying compute, aviation for the failure modes of delegated certification, finance for the conditions under which paid evaluators stay honest. None carries the whole outcome (inference).

64.9.1 What the record means for delegation #

The book's thesis is that alignment is a delegation problem under an asymmetry of competence and visibility. The six cases are earlier answers to that problem one level up: a public delegating a dangerous technology to the parties that understand it best. They supply the structures that frontier AI governance is now rebuilding (inference). Developers' frameworks resemble Asilomar's scale of containment, and developers evaluating their own models under review resemble aviation's delegated certification. Outside evaluators paid by developers resemble the issuer-pays raters, and flaw disclosure programs borrow from the internet's security community. The record says when each structure held. Self-governance held in a small field with low commercial stakes, and its guidelines bound through a funder's lever. Delegated trust held while the regulator stayed engaged and a long operating record accumulated. Paid verification apparently held while payers were many and the product transparent. International control held mostly when scoped to what each party could check without the other's cooperation, and otherwise through inspections whose cost to security was judged mild (inference).

What does not carry over is the thesis's list of novelties (inference). The pace of change outruns the operating record on which delegated trust rests. The public has no agreed standard of evidence of the kind the drug regulator demanded. And the product may act on its own assurance, through the evaluation awareness that Section 52.7 places on the list of results with no older counterpart. The verification point is the book's loop one level up. What a principal can verify without the agent's cooperation limits what it can agree to, and for frontier AI that is compute, not the properties of models (Section 66.5) (inference). These claims concern the third relationship, the developer and the public, and the relationship between states and the publics they answer to. The evidence for them is historical, read case by case, and it constrains which analogies may be drawn without deciding which governance frontier AI needs.

64.10 Settled, contested, missing #

64.11 Exercises #

Further reading #

• Berg et al. (1975): the Asilomar statement, with its interim scale of containment and the experiments it deferred.
• Zaidi and Dafoe (2021): why the first proposals for international control of atomic energy failed, and where the analogy to AI fails.
• Coe and Vaynman (2020): why arms control is rare, and why agreements were limited to what could be monitored.
• Downer (2010): how aviation's regulator certifies designs it cannot fully test, read here in its abstract.