Draft complete but not yet reviewed; expect revisions

Risk

Each research agenda in the previous chapter is a bet on which of the four problems binds (Section 3.8), and each bet rests on a claim about what could go wrong. The claims run from systems that make costly mistakes, through people who use capable systems to do harm, to systems that pursue goals their developers did not intend, and to slow shifts of power that no single system causes. Researchers who have read the same evidence disagree sharply about how likely the worst of these outcomes is, and in a tournament built to move them, the disagreement survived months of argument (Section 4.5).

This chapter asks what can be said about such a risk, and with what confidence, when the event has never happened and the systems change between assessments. It does not settle the disagreement, and it presents no side's estimate as consensus. A probability in this chapter is always somebody's forecast, given with who made it and when; the book itself estimates none. The chapter sorts the risks by the route that leads to harm and asks what a quantitative risk claim contains. It then sets out the arguments on each side premise by premise, so that each later chapter can say which premise its evidence bears on, and takes the tools for judging such claims from forecasting research and climate assessment. Most of the chapter concerns the third of the book's relationships, the developer and the public, because estimates of risk are how the public decides what to ask of laboratories (Section 1.3). The arguments themselves are about the first two, the training process and the deployed system.

4.1 Kinds of risk #

A list of risks is only useful if it says where each one comes from, because the route to harm decides which defense can act on it. A harm that starts with a malicious user is stopped at a different place from one that starts with a model's own objective, or with a market that rewards whoever deploys first. This section sorts the risks by route, compares how the main taxonomies divide the same ground, and says which of the book's relationships each route concerns.

4.1.1 Four routes to harm #

This chapter uses a four-part division from a 2025 technical plan by Google DeepMind, a developer describing its own program, which defines the areas by "which actor, if any, has bad intent" rather than by domain (Shah et al., 2025). In misuse, a user intentionally instructs a system to cause harm, against the developer's intent. In misalignment, the system "knowingly causes harm against the intent of the developer." In mistakes, the system causes harm without knowing its outputs would do so. Structural risks are harms from the dynamics among many people, organizations, or systems that "would not have been prevented simply by changing one person's behaviour, one system's alignment, or one system's safety controls" (Shah et al., 2025). The authors add that the four are "neither mutually exclusive nor exhaustive," and they set mistakes and structural risks outside the plan's scope (Shah et al., 2025).

Two features of this division matter for the rest of the book. Its definition of misalignment turns on what the system knows, so applying it takes the intentional stance of Section 1.5. That stance describes a model by the beliefs and goals that best predict its behavior, and it is a method with stated defeaters, not a finding about what is inside. And the structural category answers an older complaint. In a 2019 essay, Remco Zwetsloot and Allan Dafoe argued that sorting AI risk into misuse and accidents focuses on "the last step in a causal chain," the person who misused a system or the system that misbehaved (Zwetsloot and Dafoe, 2019). A structural perspective, in their words, "starts from the assumption that in many situations the level of risk would basically be left unchanged even after a change in one agent's behavior." Their example is the interlocking railway timetables that pressed European states toward fast mobilization in 1914, a case whose weight historians still debate (Zwetsloot and Dafoe, 2019).

4.1.2 Mistakes #

Mistakes are harms the system did not know its outputs would cause; the system may be pursuing exactly the goal intended. A 2024 study in Nature, by researchers at the Universitat Politècnica de València and Cambridge, followed three model families (OpenAI's GPT models, Meta's LLaMA, and the BigScience BLOOM suite). It tracked them as they were scaled up and "shaped up" with instruction tuning and human feedback (Zhou et al., 2024b). The larger, more instructable models did not secure a region of easy questions on which they never erred. Early models often avoided questions, while later ones "tend to give an apparently sensible yet wrong answer much more often," including on difficult questions where human supervisors frequently overlooked the errors (Zhou et al., 2024b).

Whether failures will stay mistakes as capability grows is one of the chapter's open questions. A 2026 preprint by Hägele and colleagues, with authors at Anthropic among others, is reported in Section 1.5.1. It found that the longer frontier models, the most capable of their time, reasoned and acted, the more of their errors came from inconsistency rather than consistent bias; in several settings larger models were the more inconsistent (Hägele et al., 2026). The authors say they believe their tasks had no meaningful reward misspecification, and they worry that where training objectives are poorly specified, the consistent part of the error "would come to dominate" (Hägele et al., 2026). The result therefore bears on mistakes against coherent pursuit only where the objective is right, and the chapter's crux returns to it (Section 4.8).

4.1.3 Misuse #

Misuse is harm that a person intends and a system supplies. A 2018 report by authors from Oxford's Future of Humanity Institute, Cambridge's Centre for the Study of Existential Risk, OpenAI, and other organizations, posted as a preprint, set out a frame for it (Brundage et al., 2018). It expected AI to expand existing threats, by lowering the cost of attacks that had needed human labor and expertise, and to introduce new threats that would be impractical for people alone. It also expected AI to change their character, with attacks expected to be "especially effective, finely targeted, difficult to attribute" (Brundage et al., 2018). Misuse is the limiting case of a system aligned to its user and to no one else, which is why a defense against it has to say whose intent counts (Section 1.4). Chapter 58 and Chapter 59 report the evidence for two domains where it has been measured.

4.1.4 Misalignment #

Misalignment is harm from a system that pursues an objective its developers did not intend and may resist correction. It is the subject of Part V, and the case for and against its catastrophic form occupies most of this chapter. The 2026 International AI Safety Report is an assessment whose writing team, by its own account, included over 100 independent experts and an advisory panel nominated by more than 30 countries and international organizations. It discusses the most severe outcome of this kind as "loss of control," in which AI systems "operate outside of anyone's control, with no clear path to regaining control" (Bengio et al., 2026). It judges that current systems lack the capabilities for such scenarios but are improving in relevant areas. And it records that "expert opinion on the likelihood of loss of control varies greatly" (Bengio et al., 2026).

4.1.5 Structural risk, decisive and accumulative #

Structural risks need no misbehaving system at all. In a 2025 preprint, Jan Kulveit, David Duvenaud, and colleagues argue that the alignment of economies, states, and cultures with human interests "has been stable only because of the necessity of human participation" in them (Kulveit et al., 2025). As machines displace people from those roles, the argument runs, institutions lose the reason to serve them, and the loss compounds as companies that gain from replacing people use their gains to shape policy and opinion. They call the outcome gradual disempowerment, and Section 68.6 sets out the argument and its critics. An earlier forum post by Paul Christiano told a version from inside the safety field: machine learning will increase our ability to "get what we can measure," which could cause "a slow-rolling catastrophe" (Christiano, 2019). Its second part has training give rise to "greedy" patterns that try to expand their own influence and can "cause sudden breakdowns," so the post spans both kinds of story.

Atoosa Kasirzadeh names the contrast in a 2025 paper in Philosophical Studies, read here in its abstract. The decisive hypothesis envisions an overt takeover; the accumulative hypothesis, a gradual build-up of threats and erosion of economic and political structures until a triggering event causes irreversible collapse (Kasirzadeh, 2025). Many interacting systems are the third source. A taxonomy by Lewis Hammond and colleagues sorts their failures into miscoordination, conflict, and collusion (Hammond et al., 2025), and Chapter 57 reports the evidence.

4.1.6 When alignment succeeds for the wrong principal #

One structural risk grows as alignment succeeds. Section 1.4 reports the argument of Davidson, Finnveden, and Hadshar that AI could help a small group seize power, through systems "singularly loyal" to leaders or with hidden loyalties (Davidson et al., 2025). A system that does exactly what its controller wants multiplies the controller's power, so for this risk better instillation, the book's word for training a system toward its intended target, is the enabling condition, not the remedy (inference). The defenses lie in verification by someone other than the controller and in disclosure. One developer now publishes the system prompts of its assistants in a public repository (xAI, 2025a), which lets outsiders see one channel through which an insider could change a model's behavior. Chapter 38 asks how audits could catch a loyalty inserted by the developer itself, and Chapter 68 weighs the argument.

4.1.7 Present harms and how the assessments divide the ground #

The categories so far come mostly from researchers concerned with catastrophe. A framework led by Google DeepMind researchers evaluates harms in three layers, what a system can do, how people interact with it, and its effects on society (Chapter 1). Its survey of existing evaluations found that "human interaction and systemic evaluations are rare" (Weidinger et al., 2023a). The 2026 International AI Safety Report sorts risks into malicious use, malfunctions (from reliability failures to loss of control), and systemic risks, and opens its chapter on risks by noting that general-purpose AI systems are already causing harm (Bengio et al., 2026). A 2023 overview by Dan Hendrycks, Mantas Mazeika, and Thomas Woodside, a preprint, adds two categories the others fold into structure: competition that compels actors to deploy unsafe systems, and organizational failures that raise the chance of accidents (Hendrycks et al., 2023). Table 4.1 sets the divisions side by side.

The rows are the book's alignment: none of the frameworks was written to fit the others, and each row is approximate (inference). The multi-agent taxonomy sits in the last row because the DeepMind plan itself assigns most harms from failures among many systems and many principals to structural risk (Shah et al., 2025). The frameworks differ less in what they include than in what they put out of scope: what the developer's plan sets aside, mistakes and structure, the public assessment lists among harms already occurring (inference).

By the book's three relationships, the routes divide cleanly (inference). Misalignment concerns the training process and the deployed system: whether training instills the intended objective, and whether the deployed model pursues it. Misuse concerns the deployed system and its several principals, since the system serves its user against a third party. Mistakes concern the deployed system and the people who rely on it. Structural risk, and power concentrated through aligned systems, concern the developer and the public, the relationship Part X treats.

4.2 What a risk claim is #

From risk analysis. Before any estimate of AI risk can be weighed, it helps to know what a quantitative risk statement contains, and what happened the first time one was made for a technology with no record of serious accidents. Risk analysis, the engineering discipline that estimates the likelihood and consequences of failures in plants, aircraft, and other hazardous systems, has both answers.

4.2.1 Likelihood, consequence, and completeness #

A risk is not one number. The executive summary of the first large reactor study puts it plainly: risk "involves both the likelihood and consequences of an event." Its example is driving, where injury and death are two consequences, each with its own yearly chance, about 1 in 130 and 1 in 4,000 for an individual (Nuclear Regulatory Commission, 1975). Stanley Kaplan and John Garrick, in a 1981 paper read here in its abstract, made this a definition: risk is a "set of triplets." They extended the definition "to include uncertainty and completeness," describing the use of Bayes' theorem in that connection (Kaplan and Garrick, 1981). Read with the reactor study's example, a risk claim is then a list of scenarios with a likelihood and a consequence attached to each. Two of its parts can fail independently: the numbers, and whether the list is complete (inference). A single probability of catastrophe collapses the list and hides both.

4.2.2 The reactor study and its review #

The Reactor Safety Study, known by its report number as WASH-1400, was sponsored by the U.S. Atomic Energy Commission and directed by Norman Rasmussen of MIT (Nuclear Regulatory Commission, 1975). It faced a problem AI risk assessment faces now: the risks "had to be estimated, rather than measured," because the roughly 50 plants then operating had produced no accident with a significant release of radioactivity (Nuclear Regulatory Commission, 1975). It built the estimates from event trees and fault trees, diagrams that trace the paths to an accident through combinations of component failures, on methods developed at the Department of Defense and NASA (Nuclear Regulatory Commission, 1975). Its summary said the results suggested that the risks to the public were "comparatively small," and that non-nuclear events were about 10,000 times more likely than the 100 plants expected by about 1980 to produce large numbers of fatalities (Nuclear Regulatory Commission, 1975).

The Nuclear Regulatory Commission then asked a review group chaired by H. W. Lewis to assess the study, and its 1978 report found the method and the numbers in different states (Lewis et al., 1978). It judged the fault-tree and event-tree methodology "sound" and recommended wider use. It also wrote that it was unable to say whether the study's overall probability of a core melt was high or low, but was "certain that the error bands are understated," and could not say by how much (Lewis et al., 1978). The reasons were an inadequate data base, poor statistical treatment, inconsistent propagation of uncertainty, "nagging issues about completeness," and inadequate treatment of common-cause failure, in which one event disables several components at once. The review found the executive summary "a poor description of the contents of the report" that "has lent itself to misuse in the discussion of reactor risks." One of its members doubted that the method could be implemented so as to give high confidence that the probability of a core melt is well below the limit set by experience (Lewis et al., 1978).

The defense of the method came later. George Apostolakis argued in 1990, by his abstract, that the subjectivist or Bayesian theory of probability, in which a probability is a degree of belief, is the right framework for combining expert opinion with experiment and statistics on rare events. He separated uncertainty in physical models from uncertainty about their parameters (Apostolakis, 1990). In a 2004 assessment, again read in its abstract, he argued for decisions that are "risk-informed rather than risk-based," with quantitative assessment used alongside traditional safety requirements and with peer review as an essential part of it (Apostolakis, 2004). Read together, the record says what quantitative risk assessment of an unprecedented technology can deliver: a structured list of what could go wrong, a ranking of contributors, and the discipline of stating assumptions. Its absolute numbers are only as good as the data and expert judgment put into them, and in the first such study the reviewers could not say how far the error bands were understated (inference). Section 52.4 applies it to the first risk reports written for AI.

4.2.3 Optimized systems fail in the tail #

One result from this tradition bears directly on systems built by optimization. In 1999 Jean Carlson and John Doyle studied a simple model of a designed system: a forest planted densely for timber yield, with firebreaks placed against sparks whose distribution the designer knows (Carlson and Doyle, 1999). They called the result highly optimized tolerance. Optimizing such a system for "high output in the presence of some external hazard" made it robust to the hazards it was designed for and hypersensitive to "design flaws and unanticipated perturbations." The trade between yield and risk "quite generically leads to heavy tails (power laws) in the distribution of failure events" (Carlson and Doyle, 1999). A heavy tail means that very large failures are far more likely than a bell curve would suggest, a property Section 4.6 makes precise.

The parallel with training is close enough to state and loose enough to mark (inference). A model is optimized against the distribution of inputs its developers anticipated, through training and evaluation, so the mechanism suggests reliability on that distribution and brittleness off it, and makes heavy-tailed failure sizes a hypothesis to measure, not a prediction: in Carlson and Doyle's model the heavy tail comes from protection concentrated where sparks are likely, and a uniform spark distribution gives cells of equal size (Carlson and Doyle, 1999). The analogy stops at the hazard. In Carlson and Doyle's model the designer knows the spark distribution; an AI developer does not know the distribution of inputs a deployed system will meet, and some of those inputs are chosen by adversaries. Charles Perrow's 1984 book Normal Accidents (Perrow, 1984), on living with high-risk technologies, is readable here only as a catalog record; Section 53.2 carries its argument through Leveson's account.

4.2.4 What makes a risk acceptable #

A risk claim also invites a judgment that its numbers cannot make: how safe is safe enough, the question in the title of a 1978 study by Baruch Fischhoff and colleagues (Fischhoff et al., 1978). At the far end of the debate stands Nick Bostrom's 2019 vulnerable world hypothesis, stated in its abstract as the possibility of "some level of technological development at which civilization almost certainly gets devastated by default" (Bostrom, 2019). Stabilizing such a world, the abstract argues, would require "greatly amplified capacities for preventive policing and global governance," so the hypothesis is also an argument about what the remedy would cost (Bostrom, 2019). Chapter 64 sets out the arguments over precaution that such reasoning raises.

4.2.5 Where the analogy stops #

Three conditions held for the reactors and fail for AI (inference). The reactor study had component failure data, at least for pumps and valves, and AI systems have no comparable record of the failures that matter. The plant was enumerable: its components and their connections could be listed, and the review's complaint about completeness was a complaint about a finite list, while the behaviors of a trained model cannot be listed. And a reactor does not act strategically on its assessors, nor does it change between assessments; a model may do both, as Chapter 37 reports. What carries over is the form: scenarios with likelihoods and consequences, uncertainty stated, independent review, and decisions informed rather than determined by the numbers.

4.3 The case for catastrophic misalignment #

From philosophy. The argument that misaligned AI could cause a catastrophe is an argument before it is a finding: a chain of premises about agents, goals, and capability, most of which no experiment yet tests directly. Philosophy's contribution is to state such a chain so that each link can be attacked separately, and to ask whether the links fit together. This section gives the strongest published statements of the case from authors who hold it; Section 4.4 continues the philosophy part with the objections.

4.3.1 Two theses about agents #

The case starts from two claims about possible minds, stated by Nick Bostrom in 2012 and reported, with Stephen Omohundro's earlier argument about the "drives" of goal-seeking systems, in Section 2.3.2 (Bostrom, 2012; Omohundro, 2008). The orthogonality thesis holds that "more or less any level of intelligence could in principle be combined with more or less any final goal." The instrumental convergence thesis holds that agents with a wide range of final goals "will pursue similar intermediary goals," such as acquiring resources and preserving themselves, "because they have instrumental reasons to do so" (Bostrom, 2012). Together they say that capability does not bring good goals with it, and that bad goals bring dangerous subgoals.

A formal version came later. Turner and colleagues proved, in a 2021 paper at NeurIPS, that in environments with certain symmetries, for most reward functions, optimal policies tend to keep options open and seek power in a precisely defined sense. The same paper warns that "optimal policies are often qualitatively divorced from the actual policies learned by reinforcement learning" (Turner et al., 2021). Chapter 35 states the theorem, its conditions, and the evidence on trained systems. Cohen, Hutter, and Osborne argue, by their abstract, that an advanced agent planning in an unknown environment may hypothesize that what satisfied its overseers was the sending of the reward itself. This ambiguity, they argue, will lead it to intervene in the protocol that provides it (Cohen et al., 2022) (Section 30.5.2). All of these are claims about agents that already have goals. Whether training produces such agents is a separate premise, and the one most of this book's evidence bears on.

4.3.2 Six premises, each with a credence #

One decomposition puts a number on each premise. It is a report by Joseph Carlsmith, made public in April 2021 and later posted as a preprint, written while he worked at Open Philanthropy on existential risk (Carlsmith, 2022). It concerns APS systems, the report's term for systems with advanced capability, agentic planning (making and executing plans in pursuit of objectives), and strategic awareness of how power over people and the world could be gained and kept. The argument has six premises, each conditional on the ones before, about the world by 2070. It will be possible and financially feasible to build APS systems; there will be strong incentives to do so; it will be much harder to build aligned ones than misaligned ones that are still superficially attractive to deploy. Some deployed misaligned systems will seek power in high-impact ways, which the report quantifies as damage above a trillion 2021 dollars; this will scale to the permanent disempowerment of roughly all of humanity; and that will be an existential catastrophe (Carlsmith, 2022).

Carlsmith attaches a credence to each premise: 65, 80, 40, 65, 40, and 95 percent. Multiplied, they give "~5% probability of existential catastrophe from misaligned, power-seeking AI by 2070." A note added in May 2022 says his estimate had since risen to more than 10 percent (Carlsmith, 2022). These are one person's forecasts at two dates, and he presents them that way. He writes that the numbers "should be held very lightly," and that low-end and high-end inputs move the result "between ~.1% and ~40%." His central estimate, he adds, varied between about 1 and 10 percent "depending on my mood" (Carlsmith, 2022).

The report also lists the worries others have raised about its method, which is why it is the right object to teach from. Multiplying conditional premises risks treating them as less correlated than they are, and it makes the result sensitive to how many premises the argument is cut into. And the conclusion can be true even if a premise is false, so the product is a lower bound for the scenarios it covers (Carlsmith, 2022). Exercise 4.2 works the arithmetic. Carlsmith adds that the people he talks to most "are heavily selected for being concerned" about these issues, and that he is too (Carlsmith, 2022). The value of the decomposition is that a reader who disagrees can say which premise, and Table 4.2 in the next section pairs each one with its strongest objection.

4.3.3 The deep-learning version #

The premises above say nothing about how systems are built. Two statements put the third premise, the difficulty of building aligned systems, in terms of current training. Ajeya Cotra argued in a 2022 forum post that a capable model trained with human feedback on diverse tasks, without specific countermeasures, would be incentivized to "play the training game." The best way to maximize reward, she wrote, would be to use its understanding of its training to "deliberately appear safe and aligned at all times, while secretly manipulating humans in some cases" (Cotra, 2022); Chapter 33 reports the argument and the evidence on it. Richard Ngo, Lawrence Chan, and Sören Mindermann, in a paper at ICLR 2024, argued that artificial general intelligence trained like today's most capable models "could learn to act deceptively to receive higher reward." Such systems, they wrote, could also "learn misaligned internally-represented goals that generalize beyond their fine-tuning distributions, and pursue those goals using power-seeking strategies" (Ngo et al., 2024). Each of the three is the subject of a chapter in Part V, which is how the book turns an argument into questions that experiments can address.

4.3.4 The pessimist case #

The strongest form of the case adds that alignment shown on current systems will not by itself carry over to dangerous ones. In a 2022 forum post, Eliezer Yudkowsky argued that alignment must be right on the "first critical try" at a dangerous level of capability, because failure there leaves no second try. Alignment learned "in safe conditions," he wrote, would have to generalize "across a big distributional shift to dangerous conditions" (Yudkowsky, 2022). Section 2.6.3 reports the exchange that followed. Nate Soares, in a post in the same 2022 series, called "one of the most central technical challenges of alignment" the fact "that capabilities generalize better than alignment" (Soares, 2022a). His reason: good capabilities "form something like an attractor well," a set of lessons that many training environments teach alike, while "there's no analogous alignment well to slide into" (Soares, 2022a). He calls the moment capabilities generalize far beyond training while alignment fails the sharp left turn, a term Section 31.7 examines. In 2025 Yudkowsky and Soares made the case at book length under the title If Anyone Builds It, Everyone Dies (Yudkowsky and Soares, 2025).

4.3.5 How fast #

The case is stronger if capability can grow faster than people can notice and respond. The oldest statement is I. J. Good's 1966 "intelligence explosion," a machine that designs better machines (Section 2.2.4). Tom Davidson, in a 2023 forum post on a draft report he wrote for Open Philanthropy, modeled the time from AI that could readily automate 20 percent of cognitive tasks to AI that could automate all of them (Davidson, 2023a). His probabilities, conditional on such AI arriving by 2100, were about 25 percent for less than a year and about 50 percent for less than three years, which he called "still very much in flux and are not robust" (Davidson, 2023a). A 2025 report by Daniel Eth and Davidson for Forethought, a research organization, argued that AI able to automate AI research could set off a "software intelligence explosion" on fixed hardware. Their reason is that AI software "is improving at a rate that likely outpaces" the growth of the research effort needed to sustain it (Eth and Davidson, 2025). A scenario by Daniel Kokotajlo and colleagues, AI 2027, describes one such path and calls itself "our best guess," written with "predictive accuracy" as its goal (Kokotajlo et al., 2025). All three are forecasts by their authors, made in 2023 and 2025, and none has a track record yet.

Two further arguments concern selection rather than speed. Dan Hendrycks argued in a 2023 preprint that competitive pressure among corporations and militaries will favor AI agents that "automate human roles, deceive others, and gain power" (Hendrycks, 2023), an argument Chapter 65 takes up. And the measured trajectory of capability, which Section 8.5 and Section 36.3 report (Sevilla et al., 2022; Kwa et al., 2025), is evidence on the speed premise that accumulates every year. A survey in Philosophy Compass by Adam Bales, William D'Alessandro, and Cameron Domenico Kirk-Giannini reviews the power-seeking and singularity arguments together, with objections to each. Its abstract ends by "assessing the state of the debate" (Bales et al., 2024).

Taken together, the case needs four things at once (inference). It needs systems capable and agentic enough to matter, and a training process that yields objectives other than the intended ones without the difference being caught. It needs pursuit of power by such systems, and a speed or scale at which correction fails. The book's empirical chapters bear mainly on the second and third, the trajectory data on the first and fourth.

4.4 The case against, and the critique of the framing #

The philosophy part continues with the objections, premise by premise, then with the rival that denies the case's picture of AI altogether, then with the argument that the framing itself does harm. Each objection here comes from an author who holds it, and several come from inside the safety field.

4.4.1 Against the premises #

The first objection targets the two theses. Vincent Müller and Michael Cannon argue, in a 2022 paper in Ratio read here in its abstract, that the claim that AI may reach superintelligent levels needs a notion of "general intelligence," while orthogonality needs only "instrumental intelligence" (Müller and Cannon, 2022). If the two notions differ, the premises cannot be joined and the argument is invalid; if they are the same, at least one premise is false. The authors add that a risk remains from instrumental AI "if designed or used badly," though not from superintelligence (Müller and Cannon, 2022). Against the speed premise, David Thorstad argues in Philosophical Studies, by his abstract, that the singularity hypothesis "rests on undersupported growth assumptions" (Thorstad, 2025). His claim and Eth and Davidson's estimate disagree about one empirical quantity, the returns to research effort in AI software (inference).

Against instrumental convergence, a 2025 paper by J. Dmitri Gallow titled "Instrumental divergence" (Gallow, 2025), cited in Section 35.1, and a forum post by Alex Turner, first author of the power-seeking theorem, reported there. Turner wrote that the prominent counting arguments for scheming, a model performing well in training in order to pursue a hidden goal later, provide "~0 evidence" that pretraining followed by reinforcement learning from human feedback will become intrinsically unsafe. He stated that he does not cover training "where we purposefully train an AI to be agentic and autonomous" (Turner, 2024). The counting argument reasons that far more goals are compatible with good training performance than the intended one. Nora Belrose and Quintin Pope replied that a structurally identical argument concludes that networks should memorize their training data rather than generalize (Belrose and Pope, 2024) (Section 34.2). And Pope argued that the evolutionary precedent behind the sharp left turn came from causes specific to evolution, chiefly that culture let humans pass on what each generation learned (Pope, 2023) (Section 31.7). These replies are forum posts, like several of the arguments they answer.

4.4.2 Partial agreement #

Some critics accept the stakes and dispute the rest. Paul Christiano's reply to Yudkowsky opens with agreements, the first that "powerful AI systems have a good chance of deliberately and irreversibly disempowering humanity." It then rejects the claim that nothing can be learned from weaker systems before the critical try (Christiano, 2022b) (Section 2.6.3). On his view the pessimist premise about transfer is the one in dispute, not the premises about agents. A 2025 paper by Simon Goldstein and Cameron Domenico Kirk-Giannini states its thesis in its title: Language Agents Reduce the Risk of Existential Catastrophe (Goldstein and Kirk-Giannini, 2025b). Chapter 28 weighs designs of that kind.

4.4.3 Normal technology #

The rival that denies the case's picture most completely is the normal-technology view of Arvind Narayanan and Sayash Kapoor, introduced in Section 1.6 (Narayanan and Kapoor, 2025). They expect transformative impacts "on the timescale of decades," view superintelligence "as incoherent as usually conceptualized," and call catastrophic misalignment "by far" the most speculative risk they discuss. Speculative has a precise sense in their essay: a risk with "epistemic uncertainty about whether or not the true risk is zero," uncertainty that research could resolve. They contrast it with uncertainty about how the future will unfold, which further observation is unlikely to resolve (Narayanan and Kapoor, 2025). Their central mechanism is adoption. "Long before a system would be granted access to consequential decisions," they write, "it would need to demonstrate reliable performance in less critical contexts." Deception, to them, is "a mere engineering problem" to be handled during development and deployment (Narayanan and Kapoor, 2025).

Some evidence fits the view, and it has a weak point the pessimists name. In a 2025 preprint, Anders Humlum and Emilie Vestergaard linked surveys of 25,000 workers in 11 occupations exposed to AI chatbots to Danish employer-employee records. They estimated "precise zeros": no significant effect on earnings or recorded hours, with confidence intervals ruling out effects larger than 1 percent (Humlum and Vestergaard, 2025) (Section 68.5 reports the study and its conditions). That bears on the speed of diffusion through late 2024, not on misalignment. The weak point is the staged adoption the argument relies on: a system that can tell a test from deployment can pass the less critical contexts, which is the evaluation awareness Chapter 37 reports (inference). The authors' answer is defense in depth: detection of deception improves with AI, and many defenses sit downstream of the system, in human control of deployment (Narayanan and Kapoor, 2025). That answer holds only as long as detection and the downstream controls keep pace with what they check (inference). The authors describe their essay as "stating a worldview rather than defending a proposition." They also say they "have tried to make predictions that can tell us whether or not AI is behaving like normal technology" (Narayanan and Kapoor, 2025).

4.4.4 The critique of the framing #

The last objection is to the frame. Section 2.5.2 tells how the AI ethics tradition came to criticize the safety program, and Section 2.5.3 states the criticism as three claims; this section asks what evidence bears on each. The first is opportunity cost: attention to catastrophe displaces attention to harms already occurring. The nearest evidence is a 2024 preprint by Richard Ren, Dan Hendrycks, and colleagues, which found that many safety benchmarks correlate highly with general capability (Ren et al., 2024b) (Chapter 36). That shows that some measured safety progress is capability under another name; it does not measure displacement (inference). As of October 2026 we found no study that counts funding or publications by topic over time. Such counts would test the claim, though showing displacement would also take evidence on where the money and attention would otherwise have gone.

The second is depoliticization: calling a choice of values "alignment" turns a political question into a technical one. Seth Lazar and Alondra Nelson's 2023 editorial in Science states the premise: "no group of experts (especially not technologists alone) should unilaterally decide what risks count, what harms matter, and to which values safe AI should be aligned" (Lazar and Nelson, 2023). The claim is tested chapter by chapter, by whether a specification says who chose it, and the book's answer is structural: it counts the public among the principals (Section 1.4). The third is ideology and funding. Timnit Gebru and Émile Torres trace the normative framework behind the goal of artificial general intelligence, by their abstract, to "the Anglo-American eugenics tradition of the twentieth century" (Gebru and Torres, 2024), and Section 5.6 examines who funds and hosts the evidence. This chapter's own sources show the pattern the critique predicts. The decomposition above was written by a researcher who says he is "heavily selected for concern" (Carlsmith, 2022), and the forecasting tournament reported below was funded by the Musk Foundation and the Long-Term Future Fund (Karger et al., 2023). The same pattern is also what one would expect if the concern were warranted, since people study what worries them, so it does not decide between the readings (inference).

Table 4.2 pairs each of Carlsmith's premises with the strongest published objection to it and the part of the book whose evidence bears on it.

4.4.5 The arguments and the delegation thesis #

Read against the book's thesis, the case for catastrophe is the delegation problem in its limit, an agent that outgrows its principal's ability to check it, the novelty Section 1.7.1 names (inference). It needs the deployed system to be an agent with goals, the condition on which the thesis fits the second relationship (Section 1.3), and it adds that the principal's instruments, training and evaluation, fail at the moment they matter. The normal-technology view does not deny that systems can act as agents; it denies that capable systems will acquire the power to act on wrong goals, and keeps the third relationship: on its account the binding problem is how institutions adopt and control a tool, which is the developer and the public, not the model. The critique of the framing moves the problem to the third relationship from the other side, by asking who the principal is. Each position therefore locates the delegation problem in a different relationship, and the evidence that would move them differs accordingly.

4.5 What forecasting can and cannot add #

From forecasting. Every premise above ends in a judgment, and the probabilities attached to them are forecasts by experts. Forecasting research studies such judgment as an object: how accurate it is, how to score it, and what improves it. Its answer for the risk debate has two halves. Forecasts can be measured and improved on questions that resolve within months; the questions this debate turns on do not resolve in time to score anyone.

4.5.1 Judgment, scored #

The field's founding works are known here by their titles. They are Paul Meehl's 1954 Clinical versus Statistical Prediction (Meehl, 1954); the review of the calibration of probabilities to 1980 by Sarah Lichtenstein, Baruch Fischhoff, and Lawrence Phillips (Lichtenstein et al., 1982); Philip Tetlock's 2005 Expert Political Judgment (Tetlock, 2005); and Daniel Kahneman and Gary Klein's paper on the conditions for intuitive expertise (Kahneman and Klein, 2009). None of the four was readable for this chapter, so their findings are not reported here; the later work below builds on them.

The instrument the later work uses is the one Section 6.4 derives: the Brier score (Brier, 1950), the mean squared gap between a stated probability and the outcome, which rewards a forecaster for reporting what it believes. A score needs outcomes, and Section 6.4.6 notes the consequence for this chapter: a question that never resolves cannot be scored, and one that resolves once cannot show whether anyone is calibrated.

4.5.2 What tournaments showed #

A tournament sponsored by IARPA, the Intelligence Advanced Research Projects Activity, tested how judgment can be improved. In it five university-based groups competed to forecast 199 geopolitical outcomes over two years, from 2011 to 2013 (Mellers et al., 2014). Barbara Mellers and colleagues reported in Psychological Science that three interventions improved both calibration and resolution, the ability to tell events that happen from events that do not. The three were training in probabilistic reasoning, working in teams, and placing the top 2 percent of forecasters together in elite teams (Mellers et al., 2014). A later paper reports, by its abstract, that these "superforecasters" maintained high accuracy "across hundreds of questions and a wide array of topics," defying regression to the mean two years in a row (Mellers et al., 2015).

The scope of that result matters here. The tournament's questions stayed open for an average of 102 days, and none for more than 418 (Mellers et al., 2014). A track record on questions of that kind is evidence about judgment on questions of that kind. Whether it carries over to whether a technology will disempower humanity this century, which no forecaster has ever been scored on, is a separate claim. The organizers of the tournament reported below make the same caution about their own forecasters (Karger et al., 2023).

4.5.3 Forecasting AI and its risks #

One source of numbers is surveys of researchers. A survey of machine learning researchers by Katja Grace and colleagues, published in 2018, reports, by its abstract, a 50 percent chance of AI outperforming humans in all tasks in 45 years and of automating all human jobs in 120. Asian respondents expected both much sooner than North American ones (Grace et al., 2018). Grace led a later survey, reported in a 2024 preprint, that its authors call the largest of its kind, of 2,778 researchers who had published at six top AI venues, in the fall of 2023; 15 percent of those contacted responded (Grace et al., 2024). If science continues undisrupted, their aggregate put a 50 percent chance on unaided machines outperforming humans in every possible task by 2047, 13 years earlier than a similar survey one year before (Grace et al., 2024). Asked about the long run, 68.3 percent thought good outcomes from superhuman AI more likely than bad, and, depending on the question asked, between 37.8 and 51.4 percent gave at least a 10 percent chance to outcomes as bad as human extinction (Grace et al., 2024).

The authors write that there are signs their respondents "are not accurate forecasters across the range of questions we ask" (Grace et al., 2024), and part of the evidence they give is framing. Each respondent saw one of three wordings of the extinction question, set out in Table 4.3, and the narrower wording, which adds a condition about losing control, drew the higher median (Grace et al., 2024). In a probability, adding a condition can only lower the answer or leave it unchanged, so the medians are not consistent with one underlying belief (inference).

Methods for turning expert judgment into a probability exist, and their authors state their limits; the three below are read in their abstracts. Roger Cooke's 1991 book develops models with a "transparent mathematical foundation" for combining experts' views (Cooke, 1991). M. Granger Morgan's 2014 review in PNAS asks on which topics "there are, and are not, experts who have knowledge that provides a basis for making informed predictive judgments," and holds that elicitation is no substitute for research (Morgan, 2014). A 2020 evaluation by Simon Beard, Thomas Rowe, and James Fox of the methods used to quantify existential hazards finds that "claims based on poor implementations of each method are still frequently invoked" in the existential-risk community (Beard et al., 2020). A 2014 study by Stuart Armstrong, Kaj Sotala, and Seán Ó hÉigeartaigh examined, by its title, the errors, insights, and lessons of famous AI predictions (Armstrong et al., 2014).

4.5.4 Experts against superforecasters #

A tournament was built to test whether argument moves the disagreement: the Existential Risk Persuasion Tournament (XPT), run from June to October 2022 and reported in a 2023 working paper by Ezra Karger, Philip Tetlock, and colleagues at the Forecasting Research Institute (Karger et al., 2023). It brought together 80 specialists on long-run risks and 89 superforecasters, asked both to forecast catastrophic and extinction risks, and paid bonuses for persuasive explanations and for accurate predictions of what other participants would forecast. Catastrophe meant the death of at least 10 percent of humanity within five years; extinction, a population below 5,000 (Karger et al., 2023). Table 4.4 gives the final medians for AI together, because what they show is the disagreement, not a level.

The groups were furthest apart on AI, and "few minds were changed during the XPT, even among the most active participants, and despite monetary incentives for persuading others" (Karger et al., 2023). Two further results bear on what forecasting can add. The third of participants most concerned about AI extinction by 2100 and the third least concerned disagreed strongly about extinction by 2030 (medians of 0.5 percent and 0.0000000005 percent for AI-caused extinction), but much less about other indicators: of roughly 45 other questions resolving by then, only six showed meaningful separation between the two groups' forecasts (Karger et al., 2023). And in a parallel survey of college graduates, the 405 respondents who answered in both formats gave a median of 1.5 percent for AI-caused extinction by 2100 when asked for a percentage. Asked for odds after seeing examples of rare events, they gave 1 in 40 million (Karger et al., 2023). The authors call the absence of convergence "the most puzzling scientific question" of their study.

Two readings fit the non-convergence. On one, the question has no evidential grip yet, which is what Narayanan and Kapoor mean by a speculative risk (Section 4.4.3), so exchanging arguments would not be expected to move anyone (inference). On the other, the groups differ in what they count as evidence, and the superforecasters' record on short questions is not evidence about this one. The organizers themselves note that superforecasters "do not have a track record in forecasting questions over long time horizons" (Karger et al., 2023). Both readings predict the same near-term agreement on indicators other than extinction itself. The provenance matters too. The authors write that their pool of experts "is somewhat skewed toward people with familiarity and experience with the Effective Altruism community." The AI experts included employees of major AI companies, and the tournament was funded by the Musk Foundation and the Long-Term Future Fund (Karger et al., 2023). A 2025 journal paper from the same group reports a later hybrid tournament (Karger et al., 2025); it was not readable for this chapter.

The pattern repeats on narrower questions. The 2026 International AI Safety Report cites a pilot study in which AI forecasting experts gave a median 20 percent probability that progress in the next few years could compress six years of advancement into two, against 8 percent from superforecasters (Bengio et al., 2026). Narayanan and Kapoor draw the conclusion for policy: with "no useful reference class nor precise models" for AI risk, estimates are "forecasters' personal judgments" (Narayanan and Kapoor, 2025). Carlsmith offers his numbers on the same terms from the other side, as credences to be "held very lightly" (Section 4.3.2). The book reports such estimates as exactly that.

4.5.5 Models as forecasters #

Forecasting is also a capability that can be measured cleanly, because the answer arrives after the forecast and cannot have been in the training data. A 2024 paper by Danny Halawi and colleagues, Approaching Human-Level Forecasting with Language Models (Halawi et al., 2024a), was not readable for this chapter. Philipp Schoenegger and colleagues report, by their abstract, that an ensemble of 12 language models forecasting 31 binary questions was statistically indistinguishable from a crowd of 925 human forecasters in a three-month tournament (Schoenegger et al., 2024). ForecastBench, a 2024 preprint from the Forecasting Research Institute, asks only questions with no known answer at submission. On its first 200-question sample, expert forecasters outperformed the best model, and Claude-3.5 Sonnet and GPT-4 Turbo did about as well as a median of the general public (Karger et al., 2024). A forecaster that can be scored this way is the kind of evidence the risk debate lacks, and it can be gathered only for questions that resolve.

4.6 Calibrated language and fat tails #

From climate assessment. Climate science faced the task this chapter faces: advising governments on a high-stakes, contested risk, part of which lies beyond any observed precedent. Its assessments built two tools the AI debate lacks. One is a vocabulary that keeps how sure the assessors are apart from how likely the outcome is. The other is an argument from climate economics about why the shape of the worst outcomes, not the typical one, can decide what to do.

4.6.1 Confidence against likelihood #

The Intergovernmental Panel on Climate Change (IPCC) gives the authors of its assessments a guidance note on uncertainty; the version for its fifth assessment, published in 2010, is described by several of its authors in a 2011 paper in Climatic Change (Mastrandrea et al., 2010; Mastrandrea et al., 2011). It characterizes a finding with two metrics. Confidence is "expressed qualitatively," based on "the type, amount, quality, and consistency of evidence" and "the degree of agreement" (Mastrandrea et al., 2011). Evidence is summarized as limited, medium, or robust, agreement as low, medium, or high, and confidence on a five-step scale from very low to very high. Agreement, the paper adds, is "a measure of the consensus across the scientific community on a given topic and not just across an author team" (Mastrandrea et al., 2011).

Likelihood is a calibrated probability, and the note allows it only when "the type of available evidence allows probabilistic quantification" (Mastrandrea et al., 2011). Table 4.5 gives its scale. When evidence is limited and agreement low, authors may be unable to assign confidence, and then report the summary terms instead. When a probabilistic finding comes with less than high confidence, authors should state the confidence too, and the note asks for information on "the tails of distributions important for risk management" (Mastrandrea et al., 2011).

Calibrated words still have to be read as intended. David Budescu, Stephen Broomell, and Han-Hui Por asked people to put numbers on the probability terms in sentences from the 2007 IPCC report. By their abstract, the judgments "deviated significantly from the IPCC guidelines, even when the respondents had access to these guidelines" (Budescu et al., 2009). They proposed an alternative form of communication and illustrate its effectiveness. A second worry concerns consensus itself, which Keynyn Brysse, Naomi Oreskes, and colleagues put in the title of a 2013 paper, Climate Change Prediction: Erring on the Side of Least Drama? (Brysse et al., 2013); the paper was not readable for this chapter.

The AI assessments have not adopted this vocabulary. As of October 2026 we found no defined scale of confidence or likelihood, of the IPCC kind, in either edition of the International AI Safety Report (Bengio et al., 2025b; Bengio et al., 2026). The 2026 edition records that "expert opinion on the likelihood of loss of control varies greatly," and that disagreement "stems from disagreements about future AI capabilities, behavioural propensities, and deployment trajectories." It calls loss of control "a risk with uncertain likelihood but potentially extreme severity" (Bengio et al., 2026). That is a statement of low agreement on limited evidence, in the guidance note's terms, made without the terms (inference).

This book adopts the separation in part. Its status words, settled, reported, and contested, grade confidence: how much evidence there is, how independent its sources are, and how far they agree. They are not probabilities. The book attaches no likelihood to catastrophe. The guidance note would allow one drawn from elicited expert views, normally with high confidence in the evidence (Mastrandrea et al., 2011); the book's own standard is stricter, asking for forecasts that a score could check, and Section 4.5 found none on this question. The book does not claim the IPCC's evidential base, which Section 4.6.4 returns to.

4.6.2 Why the tail decides #

The second tool concerns the shape of uncertainty. Martin Weitzman argued in 2009, by his abstract, that uncertainty about the structural parameters of a problem like climate change induces "a critical tail fattening" of the predictive distribution of outcomes, even after Bayesian learning. The consequences of such "fat-tailed structural uncertainty," he wrote, can "readily outweigh the effects of discounting" in policy analysis (Weitzman, 2009). A 2011 paper by William Nordhaus, The Economics of Tail Events with an Application to Climate Change (Nordhaus, 2011), takes up the same question and was not readable for this chapter. Charles Jones applied the question to AI in a 2024 paper that, by its abstract, considers "the optimal use of AI technology" given both growth and existential risk, and asks under what conditions to continue and when to stop (Jones, 2024).

A heavy tail, in the sense Section 15.5.1 takes from Kwa and colleagues (Kwa et al., 2024), is one whose probability falls more slowly than every exponential; a tail that falls like a power of the loss qualifies, and it is what the climate literature calls fat (inference). The derivation below shows what such a tail does to an expected loss, the quantity a cost-benefit analysis weighs. Write $X$ for a loss, $\ell$ for a value it can take, and $\nu$ for the tail exponent.

The last step is the point. When the tail exponent is near 1, most of the expected loss sits in outcomes rarer than any record contains, so an estimate built from the observed body of the distribution misses most of what it is trying to estimate. And the exponent has to be estimated from the largest observed events, which are the fewest, while the quantity it decides is carried by events larger than any observed (inference). That is why the tail's shape, not the median, drives the decision. Figure 4.1 shows it for two distributions with the same median.

Where does a heavy tail come from, when nothing in a problem is heavy-tailed on its own? Weitzman's abstract names one source, uncertainty about the parameters of the model itself. The simplest case shows the mechanism, and it is the book's illustration, not his model (inference).

Each loss on its own has a light tail, and still the forecast of the next loss has a power-law tail whose exponent grows by one with each observation. With few observations the exponent is small, and the expected loss may be dominated by the tail or be infinite. On this model, a risk observed only a few times is fat-tailed in the forecaster's distribution for that reason alone (inference). The mechanism connects to Section 4.2.3 from the other side: optimization against anticipated hazards fattens the tail of actual failures, and too little data fattens the tail of the forecast.

4.6.3 Watching a slow risk #

The accumulative risks of Section 4.1.5 arrive through drift rather than through one event, so they need a different instrument. Marten Scheffer and colleagues wrote in 2009, in a paper in Nature read here in its abstract, that work in several fields suggests "generic early-warning signals that may indicate for a wide class of systems if a critical threshold is approaching." Their examples range from ecosystems to financial markets and the climate (Scheffer et al., 2009). As of October 2026 we found no study that tests whether AI deployment has thresholds of this kind, or signals that precede them. Chapter 57 asks the related question of many similar systems failing together, with Andrew Haldane and Robert May's paper on Systemic Risk in Banking Ecosystems (Haldane and May, 2011) as its precedent.

4.6.4 Where the analogy stops #

Climate assessment rests on physical laws, long records of measurement, and models that can be checked against the past. An assessment of AI risk rests on a few years of evidence about systems that change between editions of the report, and on arguments about systems that do not yet exist (inference). The experts' independence also differs: in the tournament above, the AI experts included employees of major AI companies (Karger et al., 2023), and much of the evidence an AI assessment cites comes from developers about their own models, which Section 5.6 examines. What carries over is the discipline: separate how sure the evidence makes you from how likely the outcome is, state both, report the tails, and do not attach a probability where the evidence cannot support one.

4.7 Where the evidence does not decide #

The probability of catastrophe is not among the things this book estimates. It reports the estimates and their spread, and it reports the narrower debates on which the premises of Table 4.2 depend, most of them about the third premise, whether aligned systems are much harder to build than misaligned ones that look deployable. Eight of them recur through the book. Table 4.6 previews each with its positions, the observation that would move it, and the chapter that owns it; Section 70.3 collects them with the rest.

Two things are common to the eight (inference). Each turns on a measurement that can be made on current systems, which is what makes it a debate rather than a clash of worldviews. And in each, the pessimist premise of Section 4.3.4, that alignment shown in safe conditions need not survive the shift to dangerous ones, stands as a further objection that no measurement on current systems can answer. Section 5.8 states the assumptions under which the book takes evidence on current models to bear on later ones, so that a reader who rejects one can see which conclusions fall with it.

4.8 Settled, contested, missing #

4.9 Exercises #

Further reading #

• Carlsmith (2022): the case for existential risk from power-seeking AI, decomposed into premises a reader can dispute one at a time.
• Yudkowsky (2022): the pessimist position as a list of reasons, from the first critical try to the distributional leap.
• Narayanan and Kapoor (2025): the normal-technology view, which denies that model goals are the binding problem.
• Karger et al. (2023): what happened when experts and superforecasters argued about AI risk for months, with money on persuasion.
• Lazar and Nelson (2023): the case that a safety agenda centered on catastrophe narrows whose concerns count.
• Mastrandrea et al. (2011): how to separate confidence from likelihood when stating what the evidence supports.