Draft complete but not yet reviewed; expect revisions

Regulation, Liability, and Standards

In brief

Ideas

As of October 2026, regulation leaves the developer's knowledge advantage in place: enforcement runs mostly on developer-supplied information, and no instrument lets a state stop a training run (§63.3.2).
Harms too large to pay or too diffuse for suit favor rules in advance, while the developer's greater knowledge favors leaving precautions to it; the statutes regulate its own risk plan (§63.1.3).
Insurers can regulate care only if they can see it, and correlated losses across one model's deployments leave a variance per policy that no larger pool averages away (§63.5.3).
The release decision sets who can misuse a model and who can check it, and none of the instruments of October 2026 weighs the two (§63.7.3).

Evidence

Settled: as mathematics within its model, strict liability controls how often an injurer acts and negligence does not; an injurer unable to pay the harm escapes strict liability, and negligence if its assets are low enough (§63.1.1).
Reported: the UK's AI Security Institute reports finding vulnerabilities in the safeguards of every system it tested (§63.2.5).
Contested: whether compute thresholds are a workable trigger for duties; both sides treat compute as a filter, not a measure of risk, and disagree on how long it stays a good one (§63.2.6).

Open

Crux: can a state evaluate frontier models without relying on developers for access, data, and staff? Evaluators finding and acting on something undisclosed, under access developers did not control, would settle it (§63.3.1).
Missing: as of October 2026, any enforcement action under California's or New York's frontier statutes or the AI Act's rules for general-purpose models, and loss data to price AI catastrophe risk (§63.9).

The previous chapter showed developers writing rules for themselves (Chapter 62). This chapter asks what states do about it. A regulator faces the problem this book keeps returning to, in its oldest institutional form. The firms it regulates know more about their products' risks than it does, the harms it worries about are rare and may be catastrophic, and the evidence it would need comes mostly from the firms themselves. Law has several answers. It can let harm happen and make the firm pay; it can require approval before a product is sold; it can require the firm to manage risk by a plan the state reviews; it can recruit insurers, courts, and outside researchers to watch on its behalf.

The chapter starts with the economics of when each answer fits, then reads the statutes and executive orders in force as of October 2026 for how they are built, with their provisions in Section I.1, and asks what regulatory theory predicts about them. Two sections follow on regulators that are not agencies, courts and insurers, and one on the industry most proposals invoke as a model, drugs and medical devices. The last section treats the release of model weights as a regulatory decision that sets both how much misuse is possible and how much outside verification is. Nearly every claim concerns the third of the book's relationships, the developer and the public; liability for harm to users also concerns the second, the deployed system and the people it serves. Compute governance and the design of international agreements are Chapter 66's; disclosure mandates and the auditing profession are Chapter 67's.

63.1 Liability or rules in advance #

From economics. A state that wants a dangerous activity done carefully has two broad instruments. It can wait for harm and make the injurer pay, or it can set rules that bind before any harm occurs, which is ex ante regulation: a standard, a license, a required test. Paying after harm comes in two forms. Under strict liability the injurer pays for the harm it causes whether or not it was careless. Under negligence it pays only if it failed to take due care, the level of precaution a court judges reasonable. The economic analysis of law asks which instrument best balances the value of the activity, the cost of precautions, and the harm that remains. Guido Calabresi's 1970 book carries the program in its title, The Costs of Accidents: A Legal and Economic Analysis; this book could read only its catalog record (Calabresi, 1970).

63.1.1 What strict liability controls that negligence does not #

The two forms of liability differ in a respect that is easy to miss. An injurer chooses two things: how careful to be each time it acts, and how much it acts. A court applying negligence can check the first, since due care is a standard of conduct, but it rarely asks whether the activity should have been done so often. Steven Shavell's 1980 paper "Strict Liability versus Negligence" compares the two rules (Shavell, 1980). Only its catalog record could be read, so the derivation below is this book's own statement of the model.

Derivation Care, activity, and an injurer that cannot pay

An injurer chooses its care $e$ , measured as the money it spends on precaution each time it acts, and its activity level $v$ , how often it acts. Acting $v$ times is worth $U(v)$ to it, with $U$ increasing and concave (each extra unit of activity is worth less than the last). Each act causes an accident with probability $p_{\text{acc}}(e)$ , decreasing and convex in care (each extra dollar of care removes less risk than the last), and an accident does harm $\mathrm{Harm}$ to someone else.

The social optimum. Welfare is the activity's value minus the cost of care and the expected harm, $S(e, v) = U(v) - v\,[\,e + p_{\text{acc}}(e)\,\mathrm{Harm}\,]$ . The bracket is the full cost of one act. The best care $e^\ast$ minimizes it, so $-p_{\text{acc}}'(e^\ast)\,\mathrm{Harm} = 1$ : a last dollar of care saves a dollar of expected harm. The best activity level $v^\ast$ sets the value of one more act equal to its full cost, $U'(v^\ast) = e^\ast + p_{\text{acc}}(e^\ast)\,\mathrm{Harm}$ .
Strict liability. The injurer pays $\mathrm{Harm}$ for every accident, so its payoff is exactly $S(e, v)$ . It chooses $e^\ast$ and $v^\ast$ .
Negligence. The court sets due care at $e^\ast$ and makes the injurer pay only if its care fell short. Taking due care costs less than paying for every accident, so the injurer takes $e^\ast$ and then pays nothing for the accidents that still happen. Its payoff is $U(v) - v\,e^\ast$ , and it chooses $v_{\mathrm{N}}$ with $U'(v_{\mathrm{N}}) = e^\ast$ . Since $U'$ falls as $v$ rises and $e^\ast$ is less than the full cost, $v_{\mathrm{N}} > v^\ast$ : the injurer is careful each time and acts too often, because the residual harm $v\,p_{\text{acc}}(e^\ast)\,\mathrm{Harm}$ falls on others.
An injurer that cannot pay. Let the injurer's assets be $W < \mathrm{Harm}$ , so that strict liability takes at most $W$ per accident. Its payoff becomes $U(v) - v\,[\,e + p_{\text{acc}}(e)\,W\,]$ , as if the harm were $W$ . It chooses care $e_W$ with $-p_{\text{acc}}'(e_W)\,W = 1$ , which is below $e^\ast$ because the harm it faces is smaller, and a per-act cost below the true one, so its activity $v_W$ exceeds $v^\ast$ . Strict liability now under-deters on both margins. Under negligence the injurer still takes due care while $e^\ast$ costs less than its cheapest negligent choice, $\min_e\,[\,e + p_{\text{acc}}(e)\,W\,]$ , and that fails once $W$ is small enough beside $\mathrm{Harm}$ .
A worked case. With $p_{\text{acc}}(e) = \exp(-e)$ and $U(v) = 2\sqrt{v}$ , step 1 gives $e^\ast = \log \mathrm{Harm}$ and $v^\ast = 1/(\log \mathrm{Harm} + 1)^2$ ; negligence gives $v_{\mathrm{N}} = 1/(\log \mathrm{Harm})^2$ ; and the injurer with assets $W$ chooses $e_W = \log W$ and $v_W = 1/(\log W + 1)^2$ (for $\mathrm{Harm}, W > 1$ ). The smaller $W$ is beside $\mathrm{Harm}$ , the further care falls and activity rises. Under negligence the same injurer takes due care while $\log \mathrm{Harm} \le \log W + 1$ , that is, while $W \ge \mathrm{Harm}/\exp(1)$ .

Step 4 is the step that matters for frontier development. Liability deters only up to what the injurer can pay, and a harm larger than its assets is, to the injurer, a harm the size of its assets (Exercise 63.1 works the numbers). Strict liability then controls neither care nor activity, and negligence, which never controlled activity, keeps control of care only while the assets are not too far below the harm. That is one of the reasons the next paper gives for rules in advance.

63.1.2 When rules in advance do better #

Shavell's 1984 paper, read here in its 1983 working-paper version, asks when society should rely on liability and when on regulation of safety (Shavell, 1984). It names four determinants. The first is a difference in knowledge: where private parties know more about the risks and the cost of reducing them, letting them decide, under liability, is better; where the authority knows more, regulation is. The second is that parties may be unable to pay for the full harm: "liability exceeding a party's assets would be seen by him only as liability equal to his assets." The third is that parties may not face suit, because harm is dispersed over many victims, appears long after the act, or is hard to attribute to anyone. The fourth is administrative cost, which under liability is incurred only when harm occurs (Shavell, 1984).

Shavell then reads the record. Activities like felling a tree in one's yard are left to liability, because the party knows far more than any authority about its own tree and harm is rare and easy to trace. Activities that are much regulated, such as fire safety, food and drugs, pollutants, and the transport of explosives, are those where "the two affirmative arguments for regulation" outweigh the regulator's lack of information and its administrative cost (Shavell, 1984). Two of his finer points matter below. Verifying compliance is cheaper when a rule requires "the presence of particular things," such as guard rails or lifeboats, than when it requires modes of behavior, and cheaper when behavior is hard to conceal. And the two instruments should be used together: complying with a regulation should not relieve a party of liability, since some parties ought to do more than the regulation demands (Shavell, 1984).

63.1.3 What the determinants say about frontier models #

Read against frontier development, the determinants disagree (inference). The second and third point to rules in advance. A catastrophic harm from a frontier model could exceed any developer's assets, and a harm spread across many victims, or traced only with difficulty to one model among several, is the kind of harm for which Shavell expects suits not to be brought. The first points the other way. The developer knows more about its model's risks than any regulator, which in Shavell's analysis favors leaving the choice of precautions to it. His exception, an authority that holds expertise private parties lack, describes toxic substances and small firms, not laboratories whose staff are the experts. The fourth also cuts against regulation: what a frontier rule must verify is conduct during training and evaluation, which is behavior rather than the presence of a thing, and can be concealed. For ordinary harms, such as a model's wrong answer to one user, the second and third determinants weaken and liability does more of the work.

The statutes of the period follow Shavell's advice on joint use, at least on paper. New York's frontier statute declares its duties "cumulative with any other duties or obligations imposed under other law" (Legislature, 2026), so compliance does not shield a developer from suit. The European Union's product liability directive exempts a producer when a defect "is due to compliance of the product with legal requirements," a narrower shield that covers only the defect the requirement caused (European Parliament and Council of the European Union, 2024a). Both instruments are read in Section 63.2 and Section 63.4.

The case that frontier models need regulation was made in these terms. A 2023 white paper by Markus Anderljung and twenty-three coauthors, from policy institutes, universities, and several developers among them Google DeepMind and OpenAI, focuses on "what we term 'frontier AI' models" (Anderljung et al., 2023) (preprint). The term is the authors' own, and Chapter 62 records its critics. The paper argues that such models pose "a distinct regulatory challenge." In its words, "dangerous capabilities can arise unexpectedly; it is difficult to robustly prevent a deployed model from being misused; and, it is difficult to stop a model's capabilities from proliferating broadly." It proposes standard setting, registration and reporting, and mechanisms to ensure compliance, and holds that "Industry self-regulation is an important first step" but that government intervention will be needed (Anderljung et al., 2023). In Shavell's terms, the first challenge erodes the developer's advantage in knowledge, since a capability that surprises its developer is one the developer did not know about, and the third is the release question of Section 63.7 (inference).

Other proposals mixed the instruments. Matthew Scherer's 2015 analysis, by its abstract, ends with "a proposal for an indirect form of AI regulation based on differential tort liability" (Scherer, 2015) (preprint). Jess Whittlestone and Jack Clark, the second then at Anthropic, proposed that governments "invest in initiatives to measure and monitor various aspects of AI research, deployment, and impacts" (Whittlestone and Clark, 2021) (preprint), an attack on the first determinant, since a state that measures narrows the knowledge gap that favors liability (inference). Allan Dafoe's survey chapter frames governance by "institutional fit and adaptation to the externalities produced by AI" (Dafoe, 2023), and Anka Reuel, Ben Bucknall, and colleagues catalog the open problems of technical AI governance, a field that by their abstract can help by "designing mechanisms for enforcement, incentivization, or compliance" (Reuel et al., 2024b) (preprint).

The theory stops at its menu. It compares liability with rules the regulator writes, and when the determinants conflict, as here, it does not say what to do beyond using both. Practice has chosen a third instrument: require the firm that knows most to write and follow its own risk management, and regulate that. Section 62.5 teaches this as management-based regulation, the family in which Coglianese and Lazer place enforced self-regulation, and the statutes below are built on it. The theory also prices harm after the fact. For a harm that cannot be undone or paid for, liability deters only through the developer's expectation of a judgment it would never pay in full. Step 4 of the derivation says that is too weak (inference).

Sources cited in Section 63.1 10

Calabresi (1970) The costs of accidents: a legal and economic analysis
Shavell (1980) Strict Liability versus Negligence
Shavell (1984) Liability for Harm versus Regulation of Safety
Legislature (2026) Transparency and safety requirements for developers of artificial intelligence frontier models (New York Laws of 2026, Chapter 96)
European Parliament and Council of the European Union (2024a) Directive (EU) 2024/2853 (Product Liability Directive)
Anderljung et al. (2023) Frontier AI Regulation: Managing Emerging Risks to Public Safety
Scherer (2015) Regulating Artificial Intelligence Systems: Risks, Challenges, Competencies, and Strategies
Whittlestone and Clark (2021) Why and How Governments Should Monitor AI Development
Dafoe (2023) AI Governance
Reuel et al. (2024b) Open Problems in Technical AI Governance

63.2 Statutes and orders as of October 2026 #

What had been enacted by October 2026, and how is it built? This section reads each instrument from the text its issuing body published for what triggers it, what it requires of whom, and who enforces it; Section I.1 holds the provisions, thresholds, and dates, and Table I.1 summarizes them. Specifications of model behavior that states write are Section 17.5's, and coordination among states is Chapter 66's.

63.2.1 The European Union #

As of October 2026 the European Union's AI Act, Regulation (EU) 2024/1689, is the most complete instrument, and Section I.1.2 sets out its provisions, thresholds, and dates. Beside tiers of duty drawn by use, it regulates general-purpose AI models themselves, with a stricter tier for models "with systemic risk," presumed from the computation used to train them, which the Commission may also designate (European Parliament and Council of the European Union, 2024b). Providers in that tier must evaluate their models with adversarial testing, assess and mitigate systemic risks, report serious incidents, and protect cybersecurity, and may show compliance through a code of practice, which Section 62.5 reports (European Parliament and Council of the European Union, 2024b; European Commission, 2025). Enforcement against model providers lies with the Commission, through the AI Office, with graduated powers: requests for documentation, evaluations for which it may compel access to the model, requests for mitigation, and, last, a requirement to "restrict the making available on the market, withdraw or recall the model," with fines that also apply to refusing access for an evaluation (European Parliament and Council of the European Union, 2024b).

Michael Veale and Frederik Zuiderveen Borgesius, reading the 2021 draft against "the structure of EU product safety regimes over the last four decades," found that different rules for different risk levels "make sense" but that the enforcement regime engendered "significant concern" (Veale and Zuiderveen Borgesius, 2021). Philipp Hacker, Andreas Engel, and Marco Mauer, by their abstract, argued that the rules "must match the specificities of pre-trained models" (Hacker et al., 2023); the final text adds a model-level tier.

63.2.2 The United States: orders and standards #

As of October 2026 federal policy had been made by executive order, and it reversed; Section I.1.7 sets out the orders. An order of October 2023 required developers of the largest models to report their training, the security of their weights, and red-team results (Executive Office of the President, 2023), and it was revoked in January 2025 (Executive Office of the President, 2025c). Later orders and a 2025 action plan set a policy of global AI dominance (Executive Office of the President, 2025d; House, 2025), created a task force to challenge state AI laws (Executive Office of the President, 2025a), and offered developers a voluntary route for giving the government access to models before release, while authorizing no licensing or preclearance (Executive Office of the President, 2026). The federal standards are voluntary (Tabassi, 2023).

63.2.3 The United States: the states #

California's SB 1047 would have required a developer, before training, to implement "the capability to promptly enact a full shutdown" and to retain a third-party auditor every year from 2026 (Legislature, 2024). It was vetoed in September 2024. The report the governor commissioned from three scholars, published on 17 June 2025, rests its principles on "trust but verify," naming "Whistleblower protections, third-party evaluations, and public-facing information sharing" as key instruments and asking that thresholds adapt "not only by updating specific threshold values but also by revising or replacing metrics if needed" (Joint California Policy Working Group on AI Frontier Models, 2025).

As of October 2026 two states had frontier statutes, California's SB 53 of September 2025 (Legislature, 2025b) and New York's RAISE Act of December 2025 (Legislature, 2025a), whose article a law of March 2026 replaced with effect from 1 January 2027 (Legislature, 2026). Section I.1.10 sets out their provisions, New York's as replaced. Both cover models above a threshold of training compute, with a further condition of revenue for most duties, and both have the developer write and publish its own framework, publish a transparency report before or when it deploys a new or substantially modified frontier model, send summaries of risk from internal use, report safety incidents, and answer to the Attorney General for civil penalties (Legislature, 2025b; Legislature, 2026). California adds whistleblower protections and a yearly review of its definitions that weighs "the external verifiability of determining whether a person or foundation model is covered" (Legislature, 2025b). New York adds a disclosure statement that large developers file with a new office in its Department of Financial Services, whose costs they share (Legislature, 2026). The framework duty is read in Section 61.5.3, the incident duty in Section I.1.16, and the whistleblower protections in Section I.1.13. Both statutes require the kind of reports and disclosures the December 2025 order asks the Commerce Department to scrutinize (Executive Office of the President, 2025a). Whether its evaluation named either, the sources read here do not say.

63.2.4 China, Japan, and Korea #

China's interim measures for generative AI services of July 2023 could be read only as the page announcing them (Cyberspace Administration of China et al., 2023). Matt Sheehan reports that their April 2023 draft "reinforced many boilerplate content mandates ('embody Core Socialist Values')" and required providers to ensure the "truth, accuracy, objectivity, and diversity" of training data (Sheehan, 2023). The 2021 national ethical norms make one of six basic requirements that AI be controllable and trustworthy, with people keeping the right to stop an AI system at any time (the book's translation) (National Governance Committee for New Generation Artificial Intelligence, 2021). Huw Roberts and colleagues read the 2017 plan's aim "to become the world leader in AI by 2030" (Roberts et al., 2021). As of October 2026 Japan's AI law of 2025 is a promotion law whose English text has no penalty provision (Government of Japan, 2025), and South Korea's Framework Act, in force from January 2026, defines "high-impact AI" by area of use (Korea, 2025); Section I.1.14 sets out both, with a 2026 tracking report on China's guidance (AI, 2026).

63.2.5 State evaluators and standards #

The United Kingdom's institute, announced in November 2023, observed that evaluations of the most advanced systems "are taking place inside the top AI tech companies" and that "Governments and any external parties are unable to verify the results of these evaluations." It also stated that "The Institute is not a regulator" and "will not hold responsibility for any release decisions" (Department for Science and Technology, 2023). Now the AI Security Institute, it describes itself as partnering with "companies at the frontier of AI" (UK AI Security Institute, 2025a), and reports finding vulnerabilities in the safeguards of every system it tested (UK AI Security Institute, 2025b). The US standards center's September 2026 assessment of an open-weight model is reported in Section 58.6 (Center for AI Standards and Innovation, 2026); the international scientific assessment, which "does not endorse any particular policy or regulatory approach," in Section 66.6 (Bengio et al., 2026). ISO/IEC 42001 "specifies requirements for establishing, implementing, maintaining, and continually improving" an organization's AI management system (42, 2023).

63.2.6 What triggers a duty #

As of October 2026 most of these duties are triggered by training compute (Table I.1). The AI Act defines its stricter tier by "high impact capabilities evaluated on the basis of appropriate technical tools and methodologies, including indicators and benchmarks," and presumes them above a threshold of training compute (European Parliament and Council of the European Union, 2024b); one federal order uses a capability benchmark alone. Lennart Heim and Leonie Koessler, of the Centre for the Governance of AI, argue in a 2024 preprint that compute is "considerably more suitable than other proposed metrics to serve as an initial filter to trigger additional regulatory requirements and scrutiny." It remains "an imperfect proxy for risk" that "should not be used in isolation" (Heim and Koessler, 2024). Sara Hooker's 2024 essay, also a preprint, argues that "the relationship between compute and risk is highly uncertain and rapidly changing," that the choice of $10^{26}$ and $10^{25}$ "has not been justified," and that an underspecified threshold "is a lesson in Goodhart's Law" (Hooker, 2024). Both treat a threshold as a filter, not a measure of risk, and they disagree about how long it stays a good filter (inference). SB 53's review clause asks for exactly the external verifiability Heim and Koessler claim for compute. Compute as a lever of international verification is Section 66.5's.

Sources cited in Section 63.2 31

European Parliament and Council of the European Union (2024b) Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence
European Commission (2025) The General-Purpose AI Code of Practice
Veale and Zuiderveen Borgesius (2021) Demystifying the Draft EU Artificial Intelligence Act — Analysing the good, the bad, and the unclear elements of the proposed approach
Hacker et al. (2023) Regulating ChatGPT and other Large Generative AI Models
Executive Office of the President (2023) Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence
Executive Office of the President (2025c) Initial Rescissions of Harmful Executive Orders and Actions
Executive Office of the President (2025d) Removing Barriers to American Leadership in Artificial Intelligence
House (2025) Winning the Race: America’s AI Action Plan
Executive Office of the President (2025a) Ensuring a National Policy Framework for Artificial Intelligence
Executive Office of the President (2026) Promoting Advanced Artificial Intelligence Innovation and Security
Tabassi (2023) Artificial Intelligence Risk Management Framework (AI RMF 1.0)
Legislature (2024) SB-1047 Safe and Secure Innovation for Frontier Artificial Intelligence Models Act.
Joint California Policy Working Group on AI Frontier Models (2025) The California Report on Frontier AI Policy
Legislature (2025b) SB-53 Artificial intelligence models: large developers.
Legislature (2025a) Responsible AI Safety and Education Act (New York RAISE Act)
Legislature (2026) Transparency and safety requirements for developers of artificial intelligence frontier models (New York Laws of 2026, Chapter 96)
Cyberspace Administration of China et al. (2023) 生成式人工智能服务管理暂行办法 (Interim Measures for the Management of Generative Artificial Intelligence Services)
Sheehan (2023) China’s AI Regulations and How They Get Made
National Governance Committee for New Generation Artificial Intelligence (2021) 新一代人工智能伦理规范 (Ethical Norms for New Generation Artificial Intelligence)
Roberts et al. (2021) The Chinese approach to artificial intelligence: an analysis of policy, ethics, and regulation
Government of Japan (2025) Act on Promotion of Research and Development, and Utilization of Artificial Intelligence-related Technology
Korea (2025) Framework Act on the Development of Artificial Intelligence and Establishment of Trust
AI (2026) State of AI Safety in China (2026)
Department for Science and Technology (2023) Introducing the AI Safety Institute
UK AI Security Institute (2025a) AISI Research Agenda
UK AI Security Institute (2025b) Frontier AI Trends Report
Center for AI Standards and Innovation (2026) CAISI's Assessment of Z.ai's GLM-5.3 Cyber Capabilities
Bengio et al. (2026) International AI Safety Report 2026
42 (2023) ISO/IEC 42001:2023 Information technology - Artificial intelligence - Management system
Heim and Koessler (2024) Training Compute Thresholds: Features and Functions in AI Regulation
Hooker (2024) On the Limitations of Compute Thresholds as a Governance Strategy

63.3 What regulatory theory predicts #

From regulatory theory. Regulatory theory studies how states govern firms that know more than they do, and it makes predictions about instruments like those just read. Enforced self-regulation and management-based regulation, the family the frameworks and the state statutes belong to, are taught in Section 62.5. This section takes four other questions: whether the regulator will come to serve the regulated, how a regulator escalates, how it targets its effort, and how jurisdictions compete.

63.3.1 Capture through expertise #

Capture is the condition in which regulation serves the industry it regulates. Section 61.6.1 states the theory from the abstracts that could be read. George Stigler describes the political process that lets "relatively small groups" obtain regulation (Stigler, 1971). In Jean-Jacques Laffont and Jean Tirole's agency-theoretic account, an interest group's power is tied to the informational asymmetry between the industry and its political principal (Laffont and Tirole, 1991). An edited volume argues that capture can be diagnosed, measured, and limited (Cambridge University Press, 2013). Julia Black's paper on decentring regulation, whose subtitle concerns "the role of regulation and self-regulation in a 'post-regulatory' world," could be read only as a catalog record (Black, 2001).

The route to capture that matters here is expertise (inference). The classic theory, as far as its abstracts show, concerns groups that obtain regulation through the political process; a frontier regulator also depends on the regulated for three inputs it cannot make itself. The first is access to the models. The UK institute's founding document says that evaluations of the most advanced systems take place inside the companies and cannot be verified from outside, and the institute describes itself as partnering with frontier companies (Department for Science and Technology, 2023; UK AI Security Institute, 2025a). The US action plan has its evaluator work "in partnership with frontier AI developers," and the 2026 order makes pre-release access voluntary (House, 2025; Executive Office of the President, 2026). Of the instruments in force as of October 2026, only the AI Act compels access to a model, with a fine for refusing it (European Parliament and Council of the European Union, 2024b). The second input is data: the state statutes' incident reports and internal-use summaries are written by the developer (Legislature, 2025b; Legislature, 2026). The third is people. Hooker's essay, citing others, describes "a severe shortage of technical staff with AI experience within government" (Hooker, 2024) (preprint), so evaluators recruit from the pool the developers also draw on.

Stated as a hypothesis to measure, the expertise channel predicts two things (inference). A state evaluator whose access the developer grants will publish fewer findings adverse to that developer than one whose access is compelled, holding the models fixed. And the flow of staff between evaluators and developers will predict the evaluator's later findings about its staff's former employers. Both are measurable, from access terms, published findings, and employment histories, and as of October 2026 we found no study that measures either. The strongest counter-reading is that dependence is not capture. An evaluator that cooperates with developers learns more about their systems than one kept at arm's length, which is the trade-off between control and technical information that Section 61.1.3 reports from Kathleen Bawn's model. The two readings make different predictions only when a finding would cost the developer something, which is the case the crux of this chapter asks for.

63.3.2 Escalation #

A regulator needs a way to move from asking to compelling. Ian Ayres and John Braithwaite's Responsive Regulation, read here in its abstract, argues from studies in four countries that regulation "does not have to proceed with an adversarial tone, nor does it have to be 'soft' or 'hard' to be effective" (Ayres and Braithwaite, 1992). The abstract does not say how such a regulator escalates, so what follows is this book's own reading, built from the inspection game of Section 14.6 (inference).

In that game an inspector who cannot commit gains nothing from a harsher penalty: the penalty lowers how often it inspects and leaves the violation rate where it was. An inspector who can announce its rate and keep to it deters violation with any rate above a level that falls as the penalty rises (Section 14.6). Applied to a regulator, the instruments it can escalate to matter only if the developer believes they will be used, and only through the probability that a violation is detected. Table 63.1 lays out the instruments of October 2026 from the mildest to the most severe.

Table 63.1 The instruments enacted by October 2026 ordered from persuasion to the most severe sanction, as their texts state them; New York's take effect on 1 January 2027

Step	European Union	United States, federal	California and New York
Cooperation	Codes of practice; structured dialogue before requests (European Parliament and Council of the European Union, 2024b; European Commission, 2025)	Evaluation in partnership with developers; voluntary pre-release access (House, 2025; Executive Office of the President, 2026)	Frameworks the developer writes (Legislature, 2025b; Legislature, 2026)
Information	Requests for documentation; evaluations with compelled access (European Parliament and Council of the European Union, 2024b)	None compelled	Transparency reports, incident reports, internal-use summaries; in New York, a disclosure statement filed with a state office (Legislature, 2025b; Legislature, 2026)
Money	Fines up to 3% of worldwide turnover or €15 million (European Parliament and Council of the European Union, 2024b)	None	Up to $1 million per violation in California; up to $1 million and then $3 million in New York (Legislature, 2025b; Legislature, 2026)
Stopping a product	Restriction, withdrawal, or recall of a model on the market (European Parliament and Council of the European Union, 2024b)	No instrument; the June 2026 order authorizes no licensing or preclearance and leaves existing agency authority in place (Executive Office of the President, 2026)	No instrument
Stopping development	No instrument	No instrument	No instrument; New York bars developing a frontier model without a current disclosure statement filed and the developer's share of the office's costs paid; the statement holds the information the office requires, such as the developer's identity, offices, owners, and contacts (Legislature, 2026)

Two features of the table follow from the texts (inference). Most steps above cooperation run mainly on information the developer supplies. The other routes are the EU's compelled evaluations, complaints by downstream providers, and alerts from its scientific panel (European Parliament and Council of the European Union, 2024b), and in California reports by members of the public and disclosures by employees the statute protects (Legislature, 2025b). The probability of detection, which the inspection game says carries the deterrent, therefore rests largely on the regulated party's own reports. And the top row is empty everywhere: no instrument read here lets a state stop a training run. New York's bar on developing without a disclosure statement is lifted by filing the developer's particulars and paying its share of the office's costs, not by any finding about the model. The most severe step that exists acts on a model already on the market, and Section 63.7 explains why it reaches a model whose weights have been published only in name.

63.3.3 Targeting by risk #

A regulator with a fixed budget must decide where to look. The AI Act's recitals make that choice explicit, tailoring "the type and content of such rules to the intensity and scope of the risks that AI systems can generate" (European Parliament and Council of the European Union, 2024b). Julia Black and Robert Baldwin's paper on risk-based regulation could be read only by its title, which asks for regulation that is "really responsive" (Black and Baldwin, 2010), and the field's reference handbook by its abstract, which notes that regulation can be "enabling or facilitative" as well as restrictive (Oxford University Press, 2010). Targeting needs a measure of risk, and the model tier uses compute in its place, with the Commission's power to designate a model as the correction (European Parliament and Council of the European Union, 2024b). That is the threshold dispute of Section 63.2.6 seen from the regulator's side (inference). Shavell's fourth determinant asks that administrative effort be focused on the parties most likely to cause harm. A compute threshold focuses it on the largest training runs, which is the same thing only while scale tracks harm.

63.3.4 New designs #

Some proposals change who regulates. Jack Clark, writing from an OpenAI address, and Gillian Hadfield proposed in 2019 that "governments create a market for regulation in which private sector organizations compete to achieve regulatory outcomes set by a government regulator." Unlike self-regulation, "the overall outcomes that must be achieved by regulation are determined by the government and not the regulated entity itself" (Clark and Hadfield, 2019) (preprint). Their later version presents the design as an answer to a technical deficit in legislatures and a democratic deficit in "Overreliance on industry to provide technical standards" (Hadfield and Clark, 2023) (preprint). Hadfield's 2026 perspective adds registration regimes for frontier models and "registration and identification regimes for autonomous agents" (Hadfield, 2026), and Alan Chan and colleagues propose infrastructure for "attributing actions, properties, and other information to specific agents, their users, or other actors" (Chan et al., 2025) (preprint), which would supply the identifiable defendant liability needs (Section 63.4) (inference). A regulatory market moves the expertise problem rather than solving it: the licensed private regulator needs the same access, data, and staff as a state evaluator, and is paid by the firms it oversees, the issuer-pays problem of Section 67.2 (inference).

63.3.5 Competition among jurisdictions #

A jurisdiction's rules can govern firms far outside it. Anu Bradford's article, read here in its 2012 form, argues that the EU "has a strong and growing ability to promulgate regulations that become entrenched in the legal frameworks of developed and developing markets alike." It gives five conditions: market power, regulatory capacity, a preference for strict rules, a predisposition to regulate inelastic targets, and the nondivisibility of standards (Bradford, 2020). Charlotte Siegmann and Markus Anderljung's 2022 report "tentatively" concludes that both a de facto effect, firms changing their products everywhere, and a de jure effect, other jurisdictions copying the rules, "are likely for parts of the EU regulatory regime" (Siegmann and Anderljung, 2022) (preprint). Nathalie Smuha argues that "the 'race to AI' also brings forth a 'race to AI regulation'" (Smuha, 2021).

Bradford's conditions split for frontier models (inference). Access to the EU's market is an inelastic target, but where a model is trained is not, and the AI Act regulates models placed on the EU market rather than training runs. A model's weights are one artifact served everywhere, which favors nondivisibility, while a deployment's safeguards can differ by country. The competition of the period was also within one country: the US orders of 2025 and 2026 contest state statutes in the name of a single national framework (Executive Office of the President, 2025a). On precaution, Cass Sunstein's critique of the precautionary principle could be read only by its title (Sunstein, 2003). Jack Stilgoe and colleagues' framework of responsible innovation, by its abstract, has four dimensions, "anticipation, reflexivity, inclusion and responsiveness" (Stilgoe et al., 2013).

63.3.6 Where the theory stops #

Regulatory theory grew up on industries whose products change slowly relative to the regulator's learning. A frontier model generation is shorter than a rulemaking cycle, so a regulator assessing a model is assessing one already being superseded (inference). The texts admit it. The AI Act lets the Commission amend its compute threshold by delegated act, and SB 53 orders its definitions reviewed every year (European Parliament and Council of the European Union, 2024b; Legislature, 2025b). Chapter 64 calls the general form of this problem the dilemma of control (Section 64.1): a technology is easiest to steer before its effects are known.

Sources cited in Section 63.3 26

Stigler (1971) The Theory of Economic Regulation
Laffont and Tirole (1991) The Politics of Government Decision-Making: A Theory of Regulatory Capture
Cambridge University Press (2013) Preventing Regulatory Capture
Black (2001) Decentring Regulation: Understanding the Role of Regulation and Self-Regulation in a 'Post-Regulatory' World
Department for Science and Technology (2023) Introducing the AI Safety Institute
UK AI Security Institute (2025a) AISI Research Agenda
House (2025) Winning the Race: America’s AI Action Plan
Executive Office of the President (2026) Promoting Advanced Artificial Intelligence Innovation and Security
European Parliament and Council of the European Union (2024b) Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence
Legislature (2025b) SB-53 Artificial intelligence models: large developers.
Legislature (2026) Transparency and safety requirements for developers of artificial intelligence frontier models (New York Laws of 2026, Chapter 96)
Hooker (2024) On the Limitations of Compute Thresholds as a Governance Strategy
Ayres and Braithwaite (1992) Responsive Regulation
European Commission (2025) The General-Purpose AI Code of Practice
Black and Baldwin (2010) Really Responsive Risk-Based Regulation
Oxford University Press (2010) The Oxford Handbook of Regulation
Clark and Hadfield (2019) Regulatory Markets for AI Safety
Hadfield and Clark (2023) Regulatory Markets: The Future of AI Governance
Hadfield (2026) Legal infrastructure for transformative AI governance
Chan et al. (2025) Infrastructure for AI Agents
Bradford (2020) The Brussels Effect
Siegmann and Anderljung (2022) The Brussels Effect and Artificial Intelligence: How EU regulation will impact the global AI market
Smuha (2021) From a ‘race to AI’ to a ‘race to AI regulation’: regulatory competition for artificial intelligence
Executive Office of the President (2025a) Ensuring a National Policy Framework for Artificial Intelligence
Sunstein (2003) Beyond the Precautionary Principle
Stilgoe et al. (2013) Developing a framework for responsible innovation

63.4 Liability for AI harms #

From law. Courts regulate after the fact, and Section 63.1 says when that is enough. This section asks which doctrines could attach liability for harm caused by a model or an agent, and what gap each leaves. The doctrine of agency law applied to AI agents, including the liability of an agent's principal for its wrongs, is reported in Section 56.8; legal personhood for AI, in Section 21.6.4. Here the question is how the doctrines work as regulation.

63.4.1 Whom the law holds #

Every doctrine here reaches a person behind the system; they differ in which person. Three works could be read only by their titles: Alan Sykes's 1984 paper on the economics of vicarious liability, the rule that holds one party liable for another's wrongs (Sykes, 1984); David Vladeck's "Machines Without Principals: Liability Rules and Artificial Intelligence" (Vladeck, 2014); and Ian Ayres and Jack Balkin's "The Law of AI is the Law of Risky Agents without Intentions" (Ayres and Balkin, 2024) (preprint). Section 56.8 reports the usual condition for holding a principal liable for an agent: the agent's conduct must be a tort, a civil wrong for which the law awards damages. The titles name the difficulty (inference): liability doctrines assume a wrongdoer with a principal and an intention, and a model that acts may have neither in the doctrines' sense.

63.4.2 Product or conduct #

Is a harmful output a defective product or careless conduct? The answer decides between the two rules of Section 63.1. Ryan Abbott, in a 2016 working paper later published in a law review, writes that manufacturers of "computer tortfeasors are likely strictly responsible for their harms" under current law. He argues that where a manufacturer can show that an autonomous machine "is safer than a reasonable person, the manufacturer should be liable in negligence rather than strict liability," which would "treat a computer tortfeasor as a person rather than a product" (Abbott, 2016). The proposal uses the activity-level effect of Section 63.1 on purpose (inference): negligence lets the injurer act more, and Abbott wants more automation where the machine is safer than the person it replaces. A frontier model's worst harms have no human counterpart to be safer than.

The European Union has taken the product side. Its product liability directive of 2024 treats software as a product, lists "AI systems" among the software it covers, and makes liability no-fault; as of October 2026 it governed no product yet, since it applies to products placed on the market after 9 December 2026 (European Parliament and Council of the European Union, 2024a). Section I.1.6 sets out its provisions. It answers part of the causation problem: a national court shall presume a product defective, or the defect the cause of the damage, when the claimant faces excessive difficulties of proof and shows that the defect or the causal link is likely, and the defendant may rebut (European Parliament and Council of the European Union, 2024a). Its limits are as specific. It compensates natural persons for listed kinds of damage, lets a producer escape liability by showing that the state of knowledge did not allow the defect to be discovered, and does not apply to open-source software supplied outside a commercial activity (European Parliament and Council of the European Union, 2024a). The development-risk defense matters most for frontier models (inference): a capability its own developer did not know about, the first of the challenges Anderljung and colleagues list, is the kind of defect the defense covers, if the state of knowledge did not allow its discovery.

On the conduct side, Section 60.4.3 reports two arguments that bear on users rather than developers. One is Andrew Selbst's, that users of inscrutable decision tools cannot foresee their errors, which strains negligence; the other is Madeleine Clare Elish's account of the moral crumple zone, in which blame for an automated failure falls on the nearest human operator (Selbst, 2020; Elish, 2019). Read together, they point to two ways negligence left alone misses the developer: injured people go uncompensated when no operator was at fault (Selbst, 2020), or blame falls on the human in the loop for failures the developer's design made likely (Elish, 2019) (inference). Corporate law has the mirror problem. Mihailis Diamantis argues that the law has defined corporate misconduct through the misconduct of employees, so that "corporations will become increasingly immune to civil and criminal liability as they transfer responsibility from employees to algorithms." He proposes treating firms as if they "know" what their servers store and "intend" what their automated systems decide (Diamantis, 2019).

63.4.3 Catastrophic harm #

The hardest case is the one step 4 of the derivation describes: a harm larger than any defendant can pay. Gabriel Weil's proposal, read in its abstract, starts from the claim that advanced AI "can generate physical-harm risks whose extreme downside may be practically non-compensable," which existing tort doctrine "is likely to underinternalize" (Weil, 2024) (preprint). He proposes two remedies. First, training or deploying a defined subset of frontier systems would be treated "as an abnormally dangerous activity," triggering strict liability for physical harms "within the risk that makes the activity dangerous." Second, for smaller "warning-shot" injuries whose precautions overlap with those that would reduce the uninsurable downside, courts could award punitive damages "calibrated to internalize the expected value of that downside." The abstract also mentions "capability-scaled liability insurance" as a legislative complement (Weil, 2024).

The first remedy is the strict-liability step of the derivation applied to the largest harms, and it inherits step 4: strict liability for a harm the defendant cannot pay controls neither care nor activity (inference). The second remedy is the interesting one. It charges the expected cost of the catastrophe on harms small enough to be paid, which is the only point at which a solvent defendant exists. It needs three things the derivation does not supply: small harms that occur and are litigated, precautions that reduce both the small and the large harm, and a court able to estimate the expected value of a catastrophe that has not happened. The last moves Shavell's knowledge determinant from the regulator's office into the courtroom (inference).

63.4.4 Personhood as a liability question #

Giving an AI system legal personhood would change whom the law holds. Section 21.6.4 reports the arguments: Shawn Bayern's that business-entity law could already let an autonomous system emulate a legal person's private-law rights, and Joanna Bryson, Mihailis Diamantis, and Thomas Grant's that the difficulty of holding "electronic persons" accountable outweighs the moral interests personhood might protect (Bayern, 2016; Bryson et al., 2017). Lawrence Solum's 1992 article is cited there for its title (Solum, 1992). The proposal that private-law rights could lower a misaligned system's incentive to resist, by Peter Salib and Simon Goldstein, is Section 21.7.4's (Salib and Goldstein, 2024) (preprint). For this chapter the relevant warning is Bryson and colleagues': creating new legal persons has produced "entities that are accountable but unfunded, or fully-financed but unaccountable" (Bryson et al., 2017). A legal person that cannot pay is step 4 of the derivation in another form (inference).

63.4.5 Where liability stops #

A tort suit needs an identifiable injury, a cause that can be proven, and a defendant that can pay. Diffuse harms, spread thinly over many people, fail the first; opaque systems strain the second; catastrophic harms fail the third (inference). The European directive is the period's most direct attack on the second, through its presumptions, and leaves the first and the third where they were: it compensates natural persons for listed kinds of damage, and a presumption of causation does not make a defendant solvent. The next section turns to the institution built to stand between a defendant and a judgment it cannot pay.

Sources cited in Section 63.4 13

Sykes (1984) The Economics of Vicarious Liability
Vladeck (2014) Machines Without Principals: Liability Rules and Artificial Intelligence
Ayres and Balkin (2024) The Law of AI is the Law of Risky Agents without Intentions
Abbott (2016) The Reasonable Computer: Disrupting the Paradigm of Tort Liability
European Parliament and Council of the European Union (2024a) Directive (EU) 2024/2853 (Product Liability Directive)
Selbst (2020) Negligence and AI’s Human Users
Elish (2019) Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction
Diamantis (2019) The Extended Corporate Mind: When Corporations Use AI to Break the Law
Weil (2024) Closing the AI Accountability Gap: Strict Liability and Punitive Damages for Advanced Artificial Intelligence
Bayern (2016) The Implications of Modern Business–Entity Law for the Regulation of Autonomous Systems
Bryson et al. (2017) Of, for, and by the people: the legal lacuna of synthetic persons
Solum (1992) Legal Personhood for Artificial Intelligences
Salib and Goldstein (2024) AI Rights for Human Safety

63.5 Insurance as a regulator #

From insurance. Where liability cannot reach, an insurer can stand between the injurer and the judgment, and in doing so it can regulate. Insurers price risk, inspect the insured, and condition coverage on precautions, so a liability rule backed by insurance can become continuing oversight. They also refuse risks they cannot price, and the refusal is itself information.

63.5.1 Insurers as private regulators #

The usual worry runs the other way. Moral hazard is the weaker incentive to avoid a loss that an insured party has once someone else bears it. Omri Ben-Shahar and Kyle Logue call it "one of the cornerstones of the economics of information." They summarize the classic answer, for which they cite Steven Shavell's 1979 paper and Mark Pauly's comment, among others: "insurance must be partially pared down to give people incentives to prevent harms," which is what deductibles and copayments do (Ben-Shahar and Logue, 2012). Both cited papers could be read here only as catalog records (Shavell, 1979; Pauly, 1982).

Ben-Shahar and Logue's own argument, in a 2012 article, is "the opposite proposition": insurance "can reduce and in some cases solve, rather than create or exacerbate, the moral hazard" (Ben-Shahar and Logue, 2012). Insurers assemble actuarial data that tell them which precautions reduce losses, and they use it to set differentiated premiums, deductibles, private safety codes, and inspections. Policyholders know which precautions they took, but "often lack the expertise to quantify the effect of the precaution on risk reduction." Refusal is one of the tools. Insurers "often will not issue products liability coverage to a manufacturer that does not have a system in place for maintaining quality control with respect to safety issues or for safety testing its product," and ski resorts' insurers make lift inspections a condition of coverage. Where an activity cannot go ahead without insurance, "insurers have de facto control over access" to it (Ben-Shahar and Logue, 2012).

Shavell's 1984 paper adds the condition on which this depends. Whether mandated liability insurance restores the incentive that limited assets destroy "depends on whether insurers can easily determine risk-reducing behavior and link the premium charged (or the conditions of coverage) to such behavior." If they cannot, the insured's motive to take care "may be insufficient, and plausibly lower than if they did not own coverage" (Shavell, 1984).

63.5.2 When a risk is called uninsurable #

Howard Kunreuther and colleagues' 1993 paper on insurer ambiguity and market failure could be read only by its title (Kunreuther et al., 1993). Dwight Jaffee and Thomas Russell's 1997 paper begins from a case. After the Northridge earthquake of 1994, companies "representing 93% of the homeowner's insurance market either stopped writing homeowner's insurance or imposed strict limits on the policies they were willing to sell" (Jaffee and Russell, 1997). They list the three factors the insurance literature often blames for an uninsurable risk: adverse selection and moral hazard, a risk "too large" in some sense, and a probability "not susceptible to precise actuarial calculation." They argue that none of the three is sufficient for catastrophe insurance. A catastrophe is "infrequent, local, and unlikely to be correlated with the price of a global market index, so it is certainly diversifiable"; the obstacle is instead the capital market problem of matching "a smooth flow of annual premium receipts to a highly nonsmooth flow of annual loss payments" (Jaffee and Russell, 1997).

Their argument cuts two ways for AI (inference). It removes the easy reason to call a catastrophic AI risk uninsurable: size and imprecise probabilities did not stop marine insurance either. But their reason for optimism is that a catastrophe is local and uncorrelated with the wider economy. An AI failure large enough to motivate frontier regulation need not be either.

63.5.3 Why correlation breaks pooling #

An insurer's business rests on what Ben-Shahar and Logue call the law of large numbers, "reducing the variance by increasing the size of the pool" (Ben-Shahar and Logue, 2012). That works only for losses that are close to independent. Section 57.4 derives how much of a deployment of copies fails at once when the copies share the situations they meet; the same arithmetic says how much capital an insurer of those deployments must hold.

Derivation Capital per policy when losses are correlated

An insurer covers $n_{\text{inst}}$ deployed instances of a model, one policy each. Each instance causes a loss of 1 with average probability $p_{\text{inst}}$ , and any two instances' losses have correlation $\rho$ . The premium covers the expected loss, $p_{\text{inst}}$ per policy. Capital covers the rest, up to a loss the insurer must be able to pay in all but one year in a hundred.

The share of policies with a loss. Its mean is $p_{\text{inst}}$ , and (57.2) gives its variance, $p_{\text{inst}}(1 - p_{\text{inst}})\,[1 + (n_{\text{inst}} - 1)\rho] / n_{\text{inst}}$ .
Capital per policy. Where the share is roughly normal, as it is for a large pool of independent losses, the capital per policy needed at the 99th percentile is about $\Phi^{-1}(0.99) \approx 2.33$ standard deviations of the share: $k_{\text{pol}} \approx 2.33\,\sqrt{\frac{p_{\text{inst}}(1 - p_{\text{inst}})\,[1 + (n_{\text{inst}} - 1)\rho]}{n_{\text{inst}}}}.$
Independent losses. With $\rho = 0$ , $k_{\text{pol}}$ falls as $1/\sqrt{n_{\text{inst}}}$ : a pool a hundred times larger needs a tenth of the capital per policy. This is why pooling works.
Correlated losses. With $\rho > 0$ , as the pool grows the bracket grows with it, and the standard deviation of the share tends to $\sqrt{\rho\, p_{\text{inst}}(1 - p_{\text{inst}})}$ , a floor no pool can lower. In a model where instances share a year's failure rate, as in Figure 63.1, the share tends to that rate itself, which pooling cannot average away. If that rate were roughly normal, $k_{\text{pol}}$ would tend to $2.33\sqrt{\rho\, p_{\text{inst}}(1 - p_{\text{inst}})}$ , which is $\sqrt{\rho}$ times what (63.1) gives a single policy, so a correlation of 0.1 would leave about a third of it however large the pool. The exact capital is the 99th percentile of the rate minus $p_{\text{inst}}$ , which depends on how the rate is distributed, not on $\rho$ alone.
Perfect correlation. With $\rho = 1$ every policy loses together, so the share is 0 or 1. If $p_{\text{inst}}$ exceeds 1 percent, the 99th percentile is a total loss, and the insurer needs capital of $1 - p_{\text{inst}}$ per policy whatever the size of the pool. The normal approximation of step 2 misstates this case; Exercise 63.2 works it exactly.

Figure 63.1 Capital an insurer must hold per policy, beyond the premium, to pay its losses in all but one year in a hundred, as the pool of insured deployments grows. Computed exactly from a stated model: each deployment fails with average probability p, and failures are independent (orange) or share situations with correlation ρ (blue), in which case the number of failures is beta-binomial, the count that results when the year's failure rate itself varies. The dotted line is the floor the correlated curve approaches, the 99th percentile of the failure rate itself minus p. The zigzags at small pools come from counting whole losses. Both parameters are illustrative.

Figure 63.1 shows the floor. With a 2 percent loss rate and a correlation of 0.05, one policy alone needs capital of 0.98 of a loss beyond its premium, a thousand independent policies about 0.01 each, and a thousand correlated policies about 0.13 each, almost exactly the floor, which no larger pool lowers. The exact floor is larger than the 0.07 that step 4's normal approximation gives, because in this model the year's loss rate is skewed and the rare bad years carry the tail. A failure shared by every deployment of one model is a positive $\rho$ within one insurer's book. A failure shared by models of one lineage raises it across books (inference); Section 52.3 and Section 57.4 report the studies of correlated errors among language models.

63.5.4 Insuring AI #

Anat Lior's 2022 article argues that "An appropriate policy response, however, must include insurance as a regulatory mechanism," able to "incentivize and channel the behavior of regulated entities," and offers a framework for which policies should be bought and by whom (Lior, 2022). Weil's "capability-scaled liability insurance," reported in Section 63.4, is the same instrument tied to a capability measure (Weil, 2024). Read with the derivation, two inferences follow (inference). Where catastrophic and correlated AI risk proves uninsurable, the refusal tells a regulator what the market believes about the size and the ambiguity of that risk, which is information a regulator otherwise lacks. And a mandate to insure cannot create capacity that capital markets will not supply: a statute can require a policy, not the capital behind it.

63.5.5 Where the analogy stops #

Underwriting works from loss data, and there are no loss data for the harms that motivate frontier regulation. The insurer's information about the insured's care would come, again, from the insured, so an insurer of a frontier developer faces the regulator's knowledge problem and Shavell's condition with it: insurance disciplines care only if the insurer can see it (inference). Ben-Shahar and Logue's insurers know more than their policyholders about which precautions work because they "assemble large actuarial databases" across many policyholders (Ben-Shahar and Logue, 2012); for frontier models the developers hold the data, and an insurer would have only a handful of policyholders.

Sources cited in Section 63.5 8

Ben-Shahar and Logue (2012) Outsourcing Regulation: How Insurance Reduces Moral Hazard
Shavell (1979) On Moral Hazard and Insurance
Pauly (1982) The economics of moral hazard: comment
Shavell (1984) Liability for Harm versus Regulation of Safety
Kunreuther et al. (1993) Insurer ambiguity and market failure
Jaffee and Russell (1997) Catastrophe Insurance, Capital Markets, and Uninsurable Risks
Lior (2022) Insuring AI: The Role of Insurance in Artificial Intelligence Regulation
Weil (2024) Closing the AI Accountability Gap: Strict Liability and Punitive Damages for Advanced Artificial Intelligence

63.6 What drug and device regulation teaches #

From medicine. The regulator most often invoked as a model is the one that approves drugs before sale. Andrew Tutt's 2016 paper, titled "An FDA for Algorithms," could be read only by its title (Tutt, 2016), and Daniel Carpenter's study of the agency's gatekeeping power, Reputation and Power, only as a catalog record (Carpenter, 2014); the book reports neither. Three other chapters already report what can be read. Section 52.6 reports the 1962 drug statute's demand for "substantial evidence" before approval and its rule for withdrawing an approval on new evidence, the review by Jonathan Darrow and colleagues of expedited approvals, and Boris Babic and colleagues on algorithms that change after approval (United States Congress, 1962; Darrow et al., 2020; Babic et al., 2019). Section 54.5.3 reports how drug safety is watched after approval, and Section 64.5 how the regime was built. This section asks a narrower question: which of the drug regime's instruments has a counterpart for frontier models, as of October 2026.

63.6.1 The regime and its counterparts #

The regime moves from a burden of proof on the sponsor before sale, through a labeled use and reports of harm, to active surveillance and withdrawal. Its active surveillance is now a distributed system: by the abstract of Richard Platt and colleagues' 2018 account, the agency's Sentinel Initiative "has matured from a pilot program designed to assess potential drug-safety signals in insurance claims" (Platt et al., 2018). Table 63.2 sets each instrument beside the nearest counterpart in the instruments of Section 63.2.

Table 63.2 The drug regime's instruments and their nearest counterparts for frontier models as of October 2026

Instrument	In drug regulation	Nearest counterpart for frontier models	Who decides
Burden of proof before sale	Approval refused without substantial evidence of effect from well-controlled investigations (United States Congress, 1962)	None: no approval anywhere, and the June 2026 order authorizes no US preclearance (Executive Office of the President, 2026)	The developer, subject to later enforcement
Staged evidence	Investigational use on conditions, before approval (United States Congress, 1962)	Internal deployment and limited release (Section 52.6); SB 53's summaries of risk from internal use (Legislature, 2025b)	The developer
Labeled use	Approval for the conditions of use in the labeling (United States Congress, 1962)	Information on capabilities and limitations for downstream providers (European Parliament and Council of the European Union, 2024b)	The developer
Reports of harm	The approval holder's required records and reports of clinical experience (United States Congress, 1962); voluntary reports of adverse reactions (Section 54.5.3)	Incident reports within 15 days (California), 72 hours (New York, from 1 January 2027), or without undue delay (EU) (Legislature, 2025b; Legislature, 2026; European Parliament and Council of the European Union, 2024b)	The developer reports; the state receives
Active surveillance	Sentinel's queries of distributed records (Platt et al., 2018)	Proposed, not built (Stein et al., 2024)	No one
Withdrawal	Required when new evidence shows the drug is not shown to be safe (United States Congress, 1962)	The Commission may require withdrawal or recall of a model on the EU market (European Parliament and Council of the European Union, 2024b)	The regulator, in the EU only

The table's last column is the finding (inference). In drug regulation the regulator decides at the two ends, before sale and at withdrawal, and the manufacturer cannot sell without its approval. For frontier models, the developer decides at every step except withdrawal in the EU, and the counterpart instruments rest on reports the developer writes. Merlin Stein, Jamie Bernardi, and Connor Dunlop's 2024 position paper argues for the missing row. "Pre-deployment information is insufficient," they write; post-deployment monitoring of AI is "driven by civil society, with limited capacity," and governments, they argue, are best placed to take the lead in linking usage, incident, and impact data to risk management (Stein et al., 2024) (preprint). Among their precedents is the drug regulator's monitoring of "population-level impacts of drugs linked to individual doctor observations" (Stein et al., 2024).

63.6.2 Where the analogy stops #

Section 52.6 names four places where the drug analogy stops: AI deployments neither randomize nor register their protocols, their harms fall largely on third parties, a general-purpose model has no indication, and every AI case is approved by the developer that wrote it. One more limit belongs to regulation (inference). The 1962 statute could shift the burden of proof onto the sponsor because it could name what would count as proof, "adequate and well-controlled investigations," a design that existed before the gatekeeper did (United States Congress, 1962). A statute that shifted the burden onto a frontier developer would need a comparable design for showing a general-purpose model safe enough, and Section 52.1 shows that testing alone cannot supply one at the failure rates that matter. Section 52.6 reports Darrow and colleagues' review, read in its abstract: the regulator increasingly accepted less data and shortened its reviews, and in 2018 industry's fees paid about 80 percent of the salaries of the staff who review new drugs (Darrow et al., 2020). The abstract does not say the fees caused the change. Read as the dependence of Section 63.3.1 in a mature regime, carried by money rather than expertise, it is this book's reading (inference).

Sources cited in Section 63.6 11

Tutt (2016) An FDA for Algorithms
Carpenter (2014) Reputation and Power
United States Congress (1962) Drug Amendments of 1962 (Kefauver-Harris Amendments)
Darrow et al. (2020) FDA Approval and Regulation of Pharmaceuticals, 1983-2018
Babic et al. (2019) Algorithms on regulatory lockdown in medicine
Platt et al. (2018) The FDA Sentinel Initiative — An Evolving National Resource
Executive Office of the President (2026) Promoting Advanced Artificial Intelligence Innovation and Security
Legislature (2025b) SB-53 Artificial intelligence models: large developers.
European Parliament and Council of the European Union (2024b) Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence
Legislature (2026) Transparency and safety requirements for developers of artificial intelligence frontier models (New York Laws of 2026, Chapter 96)
Stein et al. (2024) The Role of Governments in Increasing Interconnected Post-Deployment Monitoring of AI

63.7 Release, open weights, and outside access #

Whether a developer may publish a model's weights, the trained parameters that are the model, is a regulatory decision with two sides. Published weights let anyone run, study, and modify the model, including removing its safeguards; they also let anyone outside the developer check it. The misuse side of this question is reported in other chapters, and the verification side in another. This section reads what the instruments of October 2026 say about release, recalls the evidence on each side, and asks what would move the question.

63.7.1 What the instruments say about release #

As of October 2026 the instruments split on release, and Section I.1.17 sets out what each says. The AI Act relieves providers of models released under a free and open-source licence of some documentation duties, but not for models with systemic risk (European Parliament and Council of the European Union, 2024b). The Act's most severe power, to require a provider to "withdraw or recall the model," presumes a model whose copies the provider controls (inference). For a model whose weights are already public, withdrawal from the market removes the provider's own offering and no one else's copies.

Federal policy in the United States leaves the release decision to the developer (House, 2025), and its June 2026 order authorizes no licensing or preclearance of release (Executive Office of the President, 2026). California's statute and New York's 2026 article both require cybersecurity only for unreleased weights and leave release alone (Legislature, 2025b; Legislature, 2026). The EU product liability directive excludes open-source software supplied outside a commercial activity (European Parliament and Council of the European Union, 2024a).

SB 53 also writes a baseline into law. Its definition of catastrophic risk excludes risk from information a model outputs "if the information is otherwise publicly accessible in a substantially similar form from a source other than a foundation model" (Legislature, 2025b). That is the marginal-risk idea of Section 59.1, which measures a model's risk against what exists without it (Kapoor et al., 2024b), with one choice made explicit: the baseline excludes other foundation models, so a closed developer cannot argue that an open model already provides the same thing (inference).

The US government's 2024 review of models with widely available weights, whose conclusion that the evidence could not yet establish whether restrictions were warranted Section 48.4.3 reports, made two further points (National Telecommunications and Information Administration, 2024). It recommended "Assessing the lag time between developers introducing capabilities in leading proprietary models, and those same capabilities being made available in open models." It also warned that without changes in research and monitoring, "any evidence of risks that would justify possible policy interventions to restrict the availability of model weights might arise only after AI models with those capabilities, closed or open, have been developed or released" (National Telecommunications and Information Administration, 2024). A release cannot be undone, so evidence that arrives after it can inform only the next release (inference).

63.7.2 The misuse side #

Two chapters carry the evidence. Section 48.4.1 reports that a little fine-tuning removes safety training, a result that led its authors to call releasing a fine-tunable model releasing "its evil twin" (Murphy et al., 2025), and Section 48.4.3 the safeguards meant to survive tampering. Section 59.1 reports Sayash Kapoor and colleagues' marginal-risk framework (Kapoor et al., 2024b). Section 59.4 weighs it against Elizabeth Seger and colleagues' case against open release of some models and Doni Bloomfield and colleagues' account of open genomic weights refined by others within weeks (Seger et al., 2023b; Bloomfield et al., 2024). A hackathon study by Anjali Gopal and colleagues is recalled for its design in Section 59.2.2 (Gopal et al., 2023). Rishi Bommasani and colleagues' policy forum, whose readable record is one sentence, adds that "Different policy proposals may disproportionately affect the innovation ecosystem" (Bommasani et al., 2024).

What release rules add is the range between closed and open. Irene Solaiman, writing from Hugging Face, proposes "six levels of access to generative AI systems: fully closed; gradual or staged access; hosted access; cloud-based or API access; downloadable access; and fully open," framed by "the tension between concentrating power and mitigating risks" (Solaiman, 2023) (preprint). Toby Shevlane's structured access, in which the developer mediates use through an interface, is reported in Section 38.8 (Shevlane, 2022) (preprint), and Shevlane and Allan Dafoe's argument about when publication helps attackers more than defenders in Section 59.4 (Shevlane and Dafoe, 2019) (preprint). Nick Bostrom's 2017 analysis found that "Short-term impacts of increased openness appear mostly socially beneficial in expectation," while the implications of later impacts "are complex" (Bostrom, 2017).

63.7.3 The verification side #

Outside verification needs access, and open weights are access that outsiders control. Section 38.8 reports Stephen Casper and colleagues' argument that black-box access, querying a model's outputs, is insufficient for rigorous audits (Casper et al., 2024a). It also reports that the outside auditing studies it collects built their test models on open models, that developers' terms of service deter good-faith evaluation (Longpre et al., 2024a) (preprint), and that infrastructure now lets outside researchers reach the internals of large open models remotely (Fiotto-Kaufman et al., 2024) (preprint). Andreas Liesenfeld and Mark Dingemanse, reported there too, find that many models called open are "'open weight' at best" (Liesenfeld and Dingemanse, 2024). David Widder, Meredith Whittaker, and Sarah Myers West find that "openness alone does not perturb the concentration of power in AI," while granting that open AI "allows some forms of oversight and experimentation on top of existing models" (Widder et al., 2024). Research outside the largest developers depends on what is released: the Masakhane community's participatory machine translation for African languages released "Benchmarks, models, data, code, and evaluation results" for over 30 languages (Nekoto et al., 2020), and the Aya model, titled an instruction-finetuned open-access multilingual language model, could be read only by its title (Üstün et al., 2024). The release decision therefore sets who can check a model as well as who can misuse it, and no instrument read here weighs the two (inference).

63.7.4 From criminology: displacement under a release rule #

From criminology. A safeguard on one model changes a misuser's situation, not the misuser, and criminology has measured whether blocking one route moves crime to another. Section 48.7.1 reports the measurements: displacement observed in a minority of the observations in evaluations of situational crime prevention and, in the 13 studies that allowed a net assessment, spatial displacement usually smaller than the effect it offset, and a study of police action against markets that sell denial-of-service attacks (Guerette and Bowers, 2009; Collier et al., 2019). Here the question takes a regulatory form (inference). A rule that binds only closed models is a safeguard on one class of targets, and its value depends on whether misuse moves to open models. The booter study's lesson was that removing supply works when supply is concentrated, and a closed frontier model is concentrated supply while a published one is not. As of October 2026 we found no study that measures displacement from closed to open models after a rule or safeguard was introduced.

Section 48.7.2 gives three reasons the street-crime figures reassure less for models, and Section 59.4 a fourth. Regulation adds one more (inference). A rule's reach ends at its jurisdiction, and displacement can cross it. The open-weight model whose cyber capabilities Section 58.6 reports, which a government assessment called "the most cyber-capable open-weight model released to date," came from a company based in China (Center for AI Standards and Innovation, 2026). The AI Act applies to a provider outside the Union that places a model on the Union market (European Parliament and Council of the European Union, 2024b), but once the weights are public, a withdrawal removes only the provider's own offering (Section 63.7.1) (inference).

63.7.5 Where the evidence leaves the question #

Both sides rest mostly on argument. Two measurements would move the question (inference). The first is the lag between a capability appearing in closed models and in open ones, which the 2024 review recommended tracking (National Telecommunications and Information Administration, 2024); the government assessment of September 2026 that Section 58.6 reports is one point on it, for cyber capability (Center for AI Standards and Innovation, 2026). The second is a count of what outside researchers found using open access that developers had missed. As of October 2026 we found no systematic count of that kind. Until both exist, release rules are written under the timing problem the review named: the evidence for restricting a release arrives after it.

Sources cited in Section 63.7 27

European Parliament and Council of the European Union (2024b) Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence
House (2025) Winning the Race: America’s AI Action Plan
Executive Office of the President (2026) Promoting Advanced Artificial Intelligence Innovation and Security
Legislature (2025b) SB-53 Artificial intelligence models: large developers.
Legislature (2026) Transparency and safety requirements for developers of artificial intelligence frontier models (New York Laws of 2026, Chapter 96)
European Parliament and Council of the European Union (2024a) Directive (EU) 2024/2853 (Product Liability Directive)
Kapoor et al. (2024b) On the Societal Impact of Open Foundation Models
National Telecommunications and Information Administration (2024) Dual-Use Foundation Models with Widely Available Model Weights Report
Murphy et al. (2025) Jailbreak-Tuning: Models Efficiently Learn Jailbreak Susceptibility
Seger et al. (2023b) Open-Sourcing Highly Capable Foundation Models: An evaluation of risks, benefits, and alternative methods for pursuing open-source objectives
Bloomfield et al. (2024) AI and biosecurity: The need for governance
Gopal et al. (2023) Will releasing the weights of future large language models grant widespread access to pandemic agents?
Bommasani et al. (2024) Considerations for governing open foundation models
Solaiman (2023) The Gradient of Generative AI Release: Methods and Considerations
Shevlane (2022) Structured access: an emerging paradigm for safe AI deployment
Shevlane and Dafoe (2019) The Offense-Defense Balance of Scientific Knowledge: Does Publishing AI Research Reduce Misuse?
Bostrom (2017) Strategic Implications of Openness in AI Development
Casper et al. (2024a) Black-Box Access is Insufficient for Rigorous AI Audits
Longpre et al. (2024a) A Safe Harbor for AI Evaluation and Red Teaming
Fiotto-Kaufman et al. (2024) NNsight and NDIF: Democratizing Access to Open-Weight Foundation Model Internals
Liesenfeld and Dingemanse (2024) Rethinking open source generative AI: open washing and the EU AI Act
Widder et al. (2024) Why ‘open’ AI systems are actually closed, and why this matters
Nekoto et al. (2020) Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages
Üstün et al. (2024) Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model
Guerette and Bowers (2009) Assessing the Extent of Crime Displacement and Diffusion of Benefits: a Review of Situational Crime Prevention Evaluations*
Collier et al. (2019) Booting the Booters
Center for AI Standards and Innovation (2026) CAISI's Assessment of Z.ai's GLM-5.3 Cyber Capabilities

63.8 What regulation means for delegation #

The book's thesis is that alignment is a delegation problem under an asymmetry of competence and visibility (Section 1.7.1). Regulation is where the public tries to act on that problem through its own agents, and this chapter's record says the asymmetry survives the attempt. The older structures carry over in form (inference). Shavell's determinants explain why states want rules in advance for frontier models, since the largest harms exceed any developer's assets and resist suit, and why they nonetheless leave the choice of precautions to the developer, since the developer knows most. The instrument they settled on, a plan the developer writes and the state enforces, is the management-based regulation of Section 62.5, and the statutes of 2025 and 2026 are its legal form.

What does not carry over is the regulator's footing (inference). Shavell's regulators know enough about fire exits and pasteurization to write rules; Ben-Shahar and Logue's insurers know more than their policyholders because they hold the loss data; the drug regulator could demand a kind of proof that already existed. A frontier regulator has none of these. The steps of Table 63.1 above voluntary cooperation run mainly on information the developer supplies; the AI Act's evaluation power is the only instrument that compels access, and the evaluators that states built depend on developers for it. The developer also answers to several state principals with conflicting aims, federal orders against state statutes, which is the common agency of Section 14.4. The delegation therefore has another link: the public delegates to states, states delegate measurement back to the regulated, and capture theory's informational asymmetry, recalled in Section 63.3.1, describes that second link. The release rules of Section 63.7 decide whether a third party outside both can check either.

Most of the chapter concerns the third relationship, the developer and the public. The liability sections also concern the second, the deployed system and the people it serves: the European directive compensates natural persons harmed by a defective model, and negligence left alone leaves the injured uncompensated or the human operator blamed (Section 63.4). None of the chapter's evidence bears on the first relationship, the training process and its signal, except through what a statute requires developers to test.

63.9 Settled, contested, missing #

Research status Settled, contested, missing

Settled, as mathematics within their models: that strict liability, unlike negligence, controls the activity level, and that a cap on what the injurer can pay leaves care and activity uncontrolled under strict liability, and care under negligence once the cap is low enough, in the model of care and activity derived in Section 63.1.1; and that correlated losses put a floor under the variance of an insurer's losses per policy, however large the pool (Section 63.5.3), which in the beta-binomial model of Figure 63.1 is also a floor under its capital. Shavell's four determinants are an analytic framework, applied by its author to which activities are regulated and which are left to liability (Shavell, 1984).
Reported, not yet replicated: the measurements of displacement in situational crime prevention, which aggregate many evaluations of street crime and have not been made for models (Guerette and Bowers, 2009); one government assessment of how far the most capable open-weight model lagged the frontier, for cyber capability only (Center for AI Standards and Innovation, 2026); one state evaluator's report that it found vulnerabilities in the safeguards of every system it tested (UK AI Security Institute, 2025b).
Contested: whether frontier models should be governed by rules in advance, by liability, or by regulating developers' own risk management; whether compute thresholds are a workable trigger (Heim and Koessler, 2024; Hooker, 2024); whether open weights add more marginal risk than they add verification (Kapoor et al., 2024b; Seger et al., 2023b); whether state evaluators' dependence on developers is capture or productive cooperation; whether federal action in the United States will preempt the state frontier statutes (Executive Office of the President, 2025a). Whether catastrophic AI risk is insurable the evidence does not decide: the argument that catastrophe risk is not inherently uninsurable (Jaffee and Russell, 1997) assumes losses uncorrelated with the wider economy, which has not been tested for AI.
Missing: as of October 2026, any enforcement action we could find under SB 53, the RAISE Act, or the AI Act's provisions for general-purpose models; loss data that would let insurers price AI catastrophe risk; a measurement of displacement from closed to open models after a safeguard or rule was introduced; a study of the expertise channel, relating evaluators' access terms or staff flows to their findings; a systematic count of what outside researchers found with open access that developers missed; and a state evaluator's published finding that a developer had not disclosed something material.
Crux: whether a state can evaluate frontier models without depending on the developers it regulates for access, data, and staff. A record of state evaluators finding and acting on something a developer did not disclose, under access the developer did not control, would settle it; the AI Act's compelled evaluations and evaluations of published weights are the two routes by which such access already exists. This is one of the deciding results of the debate Chapter 70 collects as whether developers' commitments bind.

Sources cited in Section 63.9 10

Shavell (1984) Liability for Harm versus Regulation of Safety
Guerette and Bowers (2009) Assessing the Extent of Crime Displacement and Diffusion of Benefits: a Review of Situational Crime Prevention Evaluations*
Center for AI Standards and Innovation (2026) CAISI's Assessment of Z.ai's GLM-5.3 Cyber Capabilities
UK AI Security Institute (2025b) Frontier AI Trends Report
Heim and Koessler (2024) Training Compute Thresholds: Features and Functions in AI Regulation
Hooker (2024) On the Limitations of Compute Thresholds as a Governance Strategy
Kapoor et al. (2024b) On the Societal Impact of Open Foundation Models
Seger et al. (2023b) Open-Sourcing Highly Capable Foundation Models: An evaluation of risks, benefits, and alternative methods for pursuing open-source objectives
Executive Office of the President (2025a) Ensuring a National Policy Framework for Artificial Intelligence
Jaffee and Russell (1997) Catastrophe Insurance, Capital Markets, and Uninsurable Risks

63.10 Exercises #

Exercise 63.1 Care and activity under three rules

Use the worked case of the derivation in Section 63.1.1: $p_{\text{acc}}(e) = \exp(-e)$ , $U(v) = 2\sqrt{v}$ , and harm $\mathrm{Harm} = \exp(3) \approx 20.1$ . Find the social optimum $(e^\ast, v^\ast)$ , the activity level under negligence, and the choices of an injurer under strict liability whose assets are $W = \exp(1) \approx 2.72$ . For each, compute the expected harm $v\,p_{\text{acc}}(e)\,\mathrm{Harm}$ and the social welfare $S(e, v)$ . Which rule is worse than banning the activity?

Solution

The full cost of one act is $e + \exp(-e)\,\mathrm{Harm}$ , minimized where $\exp(-e)\,\mathrm{Harm} = 1$ , so $e^\ast = \log \mathrm{Harm} = 3$ and the full cost is $4$ . With $U'(v) = 1/\sqrt{v}$ , $v^\ast = 1/16$ . Negligence charges the injurer only its care, 3 per act, so $v_{\mathrm{N}} = 1/9$ . The capped injurer acts as if the harm were $W$ : $\exp(-e)\,W = 1$ gives $e_W = 1$ , a perceived cost of $2$ per act, and $v_W = 1/4$ . Expected harm is $1/16 \approx 0.06$ at the optimum, $1/9 \approx 0.11$ under negligence, and $\tfrac14 \exp(-1)\exp(3) = \exp(2)/4 \approx 1.85$ under capped strict liability, about thirty times the optimum. Welfare is $2\sqrt{v} - v\,(e + \exp(-e)\,\mathrm{Harm})$ : $0.25$ at the optimum, $0.22$ under negligence, and $1 - (1 + \exp(2))/4 \approx -1.10$ under capped strict liability. Banning the activity gives zero, so with these numbers a liability rule against a defendant that cannot pay is worse than a ban, while negligence loses only a little to the optimum. Against the capped injurer, negligence would fail too: due care at 3 per act costs more than its cheapest negligent choice, care 1 at 2 per act, since $W$ is below $\mathrm{Harm}/\exp(1) \approx 7.4$ . A rule in advance that fixed care at 3 and activity at $1/16$ would reach the optimum, if the regulator knew those numbers.

Exercise 63.2 Capital for a pool of deployments

An insurer covers $n_{\text{inst}} = 100$ deployments, each causing a loss of 1 with probability $p_{\text{inst}} = 0.02$ . Find the capital beyond the premium it needs to pay its total loss in all but one year in a hundred when losses are independent, and when they are perfectly correlated. Compare each with the normal approximation of (63.1).

Solution

The premium covers the expected loss, $100 \times 0.02 = 2$ . Independent losses make the total binomial: $\Prob(\text{total} \le 5) \approx 0.985$ and $\Prob(\text{total} \le 6) \approx 0.996$ , so the 99th percentile is 6 and the capital beyond the premium is 4, or 0.04 per policy. The approximation gives $2.33\sqrt{0.02 \times 0.98 / 100} \approx 0.033$ per policy, close. Perfectly correlated losses make the total 0 with probability 0.98 and 100 with probability 0.02; since $0.02 > 0.01$ , the 99th percentile is 100, and the capital beyond the premium is 98, or 0.98 per policy, a pool of one. The approximation gives $2.33\sqrt{0.0196} \approx 0.33$ per policy, a third of what is needed, because a loss that is all or nothing is far from normal. Correlation multiplies the capital by about 25 here, and the multiple grows with the pool.

Exercise 63.3 The ladder and the inspection game

Place each of these on the rows of Table 63.1: the June 2026 executive order, the NIST AI Risk Management Framework, the EU code of practice, California's SB 53, New York's RAISE Act, and the AI Office's fining power. What occupies the top row? Then use the inspection game of Section 14.6 to argue which matters more for deterrence, the size of the EU's fine or its power to compel access for an evaluation.

Solution

The June 2026 order, the NIST framework, and the code of practice sit at cooperation: voluntary access, a voluntary standard, and a compliance route the provider chooses. SB 53 and the RAISE Act, as replaced in 2026, sit at information and money: reports and frameworks, enforced by civil penalties. The AI Office's fines sit at money, and its withdrawal power at stopping a product. The top row, stopping development, is empty in every jurisdiction. In the inspection game, a penalty deters only through the probability that a violation is detected, and a regulator that cannot commit to inspecting gains nothing from a harsher penalty, since the penalty only lowers how often it inspects. Most instruments detect violations mainly through the developer's own reports, with complaints, alerts, public reports, and protected employees as the other routes, so the developer shapes much of the detection probability. Compelled access for an evaluation is the one route by which the regulator sets it. On this reading it matters more than the size of the fine, which is multiplied by a detection probability the fine does not change.

References

42, I. J. 1. (2023). ISO/IEC 42001:2023 Information technology - Artificial intelligence - Management system. International Organization for Standardization. non-peer-reviewed Cited in §63.2
Abbott, R. B. (2016). The Reasonable Computer: Disrupting the Paradigm of Tort Liability. SSRN Electronic Journal. working paper Cited in §63.4
AI, C. (2026). State of AI Safety in China (2026). Concordia AI. non-peer-reviewed Cited in §63.2
Anderljung, M., Barnhart, J., Korinek, A., Leung, J., O'Keefe, C., Whittlestone, J., … Wolf, K. (2023). Frontier AI Regulation: Managing Emerging Risks to Public Safety. arXiv. preprint Cited in §63.1
Ayres, I., and Balkin, J. M. (2024). The Law of AI is the Law of Risky Agents without Intentions. Elsevier BV. preprint Cited in §63.4
Ayres, I., and Braithwaite, J. (1992). Responsive Regulation. Oxford University PressNew York, NY. Cited in §63.3
Babic, B., Gerke, S., Evgeniou, T., and Cohen, I. G. (2019). Algorithms on regulatory lockdown in medicine. Science. Cited in §63.6
Bayern, S. (2016). The Implications of Modern Business–Entity Law for the Regulation of Autonomous Systems. European Journal of Risk Regulation. Cited in §63.4
Ben-Shahar, O., and Logue, K. (2012). Outsourcing Regulation: How Insurance Reduces Moral Hazard. Michigan Law Review. Cited in §63.5
Bengio, Y., Clare, S., Prunkl, C., and Murray, M. (2026). International AI Safety Report 2026. International AI Safety Report. non-peer-reviewed Cited in §63.2
Black, J. (2001). Decentring Regulation: Understanding the Role of Regulation and Self-Regulation in a 'Post-Regulatory' World. Current Legal Problems. Cited in §63.3
Black, J., and Baldwin, R. (2010). Really Responsive Risk-Based Regulation. Law & Policy. Cited in §63.3
Bloomfield, D., Pannu, J., Zhu, A. W., Ng, M. Y., Lewis, A., Bendavid, E., … Inglesby, T. (2024). AI and biosecurity: The need for governance. Science. Cited in §63.7
Bommasani, R., Kapoor, S., Klyman, K., Longpre, S., Ramaswami, A., Zhang, D., … Liang, P. (2024). Considerations for governing open foundation models. Science. Cited in §63.7
Bostrom, N. (2017). Strategic Implications of Openness in AI Development. Global Policy. Cited in §63.7
Bradford, A. (2020). The Brussels Effect. Oxford University PressNew York. Cited in §63.3
Bryson, J. J., Diamantis, M. E., and Grant, T. D. (2017). Of, for, and by the people: the legal lacuna of synthetic persons. Artificial Intelligence and Law. Cited in §63.4
Calabresi, G. (1970). The costs of accidents: a legal and economic analysis. Yale University Press. Cited in §63.1
(2013). Preventing Regulatory Capture. Cambridge University Press. Cited in §63.3
Carpenter, D. (2014). Reputation and Power. Princeton University Press. Cited in §63.6
Casper, S., Ezell, C., Siegmann, C., Kolt, N., Curtis, T. L., Bucknall, B., … Hadfield-Menell, D. (2024a). Black-Box Access is Insufficient for Rigorous AI Audits. The 2024 ACM Conference on Fairness, Accountability, and Transparency. Cited in §63.7
Center for AI Standards and Innovation (2026). CAISI's Assessment of Z.ai's GLM-5.3 Cyber Capabilities. NIST. non-peer-reviewed Cited in §63.2 §63.7 §63.9
Chan, A., Wei, K., Huang, S., Rajkumar, N., Perrier, E., Lazar, S., Hadfield, G. K., and Anderljung, M. (2025). Infrastructure for AI Agents. arXiv. preprint Cited in §63.3
Clark, J., and Hadfield, G. K. (2019). Regulatory Markets for AI Safety. arXiv. preprint Cited in §63.3
Collier, B., Thomas, D. R., Clayton, R., and Hutchings, A. (2019). Booting the Booters. Proceedings of the Internet Measurement Conference. Cited in §63.7
Cyberspace Administration of China, National Development and Reform Commission, Ministry of Education, Ministry of Science and Technology, Ministry of Industry and Information Technology, Ministry of Public Security, and National Radio and Television Administration (2023). 生成式人工智能服务管理暂行办法 (Interim Measures for the Management of Generative Artificial Intelligence Services). Cyberspace Administration of China, Order No. 15. non-peer-reviewed Cited in §63.2
Dafoe, A. (2023). AI Governance. The Oxford Handbook of AI Governance. Cited in §63.1
Darrow, J. J., Avorn, J., and Kesselheim, A. S. (2020). FDA Approval and Regulation of Pharmaceuticals, 1983-2018. JAMA. Cited in §63.6
Department for Science, I., and Technology (2023). Introducing the AI Safety Institute. UK Government (GOV.UK). non-peer-reviewed Cited in §63.2 §63.3
Diamantis, M. (2019). The Extended Corporate Mind: When Corporations Use AI to Break the Law. SSRN Electronic Journal. working paper Cited in §63.4
Elish, M. C. (2019). Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction. Engaging Science, Technology, and Society. Cited in §63.4
European Commission, A. O. (2025). The General-Purpose AI Code of Practice. European Commission. non-peer-reviewed Cited in §63.2 §63.3
European Parliament and Council of the European Union (2024a). Directive (EU) 2024/2853 (Product Liability Directive). Official Journal of the European Union. non-peer-reviewed Cited in §63.1 §63.4 §63.7
European Parliament and Council of the European Union (2024b). Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence. Official Journal of the European Union (OJ L 2024/1689); verified on the European Commission AI Act page. non-peer-reviewed Cited in §63.2 §63.3 §63.6 §63.7
Executive Office of the President (2023). Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence. Federal Register, 88 FR 75191 (Executive Order 14110). non-peer-reviewed Cited in §63.2
Executive Office of the President (2025a). Ensuring a National Policy Framework for Artificial Intelligence. Federal Register, 90 FR 58499 (Executive Order 14365). non-peer-reviewed Cited in §63.2 §63.3 §63.9
Executive Office of the President (2025c). Initial Rescissions of Harmful Executive Orders and Actions. Federal Register. non-peer-reviewed Cited in §63.2
Executive Office of the President (2025d). Removing Barriers to American Leadership in Artificial Intelligence. Federal Register, 90 FR 8741 (Executive Order 14179). non-peer-reviewed Cited in §63.2
Executive Office of the President (2026). Promoting Advanced Artificial Intelligence Innovation and Security. Federal Register, 91 FR 34565 (Executive Order 14409). non-peer-reviewed Cited in §63.2 §63.3 §63.6 §63.7
Fiotto-Kaufman, J., Loftus, A. R., Todd, E., Brinkmann, J., Pal, K., Troitskii, D., … Bau, D. (2024). NNsight and NDIF: Democratizing Access to Open-Weight Foundation Model Internals. arXiv. preprint Cited in §63.7
Gopal, A., Helm-Burger, N., Justen, L., Soice, E. H., Tzeng, T., Jeyapragasan, G., … Esvelt, K. M. (2023). Will releasing the weights of future large language models grant widespread access to pandemic agents? arXiv. preprint Cited in §63.7
Government of Japan (2025). Act on Promotion of Research and Development, and Utilization of Artificial Intelligence-related Technology. Japanese Law Translation. non-peer-reviewed Cited in §63.2
Guerette, R. T., and Bowers, K. J. (2009). Assessing the Extent of Crime Displacement and Diffusion of Benefits: a Review of Situational Crime Prevention Evaluations*. Criminology. Cited in §63.7 §63.9
Hacker, P., Engel, A., and Mauer, M. (2023). Regulating ChatGPT and other Large Generative AI Models. 2023 ACM Conference on Fairness Accountability and Transparency. Cited in §63.2
Hadfield, G. K. (2026). Legal infrastructure for transformative AI governance. Proceedings of the National Academy of Sciences. Cited in §63.3
Hadfield, G. K., and Clark, J. (2023). Regulatory Markets: The Future of AI Governance. arXiv. preprint Cited in §63.3
Heim, L., and Koessler, L. (2024). Training Compute Thresholds: Features and Functions in AI Regulation. arXiv. preprint Cited in §63.2 §63.9
Hooker, S. (2024). On the Limitations of Compute Thresholds as a Governance Strategy. arXiv. preprint Cited in §63.2 §63.3 §63.9
House, T. W. (2025). Winning the Race: America’s AI Action Plan. The White House. non-peer-reviewed Cited in §63.2 §63.3 §63.7
Jaffee, D. M., and Russell, T. (1997). Catastrophe Insurance, Capital Markets, and Uninsurable Risks. The Journal of Risk and Insurance. Cited in §63.5 §63.9
Joint California Policy Working Group on AI Frontier Models (2025). The California Report on Frontier AI Policy. Office of the Governor of California. non-peer-reviewed Cited in §63.2
Kapoor, S., Bommasani, R., Klyman, K., Longpre, S., Ramaswami, A., Cihon, P., … Narayanan, A. (2024b). On the Societal Impact of Open Foundation Models. arXiv. preprint Cited in §63.7 §63.9
Korea, R. o. (2025). Framework Act on the Development of Artificial Intelligence and Establishment of Trust. Center for Security and Emerging Technology (translation). non-peer-reviewed Cited in §63.2
Kunreuther, H., Hogarth, R., and Meszaros, J. (1993). Insurer ambiguity and market failure. Journal of Risk and Uncertainty. Cited in §63.5
Laffont, J.-J., and Tirole, J. (1991). The Politics of Government Decision-Making: A Theory of Regulatory Capture. The Quarterly Journal of Economics. Cited in §63.3
Legislature, C. S. (2024). SB-1047 Safe and Secure Innovation for Frontier Artificial Intelligence Models Act. California Legislative Information (enrolled 09/03/24; vetoed). non-peer-reviewed Cited in §63.2
Legislature, N. Y. S. (2025a). Responsible AI Safety and Education Act (New York RAISE Act). non-peer-reviewed Cited in §63.2
Legislature, C. S. (2025b). SB-53 Artificial intelligence models: large developers. California Legislative Information (chaptered 09/29/25). non-peer-reviewed Cited in §63.2 §63.3 §63.6 §63.7
Legislature, N. Y. S. (2026). Transparency and safety requirements for developers of artificial intelligence frontier models (New York Laws of 2026, Chapter 96). non-peer-reviewed Cited in §63.1 §63.2 §63.3 §63.6 §63.7
Liesenfeld, A., and Dingemanse, M. (2024). Rethinking open source generative AI: open washing and the EU AI Act. The 2024 ACM Conference on Fairness, Accountability, and Transparency. Cited in §63.7
Lior, A. (2022). Insuring AI: The Role of Insurance in Artificial Intelligence Regulation. Harvard Journal of Law & Technology. Cited in §63.5
Longpre, S., Kapoor, S., Klyman, K., Ramaswami, A., Bommasani, R., Blili-Hamelin, B., … Henderson, P. (2024a). A Safe Harbor for AI Evaluation and Red Teaming. arXiv. preprint Cited in §63.7
Murphy, B., Bowen, D., Mohammadzadeh, S., Tseng, T., Broomfield, J., Gleave, A., and Pelrine, K. (2025). Jailbreak-Tuning: Models Efficiently Learn Jailbreak Susceptibility. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Cited in §63.7
National Governance Committee for New Generation Artificial Intelligence (2021). 新一代人工智能伦理规范 (Ethical Norms for New Generation Artificial Intelligence). Ministry of Science and Technology of the People's Republic of China. non-peer-reviewed Cited in §63.2
National Telecommunications and Information Administration (2024). Dual-Use Foundation Models with Widely Available Model Weights Report. U.S. Department of Commerce, NTIA. non-peer-reviewed Cited in §63.7
Nekoto, W., Marivate, V., Matsila, T., Fasubaa, T., Fagbohungbe, T., Akinola, S. O., … Bashir, A. (2020). Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages. Findings of the Association for Computational Linguistics: EMNLP 2020. Cited in §63.7
(2010). The Oxford Handbook of Regulation. Oxford University Press. Cited in §63.3
Pauly, M. V. (1982). The economics of moral hazard: comment. Readings in the Economics of Contract Law. Cited in §63.5
Platt, R., Brown, J. S., Robb, M., McClellan, M., Ball, R., Nguyen, M. D., and Sherman, R. E. (2018). The FDA Sentinel Initiative — An Evolving National Resource. New England Journal of Medicine. Cited in §63.6
Reuel, A., Bucknall, B., Casper, S., Fist, T., Soder, L., Aarne, O., … Trager, R. (2024b). Open Problems in Technical AI Governance. arXiv. preprint Cited in §63.1
Roberts, H., Cowls, J., Morley, J., Taddeo, M., Wang, V., and Floridi, L. (2021). The Chinese approach to artificial intelligence: an analysis of policy, ethics, and regulation. AI & SOCIETY. Cited in §63.2
Salib, P., and Goldstein, S. (2024). AI Rights for Human Safety. Elsevier BV. preprint Cited in §63.4
Scherer, M. U. (2015). Regulating Artificial Intelligence Systems: Risks, Challenges, Competencies, and Strategies. Elsevier BV. preprint Cited in §63.1
Seger, E., Dreksler, N., Moulange, R., Dardaman, E., Schuett, J., Wei, K., … Gupta, A. (2023b). Open-Sourcing Highly Capable Foundation Models: An evaluation of risks, benefits, and alternative methods for pursuing open-source objectives. arXiv. preprint Cited in §63.7 §63.9
Selbst, A. D. (2020). Negligence and AI’s Human Users. Boston University Law Review. Cited in §63.4
Shavell, S. (1979). On Moral Hazard and Insurance. The Quarterly Journal of Economics. Cited in §63.5
Shavell, S. (1980). Strict Liability versus Negligence. The Journal of Legal Studies. Cited in §63.1
Shavell, S. (1984). Liability for Harm versus Regulation of Safety. The Journal of Legal Studies. Cited in §63.1 §63.5 §63.9
Sheehan, M. (2023). China’s AI Regulations and How They Get Made. Carnegie Endowment for International Peace. non-peer-reviewed Cited in §63.2
Shevlane, T. (2022). Structured access: an emerging paradigm for safe AI deployment. arXiv. preprint Cited in §63.7
Shevlane, T., and Dafoe, A. (2019). The Offense-Defense Balance of Scientific Knowledge: Does Publishing AI Research Reduce Misuse? arXiv. preprint Cited in §63.7
Siegmann, C., and Anderljung, M. (2022). The Brussels Effect and Artificial Intelligence: How EU regulation will impact the global AI market. arXiv. preprint Cited in §63.3
Smuha, N. A. (2021). From a ‘race to AI’ to a ‘race to AI regulation’: regulatory competition for artificial intelligence. Law, Innovation and Technology. Cited in §63.3
Solaiman, I. (2023). The Gradient of Generative AI Release: Methods and Considerations. arXiv. preprint Cited in §63.7
Solum, L. B. (1992). Legal Personhood for Artificial Intelligences. North Carolina Law Review. Cited in §63.4
Stein, M., Bernardi, J., and Dunlop, C. (2024). The Role of Governments in Increasing Interconnected Post-Deployment Monitoring of AI. arXiv. preprint Cited in §63.6
Stigler, G. J. (1971). The Theory of Economic Regulation. The Bell Journal of Economics and Management Science. Cited in §63.3
Stilgoe, J., Owen, R., and Macnaghten, P. (2013). Developing a framework for responsible innovation. Research Policy. Cited in §63.3
Sunstein, C. R. (2003). Beyond the Precautionary Principle. University of Pennsylvania Law Review. Cited in §63.3
Sykes, A. O. (1984). The Economics of Vicarious Liability. The Yale Law Journal. Cited in §63.4
Tabassi, E. (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). National Institute of Standards and Technology (U.S.). non-peer-reviewed Cited in §63.2
Tutt, A. (2016). An FDA for Algorithms. SSRN Electronic Journal. Cited in §63.6
UK AI Security Institute (2025a). AISI Research Agenda. UK AI Security Institute. non-peer-reviewed Cited in §63.2 §63.3
UK AI Security Institute (2025b). Frontier AI Trends Report. UK AI Security Institute. non-peer-reviewed Cited in §63.2 §63.9
United States Congress (1962). Drug Amendments of 1962 (Kefauver-Harris Amendments). United States Statutes at Large 76. non-peer-reviewed Cited in §63.6
Üstün, A., Aryabumi, V., Yong, Z., Ko, W.-Y., D’souza, D., Onilude, G., … Hooker, S. (2024). Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Cited in §63.7
Veale, M., and Zuiderveen Borgesius, F. (2021). Demystifying the Draft EU Artificial Intelligence Act — Analysing the good, the bad, and the unclear elements of the proposed approach. Computer Law Review International. Cited in §63.2
Vladeck, D. C. (2014). Machines Without Principals: Liability Rules and Artificial Intelligence. Washington Law Review. Cited in §63.4
Weil, G. (2024). Closing the AI Accountability Gap: Strict Liability and Punitive Damages for Advanced Artificial Intelligence. Elsevier BV. preprint Cited in §63.4 §63.5
Whittlestone, J., and Clark, J. (2021). Why and How Governments Should Monitor AI Development. arXiv. preprint Cited in §63.1
Widder, D. G., Whittaker, M., and West, S. M. (2024). Why ‘open’ AI systems are actually closed, and why this matters. Nature. Cited in §63.7

Regulation, Liability, and Standards

63.1 Liability or rules in advance #

63.1.1 What strict liability controls that negligence does not #

63.1.2 When rules in advance do better #

63.1.3 What the determinants say about frontier models #

63.2 Statutes and orders as of October 2026 #

63.2.1 The European Union #

63.2.2 The United States: orders and standards #

63.2.3 The United States: the states #

63.2.4 China, Japan, and Korea #

63.2.5 State evaluators and standards #

63.2.6 What triggers a duty #

63.3 What regulatory theory predicts #

63.3.1 Capture through expertise #

63.3.2 Escalation #

63.3.3 Targeting by risk #

63.3.4 New designs #

63.3.5 Competition among jurisdictions #

63.3.6 Where the theory stops #

63.4 Liability for AI harms #

63.4.1 Whom the law holds #

63.4.2 Product or conduct #

63.4.3 Catastrophic harm #

63.4.4 Personhood as a liability question #

63.4.5 Where liability stops #

63.5 Insurance as a regulator #

63.5.1 Insurers as private regulators #

63.5.2 When a risk is called uninsurable #

63.5.3 Why correlation breaks pooling #

63.5.4 Insuring AI #

63.5.5 Where the analogy stops #

63.6 What drug and device regulation teaches #

63.6.1 The regime and its counterparts #

63.6.2 Where the analogy stops #

63.7 Release, open weights, and outside access #

63.7.1 What the instruments say about release #

63.7.2 The misuse side #

63.7.3 The verification side #

63.7.4 From criminology: displacement under a release rule #

63.7.5 Where the evidence leaves the question #

63.8 What regulation means for delegation #

63.9 Settled, contested, missing #

63.10 Exercises #

Further reading #

References