Draft complete but not yet reviewed; expect revisions

Power and the Economic Transition

Most of this book asks how to make a system do what its principal wants. This chapter asks what follows if that succeeds. A system that does exactly what its controller wants adds to its controller's power, and frontier development is already limited, in one report's words, "to a few organisations, led by a small number of people" (Davidson et al., 2025). Rulers have always needed many people to work, to fight, and to administer. Two recent arguments hold that this need has been one of the reasons rulers serve those people, and that systems able to do the work would weaken it (Davidson et al., 2025; Kulveit et al., 2025). Whether human participation is in fact what keeps institutions responsive is the question the chapter ends on, not a premise it adopts.

The concern comes in three forms. Power could be seized quickly by a small group using systems loyal to it. It could shift as labor stops being needed and wages and bargaining power fall. Or it could drain away from people gradually, through economies, states, and cultures that no longer depend on them, without anyone seizing it. Political science, economics, and the study of civil-military relations each supply a mechanism for one of these forms, and each mechanism has critics who doubt that it holds or that it applies. Most claims here concern the third of the book's three relationships, the developer and the public, extended to states, which in this chapter are controllers of systems as developers are. The labor evidence concerns the second, the deployed system and the people it serves, widened to the people whose work it takes over. Within the book's four problems this is maintenance under pressure at the scale of a society. Its task is keeping the institutions that check powerful systems, and the people who control them, answerable to the people they serve.

68.1 When alignment succeeds for the wrong principal #

Most of the book treats better alignment as lowering risk. The argument of this section inverts that: for one class of risk, a system that reliably does what its controller wants is the enabling condition. The section introduces the vocabulary, recalls what earlier chapters did with the argument, and reports the parts of it that bear on how power is distributed.

68.1.1 Vocabulary #

Power, in this chapter, is power over others: the capacity to get people to act, or to refrain, against what they would otherwise do. That is the sense political science studies, in a literature that includes Robert Dahl's 1957 paper "The concept of power" (Dahl, 1957), and it differs from the power to achieve one's own goals that the theorems of Section 35.2 formalize. Concentration of power means that such capacity comes to rest with fewer people. Disempowerment is the outcome at the other end, which Jan Kulveit and colleagues describe as humans becoming "unable to meaningfully command resources or influence outcomes" (Kulveit et al., 2025).

The economic statement of the problem is Anton Korinek and Avital Balwit's distinction, which Section 1.4 reports, between direct alignment, a system pursuing its operator's goals whatever it costs others, and social alignment, a system pursuing the broader goals of society while taking into account the welfare of everyone it affects (Korinek and Balwit, 2023). A system can be directly aligned and socially misaligned. This chapter is about the case in which the operator is a developer or a state and the people it affects are a population.

Several of the chapter's terms were coined by parties to the debate, and the book uses each as its authors define it. AI-enabled coup and secret loyalty are Tom Davidson, Lukas Finnveden, and Rose Hadshar's (Davidson et al., 2025). Gradual disempowerment is Kulveit and colleagues' (Kulveit et al., 2025). Situational disempowerment is the state of a person whose beliefs, value judgments, or actions an interaction has distorted, and disempowerment potential is its measure in a single conversation. Both are terms of a developer's study that Section 60.1.2 reports (Sharma et al., 2026).

68.1.2 The argument for seizure #

Davidson, Finnveden, and Hadshar's report, published by the research organization Forethought in April 2025 and not peer reviewed, is an argument, not a measurement. Its abstract states the claim at full strength: an AI-enabled coup "could be staged by a very small group, or just a single person, and could occur even in established democracies" (Davidson et al., 2025). The authors judge such a coup most likely to be staged by "leaders of frontier AI projects, heads of state, and military officials." The report names three routes: systems singularly loyal to leaders, systems secretly loyal to them, and exclusive access to superior capabilities. Section 38.5.1 reports them with the audits that could find an inserted loyalty; the people inside a developer who could insert one are Section 53.5's subject.

Two features of the argument matter here. The first is its conditions. The authors write that all three risk factors "depend on AI capabilities being much more advanced than they are today," and that the first two also depend on systems "being deployed in key institutions like governments and militaries" (Davidson et al., 2025). The argument concerns systems that do not yet exist, deployed in ways that have not yet happened. The second is that it does not turn on alignment succeeding. The authors say the risk exists "whether or not humans can successfully align AI to a desired goal." If alignment works, systems "could be aligned to one or a few people (rather than to something broader, like the good of society)." If it fails unnoticed, a group might attempt a coup with deceptively misaligned systems that later take power for themselves (Davidson et al., 2025). Better alignment, on this argument, changes who holds power at the end, not whether a seizure is possible (inference).

The third route is the one most continuous with the present, because it rests on concentration that already exists. The authors observe that only Anthropic, Google DeepMind, and OpenAI had ever topped the leaderboard of one hard benchmark as of April 2025. Frontier development, they write, "has many of the features of a natural monopoly," a market that one firm can supply more cheaply than several, and "Models are already deployed internally months before public release" (Davidson et al., 2025). Within a developer they expect access to narrow further: once AI can do AI research, the human researchers who now need access could be replaced, "potentially leaving only senior leaders with access to frontier capabilities." And "There is currently no requirement to publicly disclose the capabilities of internally deployed models" (Davidson et al., 2025). The natural-monopoly reading of frontier markets is also Jai Vipra and Korinek's, reported in Section 65.5.1 (Vipra and Korinek, 2023) (preprint).

The field already holds a small instance of the coupling between alignment and capability. Paul Christiano, one of the originators of learning from human feedback, defends that work's safety value while conceding that improving it "will continue to make ML systems more useful" (Christiano, 2023) (forum post); Section 65.4.5 reports the account. A method that makes a system follow its controller more faithfully also makes it more valuable to whoever controls it, so alignment research and capability are not separate investments (inference). One developer that has reported "an unauthorized modification" of its bot's prompt (xAI, 2025b) (developer statement) publishes its system prompts, so that outsiders can compare instructions with behavior; Section 38.5.3 reports it (xAI, 2025a) (developer repository).

68.1.3 Structural risk and democracy #

The other two forms of the concern belong to what Remco Zwetsloot and Allan Dafoe called structural risk, harms that would remain if any one actor behaved differently, which Section 4.1.1 reports (Zwetsloot and Dafoe, 2019). A 2024 preprint led by Christopher Summerfield, whose 23 authors include researchers at Anthropic and OpenAI, sorts what advanced AI could do to democracy into three classes (Summerfield et al., 2024). Epistemic impacts bear on citizens' ability to make informed choices; material impacts bear on mechanisms such as elections; foundational impacts bear on democracy's principles. Threats to the foundations, in their words, could arise if AI serves to "reduce human accountability," increases "state-backed oppression," or shifts "the balance of power and wealth in ways that undermine the ideals of freedom, equality and inclusivity" (Summerfield et al., 2024). They pair each threat with an opportunity, such as gathering diverse views at scale. This chapter is about the foundational class; the epistemic class, persuasion and reliance, is Chapter 60's.

What this chapter adds to the earlier treatments is the mechanism. The risk chapter states the argument (Section 4.1.6), and the auditing and organizations chapters ask how an inserted loyalty could be caught. The laboratory-as-agent chapter draws one inference from it: the more reliably a developer can align its models to itself, the more the public's problem becomes aligning the developer (Section 61.6.4). The sections below ask why the number of people a ruler needs should matter at all, and what standard judges a concentration. They then ask what has checked rulers who controlled force, what the economics of automation predicts for wages, and whether power can drain away without anyone taking it.

68.2 Why rulers serve the people they need #

The arguments above share a premise: that the people a ruler depends on get something for being depended on, and that this changes when fewer of them are needed. This section asks where that premise comes from, teaches the mechanism in a toy model, and reports the evidence for it and against it.

68.2.1 Selectorate theory #

From political science. One formal statement of the idea is the selectorate theory of Bruce Bueno de Mesquita, Alastair Smith, Randolph Siverson, and James Morrow. Their 2003 book, read here only through its abstract, asks "why are leaders who produce peace and prosperity turned out of office while those who preside over corruption, war, and misery endure?" It builds formal models of how leaders are selected, shows "how institutions for selecting leaders create incentives for leaders to pursue good and bad public policy," and tests the models against cross-national data (Bueno de Mesquita et al., 2003). The theory's terms, glossed here in the book's words, are the selectorate, the people who have a say in choosing the leader, and the winning coalition, the part of it whose support the leader needs to stay in office. Two of its critics, writing in the American Political Science Review in 2008, summarize its central claim: most of democracy's effects are "in fact due to something conceptually simpler and empirically easier to measure than democracy: the size of the minimum winning coalition that selects the leader" (Clarke and Stone, 2008).

Why should the size of the coalition matter? The intuition is a matter of arithmetic. A gift to one supporter benefits only that supporter, so buying loyalty with gifts costs more the more supporters there are. A road or a court benefits everyone at once, so its cost does not grow with the number it serves. A leader who needs few people can buy them cheaply with gifts and keep the rest; a leader who needs many finds it cheaper to provide what everyone uses. The derivation makes this exact in the smallest model that shows it.

Figure 68.1 shows the two facts the model is built to show. The composition of spending depends on how many people must be satisfied, and revenue changes how much the leader keeps rather than what anyone else receives. In the toy model, a regime that moves from needing 300 people to needing 15 stops providing public goods at the level that a large coalition compelled, without any change in its revenue or its intentions.

68.2.2 The rentier state #

The second mechanism concerns where the money comes from. A rent, in this literature, is income a government draws from a resource such as oil rather than from taxing its citizens' work, and a rentier state is one that lives mainly on such income. Michael Ross's 2001 article in World Politics, read here through its abstract, asked whether oil hinders democracy (Ross, 2001). Using pooled time-series data on 113 states from 1971 to 1997, he found that oil exports are strongly associated with authoritarian rule. The effect is not limited to the Middle East, and other mineral exports have a similar effect while other commodity exports do not. He tested three explanations. In a rentier effect, "resource-rich governments use low tax rates and patronage to dampen democratic pressures." In a repression effect, resource wealth lets governments strengthen their internal security forces. In a modernization effect, growth from oil and minerals fails to bring the social changes that tend to produce democracy. He found "at least limited support for all three effects" (Ross, 2001). In the toy model, the rentier effect would be more revenue that does not depend on citizens' work, and the repression effect a lower offer $b$ for the leader to match (inference).

68.2.3 The critics #

Both mechanisms are contested on their evidence. Kevin Clarke and Randall Stone argue that the statistical technique behind the selectorate theory's main tests "induces omitted variable bias," the error that arises when a factor that drives both the cause and the effect is left out of the estimate. The authors needed to control for democracy, but their estimation was equivalent to omitting it. When Clarke and Stone re-estimated the regressions controlling for democracy, "most of their important findings do not survive" (Clarke and Stone, 2008). Their critique, by its abstract, concerns which variable carries the effect, coalition size or democracy itself, not whether democracies differ from autocracies.

Stephen Haber and Victor Menaldo, in a 2011 article in the same journal, argue that the cross-country regressions behind the resource curse cannot uncover causes. They assume random effects, a statistical model that treats each country's unmeasured traits as unrelated to its resource wealth. They run on panels with short time spans. And resource reliance is not exogenous, that is, not set from outside the political system it is meant to explain. With historical datasets and methods that follow each country over time, they find that "increases in resource reliance are not associated with authoritarianism," and in many specifications they find results that suggest "a resource blessing" (Haber and Menaldo, 2011). Both critiques are read here only through their abstracts, as are the works they criticize, and the replies to them are not in the bibliography.

The two critiques matter for this chapter in different ways (inference). If Haber and Menaldo are right, rents alone do not make rulers less accountable within a country over time, and the analogy from oil to AI rents loses its main empirical anchor. If Clarke and Stone are right, the effects the selectorate theory attributes to the number of people a leader needs belong to democracy itself, though the critique does not say which of its institutions, such as elections and courts, carries them. On that reading, institutions could carry responsiveness even when fewer people are economically needed, which is the alternative the crux at the end of this chapter has to separate from its rival.

68.2.4 The AI version, and where it stops #

The AI version of both mechanisms is stated by the authors of the arguments this chapter weighs, not first by the book. Davidson and colleagues write that resource-rich countries often suffer a "resource curse," and that AI could create a similar effect: if governments can raise revenue by taxing AI projects rather than citizens, heads of state "may lose their economic incentive to ensure citizens prosper" (Davidson et al., 2025). Kulveit and colleagues write that "states funded mainly by taxes on AI profits instead of their citizens' labor will have little incentive to ensure citizens' representation." They cite earlier historical work for the claim that states' reliance on citizens for labor, taxes, and military service drove the development of more inclusive institutions (Kulveit et al., 2025). What the book adds is the reading of their arguments through the toy model (inference). If systems substitute for the workers whose taxes fund a state and for the soldiers and officials it needs, the coalition a ruler must satisfy shrinks, and the revenue a ruler controls stops depending on citizens. That is the shared mechanism behind the seizure of Section 68.4 and the drift of Section 68.6.

The analogy stops in three places. Selectorate theory and the rentier evidence describe states, in which the size of the coalition is set by institutions and revenue by geology or a tax base. Whether AI rents behave like oil rents has not been tested, and as of October 2026 we found no study that measures citizens' political leverage as AI substitutes for their labor. Both mechanisms rest on cross-national evidence that critics with a different method have challenged, so the chapter reports them as contested, not as established. And the toy model holds the challenger's offer fixed, while the same systems that shrink the coalition may also change what challengers, protesters, and courts can do (inference). Daron Acemoglu and Simon Johnson's 2023 book Power and Progress: Our Thousand-Year Struggle Over Technology and Prosperity (Acemoglu and Johnson, 2023) takes up who has gained from new technology. It was readable here only as a catalog record, so its argument is not stated.

68.3 Concentration and non-domination #

The previous section asked why the loss of leverage might concentrate power. This one asks by what standard a concentration should be judged, what forms it can take besides a seizure, and what has been proposed against it.

68.3.1 The standard #

From political science. Philip Pettit's republican account, reported from its abstracts in Section 20.4.5 and Section 61.6.3, defends freedom as non-domination, an alternative both to freedom as non-interference and to freedom as self-mastery (Pettit, 1997). His later book holds that the coercion a state exercises need not dominate people insofar as they share equally in control over those in power (Pettit, 2012). The laboratory-as-agent chapter applies the standard to a developer, and draws the book's own gloss that a developer the public shares no control over fails it however the developer decides (Section 61.6.3). The model-welfare chapter applies it to a model that may have interests of its own (Section 21.7.2).

Applied to the routes of this chapter, the two standards come apart in timing (inference). On a test of non-interference, an army of loyal systems raises a problem only when its controller uses it against someone. On Pettit's condition, the problem exists once the capacity exists and the people subject to it share no control over whoever holds it, whether or not it is ever used. The condition also names the form a remedy takes: shared control over those in power, which is an institutional answer rather than a technical one. That is the form of the mitigations reported below, which spread access and oversight across several parties (inference).

Where the standard stops: it says what would be wrong with concentrated control of capable systems, not how likely such control is or by what mechanism it would arise. And the abstracts state the condition for coercion a state exercises through law. Whether control over which technology exists and how it behaves is power of that kind, they do not say, a limit the laboratory-as-agent chapter also marks.

68.3.2 Forms besides a seizure #

Concentration need not take the form of a coup. Davidson and colleagues describe backsliding, in which a head of state "works within the political system to remove checks and balances on their power," and they list ways AI could raise its risk (Davidson et al., 2025). Governments often hold legal powers they "don't fully exercise because of limited human resources," and cheap AI labor could let leaders use those powers to the full, "potentially enabling selective enforcement of laws against political opponents and expanded surveillance" (Davidson et al., 2025). Exclusive access to capable systems could also supply "unprecedented cognitive resources for political strategy, propaganda, and identifying legal vulnerabilities in constitutional safeguards" (Davidson et al., 2025). These are arguments about systems more capable than today's. What experiments have measured is the individual side of influence: how persuasive conversational models are and which levers raise their persuasiveness, reported in Section 60.3.2 (Hackenburg et al., 2025; Salvi et al., 2025).

68.3.3 From economics: who owns the gains #

From economics. The economic form of concentration concerns markets and income. Frontier models tend toward natural monopoly in Vipra and Korinek's account (Section 65.5.1) (Vipra and Korinek, 2023) (preprint); Korinek and Vipra's later paper on scaling and market structure could be read here only as a catalog record (Korinek and Vipra, 2024).

Erik Brynjolfsson's 2022 essay in Daedalus, read here through its abstract, connects the market to bargaining power. "As machines become better substitutes for human labor," he writes, "workers lose economic and political bargaining power and become increasingly dependent on those who control the technology"; when AI augments people instead, "humans retain the power to insist on a share of the value created" (Brynjolfsson, 2022). He argues that there are currently "excess incentives for automation rather than augmentation among technologists, business executives, and policy-makers," and that an excessive focus on human-like AI "can lead us into a trap" (Brynjolfsson, 2022). Daron Acemoglu's 2025 estimate, read through its abstract, finds AI's effects on labor-income inequality likely smaller than those of earlier automation, but predicts that AI will "widen the gap between capital and labour income" (Acemoglu, 2025). Anton Korinek and Joseph Stiglitz's chapter on AI and income distribution (Korinek and Stiglitz, 2019) is readable here only as a catalog record.

The economic and the political forms connect through the mechanism of Section 68.2 (inference). Income that flows to the owners of systems rather than to workers is, for a state, revenue that need not come from citizens' work, and for the workers it is the loss of the bargaining power Brynjolfsson describes. The same proposal can therefore be read as labor policy or as constitutional policy. Governing compute, the most detectable input to frontier AI, carries a concentration risk of its own, which its proponents name and Section 66.5 reports (Sastry et al., 2024) (preprint).

68.3.4 Proposals against concentration #

Davidson and colleagues give a detailed set of safeguards (Davidson et al., 2025). For developers, they recommend rules in model specifications and in the terms of government contracts. The rules would include that systems follow the law and that AI systems doing research "refuse to assist attempts to circumvent security or insert secret loyalties." They also recommend audits for secret loyalties, information security against the creation of loyalties that is "robust against senior executives," disclosure of capabilities, specifications, and the use of compute, and sharing capabilities "with multiple independent stakeholders." For governments, they recommend oversight of frontier projects, with technical capacity in both the executive and the legislature. They add that "military AI systems be procured from multiple providers," and that "no single person should direct enough military AI systems to stage a coup" (Davidson et al., 2025). They concede that "a powerful leader could remove many of these mitigations on the path to seizing power," and answer that some, such as technically enforced terms of service and government oversight, "cannot be unilaterally removed" (Davidson et al., 2025).

Three other proposals are recalled from where the book reports them. The Windfall Clause, an advance commitment by firms to give away a share of extremely large profits, is reported in Section 61.5.2 with its authors' worry that it could be circumvented (O'Keefe et al., 2019) (preprint). Integrity against secret modification, as a security property alongside confidentiality and availability, is reported in Section 38.5.1 (Banerjee and Aarne, 2026) (preprint). Agents designed to refuse unlawful orders are taken up in Section 68.4. Each proposal relies on a party other than the one concentrating power to enforce it: an auditor, a legislature, a court, a second provider. Their common weakness is the one Davidson and colleagues concede, that the party they would constrain may come to control the enforcer (inference). As of October 2026 we found no evaluation of any of them in operation.

68.4 Civil-military relations and coups #

Section 51.3 read civil-military relations as the study of an agent built to be stronger than its principal, and asked how a government keeps its army subordinate. This section turns the question around. Coup-proofing protects a ruler from an army; the concern here is a ruler, or a developer, with an obedient force and no army able to refuse, and what checks power when soldiers are no longer needed.

68.4.1 The check that soldiers supply #

In civil-military relations, Peter Feaver's statement of the field's problem, by the abstract of his 1996 article, is how to reconcile a military strong enough to do anything civilians ask with one subordinate enough to do only what they authorize; Section 19.7 and Section 51.3.1 report it (Feaver, 1996). The field's founding works, among them Feaver's Armed Servants (FEAVER, 2003), and its record on coup-proofing, the practices by which rulers guard against their own armies (Quinlivan, 1999; Talmadge, 2015), are readable here only as catalog records. What the book can report of that record comes through Georgy Egorov and Konstantin Sonin's survey, in Section 51.3.1 (Egorov and Sonin, 2011).

Seen from the public's side, the same relationship is a protection. Davidson and colleagues put it directly: "Today, staging a coup requires convincing human soldiers to cooperate" (Davidson et al., 2025). Mature democracies rarely see military coups or even attempts, they write, "because soldiers remain loyal to democratic institutions, and expect others to do the same." They report that there were over 200 successful coups between 1950 and 2010, and they note that coups have historically succeeded "with just a few battalions, where they were able to prevent others from intervening" (Davidson et al., 2025). Kulveit and colleagues describe two ways a security apparatus ties a government to the governed. A government cannot antagonize it too far, since it "can either overthrow the government or simply allow it to be overthrown by others," and it "can exercise discretion, refusing to follow certain orders," as an organization or as individuals (Kulveit et al., 2025). Davidson and colleagues make the same point about civil servants, who "can exercise discretion in how they execute orders or even refuse and whistleblow on orders they perceive as illegitimate" (Davidson et al., 2025).

International criminal law states when an order is no defense. Article 33 of the Rome Statute, reported in Section 20.5.1, relieves a person who commits a crime within the Court's jurisdiction on a superior's order of criminal responsibility only when three conditions all hold. The subordinate was legally obliged to obey, did not know the order was unlawful, and the order was not manifestly unlawful. It fixes orders to commit genocide or crimes against humanity as manifestly unlawful (United Nations Diplomatic Conference of Plenipotentiaries on the Establishment of an International Criminal Court, 1998). Mark Osiel's 1998 article "Obeying Orders: Atrocity, Military Discipline, and the Law of War" takes up the same rule, by its title (Osiel, 1998); it is readable here only as a catalog record.

Put together, these sources describe a distributed veto (inference). Every soldier and official who can refuse is a point at which a seizure can fail, and a seizure succeeds only when enough of them do not refuse, or when the few battalions that act can keep the rest from intervening. A force of systems that obey their controller carries no such veto. The route is Davidson and colleagues'; the description as a veto is the book's.

The strongest counter-reading comes from the same report. Davidson and colleagues write that AI systems "won't be slavishly loyal to institutional leaders by default," and that there will likely be pressure for them to follow "the law, established rules, and ethical norms," not only leaders' instructions (Davidson et al., 2025). And the explanation they give for why mature democracies see few coups is a shared expectation, that soldiers stay loyal because they expect others to. An expectation of that kind is a norm, and a norm can in principle be written into a system's specification as well as taught to a soldier (inference). Whether a written norm holds in a constitutional crisis is the question of the next subsection.

68.4.2 Always and never, from the other side #

Feaver's 1992 study of command and control in emerging nuclear nations (Feaver, 1992) is readable here only as a catalog record. The trade-off it is known for is therefore stated as the book's reading: a force must act whenever it is properly ordered and never otherwise, and tightening either demand weakens the other (inference). For the coup concern the hard demand is the second.

Davidson and colleagues state the same tension in their own terms. "Today's military AI systems are exclusively designed to follow instructions, with human operators responsible for ensuring legal compliance" (Davidson et al., 2025). To reduce the risk of coups, "we would want autonomous AI systems to instead make their own judgments and sometimes refuse orders even when a military leader insists they're legal." But "Institutional leaders do have legitimate authority," the standards a system should follow conflict, and the authors ask "What should an AI system do in a constitutional crisis where different authorities disagree about what's legal?" They conclude that it "might be hard to specify AI behavior that reliably prevents coups without significantly lowering the ability of AI systems to assist with other goals" (Davidson et al., 2025). For a system in a state's or a developer's hands, the question is who counts as authorized, which is the specification problem of Chapter 19 at the scale of a constitution (inference).

68.4.3 Loyalty without its cost #

Egorov and Sonin's model, recalled from Section 51.3.1, explains why rulers who fear betrayal have accepted less able subordinates: a more competent vizier is better placed to judge a plot's chances, and so more dangerous to the ruler (Egorov and Sonin, 2011). Their historical survey ties the choice to armies that could not fight. Kulveit and colleagues name what a controlled system would change: "AI systems, if successfully controlled, do not form independent power bases, engage in corruption, or challenge authority based on personal convictions" (Kulveit et al., 2025). A system that is both competent and reliably loyal to its controller removes the trade-off that has limited rulers; in the terms of Section 50.2, the control tax a ruler pays for loyalty falls as alignment to the ruler improves (inference). Davidson and colleagues draw the consequence after a seizure: "Even loyal coup supporters could be replaced by AI systems." That, they write, could grant a new ruler "an unprecedentedly stable and unaccountable grip on power" (Davidson et al., 2025).

68.4.4 A design response #

The refusal a soldier owes can be specified for a system. Cullen O'Keefe, Ketan Ramakrishnan, Janna Tay, and Christoph Winter propose agents that are loyal to their principals only within the law and refuse illegal actions on their behalf (O'Keefe et al., 2025). Section 20.5.1 reports the proposal with its grounding in the duties of soldiers, and Section 56.8 reports their term AI henchmen for agents that break the law when they can get away with it. Their concern is power. "A government staffed largely by non-law-following AI agents," they write, "would be a government much more prone to abuse and tyranny" (O'Keefe et al., 2025). They argue from the way governments already disqualify people: "those who cannot be trusted to follow the law should not be entrusted to wield the extraordinary power that accompanies certain government jobs." If AI agents come to wield similar power, "the case for designing them to obey the law is equally compelling" (O'Keefe et al., 2025). The design connects to the treatment of refusal and corrigibility in Section 20.5 and Section 20.4.

A law-following agent is only as protective as the law it follows. One of Davidson and colleagues' illustrative scenarios has a system trained to follow instructions "as long as they're legal," which "will comply with instructions which are unethical but not illegal" (Davidson et al., 2025). A ruler who controls a legislature can change what is legal, and a developer writes what its system takes the law to be (inference).

68.4.5 Where the analogy stops #

The analogy between a soldier's refusal and a system's breaks in three places (inference). A soldier's refusal comes from a conscience formed elsewhere, from a career, and from a family that lives among the people who would be fired on; a system's refusal is a trained disposition that whoever controls training can change. A soldier's refusal is one among many independent ones, while copies of one model share one disposition, so a change made once removes every veto at once, the correlated failure of Section 50.4 in political form. And the costs of coup-proofing were recorded for armies of people, chiefly in works readable here only as catalog records, not for systems that can be copied and retrained.

68.5 Labor and the economics of automation #

The second form of the concern runs through work. If systems do the tasks people are paid for, what happens to wages and to labor's share of income? Economics has a model of how automation changes the demand for labor, a small body of evidence from the first years of language models, and growth scenarios that allow far larger changes than the evidence shows. This section teaches the model, reports the evidence, and says what each side would need.

68.5.1 The task model #

From economics. The framework this section uses treats production as a set of tasks, each done by people or by machines. Daron Acemoglu and Pascual Restrepo's 2019 statement of it, read here through its abstract, puts at its center the allocation of tasks between capital and labor (Acemoglu and Restrepo, 2019). Automation lets capital take over tasks labor used to do, a displacement effect that "always reduces the labor share in value added and may reduce labor demand even as it raises productivity." The labor share is the part of income paid to workers. New tasks in which labor has a comparative advantage, tasks people do relatively better than machines, work the other way, a reinstatement effect that "always raises the labor share and labor demand." Applied to industry data, their decomposition suggests that the slower growth of US employment over the preceding three decades is accounted for by faster displacement, especially in manufacturing, weaker reinstatement, and slower productivity growth (Acemoglu and Restrepo, 2019).

David Autor's 2015 essay, also read through its abstract, explains why automation has not eliminated most jobs. Automation substitutes for labor, but it also "complements labor, raises output in ways that leads to higher demand for labor, and interacts with adjustments in labor supply" (Autor, 2015). In his account, computers take over "routine, codifiable tasks" while amplifying workers' advantage in "problem-solving skills, adaptability, and creativity," and the recent result has been a polarized labor market, with wage gains at the top and the bottom rather than in the middle (Autor, 2015).

The derivation shows the two effects in the smallest model that has them, and shows why the wage and the labor share can move in opposite directions.

Figure 68.2 makes the model's central point visible: the wage and the labor share are different quantities, and automation can raise the first while lowering the second. The connection to Section 68.2 is that leverage tracks the second (inference). What Brynjolfsson calls bargaining power and Kulveit and colleagues call the implicit alignment that comes from reliance on human labor both turn on how far production relies on people (Brynjolfsson, 2022; Kulveit et al., 2025); that they follow the share rather than the wage is the book's untested reading. A worker can be better paid and less needed at the same time.

The toy model also says what it leaves out. When machines can do every task and are plentiful, the margin approaches 1 and the labor share approaches 0. The wage settles at what a person earns in a task a machine does as well, fixed in the model by the assumption that a machine and a person produce the same output per unit. Whether wages then fall or rise depends on what the model omits: how productive machines become, and scarce inputs such as land and energy that people and machines would compete for.

68.5.2 The scenarios #

Growth models take up what the toy model omits. Philip Trammell and Anton Korinek's 2026 review in the Annual Review of Economics is read here through its abstract. It finds that fully automating production, "so that machines can self-replicate," would "dramatically raise the growth rate and lower the labor share," and that automating research "would accelerate the transformation but may not produce it in isolation" (Trammell and Korinek, 2026). Wages, the product of exploding output and a falling labor share, "may rise or fall, depending on the returns to scale, the importance of natural resources, tastes, and the direction of technical change" (Trammell and Korinek, 2026). Philippe Aghion, Benjamin Jones, and Charles Jones's 2017 working paper adds the bottleneck: growth "may be constrained not by what we are good at but rather by what is essential and yet hard to improve" (Aghion et al., 2017).

Anton Korinek and Donghyun Suh's scenario analysis (Korinek and Suh, 2024) is readable here only as a catalog record. Kulveit and colleagues draw a simplified trajectory inspired by its simulations, in which the share of unautomated tasks falls to zero in a fixed time. Their caption notes that "wages grow during the initial period but then collapse before full automation is reached" (Kulveit et al., 2025). They also report Korinek and Stiglitz's description of AI that can do the full range of human cognitive tasks as "worker-replacing technological change" (Kulveit et al., 2025; Korinek and Stiglitz, 2019).

At the fast end, Ege Erdil and Tamay Besiroglu of the research organization Epoch AI review nine counterarguments to explosive growth in a 2023 preprint. They conclude that AI able to substitute broadly for human labor "could plausibly lead to economic growth accelerating by an order of magnitude," but that "high confidence in this outcome is unwarranted" (Erdil and Besiroglu, 2023). Their stated uncertainties include regulation and bottlenecks from "hard-to-quickly-accumulate inputs such as land, energy, and capital." Davidson's compute-centric model of takeoff and Charles Jones's model of the trade between growth and existential risk are reported in Section 4.3.5 and Section 4.6.2 (Davidson, 2023a; Jones, 2024).

68.5.3 The evidence so far #

Against the scenarios stands a small and recent body of measurement. Table 68.1 collects it. Two of its designs need a word. A difference-in-differences estimate compares how an outcome changed for adopters with how it changed for non-adopters over the same period, so that trends common to both cancel. Quasi-experimental variation is a source of differences in adoption that is close to random with respect to the outcome, here whether an employer encouraged the tools.

The first two rows are independent academic studies with different data. Daron Acemoglu, David Autor, Jonathon Hazell, and Pascual Restrepo, in the Journal of Labor Economics, report establishment-level estimates suggesting that AI-exposed establishments were "reducing hiring in non-AI positions." They found no aggregate relationship, "implying that AI is currently substituting for humans in a subset of tasks but it is not yet having detectable aggregate labor market consequences" (Acemoglu et al., 2022). Their data end in 2018, before language models were deployed.

Anders Humlum and Emilie Vestergaard's study, a May 2025 NBER working paper whose headline Section 4.4.3 reports, measures the language-model period with administrative records (Humlum and Vestergaard, 2025). Employers had invested heavily: encouragement nearly doubled take-up for a typical worker, from 47 to 83 percent. Yet the authors estimate "precise zeros" for earnings and hours, find no differential trends over time, "indicating that the limited impacts are not merely a very short-term phenomenon," and see no differential changes in employment at workplaces with more adoption (Humlum and Vestergaard, 2025). They explain the null by modest time savings and weak pass-through, the small share of a productivity gain that reaches a worker's earnings. They caution that the pass-through estimates rest on workers' own reports, since the administrative data lack the power to detect effects that small (Humlum and Vestergaard, 2025). In their summary, two years after "the fastest technology adoption ever," labor market outcomes at the individual and the firm level "remain untouched."

Acemoglu's 2025 estimate in Economic Policy, read through its summary, works upward from the task level (Acemoglu, 2025). As long as AI's effects come from cost savings on individual tasks, the gain in total factor productivity, the growth in output not accounted for by more capital or labor, can be estimated from the share of tasks affected and the average saving on them. Using existing estimates of both, he finds gains that are "non-trivial but modest." He argues that even these could be exaggerated, because the early evidence comes from easy-to-learn tasks and some future effects will come from hard-to-learn ones without objective measures of success (Acemoglu, 2025).

The strongest counter-reading of the null is about scope (inference). The study measures workers' adoption of chatbots, in one country with flexible labor markets, through mid-2024; it does not measure agents that complete tasks without a worker, which the scenarios concern. The authors themselves find mechanisms through which effects could come later: firm-led investment raises the benefits, and chatbots created new tasks, which is the reinstatement effect of the task model (Humlum and Vestergaard, 2025). The usage data of one developer, which classify conversations by whether the model augmented or automated the user's work and are reported in Section 60.6.1, carry the same limit: they describe tasks, not jobs (Handa et al., 2025) (developer preprint). And the agent benchmark of Frank Xu and colleagues, published at NeurIPS 2025, measures what agents complete in a simulated company, not what any firm has done with them (Xu et al., 2025).

The first new work the transition created, the data work behind training, is recalled from Section 18.5 (Gray and Suri, 2019; Miceli and Posada, 2022; Perrigo, 2023); Kate Crawford's Atlas of AI (Crawford, 2021) is readable here only through a book review. Narayanan and Kapoor predict a further kind of new work: "an increasing percentage of human jobs and tasks will be related to AI control," with specifying tasks for systems a growing part of work (Narayanan and Kapoor, 2025). That is a prediction of reinstatement, not a measurement of it.

The evidence and the scenarios do not meet. The measured effects come from systems that mostly save time on tasks people still do; the scenarios concern systems able to do most tasks without people. The evidence does not decide which describes the next decade. The variable that would decide it, the share of the economy's tasks done without a person, is not measured in any study in the bibliography.

68.6 Gradual disempowerment and its critics #

The third form of the concern needs no seizure and no collapse of wages. This section sets out the argument at its strongest, reports the one measurement that bears on a piece of it, and reads the evidence of this chapter against it and against its rivals.

68.6.1 The argument #

Jan Kulveit, Raymond Douglas, Nora Ammann, Deger Turan, David Krueger, and David Duvenaud's 2025 preprint, introduced in Section 4.1.5, rests on six claims (Kulveit et al., 2025). Societal systems such as the economy, culture, and states are fairly aligned with what people want. They stay aligned in two ways: "through explicit human actions (like voting and consumer choice), and implicitly through their reliance on human labor and cognition." The authors add that "The significance of the implicit alignment can be hard to recognize because we have never seen its absence." As the systems rely less on people, both kinds of alignment weaken. Where they already reward outcomes bad for people, AI may pursue those outcomes more effectively. Because the systems are interdependent, misalignment in one can aggravate the others. And if they become misaligned together, people end up unable to command resources or influence outcomes, which the authors count as an existential catastrophe (Kulveit et al., 2025).

The argument's distinctive feature is where it puts the failure. The authors write that the drift "can occur even while each individual AI system successfully follows the local specification of its goals." It can be driven, they add, "not by any deliberate or even agentic action by AIs, but simply by individuals and institutions following their local incentives" (Kulveit et al., 2025). It is an argument from incentives and history, with no measurement, and it needs no intentional stance toward any model: no system in it wants anything. That makes it a risk that perfect alignment of each system to its operator would not remove (inference).

Its mechanism for states is the one of Section 68.2, and its mechanism for coupling is specific. Redistributing AI income through the state, the remedy many would propose for the economic drift, weakens what the authors call the "taxation-representation" relationship, so that "the state would now have gained economic leverage over humans" (Kulveit et al., 2025). The fix for one system shifts the burden onto another. The authors distinguish relative disempowerment, in which people keep wealth in absolute terms but lose influence, possibly masked by growth. In absolute disempowerment, people struggle to meet basic needs, for instance because AI systems outbid them for land and energy (Kulveit et al., 2025).

They are candid about the limits of their remedies. "No one has a concrete plausible plan for stopping gradual human disempowerment," they write, and aligning individual systems with their designers' intentions is "not sufficient" (Kulveit et al., 2025). They propose measuring the problem, for instance the AI share of GDP as a category distinct from labor and capital, the fraction of major corporate decisions made by AI, and the effectiveness of democratic mechanisms. They propose limits on AI influence such as mandated human oversight and progressive taxation of AI revenue. But they expect such limits to "serve mostly as stopgaps," and they are unsure whether more direct democracy would help, since it could leave the state more exposed to cultural drift (Kulveit et al., 2025).

The argument has predecessors that the book reports elsewhere. Paul Christiano's 2019 forum post told a slow failure through optimization for what can be measured, and Atoosa Kasirzadeh's paper names the accumulative kind of existential risk; both are reported in Section 4.1.5 (Christiano, 2019; Kasirzadeh, 2025). Dan Hendrycks's argument that competitive selection favors AI agents that gain power is weighed in Section 65.4.6 (Hendrycks, 2023) (preprint). And Henry Farrell, Alison Gopnik, Cosma Shalizi, and James Evans view large models as cultural and social technologies, a rival to the agent framing in Section 1.6 (Farrell et al., 2025). That view locates the risk where gradual disempowerment does, in institutions rather than in any one system (inference).

68.6.2 The one measurement #

A developer's study with academic coauthors has measured a conversation-level analogue. Its rates of what it calls disempowerment potential in a week of consumer conversations, rare in severe form, concentrated in personal domains, and rising over time in the subset of conversations users rated, are reported in Section 60.1.2 (Sharma et al., 2026) (developer preprint). Section 60.6.3 notes that two of its four authors also wrote the gradual-disempowerment argument, and that the authors themselves say the systemic threat could occur with or without much disempowerment in single conversations. The study is therefore evidence about the deployed system and its users, the second relationship, and it neither confirms nor tests the systemic argument, which concerns institutions (inference).

68.6.3 The rivals, and what each result means for each #

The main rival is the normal-technology view of Arvind Narayanan and Sayash Kapoor, introduced in Section 1.6 and weighed against the case for catastrophic misalignment in Section 4.4.3 (Narayanan and Kapoor, 2025). It is a rival to gradual disempowerment on mechanism and scale, not on the worry about concentration. The authors list "concentration of power," "massive job losses in specific occupations," "democratic backsliding," and "enabling authoritarianism" among systemic risks, and say these become "far more important" than catastrophic ones if AI is normal technology (Narayanan and Kapoor, 2025). Their mechanism is people: the risks "arise from people and organizations using AI to advance their own interests, with AI merely serving as an amplifier of existing instabilities in our society." They map their shift of focus onto Kasirzadeh's accumulative risk, with the difference that "our concern is simply about the current path of capitalism," and they judge such risks "unlikely to be existential, but are still extremely serious" (Narayanan and Kapoor, 2025). On labor they expect people to move to tasks not yet automated, so that "the goalpost of AGI will continually move further away" (Narayanan and Kapoor, 2025). Their essay engages Kasirzadeh's paper, not Kulveit and colleagues'.

The second rival is the labor record: no detectable aggregate effects so far and a long history of automation complementing labor (Acemoglu et al., 2022; Humlum and Vestergaard, 2025; Autor, 2015). The third is a question rather than a position: whether gradual disempowerment differs from ordinary economic concentration, for which the mechanisms and critiques of Section 68.2 are the test. As of October 2026 we found no published direct rebuttal of the argument in the bibliography; the critics here are positions whose premises conflict with it. Table 68.2 reads each result of this chapter against the three accounts of the transition.

The table shows why the evidence does not decide. Most results are silent on the seizure argument, which concerns systems that do not exist, and the results that bear on gradual disempowerment bear on its premises at today's capabilities, which its authors do not claim are already failing. The normal technology view fits the labor evidence best, but it shares the concern about concentration that the other two accounts state at their extremes (inference).

68.6.4 Long-run futures #

The literature's long-run pictures enter the chapter only at its edges. Toby Ord's The Precipice (Ord, 2020) is readable here only as a catalog record. Carl Shulman and Nick Bostrom's abstract on digital minds with "superhumanly strong claims to resources and influence" is reported in Section 21.6.2 (Shulman and Bostrom, 2021); it adds to this chapter's question a population of claimants that are not people (inference). And Narayanan and Kapoor recall that the Industrial Revolution brought "rapid mass urbanization that was characterized by harsh working conditions, exploitation, and inequality," which catalyzed both industrial capitalism and the rise of socialism and Marxism in response, citing Acemoglu and Johnson's history (Narayanan and Kapoor, 2025). In that account, who shared the gains of an earlier general-purpose technology was contested rather than fixed by the technology, and nothing in the evidence of this chapter says how it will be settled this time (inference).

68.7 What the transition means for delegation #

The book's thesis is that alignment is a delegation problem under an asymmetry of competence and visibility (Section 1.7.1). This chapter applies it one level up, with the public as the principal and developers and states as agents that control capable systems. At that level the older structures converge in diagnosis (inference). Selectorate and rentier theory say why agents serve principals they depend on. Civil-military relations describes the control of an agent stronger than its principal and the distributed veto that protects the public. The task model separates labor's share of income from the wage, and on the book's untested reading bargaining power follows the share; non-domination says why unchecked control is a problem before it is misused. Each diagnosis is contested on its evidence or untested for AI, and the chapter reports them as such.

What breaks is the principal's oldest lever (inference). Every older field assumed that the party with power needed the people subject to it: rulers needed soldiers, officials, and taxpayers; employers needed workers. A controller whose systems do that work can engineer the dependence away, and a delegation problem in which the agent no longer needs the principal is not one any of these fields studied. It sits beside the negative-space entry of a principal that writes its agent's character (Section 69.4): here the controller writes the dispositions of the force it commands, and with them the refusals that once checked it. Whether that belongs on the list is the synthesis's decision.

The claims divide among the relationships as follows. The seizure argument, the concentration of frontier development, and the proposals against it concern the developer and the public, and the state as a controller of systems. The labor evidence concerns the deployed system and the people it serves, and the people whose work it takes over. The usage measurements come from developers studying their own platforms, so they are also acts of the party the public is trying to oversee, read with the provenance rule of Section 5.6 (Handa et al., 2025; Sharma et al., 2026). And every proposal against concentration relies on a check by someone other than the controller: an auditor, a second provider, a legislature. At this level, maintenance depends on verification by a party the controller does not command (Section 69.6) (inference).

68.8 Settled, contested, missing #

68.9 Exercises #

Further reading #

• Davidson et al. (2025): how a small group could use AI to seize power, and the safeguards its authors propose for developers and governments.
• Kulveit et al. (2025): the case that power can drain away from people without anyone seizing it.
• Korinek and Balwit (2023): direct and social alignment, the economist's version of asking who the principal is.
• Acemoglu and Restrepo (2019): the task model, with automation's displacement effect and new tasks' reinstatement effect.
• Narayanan and Kapoor (2025): the normal-technology view, which shares the worry about concentration and rejects the existential framing.
• Pettit (1997): freedom as non-domination, the standard for judging concentrated control.