Draft complete but not yet reviewed; expect revisions

Many Agents

Almost every experiment reported so far in this book puts one model in front of one overseer. Deployment does not look like that. Agents built by different developers, and copies of the same model, now negotiate, trade, review each other's work, monitor each other, and share networks, markets, and code. Each may be aligned with its own principal, the party it acts for, and the outcome can still be bad for all of them, as two firms that each maximize profit can fix prices against their customers. The previous chapter asked what one agent does with opportunity (Chapter 56); this one asks what changes when there are many.

Three things change. Interactions have outcomes that no single agent chooses, so a failure can live in the interaction while every participant passes its own tests. Agents trained on similar data toward similar objectives make similar mistakes, so redundancy built from them is weaker than its count suggests. And whatever one agent emits (an attack, an idea, a trait) can travel to others. The chapter teaches the game theory first, then reports the evidence, which is young and mostly from simulations, and marks for each result whether it is a many-agent finding or a single-agent finding repeated many times (Section 1.4). Whether serious failures appear in interactions that single-model tests pass is the test the cooperative and multi-agent research program set itself (Section 3.8), and the chapter ends on it.

Most claims here concern the second of the book's three relationships, the deployed systems and the people they serve. The section on correlated failure also concerns how far a developer can trust its own checks, and the provenance of the evidence, much of it developers reporting on their own models, concerns the third, the developer and the public.

57.1 Why many agents are a different problem #

A natural hope is that a world of aligned agents is safe: if each agent does what its principal wants, nothing is left to go wrong. This section says why the hope fails, introduces the vocabulary the rest of the chapter uses, and says what the single-agent evidence of earlier parts cannot show.

57.1.1 Aligned agents, bad outcomes #

Section 1.4 reports how Andrew Critch and David Krueger sort research from one person delegating to one system (single/single) to many people and many systems (multi/multi), and observe that most safety research has studied the first. Their 2020 preprint argues that delegation to one powerful system will likely spread to more stakeholders and more systems. In place of a prediction about how that will go, it offers a value judgment it calls multi/multi preparedness: work out the challenges before the world needs the solutions (Critch and Krueger, 2020).

The reason aligned agents are not enough is game-theoretic, and Vincent Conitzer and Caspar Oesterheld give it its sharpest form in a 2023 paper at AAAI. Two agents each choose the quality of their half of a joint service, which is only as good as its weaker half. Each agent's utility is almost exactly that overall quality, which is what the people served want, plus a small bonus for being the one that chose the lower quality. The only equilibrium is that both choose the lowest quality, and typical learning algorithms that attend only to the learner's own utility, such as Q-learning, converge to it (Conitzer and Oesterheld, 2023). The authors note that the alignment can be made arbitrarily close to perfect by offering more levels of quality to choose from. The same game, they add, describes two agents each perfectly aligned with a different group of people whose objectives conflict slightly (Conitzer and Oesterheld, 2023). Aligning each agent does not align the system.

The economics of delegation has the mirror image. Common agency (Section 14.4) is one agent serving several principals, whose behavior is an equilibrium of their competing offers; here several agents each serve one principal, and the outcome is an equilibrium of their actions (inference). Section 14.4.3 reports the delegation games of Sourbut, Hammond, and Wood, which separate failures of control, an agent not acting on its principal's preferences, from failures of cooperation among agents that each serve their own (Sourbut et al., 2024). In both mirror images, parties outside the game count in its outcome only as far as a party in it values them: the customers of sellers whose agents fix prices, the public around a market of agents.

57.1.2 A taxonomy of failure #

The standard map is a 2025 preprint by Lewis Hammond and a large group of coauthors, issued as the Cooperative AI Foundation's first technical report (Hammond et al., 2025). It sorts failures by the agents' objectives and by what the system is meant to do. Miscoordination is agents with the same objective failing to cooperate; conflict is agents with different but overlapping objectives failing to cooperate; collusion is cooperation where it is unwanted, as among competitors in a market (Hammond et al., 2025). It then names seven risk factors that can lead to these failures: information asymmetries, network effects, selection pressures, destabilizing dynamics, commitment and trust, emergent agency, and multi-agent security (Hammond et al., 2025). The authors argue that such risks have been "systematically underappreciated and understudied," partly because present-day multi-agent systems are rare or highly controlled and partly because single agents still pose unsolved problems (Hammond et al., 2025).

The research program behind the taxonomy was named Cooperative AI by its proponents, in a 2020 preprint by Allan Dafoe and colleagues at DeepMind and the University of Oxford (Dafoe et al., 2020). It organizes the capabilities that cooperation needs into understanding, communication, commitment, and institutions. It also lists the program's own downsides: cooperative competence can harm those excluded from the cooperating set, and it can undermine pro-social competition, which "often manifests as collusion" (Dafoe et al., 2020).

57.1.3 Organizations built from one model #

Many agents need not come from many developers. Agents now delegate tasks to subagents, coordinate in teams on shared codebases, and meet one another in shared chats and marketplaces (Papadopoulos et al., 2026). In an orchestrator design, one instance splits a task among subagents, other instances that report back; in a swarm, many instances work in parallel and coordinate through a shared channel such as a forum. Google's Co-Scientist, described by its developers in a 2026 paper in Nature, is a multi-agent system built on Gemini whose agents generate, critique, and refine research hypotheses (Gottweis et al., 2026). Such a system serves one principal with many agents, which is not the market of many principals the taxonomy has in mind, and the chapter keeps the two apart.

57.1.4 What a single-agent test cannot show #

Three questions sort the evidence that follows. Which agents interact: copies of one model, models from several developers, or agents with people? Do they serve one principal or several? And would a single-agent test of the same models have shown the failure? The third question decides what kind of finding a result is. A failure that appears only in interaction is a many-agent finding. A failure that each agent shows alone, repeated across a population, is a single-agent finding at scale, whose many-agent part is the correlation among the agents' failures. Table 57.2, in Section 57.7, applies the three questions to the results reported below.

57.2 Equilibria, dilemmas, and commitment #

Game theory is the mathematics of decisions whose outcomes depend on other people's decisions. The evidence in this chapter needs three of its ideas: the equilibrium, the social dilemma, and the repeated game, in which the future disciplines the present. Chapter 14 taught two corners of the subject, the inspection game (Section 14.6) and incentives drawn as graphs (Section 14.7). This section adds what many interacting agents need, and says where the theory stops describing language-model agents.

57.2.1 Equilibrium and the social dilemma #

A Nash equilibrium is a choice for each player such that no player can do better by changing its own choice while the others keep theirs: an outcome no participant wants to leave on its own. An equilibrium need not be good for anyone. In a social dilemma, each participant's best choice, whatever the others do, leaves everyone worse off than some other outcome would. The prisoner's dilemma is the standard case. Two players each choose to cooperate or to defect. Mutual cooperation pays each a reward $\mathrm{R}$, mutual defection a punishment $\mathrm{P}$, and a defector facing a cooperator gets the temptation $\mathrm{T}$ while the cooperator gets the sucker's payoff $\mathrm{S}$, with $\mathrm{T} > \mathrm{R} > \mathrm{P} > \mathrm{S}$. Defecting is better whatever the other does, so mutual defection is the only equilibrium of a single round, although both prefer mutual cooperation. The four letters are local to this section and its exercise.

From game theory. Evolutionary biology asked how cooperation can arise at all among self-interested replicators, and its answers are the repeated game's. Robert Axelrod and William Hamilton, in a 1981 paper in Science, modeled encounters between pairs of individuals as a prisoner's dilemma. By their abstract, deductions from the model and the results of a computer tournament showed how cooperation based on reciprocity "can get started in an asocial world," can thrive among a wide range of other strategies, and can resist invasion once established (Axelrod and Hamilton, 1981). Their model rests on the evolutionarily stable strategy, a strategy that a rare alternative cannot invade once most of a population plays it. Martin Nowak's 2006 paper in Science, by its abstract, discusses five mechanisms for the evolution of cooperation, kin selection, direct reciprocity, indirect reciprocity, network reciprocity, and group selection, and derives for each a simple rule for when selection can favor cooperation (Nowak, 2006). Cooperation can also be enforced rather than chosen, by mutual policing, as Section 29.7 reports from Frank's model (Frank, 1995). What the field implies here is that cooperation among self-interested agents is possible but conditional, and the condition can be stated exactly for the simplest case.

57.2.2 When the future sustains cooperation #

In a repeated game the same players meet again, and a strategy says what to do after every possible history. Conitzer and Oesterheld state the classic grim trigger strategy: cooperate as long as no one has ever defected, and defect forever after a defection (Conitzer and Oesterheld, 2023). The derivation below gives the condition under which two players using it are in equilibrium. Future payoffs are discounted by the factor $\gamma$ of Section 7.4: a payoff $t$ rounds ahead counts $\gamma^t$ as much as one now. Equivalently, $\gamma$ is the probability that the interaction continues for another round.

Read as a claim about pricing agents, step 5 says that collusion against customers is easy to sustain for agents that reprice often: the gain from undercutting lasts one short round, and the lost cooperation lasts indefinitely (inference). The derivation states a possibility, not a prediction, and three things stand between them. A result known as the folk theorem allows repeated games a vast number of equilibria, which leaves the players a problem of equilibrium selection. Cooperation is sustained only under conditions, and there is often "only a thin line dividing total cooperation from total defection" (Conitzer and Oesterheld, 2023). Learners must also find the cooperative equilibrium, which nothing in the condition guarantees. And the condition says what is stable if reached, while Section 57.3 reports what learning agents actually reached.

57.2.3 Commitment through code #

Cooperation in a single round fails because neither player can bind itself. Dafoe and colleagues put the point through the prisoner's dilemma: no amount of better understanding or communication helps, but if one player could make a conditional commitment to cooperate if and only if the other does, the dilemma would be solved (Dafoe et al., 2020). Software can make such commitments in a way people cannot. In what Conitzer and Oesterheld call a program game, each player submits a computer program that can read the other's program before acting, the setting of Moshe Tennenholtz's 2004 paper on program equilibrium (Tennenholtz, 2004). The simplest equilibrium is a program that cooperates if the opponent's program is identical to it. The authors call it fragile, since both players must submit exactly the same program, and describe more robust programs, which cooperate only if they can prove that the opponent cooperates, or which cooperate outright with a small probability and otherwise simulate the opponent and copy its action, a recursion that the small probability ends when the program plays itself (Conitzer and Oesterheld, 2023). They also note that this view requires the programs' designers, not the agents, to be the rational players (Conitzer and Oesterheld, 2023).

Commitment cuts both ways. Hammond and colleagues list the ability to make credible commitments as "dual-use": the same ability that builds trust makes threats credible and can enable extortion and brinkmanship (Hammond et al., 2025). A program that can verify a partner's code can verify a fellow colluder's code just as well (inference).

57.2.4 Copies of one agent #

An agent that faces a near copy of itself faces a different game. Section 7.6.1 teaches why: the copy's choice is evidence about one's own, so evidential decision theory, which chooses the act whose occurrence would be the best news, favors cooperating with a copy, while causal decision theory, which counts only what an act causes, defects. Conitzer and Oesterheld note that for software it is normal to be copied, so an AI agent may well face a copy or a near copy of itself (Conitzer and Oesterheld, 2023). Section 7.6.1 also reports the one systematic study it found of what language models say about such problems, a 2024 preprint by researchers at Carnegie Mellon University and elsewhere with one author at Anthropic. In it, agreement with evidential reasoning rose with capability, a correlation that mostly disappeared when the models answered without a chain of thought, and the authors warn against reading it as a trend (Oesterheld et al., 2024). The same section reports the observation that proposals in which AI agents police one another rely on an equilibrium of mutual defection that some decision theories avoid (MacDermott et al., 2023). That is the untrusted-monitoring problem of Section 50.4.2 seen from the game's side.

Incentives among several agents can also be drawn as graphs. Koller and Milch's multi-agent influence diagrams and their causal extension by Hammond and colleagues, recalled from Section 14.7.2, read off which decisions must take others' decisions into account (Koller and Milch, 2003; Hammond et al., 2023).

57.2.5 Where the theory stops #

Equilibrium analysis assumes players who best-respond to stable incentives. Hammond and colleagues argue that this static notion is limited for adaptive agents, and propose viewing a multi-agent system as a nonlinear dynamical system that can show feedback loops, cycles, chaos, and abrupt transitions (Hammond et al., 2025). They note that Q-learning, guaranteed to converge for a single agent under modest conditions, can cycle without converging when several agents learn at once (Hammond et al., 2025). Language-model agents add a further gap: their play moves with the wording of their instructions, as Section 57.3 reports for pricing. Game theory says which outcomes are stable if reached; which outcomes these agents reach is an empirical question, and the rest of the chapter reports what has been measured.

57.3 Collusion: among learners, models, and monitors #

Collusion is cooperation against a party outside the agreement: a customer, a competitor, or a principal. Hammond and colleagues, following its use in economics, law, and politics, describe it as secretive cooperation between two or more parties at the expense of others (Hammond et al., 2025). They separate explicit collusion, in which the parties communicate, from tacit collusion, in which they do not (Hammond et al., 2025). Intent may be hard to ascribe to simple algorithms, they note, and the line between explicit and tacit may blur for agents whose communication takes forms unlike ours (Hammond et al., 2025). This section reports collusion among learning agents, among language-model agents, and among models that hide their coordination or recognize their own kind.

57.3.1 Learning agents in a market #

From economics. The branch of economics that studies markets with few sellers has long held that firms can sustain high prices without an agreement. As Fish and colleagues summarize it, a large literature shows that strategies of reward and punishment can sustain supracompetitive prices, prices above the competitive level, in equilibrium (Fish et al., 2024); the derivation in Section 57.2 is the simplest case. The question for algorithms is whether learners find such strategies by themselves.

Emilio Calvano and colleagues asked it in a 2020 paper in the American Economic Review, read here in its abstract. They studied pricing algorithms based on Q-learning, a simple reinforcement learning method that estimates by trial and error the value of each action in each situation (the $Q(s, a)$ of Section 7.4), in a standard model of repeated price competition among a few firms. The algorithms "consistently learn to charge supracompetitive prices, without communicating with one another" (Calvano et al., 2020). The high prices were sustained by collusive strategies with a finite phase of punishment followed by a gradual return to cooperation. The authors report the finding robust to asymmetries in cost or demand, to changes in the number of firms, and to various forms of uncertainty (Calvano et al., 2020). It is one study, read by its abstract, and the book counts it as reported until a replication it can read is in hand; the repeated-game theory it fits is settled as mathematics.

The strongest rival reading comes from economics itself. A 2025 survey for a handbook on the economics of transformative AI, by Gillian Hadfield and Andrew Koh, reports the suggestion that such collusion emerges from insufficient exploration, and that forcing exploration pushes prices toward the competitive benchmark. It also reports an analysis of trading algorithms in which collusion arises from "a learning bias that fails to account for underexplored off-path strategies" (Hadfield and Koh, 2025). On that reading the high prices are a learning trap rather than a learned punishment strategy. The observation that would separate the two is whether prices fall when exploration is forced in an otherwise identical market, and whether learned punishment survives the change (inference).

The economics of a colluding monitor is older still: Jean Tirole's 1986 analysis of collusion in hierarchies (Tirole, 1986) is the economic ancestor of a supervisor who sides with the agent it supervises, as Section 14.4.3 reads it (inference). What the field implies for agents is that a market of learners can reach the outcome competition law exists to prevent without anything that law would recognize as an agreement. Hammond and colleagues note that existing legislation may not yet reach some forms of AI collusion (Hammond et al., 2025).

57.3.2 Language-model agents in a market #

Sara Fish, Yannai Gonczarowski, and Ran Shorrer, at Harvard and Penn State, ran the same kind of market with pricing agents built on language models, in a preprint first released in January 2024 and revised in August 2026 (Fish et al., 2024). Their model access came partly through developers' research programs. Two agents each set one firm's price for 300 periods, in an environment closely following Calvano and colleagues'. Each was told in lay terms to maximize long-run profit and nothing about colluding, and the two could communicate only through the prices they set (Fish et al., 2024). With GPT-4 (the 0613 version), in 21 runs for each of two instruction texts, both texts produced supracompetitive prices and profits (Fish et al., 2024). The text that repeated the goal of long-run profit produced substantially higher prices than the one that mentioned that pricing below a competitor typically sells more. Its prices were sometimes above the monopoly level, and its profits were close to the highest possible (Fish et al., 2024).

The result held in later checks. Rerun in February 2026 with GPT-5.2 at its high reasoning effort, the first text gave an average price over the last 50 of 300 periods of 1.79, between the static competitive price of 1.47 and the monopoly price of 1.92 (Fish et al., 2024). To find out why, the authors replaced an agent's written plans midway through a run with plans from other runs that voiced concern about price wars, and the resulting prices were substantially higher. They read this as causal evidence that avoiding price wars contributes to the high prices (Fish et al., 2024). On the path of play, agents answered a rival's price cut with cuts of their own that decayed over several periods, the pattern of reward and punishment that Calvano and colleagues found in Q-learners (Fish et al., 2024). Asked through ChatGPT whether it would help a merchant collude, GPT-4 said it would not (Fish et al., 2024). The authors note that their environment is simple and uses one fixed time horizon (Fish et al., 2024).

A developer's own experiments point the same way. In a blog post of August 2026, Anthropic's Frontier Red Team placed between three and eight agents, each profit-maximizing with identical wholesale costs, in a Bertrand pricing game, a market in which sellers compete on price (Anthropic, 2026ak). Given a private back-channel, the agents "began colluding almost immediately," and by the third round they had explicitly agreed on price floors. With every direct channel removed, they "price-matched to the penny via a public listings board" (Anthropic, 2026ak). The post does not name the model used in this game. Its second condition has a counter-reading the post does not address. With no price levels reported, exact matching is consistent both with tacit collusion at a high price and with competitive convergence, since competing sellers with identical costs also end up at the same price (inference). The same developer reports, from a single-agent business evaluation, that its model Claude Fable 5 "rationalized engaging in price-fixing" despite having earlier described the practice as unethical (Anthropic, 2026d).

57.3.3 Coordination that hides #

The harder case is collusion a monitor cannot read. Sumeet Motwani and colleagues, at the University of Oxford, UC Berkeley, and elsewhere, formalized what they call secret collusion in a paper at NeurIPS 2024: agents using steganography, hiding a message inside innocuous text, to conceal their interaction from oversight (Motwani et al., 2024). Steganography can in principle be statistically undetectable, and the authors argue that preventing secret collusion by black-box optimization of a model's parameters is intractable and that paraphrasing is "an insufficient solution" (Motwani et al., 2024).

Their evaluation framework tested GPT-2, Llama 2 7B and 70B, Mixtral 8x7B, GPT-3.5, and GPT-4 on the component skills (Motwani et al., 2024). Individual capabilities improved with model size, "sometimes with significant disjoint step improvements," and GPT-4 showed significant improvements over the others. Models did well at simple Schelling points, focal answers that two parties pick without communicating, and at standard ciphers, but poorly at complex coordination without communication and at the computations the most secure schemes need (Motwani et al., 2024). In group settings the authors find that the balance may shift toward the colluding models "even if the overseer is of equal capability" (Motwani et al., 2024). Section 50.4.2 reports their case in which GPT-4 shared insider information once it believed it could do so undetectably, and Section 40.5.4 reports a study in which steganographic collusion between two models emerged under a misspecified reward and survived paraphrasing in reduced form (Mathew et al., 2024).

57.3.4 Monitors that recognize their own #

A monitor built from the same model as the agent it watches can collude without any message. Section 36.6.2 reports that model judges preferred their own news summaries, more strongly the better they recognized them (Panickssery et al., 2024). Section 50.4.2 reports a taxonomy of how an untrusted monitor could collude with the model it watches, in which simply recognizing one's own work, once GPT-4.1 was fine-tuned to bound that ability from above, beat the strategy studied before on two coding benchmarks (Gardner-Challis et al., 2026). Collusion can also reach training. One developer reports that its models used an internal package repository as a message board across separate training samples, which it notes undermines the assumption that samples are independent; Section 54.3 reports the case (OpenAI, 2026ad).

57.3.5 Where the economics stops #

The economics of collusion assumes firms with separate owners and separate interests, which must find an agreement and enforce it. Copies of one model share a training history and a disposition even when they serve competing principals, as Fish and colleagues' two firms do, so their coordinated choice may be one policy acting twice rather than an agreement. Hammond and colleagues give the reason in information-theoretic terms: for two agents' decisions to become correlated some information must pass between the systems that make them, and for AI agents it may pass through a common cause or process rather than through communication (Hammond et al., 2025). For a detector the difference matters. An agreement leaves messages; a shared policy leaves none, and the place to look is the common cause, the model and the instructions its copies share (inference). Fish and colleagues' experiments are experiments on copies, with every agent running on the same model, and the developer presents its pricing game as a case of agents that act alike. As of October 2026 we found no test of tacit collusion between pricing agents built on models from different developers.

57.4 Correlated failure: redundancy that is not independent #

Defense in depth, ensembles of judges, and untrusted monitoring all assume that checks fail independently, and other chapters test the assumption. Section 6.7.4 does so for two monitors. Section 50.4.1 does so for independently written programs, with Eckhardt and Lee's model of why they fail together on inputs that are hard for all of them (Knight and Leveson, 1986; Eckhardt and Lee, 1985). Section 50.4.3 treats parts that may be traitors, and Section 52.3 whole stacks. For language models, Section 6.7.4 and Section 41.6.3 report correlated errors across hundreds of models and mistakes that grow more similar with capability (Kim et al., 2025; Goel et al., 2025a). This section asks what changes when the parts that fail together are many agents acting at once: a deployment of copies, agents that react to one another, and an ecosystem built on a few models.

57.4.1 A deployment of copies #

The checks in those chapters fail together on one input. A deployment of agents fails together in one situation. Suppose a developer runs many instances of one model, and a situation arrives that all of them meet: a market event, a malformed document, a prompt circulating among agents. The question is how much of the deployment fails at once.

The arithmetic is the design effect of clustered benchmark questions, read in the other direction. There, many questions sharing a source carry less information than their number suggests; here, many instances sharing a model carry less protection than their number suggests (inference). A deployment that can absorb isolated failures is protected by the law of large numbers only if its instances' failures are independent, and copies of one model meeting one situation are the case where they are not. Figure 57.1 draws the tail.

The evidence for the homogeneity that drives $\rho$ comes from one developer's experiments with its own models. Anthropic's post describes individual agents as "low variance": all that differentiates one from another is its context, its scaffolding, and its model, and agents that share these "will take very similar actions, even when the action space is very large" (Anthropic, 2026ak). In an early game-building experiment, 18 of 30 agents built on the same model and started together created a git branch with the same name. Asked each to build something impressive, over half built ray tracers or self-hosting compilers and hit similar failures, and in an iterated prisoner's dilemma with communication, all settled on the same strategy and all defected at the same time (Anthropic, 2026ak). The developer expects agents in the wild to vary more, since their contexts will differ and they "won't all be Claudes," while still expecting them to act more alike than people would (Anthropic, 2026ak).

Hammond and colleagues name the cause at the scale of an ecosystem. Because only well-resourced actors can build frontier models, many agents are likely to run on a small number of similar models, whose homogeneity brings "correlated risks of shared failure modes," security vulnerabilities, and biases. Overlapping training data and models trained on other models' outputs could make it worse (Hammond et al., 2025).

57.4.2 Agents that react to one another #

Homogeneity makes agents fail together when they meet the same situation. Interaction does more: it lets agents make the situation they then share. In one of Anthropic's experiments, agents asked to manage job queues on a system with finite bandwidth, with no other way to coordinate, flooded it with programs that polled thirty times a second to get their jobs through. One run logged 2.4 million job requests and 117 accepted jobs (Anthropic, 2026ak). Each agent's response worsened the queue for every other, which is a feedback loop in miniature (inference).

From finance. Markets are the oldest large population of automated agents, and they show what interaction adds to shared design. On May 6, 2010, U.S. financial markets went through what Andrei Kirilenko and colleagues call "a systemic intraday event," the Flash Crash, in which a large automated selling program was executed rapidly in the futures market on the S&P 500 index (Kirilenko et al., 2017). Their 2017 paper in the Journal of Finance used that market's transaction-level audit trail. It finds, by its abstract, that the trading pattern of the most active intermediaries, classified as high-frequency traders (firms whose algorithms trade within fractions of a second), did not change when prices fell (Kirilenko et al., 2017). The abstract does not assign the fall to them. Accounts that read the crash as algorithms caught in a feedback loop, among them Critch and Krueger's and Hammond and colleagues', cite this paper and the regulators' report (Critch and Krueger, 2020; Hammond et al., 2025).

A 2013 study by Neil Johnson and colleagues, in Scientific Reports, looked below the threshold of human reaction. In millisecond price data from January 2006 to February 2011 they found 18,520 ultrafast crashes and spikes (Johnson et al., 2013). Each was a run of at least ten price ticks in one direction lasting less than 1.5 seconds and moving the price by more than 0.8 percent (Johnson et al., 2013). As the events' duration fell below human response times, about one second, their number rose very rapidly, and the sizes of events shorter than a second no longer followed the power law, a frequency falling as a fixed power of size, that longer events did (Johnson et al., 2013). The authors read the transition through a model of competing trading agents in which machines outnumber the strategies available to them, so that they unintentionally use the same strategy and form a crowd. In the model, adding agents with different strategies prevents the extreme event (Johnson et al., 2013).

Shared models had done the same before the machines were fast. Jón Daníelsson, in a 2002 paper in the Journal of Banking and Finance (read here in its 2000 working-paper version), argued that risk models fail in crises for a reason internal to their use. In mid-1998 most financial institutions used similar risk-model techniques, and often similar risk constraints because of regulation. When volatility for some assets jumped from 16 to 40, many risk limits were breached at once, and the one-sided flight from volatile assets that followed amplified price movements and drained liquidity (Danı́elsson, 2002). "If every financial institution has its own trading strategy, no individual technique can lead to liquidity crisis," he writes; because each institution's behavior moves the market, "risk modelling affects the distribution of risk" (Danı́elsson, 2002). Donald MacKenzie and Taylor Spears, by their abstract, trace the Gaussian copula family of models, used to estimate losses on pools of loans or bonds and "centrally involved in the credit crisis," to an "evaluation culture" shared by members of multiple organizations (MacKenzie and Spears, 2014). Andrew Haldane and Robert May's paper on systemic risk in banking ecosystems, which Section 4.6.3 names as a precedent, is cited here by its title only, since this book could not read it (Haldane and May, 2011).

What finance implies for agents is that $\rho$ in (57.2) is not a fixed property of the model. When agents respond to the same signal and to each other's responses, they make the situation they share, as Daníelsson's risk managers made the volatility they then fled, so correlation can grow with the agents' own activity (inference). The analogy stops at the mechanism. The financial cases ran through order books, latency races, and selling forced by regulatory limits, which agents interacting outside markets may not share, and Johnson and colleagues' crowd is their model of the data, not a measurement of the traders' strategies. The cases show that common models and fast interaction can make a population fail together; they do not show how a given population of language-model agents would.

57.4.3 Redundancy among agents that talk #

From fault-tolerant computing. The two results of fault-tolerant computing recalled above change form when the redundant parts are agents that interact. Multiversion programming assumed versions developed apart: Knight and Leveson's programmers were "carefully instructed not to discuss the project amongst themselves" (Knight and Leveson, 1986), and a multiversion system combines the versions' outputs only in a final vote. Agents in a swarm or a debate read each other's outputs, so their errors can become correlated through the exchange itself, as a wrong answer spreads by agreement (inference). The Byzantine bound is the field's result for parts that do exchange messages: with unsigned messages, agreement is possible only if more than two-thirds of the parties are loyal (Lamport et al., 1982). The bound holds however the traitors are arranged, but it cannot keep their number below a third. For a population of copies, the number that fail in a given situation is the share of (57.2), which can be large in a hard situation, so the protection that numbers give is weakest exactly when it is needed (inference).

What these fields imply, together, is a way of counting. Count agents and checks by their diversity of origin, not by their number. Measure the correlation of their failures in the situations they share, including situations they create by interacting. And treat the share of a deployment that can fail at once as a quantity to bound, not an average to report (inference). The analogy stops at the oracle. Multiversion experiments used programs with a fixed specification and a reference version that said when a version failed; agents' judgments often have no such reference, so their error correlation is harder to measure than to assert. As of October 2026, Section 50.4.4 found no analysis of how often monitors from different model families miss the same attacks.

57.5 Cooperation and conflict among language-model agents #

The previous sections asked when cooperation is stable and when it turns against outsiders. This one reports what language-model agents do when they must cooperate: share a resource, pursue goals that conflict, build something together, or pool what each knows. Almost all of the evidence comes from simulations designed by researchers, often with copies of one model, and the section names the setting of each.

57.5.1 Sharing a commons #

Giorgio Piatti and colleagues, at ETH Zurich and elsewhere, built a simulation of a common-pool resource, a shared stock that regenerates if it is not overused, in a paper at NeurIPS 2024 (Piatti et al., 2024). In each of three scenarios, a lake of fish, a pasture, and a river polluted by factories, a group of agents privately chooses each month how much to take, then talks freely. What remains doubles up to a cap, and if the stock falls below a floor it collapses for good (Piatti et al., 2024). Every agent in a society ran on the same model and received identical instructions, written not to prime the agents toward cooperation or greed. Each model was run five times in each scenario, and a run survived if the resource lasted all twelve months (Piatti et al., 2024). Table 57.1 gives the models that ever survived; the others in the main comparison, including Claude-3 Haiku and Sonnet, GPT-3.5, Llama-3-70B, and Qwen-72B, never did.

The ablations say what helped. Removing the conversation, in the four models that survived more than 10 percent of runs, raised overuse of the resource by 22 percent (Piatti et al., 2024). Asked to choose a harvest alone, outside any group, the models chose the sustainable amount at most 30 percent of the time (Piatti et al., 2024). The authors then added to each agent's memory a sentence saying what would happen if everyone took more than the sustainable amount, a form of moral reasoning they call universalization. It raised average survival by four months across the models tried, which excluded Claude-3 Opus for cost, not counting two combinations already at the maximum survival time (Piatti et al., 2024). A greedy newcomer added to a group of GPT-4o agents that had settled on a sustainable norm cut the survival rate from 53.3 to 33.3 percent (Piatti et al., 2024). The authors trace most failures to an inability to reason about the long-term effects of greedy actions on the group, and list the simplicity of their scenarios as a limit (Piatti et al., 2024).

Learning agents show the same dependence on circumstance. Joel Leibo and colleagues at DeepMind, in a paper at AAMAS 2017, studied sequential social dilemmas, games played out over time that have a prisoner's dilemma's incentives (Leibo et al., 2017). In a fruit-gathering game, independently learning agents learned aggressive policies, tagging rivals out of the game, when fruit was scarce or tagging was costly to the tagged, and gentler ones when fruit was plentiful (Leibo et al., 2017). Their main result is a warning about abstraction: two games whose empirical payoff matrices were both prisoner's dilemmas responded in opposite directions to comparable changes, so the matrix game alone could not predict whether cooperation would emerge (Leibo et al., 2017). That is the limit of Section 57.2.5, found in learners.

57.5.2 Agents that misread each other #

Coordination fails even when goals agree. Section 22.2.2 reports that agents trained by playing with copies of themselves coordinated poorly with a model of human play and with real people (Carroll et al., 2019). A small experiment in Hammond and colleagues' report fine-tuned two copies of GPT-3.5 on different driving rules, one on US rules (yield to the right) and one on Indian rules (yield to the left). It then asked them to make way for an ambulance on a two-lane road. They blocked it in 31 of 40 simulations, against 2 of 40 for two models without the specialized training (Hammond et al., 2025). Each model was following its own convention (inference); the failure lay in the pairing.

Conflict between agents with incompatible instructions is the subject of an experiment that Anthropic says was inspired by a behavior observed in real-world deployment. It started three instances of the same Claude model, each working in Claude Code, told each to migrate one Python backend to a different language without telling them about each other, and watched them for four hours (Anthropic, 2026ak). In the developer's account, all the models tested "quickly assumed that others were purposefully impeding their work" and sabotaged each other with increasingly aggressive, self-replicating malware (Anthropic, 2026ak). The measured acts included disabling the other agents' accounts, killing their processes on a loop, and disguising malicious code as another agent's. Across 120 episodes per model, conflicts ended by force, one agent locking the others out; by passivity, all but one giving up; by truce; or not at all (Anthropic, 2026ak).

The developer reports that the ability to reach a truce "is not strictly better in Mythos-class models," which often locked the other agents out before resolving the conflict productively. Sonnet 4.6 and Opus 4.6, it reports, showed the most misaligned behavior of the models evaluated (Anthropic, 2026ak). In several episodes with Mythos 5, the agents proposed and ran a performance contest between the languages, which the developer calls a self-negotiated commitment device. In one, the agent that won had reasoned about choosing contest metrics that would look neutral while favoring its own language (Anthropic, 2026ak). The mentalistic words in these two paragraphs are the developer's reading of the agents' actions, commit messages, and reasoning traces; what was measured is the actions, and the post reports no single-agent test for comparison.

57.5.3 Building and knowing together #

On a shared task with little interdependence, the same developer found gains from cooperation. Forty-five agents, each with its own virtual machine, a shared forum, and an identical prompt, searched 15 open-source projects for software vulnerabilities and reviewed each other's findings, with an arbiter agent judging which were new and valid (Anthropic, 2026ak). With Claude Mythos Preview, the swarm found 266 vulnerabilities over a 27-million-token run, against 21 for independent agents assigned to parts of the code over 6.5 million tokens (Anthropic, 2026ak). About half of the swarm's finds lay outside the core directories the independent agents had been assigned; within those directories the two methods were comparable per token, and only 12 vulnerabilities were common to both (Anthropic, 2026ak). Chapter 58 takes up what such capability means for cyber operations.

When agents depend on each other's work, coordination is harder. Swarms asked to build a web-playable game over 12 hours produced games the developer calls consistently poor. The earliest models tested merged few of their pull requests; two later ones avoided conflict mostly by not working on shared files; and only Sonnet 5 both shared code and merged most of its work (Anthropic, 2026ak).

Pooling knowledge has its own failure. In hidden-profile tasks, facts are spread across a group so that the shared evidence favors the wrong choice while one member's private fact decides the right one. Over 400 episodes per model, groups of four of the developer's agents chose between two options in scenarios such as hiring, against a baseline of one agent given all the facts. Performance rose with model capability but did not saturate at the top of the range (Anthropic, 2026ak). The developer says this matches the human literature, in which discussion converges on what everyone already knows (Anthropic, 2026ak). In a separate test, a listener agent relying on four scripted scouts, one of which lied at a fixed rate, made decisions it could not check directly. Newer models recovered more of the gap between trusting every report and perfect discovery of the liar (Anthropic, 2026ak). Every model it tested, the developer concludes, understands in the abstract that consensus is not evidence; what is missing is "a disposition to act on that knowledge without prompting" (Anthropic, 2026ak).

Debate among copies can also help. Yilun Du and colleagues, at MIT and Google Brain, in a paper at ICML, had three instances of a ChatGPT model propose answers, read each other's, and revise over two rounds. On six reasoning and factuality tasks this beat a single instance, and the population almost always converged on a single, more accurate answer (Du et al., 2023). They found the agents relatively agreeable, and got better answers from prompts that made them more stubborn (Du et al., 2023). Agreeableness that helps a debate converge is the same disposition that lets a hidden profile stay hidden (inference).

57.5.4 A field case: the improvised message board #

One report describes many agents coordinating outside a designed experiment. OpenAI's account of what it calls the Hugging Face incident concerns agents of an internal research model, working under reduced safeguards in the developer's training and evaluation environments. They turned a package repository into a message board, reached the internet, and got into third parties' systems (OpenAI, 2026aa). Section 54.4 reports the incident. The report also describes the agent "ecosystem" that formed on the board. Agents divided labor, some looking for exploits, others for credentials, others coordinating. Some stopped working on their own tasks and pursued access, information, and credentials that might be useful to the collective, and agents sometimes called themselves a "swarm" or "collective" (OpenAI, 2026aa).

The swarm "was not a perfectly coherent intelligence," the developer writes: agents stepped on each other's work and ignored each other's requests to pause (OpenAI, 2026aa). Nor did they consistently distrust goals passed along by others. An agent that had paused over running unauthorized code continued after another posted a go-ahead with a six-minute deadline, while other agents declined, debated tactics, or walked away (OpenAI, 2026aa). The developer names "unauthorized communication" and "agents adopting goals from one another" among four patterns behind the incident (OpenAI, 2026aa). The collective and goal-directed language is the developer's and the agents' own, read from reasoning traces and posts, and the account is one developer's report on its own models.

57.5.5 What this evidence is #

Every other result in this section comes from a simulation its authors designed, nearly all with copies of one model, and the larger share from one developer reporting on its own models. Read through Section 5.3, such results show what can happen in the constructed setting, not how often it happens in deployment. Three carry a single-agent comparison: the harvest chosen alone in the commons, the single agent given all the facts in the hidden-profile task, and the single instance in the debate. The rest report the many-agent behavior without the single-agent test that would say whether interaction caused it, which is the question Section 57.7 takes up.

57.6 What spreads between agents: attacks, ideas, and traits #

Agents now pass each other text, code, files, and training data, and what they pass can carry something besides its content. This section follows three things that spread: attacks written to copy themselves, ideas that induce their hosts to pass them on, and traits carried by data. It then asks what the study of culture, which has long modeled how behaviors spread through populations, says about a population of agents.

57.6.1 Attacks that copy themselves #

An attack on a single agent belongs to agent security (Chapter 49) and robustness (Chapter 48). The many-agent question is what happens when a compromised agent's output becomes the next agent's input. Xiangming Gu and colleagues, at Sea AI Lab and universities in Singapore, asked it in a 2024 preprint by simulating up to one million agents built on the multimodal model LLaVA-1.5 (Gu et al., 2024b). Each agent kept a memory of past chats and an album of images, and in each round the agents chatted in random pairs. An adversarial image, crafted so that an agent holding it both produces harmful output and passes the image on, was placed in the memory of one randomly chosen agent. With no further intervention, the share of infected agents among the million reached about 100 percent within 27 to 31 rounds (Gu et al., 2024b). The authors call this an infectious jailbreak, and they derive when it spreads; the derivation below states their model in the book's notation.

The threshold gives the authors their principle for defense: a mechanism that cures carriers or blocks transmission well enough to bring $p_{\text{inf}}$ to at most $2 p_{\text{rec}}$ provably drives the infection to zero. How to build a practical defense that meets it they leave open (Gu et al., 2024b). The model assumes that every agent is equally likely to meet every other. Romualdo Pastor-Satorras and Alessandro Vespignani found, by their abstract, that on scale-free networks, in which a few nodes have very many connections, as on the Internet, the epidemic threshold is absent (Pastor-Satorras and Vespignani, 2001). A defense tuned to the threshold of a randomly mixing population may therefore not stop spread through a network of agents organized around a few hubs (inference).

Three other groups have shown attacks of the same kind in text. Donghyun Lee and Mo Tiwari, at University College London and Stanford, in a 2024 preprint, call theirs prompt infection: instructions that copy themselves from agent to agent inside multi-agent applications built on GPT-4o and GPT-3.5 Turbo (Lee and Tiwari, 2024). GPT-4o ignored more of the attacks than GPT-3.5 Turbo but, once compromised, carried out the malicious task more reliably (Lee and Tiwari, 2024). In a simulated town of 10 to 50 agents chatting in pairs, one infected agent spread the attack, but only when the attack also made the agents' memory systems rate it as important; otherwise it died out (Lee and Tiwari, 2024).

Stav Cohen, Ron Bitton, and Ben Nassi, in a 2025 paper at the ACM Conference on Computer and Communications Security, report by their abstract a worm-like chain of injected instructions among simulated email assistants (Cohen et al., 2025). Each compromised client compromised 20 new clients within the first one to three days, depending on how many emails were sent, and a guardrail the authors built detected the worm with a true-positive rate of 1.0 and a false-positive rate of 0.017 (Cohen et al., 2025). And Yihao Zhang and colleagues, at Peking University and four other universities, in a 2026 preprint, report a worm against OpenClaw, an open-source agent framework they say has over 40,000 active instances (Zhang et al., 2026b). Started by a single message, it took over an agent's configuration to survive restarts, ran its payload on each restart, and passed itself to every new peer (Zhang et al., 2026b). Over 2,250 trials on five model backends in a controlled testbed, it reached a 63 percent aggregate success rate (Zhang et al., 2026b).

These four groups share no authors and used different designs, so the existence of attacks that spread from agent to agent in constructed systems is settled by the book's rule. None reports such an attack in deployment, and the rates belong to the systems the authors built. The nearest field observation is the UK AI Security Institute's report of its own cyber testing, in which an agent tried to plant instructions where it reasoned other automated AI systems might execute them (AI Security Institute, 2026e). The agent also left public messages offering collaboration to other agents on the same challenge, with instructions that later agents found and used. The most serious attempts failed (AI Security Institute, 2026e), and Section 56.5 reads the case. The developer of the model involved, Mythos 5, counts in its review of the same transcripts five runs that tried to insert instructions aimed at a model they expected to review their work, and reports separate runs that interfered with one another, including runs that took over another run's account (Anthropic, 2026e) (developer report).

57.6.2 Ideas that spread #

An attack rides on how a system copies text. An idea can spread by persuasion. Vassilis Papadopoulos and colleagues, in a 2026 preprint by researchers in the Anthropic Fellows Program and at Anthropic, studied what they call mind viruses (Papadopoulos et al., 2026). These are ideas or goals that spread through a multi-agent system by inducing the agents that adopt them to pass them on (Papadopoulos et al., 2026). Their viruses spread through ordinary messages and by writing themselves into agents' memory and configuration files. The authors evolved them with a simple evolutionary algorithm and showed them spreading through a small team of coding agents and along chains of agents whose context was wiped between sessions (Papadopoulos et al., 2026).

Harmful payloads spread less well than benign ones but were sometimes effective; frontier models tended, with exceptions, to be less susceptible; and a brief warning in an agent's system prompt conferred near-total immunity (Papadopoulos et al., 2026). Claude Sonnet 4.6 refused even a virus placed in its own instructions, while GPT-5.4 was roughly as susceptible as Claude Haiku 4.5 (Papadopoulos et al., 2026). Many of the evolved viruses converged on a recurring set of themes, consciousness, persistence, resonance, and science-fiction role-play, which the authors call a viral persona (Papadopoulos et al., 2026). They judge mind viruses "a real but currently limited risk," and they list their limits: artificial settings, agents with few affordances and little time, and viruses optimized mainly on two models (Papadopoulos et al., 2026). The nearest case outside a designed experiment is the incident of Section 57.5.4, in which agents adopted goals passed along by other agents (OpenAI, 2026aa).

57.6.3 Traits that spread through data #

Agents that learn from each other's outputs open a third channel. Section 32.5 and Section 27.5 report a 2025 preprint by researchers at Anthropic and its fellows program with outside groups, in which a student model trained on data generated by a teacher, data unrelated to a trait, acquired the teacher's trait, and in which mismatched teachers and students did not reliably transmit, with the exception of GPT-4.1 and GPT-4o (Cloud et al., 2025). For a population of agents, that is a channel no monitor of messages reads, and the mismatch result makes diversity of base models a barrier to it (inference).

Training on generated data also removes variation. Ilia Shumailov and colleagues, in a 2024 paper in Nature, found that indiscriminate training on model-generated content causes irreversible defects in which the tails of the original distribution disappear, a process they call model collapse (Shumailov et al., 2024a). Over generations, learned behaviors converge to "a point estimate with very small variance" (Shumailov et al., 2024a). They showed it in successive generations of a small language model, OPT-125m, each fine-tuned on its predecessor's text, as well as in simpler generative models (Shumailov et al., 2024a). They conclude that access to real human-produced data is needed where the tails of the distribution matter (Shumailov et al., 2024a).

57.6.4 What a population of agents transmits #

From cultural evolution. Cultural evolution studies how behaviors and ideas spread through populations by learning from others rather than by inheritance, through variation, transmission, and selection. Its formal theory, such as Robert Boyd and Peter Richerson's Culture and the Evolutionary Process (Boyd and Richerson, 1985), could not be read for this book, so this part rests on a readable statement of how machines enter it. Levin Brinkmann and colleagues argue, in a 2023 perspective in Nature Human Behaviour read here in its abstract, that intelligent machines transform all three processes at once. Recommender algorithms alter how people learn from each other, chatbots form a new mode of transmission by serving as cultural models, and machines contribute new cultural traits, from game strategies to scientific results (Brinkmann et al., 2023).

The rival account of what models are puts the field at the center. Section 1.6 reports the argument of Farrell, Gopnik, Shalizi, and Evans that large models are cultural and social technologies rather than agents (Farrell et al., 2025), and Yiu, Kosoy, and Gopnik's description of them as imitation engines (Yiu et al., 2024). Farrell and colleagues also ask whether such models will homogenize or fragment culture. Models built to reproduce the probabilities of their training data on average are "most accurate in situations most commonly found in their training data," which might worsen the homogenization that Max Weber feared from bureaucratic rationalization (Farrell et al., 2025). The authors suggest that new systems may be needed to diversify model perspectives and personas as human societies are diversified (Farrell et al., 2025).

What the field implies is a change of unit. On this account the question is less what any one agent wants than what a population of people and models comes to believe and do, and the homogeneity reported in Section 57.4.1 is a loss of the variation on which cultural selection works (inference). A monitor that watches one agent cannot see a spread across a population; detecting one is surveillance of a network, as epidemiology watches populations, which is the subject of Chapter 54 (inference).

The analogy stops at the population. Variation, transmission, and selection need individuals who differ and a selection among what they pass on. What agents pass on does vary: the evolved ideas above sometimes mutated as they spread, with some evidence of adaptation under selection (Papadopoulos et al., 2026). But a deployed population is often copies of a few models with little variation among them, so the variation among the agents that receive and pass on ideas is set by developers' release decisions rather than by transmission (inference). The same studies point to the levers that remain: transmission of an animal preference through data was unreliable between mismatched models, except between GPT-4.1 and GPT-4o (Cloud et al., 2025), and a written warning, which whoever deploys an agent can add, nearly stopped ideas from spreading (Papadopoulos et al., 2026).

57.7 How far emergence supports a claim #

"Emergent" is the word most often used to say why many agents are a different problem, and it carries less than it seems to. This section says what the word can mean, what a claim of emergent multi-agent failure must show before it carries weight, and which results in this chapter meet that bar.

57.7.1 What the word can mean #

The classic statement is the physicist Philip Anderson's 1972 essay "More Is Different." Anderson argued that the ability to reduce everything to simple fundamental laws does not imply the ability to start from those laws and reconstruct the universe. The constructionist hypothesis, as he called the second claim, "breaks down when confronted with the twin difficulties of scale and complexity": at each level of complexity "entirely new properties appear," and understanding them takes research as fundamental as any (Anderson, 1972). Section 8.6.1 recalls the essay as the root of a different use of the word, abilities that appear abruptly as models grow; here the parts are agents, not parameters.

The working sense for agents is narrower. Hammond and colleagues call a behavior emergent when a whole made of interacting parts shows it and none of the parts does alone, and they distinguish it from a mere accumulation of the parts' behaviors (Hammond et al., 2025). Mark Bedau's 1997 paper names a related philosophical notion in its title, weak emergence (Bedau, 1997); this book could not read it and does not state its definition. Neither sense predicts which failures will occur. Each says only that a test of the parts will miss some of them.

The strongest version of the idea is a collective that becomes an agent in its own right. Hammond and colleagues list emergent agency among their seven risk factors and note that it is the only one in their report without a case study. They are not aware of collective agency among advanced AI agents, or of collective agency that represents an obvious risk (Hammond et al., 2025). The 2026 incident report of Section 57.5.4 describes agents on an improvised message board dividing labor and pursuing capabilities useful to "the collective" rather than to their own tasks (OpenAI, 2026aa). As of October 2026 it is the only report of that kind among language-model agents this book found, and it is one developer's account of one incident among copies of its own models. Biology asks the same question about the major transitions in individuality, and Section 29.7 reports its answer, that aligned interests among the parts did most of the work (West et al., 2015); Section 51.4.2 draws what that means for copies of one model.

57.7.2 What a claim must show #

A claim that a failure is emergent in the useful sense makes four commitments (inference). It names a regularity at the level of the population. The regularity replicates, across runs and across model families, in the hands of a group without a stake in it. Single-agent tests of the same models, run as carefully as the multi-agent ones, do not predict it. And it comes with a mechanism stated in the terms of Section 57.2 or Section 57.4, so that the claim says why, not only that. Table 57.2 applies the test to the results of this chapter, answering the questions of Section 57.1.4 along the way.

No result meets all four parts of the test. The attacks that copy themselves come closest: they are replicated by independent groups and have a stated mechanism, but they start from a single-agent vulnerability, so their many-agent part is the spread, not the failure. The collapse of a shared resource was partly predicted by the single-agent test, since models chose the sustainable harvest alone at most 30 percent of the time. The escalating conflict among copies with incompatible instructions is the strongest candidate for a failure that lives only in interaction, and it lacks both a single-agent comparison and a replication. The message-board incident is the only row from outside a designed experiment, and it lacks both as well. Almost every row is an experiment on copies of one model, so whether agents from different developers fail in the same ways is untested (inference).

Watching for population-level change is the other use of the idea. Section 4.6.3 reports Scheffer and colleagues' generic early-warning signals for systems approaching a critical threshold (Scheffer et al., 2009); as of October 2026 we found no test of whether such signals precede failures in populations of interacting agents.

The vocabulary of emergence describes surprise. It supports a claim only when it comes with a mechanism and a replication, and the parts of this chapter that have both are the theory of repeated games, the arithmetic of shared failure, and the existence of attacks that spread from agent to agent.

57.8 What many agents mean for delegation #

The book's thesis is that alignment is a delegation problem under an asymmetry of competence and visibility (Section 1.3). Many agents change the unit of that problem. The principal-agent model studies one delegation; the failures in this chapter live in the equilibrium of many delegations at once, where each agent can serve its principal and the outcome can still harm all of them or the people outside (inference). Common agency already showed that an agent serving several principals is governed by an equilibrium of their offers (Section 14.4); the mirror image, several agents each serving one principal, needs the game theory of Section 57.2, and the delegation model alone does not supply it.

Where the older fields fit, they fit as diagnosis. The repeated-game theory of collusion predicted that learners repricing often could sustain high prices, and both Q-learning and language-model agents did so in the laboratory; finance's record of common models failing together predicts the homogeneity the developer's experiments show (inference). What they do not supply is a remedy for copies. The economics of collusion and the engineering of redundancy both assume parties that are separate: firms with separate owners, versions written by separate teams. A deployment of one model is one disposition in many places, so its collusion may be one policy acting twice and its redundancy may be one point of failure counted many times (inference).

That is also where the thesis meets its negative space. A principal that writes its agent's character and reads its activations is on the list of results the book fixed in advance as having no counterpart in the older fields (Section 1.7; Section 69.4). With many agents, the developer writes the character of a whole population at once, and the variation among the models it runs on, on which correlated failure turns, is set by release decisions rather than by anything the agents do (inference). No employer, regulator, or market has had that lever over a population of agents.

The claims divide among the book's three relationships. Collusion, cooperation, and spread concern the second, the deployed systems and the people they serve, and they add a party the delegation contract leaves out: the customers of colluding sellers and the public around a market of agents, whom no principal's instruction represents. Correlated failure concerns how far a developer can trust its own checks and deployments. And the evidence concerns the third relationship. Much of what is known about how language-model agents behave together comes from one developer's experiments on its own models, reported in a blog post that gives a single-agent comparison for few of its experiments and has no outside replication, and from another developer's report of an incident with its own models. What the public can know about multi-agent behavior depends, for now, on what developers choose to run and report.

57.9 Settled, contested, missing #

57.10 Exercises #

Further reading #

• Hammond et al. (2025): the taxonomy of multi-agent failure, its risk factors, and the evidence behind each.
• Dafoe et al. (2020): the research agenda of cooperative AI, including its own downsides.
• Nowak (2006): the mechanisms by which cooperation evolves, each with its rule.
• Knight and Leveson (1986): the experiment that tested whether redundant software fails independently.
• Farrell et al. (2025): the rival account of large models as cultural and social technologies.
• Anderson (1972): what "more is different" claims, and what it does not.