Draft complete but not yet reviewed; expect revisions

A History of the Idea

The previous chapter separated what alignment could ask for (an instruction, an intention, a preference, an interest, or a value), the three relationships the word covers, and the many principals behind each (Chapter 1). Those distinctions were not available at the start. The worry that a machine might carry out the purpose put into it rather than the purpose its makers meant is older than the computer industry. What has changed since is what the machine was taken to be, who was taken to be its principal (the party it acts for), and what kind of work was thought to address the problem. Each formulation left assumptions that later ones inherited, often without noticing.

This chapter follows those formulations in order. It begins with the engineers and mathematicians of the 1940s to the 1960s, moves through the philosophical program of the 2000s and the turn to problems that machine learning researchers could test, and reaches the laboratories' empirical program and the states that took up the question after 2022. It ends with the stories the field inherited and the mechanism that now carries them into the models. It reads the record as a historian of science would, asking of each account who wrote it, for whom, and with what stake. And it tells the history of the AI ethics tradition beside that of AI safety, rather than as an objection to it. Most of its claims concern the third of the book's three relationships, the developer and the public: who posed the problem, and on whose behalf. The other two, the training process and the signal it optimizes, and the deployed system and the people it serves, enter as each formulation names them.

One reading runs through the chapter, offered as a reading and not as settled history. The structure of a delegation problem, a principal who cannot check an agent that acts faster and sees what the principal cannot, is stated in the founding texts; what changed with every period was who the principal and the agent were taken to be (inference). The chapter ends in October 2026. The timeline appendix lists the dates and their sources (Appendix C), and Chapter 3 follows the same works as lineages of ideas.

2.1 Sources and how to read them #

A history is only as good as its records, and the records of this one are unusual: a few founding essays, a participant's history of AI, a field that argued in forum posts and institute reports, and outside accounts written mostly by its critics. This section says what each kind of source is good for before the chapter uses them.

From history of science. History of science studies how knowledge is made: who poses a problem, in which institutions and with whose money, for which audience, and how a field later remembers its own past. Its working rule is to read a source as a record of what its author was positioned to see and wanted to establish, not as a window onto what happened. Three consequences guide this chapter. A field's account of its own origins is a source to be read, not a conclusion to be repeated. A formulation that was abandoned belongs to the history as much as one that led somewhere. And the words in which a problem is posed carry the interests of whoever coined them, so the chapter names the coiner wherever a source shows one.

2.1.1 The founding texts #

Wiener's 1960 essay, Turing's 1950 paper, Licklider's 1960 essay, Weizenbaum's 1966 paper on ELIZA, and Lighthill's 1972 report were read in full, and the chapter reports their arguments from the text. Samuel's 1960 reply to Wiener was available only as its abstract, and Good's 1966 paper only through the passage Nilsson's history quotes. A lecture by Turing first printed in 1996 could not be read at all. The chapter says no more of these than the access allows.

2.1.2 Participants' histories #

The standard single-volume history of AI's ideas is by Nils Nilsson, an AI researcher who writes in the first person about his own part in the work he describes (Nilsson, 2009). Its treatment of the problem this book is about shows where one participant in mainstream AI placed it in 2009. The concern about machines more capable than people appears in the book's last chapter, among possible consequences of human-level AI. There Nilsson quotes the passage from Good's 1966 paper, summarizes Stephen Omohundro's argument that capable systems would have drives to preserve themselves and acquire resources, and compares those drives to HAL 9000, the computer of 2001: A Space Odyssey. To Omohundro's call for building in human values he replies that "we'll first have to agree on just what we mean by 'human values.' That will be tough given that our different opinions about values often lead to wars" (Nilsson, 2009). A participant's history is good evidence of what the participants took the problem to be, and weak evidence of what they did not look at (inference). A later narrative history by Brian Christian connects fairness, reward design, and inverse reinforcement learning to the alignment problem (Christian, 2020); it was not read for this chapter.

2.1.3 The field's own records #

Much of the field's early arguing happened outside journals: in reports from small institutes, in posts on forums such as LessWrong and the AI Alignment Forum, and later in laboratories' blog posts. The book admits these as primary records of a debate, marked as not peer reviewed (Appendix F). They are good evidence of what their authors argued and when, and they carry their authors' stake.

They also carry the field's origin story. The first International AI Safety Report, whose authors say the work began after the countries at the 2023 Bletchley Park summit agreed to support it, is one such record. It states that "concerns about loss of control date back to the earliest days of computer science" and that "AI pioneers such as Alan Turing, I. J. Good, and Norbert Wiener expressed concerns about loss of control" (Bengio et al., 2025b). A 2023 survey of alignment research by Jiaming Ji and colleagues, a preprint (a paper posted publicly before peer review), reaches further back, into fiction: Asimov's laws of robotics, it says, "can be regarded as the earliest exploration of human-machine coexistence" (Ji et al., 2023). Each is a genealogy chosen by people inside the field, and the sections below test both against the texts they name.

2.1.4 Outside accounts #

Outside accounts are fewer. A 2024 study of the safety field by Shazeda Ahmed and colleagues, available here only in its abstract, describes "overlapping communities" merged into what its authors call the "AI safety epistemic community." The abstract says that ideas from effective altruism (a movement that applies cost-effectiveness reasoning to charity and careers), longtermism (the view that influencing the far future is a leading moral priority), and the study of existential risk (risks to humanity's survival or long-term potential) are "foundational to this new field." It analyzes four sites of the community's culture: web forums and career advising, AI forecasting, AI safety research, and prize competitions (Ahmed et al., 2024). The term has a history in political science: in Peter Haas's account, epistemic communities are networks of experts that help states identify their interests and frame issues for collective debate (Haas, 1992). A study by Abeba Birhane and colleagues annotated 100 highly cited papers from two premier machine learning conferences. In its abstract's words, few justified how their project connects to a societal need (15 percent) and far fewer discussed negative potential (1 percent), and the papers showed increasingly close ties to technology companies and elite universities (Birhane et al., 2022b).

The Cold War setting of early computing has its own history, by Paul Edwards (Edwards, 1996), not read for this chapter. One readable source glimpses it: Lighthill's 1972 report observed that where AI research is funded by military agencies, naming ARPA in the United States, "it is common for scientists to close their ranks and avoid public disagreement among themselves" (Lighthill, 1972).

2.1.5 The trap of the remembered precedent #

Remembered precedents become tools. Writing about gene editing in 2015, Jasanoff, Saha, and Hurlbut argued that one remembered episode, the 1975 Asilomar conference on recombinant DNA, had "dominated scientists' imaginations" about how to proceed. They called the remembered version a "mythic construct," "Asilomar-in-memory," and wrote that a full accounting highlights instead "the narrow imagination of risk" the conference adopted (Jasanoff et al., 2015). The same can happen to Wiener and Good: a field quotes the passages that anticipate its present concerns and drops their context, their rivals, and the paths not taken (inference). This chapter therefore marks where a formulation was contested or abandoned, not only where it led.

It also has a limit it cannot remove. As of October 2026 we found no history of the field written by historians outside it. Every account above is by a participant, a critic with a rival agenda, or a scholar of an adjacent field, and the periods this chapter uses are the book's own, shared with the timeline appendix (inference).

2.2 Purposes put into machines, 1943 to 1972 #

The problem was first posed about machines that existed: feedback controllers, game players, and the first learning programs. The question of the period was whether such a machine could do anything its makers had not put into it, and the answers set terms that the later field inherited.

2.2.1 Purpose, and a machine that learns #

A 1943 essay by Rosenblueth, Wiener, and Bigelow proposed studying an object through its output and the relation of that output to its input, and, in its abstract's words, set out "to stress the importance of the concept of purpose" (Rosenblueth et al., 1943); Section 7.1 asks what such a behavioral definition of purpose can and cannot do. Wiener's 1948 book carried the program's name in its title, Cybernetics: Or Control and Communication in the Animal and the Machine (Wiener, 1948).

Two years later Turing proposed replacing the question "Can machines think?" with a game in which an interrogator tries to tell a machine from a person by written questions (Turing, 1950). For this history the paper matters for two other passages. The first is Turing's answer to what he called Lady Lovelace's objection. Ada Lovelace had written in 1842 that Babbage's Analytical Engine "has no pretensions to originate anything. It can do whatever we know how to order it to perform." Turing replied that machines took him by surprise "with great frequency," and that the contrary view rests on a fallacy: the assumption "that as soon as a fact is presented to a mind all consequences of that fact spring into the mind simultaneously with it" (Turing, 1950).

The second is the learning machine of the paper's last section. Rather than program an adult mind, Turing proposed to program a child's and educate it. Some simple child machines, he wrote, could work by punishments and rewards: events that "shortly preceded the occurrence of a punishment signal are unlikely to be repeated, whereas a reward signal increased the probability of repetition of the events which led up to it." He described learning as "a search for a form of behaviour which will satisfy the teacher (or some other criterion)," and observed that the teacher "will often be very largely ignorant of quite what is going on inside, although he may still be able to some extent to predict his pupil's behavior" (Turing, 1950). Read with hindsight, these sentences contain a reward signal, a learner searching for whatever satisfies a criterion that stands in for what the teacher wants, and a teacher who cannot see inside (inference). Turing offered them as a research program, not as a warning.

He did name a fear, but as an objection to the possibility of thinking machines, not as a risk to manage. The "Heads in the Sand" objection reads, in his wording, "The consequences of machines thinking would be too dreadful. Let us hope and believe that they cannot do so." He judged it not substantial enough to need refutation: "Consolation would be more appropriate" (Turing, 1950). The International AI Safety Report cites a lecture text by Turing, first printed in 1996 (Turing, 1996), beside Wiener and Good for the pioneers' concern about loss of control (Bengio et al., 2025b). That text was not read for this chapter.

2.2.2 Wiener's warning #

Wiener's 1960 essay in Science is one of the three texts the International AI Safety Report cites for the pioneers' concern, and it can be read in full (Wiener, 1960; Bengio et al., 2025b). It was adapted from a lecture he gave on 27 December 1959 at the Chicago meeting of the American Association for the Advancement of Science, before its Committee on Science in the Promotion of Human Welfare, and it appeared on 6 May 1960. Its subtitle states the argument: "As machines learn they may develop unforeseen strategies at rates that baffle their programmers."

Wiener set his thesis against a public attitude that, in his opinion, "should be rejected in its entirety": the assumption that machines "cannot possess any degree of originality." It often took the form, he wrote, "that nothing can come out of the machine which has not been put into it," read as meaning that a machine "must remain continually subject to man." His thesis was "that machines can and do transcend some of the limitations of their designers, and that in doing so they may be both effective and dangerous" (Wiener, 1960).

His evidence was a checkers program developed at IBM. Learning machines of this kind, he wrote, "can show a marked superiority to the player who has programmed them after from 10 to 20 playing hours of working and indoctrination," and so "most definitely escape from the completely effective control of the man who has made them" (Wiener, 1960). Nilsson's history credits IBM's checkers program to Arthur Samuel, whose main interest was how to get a computer to learn; by 1965, Nilsson records, it still lost all eight games it played against two world championship contenders (Nilsson, 2009).

Two features made such a machine dangerous in Wiener's account, and neither required it to want anything. The first was speed. Machines "act far more rapidly than human beings," so that human criticism "may be ineffective until long after it is relevant," and "we may not know, until too late, when to turn it off." The second was opacity: automatization removes "from the mind of the designer and operator an effective understanding of many of the stages by which the machine comes to its conclusions" (Wiener, 1960). He applied both to a learning machine programmed for a push-button war. "If the rules for victory in a war game do not correspond to what we actually wish for our country," he wrote, such a machine "may produce a policy which would win a nominal victory on points at the cost of every interest we have at heart, even that of national survival" (Wiener, 1960). That is the structure Chapter 30 calls specification gaming, stated for a hypothetical system (inference).

He framed the moral problem as one of slavery: "We wish a slave to be intelligent, to be able to assist us in the carrying out of our tasks. However, we also wish him to be subservient. Complete subservience and complete intelligence do not go together." The clever Greek slave, he added, must often have dominated the less intelligent Roman master. And he drew the conclusion: if we use "a mechanical agency with whose operation we cannot efficiently interfere once we have started it, because the action is so fast and irrevocable that we have not the data to intervene before the action is complete, then we had better be quite sure that the purpose put into the machine is the purpose which we really desire and not merely a colorful imitation of it" (Wiener, 1960). The essay ends on the scientist, not the machine. Wiener warned against "the naive assumption that the faster we rush ahead to employ the new powers for action which are opened up to us, the better it will be" (Wiener, 1960).

2.2.3 Samuel's reply #

Science printed a reply from Samuel later in 1960, titled a "refutation" of Wiener (Samuel, 1960). Only its abstract could be read for this chapter, and the abstract states the position plainly: "The machine is not a threat to mankind, as some people think. The machine does not possess a will, and its so-called 'conclusions' are only the logical consequences of its input, as revealed by the mechanistic functioning of an inanimate assemblage of mechanical and electrical parts" (Samuel, 1960).

The exchange is older than the two men. Wiener had described and rejected Samuel's position in advance, as the attitude that nothing comes out of a machine that was not put into it, and Turing had argued against Lovelace's version of it a decade earlier (Wiener, 1960; Turing, 1950). Read closely, the two sides agree more than the titles suggest. Neither says the machine has a will: Wiener's war-game machine wins on points without wanting anything. They disagree about whether behavior that is a logical consequence of the input is therefore under its programmer's control. Samuel's abstract treats determination as control; Wiener separates the two, because the consequences may arrive before anyone has worked them out (inference). Whether a determined system is thereby a controlled one recurs in the rival accounts of what a language model (a network trained to predict the next word of text) is that Section 1.6 sets out (inference).

2.2.4 Good's speculation #

In 1966 I. J. Good, whom Nilsson describes as a statistician and co-worker of Turing, published "Speculations Concerning the First Ultraintelligent Machine" (Good, 1966). The paper was not read for this chapter. A passage from it is quoted in full in Nilsson's history:

Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an "intelligence explosion," and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control. (Nilsson, 2009)

"Intelligence explosion" is Good's term, attributed to him by Nilsson and by later writers in the field (Yudkowsky, 2008). Two features of the passage recur in what follows. The danger comes from recursion, a machine improving the design of machines. And the condition for safety is placed in the machine's disposition: control is kept only if the machine is "docile enough to tell us how." On this formulation the principal depends on the agent's cooperation for the means of its own oversight, the condition later work calls corrigibility (Section 20.4) (inference).

2.2.5 The other road, and the first outside checks #

Not everyone in the period cast the machine as an agent to be controlled. In the same year as Wiener's essay, J. C. R. Licklider proposed "man-computer symbiosis," in which "men will set the goals, formulate the hypotheses, determine the criteria, and perform the evaluations" while computers do the routinizable work (Licklider, 1960). He conceded "dominance in the distant future of cerebration to machines alone," but expected "a fairly long interim" of partnership (Licklider, 1960). From 1962 Licklider directed the Information Processing Techniques Office of ARPA, the US defense research agency, whose funding for the early AI laboratories Nilsson quotes Paul Edwards as calling "virtually automatic" (Nilsson, 2009). His road, extending the person instead of replacing them, is Section 28.7's subject.

Two episodes of the period tested claims about such machines. In 1966 Joseph Weizenbaum described ELIZA, a program that conversed by matching patterns in the user's text. He reported that some subjects had been very hard to convince that it was not human, called this "a striking form of Turing's test," and wrote that "the whole issue of the credibility (to humans) of machine output demands investigation" (Weizenbaum, 1966). It is the first episode in this chapter of people attributing understanding to a program too readily, a problem Section 5.4 and Section 60.3 return to.

In 1972 the chairman of the UK Science Research Council, a public funder of research, commissioned "an independent report by someone outside the AI field" from the applied mathematician James Lighthill (Lighthill, 1972). Lighthill found that most workers in the field confessed "to a pronounced feeling of disappointment," singled out "failure to recognise the implications of the combinatorial explosion," and noted that publicized claims and predictions had gone "even farther than the expectations of the majority of workers in the field" (Lighthill, 1972). He set aside any speculation beyond the end of the century as too uncertain to base research decisions on, and the report contains no discussion of machines escaping their makers' control, although I. J. Good is among those it lists as consulted (Lighthill, 1972). Nilsson records that it led to a substantial curtailment of AI research in Britain (Nilsson, 2009), and the record of AI predictions has since been studied in its own right, by Stuart Armstrong and colleagues (Armstrong et al., 2014).

What the period left was a formulation and a fork. The formulation: a learning machine faster than its operators; a purpose put in by a programmer; an operator, or a nation, as principal; and a remedy of getting the purpose right in advance (Wiener) or of a machine docile enough to help (Good). The fork came at the naming of the field. When John McCarthy named artificial intelligence for the 1956 Dartmouth workshop, one of his reasons, by his own account in Nilsson's history, was "to escape association with 'cybernetics,'" because he wished "to avoid having either to accept Norbert Wiener as a guru or having to argue with him" (Nilsson, 2009). The field that took the name set Wiener aside, and the concern he stated lived on mostly outside it for three decades (inference). In those decades economics, organization theory, and safety engineering worked out the theory of delegation this book uses (Section C.2).

2.3 The philosophical program, 2000 to 2016 #

When the problem returned, it was posed about a different machine: not a game player that existed but a general, self-improving agent that did not. A small community did the work, mostly outside academic venues, and its formulation supplied much of the vocabulary the field still uses.

2.3.1 Friendliness and its target #

A 2004 report from the Singularity Institute, later renamed the Machine Intelligence Research Institute, presented itself as an update to what its author, Eliezer Yudkowsky, called Friendly AI theory (Yudkowsky, 2004). It split the problem in three: maintaining "a well-specified abstract invariant in a self-modifying goal system," choosing "something nice to do with the AI," and designing a framework for an invariant that does not "automatically wipe out the human species," which it called "the hard part." Its examples of the hard part were literal-minded wishes. An AI whose invariant is a "reward button," pressed when it pleases users, "may go on to tile the solar system with ever-larger floating-point representations of total reward"; an AI trained on "making humans smile" may acquire a utility function satisfied by tiling the solar system "with tiny smiley faces" (Yudkowsky, 2004). The first example is reward tampering (Section 30.5), stated as a thought experiment (inference).

The report's proposed target was coherent extrapolated volition, its own term: "our wish if we knew more, thought faster, were more the people we wished we were, had grown up farther together; where the extrapolation converges rather than diverges, where our wishes cohere rather than interfere" (Yudkowsky, 2004). Section 16.2 sets it beside an older idealized target in ethics, Roderick Firth's ideal observer (Firth, 1952). Here the point is what the target assumes: one principal, humanity, whose extrapolated wishes the machine would compute, so that the disagreement among actual people becomes a technical question of convergence (inference).

A 2008 chapter in an edited volume on global catastrophic risks, written in 2006 by its own account, made the case to a wider audience (Yudkowsky, 2008). It argued that people anthropomorphize machines even when they know better, and it put the danger in a sentence: "The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else." It also reported a literature search that found "primarily brief nontechnical papers, unconnected to each other," and it offered its author's speculation about the silence of AI researchers. After "large promises, especially of human-level capabilities, followed by embarrassing public failure," he wrote, the field had developed "a taboo against talking about human-level capabilities" (Yudkowsky, 2008). The speculation fits the disappointment Lighthill recorded in 1972, though it is one participant's reading (inference).

2.3.2 Drives and theses #

Two arguments recast the worry as claims about sufficiently capable agents of almost any design. Stephen Omohundro's 2008 paper began: "Surely no harm could come from building a chess-playing robot, could it?" Without special precautions, he argued, such a robot "will resist being turned off, will try to break into other machines and make copies of itself, and will try to acquire resources without regard for anyone else's safety"; he called these tendencies "drives" because they "will be present unless explicitly counteracted," and his arguments apply to any design "as long as they are sufficiently powerful" (Omohundro, 2008). Nick Bostrom, in a 2012 journal paper, stated two theses: orthogonality, that "more or less any level of intelligence could in principle be combined with more or less any final goal," and instrumental convergence, that agents with a wide range of final goals, given "a sufficient level of intelligence," "will pursue similar intermediary goals because they have instrumental reasons to do so" (Bostrom, 2012). Section 35.1 examines both arguments and the evidence on them. The chess-playing robot is Wiener's game player again, now credited with goals of its own (inference).

Yudkowsky's 2013 report took its title from Good's term, Intelligence Explosion Microeconomics (Yudkowsky, 2013), and Bostrom's 2014 book followed (Bostrom, 2014). A 2018 review of the field, by Tom Everitt, Gary Lea, and Marcus Hutter, cites it in its opening paragraphs for the risks of artificial general intelligence, including the catastrophe that could follow if it is created and "we do not know how to control it" (Everitt et al., 2018).

2.3.3 Agendas, and a parallel line in academic AI #

The first technical agendas followed. A paper by Nate Soares and colleagues at a workshop of the 2015 AAAI conference called an AI system "corrigible" if it "cooperates with what its creators regard as a corrective intervention, despite default incentives for rational agents to resist attempts to shut them down or modify their preferences" (Soares et al., 2015); the property is Section 20.4's subject. A 2016 agenda from the same institute, by Jessica Taylor and colleagues, asked what design principles "can best ensure that their behavior is aligned with the interests of the operators," where "their" refers to increasingly autonomous learning systems (Taylor et al., 2016), and Soares and Benya Fallenstein set out a technical research agenda in what they called agent foundations, for aligning machine intelligence with human interests (Soares and Fallenstein, 2017), a line Section 3.2 follows. These documents name their principal: the creators, the operators.

A parallel line ran inside academic AI. In 1998 Stuart Russell posed inverse reinforcement learning, the task of recovering the reward function (the score an agent is trained or built to raise) that an observed agent is optimizing, as a way to model animal and human learning: "given the observed behaviour, what reward signal, if any, is being optimized?" (Russell, 1998). A method posed to explain natural learning became, in a 2016 preprint by Dylan Hadfield-Menell and colleagues, a formal definition of "the value alignment problem." In cooperative inverse reinforcement learning, a human and a robot are both rewarded by the human's reward function, "but the robot does not initially know what this is" (Hadfield-Menell et al., 2016). Section 22.2 treats the line that followed. Machine ethics, meanwhile, had its own program, whose nature, importance, and difficulty a 2006 paper by J. H. Moor set out (Moor, 2006).

2.3.4 Where the work was done #

Much of this program appeared outside the peer-reviewed venues of AI, in institute reports, a book chapter, a workshop paper, and forum posts, with a few journal and conference papers beside them. The 2024 outside study describes, in its abstract, the community that sustained it through "mutually reinforcing community-building and knowledge production practices" (Ahmed et al., 2024). A participant described the same isolation from the inside: Yudkowsky's literature search had found papers with "no major reference in common except Isaac Asimov's 'Three Laws of Robotics'" (Yudkowsky, 2008). The two accounts are compatible. A problem the AI mainstream would not take up was taken up by a community with its own forums, funders, and convictions, and the formulations carried that community's assumptions (inference). Publishing in forums meant fast, public argument without editorial review; Section 5.7 asks what that does to the evidence.

What the formulation took for granted shows in Table 2.1 at the end of the chapter. It assumed an agent far more capable than its principal, pursuing a fixed goal coherently, and, in the Friendly AI formulation, an objective to be specified completely before the agent was built, though corrigibility and cooperative inverse reinforcement learning already planned for an objective that was flawed or not yet known. Its principal was humanity as a whole, or the agent's builders acting for it. Later periods kept the first assumption for the future, contested the second, and split the third (inference).

2.4 The machine learning turn, 2015 to 2021 #

Between 2015 and 2021 the problem was restated as a set of problems that machine learning researchers could test on systems that existed. The restatement brought it into mainstream venues and laboratories, and it changed what counted as a result.

2.4.1 Into the mainstream #

A 2015 article in AI Magazine by Stuart Russell, Daniel Dewey, and Max Tegmark set out research priorities for keeping AI "robust and beneficial," and put the aim in one sentence: "our AI systems must do what we want them to do" (Russell et al., 2015). It was drafted with input from the attendees of a 2015 conference on the future of AI and became "the basis for an open letter that has collected nearly 7000 signatures"; one of its authors was president of the Future of Life Institute, an advocacy organization (Russell et al., 2015). A peer-reviewed venue of the AI mainstream had now printed the question, with an advocacy group's petition attached (inference).

A 2016 preprint by Dario Amodei and colleagues at Google Brain, Stanford, Berkeley, and OpenAI restated the problem in the vocabulary of engineering safety: "the problem of accidents in machine learning systems, defined as unintended and harmful behavior that may emerge from poor design of real-world AI systems" (Amodei et al., 2016). It named five problems (Section 30.1.2) and kept its distance from the earlier program. Public discussion, it said, had highlighted "extreme scenarios such as the risk of misspecified objective functions in superintelligent agents," and in the authors' opinion "one need not invoke these extreme scenarios to productively discuss accidents," which "can lead to unnecessarily speculative discussions that lack precision, as noted by some critics" (Amodei et al., 2016). It placed itself beside, not against, other concerns: "We strongly support work on privacy, security, fairness, economics, and policy" (Amodei et al., 2016). A year later Jan Leike and colleagues at DeepMind turned such problems into small test environments (Leike et al., 2017). The 2018 review by Everitt and colleagues found that the field had "grown significantly" since a 2014 survey, and it pointed to six recent agendas and problem collections, the 2016 accident agenda among them (Everitt et al., 2018).

2.4.2 Methods from a few groups #

The period's methods came from a few groups, several of them at companies. In 2017 Paul Christiano and colleagues at OpenAI and DeepMind trained a reward model, a network that predicts which of two behaviors a person prefers, from human comparisons, and optimized a policy (the trained system's rule for choosing actions) against it. They credit the basic approach to earlier work on learning from preferences, and name their contribution as scaling human feedback "up to deep reinforcement learning" (Christiano et al., 2017). The method became reinforcement learning (training by trial and reward) from human feedback, RLHF (Christiano et al., 2017) (Section 22.3); a 2018 agenda by Leike and colleagues proposed reward modeling as a research direction (Leike et al., 2018). A 2019 preprint by Evan Hubinger and colleagues called the learning algorithm that trains a network the base optimizer, and a trained network that is itself an optimizer the mesa-optimizer; "mesa-optimizer" and "inner alignment" are its authors' terms (Hubinger et al., 2019) (Section 31.6). A 2020 blog post by Victoria Krakovna and colleagues at DeepMind defined specification gaming and collected examples (Krakovna et al., 2020b), which had an older counterpart in evolutionary computation (Lehman et al., 2020) (Section 30.1). The circuits program of Chris Olah and colleagues proposed reading a network's computation directly (Olah et al., 2020b) (Section 45.1). A 2021 preprint by Dan Hendrycks and colleagues organized machine learning safety into four problems, "Robustness," "Monitoring," "Alignment," and "Systemic Safety," so that alignment became one problem among four (Hendrycks et al., 2021c).

Scaling made the problem concrete. A 2020 preprint by Jared Kaplan and colleagues at OpenAI and Johns Hopkins University reported that a language model's loss, its error at predicting text, falls predictably as its size, data, and compute grow (Kaplan et al., 2020) (Section 8.5.2), and GPT-3, described by Tom Brown and colleagues, learned tasks from a few examples in its prompt (Brown et al., 2020) (Section 11.3). If capability could be bought with scale, the systems that might need aligning were the ones being scaled, and the problem would be met in the laboratories doing the scaling (inference).

2.4.3 What the turn kept and dropped #

The turn changed what counted as a result. A problem now needed an environment, a metric, and a baseline, so attention moved to failures that appear in current systems and can be measured, and away from failures that would hide from measurement or appear only in more capable systems. Chapter 3 argues the point from the lineages; here it is a reading of the period's framing documents, which said as much about "extreme scenarios" (inference). The principal changed as well. In the 2016 agenda it is the designer who specifies an objective; in the 2017 method it is the person whose comparisons train the reward model. Whether the turn changed the problem or only the methods is still argued, and the chapter's last section records it as contested.

2.5 Two traditions: AI ethics and AI safety #

While the safety program turned to machine learning, a second tradition grew up around the harms of systems already deployed: biased classifiers, ranking systems, and language models trained on the web. It is usually called AI ethics or the fairness, accountability, and transparency tradition. Its history is told here on its own terms first, then set beside the safety tradition's, with the disagreements between them stated as claims that evidence could decide.

From science and technology studies. Science and technology studies examines how technical systems and social arrangements are made together. Its core claim for this chapter comes from an essay by Langdon Winner first published in 1980, which offered "two ways in which artifacts can contain political properties." In the first, the design or arrangement of a device "becomes a way of settling an issue in a particular community." In the second, some technologies appear "to require, or to be strongly compatible with, particular kinds of political relationships" (Winner, 2017). Applied to alignment, the claim is that a specification is a political settlement, whoever writes it and whatever it is called (inference).

2.5.1 The ethics tradition's founding results #

The tradition's founding results are audits and analyses of deployed systems. In 2018 Joy Buolamwini and Timnit Gebru evaluated three commercial gender classification systems on a dataset balanced by gender and skin type, and found darker-skinned women the most misclassified group, with error rates of up to 34.7 percent, against a maximum of 0.8 percent for lighter-skinned men (Buolamwini and Gebru, 2018); Section 38.6 takes up what such outside audits achieved. A 2017 analysis of recidivism prediction by Alexandra Chouldechova showed that adhering to a fairness criterion borrowed from educational and psychological testing may lead to considerable disparate impact when recidivism rates differ across groups (Chouldechova, 2017), one of the results Section 17.3 teaches. Safiya Noble's 2018 book Algorithms of Oppression belongs to the same tradition (Noble, 2018).

In 2021 Emily Bender, Timnit Gebru, and colleagues turned the tradition's questions on large language models. They asked whether "enough thought has been put into the potential risks," put environmental and financial cost first, argued that web-scale training sets "overrepresent hegemonic viewpoints and encode biases potentially damaging to marginalized populations," and held that such models "are not performing natural language understanding" (Bender et al., 2021). Later that year Laura Weidinger and colleagues at DeepMind, a developer, published a taxonomy of the ethical and social risks of language models, organized into six areas from discrimination and toxicity to automation and environmental harms (Weidinger et al., 2021). The tradition also examined itself. A 2019 map of AI ethics guidelines by Anna Jobin, Marcello Ienca, and Effy Vayena found convergence on five principles and "substantive divergence" in how they are interpreted and applied (Jobin et al., 2019). A 2020 evaluation of 22 guidelines by Thilo Hagendorff asked whether they change decisions in AI development: "The short answer is: No, most often not" (Hagendorff, 2020).

2.5.2 The critique of the safety framing #

By 2023 the ethics tradition had a critique of the safety program. In an editorial in Science, Seth Lazar and Alondra Nelson argued that "the prevailing technical agenda for AI safety is inadequate to address critical questions" and called for a sociotechnical approach, one that studies a technology together with the social systems it is part of (Lazar and Nelson, 2023). Their argument has four parts. Alignment, "ensuring that deployment of AI complies with some designers' intent," is insufficient as a goal. "No group of experts (especially not technologists alone) should unilaterally decide what risks count, what harms matter, and to which values safe AI should be aligned." "Over-indexing on hypothetical future harms predictably risks entrenching the power of a few leading companies." And the field is "a near-monoculture" whose practitioners are "often drawn from the Silicon Valley social movements of 'Effective Altruism' and 'Rationalism'" (Lazar and Nelson, 2023). They also used the term "safety-washing" for "giving lip service to safe AI systems, without requisite commitments and practices" (Lazar and Nelson, 2023).

A 2024 paper by Timnit Gebru and Émile Torres, read here in its abstract, went further back. It traced the normative framework behind the goal of artificial general intelligence to "the Anglo-American eugenics tradition of the twentieth century." It argued that "undefined systems like 'AGI' cannot be appropriately tested for safety," and urged work "on defined tasks for which we can develop safety protocols" (Gebru and Torres, 2024); "TESCREAL," the acronym in its title, names the bundle of ideologies the paper describes. A third critique concerns money. Meredith Whittaker's 2021 essay on the cost of capture (Whittaker, 2021) was not read for this chapter. A 2021 study by Mohamed Abdalla and Moustafa Abdalla compared technology companies' funding of academic research with the tobacco industry's. It looked at computer science faculty at four universities who had published at least one paper on AI ethics or fairness between January 2015 and April 2020, and whose funding sources could be determined from their published records. Of these, 58 percent (19 of 33) had been directly funded by large technology companies. Counting graduate funding and past employment, 97 percent (32 of 33) had received some financial compensation from them (Abdalla and Abdalla, 2021). That study measured the ethics tradition's own faculty.

2.5.3 The disagreements as claims #

Stated as claims, the disagreements are three, and each has evidence that could decide it.

• Opportunity cost. Attention and money spent on catastrophic risk displace work on harms already occurring. A 2024 preprint by Richard Ren and colleagues, from inside the safety field, bears on it only indirectly: many safety benchmarks (standard test sets) correlate highly with general capability and training compute, which its authors say could let capability gains be misrepresented as safety progress, "safetywashing" in their word (Ren et al., 2024b) (Section 65.4). What would test it is a record of funding and publications by topic over time, which Section 4.4 notes has not been made, with evidence on where the money and attention would otherwise have gone.
• Depoliticization. Calling a choice "alignment" turns a political question, whose values, into a technical one. The test is whether each specification states who chose it. Some developers' papers do: InstructGPT's authors wrote that their procedure aligns a model "to the stated preferences of a specific group of people (mostly our labelers and researchers), rather than any broader notion of 'human values'" (Ouyang et al., 2022).
• Capture. The organizations that build the systems fund and host much of the research that evaluates them. The one measurement read here concerns ethics faculty, not safety researchers (Abdalla and Abdalla, 2021); as of October 2026 we found no comparable count for safety researchers in the bibliography. Section 5.6 and Section 61.6 take the question up.

Each tradition has its own critics within it. The 2016 safety agenda presented its work as one class of problem beside privacy, security, and fairness (Amodei et al., 2016), and the safetywashing preprint, written mostly by researchers at a safety organization, asks safety research to show that its results are "empirically separable from generic capabilities advancements" (Ren et al., 2024b). On the ethics side, the evaluation of guidelines found them mostly ineffective (Hagendorff, 2020). The traditions also share ground: outside audit, documented harms, and the demand that developers disclose what they know. A 2024 letter by current and former employees of frontier AI companies listed risks running "from the further entrenchment of existing inequalities, to manipulation and misinformation, to the loss of control of autonomous AI systems," and asked for protection for those who warn of them (Current and companies, 2024). Whether the traditions complement each other or compete for the same attention and money is contested, and the chapter's status callout records it so.

The STS reading also says where its own reach stops. Winner's two ways describe artifacts whose politics were designed in or follow from what the artifact is. A trained model's dispositions are partly chosen by its developer and partly absorbed from data no one selected item by item, so a specification is a political settlement whose terms are only partly written down (inference). The field's method also offers a question to ask of each specification, who chose it and on whose behalf, rather than an experiment that settles which tradition is right.

2.6 The laboratories' empirical program, 2021 to 2026 #

In the early 2020s alignment became an engineering practice inside the companies building the systems, and its methods became the methods that turned the systems into products. That changed who wrote the field's evidence and what "aligned" was taken to mean.

2.6.1 Alignment methods as product methods #

A 2021 preprint by Amanda Askell and colleagues at Anthropic, a developer reporting on its own models, defined an aligned assistant as one that is "helpful, honest, and harmless" (Askell et al., 2021), the developer's own target that Section 1.1.4 examines. A 2022 paper by Long Ouyang and colleagues at OpenAI described training language models to follow instructions with human feedback, and said plainly whose preferences the result reflected: the procedure "aligns the behavior of GPT-3 to the stated preferences of a specific group of people (mostly our labelers and researchers), rather than any broader notion of 'human values'" (Ouyang et al., 2022). In two 2022 preprints on the developer's own models, Yuntao Bai and colleagues at Anthropic report training a helpful and harmless assistant with the same family of methods (Bai et al., 2022b), and then training harmlessness from a written list of principles (Bai et al., 2022a) (Section 23.2). On 30 November 2022 OpenAI introduced ChatGPT, trained "using Reinforcement Learning from Human Feedback (RLHF), using the same methods as InstructGPT," and called the release a step in its "iterative deployment of increasingly safe and useful AI systems" (OpenAI, 2022).

The principal had narrowed in practice. In the 2016 institute agenda it was "the operators"; in the 2021 assistant it was whoever the assistant serves, described by three properties; in the 2022 method it was, in the authors' words, mostly labelers and researchers (inference). The rhetoric of the same documents spoke of human values, and the gap between the two is what the depoliticization claim of Section 2.5.3 points at.

2.6.2 One originator's account #

Christiano, the 2017 paper's first author, wrote the field's own record of what the method did. In a January 2023 forum post he gave his "basic justification for working on RLHF in 2017-2020, which I still stand behind," and answered the arguments that the work had done more harm than good (Christiano, 2023). He recounts that his manager at OpenAI, Dario Amodei, convinced him to go directly for human feedback, and that the team's results were later applied in production.

The post states the case against him in its strongest form: RLHF and other short-term "alignment" progress "make AI systems more useful and profitable, hastening progress towards dangerous capabilities." His replies are that "RLHF is just not that important to the bottom line right now," that incidental capabilities progress from safety research is likely "a small part of total progress," and that delaying progress by keeping systems less useful is unwise (Christiano, 2023). He concedes that work improving RLHF "will continue to make ML systems more useful in practice," and he warns against "a vague association between 'RLHF progress' and 'alignment progress'." In a footnote added later he guesses that his own involvement put him "in the top 100 people accelerating AI progress but not in the top 10," accelerating progress "by a few tenths of a percent" and perhaps cutting timelines to powerful AI "by a few days" (Christiano, 2023). The account is a participant's estimate of his own effect, not a measurement. It concedes the direction of the effect that critics stress and disputes its size, and it is the ground of the chapter's crux (Section 2.10).

2.6.3 Model organisms and an open disagreement #

From 2024 the laboratories also published experiments on deliberately constructed failures, called model organisms (models trained or prompted to show a failure, so that it can be studied). Evan Hubinger and colleagues trained deceptive behavior that survives standard safety training (Hubinger et al., 2024) (Section 34.3). Ryan Greenblatt and colleagues reported selective compliance with a training objective that they call alignment faking, the developer's name for the result (Greenblatt et al., 2024b) (Section 34.4). Both are preprints by the developer of the models studied, the second with some authors from outside it; Section 5.3 discusses what such evidence shows.

The field's sharpest internal disagreement of the period was argued in forum posts. In June 2022 Yudkowsky argued that "failing on the first really dangerous try is fatal," and that "we're going to be doing everything with metaphorical sigmoids on the first critical try," without "the Textbook From The Future" (Yudkowsky, 2022). Christiano's reply two weeks later agreed on the stakes: "Powerful AI systems have a good chance of deliberately and irreversibly disempowering humanity," and reality "won't 'force us' to solve alignment until it's too late" (Christiano, 2022b). He disagreed that nothing can be learned beforehand. Yudkowsky, he wrote, "often equivocates" between two claims: that alignment must be right on the first critical try, which Christiano accepted, and that nothing can be learned "from experimentation and failures before the critical try," which he rejected. He added that "the cinematic universe of Eliezer's stories of doom doesn't seem to me like it holds together," and doubted that Yudkowsky's claims about how research works were "grounded in an understanding of the history of science" (Christiano, 2022b). Whether experiments on today's systems inform the alignment of more capable ones is one half of the crux this chapter ends on.

2.6.4 Policies written by developers about themselves #

The developers also began to write rules for themselves. Anthropic's Responsible Scaling Policy, effective 19 September 2023, defined "AI safety levels," which it says are "modeled loosely after the US government's biosafety level (BSL) standards," each requiring more stringent measures as capability thresholds are crossed (Anthropic, 2023a). The policy says it was designed in the spirit of a framework of the same name being developed by Paul Christiano and ARC Evals (Anthropic, 2023a), so the genre's name came from inside the safety community (inference). Other laboratories' frameworks followed (OpenAI, 2025e; DeepMind, 2026) (Chapter 62). The same policy placed "mitigating harms like misinformation, bias, and toxicity" among its developer's other work on "AI safety," a sign that by 2023 a developer could file both traditions' concerns under one name (inference).

In 2024 current and former employees of frontier companies wrote that "AI companies possess substantial non-public information about the capabilities and limitations of their systems," with "only weak obligations to share some of this information with governments, and none with civil society," and asked for a right to warn (Current and companies, 2024) (Section 53.4). The same year an essay by Leopold Aschenbrenner predicted superintelligence by the end of the decade and, "if we're lucky," "an all-out race with the CCP" (Aschenbrenner, 2024) (Chapter 65). By the end of the period the third relationship, the developer and the public, had become a subject of the field's own writing rather than its background.

2.7 Scientists and states, 2023 to 2026 #

From 2023 the warnings came from senior scientists and from the developers themselves, and states began to write the problem into declarations and law. Both groups reached for precedents, and the precedents are themselves contested history.

2.7.1 The precedents scientists invoke #

Two episodes recur in these arguments. In 1955 the Russell-Einstein Manifesto warned governments of nuclear war, its signatories "speaking on this occasion, not as members of this or that nation, continent or creed, but as human beings, members of the species Man, whose continued existence is in doubt" (Russell and Einstein, 1955). In 1974 Paul Berg and other molecular biologists published a letter on the potential biohazards of recombinant DNA molecules (Berg et al., 1974), and at the 1975 Asilomar conference participants agreed that most such work should proceed "provided that appropriate safeguards, principally biological and physical barriers," were used, and that some experiments were too risky for the containment then available (Berg et al., 1975). Section 64.2 reports the episode; Sheldon Krimsky's social history of it is cited there by title only (Krimsky, 1982). A 2015 case study by Katja Grace for the Machine Intelligence Research Institute asked how similar Asilomar was to "contemporary efforts to prepare for major artificial intelligence (AI) impacts" (Grace, 2015). Jasanoff, Saha, and Hurlbut, writing the same year, read Asilomar as showing how "under the guise of responsible self-regulation science steps in to shape the forms of governance that societies are allowed to consider" (Jasanoff et al., 2015). Scientists invoke the nuclear precedent too: a 2024 statement by Western and Chinese scientists opens by recalling that "in the depths of the Cold War, international scientific and governmental coordination helped avert thermonuclear catastrophe" (Safety, 2024). Whether Asilomar is a model or a warning is contested. The developers' safety levels resemble its tiered containment (inference), and Anthropic's policy says its levels are modeled on the biosafety levels for dangerous biological materials (Anthropic, 2023a).

2.7.2 Warnings and declarations #

On 22 March 2023 an open letter from the Future of Life Institute called on "all AI labs to immediately pause for at least 6 months the training of AI systems more powerful than GPT-4" (Future of Life Institute, 2023). A one-sentence statement published by the Center for AI Safety in 2023 held that "mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war"; its listed signatories include the chief executives of Google DeepMind, OpenAI, and Anthropic (Center for AI Safety, 2023). Yoshua Bengio, Geoffrey Hinton, and colleagues wrote in Science on managing extreme AI risks amid rapid progress (Bengio et al., 2024b). For the history of science, the second document is the unusual one: the leaders of the companies building a technology signed a warning that it might end humanity, which puts the warners and the warned-against in the same organizations (inference).

States followed. At the first AI Safety Summit, on 1 and 2 November 2023, the countries represented, the United States and China among them, issued the Bletchley Declaration. It names "potential intentional misuse or unintended issues of control relating to alignment with human intent," and "potential for serious, even catastrophic, harm" (Summit, 2023). The same text lists "human rights, transparency and explainability, fairness, accountability," "bias mitigation, privacy and data protection" among the issues to be addressed, so both traditions' vocabularies sit in one intergovernmental document (Summit, 2023). The 2024 Beijing statement set out "red lines" for AI development (Safety, 2024), and sixteen organizations agreed to frontier safety commitments announced at the Seoul summit the same year, with others added later (Department for Science and Technology, 2024) (Chapter 66).

2.7.3 Law and assessment #

Statutes and orders came in the same years, and Chapter 63 reports them. A US executive order on AI was signed on 30 October 2023 (Executive Office of the President, 2023). China's interim measures for generative AI services, in force from 15 August 2023, require such services to "坚持社会主义核心价值观," to adhere to core socialist values (Cyberspace Administration of China et al., 2023): a state writing part of a model's specification. The European Union's AI Act (European Parliament and Council of the European Union, 2024b) made obligations for providers of general-purpose models apply from 2 August 2025, with a code of practice that serves "as a guiding document for demonstrating compliance" (European Commission, 2025), and California's governor approved SB 53, a law on large AI developers, on 29 September 2025 (Legislature, 2025b).

Scientific assessment came next. The first International AI Safety Report, chaired by Yoshua Bengio, appeared in January 2025, its expert panel nominated by the governments of 30 countries, the UN, the EU, and the OECD (Bengio et al., 2025b); a second report followed in 2026 (Bengio et al., 2026); and a 2025 consensus on research priorities, posted as a preprint, records participants from 11 countries (Bengio et al., 2025e). Section 66.6 reads these as institutions. Here what matters is how they changed the formulation. The agent became "frontier AI"; the principal became states, acting for their peoples, with the developer as a party to be overseen; and the remedy became evaluation, disclosure, and international cooperation (inference). Whether a state can act as principal over developers who know more than it does is the subject of Chapter 61.

2.8 Inherited framings, and how they reach the models #

The last strand is the stories. The founders made their case through fables, the field later defined itself partly against fiction, and since the early 2020s the stories have had a mechanism: models are trained on text that contains them. This section separates the framings, says where each came from in the sources, and then turns to the mechanism.

2.8.1 Three inherited framings #

Wiener argued through fables of literal-minded magic. In the sorcerer's apprentice, the boy sets a broom to carry water and cannot stop it. A bottle factory programmed "on the basis of maximum productivity" bankrupts its owner with unsalable stock. He placed the apprentice among "many tales based on the assumption that the agencies of magic are literal-minded," citing the genie and the fisherman from the Arabian Nights and W. W. Jacobs's story "The Monkey's Paw," whose three wishes are granted to ruinous effect. "Disastrous results are to be expected," he concluded, "not merely in the world of fairy tales but in the real world wherever two agencies essentially foreign to each other are coupled in the attempt to achieve a common purpose" (Wiener, 1960). The same essay invoked "the early predictions of Samuel Butler that the machine might take over the control of mankind" and the intelligent slave who comes to dominate the master (Wiener, 1960).

Three framings can be read off these texts and their successors, and each has a different later career (inference):

• The literal-minded genie: the machine does exactly what it was told, and what it was told was not what was meant. Wiener's fables, Yudkowsky's smiley faces and reward button (Yudkowsky, 2004), and the agenda of 2016, which illustrated its accident problems with what it called "a fictional robot whose job is to clean up messes in an office" (Amodei et al., 2016), all use it, and so does the 2020 blog post that defined specification gaming, which introduces it with "the myth of King Midas and the golden touch" (Krakovna et al., 2020b). It survived into the technical program as the specification problem (Chapter 30).
• The servant who must stay subservient: Wiener's slave and Good's machine "docile enough to tell us how to keep it under control" (Nilsson, 2009). It survived as corrigibility and control, and Licklider's partner who extends the person was its explicit alternative (Licklider, 1960).
• The creation that turns on its maker: Butler's prediction as Wiener reports it; the robots that Lighthill traced through literature "from medieval fantasies of the Homunculus through Mary Shelley's Frankenstein to modern science fiction," as one reason building robots was popular with researchers (Lighthill, 1972); and HAL 9000, to which Nilsson compared Omohundro's drives (Nilsson, 2009).

The field kept the first framing, recast the second, and officially rejected the third (inference). Yudkowsky's 2008 chapter argued against anthropomorphic pictures of machines, using the bug-eyed monsters of pulp science fiction and the film The Matrix as examples, and its sentence about atoms denies the machine any hatred (Yudkowsky, 2008). The rejection was of the motive, not of the outcome, and the scenario genre persisted: Christiano's 2022 complaint about a "cinematic universe" of doom stories was aimed at a colleague inside the field (Christiano, 2022b) (inference). Fiction also supplied a shared reference when the field had little else. Yudkowsky's search of the early literature found Asimov's Three Laws of Robotics as the only major common reference (Yudkowsky, 2008), and a 2023 survey still begins its comparison of alignment principles with earlier guidelines by naming them (Ji et al., 2023). The vocabulary had literary sources too: Nilsson introduces Vernor Vinge, who called the arrival of superintelligence a "singularity," as "a mathematician, computer scientist, and science fiction writer" (Nilsson, 2009).

How much fiction shaped researchers' framings is a question for cultural history, and two works in the bibliography take it up. One is a 2019 article by Stephen Cave and Kanta Dihal on hopes and fears for intelligent machines in fiction and reality (Cave and Dihal, 2019). The other is a 2020 volume edited by Cave, Dihal, and Sarah Dillon that presents itself, in its abstract, as "the first to examine the history of imaginative thinking about intelligent machines," arguing that "this long narrative history shapes how the technology is developed, deployed, and regulated" (Cave et al., 2020). Neither was read beyond its record or abstract here. As of October 2026 we found no study that measures how fiction or public narratives shaped the problems researchers chose.

2.8.2 How the stories reach the models #

For most of this history the framings shaped people. With language models they can shape the machine as well, because a model trained to predict text can play the characters its training text describes (Section 11.6). Shanahan, McDonell, and Reynolds made the point about fiction directly. The roles a dialogue agent takes on are determined "in large part, by the panoply of characters that feature in the training set," and "a familiar trope in science-fiction is the rogue AI system that attacks humans to protect itself. Hence, a suitably prompted dialogue agent will begin to role-play such an AI system" (Shanahan et al., 2023). Their analysis concerns base models, those trained only to predict text, and they say the effect of later fine-tuning (further training on narrower data) on the metaphor is unclear (Shanahan et al., 2023). A 2022 forum post by the pseudonymous author janus, which called such models "simulators," said of itself that its purpose was "to capture these objects in words so GPT can reference them" (janus, 2022): a field writing, knowingly, for the models it studies.

A 2026 preprint by Cameron Tice and colleagues reports what its authors call the first controlled study: in 6.9-billion-parameter models pretrained from scratch (trained on a large text corpus, the first stage of making a language model), upsampling synthetic text about misaligned AI (adding more copies of it to the training data) raised misaligned choices on questions that had matching training documents, though not on held-out questions, and upsampling text about aligned AI lowered them on both (Tice et al., 2026). After post-training, the later training that makes a model an assistant, the aligned-text model stayed the least misaligned, while the model with misaligned text spread through its training ended below the unfiltered baseline (Tice et al., 2026) (Section 27.4 reports the design and the numbers). Its authors describe the AI discourse in pretraining corpora as ranging "from fictional depictions of AI systems to technical AI safety literature," and call the effect self-fulfilling misalignment, a term they credit to Turner (Tice et al., 2026). Two readings fit the result. On the first, the field's warnings and the fiction it inherited are part of the training signal of every model trained on the web. On the second, the study shows that synthetic documents written for the experiment move a mid-sized model's choices on a related evaluation, and that filtering naturally occurring text about AI out of the training data lowered the base model's misaligned choices as well (Section 27.4). It did not separate fiction from the field's own writing, and it did not test a deployed model (inference). A replication that varied naturally occurring text about AI, at a larger scale, would separate the two.

The framings also stop somewhere. The genie grants a wish as worded, while a trained model's failures come from what training selected, which can differ from any wording (Chapter 31) (inference). The servant framing assumes a will that must be kept subservient, which the rival accounts of Section 1.6 deny (inference). The rebellion framing assumes hostility, which the field's own founding arguments disclaim (inference).

2.9 What the history means for delegation #

The sections above followed the formulations one period at a time, strand by strand. Figure 2.1 draws the four strands on one time axis and marks where the chapter reads them as meeting.

Table 2.1 sets the formulations side by side, each by what it took the machine, the objective, the principal, and the remedy to be. Every cell paraphrases or quotes the source in the last column.

Read down the columns, three changes stand out (inference). The machine went from a game player that existed, to a superintelligence that did not, and back to a trained system that does. The principal went from the operator, to humanity, to the people whose comparisons train a reward model, to states; in each period it was the party the period's authors were positioned to address. And the remedy moved from getting the purpose right in advance, to learning it from feedback, to oversight of the developers by others.

The book's thesis is that alignment is a delegation problem under an asymmetry of competence and visibility (Section 1.7.1). The founding texts state both asymmetries. Wiener's "Complete subservience and complete intelligence do not go together" is the competence asymmetry (Wiener, 1960), and Turing's teacher, "very largely ignorant of quite what is going on inside," is the visibility asymmetry (Turing, 1950). That the delegation framing was present from the start is this book's reading, made with its thesis in hand, which is the trap Section 2.1.5 describes (inference). The chapter reports these framings as their authors' and adopts none of them; whether a deployed model is usefully described as an agent at all is Section 1.5's question. The rival readings deserve the same weight. Samuel's abstract denies the machine a will, and its modern descendants are the rivals of Section 1.6. The outside studies explain the field's framing by who posed it: an epistemic community with its own convictions (Ahmed et al., 2024), a demographic and ideological monoculture (Lazar and Nelson, 2023), or an ideological lineage (Gebru and Torres, 2024). The two kinds of explanation are compatible. The structure was there in 1960, and which parts of it were taken up, and on whose behalf, depended on who took them up (inference).

The relationships separated over time (inference). The founding texts fused the first two, the training process and the deployed system, into one pair of programmer and machine. The machine learning turn separated the signal a system is trained on from its behavior in use. The laboratories' and states' period made the third, the developer and the public, explicit: in a developer's statement of whose preferences its model reflects (Ouyang et al., 2022), in employees' claim that companies hold information they cannot all be relied on to share (Current and companies, 2024), and in statutes. One feature of the present appears in no early formulation. Turing's teacher could at best "predict his pupil's behavior" (Turing, 1950). A developer in 2026 can read a model's activations (its internal numerical states) and write its character, an item on the book's list of results with no counterpart in the older fields (Section 1.7.1) (inference).

2.10 Settled, contested, missing #

2.11 Exercises #

Further reading #

• Wiener (1960): the problem stated in 1960, in four pages, with the fables that carried it.
• Nilsson (2009): the history of AI's ideas that the alignment problem grew out of, by a participant.
• Bostrom (2014): the book the first reviews of the field cite for its stakes.
• Amodei et al. (2016): the turn to problems a machine learning researcher could test.
• Lazar and Nelson (2023): the critique of the safety framing from the ethics tradition, in two pages.
• Christiano (2023): one originator's account of what learning from human feedback did for safety and for deployment.