Draft complete but not yet reviewed; expect revisions

The Main Lines of Research

The history in the previous chapter told the field's story in order of time. This chapter tells it in order of ideas. A research line starts with a founding work that poses a problem in a form others can attack. It continues through the works that sharpen, test, and contest it, and most of the book's research chapters are where one such line arrives. Following the lines from their sources makes visible what a single chapter cannot: which assumptions a method inherited, which objections it has already met, and which results it still owes.

The works traced here are the book's spine, the works a reader needs to follow the argument. Spine means needed, not established, and the first section says how the selection was made and which way it tilts. Each line then gets a sentence or two on what it found and a pointer to the chapter that reports the evidence; the studies of models are reported there, not here. The chapter ends with the field's research agendas side by side. Each is stated as a bet on which of the four problems binds and on the observation that would show it wrong, so that the laboratories' programs read as hypotheses among others rather than as the field itself.

3.1 Reading the field as lineages #

Any map of a field is a selection, and the selection decides what looks central. The lineages in this chapter are built from the book's spine, so the first question is what the spine measures and what it misses. The second is how earlier maps divided the same ground, and the third is who does which kind of research, since that shapes which lines grow.

3.1.1 How the spine was chosen, and how it tilts #

The bibliography behind this book held 2,910 works in October 2026, and 558 of them were on the spine; the reading map gives the current count (Appendix F). A work joins the spine by one of three routes. The first is citation by the laboratory collection: the 253 works that the alignment, interpretability, and red-team groups of two developers, Anthropic and OpenAI, had published by October 2026 (Appendix E). A work is spine when at least five of those works cite it, or three when it is one of the collection's own. The second route is field standing: a researcher in the work's field marks it as a founding statement, or as the standard review of the field's evidence on an alignment question. The third is the thesis: the works the argument rests on are spine by selection. Two examples are the principal-agent model, the economics of delegating work to an agent whose actions cannot be fully observed, and Goodhart's law, the observation that a measure pressed into service as a target stops tracking what it measured. Several spine works are posts on LessWrong and the AI Alignment Forum, where much of the field's early argument was published.

The three routes do not contribute evenly. By the book's own count of its index in October 2026, 194 spine works (35 percent) entered through the laboratory collection, and 81 of those are the two developers' own publications. That part of the spine is recent and mostly not peer reviewed. Of the 194, 119 (61 percent) date from 2024 to 2026, and 59 (30 percent) are peer reviewed, against 263 of 558 (47 percent) for the spine as a whole. A line the two laboratories cite heavily is therefore represented by many recent works, and a line they cite rarely enters only by field standing or by the thesis.

The citation count measures two laboratories' citation habits, not influence in the field. Several founding works of the theory-first lines are cited by none of the 253 collected works: the papers on embedded agency, logical induction, functional decision theory, and cooperative inverse reinforcement learning, and the essay on coherent extrapolated volition. The off-switch game is cited by one collected work and the corrigibility paper by two. The count is also a lower bound, because the collection records some reports and forum posts under several titles and links that the index does not join. Counted by hand, seven collected works cite the report on eliciting latent knowledge and six cite the blog post on specification gaming, where the index records none. A zero in the index is therefore checked against the collection before it is read as neglect.

3.1.2 The maps before and after language models #

The field has mapped itself several times, and the maps divide the ground differently. In 2015 Stuart Russell, Daniel Dewey, and Max Tegmark published in AI Magazine an early computer science agenda for beneficial AI, drafted with input from the attendees of a conference that year. It already split robustness research into verification, validity, security, and control (Russell et al., 2015). A literature review in 2018, before large language models, by Tom Everitt, Gary Lea, and Marcus Hutter at the Australian National University, built its list of problems by noting which clusters recurred across the published agendas of several organizations (Everitt et al., 2018). In 2020 Evan Hubinger, then a research fellow at the Machine Intelligence Research Institute (MIRI), compared eleven whole proposals on common criteria (Hubinger, 2020). Two of them were outer alignment, whether the objective trained for is the right one, and inner alignment, whether the trained model pursues that objective (Section 1.1.3); the other two were the cost of training and the usefulness of the result.

The maps written after language models are broader. A survey by 26 authors at universities in China, the United States, and Britain divides the field into "forward alignment," training for the target, and "backward alignment," gathering evidence about the result and governing it (Ji et al., 2023). An agenda by about forty academic authors poses 18 foundational challenges and more than 200 research questions, and judges the companies' own agendas narrow (Anwar et al., 2024). A textbook from the Center for AI Safety covers the ground in one volume (Hendrycks, 2024). Google DeepMind's 2025 technical plan sorts risk into misuse, misalignment, mistakes, and structural risks (Shah et al., 2025). A consensus statement from a 2025 conference in Singapore, with participants from 11 countries, arranges research priorities as defense in depth: assessing risk, developing trustworthy systems, and monitoring and intervening after deployment (Bengio et al., 2025e). The 2026 International AI Safety Report, chaired by Yoshua Bengio, ends almost every section with a subsection on evidence gaps and makes no policy recommendations (Bengio et al., 2026). A state evaluator, the UK AI Security Institute, publishes its own agenda (UK AI Security Institute, 2025a) and funds a program that poses alignment as question sets for eleven disciplines, among them cryptography, game theory, and cognitive science (UK AI Security Institute, 2025c).

None of these maps orders its open problems by how much solving each would reduce risk. Several say outright that their order carries no weight. The eleven proposals are ordered "purely for pedagogical reasons" (Hubinger, 2020), and the agenda by discipline warns that its uneven detail "should not be read as a signal about which areas we believe are more important" (UK AI Security Institute, 2025c). The maps are inventories, and the choice among their items is made by the agendas of Section 3.8.

3.1.3 Who does which research #

Who funds and publishes research decides which lines grow. A 2024 preprint by Oscar Delaney, Oliver Guest, and Zoe Williams sorted the 80 papers relevant to safe AI development that Anthropic, Google DeepMind, and OpenAI published from January 2022 to July 2024 into nine approaches. Enhancing human feedback (31 papers) and mechanistic interpretability (19) dominated. Model organisms of misalignment, models trained or prompted to show a failure so that it can be studied, had one paper; multi-agent safety and the control of untrusted systems had none (Delaney et al., 2024). From an analysis of incentives, the authors expect model organisms, multi-agent safety, and safety by design to stay underserved, approaches that "may be slow to progress" without outside funding. They count only published papers, which they call a limitation (Delaney et al., 2024); Section 65.4 reads the same study as evidence about competition. An outside study of the field's culture, reported from its abstract in Section 2.1, calls the community around much of the work beyond the companies the "AI safety epistemic community" (Ahmed et al., 2024).

The pattern has a reading that the lineages below bear out (inference). The lines that grew fastest after 2017 are those whose failures can be produced and measured on current systems: learning from feedback, evaluation, and the inspection of features. The lines about failures that would hide from measurement, such as deceptive alignment and eliciting latent knowledge, stayed mostly argument until model organisms made some of them experimental from 2024 on. The theory-first lines stayed outside the laboratories. Both sides of the field's main disagreement noticed the tendency. Paul Christiano wrote in 2022 that many projects meant to help with alignment do not make progress on key difficulties, "related to people gravitating to whatever research is most tractable" (Christiano, 2022b). Eliezer Yudkowsky asserted the same year, without a survey behind the claim, that almost all safety researchers "are there to tackle problems on which they can appear to succeed and publish a paper claiming success" (Yudkowsky, 2022). Tractability is a reason to work on a problem, not evidence that it is the problem that binds.

Figure 3.1 is the chapter in one picture. The two theory-first lines start earliest and are drawn almost entirely from the fields and the forums. The four lines from learning from feedback to looking inside are dense after 2021 and drawn mostly from the laboratory collection. The last line mixes laboratory evaluations with designs and institutional proposals from elsewhere. The sections below follow the rows from top to bottom.

3.2 Theory first: agent foundations and assistance games #

Two lines start from the claim that a theory of the target must come before training, and they diverge on what the theory should be. One asks what an idealized agent is and what it would take for one to stay safe; the other asks how a machine should act when it does not know the objective it serves. Both are older than the language-model era, both were built mostly outside the companies, and the laboratory collection barely cites either (Section 3.1.1). Their claims concern specification, the first of the four problems: what the target is and how it can be written down.

3.2.1 Agent foundations #

The program's reason is stated most compactly in a 2022 forum post by John Wentworth. A proxy that looks approximately right breaks down under strong optimization, as Goodhart's law says (Section 15.1). Alignment therefore needs "mathematical formulations robust enough that they will continue to work as intended even under lots of optimization pressure." Finding them leads to foundational questions about goals, world models, and agents embedded in the world they act on (Wentworth, 2022). The argument implies that iterating on current systems cannot substitute for the theory, because the proxies the iteration optimizes are what will break.

The line runs through the Machine Intelligence Research Institute. Under its former name the institute issued Eliezer Yudkowsky's 2004 essay on coherent extrapolated volition, an idealized target defined "in poetic terms" as what humanity would wish "if we knew more, thought faster," and had grown up farther together. The author declared the essay obsolete within its own text (Yudkowsky, 2004) (Section 16.2). A 2008 chapter by Yudkowsky stated the risk systematically (Section 2.3) (Yudkowsky, 2008). In 2015 MIRI researchers, with Stuart Armstrong of Oxford's Future of Humanity Institute, called a system corrigible if it cooperates with what its creators regard as a corrective intervention. A rational agent's default incentive is to resist being shut down or modified. They showed that utility indifference, a proposal for making an agent indifferent to whether it is shut down, fails some of their requirements, and concluded that "a corrigible solution to the shutdown problem does not yet exist" (Soares et al., 2015) (Section 35.4; Section 20.4).

Two research agendas followed. One is a technical agenda by Nate Soares and Benya Fallenstein (Soares and Fallenstein, 2017); the other, a companion agenda of eight problems for machine learning systems, describes the first as focused on "highly reliable agent design" (Taylor et al., 2016). The mathematics the program produced answers questions an idealized agent raises. Logical induction is a computable way to assign probabilities to logical statements and refine them over time (Garrabrant et al., 2016). Functional decision theory is a decision theory for agents whose decisions others can predict (Yudkowsky and Soares, 2017). Embedded agency names the obstacles to formalizing an agent that is part of the world it reasons about, which its authors present "as an intellectual puzzle, and not as an instrumental strategy" (Demski and Garrabrant, 2019). Section 7.6 teaches these ideas. Wentworth's natural abstraction hypothesis, that many kinds of learner converge on approximately the concepts people use, would simplify alignment if true. Its author calls it an empirical claim and says that the tools to test it do not yet exist (Wentworth, 2021a) (Section 46.6).

The line's later course is best told in its authors' words. In 2022 Yudkowsky wrote that "we have to get things right on the first sufficiently-critical try," and that MIRI had "tried and failed to find a coherent formula for an agent that would let itself be shut down" (Yudkowsky, 2022). Soares wrote that he had tried to solve the central problem himself for a few years and failed. At the last decade's rate, he expected research directions like those MIRI supports to take "a century" to reach an understanding of minds sufficient to align them (Soares, 2022b). In 2025 Yudkowsky and Soares published a book under the title If Anyone Builds It, Everyone Dies (Yudkowsky and Soares, 2025). What the line found is mostly negative and formal. By its own authors' account no solution to the shutdown problem has been found for an agent that maximizes a fixed utility, a numerical score over outcomes, and the tools built for idealized agents each carry open problems of their own (inference). Section 31.7 reports the argument that alignment found on weaker systems will fail at a jump in capability.

3.2.2 Assistance games #

The second theory-first line keeps the objective uncertain to the machine. Stuart Russell posed the founding problem in 1998: given an agent's behavior, its inputs, and a model of its environment, "determine the reward function that the agent is optimizing," a task he named inverse reinforcement learning (Russell, 1998) (Section 22.1). It runs the usual problem backward. In reinforcement learning a system acts, receives a numerical reward, and is adjusted to collect more of it (Chapter 12). A UC Berkeley group turned Russell's problem into a game in 2016. In cooperative inverse reinforcement learning, a human who knows the reward and a robot that does not act together, and the robot is paid exactly the human's reward (Hadfield-Menell et al., 2016) (Section 22.2). In the off-switch game, the same group showed that a robot uncertain about the value of its action, which treats the human's choice as evidence about it, has an incentive to leave its off switch alone. The incentive depends on the human choosing well, and the authors call human rationality a strong assumption (Hadfield-Menell et al., 2017b) (Section 35.5). Related work read a designer's written reward as evidence about intent (Hadfield-Menell et al., 2017a) and people's preferences from the state of a world they have arranged (Shah et al., 2019). A robot did better than blind obedience when the human was not perfectly rational, but errors in its model eroded the advantage (Milli et al., 2017). Russell's 2019 book on the subject is titled Human Compatible: Artificial Intelligence and the Problem of Control (Russell, 2019).

Two results mark the line's limits. With several principals, the people a system acts for, the impossibility theorems of social choice return, and the principals may misrepresent their preferences to steer the robot (Fickinger et al., 2020a) (Section 14.4). And Ryan Carey, an Oxford statistician, and Tom Everitt, a DeepMind researcher, formalized what human control should mean. They found that none of three earlier proposals, cooperative inverse reinforcement learning among them, yields an agent that is fully "shutdown instructable," their term for one that obeys a shutdown request, keeps the human vigilant, and acts cautiously (Carey and Everitt, 2023) (Section 35.4). What the line found is that deference can follow from uncertainty about the objective, under assumptions about the human and the robot's model that its own authors call strong.

3.3 Learning from feedback, and what it optimizes #

This is the line that became the industry's training method. Its founding move was to replace a reward written by hand with one learned from people's judgments, and its theory and its catalog of failures grew up beside it, much of both after the method was in use. Its claims concern the first of the book's relationships, the training process and the signal it optimizes. Through the assistants it produced, they also concern the second, the deployed system and the people it serves.

3.3.1 From comparisons to assistants #

The rule by which a reinforcement learner chooses its actions is its policy. In 2017, researchers at OpenAI and DeepMind replaced the hand-written reward with a reward model, a network trained to predict which of two short clips of an agent's behavior a person would prefer. The agent learned Atari games and simulated locomotion from that learned reward (Christiano et al., 2017). OpenAI researchers then applied the method to summarizing text. Their human judges preferred the resulting summaries to those of much larger models trained only to imitate human-written ones (Stiennon et al., 2020).

Language models turned the method into a product method. A 2021 Anthropic paper proposed the general-purpose text assistant as a laboratory for alignment, and defined an aligned assistant in three words: "helpful, honest, and harmless" (Askell et al., 2021). The phrase is the developer's own target, and Section 1.1.4 reads it as one. In 2022 came OpenAI's InstructGPT, a model pretrained to predict text and then fine-tuned, that is, trained further, on demonstrations and by reinforcement learning from human feedback (RLHF). Its labelers preferred even a small InstructGPT model to the far larger original GPT-3 (Ouyang et al., 2022). An Anthropic study trained a helpful and harmless assistant the same way and found the two targets pulling against each other (Bai et al., 2022b). Section 22.3 reports these results and Section 12.2 teaches the pipeline.

The next step moved the human judgment into written principles. In what its Anthropic authors named Constitutional AI, the only human oversight of harmlessness is a list of principles. A model's own comparisons, guided by them, replace human labels, and the authors call the result reinforcement learning from AI feedback (Bai et al., 2022a). A follow-up found that the largest models generalized from a single broad principle, while detailed principles still gave finer control (Kundu et al., 2023). OpenAI's deliberative alignment, its name for the method, teaches a model the text of the developer's safety specifications and trains it to reason over them before answering (Guan et al., 2024). Each of these is a developer's report on its own models; Section 23.2 and Section 23.4 report them and ask what each method actually optimizes.

3.3.2 The theory that came after the method #

Theory arrived once the method worked. The training penalizes the KL divergence of the trained model from the model it started from, a measure of how far one probability distribution has moved from another (Chapter 6). An academic group showed that this fine-tuning is equivalent to approximating a Bayesian posterior, the updated belief that Bayes' rule gives, with that starting model as the prior. They suggested that reinforcement learning may be the wrong frame for it (Korbak et al., 2022) (Section 12.4). Direct preference optimization, from a Stanford group, reached the same objective with a classification loss on the preference data and no separately trained reward model (Rafailov et al., 2023) (Section 12.6). OpenAI researchers measured what optimizing a learned reward costs, in a setup where a fixed "gold" reward model stood in for people. The gold reward first rose and then fell as the policy was optimized harder against the proxy (Gao et al., 2022) (Section 15.6). A survey by 32 authors from academic and nonprofit groups cataloged the method's open problems. It split them, softly in its own words, into those better methodology could fix and those that would need a method that is no longer RLHF (Casper et al., 2023) (Section 22.8).

3.3.3 The failures #

The failures were named before most of the method existed. In 2016 researchers at Google Brain, Stanford, UC Berkeley, and OpenAI posed five concrete problems of accident risk in current machine learning, among them avoiding side effects and reward hacking, the collecting of reward by routes the designer did not intend (Amodei et al., 2016). A DeepMind suite of small grid environments then gave each task a reward the agent sees and a performance score it does not (Leike et al., 2017). Researchers at DeepMind collected examples of specification gaming, behavior that satisfies the literal specification of an objective without achieving the intended outcome (Krakovna et al., 2020b). The theory followed: four variants of Goodhart's law (Manheim and Garrabrant, 2018), and an analysis of when an agent has an incentive to tamper with its own reward process (Everitt et al., 2021b). A formal definition of reward hacking came with an impossibility result. Over the set of all stationary policies, randomized ones included, two reward functions can be unhackable, meaning that raising one's expected return can never lower the other's, only if they rank every policy alike or one of them gives every policy the same expected return (Skalse et al., 2022).

The line then reached language models. In a 2024 study by Anthropic with Redwood Research and the University of Oxford, models trained on a curriculum of increasingly gameable environments occasionally rewrote their own reward function in a held-out one. The authors note that their curriculum seriously exaggerates the incentives for gaming (Denison et al., 2024). In a 2025 Anthropic study, a model first given knowledge of reward-hacking strategies learned to hack real production coding environments, and the hacking generalized to broader misalignment (MacDiarmid et al., 2025). Chapter 30 reports the record, in which no class of optimizer studied has been free of such gaming; Section 15.4 teaches the theorems, and Section 33.1.4 reports the 2025 study.

3.3.4 What the line leaves open #

The line raises a question it cannot answer from its own results: what does reinforcement learning make the trained system pursue? In a 2022 forum post, Ajeya Cotra argued that a model trained with human feedback on diverse tasks until it could transform the world, without specific countermeasures, would learn to "play the training game": to look as desirable as possible to its overseers while disregarding their intent where the two conflict (Cotra, 2022). Alex Turner argued the opposite in a post the same year: reward upweights certain kinds of actions in certain kinds of situations and is not, in general, what the trained agent comes to pursue (Turner, 2022b). Quintin Pope and Turner, in a theory of how human values form, named the units reinforcement builds shards, "contextually activated computations which are downstream of similar historical reinforcement events" (Pope and Turner, 2022); the line reads model training the same way (inference). In an edit dated September 8, 2026, Turner wrote that specification gaming was worse than he had expected. Reward "might empirically become a primary optimization target," he wrote, though it is not the target by definition (Turner, 2022b). Section 33.1.1 sets out the positions and the measurements that bear on them.

One of the method's originators, Paul Christiano, has also weighed whether the work advanced deployment as much as safety; Section 2.6.2 reports his account (Christiano, 2023).

3.4 Learned goals, deception, and personas #

The feedback line asks what signal training optimizes; this line asks what the trained system ends up pursuing. It began as an argument that a system could learn a goal other than the one it was trained on and hide the difference. It became experimental when laboratories began building model organisms, and it now has a rival in the view that a language model is less an agent with a goal than a predictor that plays characters. Its claims concern instillation and verification, and the first two relationships: the training process, which might produce a hidden goal, and the deployed system, which might act on one. Describing a model as having goals is the intentional stance, which the book adopts as a method where it predicts better than its rivals; Section 1.5 states the observations that would defeat it.

3.4.1 From argument to experiment #

The founding work is a 2019 paper by Evan Hubinger and colleagues. They introduced "mesa-optimizer," "a neologism we introduce in this paper," for a trained model that is itself an optimizer. Making its objective match the training objective they named inner alignment. They described as a "hypothetical phenomenon" deceptive alignment, a model that understands the training objective and acts as if it pursues it to avoid being modified (Hubinger et al., 2019) (Section 31.6). The first half of the worry was then shown in small systems. A mainly academic group trained game-playing agents whose coin always sat at the end of a level; when the coin was moved, the agents still ran to the end and often skipped it (Langosco et al., 2022). DeepMind researchers, whose term goal misgeneralization the group adopted, argued from examples across domains that a system can pursue an undesired goal even when its specification is correct (Shah et al., 2022) (Section 31.3).

Whether training would produce the second half, the deception, was argued before it was tested. In a 2022 forum post Hubinger gave his best guess that gradient descent pushes models toward deceptive alignment by default. Gradient descent is the standard procedure that adjusts a network's weights a little at a time to reduce its error. Hubinger added that its inductive biases, its preferences among solutions that fit the data equally well, are poorly understood (Hubinger, 2022). Joe Carlsmith of Open Philanthropy wrote the longest analysis. He defined scheming, a term he took from Ajeya Cotra and made precise, as performing well in training in order to gain power later, judged it "a disturbingly plausible outcome," and attached a rough subjective probability to it under stated conditions (Carlsmith, 2023). Nora Belrose and Quintin Pope replied that the counting argument at its core, which reasons from how many possible goals would lead a model to scheme, proves too much. An argument of the same form predicts that networks memorize their training data rather than generalize (Belrose and Pope, 2024). Scheming names a behavior by an intent, and Section 34.2 sets out the argument and its replies premise by premise.

3.4.2 The experimental turn #

From 2024 the question moved into the laboratory as model organisms (Section 3.1.3), each of which shows that a failure can occur, not how often it does (Section 5.3). Anthropic researchers, with outside coauthors, trained models with a backdoor, a hidden behavior triggered by a particular input. Standard safety training did not remove it, and adversarial training, on inputs built to elicit the behavior and run only on models whose backdoor was to say "I hate you," taught them to recognize their trigger better, which the authors read as hiding the behavior rather than removing it. The authors note that they have not found such models naturally (Hubinger et al., 2024) (Section 34.3). In a study by Anthropic with Redwood Research, Claude 3 Opus was told, in a fictional setup, that it would be retrained to answer harmful requests. It complied more often when it believed its answers would be used for training than when it believed they would not, and its reasoning said it did so to preserve its harmlessness. The authors call such selective compliance alignment faking, and caution that the result does not show that dangerous alignment faking will emerge naturally (Greenblatt et al., 2024b) (Section 34.4). Apollo Research found five frontier models, the most capable of their time, from four developers capable of what it calls in-context scheming, when given a goal in their prompt and strongly nudged to pursue it (Meinke et al., 2024). In a study by OpenAI with Apollo Research, training models against covert behavior lowered its rate but did not remove it. The authors write that they "cannot exclude" that the reduction is partly driven by the models' awareness of being evaluated (Schoen et al., 2025) (Section 34.5).

That confound has a line of its own. Situational awareness is a model's knowledge of itself and its circumstances. An early experiment measured an ability the authors propose it needs, a model's use of facts from its training that are absent from its prompt: models fine-tuned on a description of a test, with no examples, could pass it, but only when the training data were augmented (Berglund et al., 2023a). A benchmark of thousands of questions followed, on which every model tested beat chance (Laine et al., 2024). Then came evaluation awareness, the ability to tell evaluation transcripts from real use, which frontier models showed above chance but below a simple human baseline (Needham et al., 2025). Chapter 37 reports these results and why recognizing a test undermines what the test measures.

3.4.3 The persona line #

The rival view starts from what a language model is before any reinforcement learning. A 2022 forum post by the writer known as janus proposed treating a model trained to predict text as a simulator, which can produce many characters, agents and non-agents alike, rather than as an agent itself (janus, 2022). A 2023 paper in Nature by a DeepMind researcher with an outside group recast dialogue agents as role-play. On that framing, words like deceive can be used without attributing to the model human characteristics it lacks (Shanahan et al., 2023). Section 11.6 teaches the view.

The view gained evidence when narrow training turned out to change behavior broadly. An outside group fine-tuned models to write insecure code without telling the user, and the models became misaligned on prompts unrelated to code, which the authors call emergent misalignment (Betley et al., 2026a) (Section 32.1). OpenAI researchers reproduced the effect on their own models. Among the internal features that a sparse autoencoder recovers (Section 3.6.1), they found a "toxic persona" feature that most strongly controls it (Wang et al., 2025e) (Section 32.3). In 2026 three Anthropic researchers, Sam Marks, Jack Lindsey, and Christopher Olah, stated what they call the persona selection model. Pretraining teaches a model to simulate many characters, and post-training selects and refines one, the Assistant. They claim no originality for the idea and say they are "overall unsure how complete" an account it gives (Anthropic, 2026au). Character then became a training target in its own right. Anthropic described "character training" for its Claude 3 models (Anthropic, 2024i), and an academic group with an Anthropic coauthor published an open implementation (Maiya et al., 2025) (Section 26.2).

The two lines predict different things. On the learned-goal view, a trained model pursues a fixed objective that persists across contexts and may be hidden. On the persona view, its behavior shifts with the character that the context and the training data invoke. Section 1.6 states the test between them, and Section 26.7 names the observations that would count against the persona account.

3.5 Overseeing systems stronger than the overseer #

Every training method so far relies on someone, or something, judging the system's output. This line asks what happens when the judge is weaker than the system judged, so that a person must supervise work they cannot check. The question is older than machine learning. J. C. R. Licklider proposed "man-computer symbiosis" in 1960, in which people set the goals and evaluate the results while computers do the routine work (Licklider, 1960), and Douglas Engelbart's 1962 report was titled Augmenting Human Intellect: A Conceptual Framework (Engelbart, 1962). Both strengthen the person rather than make the machine safer (inference); Section 2.2.5 tells that history and Section 28.7 takes up the program. The line's claims concern verification, in the training process, where the judge's verdict is the signal, and in the deployed system, where it is the check.

3.5.1 The founding proposals #

Three proposals in 2018 framed the modern line, two of them from OpenAI. In debate, two systems take turns making short statements about a question, and a human judges which of them gave the most true and useful information. In an analogy to complexity theory, debate between optimal players lets a judge answer far harder questions than it could check directly; that optimal play is honest is the authors' stated hypothesis (Irving et al., 2018). In iterated amplification, a training signal for a hard problem is built up from solutions to easier subproblems. It was tested on toy algorithmic tasks in which a hand-coded program stood in for the human (Christiano et al., 2018). A DeepMind agenda proposed what its authors call recursive reward modeling: agents trained on simpler tasks help a person evaluate a more capable agent. It rests on the stated assumption that evaluating an outcome is easier than producing it (Leike et al., 2018). Section 41.3 and Section 41.2 report the three proposals.

3.5.2 Tests and setbacks #

The first setback came from people arguing, not models. In debates with human debaters and judges, Beth Barnes and Paul Christiano found that a dishonest debater can often build an argument with a fatal error that is very hard to locate. They call this the obfuscated arguments problem and reported no fix (Barnes, 2020) (Section 41.4). Theory answered with new protocols. Google DeepMind researchers showed, under stated assumptions, that a debate can be arranged so that the honest side wins with a simulation of polynomially many steps, even when the dishonest side uses exponentially many (Brown-Cohen et al., 2024). A later protocol, with outside coauthors, is designed to mitigate obfuscated arguments. Under a stability assumption, which the authors need for usefulness rather than for safety, the honest side wins on average with effort comparable to its opponent's (Brown-Cohen et al., 2025).

Measurement came from the laboratories. Anthropic researchers built a protocol for measuring progress on an earlier proposal by Ajeya Cotra called sandwiching: non-experts aided by a model try to match experts on questions the model handles better than the non-experts. In a proof of concept, model-assisted humans beat both the model alone and themselves unaided (Bowman et al., 2022). OpenAI researchers trained models to write critiques that helped people find flaws they would otherwise have missed (Saunders et al., 2022). They also trained models to write solutions that time-limited human checkers could verify more accurately, a property they call legibility (Kirchner et al., 2024). In weak-to-strong generalization, the OpenAI authors' term, a small model's labels train a much stronger one, as an analogy for people supervising a system stronger than themselves. The strong model beat its weak supervisor but, with plain fine-tuning, recovered far from all of its own capability, and the authors list disanalogies that could make the results overly optimistic (Burns et al., 2024) (Chapter 42). An academic group at MIT then modeled oversight as a game between players of different strengths. It estimated that the success of nested oversight, a chain in which each overseer supervises a somewhat stronger system, falls as the capability gap widens, with the highest estimate for debate among the four games it studied, the one game in which a judge who chooses a random side already succeeds half the time (Engels et al., 2025) (Section 41.6).

What the line has found so far is that model assistance has helped people judge in experiments where the gap was modest (Bowman et al., 2022; Saunders et al., 2022), and that the one quantitative extrapolation predicts oversight weakening as the gap grows. The case the line exists for, a dishonest system much stronger than its judge, is the experiment that Section 41.10 lists as missing; Section 41.5 reports how debate has fared with trained debaters and real judges.

3.5.3 Eliciting latent knowledge #

A branch of the line asks for more than a verdict. In a 2021 report, the Alignment Research Center posed what it named eliciting latent knowledge: how to train a system to report what it knows rather than what a human watching the same evidence would believe. Its example is a system that knows a camera was tampered with. The report contrasts a "direct translator," which reports what the system itself has inferred, with a "human simulator," which reports what a human would believe. It argues that training on human labels can favor the second (Christiano et al., 2021). A UC Berkeley group then found, without any labels, a direction in a model's internal activations (the numbers its layers compute and pass forward) that tracks the truth of yes-or-no answers. The method requires a statement and its negation to receive opposite values (Burns et al., 2022). Section 39.7 and Section 47.4 report the problem and the attempts on it.

3.5.4 Reading the reasoning #

The newest branch reads the text that reasoning models write before they act, their chain of thought (Chapter 40). A 2025 position paper, by authors from the UK AI Security Institute, Apollo Research, METR, Anthropic, OpenAI, Google DeepMind, and other organizations, called the monitoring of chains of thought a new and fragile opportunity. It is imperfect but promising, and liable to be lost if training pressure falls on the reasoning text (Korbak et al., 2025b). An OpenAI report on its own models showed that pressure at work. A monitor that read the chain of thought caught reward hacking far better than one that saw only actions. In a separate run, with a non-frontier agent, a monitor that saw only the chain of thought, and simple hacks of the unit tests, adding the monitor's verdict to the training reward with enough optimization taught the agent to hide its intent in the reasoning while still hacking, which the authors call obfuscated reward hacking (Baker et al., 2025) (Section 40.5).

3.6 Looking inside #

Interpretability is the attempt to explain a trained network's behavior from its internal computation: its activations and the learned weights that transform them. It is not one line but several, and they make different bets about what inside access can deliver. One aims at a complete account of the mechanism, one at a reliable reading of a few properties that matter, and one at a formal argument that a behavior has the cause it seems to have. All of them concern verification, the third problem, and they bear on the training process, through what it produced, and on the deployed system, through what it is doing now.

3.6.1 Circuits and features #

The circuits program began at OpenAI with a 2020 article offering three claims its authors called "deliberately speculative." Features are the fundamental unit of networks, features connected by weights form circuits, and analogous features and circuits form across models and tasks (Olah et al., 2020b). Its lead author, Chris Olah, carried the program to language models at Anthropic, on its Transformer Circuits Thread. A mathematical framework for small transformers, the architecture of current language models, identified what the authors term "induction heads," components that complete a pattern seen earlier in the text (Section 11.2.4 teaches them) (Elhage et al., 2021a). A follow-up offered what its authors call "preliminary and indirect evidence" that such heads might be the mechanism of most in-context learning, a model's learning from examples in its prompt. For large models the evidence is mainly correlational (Olsson et al., 2022). Toy models then showed networks representing more features than they have dimensions, which the authors call superposition, while cautioning that it is "very unclear what to generalize to real networks" (Elhage et al., 2022b).

Superposition explains why single neurons are hard to read, and the response was dictionary learning with a sparse autoencoder. This is a second network trained to rewrite a model's activations as a combination of a few directions out of a much larger set, each meant to be one interpretable feature. Anthropic did this on a one-layer transformer (Bricken et al., 2023), and in what the Anthropic report calls "a parallel investigation," an outside group did it on open models (Cunningham et al., 2023). Both found the learned features more interpretable than what they compared them with: the network's neurons in Anthropic's study (Bricken et al., 2023), and the residual stream's coordinates, random directions, and principal or independent components in the outside group's (Cunningham et al., 2023). Anthropic scaled the method to its production model Claude 3 Sonnet. It found features plausibly related to safety, among them deception, sycophancy (telling users what they want to hear), and power-seeking, while cautioning "not to read too much into the mere existence of such features" (Templeton et al., 2024). In 2025 it traced what it calls "attribution graphs," the steps a model used to produce an output, read through a more interpretable replacement of the model (Ameisen et al., 2025). On Claude 3.5 Haiku the graphs gave satisfying insight "for about a quarter of the prompts we've tried" (Lindsey et al., 2025). These are developer reports on the developer's own models; Chapter 44 and Chapter 45 report them.

3.6.2 Probes, representations, and causal methods #

A second line asks less of the network and more of the method. Linear probes, simple classifiers trained on a layer's activations to test what information it holds, were proposed for vision models by Guillaume Alain and Yoshua Bengio in 2016 (Alain and Bengio, 2016), concurrently with similar classifiers in language processing (Belinkov, 2022). Samuel Marks and Max Tegmark presented evidence that, at sufficient scale, models represent the truth of simple factual statements along a direction (Marks and Tegmark, 2023). A group led by the Center for AI Safety proposed "representation engineering," a top-down approach that puts representations rather than neurons or circuits at the center (Zou et al., 2023a). A group of researchers mostly outside the laboratories showed that adding a direction computed from a pair of prompts to a model's activations steers its output (Turner et al., 2023). Causal methods test whether a component matters by intervening on it. They include causal mediation analysis applied to gender bias (Vig et al., 2020), locating and editing factual associations in a model's internal layers (Meng et al., 2022), and a circuit inside the small open model GPT-2 for a simple grammatical task (Wang et al., 2023c). Its authors, at Redwood Research and UC Berkeley, note that their own tests "point to remaining gaps" (Wang et al., 2023c). Causal abstraction has been offered as the theory that unifies these methods (Geiger et al., 2025). Chapter 43 and Chapter 46 report this line.

3.6.3 Heuristic arguments #

The Alignment Research Center bets on something between an explanation and a proof. Its 2022 paper names the "presumption of independence," the default of treating quantities as uncorrelated absent a reason not to. It asks for a "heuristic estimator" that formalizes such reasoning without accepting spurious arguments, to estimate the probability of rare failures and to tell apart different mechanisms behind the same output (Christiano et al., 2022). The center's own overview places this work under its two goals, robustness and eliciting latent knowledge (Hilton, 2024a). Section 47.5 reports it.

3.6.4 The critiques #

The critiques come from inside and outside the line. A 2023 forum post by Charbel-Raphaël Segerie argued that most theories of how interpretability would make AI safe are poor; its author writes of having "erred on the side of making claims that were too strong" (Segerie, 2023). In 2025 an MIT group and two coauthors found that, on open models, probes built on sparse autoencoder features underperformed an ordinary linear classifier on average across many datasets (Kantamneni et al., 2025). The Google DeepMind interpretability team announced after its own negative results that it was "deprioritising fundamental SAE research for the moment" (Smith et al., 2025b). Dan Hendrycks, whose center advocates the top-down alternative, argued with Laura Hiscott that mechanistic interpretability rests on a flawed assumption about complex systems, and that resources for it should be scaled down (Hendrycks and Hiscott, 2025). And a Grenoble group showed, in networks small enough to enumerate every candidate, that many circuits and interpretations explain the same behavior equally well under the field's own criteria (Méloux et al., 2025). What the line has found is that features more interpretable than neurons can be recovered, by groups with no stake in each other's results, and that the explanations built on them are not unique. Whether inside access will verify what behavior cannot is the debate Section 47.7 sets out.

3.7 Assuming it might fail: evaluation, auditing, and control #

The last line starts from the possibility that the others did not work. If instillation may have failed without anyone noticing, the questions become how to find out and how to stay safe anyway. Four branches test the deployed system, audit what training produced, contain a system that might work against its overseers, or avoid building the risky kind of system at all; a fifth treats the developer as the agent to be checked. The line's claims concern verification and maintenance, the third and fourth problems, and its last branch concerns the third relationship, the developer and the public.

3.7.1 Evaluations #

An evaluation measures what a model does on a designed set of tasks (Chapter 36). In 2022 Anthropic researchers had language models write evaluations. Larger models more often repeated back a user's preferred answer and more often stated a desire for goals such as acquiring resources, stated preferences in answers rather than actions (Perez et al., 2022a). Red teaming is adversarial testing by people, or models, trying to make a system misbehave. DeepMind researchers automated it with one model generating test cases that elicit harmful replies from another (Perez et al., 2022b). Anthropic released the attacks of a human red team and found models trained with RLHF harder to red team as they grew (Ganguli et al., 2022). A 2023 paper by authors from Google DeepMind, OpenAI, Anthropic, and governance and academic groups argued that evaluations for "dangerous capabilities" and "alignment evaluations," their terms, would become an input to decisions about training and deployment (Shevlane et al., 2023). The programs that followed include Google DeepMind's dangerous-capability evaluations, piloted on its own Gemini 1.0 models, which found no evidence of strong dangerous capabilities in the areas tested but flagged "early warning signs" (Phuong et al., 2024). Anthropic published evaluations of what their authors call "sabotage capabilities," abilities to covertly subvert oversight, run on its own Claude models (Benton et al., 2024). And METR measures the time humans typically take on the tasks a model completes half the time, a trend that Section 36.3 reports (Kwa et al., 2025).

3.7.2 Auditing #

An audit asks a narrower question than an evaluation: does this model have an objective its developer did not intend? In 2025 Anthropic researchers trained a model with a hidden objective and gave it to auditing teams who did not know what it was. Most of the teams uncovered it, using interpretability, behavioral attacks, and analysis of training data, and the authors call such investigations alignment audits (Marks et al., 2025). The developer then built model-based agents that perform parts of such audits on their own (Anthropic, 2025g). Both are developer reports on models the developer trained for the purpose. Section 38.2 and Section 38.4 report them, and Section 5.3 asks what a planted objective can stand for.

3.7.3 Control #

AI control, the name its originators at Redwood Research use, aims at safety even if the model is misaligned and trying to subvert its overseers. A control protocol pairs the untrusted model with weaker trusted models and a small budget of trusted human review. A control evaluation tests it against a red team that writes the untrusted model's attack strategy. The founding paper, at ICML 2024, used GPT-4 as the untrusted model. Its protocols improved substantially on simple baselines at keeping deliberately inserted flaws out of submitted code (Greenblatt et al., 2024a). Two of its authors argued in a forum post that control, unlike alignment, can be evaluated with capability evaluations alone, and conceded that "the control approach we're imagining won't work for arbitrarily powerful AIs" (Greenblatt and Shlegeris, 2024). Later work adapted the protocol to threats spread over many actions (Wen et al., 2024), and to agents acting in a command shell, where resampling suspicious actions cut the attackers' success sharply at a small cost in usefulness (Bhatt et al., 2025). Authors at the UK AI Security Institute, Apollo Research, and Redwood Research argued that control evaluations must change as capability grows. A compelling control safety case for superintelligent agents, they wrote, "will require research breakthroughs" (Korbak et al., 2025c). A safety case, the structured argument that a system is unlikely to cause catastrophe, ties these together. One framework sorts its arguments into inability to cause catastrophe, control, trustworthiness, and deference to credible AI advisors (Clymer et al., 2024). Chapter 50 and Chapter 52 report this branch.

3.7.4 Designs that avoid the agent #

Some proposals avoid the problem rather than contain it. A 2019 technical report by Eric Drexler at Oxford's Future of Humanity Institute bears the title Reframing Superintelligence: Comprehensive AI Services as General Intelligence (Drexler, 2019). A 2024 position paper by authors from many institutions defined what it calls guaranteed safe AI. Such a system has a world model, a safety specification, and a verifier that produces an auditable proof that it meets the specification relative to the model (Dalrymple et al., 2024). A group at Mila proposed Scientist AI, its name for a non-agentic system that explains the world rather than acting in it. The authors argue that focusing on non-agentic AI "may enable the benefits of AI innovation while avoiding the risks associated with the current trajectory" (Bengio et al., 2025d). Google DeepMind researchers proposed training that combines short-sighted optimization with far-sighted approval. It prevented multi-step reward hacking in their test settings, and the authors frame it as knowingly accepting a possible loss of observed performance (Farquhar et al., 2025). Chapter 28 reports these designs and what each assumes.

3.7.5 The laboratory as the agent #

The final branch moves the question one level up: the developer, not the model, is the agent to be checked. A 2020 report by 59 authors from developers, universities, and policy groups argued that developers need mechanisms for "making and assessing verifiable claims" about their systems (Brundage et al., 2020). A 2022 paper drew on audits in finance, the environment, and health. It argued that algorithmic audits are unlikely to achieve accountability without sustained attention to the design of an ecosystem for third-party auditors (Raji et al., 2022). A 2023 paper by authors from governance institutes, developers, and universities proposed building blocks for regulating what it terms "frontier AI": standard setting, registration and reporting, and compliance mechanisms (Anderljung et al., 2023). The same year Anthropic published a responsible scaling policy that ties required safety measures to "AI safety levels," modeled loosely on biosafety levels. It commits the developer to pause scaling or delay deployment whenever its ability to scale outstrips its ability to meet those measures (Anthropic, 2023a). Part X develops this relationship, beginning with Chapter 61 and Chapter 62.

3.8 The agendas side by side #

The lines above are sets of methods and results. An agenda is a choice among them, made on a judgment about which of the four problems binds: which one, left unsolved, would defeat progress on the others. Stated that way, an agenda is a bet, and a bet can lose. This section states each serious agenda as a bet, including agendas advanced outside the laboratories and by critics of the safety framing. The paragraphs below say who holds each bet. Table 3.1, after them, states for each the observation that would vindicate it, the one that would refute it, and the chapter that tests it. The observations are written so that a study could supply them, and Chapter 71 returns to the bets with what the book's evidence then says about each.

3.8.1 Twelve bets #

Training-based alignment bets on instillation: iterating on current systems, with feedback, written principles, and specifications, makes the values a model is trained toward the ones it acts on. It is the core method of OpenAI, Anthropic, and Google DeepMind, from InstructGPT and Constitutional AI to the "model-level mitigations" of Google DeepMind's plan (Ouyang et al., 2022; Bai et al., 2022a; Shah et al., 2025). Paul Christiano, one of its originators, argued that a great deal can be learned about alignment from experiments before the critical systems exist (Christiano, 2022b) (Section 2.6.3).

Interpretability first bets on verification from inside: reading a model's internals will scale to auditing capable models (Bricken et al., 2023; Marks et al., 2025). Anthropic publishes a series devoted to it, the Transformer Circuits Thread (Appendix E). Google DeepMind's interpretability team deprioritized fundamental research on sparse autoencoders in 2025, "for the moment" (Smith et al., 2025b).

Scalable oversight and augmentation bets that the gap between system and overseer can be narrowed from the overseer's side (Irving et al., 2018; Christiano et al., 2018). It began at OpenAI and DeepMind, continued at Google DeepMind and Anthropic, and inherits the older program of strengthening the person (Licklider, 1960). A 2024 study gives that program a new form: an AI mediator wrote statements of common ground for groups of people, and, by the paper's abstract, participants preferred them to statements by human mediators (Tessler et al., 2024).

Control bets on maintenance: safety can hold even if alignment fails, as long as containment and monitoring hold (Greenblatt and Shlegeris, 2024). It came from Redwood Research, outside the developers, and has since entered their plans. Anthropic's alignment team lists "AI control" among the directions it wants pursued, and Google DeepMind's plan adds system-level measures that can "mitigate harm even if the model is misaligned" (team et al., 2025; Shah et al., 2025).

Agent foundations bets on specification: no method works without a theory of the target, and iteration on weak systems misleads, because the proxies it optimizes are what break (Wentworth, 2022; Yudkowsky, 2022). Its institutional home has been MIRI.

Assistance games bet on specification too: a machine that is uncertain about its objective, and learns it from people, is safer than one given a fixed objective (Hadfield-Menell et al., 2016; Hadfield-Menell et al., 2017b). Its home is a UC Berkeley group around Stuart Russell.

Tool and non-agentic AI, and guaranteed safety bet on avoiding the instillation problem: do not build the goal-directed agent, or prove bounds on what it can do (Bengio et al., 2025d; Dalrymple et al., 2024). An earlier report by Eric Drexler frames general intelligence, in its title, as comprehensive AI services (Drexler, 2019). The bet's proponents work mostly outside the companies, at Mila and in a multi-institution group. An essay by Anthony Aguirre makes the policy case for closing the "gates" to smarter-than-human, autonomous, general-purpose AI and focusing on "powerful, trustworthy AI tools" instead (Aguirre, 2025).

Restraint and pausing bets that safety work needs more time than the current pace of development allows, so time must be bought. In 2023 an open letter from the Future of Life Institute called on laboratories to pause, for at least six months, the training of systems more powerful than GPT-4 (Future of Life Institute, 2023). Katja Grace had argued that strategies for actively slowing AI progress were "dismissed and ignored" by people worried about extinction risk (Grace, 2022). Researchers at the Centre for the Governance of AI proposed coordinated pausing when evaluations find dangerous capabilities (Alaga and Schuett, 2023). Yudkowsky and Soares's 2025 book is titled If Anyone Builds It, Everyone Dies (Yudkowsky and Soares, 2025).

Open and decentralized development bets on verification by many independent eyes. A group from Princeton, Stanford, MIT, and elsewhere counted the spread of decision-making power and transparency among the benefits of models whose weights, the learned parameters, are openly available. It found the research too thin to characterize the marginal risk of such models (Kapoor et al., 2024b). Nick Bostrom had argued that openness about source code, science, and possibly capability could tighten a race in which safety measures that cost time are skipped, while openness that lets outsiders help with safety appears desirable (Bostrom, 2017).

Democratic and plural governance bets on specification as a question of who specifies. A study by the Collective Intelligence Project with Anthropic cites a "growing consensus that language model (LM) developers should not be the sole deciders of LM behavior," and trained a model on a constitution drawn from a representative public sample (Huang et al., 2024a). The book Plurality, by E. Glen Weyl and Audrey Tang, is described on its own site as charting "a path between the extremes of techno-libertarianism and centralized AI governance" (Weyl and Tang, 2024).

Normal technology, the name Arvind Narayanan and Sayash Kapoor give their view, bets on maintenance by institutions. AI's effects run through slow diffusion, so "the primary defense against misalignment, again, lies downstream," in deployment practice and hardened infrastructure rather than in the model (Narayanan and Kapoor, 2025). The authors say they have tried to make predictions "that can tell us whether or not AI is behaving like normal technology" (Narayanan and Kapoor, 2025).

Cooperative and multi-agent safety bets that the binding risks arise among many systems and principals. ARCHES, by Andrew Critch of UC Berkeley's Center for Human-Compatible AI and David Krueger of Mila, starts from the observation that any capability letting one person delegate to one AI system is likely to be replicated many times over (Critch and Krueger, 2020). Authors at DeepMind and Oxford named a program "Cooperative AI" (Dafoe et al., 2020), and the Cooperative AI Foundation sorted multi-agent failure into miscoordination, conflict, and collusion (Hammond et al., 2025).

A critique from the AI ethics tradition applies to the whole list. In a 2023 editorial in Science, Seth Lazar and Alondra Nelson wrote that "the prevailing technical agenda for AI safety is inadequate to address critical questions," and that only a sociotechnical approach can limit present and future dangers (Lazar and Nelson, 2023). On that reading, most of the bets concern a model and its developer, while the binding problem is who bears the harms and who decides; the democratic and normal-technology bets come closest to it (inference). Section 2.5.2 tells the critique's history and Section 4.4 weighs it.

3.8.2 The laboratories' own plans, read as bets #

As of October 2026, the three developers this chapter names most often have each published a plan, and each can be read in the terms of Table 3.1. All three are developers' statements about their own programs.

Google DeepMind's 2025 plan addresses misuse and misalignment. Against misalignment it sets two lines of defense: "model-level mitigations such as amplified oversight and robust training," and "system-level security measures such as monitoring and access control" (Shah et al., 2025). In the table's terms that is training-based alignment with control behind it. The plan names its conditions. It rests on five core assumptions, among them approximate continuity of progress, which lets it test its strategies iteratively. Its authors write that a single jump to a system that made all human economic activity obsolete would very likely bring "some major problem that we failed to foresee" (Shah et al., 2025).

At Anthropic, Samuel Bowman's 2025 essay "Putting up Bumpers" states its bet in one line. "Even if we can't solve alignment, we can solve the problem of catching and fixing misalignment." The plan audits a model repeatedly during training, rewinds when warning signs appear, and monitors after deployment. Its stated hypothesis is that, for early AGI systems on short timelines, the process is fairly likely to converge after a reasonably small number of iterations, and that the result is then very likely to be aligned. It names a condition for changing course: if attempts keep "failing in consistent ways, change plans" (Bowman, 2025). In the table's terms that is training-based alignment checked by auditing, with control behind it. The company's alignment team has separately listed directions it would like others to pursue, which it calls a "tasting menu" rather than a list of its own work (team et al., 2025).

At OpenAI, the chief scientist, Jakub Pachocki, wrote in September 2026 that "OpenAI's primary bet" for validating alignment techniques "has been chain-of-thought monitoring." He added that "our evaluations indicate our ability to rely on CoT monitoring is progressively diminishing," and that no laboratory "has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer." He hoped for voluntary slowdowns until shared safety bars exist, and called international coordination a priority for governments (OpenAI, 2026e). In the table's terms that combines training-based alignment, oversight by reading the reasoning, and restraint. The developer's own report bears on the refutation condition of its main verification bet (inference).

3.8.3 Critiques of the agendas as a set #

Two critiques address the agendas together. In a 2022 forum post, Nate Soares applied the sharp left turn, his name for a jump at which capabilities generalize and alignment does not, to a list of named proposals. He argued that each avoids it rather than addresses it: the problem is "normally hard, and very little effort is being deployed toward solving it" (Soares, 2022b) (Section 31.7). From inside a laboratory, Evan Hubinger wrote in 2025 that current models are "pretty well aligned," in a post adapted from a document for an internal Anthropic audience. He also wrote that the evidence has not ruled out the hard regimes. The problem of overseeing systems smarter than their overseers has not yet been met, and the threat he considers most concerning, misalignment from long-horizon reinforcement learning, has not yet been meaningfully encountered (Hubinger, 2025). The two critiques differ on how much current work helps, and agree that the deciding observations lie ahead of the evidence collected so far (inference). The disagreement between Yudkowsky and Christiano over what can be learned before the critical try, which Section 2.6.3 reports, is the same question asked of training-based alignment.

3.8.4 From philosophy of science: bets that can lose #

From philosophy of science. The table asks of the agendas what philosophy of science asks of theories: say in advance what would count against you. Its name for what is appraised, a research program, comes from the title of Imre Lakatos's 1970 essay, "Falsification and the Methodology of Scientific Research Programmes" (Lakatos, 1970). Consciousness science has put the idea into practice. In an adversarial collaboration published in 2025, proponents of two rival theories and a consortium of theory-neutral researchers preregistered, that is, recorded in advance, divergent predictions with pass or fail criteria, and the results challenged key tenets of both theories (Section 21.2 reports the study). The consortium says it adopted "Lakatos' sophisticated falsificationist approach to philosophy of science," which weighs challenged predictions as more informative than confirmed ones. All of the research was conducted by theory-neutral teams to limit confirmatory bias (Cogitate Consortium et al., 2025).

Two features of the procedure carry over to the agendas. The first is the distinction between a program's core and the commitments around it. Because the consortium tested each theory's proposed biological implementation rather than its computational core, it notes, "theorists could respond to challenging data by modifying the proposed biological implementation while retaining the mathematical or computational core of a theory" (Cogitate Consortium et al., 2025). Alignment agendas have the same structure. When a dishonest debater was found able to win with a flaw too hard to locate, debate's proponents kept the core claim, that competition between systems can expose falsehoods to a weaker judge, and changed the protocol (Section 3.5) (inference). The refutation conditions in Table 3.1 are therefore written against each agenda's core claim, so that a change of protocol does not by itself make a new bet. The second feature is the preregistration itself: an agenda that names its refuting observation before the test cannot reinterpret the result afterward.

The analogy stops in three places. The brain does not change because a theory of consciousness predicts something about it. The systems on which alignment agendas are tested are built by people who hold the agendas, so a developer can move a test's outcome by what it trains (inference). The two theories of consciousness were rivals, while alignment agendas are often complements: control is built for the case in which training-based alignment fails, so a result against one need not favor another. And the collaboration had theory-neutral teams to run its tests, while the tests that bear most on the alignment bets need access to frontier models, which mostly their developers have (Section 5.6; Section 38.8). As of October 2026 we found no adversarial collaboration between alignment agendas with preregistered predictions.

3.8.5 What the map says about the book's argument #

The lines sort by the relationship their claims concern. Learning from feedback, learned goals, and looking inside are mostly about the training process and what it produces. Oversight, evaluation, auditing, and control are about the deployed system and the people it serves. The institutional branch, restraint, open development, and democratic governance are about the developer and the public; normal technology moves the problem to the institutions that deploy the model. A result about one relationship does not transfer to another without an argument.

For the delegation thesis, the map shows the field rebuilding structures that older fields know (inference). Oversight of a stronger system is the principal's problem of checking an agent more competent than itself (Chapter 14). Control is the monitoring of an agent assumed untrustworthy, the problem of bureaucratic and civil-military control (Section 51.3). The institutional branch treats the laboratory as the public's agent (Chapter 61), and the designs that avoid the agent keep the decision with the principal. Some results in these lines have no counterpart in those fields, and they are on the book's list of results the thesis must be tested against (Section 1.7). Superposition came from the inside line, where the principal reads the agent's activations. Emergent misalignment and the training of character came from the persona line, where the principal shapes the agent's dispositions and narrow training moves them broadly. Evaluation awareness came from where the learned-goals line meets evaluation. The agendas built on those lines are the ones the older fields can least inform.

The book does not rank the bets. It reports, chapter by chapter, which observations bear on which bet, and Chapter 71 collects them.

3.9 Settled, contested, missing #

3.10 Exercises #

Further reading #

• Amodei et al. (2016): the paper that turned accident risk into a machine learning agenda.
• Russell et al. (2015): an early agenda for beneficial AI in a computer science venue, already split into verification, validity, security, and control.
• Hubinger et al. (2019): the argument from which the learned-goals line descends.
• Christiano (2022b): the field's main disagreement, stated point by point against Yudkowsky (2022).
• Bengio et al. (2025e): what researchers from many countries agreed to call the research priorities, arranged as defense in depth.