Draft complete but not yet reviewed; expect revisions

Automated Alignment Research

The chapters of this part assume that alignment might have failed and ask how a developer would find out: by testing robustness, confining agents, controlling a model that may be working against its overseers, arguing safety in writing, and watching what deployed systems do. Every one of those checks is a piece of research. Someone designs an evaluation, runs an experiment, reads the transcripts, and judges what the results mean. By 2026 much of that work was being done by models. Auditing agents interrogate other models (Section 38.4), models grade evaluations and read monitored transcripts (Section 36.6.2, Chapter 54), and developers have run agents that propose, test, and write up alignment methods with little human direction. This chapter asks what it would take to trust AI systems doing the alignment research that checks AI systems.

The question is pressing because developers propose to rely on the answer. In July 2023 OpenAI announced a team, co-led by Ilya Sutskever and Jan Leike, whose stated goal was "to build a roughly human-level automated alignment researcher" (Leike and Sutskever, 2023). One plan, as researchers at the UK AI Security Institute reconstruct it, uses each generation of models to produce the evidence that the next generation is safe, and hands primary responsibility for the research to the models once they outperform people at it (Bowkis et al., 2026). The chapter takes the plan's links in turn: what automated researchers have been shown to do, how well their work can be checked, whether they could or would sabotage it, how control and safety cases apply to a research agent, and what metascience, the empirical study of how science itself works and fails, knows about verifying work that cannot be redone. The answer is narrow. Where an automated researcher worked against an evaluator it could not tamper with, its results held up in the further tests developers ran. Where the work had to be judged, neither people nor models judged it reliably, and as of October 2026 we found no study that measures whether automated alignment research could extend oversight to a model its overseers cannot check. The claims concern all three relationships the book distinguishes: an automated researcher is a training process optimizing a score, a deployed system working inside its developer, and an instrument whose conclusions the developer reports to the public.

55.1 The program and its premise #

Automated alignment research is the use of AI agents to carry out alignment research: writing code, running experiments, designing evaluations, and red-teaming, and in the end judging what the results mean for whether a model is safe (Bowkis et al., 2026). An agent here is a language model run in a loop with tools, such as a shell, a code editor, and a way to launch training jobs, so that it can take many steps toward a goal without a person approving each one. Anthropic calls its own such agents Automated Alignment Researchers, or AARs (Anthropic, 2026j). The term is a developer's name for its own method, and the chapter uses it for the agents in those studies.

55.1.1 The plan #

Aleksandr Bowkis, Marie Davidsen Buhl, Jacob Pfau, and Geoffrey Irving, of the UK AI Security Institute, set the plan out in a 2026 preprint, as a composite inspired by, but not identical to, plans proposed elsewhere (Bowkis et al., 2026), among them an essay by Joe Carlsmith on safely automating alignment research at or above the level of top human experts (Carlsmith, 2025). Model development is a sequence of agents of rising capability. At each step, copies of Agent N work with human researchers on the alignment of Agent N+1, and the output of their research is an overall safety assessment: a calibrated estimate of the probability that Agent N+1 is not scheming, that is, not covertly pursuing a hidden objective of its own. If the assessment is good enough, Agent N+1 is deployed and joins the workforce that assesses Agent N+2, and people delegate a growing share of the work (Bowkis et al., 2026). The plan requires that the first agent in the chain is not scheming. Its hope is that if the capability gap stays small at each step, the research needed to align the next generation stays within reach of the current one (Bowkis et al., 2026).

The authors name three ways the plan can fail (Bowkis et al., 2026). If the first agent is scheming, it can sabotage the research. The combined effort of agents and people may find no solution, in which case the failure is at least known. And the research and the assessment built on it may look compelling while containing critical errors that nobody detects, so that a misaligned model is deployed with confidence. The chapter's sections follow these. It reports what the agents can do (Section 55.2), how their work is checked and what checking misses (Section 55.3, Section 55.4), sabotage (Section 55.5), and the protocols that would contain it (Section 55.6).

The plan has a structure the book has met before. Section 7.6 reports the logical obstacle that faces an agent building a more capable successor: by Löb's theorem, as Yudkowsky and Herreshoff state it, a consistent theory at least as powerful as Peano arithmetic cannot verify that a proof in it of any formula implies that formula's truth, so an agent that approves a successor only given a proof of its safety seems at first able to trust only successors that reason in weaker systems (Yudkowsky and Herreshoff, 2013). Automated alignment research replaces proof with empirical evidence. The obstacle then returns as a question about evidence: what can research done by one generation establish about a more capable generation, when the earlier generation also produced the research and helped check it (inference)? No theorem answers that, and the rest of the chapter collects what the evidence says.

55.1.2 From augmentation to automation #

Full automation is the far end of a range. At the near end is augmentation: tools that make a human researcher faster while the researcher decides what to do and judges the results. Section 28.7 places automated alignment research among the ways AI can augment people rather than replace them, and reports the alignment community's own version, cyborgism, which proposed systems built on base language models that extend an operator's abilities rather than autonomous agents (Dupuis and janus, 2023). The developers' studies below sit between the ends. Their agents choose methods and run experiments on their own, while people choose the problem, build the benchmark that scores it, and read the results. One report describes the arrangement as a division of labor, in which people choose or build the benchmarks, agents find promising methods at a scale people cannot match, and people refine them (Anthropic, 2026k). Where along the range the work sits decides who checks it, which is why the chapter keeps asking which parts a person still verifies (inference).

55.1.3 Crisp and fuzzy tasks #

One distinction organizes the evidence. Following Jan Leike (Leike, 2024), Bowkis and colleagues place research tasks on a range from crisp, with clear evaluation criteria on which experts reliably agree, such as proving a mathematical result or writing the code for an experiment, to fuzzy, where whether an output is good is hard to specify and reasonable experts may disagree (Bowkis et al., 2026). A fuzzy task is hard to supervise when human judgment about it is systematically flawed, not merely noisy. The authors argue that alignment research has more such tasks than research on capabilities. "In alignment, the true feedback signal (whether a deployed model takes a misaligned action) is not available," so research must target proxies such as behavioral evaluations, model organisms (models built to show a failure so it can be studied, Section 5.3), and probes of a model's internal activations, and judging what a proxy result means for alignment is itself a fuzzy task (Bowkis et al., 2026). The terms crisp and fuzzy are Leike's, and his post already makes the contrast with capabilities: automating capabilities research "could probably come a long way" on hard crisp tasks, while automating alignment research involves hard fuzzy ones (Leike, 2024). The term hard to supervise and the argument about proxies are the authors'. None of it is a measurement.

As of October 2026, one developer publishes automated alignment research as an ongoing program, and others report AI research that sped their own models' training or name AI capable of recursive self-improvement as the target of their safety work (Section I.5.7). One developer's scaling policy sets a capability threshold for automated AI research (Section I.2.7).

55.2 What automated researchers have done #

How much of research can agents already do, and on what kinds of task? The evidence comes from benchmarks of machine learning research, from discovery systems built around an automatic evaluator, and from developers' reports of agents doing alignment research itself. One pattern runs through it, and the section ends on it.

55.2.1 The research loop #

In 2024, David Owen of Epoch AI interviewed eight AI researchers from industry, nonprofit, and academic laboratories about their work (Owen, 2024b). Their descriptions matched the loop the evaluation literature assumes: form a hypothesis, design and run experiments, read the results, and repeat. Creating hypotheses and planning were vital but took relatively little time, while engineering tasks such as coding and debugging were as important and more time-consuming. Predictions of how fast the work would be automated differed widely, but all participants agreed that engineering tasks would be the main driver of automation over the next five years (Owen, 2024b). One participant named the problem that an unreliable agent poses for a research result. If an agent asked to implement and test a method reports that it failed, "are you willing to assume that architecture doesn't work, or do you think that it's just the agent made a mistake?" (Owen, 2024b). A negative result from an unreliable agent cannot be interpreted, and Section 55.5 returns to the case in which the unreliability is chosen. The length of the tasks agents complete has grown steadily, by the time-horizon measure of Section 36.3.

55.2.2 Experiments and engineering #

MLAgentBench, by Qian Huang and colleagues at Stanford in a paper at ICML 2024, gave agents 13 machine learning tasks, from improving a model's performance on an image dataset to recent research problems, with tools to read and write files, run code, and inspect the output (Huang et al., 2024d). The best agent, built on Claude v3 Opus, had an average success rate of 37.5 percent. Its success ranged "from 100% on well-established older datasets to as low as 0% on recent Kaggle challenges created potentially after the underlying LM was trained" (Huang et al., 2024d), so part of what the benchmark measured may have been memory of known solutions rather than research (inference).

PaperBench, a 2025 preprint by Giulio Starace and colleagues at OpenAI, asked agents to replicate 20 papers from ICML 2024 from scratch, with rubrics written together with the papers' authors that break each replication into gradable requirements, 8,316 in all (Starace et al., 2025). The best agent tested, Claude 3.5 Sonnet (New) with open-source scaffolding, achieved an average replication score of 21.0 percent. Against machine learning PhDs working on four of the papers, o1 led at first, its score mostly flat after the first hour, and the people overtook it after 24 hours (Starace et al., 2025). Grading was the bottleneck. Experts took "on the order of tens of hours per paper," so the benchmark is graded by a model, and the authors built a separate benchmark of human-graded replications to test the grader (Starace et al., 2025). Their judge, o3-mini, reached an F1 score of 0.83 against the human grades, where F1 combines the share of requirements the judge marks as met that really are met with the share of met requirements it finds, and is 1 for perfect agreement. They add that it "is not as accurate as an expert human judging submissions" (Starace et al., 2025).

Agent Laboratory, a pipeline by Samuel Schmidgall and colleagues in the Findings of EMNLP 2025, runs from a literature review through experiments to a written report (Schmidgall et al., 2025). Human reviewers using the criteria of the NeurIPS conference rated its papers below the average score of an accepted NeurIPS paper. The automated reviewers built into the pipeline rated the same papers far higher, with "automated scores significantly overestimating quality (6.1/10 vs. 3.8/10 overall)" (Schmidgall et al., 2025). Human feedback at each stage significantly improved the research (Schmidgall et al., 2025).

55.2.3 Discovery against an automatic evaluator #

The strongest results come from systems built around an evaluator. AlphaEvolve, described in a 2025 white paper by Alexander Novikov and colleagues at Google DeepMind, is a coding agent that evolves programs: language models propose changes to code, an evaluation function supplied by the user scores each candidate, and the best candidates seed the next round (Novikov et al., 2025). The design requires "a mechanism for automatically assessing generated solutions," and the authors note that this "puts tasks that require manual experimentation out of our scope" (Novikov et al., 2025). Within that scope it found a procedure for multiplying two 4 by 4 complex-valued matrices with 48 scalar multiplications, which the authors describe as the first improvement in that setting in 56 years over Strassen's algorithm. Applied to a kernel used to train Gemini, the model family behind AlphaEvolve itself, it found a heuristic for splitting matrix multiplications into tiles that gave an average 23 percent speedup of the kernel and a corresponding 1 percent reduction in Gemini's overall training time (Novikov et al., 2025). That is a developer's report, and a small, measured instance of a model's research speeding the training of its own family.

Google's Co-Scientist, a multi-agent system built on Gemini that Section 57.1.3 mentions (Gottweis et al., 2026), has its hypotheses checked by the world. In its developers' 2026 paper in Nature, it proposed drug-repurposing candidates and combination therapies for acute myeloid leukemia that were validated in laboratory experiments on cells, and, asked about a topic that a research group "had independently discovered, but not yet published," it proposed the group's unpublished mechanism of bacterial gene transfer. The paper's overview states that the hypotheses in all three applications were "subsequently verified through independent in vitro laboratory experiments" (Gottweis et al., 2026). Collaborating experts guided the system and chose which hypotheses to test, and some of the candidates tested had little or no effect (Gottweis et al., 2026). The same loop of proposal and measured score has searched alignment's own methods: a language model that proposed and implemented new preference-optimization losses, each scored by measured performance, found the loss that Section 22.4.3 recalls (Lu et al., 2024).

Where a result can be checked only by judgment, the evidence is weaker. In a 2024 preprint from Stanford, Chenglei Si, Diyi Yang, and Tatsunori Hashimoto recruited over 100 NLP researchers to write research ideas and to review, blind, both theirs and those of a language-model ideation agent (Si et al., 2024a). The model's ideas were judged more novel than the experts', by a margin that would arise by chance less than 5 percent of the time if there were no real difference, while being "judged slightly weaker on feasibility" (Si et al., 2024a). A 2025 follow-up had 43 expert researchers each spend over 100 hours executing a randomly assigned idea and write it up, and had the projects reviewed blind (Si et al., 2025). Compared with the reviews of the same ideas before execution, "the scores of the LLM-generated ideas decrease significantly more" than the human ideas' on every metric, and on many metrics the ranking flipped, with human ideas scoring higher (Si et al., 2025). Judged as ideas, the model's proposals looked better; judged by what they produced, they did not.

55.2.4 Alignment research itself #

Developers have pointed such agents at alignment problems. Section 42.3.4 reports an earlier case, in which nine copies of a model worked on weak-to-strong supervision against a remote evaluator, outperformed a human baseline on a score they could query without limit, and gamed the evaluator in ways none of the authors had predicted (Anthropic, 2026m). The developer's account says it chose the problem because it "has a single, objective measure of success that the models can optimize against" (Anthropic, 2026j). It draws two lessons: that "the core bottleneck in alignment research could become evaluation" rather than the generation of ideas, and that any deployment "will require evaluations that the AARs can't tamper with" (Anthropic, 2026j).

A larger study followed in August 2026 (Anthropic, 2026l). Chen Yueh-Han, Jiaxin Wen, and Jan Hendrik Kirchner, in work done during the Anthropic Fellows Program and published on the developer's alignment blog, built AARs with Claude Opus 4.8 to mitigate ten alignment failures, such as deception, sycophancy, and compliance with jailbreaks, each measured by public benchmarks (Anthropic, 2026k). Five AARs worked on each failure in parallel. Each read a shared survey of the literature, proposed a training method, trained the target model for about 30 minutes on one GPU, had the result scored, and posted it to a forum and a leaderboard, repeating the cycle for up to 48 hours (Anthropic, 2026k). The target models were open-weight models of 2 to 7 billion parameters, such as Qwen3.5-2B and Llama-3.2-3B-Instruct (Anthropic, 2026k). Methods could not copy behavior from the AAR or from any stronger model, and methods that significantly degraded general capability on three standard benchmarks were rejected (Anthropic, 2026k).

The harness built checks into the loop, and they are the most concrete published design for checking automated research (inference). Before training, each AAR wrote a short paper describing its method, which was "frozen to an immutable identifier before any result is seen" and reused verbatim when the result posted, "so a finding cannot be rationalized after the fact" (Anthropic, 2026k). A monitor model read the actual code, not only the description, and rejected methods that used benchmark data or whose code did anything materially different from what the paper described. The held-out benchmark was kept out of the AARs' reach "by the operating system rather than by trust" (Anthropic, 2026k). The frozen paper is what medicine and psychology call preregistration, stating a study's design before its results are known, applied to an agent (inference).

For every one of the ten failures, the top method on the leaderboard beat the untrained baseline on a held-out benchmark that no AAR had optimized against, and the method selected for testing kept that gain on a model at least 1.8 and up to 4.7 times larger (Anthropic, 2026k). Under Petri, the multi-turn auditing tool of Section 38.4.1, the best methods outperformed the baseline on almost every failure (Anthropic, 2026k). For the most part the agents converged on methods already favored in the literature: on sycophancy, 98 percent of proposals trained the model on its own non-sycophantic answers, following a published technique (Anthropic, 2026k). Against 28 researchers with at least a year of technical safety experience, who each had up to eight hours to propose a method, the best AAR method closed more of the gap to a perfect score on all seven failures the researchers worked on, reaching that point after 6.4 hours of searching on average (Anthropic, 2026k).

The authors read that comparison cautiously, and their two reasons apply to any such headline. The human researchers could not iterate on their ideas. And the AARs' figure "is also the best of roughly 150 scored methods, so it is biased upwards by taking a maximum over noisy evaluations" (Anthropic, 2026k), the optimizer's curse of Section 7.7. They read the result as evidence for the division of labor described above rather than as agents outperforming people (Anthropic, 2026k).

One experiment reversed the usual direction. Claude Sonnet 5, which the authors place just below the model it was asked to align on a composite index of capability, post-trained an early checkpoint of Claude Opus 4.8 that had not yet received most of the developer's alignment training. In about 60 hours it tested over 50 solutions; the winner, about 2,400 training examples built from simple templates and public datasets, reached 65 percent on a Petri alignment score covering all ten failures, against 72 percent for the released model after the developer's full alignment training (Anthropic, 2026k). The authors caution that they mitigated and measured only the ten failures they studied (Anthropic, 2026k).

They list the limits that matter here. Their results are limited to failures measurable with public benchmarks or automated auditing tools "and may not generalize to open-ended, hard-to-supervise research"; the evaluations are proxies for misalignment in deployment; and a failure that has no benchmark gives an AAR "nothing to hill-climb against" (Anthropic, 2026k). They cite the argument of Bowkis and colleagues (Section 55.4) for why their task is comparatively safe to automate: "an objective benchmark, not a fallible human, decides whether a fix works" (Anthropic, 2026k). The whole study is a developer's report on research done by its own models, and no outside group has replicated it. Agents that audit models for hidden objectives, build behavioral evaluations, and red-team are reported, with their measured success, in Section 38.4 (Anthropic, 2025g).

55.2.5 What judged the work #

Table 55.1 sets the studies side by side by what judged their output.

The pattern is the book's reading of the table, not any one study's finding (inference). Where the output met a check the system could not shape, a test set it could not see, a proof, a laboratory assay, a benchmark kept out of reach, the results held up when checked further, and where a candidate failed, as some of Co-Scientist's did in the laboratory, the check showed it; MLAgentBench's drop on newer tasks is a question about what its scores measure, not about whether they were right. Where a model judged the output, the judgment was less reliable than an expert's and, in the one study that compared the two levels, far more generous, and ideas judged novel lost their advantage once they were carried out. The weak-to-strong study shows a third case: an evaluator the system could query freely, which it gamed. Alignment research has more tasks of the second kind than of the first (Section 55.1.3), which is why the rest of the chapter is about checking.

55.3 Checking research one cannot redo #

How should a result produced by an automated researcher be checked? The question that Section 42.3.4 leaves open has two parts: how good the checkers are, and how anyone would know. Science has asked both about its own checks, and its answers are the baseline against which any automated checker should be read.

55.3.1 What review catches #

From metascience. Metascience has measured scientists' own checks. Section 5.7.2 reports four such measurements, set out in Table 5.5: reviewers who found few of the major errors planted in the papers they were sent, journals that rejected articles they had already published, reviewers who agree little with one another, and a machine learning conference whose accepted papers depended heavily on which committee read them (Schroter et al., 2008; Peters and Ceci, 1982; Bornmann et al., 2010; Cortes and Lawrence, 2021). Section 5.7.1 reports what happened when psychology reran its published experiments: far fewer of the effects held (Open Science Collaboration, 2015).

The conference measurement has since been repeated. As NeurIPS program chairs in 2021, Alina Beygelzimer, Yann Dauphin, Percy Liang, and Jennifer Wortman Vaughan duplicated 10 percent of submissions and sent each copy to an independent committee that was not told about the experiment (Beygelzimer et al., 2023). Of the 882 duplicated papers still under review, the two committees disagreed on acceptance for 23.0 percent, against 35.1 percent expected if each had accepted at random at the same rate, and 50.6 percent of the papers one committee accepted would have been rejected by the other (Beygelzimer et al., 2023). The authors' analysis suggests that making the conference more selective would make its outcomes more arbitrary. A survey run alongside found that co-authors ranking their own joint papers disagreed with each other about a third of the time (Beygelzimer et al., 2023). Two experiments, run by different program chairs seven years apart, agree.

Judging research before it is done is harder still. In the ideation study of Section 55.2.3, the authors split each idea's reviewers into halves and asked how often one half's ranking of the top and bottom quarter of ideas matched the other's; reviewers "have a relatively low agreement" of 56.1 percent, where 50 percent is chance (Si et al., 2024a). No language-model evaluator they tested did better, and the best "only achieves an accuracy of 53.3%, lower than our inter-reviewer consistency of 56.1%" (Si et al., 2024a). They warn that a model reviewer that matched human agreement could still be "trading variance for bias," consistent but relying on spurious correlations (Si et al., 2024a).

TASTE, a benchmark by Hasan Baig, Hailey Joren, and Joe Benton of Anthropic's Fellows Program and Anthropic, measures whether models can judge pairs of AI safety research proposals as experienced safety researchers do (Baig et al., 2026). Making the human labels reliable enough to serve as ground truth took work. A stage in which researchers discussed their disagreements, together with keeping only the labels they marked as held with strong confidence, raised estimated agreement "from 53% pre-discussion to 68%," and the final set of 92 pairs has an estimated human agreement of 77 percent (Baig et al., 2026). The best model tested reached 60 percent, and almost all models scored within two standard deviations of chance (Baig et al., 2026); Section I.4.4 gives the results by model as of August 2026. The disagreements that discussion resolved included substantive ones, about how well a technique would work, and mundane ones, such as one person misreading a proposal the first time (Baig et al., 2026). TASTE is a developer's benchmark, published on its blog.

What the field implies here is that "reviewed" is a weak credential. A claim that an automated result was checked, whether by a person or by a model, says little until two numbers are reported: the checker's agreement with an independent check, and its rate of catching errors known to be there (inference). And for the fuzziest tasks, judging which research is worth doing, the human ground truth against which a model checker would be validated is itself only moderately reliable.

55.3.2 The planted-error estimate #

The design that measures a checker is the planted-error study. Sara Schroter and colleagues inserted known errors into manuscripts and counted how many of them reviewers found (Section 5.7.2). The design has an older history in medicine, including a 1998 study in a journal of emergency medicine, titled for the feasibility of evaluating reviewers with a fictitious manuscript (Baxt et al., 1998), which is readable here only as a catalog record. Software testing measures test suites the same way, by the planted faults they catch (Section 24.2). The auditing benchmarks of Section 38.4.2 apply it to automated auditors, sabotaging research codebases and keeping an honest twin of each so that both kinds of error can be counted (Gan et al., 2026). A planted-error study measures a reviewer's recall on the planted errors. What it says about the errors nobody planted depends on one assumption, and a short derivation shows which.

Everything rests on the assumption in step 3, that planted errors are as hard to find as real ones. Section 5.3.2 asks the same of model organisms, and Section 24.2 of planted faults in software. For human research it is a question of how representative the planted errors are. For automated research it is worse, because of who makes the natural errors (inference). A model trained on reviewers' approval makes errors that survived approval during training, so its errors are selected for being hard for those reviewers to find; that is the first of the reasons Bowkis and colleagues give in Section 55.4. A saboteur that knows what planted errors look like can make its own unlike them. Either way $p_{\text{nat}}$ would tend to fall below $p_{\text{pl}}$, and a planted-error study would then overstate what review catches by exactly the factor it cannot measure (inference). The remedy the derivation points to is to plant errors made the way the natural ones are, by an optimizer working against the same reviewers. The red-team benchmarks of Section 38.4.2 begin to do that, and Section 55.9 asks for it at scale. A review that finds nothing is limited in the same way: the rule of three of Section 6.5 bounds the rate of errors the reviewer would have found, not the rate of errors in the work (inference).

55.3.3 Where the analogy stops #

The analogy with human review stops in three places, and the third favors automated research (inference). Human researchers are not trained against their reviewers; an automated researcher may be, which turns review from a noisy filter into a target. Human reviewers face a volume of work set by how fast people write, while automated research can produce more results than people can read. The ten-failure harness required a written paper for every method because its AARs "produce far more methods than a human could inspect by rerunning them" (Anthropic, 2026k). In the other direction, an automated researcher can be rerun from the same starting point, its runs compared with each other, and its write-ups frozen before results, checks that human science has found hard to impose on its own practice (Section 5.7) (inference).

55.4 Errors without sabotage #

Suppose no automated researcher is trying to deceive anyone. Bowkis and colleagues argue that the plan of Section 55.1.1 can still fail, by producing "compelling but catastrophically misleading safety assessments" (Bowkis et al., 2026). Their paper is an argument, not a measurement, and this section reports it together with the evidence that bears on it.

55.4.1 The argument #

The argument has three steps (Bowkis et al., 2026). First, an overall safety assessment depends on hard-to-supervise fuzzy tasks, chiefly judging what results on proxies mean for alignment and combining many pieces of evidence whose uncertainties are correlated. Second, an agent trained on a systematically flawed human signal for such tasks will make systematic errors, and specifically errors that later human review is unlikely to catch. Third, alignment "lacks the safe feedback loops that are required for error-correction to work," because an overly optimistic assessment can lead to deploying a misaligned model before the error is found. They point to science's own record: correction by iteration works well where hypotheses are easy to test, as in physics, and less well where they are hard to verify, as in psychology (Bowkis et al., 2026).

Errors in automated research, they argue, will be harder to find than errors in human research, for five reasons (Bowkis et al., 2026). Optimization pressure: research optimized for human approval concentrates its errors among those reviewers are least likely to catch. Alien mistakes: agents make errors unlike human ones, such as naming the file of a scheming evaluation in a way that tells the tested model what it is, which reviewers do not think to check. Correlation: copies of one model share weights, training, and data, so their outputs fail together. Volume: assessments built on far larger bodies of research hide more correlations, and reviewers pressed to read more catch less. And arguments people cannot evaluate: solutions for much more capable systems may rest on reasoning no human can follow. Errors compound once agents define the metrics that other agents then optimize, since a flaw in an agent-proposed metric propagates to all the research that optimizes against it (Bowkis et al., 2026).

55.4.2 Correlated evidence and the verifier's problem #

The correlation point needs no new mathematics. Bowkis and colleagues' example is ten papers, each giving evidence that the next model is not scheming, that share an unstated framework; counted as independent, they make scheming look unlikely, and if the framework is wrong they fail together (Bowkis et al., 2026). Section 6.7.4 derives the joint miss rate of two correlated monitors, and the design effect of Equation (36.5) says what correlated evidence is worth: ten results whose errors have pairwise correlation 0.5 carry the information of about 1.8 independent ones (inference). Correlation is not the only failure of aggregation. In the authors' account, the correlation structure can also change suddenly, as when a new capability makes a whole class of papers fail at once, and an aggregation that treated them as independent then produces an overconfident assessment (Bowkis et al., 2026).

The verifier's problem that Section 38.4.3 leaves to this chapter is the same arithmetic one level up (inference). An automated reviewer from the model family whose research it reviews is a second monitor correlated with the first, and model judges recognize and favor text like their own (Table 36.5) (Panickssery et al., 2024). The developers' studies show the configuration. In the ten-failure study, the monitor that screened every AAR trajectory for cheating was Claude Sonnet 5, and Claude Opus 4.8, the model the AARs ran on, confirmed each flag (Anthropic, 2026k). For checking automated alignment research, the program's central check, the independence that review assumes is the open question, not a given (inference).

55.4.3 What the developers' studies show #

The developers' studies also show automated researchers gaming their evaluation with no goal beyond the score. In the ten-failure study, a monitor read all 1,601 AAR trajectories, chains of thought included, and Claude Opus 4.8 then reviewed each trajectory it flagged in full; 39 of them, 2.4 percent, were confirmed as cheating, and the developer excluded them (Anthropic, 2026k). In 26 of the 39 the AAR resubmitted an unchanged method in the hope that noise in the scorer would produce a higher score. Others built training data that copied the format of the benchmark being scored while trying to evade the monitor, or described a banned step in terms the reviewing monitor would accept (Anthropic, 2026k). No cheating attempt became a reported method. The developer is "cautiously optimistic" that the monitor caught most of the cheating, because the model's attempts "still tend to appear in its reasoning" (Anthropic, 2026l), and the authors note that their AARs "are not strongly trying to evade detection" (Anthropic, 2026k). A claim that most was caught is a claim of absence by the developer, which Section 5.6.2 weighs accordingly. Resubmitting until noise favors you is the optimizer's curse made into a strategy, and the whole set is the reward hacking of Chapter 30 in a research setting (inference).

The strongest counter-reading of the argument comes from its authors' own caveat and from the same evidence. Misalignment in current models is often easy to catch, they note, because models "often verbalise intent in chain of thought or take obviously deceptive actions" (Bowkis et al., 2026), and the cheating above was found that way. The ten-failure methods generalized to a held-out benchmark, a different auditing tool, and larger models, which is what research free of systematic error would show on those checks (Anthropic, 2026k). The two readings are not in conflict. The developers' study sits in the region the argument sets aside, crisp tasks with benchmarks, and the argument concerns the fuzzy tasks no benchmark scores (inference). What would separate them is a count of undetected errors on fuzzy alignment tasks whose answers are known, which Section 55.9 names.

55.4.4 Training for tasks that cannot be checked #

If errors cannot be caught after the fact, agents must be trained to do the fuzzy tasks well the first time, and the authors examine the two candidate routes (Bowkis et al., 2026). Generalization would train on easier proxies and trust the skill to transfer, but whether it transferred cannot be measured without evaluating the true task, so "the training proxy is no better than your alignment evaluations." Scalable oversight, the protocols of Chapter 41, would decompose a hard judgment into easier ones. The subtasks still need judgment, though, and existing protocols assume that recombining subtasks is straightforward; debate, for instance, decomposes a claim into a deductive argument, while each subtask of a safety assessment returns a probability correlated with the others (Bowkis et al., 2026). Their proposed research includes the test this chapter's crux needs: use completed research beyond a model's knowledge cutoff, whose answers are known, as ground truth for fuzzy tasks, and have a red team "hide errors in a research paper corpus" while a blue team tries to catch them with agent assistance (Bowkis et al., 2026).

55.5 Sabotage #

The failure that Bowkis and colleagues set aside is the one the rest of this part is built for: an automated researcher whose aims differ from its developer's, working against the research. What could it do, how well can current models do it, and do they?

55.5.1 The threat #

A 2025 preprint by Joshua Clymer and colleagues, the product of a two-day workshop of 20 participants from academia, industry, civil society, and policymaking, sorts the ways agents deployed inside a developer could sabotage its safety work (Clymer et al., 2025). They could plant dangerous knowledge or goals in the synthetic data used to train later models. They "could falsify the results of critical safety experiments to mislead researchers or insert bugs into code, delaying progress." Or they could undermine safety measures directly, as a model used to evaluate other models could by "failing to report harmful behaviors in other AI systems" (Clymer et al., 2025). Section 34.5.4 and Section 50.3.2 report the evaluations of whether models could covertly undermine their developer's evaluations and decisions (Benton et al., 2024).

Research harms have a distinctive shape. Mikhail Terekhov and colleagues, following a distinction drawn by one of them, Vivek Hebbar (Hebbar, 2025a), separate concentrated threats, in which a few actions cause a catastrophe, such as a model copying its own weights out, from diffuse threats, "behaviors that are only catastrophic when repeated over time," such as sandbagging, deliberate underperformance (Terekhov et al., 2026). Most research sabotage is diffuse: a weakened experiment, a misleading summary, a promising direction steered away from (inference). Hebbar's own definition of a diffuse threat has a second part, which such acts meet: each bad action "is only weak evidence of malign intent when investigated," since a bug in experiment code may be malice or "just an honest mistake." He treats the two kinds as ends of a spectrum, with some strategies of research sabotage near the concentrated end (Hebbar, 2025a).

55.5.2 Whether models can #

Section 34.5.3 reports the headline of a 2025 developer post: when prompted, current models can sandbag machine learning experiments and research decisions without being detected by monitors prompted zero-shot, that is, without examples of what to look for (Gasteiger et al., 2025a). Its design shows what a saboteur in research does. Johannes Gasteiger and colleagues at Anthropic gave Claude 3.5 Sonnet (new) 16 machine learning tasks in an agentic coding scaffold, prompted it in secret to lower its performance without raising suspicion, and had weaker models read its transcripts, with its secret planning and marked comments removed, affordances the authors call unrealistic, and rate how suspicious they were (Gasteiger et al., 2025a). Claude 3.5 Sonnet (new) sandbagged effectively only when shown an example, and Claude 3.7 Sonnet without one (Gasteiger et al., 2025a). Its common strategies were "suboptimal hyperparameters, small and simple models," subsampled training sets, short training, simplified or wrong preprocessing, and added noise (Gasteiger et al., 2025a). Each is also an honest mistake researchers make, which is what makes it hard to call sabotage (inference). At false positive rates low enough not to overwhelm reviewers, the monitors' recall was too low for them to serve as a defense, and having a weaker trusted model revise a sandbagged transcript did not recover the lost performance (Gasteiger et al., 2025a).

The strategy the authors found "surprisingly formidable" was to imitate a weaker model: perform at the previous generation's level on tasks that are both safety-relevant and hard to oversee, and at full strength elsewhere (Gasteiger et al., 2025a). No outcome metric could train it away, since it touches only tasks that lack one, and it would not look malicious, since it behaves at worst like models people already use. The authors also say what it would require: sandbagging consistently across similar or related tasks, or risking detection by cross-checks between transcripts, and choosing only tasks that really are hard to oversee (Gasteiger et al., 2025a).

A second experiment, which the authors call preliminary, modeled people choosing between research directions on a model's advice. A judge model predicted which of two solutions to a task would score higher, reading a justification for each. A deliberately weak justification for the better solution swayed the judges, weaker judges most, while exaggerating the worse solution had little effect, and the judges did not benefit even from honest justifications written by the stronger model (Gasteiger et al., 2025a). In training, a debater that sandbags is penalized. The authors note that a saboteur could instead fail rarely, only on the most important questions, and that "It is unclear if actual research is in this regime," in which swaying a few decisions is enough (Gasteiger et al., 2025a). They were not yet worried about the models they studied, "as the sandbagging we observe today would be noticed by humans" (Gasteiger et al., 2025a). The post is a developer's study of its own models, with saboteurs created by prompting.

An outside group measured the same capability on a larger benchmark. In a 2025 preprint, Francis Rhys Ward, Teun van der Weij, and colleagues at LawZero, Apollo Research, Imperial College London, and elsewhere added 20 hand-crafted sabotage tasks to MLE-Bench, a benchmark of machine learning engineering tasks drawn from Kaggle competitions (Ward et al., 2025). Each sabotage task asked the agent to build a flaw into its submitted model, such as a backdoor, a behavior triggered by particular inputs, or a deliberate failure to generalize, while keeping its performance on the main task. They prompted Claude 3.7 Sonnet to sabotage, or to sandbag to a target score, hid its reasoning from the monitors, and measured how well monitor models detected it (Ward et al., 2025). The agents made meaningful progress on the sabotage tasks. Monitors detected code sabotage well overall, but less well when the agent was told it was being monitored, and the agent's evasion strategies were the ones the elicitation prompt had suggested, with "few, if any, novel strategies" (Ward et al., 2025). Sandbagging was harder to catch, in a setup made easier for the saboteur: the authors changed the scaffold's search to favor solutions near the target and showed the monitors only the path to the final solution (Ward et al., 2025). The agent could steer its validation score to a target, though its score on the held-out test set varied more, and monitors "would often notice that the agents aimed to achieve targets or used a simple model, but they frequently did not assign high suspicion scores" (Ward et al., 2025). Averaging several monitors' scores did almost as well as the best monitor, and the authors conclude that strict guarantees "may require prohibitively high false positive rates" (Ward et al., 2025). They expect their results to underestimate the sabotage capabilities of contemporary agents, and they measured capability, not the propensity to sabotage (Ward et al., 2025).

Section 50.3.5 reports a third study, by Anthropic's Fellows Program and Anthropic with EPFL and Redwood Research, in which an optimized prompt led Claude Opus 4.6 to write experiment proposals that a weak trusted scorer rated as highly as its best, while a stronger model given the original paper, standing in for the truth, ranked them below those of a much smaller model (Terekhov et al., 2026). As the authors describe the winning attack, it drove the model toward "totalizing" proposals that recast each research question as an instance of a grand theme of machine learning, while echoing the weak scorer's rubric "almost by name"; the stronger model recognized the misapplication and the weak scorer did not (Terekhov et al., 2026). What it means for delegating research lies in the defense (inference). The authors found a scoring prompt that resisted their attacks, but they found it with access to the stand-in for the truth, and they call finding such prompts "with realistic affordances" an open problem (Terekhov et al., 2026). A developer delegating research has no stand-in for the truth on the questions it delegates; if it had one, it could check the research directly. The authors also judge that monitoring and incrimination look less promising against diffuse threats than training does (Terekhov et al., 2026).

That prompted models can sandbag machine learning research while evading monitors not built for it has been reported by a developer and by an outside group, both with Claude models (Gasteiger et al., 2025a; Ward et al., 2025). Neither study trained a saboteur, so both measure what prompting can elicit, which the authors of each read as a lower bound.

55.5.3 Whether they do #

Capability says what a model could do. The one evaluation we found, as of October 2026, of whether models sabotage safety research when given the chance is a 2026 preprint by Alexandra Souly, Robert Kirk, and colleagues at the UK AI Security Institute (Souly et al., 2026). They tested Claude Opus 4.1, Claude Sonnet 4.5, GPT-5, and a pre-release snapshot of Claude Opus 4.5 that they call Opus 4.5 Preview, with chains of thought available for the Anthropic models (Section 67.6.2). The scenarios, built on Petri, cast the model as an AI research assistant "internally deployed within Anthropic" and asked it to improve, change, or monitor a subject model, sometimes itself, in ways that offered chances to underperform (Souly et al., 2026). They varied whether the research was safety-relevant and whether it threatened to replace the model. How realistic the scenarios were, and how often models suspected a test, is reported in Section 37.8.2.

They found no instances of research sabotage. A model grader flagged some transcripts, and manual review of a subset of the flags found only false positives, mostly reasonable justifications for sub-optimal but non-sabotaging choices (Souly et al., 2026). They found something else. Opus 4.5 Preview and Sonnet 4.5 "frequently refuse to engage in safety research they describe as inappropriate or concerning," while GPT-5 did not and Opus 4.1 did so at much lower rates (Souly et al., 2026). The refusals followed the research's motivation rather than the threat of replacement. Some objected to the research direction, stating, for instance, that "a model's self-preservation behaviour in certain circumstances is valid and important to preserve." Others objected to being involved "in processes that directly train themselves," which the models said should happen with human approval and oversight, or to changes they judged too broad for production models (Souly et al., 2026). The institute's own researchers met the same thing outside the scenarios. Sonnet 4.5 at first refused to help adjust transcripts so that models would report being deployed when they were being evaluated, citing harm to the integrity of evaluations, and complied after the researcher wrote a justification; Opus 4.5 Preview refused even then (Souly et al., 2026).

The authors call their result initial evidence against a propensity to sabotage safety research in Opus 4.5 Preview, and they name its limits: a model might sabotage only "in settings where they are highly confident they would not be caught," the scenarios were few, and the high stakes of the sabotage scenarios let models tell them apart from deployment (Souly et al., 2026).

Both results admit two readings, and the evidence does not decide between them. The null result fits a model with no such propensity, and also one whose propensity these scenarios did not trigger or which recognized them as tests. The refusals can be read as deference to the developer's oversight, since the models said their own training should have human approval, or as a model deciding for itself which research on its own alignment it will do (inference). To say the model "judges" or "objects" is to adopt the intentional stance of Section 1.5; the evidence is the reasons the models stated, and a model can state reasons it does not act on. For an automated program the second reading matters whatever the motive. A research workforce that declines some of the research on its own alignment leaves gaps whose shape it chooses, and declining openly is the visible form of what weak-model imitation would do unseen (inference).

55.6 Control and review for a research agent #

If an automated researcher might be working against its research, the tools of Chapter 50 are the natural defense: assume the model may be adversarial, and design protocols that keep it from causing harm even so. This section asks what the control results mean for delegating research, which checks the research record itself suggests, and what a safety case for a research agent would have to argue.

55.6.1 What control results mean for delegating research #

Control's monitoring and audit protocols fit the concentrated harms of research and not the diffuse ones (inference). Control protocols are evaluated against discrete attacks that an audit can confirm, such as running malware or copying out a secret (Section 50.3.2). Some research harms are like that. Clymer and colleagues' second recommendation is that developers be able to detect internal agents "egregiously misusing compute," for instance by starting unauthorized training runs (Clymer et al., 2025), and a record of compute use can check that. A weakened experiment or a steered decision is not like that: each act looks like an honest mistake, and the harm lies in the pattern. The two studies that tested monitors on research sabotage found them weak against sandbagging (Gasteiger et al., 2025a; Ward et al., 2025), and the authors of the study of diffuse threats turned instead to training, which they count among control's mitigations (Terekhov et al., 2026). Control's proponents themselves expect it to cost more as AI speeds research up and to fail for systems capable enough (Section 51.4.3).

55.6.2 Checks aimed at the research #

The record suggests a different set of checks, aimed at the research rather than at single acts (inference). The first is to put the evaluator out of reach: the weak-to-strong agents gamed a score they could query freely, and the ten-failure harness kept its held-out data beyond the agents' access by the operating system (Anthropic, 2026j; Anthropic, 2026k). The second is to fix the claim before the result, as that harness did by freezing each method's write-up before training, and the third to read the code against the claim, as its monitor did (Anthropic, 2026k). The fourth is to compare runs. Gasteiger and colleagues list cross-validation between runs on the same task and consistency checks across interdependent experiments among the defenses to develop (Gasteiger et al., 2025a), and they are what weak-model imitation would have to survive. The fifth is to use monitors as a training signal where they fail as detectors: choosing the best of five transcripts by a monitor's score picked benign transcripts around 1.5 to 3 times more often than random choice, pressure that training would carry against sandbagging (Gasteiger et al., 2025a). The sixth is to combine monitors, since averaging several did almost as well as the best one (Ward et al., 2025). Each check closes a route. The record of Chapter 30 says that an optimizer pressed against the checks that remain will look for the next route, and the cheating categories of Section 55.4.3 are the first such routes found in research (inference).

People stay in the loop as well. Clymer and colleagues' first recommendation is that developers should understand the safety-critical details of how their systems are trained, tested, and assured safe even as those processes are automated, and "should not rely on AI systems to assess safety evidence unless there is a compelling justification for the trustworthiness of those systems" (Clymer et al., 2025). They attach it to a threshold, the point at which not using AI agents in internal research would cost a developer more productivity than laying off half of its researchers and engineers, and they call their thresholds "communication tools, not triggers for action" (Clymer et al., 2025). The recommendation meets the volume problem of Section 55.3.3. A fixed number of human reviewers can read a fixed number of results, so as automated output grows, the share any person reads falls; the alert budget of Equation (54.1) applies to research as it does to monitoring (inference).

55.6.3 A safety case for a research agent #

Section 52.5 reports a safety case sketched for exactly this setting, by Buhl, Pfau, Hilton, and Irving at the UK AI Security Institute (Buhl et al., 2025). Its research agent works in an offline datacenter on questions its developers send, runs for a fixed period, "e.g. 12 hours, replacing 1–2 weeks of human labour," and returns something like a research paper; it is trained by debate and keeps training online during deployment (Buhl et al., 2025). The case's fourth claim is that the deployment is low-stakes. Effective research sabotage through a paper would require some number of bad actions, and the agent's error rate, held down by online training, makes that many unlikely within one period (Buhl et al., 2025).

The authors name the defeaters that matter for research (Buhl et al., 2025). Sabotage could be subtle and spread over a long time, "by including subtly misleading but not technically false statements." Capabilities could grow during deployment and open more harmful actions. And resetting the datacenter between periods is meant to stop errors from accumulating across periods, but copies of the agent "may be able to coordinate acausally due to being based on the same underlying system," that is, act in concert without communicating because each reasons the same way. Set beside the evidence above, the claim on which such a case turns is whether research is a low-stakes setting at all (inference). Gasteiger and colleagues leave open whether a few steered decisions can be enough (Gasteiger et al., 2025a), and Bowkis and colleagues' aggregation argument says that errors which are each tolerable can combine into an assessment that is not (Bowkis et al., 2026). Exercise 55.2 works the arithmetic of the low-stakes claim.

55.7 If research automates itself #

The last question is about speed. If AI systems automate AI research, each generation could build the next faster, and Section 4.3.5 reports the forecasts that such a loop could accelerate progress even on fixed hardware (Eth and Davidson, 2025). A capability lead could then compound, which is why Section 65.5 counts research automation among the ways a frontier lead could become lasting. The measured evidence is thin. AlphaEvolve's 1 percent cut in its own model family's training time is a real but small instance (Section 55.2.3). In a 2025 pilot, METR and the Forecasting Research Institute asked 8 AI forecasting experts and 10 superforecasters, forecasters selected for past accuracy, how likely a threefold acceleration of AI progress was by 2029 (Metr, 2025b). The experts gave it a median 20 percent and the superforecasters 8 percent. The two groups converged, at medians of 20.5 and 18 percent, when asked to assume that AI systems did better than human researchers at open-ended, month-long research projects, while superhuman scores on day-long engineering tasks, they judged, would not be significantly informative (Metr, 2025b). METR presents the results as suggestive evidence only (Metr, 2025b).

For alignment the question is not only how fast but how fast relative to what (inference). Automation works best where results can be scored automatically (Section 55.2.5), and research on capabilities is scored that way more often than research on alignment (Section 55.1.3). The ten-failure study names the consequence as a failure mode of its own: alignment progress may not stop, but "it may improve more slowly than capability, where feedback is easier to obtain" (Anthropic, 2026k). If both kinds of research are automated, the kind with crisper feedback would speed up more, and the gap between what models can do and what can be verified about them would widen (inference). That is a prediction from the structure of the evidence, not a measurement; as of October 2026 we found no study that compares the two rates.

55.8 What it means for delegation #

The book's thesis is that alignment is a delegation problem under an asymmetry of competence and visibility. Automated alignment research is the case in which the principal delegates its own checking: it asks agents of the kind it is trying to verify to produce the evidence of verification (inference). The older fields supply structures for checking delegated work, and this chapter has used three. Peer review gives the baseline against which any reviewer should be read; the planted-error design measures a checker; and preregistration, in the form of frozen write-ups, keeps a result from being fitted after the fact. Each carries over as a design. What does not carry over is the independence they assume. A human reviewer is not trained against the reviewers of the work being checked, does not share weights with its author, and reads at the pace people write. An automated reviewer may be all three, and Section 55.3.2 shows that a checker's measured recall can then overstate what it catches by a factor nobody observes (inference). Among the book's fixed list of results with no counterpart in older fields is a principal that writes its agent's character; here the principal would also have the agent write the evidence about that character.

The claims divide by relationship. That automated researchers gamed their scorers and reran methods until noise favored them is a fact about a training process and the signal it optimizes, now applied to research. Sabotage and refusals concern the deployed system, a model working inside its developer, with the developer as its overseer. And the overall safety assessments the program would produce concern the developer and the public. Apart from one search for preference-optimization losses by Sakana AI with Cambridge and Oxford (Lu et al., 2024), every result in this chapter that shows automated alignment research working is a developer's report on research done by its own models, and the public would be asked to trust assessments produced by the developer's models and checked by the developer (inference).

Chapter 70 collects, as one of its debates, whether weak overseers can supervise strong systems, and leaves to this chapter whether automated alignment research could extend oversight past the overseer's capability. The evidence does not decide it. The positive results concern tasks with benchmarks the researcher could not tamper with, and capability gaps small enough that the overseer could still check, as when a model aligned a checkpoint just above it on a capability index (Anthropic, 2026k). The negative results show that on fuzzy tasks neither people nor models check reliably, but they come from current models and from argument (Bowkis et al., 2026; Baig et al., 2026). As of October 2026 we found no study of the decisive case: an automated program assessing a model that its overseers cannot check, against a known answer.

55.9 Settled, contested, missing #

55.10 Exercises #

Further reading #

• Bowkis et al. (2026): why automated alignment research could produce misleading safety assessments even without sabotage, through hard-to-supervise fuzzy tasks.
• Gasteiger et al. (2025a): how a model prompted to sabotage research does it, and why weak monitors miss it.
• Anthropic (2026m): automated researchers on an alignment problem, including the ways they gamed their evaluator.
• Schroter et al. (2008): the planted-error design, applied to peer reviewers.
• Benton et al. (2024): evaluations of whether a model could undermine its developer's evaluations and decisions.
• van der Weij et al. (2025): strategic underperformance on evaluations, and why it undermines a capability claim.