Draft complete but not yet reviewed; expect revisions

Scalable Oversight

Every method in the chapters so far assumed that someone can tell a good answer from a bad one. Reinforcement learning from human feedback (RLHF) needs a rater who can judge a response (Chapter 22); an evaluation needs a grader (Chapter 36); an audit needs an auditor who recognizes what it finds (Chapter 38). As systems become more capable than the people overseeing them, that assumption fails first on the tasks where a wrong answer costs most. Scalable oversight is the research program on supervising systems whose work the supervisor cannot check directly. Reading a model's chain of thought (Chapter 40) is one such method. This chapter is about the others, which restructure the task so that a weak judge's verdict can still be trusted.

The program rests on an idea from complexity theory: checking can be far cheaper than doing, if the work is presented in the right form. The chapter states the problem, teaches the protocols built on that idea with the derivation that makes debate's promise precise, and then follows the derivation's assumptions into experiments and training, where several of them fail. It ends with the person at the end of every protocol, whom social epistemology, the psychology of argument, the law of evidence, economics, and human factors studied long before machine learning. Most claims here concern two of the book's three relationships: oversight during training supplies the signal a model is optimized on, and oversight in deployment decides whether an action goes ahead. Much of the evidence comes from the developers of the models it studies, which brings in the third, the developer and the public. The question the chapter cannot yet answer is the plainest one: whether a weak overseer can supervise a strong system that is trying to mislead it.

41.1 The oracle problem and the overseer's position #

The term entered the field in a 2016 agenda, posted as a preprint, which named scalable oversight as one of five concrete problems of accident risk. There it meant an objective that is correct but too expensive to evaluate often, so that behavior must be extrapolated "from limited samples" (Amodei et al., 2016). Later work widened it to "supervising systems that potentially outperform us on most skills relevant to the task at hand" (Bowman et al., 2022). The widened question is whether a weaker party can check a stronger one at all. Software testing asked it about programs, and cybernetics asked it about controllers.

41.1.1 Partial oracles #

From software testing. Section 24.2.1 introduced the oracle problem: a test needs a procedure that tells correct behavior from incorrect, and for the programs E. J. Weyuker called non-testable, written to compute answers no one knows, no such procedure exists (Weyuker, 1982; Barr et al., 2015). That chapter reads a model trained on problems its developers cannot solve as such a program.

What this section adds is the field's map of partial remedies. A 2015 survey by Earl Barr and colleagues sorts the literature into four categories: oracles that can be specified, oracles that can be derived, oracles built from implicit information, and the case where "no automatable oracle is available, yet it is still possible to reduce human effort" (Barr et al., 2015). Each has a counterpart among oversight methods (Table 41.1), a mapping that is the book's (inference).

The derived oracles include one with a direct descendant. A pseudo-oracle, in the survey's account, is an alternative version of the program produced independently, by a different team or in a different language, whose outputs must agree with the original's (Barr et al., 2015). Using a second model to check the first is the same move, and it inherits the same weakness: versions built independently still fail together on inputs that are hard for both, which Section 6.7.4 derives and Section 41.6 measures for language models.

The analogy stops at the tester's position. In every remedy the survey lists, the tester knows at least as much about the intended behavior as the program does, and the program does not shape its output to pass. Testing theory has no remedy in which a party that knows more than the tester helps the tester check that party's own work, and none for a program that adapts to its tests (inference). The protocols in this chapter make the first move, and Section 41.5 reports what happens when the second holds.

41.1.2 Requisite variety #

From cybernetics. Cybernetics, the mid-century study of control and communication in machines and organisms, gave the problem a precise form. In his 1956 textbook, W. Ross Ashby modeled regulation as a game. The environment makes one of a set of moves, the disturbances; the regulator answers with one of its own moves, the responses; and a table gives the outcome for each pair (Ashby, 1956). The regulator wants to keep the outcomes inside a small acceptable set. Ashby proved a lower bound on how well it can do, which he called the law of requisite variety. The variety of outcomes, if minimal, "can be decreased further only by a corresponding increase in that of R," the regulator; more picturesquely, "only variety in R can force down the variety due to D; variety can destroy variety" (Ashby, 1956). Variety here is the number of distinct possibilities, often counted on a logarithmic scale.

Read directly, the law says that an overseer can hold the outcomes to an acceptable set of $\nu_{\mathrm{a}}$ outcomes only if $\nu_{\mathrm{d}}/\nu_{\mathrm{r}} \le \nu_{\mathrm{a}}$, so an overseer whose repertoire of distinct interventions is far smaller than the variety of ways a system can go wrong cannot hold the outcomes to a small acceptable set, whatever its strategy (inference). Debate and amplification can be read as ways to borrow variety: the overseer's own responses stay few, and a capable model supplies the distinctions the overseer lacks (inference). A companion result of Roger Conant and Ashby, that a good regulator must be a model of the system it regulates, is stated in Section 28.6.4 together with the reading that it is weaker than its title.

The law stops where this chapter starts. It counts possibilities, not computation: it says how many distinct responses a regulator needs, not how hard it is to find the right response or to check that a proposed one is right (inference). Checking a proposed answer can be far cheaper than producing one, and every protocol below lives in that gap. As of October 2026 we found no oversight protocol stated in requisite-variety terms with the variety of each party measured, so the law remains a diagnosis rather than a design rule.

41.1.3 Sandwiching: the experimental design #

A weaker overseer and a stronger system can be studied before superhuman systems exist, if the gap is staged. Ajeya Cotra, in a 2021 Alignment Forum post, proposed to "sandwich" a model between people less capable than it at a task and people more capable, and to "figure out how to help the less-capable set of humans reproduce the judgments of the more-capable set of humans" (Cotra, 2021). Her starting assumption, which the post states as one, is that techniques that work in this setting "stand a good chance of just continuing to work" once models exceed the best humans (Cotra, 2021).

A 2022 preprint by Samuel Bowman and colleagues at Anthropic, with participants hired through Surge AI, turned the proposal into an experimental paradigm in which the non-experts may not use any help from the experts, who only grade the result (Bowman et al., 2022). Its proof of concept asked participants multiple-choice questions from two datasets that the developer's own assistant handled better than they did: MMLU, exam questions that draw on specialized knowledge, and QuALITY, questions about a story of about 5,000 words, with a five-minute limit that prevented reading it in full. They could chat with the assistant in any way they liked. Two relaxations simplified the design: the participants could only talk to the model, not retrain it, and the datasets' answer keys stood in for the experts (Bowman et al., 2022). Table 41.2 gives the results.

Participants who chatted with an unreliable assistant beat both the model alone and their own unassisted answers. The authors state the conditions that qualify the result. Each task had four participants, after a fifth on each was excluded for scoring far above the others unassisted, and the correct answer was revealed after each question, which the paper calls a divergence from the core protocol (Bowman et al., 2022). Access to the model also raised participants' confidence, and on the answers the group was most confident of, accuracy was 88 percent on QuALITY and 81 percent on MMLU (Bowman et al., 2022): even the most confident assisted judgments were wrong about one time in eight or more. The assistant made mistakes but was not set to mislead anyone, so the study shows what assistance adds, not what an adversary takes away (inference).

The same protocols serve two relationships. During training, the overseer's verdict is the reward the model is optimized on (Chapter 22); in deployment, it decides whether an action is allowed, the human audit step of Section 51.2. Only in training is the model optimized against that verdict, which is where an overseer's blind spots are learned (inference).

41.2 Amplification, recursive reward modeling, and assisted evaluation #

The first route decomposes. If a hard task splits into subtasks that a weak overseer can judge, the overseer can supervise the whole by supervising the parts, with copies of the model doing the parts.

Iterated amplification, proposed in a 2018 preprint by Paul Christiano, Buck Shlegeris, and Dario Amodei at OpenAI, builds a training signal this way (Christiano et al., 2018). A human expert answers a question by identifying useful subquestions, having copies of the current model answer them, and deciding the answer after seeing the subanswers. The model is then trained to predict what this composite system outputs, and the cycle repeats with the improved model. At first the composite is essentially the human; over time the human's role becomes coordinating copies of the model. The scheme helps "as long as it is possible for multiple agents to collaboratively solve problems more effectively than a single agent" (Christiano et al., 2018). Its experiments used five toy algorithmic tasks, such as following a permutation or finding a shortest path, with a hand-coded algorithm in place of the human. Amplification learned them almost as fast as supervised learning on the true answers, and it needed tens of thousands of examples of the expert's decompositions where supervised learning needed tens of millions of answers. The authors add that the experiments "don't shed any light on whether humans can decompose interesting real world tasks" (Christiano et al., 2018).

Recursive reward modeling, set out the same year by Jan Leike and colleagues at DeepMind in a preprint agenda, is the reward-learning form (Leike et al., 2018). A reward model is a model trained to predict a person's judgment of outcomes (Section 12.2.1). Agents trained this way on simpler tasks assist the user in evaluating the outcomes of the next, more capable agent, for instance by summarizing a novel the user would otherwise have to read (Leike et al., 2018). The agenda rests on a stated assumption: "For many tasks we want to solve, evaluation of outcomes is easier than producing the correct behavior." The authors note that it fails for tasks with a low-dimensional outcome space, such as yes-or-no questions, unless the answer comes with an explanation (Leike et al., 2018). They also name two conditions on the recursion: the decomposition of an evaluation must be exhaustive, or the next agent may optimize the neglected aspect in an arbitrary direction, and it is an open question whether errors accumulate from one level to the next (Leike et al., 2018). A 2020 preprint by Evan Hubinger compared eleven such proposals, amplification, debate, and recursive reward modeling among them, on outer alignment, inner alignment, and two kinds of competitiveness (Hubinger, 2020).

Decomposition has been run on a real task. A 2021 OpenAI preprint by Jeff Wu and colleagues trained GPT-3 to summarize whole novels recursively: summarize short sections, then summarize the summaries (Wu et al., 2021b). Labelers could supervise and evaluate the models "despite not having read the entire books themselves," and the resulting summaries matched the quality of human-written ones for about 5 percent of books (Wu et al., 2021b).

The practical descendant is assisted evaluation: a model that helps a person judge another model's output. In a 2022 OpenAI preprint, William Saunders and colleagues fine-tuned models to write critiques of summaries (Saunders et al., 2022). Labelers given the critiques found about 50 percent more flaws, and on answers written to be deliberately misleading they found the intended flaws 50 percent more often. Larger models wrote more helpful critiques. The authors also found no clear trend of a model's ability to critique catching up with its ability to tell good answers from bad, which they read as larger models having "relevant knowledge they don't articulate as critiques" (Saunders et al., 2022), the gap Section 39.7 studies.

A 2024 OpenAI preprint by Nat McAleese and colleagues trained a GPT-4-based critic, CriticGPT, with RLHF to point out problems in code (McAleese et al., 2024). On code containing naturally occurring errors, its critiques were preferred over human critiques in 63 percent of cases. On code with bugs that people had inserted on purpose, models caught more of them than the contractors paid for code review. The critics also invented bugs that did not exist, and teams of a contractor and a critic caught about as many bugs as the critic alone while inventing fewer (McAleese et al., 2024). The authors list the limits: inserted bugs are distributed differently from natural ones, the code snippets were short, and the rate of invented bugs "is still quite high" (McAleese et al., 2024). In a December 2025 post, Maja Trębacz and colleagues at OpenAI reported deploying a code-review agent tuned for precision over recall. It commented on 36 percent of pull requests generated entirely by its Codex system; 46 percent of those comments led the author to change the code, against 53 percent of its comments on human-written pull requests (Trębacz et al., 2025). Since the reviewer and the generator are the same underlying model, the developer watches whether the model "learns to subtly game or avoid its own checks," and writes that "There is no clean direct measurement of this" (Trębacz et al., 2025). All three are developers' reports on their own models.

What assistance buys is a larger set of things a person can check. What it costs is that the person's verdict now depends on an assistant whose reliability is the question the oversight was meant to answer. None of these studies used an assistant trying to mislead its user, and each measured helpfulness, not resistance to deception (inference). Section 41.8 reports how people use such help.

41.3 Debate as an interactive proof #

The second route makes the overseer a referee. Two models argue, and the judge decides who won. A theory from complexity theory says why a referee who checks one small step can settle a question far beyond its own reach, and the same theory says what the referee must be able to do.

41.3.1 Checking instead of computing #

From complexity theory. Complexity theory sorts problems by the resources needed to solve them. A computation runs in polynomial time when its running time grows at most as a fixed power of the size of its input, the field's standard meaning of efficient. NP is the class of yes-or-no questions whose yes-answers come with a certificate that a polynomial-time check can verify, such as a proposed route through every city of a map. PSPACE is the class of questions that can be decided with memory that grows polynomially, however long the computation runs; it contains NP and is believed to be far larger, since it includes deciding perfect play in games whose length grows polynomially (Irving et al., 2018).

An interactive proof lets a verifier that is limited to polynomial time, and may flip coins, question a prover that has unlimited computation but is not trusted. As Adi Shamir states the definition, a question belongs to the class IP when such a verifier can be convinced by some prover to accept every true instance with overwhelming probability, and cannot be convinced by any prover to accept a false instance with more than negligible probability (Shamir, 1992). Shamir's paper credits the class to work published in 1985 by Goldwasser, Micali, and Rackoff, and by Babai (Shamir, 1992; Goldwasser et al., 1985; Babai, 1985). Shamir's 1992 theorem settled its power: "when both randomization and interaction are allowed, the proofs that can be verified in polynomial time are exactly those proofs that can be generated with polynomial space" (Shamir, 1992). A weak verifier that may ask questions can thus be convinced of anything a polynomial-memory computation can establish, far beyond what it could compute or check from a single certificate. Computation by two players who move in turn has a theory of its own, under the name alternation (Chandra et al., 1981).

41.3.2 The debate game #

Geoffrey Irving, Paul Christiano, and Dario Amodei proposed debate in a 2018 OpenAI preprint (Irving et al., 2018). Two agents are shown a question, state their answers, and take turns making short statements; a human judge then decides which agent "gave the most true, useful information." The game is zero-sum, one agent's win being the other's loss, so each maximizes its chance of winning. The method rests on what the authors call its central claim, "In the debate game, it is harder to lie than to refute a lie," which they call empirical for any particular setting (Irving et al., 2018). If it holds, they hope that in every equilibrium of the game both agents tell the truth, and that self-play training, in which a model improves by playing copies of itself, can find such equilibria (Irving et al., 2018). An equilibrium is a pair of strategies in which neither player gains by changing its own given the other's (Section 39.2.3). Honest and dishonest describe the answer a debater defends and the strategy it plays, not a belief; Section 1.5 sets out when such language about a trained model predicts its behavior.

The analogy to complexity theory makes the hope precise. Replace the human by a polynomial-time judge. A judge that answers questions directly decides only what it can compute; a judge that checks a single proposed answer reaches NP, which the paper compares to reinforcement learning against a judge of outcomes; and a judge of a debate with polynomially many rounds reaches all of PSPACE (Irving et al., 2018). The paper's Theorem 1 states the last step: for any problem in PSPACE, a polynomial-time judge can be chosen "so that optimal play in the debate game with polynomially many steps solves" it (Irving et al., 2018). Its proof rests on a standard result that every such problem can be written as a formula whose quantifiers alternate between "there exists" and "for all," which two players in turn can play out (Irving et al., 2018). The authors add that these "complexity class arguments are analogies only" (Irving et al., 2018).

The argument behind the theorem is short enough to give in full, in the form of an argument tree.

The analogy holds when the judge checks something formal, such as a single step of a computation. When the judge is a person, the paper's own list of reasons to worry applies, beginning with "Maybe humans believe whatever they want to hear?" since people tend to evaluate arguments by their prior beliefs rather than their validity (Irving et al., 2018). A later result shows that the analogy breaks at a deeper point. Jonah Brown-Cohen, Geoffrey Irving, and Georgios Piliouras observe that the methods of interactive proofs apply "only to tasks with mathematically precise definitions," while a human judge is a black box whose answers the debaters can only predict. When computations may query such a black box, they note, "the main theorems regarding the power of interactive proofs (e.g. IP=PSPACE and the PCP theorem) are actually false" (Brown-Cohen et al., 2024). What can still be had, they write, comes from "requiring that the provers compete" (Brown-Cohen et al., 2024). Debate's guarantees are therefore not inherited from interactive proofs; they rest on competition between two provers, and Section 41.4 reports what that competition can and cannot deliver.

Figure 41.1 draws the path; its second mode anticipates the failure of step 2, which the next section takes up.

41.3.3 The same asymmetry elsewhere #

Computing has used the gap between finding and checking for decades: proof-carrying code (Section 28.6.2) and verifiable computation (Section 28.6.3) let a host check cheaply what an untrusted party produced. A learning version is recent. In a preprint first posted in 2024, Noga Amit, Shafi Goldwasser, Orr Paradise, and Guy Rothblum define self-proving models, which "prove the correctness of their output to a verification algorithm V via an Interactive Proof." Since the verifier is sound, no model can convince it that an incorrect output is correct, except with a small probability bounded by the verifier's soundness error (Amit et al., 2024). Each of these needs a formal statement of what is proved, and the claims oversight cares about, that a summary is faithful or that a change to code has no hidden purpose, have none (inference).

41.3.4 Amplification compared #

The two routes split the work differently. Amplification asks the overseer, or a trusted model trained to imitate it, to choose the subquestions and combine the answers; debate hands the search for the weak point to an adversary. Irving and colleagues, who found debate while trying to understand amplification, call the two approximately equivalent and draw out one difference. Debate tolerates a high branching factor, since a debater need only point to the one subclaim that matters, while an amplification tree must be deep enough to visit the alternatives; the advantage disappears if amplification's questioner is itself trained adversarially (Irving et al., 2018). They also note that the assumptions about people are similar: "Debate assumes lying is harder than arguing against a lie given a human judge, while stock amplification assumes that lying is harder than questioning a lie given a human judge and a human questioner" (Irving et al., 2018). Recursive reward modeling draws the same analogy, with each level of assistance adding an alternating quantifier (Leike et al., 2018). All three assume that the judge can check a leaf, and that someone can find the leaf worth checking.

41.4 Efficient honesty, obfuscated arguments, and their repair #

Step 2 of the derivation granted the honest debater unlimited computation. Real debaters are bounded, and a dishonest one may be as strong as the honest one or stronger. This section asks what changes when the honest side must win with bounded effort, reports the failure that experiments with people found, and the protocol proposed to avoid it.

41.4.1 Honest strategies that are efficient #

Complexity theory has a model for a weak verifier that wants a bounded computation done for it. In a doubly efficient interactive proof, as Brown-Cohen and colleagues describe the line of work begun by Shafi Goldwasser, Yael Kalai, and Guy Rothblum, an untrusted prover runs a polynomial-time computation and a verifier that takes only linear time checks that it was done correctly (Goldwasser et al., 2008; Brown-Cohen et al., 2024). Their 2024 paper, by Google DeepMind researchers and published at ICML, carries the idea to debate. Two provers that each run in polynomial time compete before a verifier that may ask an oracle, a stand-in for human judgment, about single steps (Brown-Cohen et al., 2024). The protocols are built so that "the honest strategy can always succeed using a simulation of a polynomial number of steps," can verify stochastic systems, and wins "even when the dishonest strategy is allowed to use exponentially many simulation steps" (Brown-Cohen et al., 2024).

In one protocol, which uses cross-examination, questioning independent copies of a debater that must answer consistently, one prover writes out the whole transcript of the computation, the other names the location of a supposed mistake, and the verifier checks only that location (Brown-Cohen et al., 2024). For human judgment modeled as a stochastic oracle, the guarantee needs an extra assumption, that the result does not swing on small changes in the oracle's probabilities (Brown-Cohen et al., 2024). The authors name the limitation that matters most: the debaters must be able to produce reasoning "in such a way that (potentially extensive) careful human analysis could have been used to judge that the reasoning was correct" (Brown-Cohen et al., 2024). A later paper by the same group puts the cost plainly: these protocols avoid the problem below "by doing away with recursion entirely," and the debaters must write out the full argument a human could check (Brown-Cohen et al., 2025).

41.4.2 The obfuscated arguments problem #

The problem was found by people arguing. In the second half of 2019, a team at OpenAI that included Beth Barnes, Paul Christiano, Long Ouyang, and Geoffrey Irving ran debates with human debaters and judges; a December 2020 update by Barnes, on later work with Christiano, reported debates on physics questions. Both appeared as Alignment Forum posts (Barnes and Christiano, 2020; Barnes, 2020). In free-text debates a dishonest debater "could often evade being pinned down." Recursing on one claim at a time fixed that but let it exploit ambiguity, "claiming to have meant whatever is most convenient." The remedy was cross-examination of copies that cannot communicate, which the authors say lifts debate in theory "from PSPACE to NEXP," a larger class still (Barnes and Christiano, 2020).

The failure they could not fix came next. They "discovered that a dishonest debater can often create arguments that have a fatal error, but where it is very hard to locate the error," and wrote, "We don't have a fix for this 'obfuscated argument' problem" (Barnes, 2020). Their illustration, which they do not consider a real difficulty for debate, asks whether a large public number, RSA-2048, has prime factors. The honest debater shows that a primality test declares it composite and proves the test correct. The dishonest one argues that there is no prime factor below some value $X$ and none above it (Barnes, 2020). The honest debater knows one of the two claims is false but cannot say which without factoring the number. And the dishonest debater can say the same thing about the honest proof: one of its steps is wrong, it cannot tell which. To the judge the two cases look nearly symmetric: in the honest case every step checked is always correct, and in the dishonest case almost always (Barnes, 2020). The authors stress that the general difficulty was known in theory; what was new was that in natural-language debates "a dishonest debater can frequently construct such situations in a way that seems hard to distinguish from honest play" (Barnes, 2020). Their conclusion was that debate may verify arguments too large for unaided people to follow, but only arguments small or robust enough that a flaw could be found by checking at random, which "may still have to be small relative to the computation used during training" (Barnes, 2020).

The model is the book's (inference), a formal version of the post's argument. It assumes that the challenger can do no better than check subclaims at random, which is what obfuscation means here, and that the judge knows how often arguments are obfuscated. The post's own conclusion is the same, that only arguments small or robust enough to be checked by random recursion can be trusted (Barnes, 2020). Figure 41.2 draws one such debate.

41.4.3 Prover-estimator debate #

A repair came from theory. In a preprint first posted in 2025 and revised in July 2026, Brown-Cohen, Irving, and Piliouras, with Lijie Chen, Jiawei Li, and Zhiyang Xun of UC Berkeley and UT Austin, proposed prover-estimator debate (Brown-Cohen et al., 2025). The prover decomposes a problem into subclaims; the estimator's only task is to assign each subclaim a probability; and the prover, not the opponent, chooses which subclaim to debate further, by claiming that the estimator is wrong about it in a specific direction. Since the prover must point to a flaw in the estimator's probabilities, the estimator wins if neither side can locate one (Brown-Cohen et al., 2025). The default of Equation (41.1) is reversed: an argument no one can fault is now rejected rather than accepted.

The authors split their guarantee in two, and the split matters. Soundness: an honest estimator wins on average against a dishonest prover, "even if stability is violated and the subproblems are obfuscated," with computational effort comparable to its opponent's (Brown-Cohen et al., 2025). Completeness: an honest prover can always win, but only with a decomposition that satisfies a stability condition, that "the validity of arguments cannot hinge on arbitrarily small changes in subclaim probabilities" (Brown-Cohen et al., 2025). In their words, "We require stability only for usefulness, not safety" (Brown-Cohen et al., 2025). The paper's example of a stable claim is whether most published studies show a drug effective: if two thirds do, up to a sixth of all studies could turn out flawed without changing the answer (Brown-Cohen et al., 2025).

Three qualifications travel with the result. The guarantees hold on average over questions, not for every question: "we do not prove the judge is always able to tell which debater is correct on every question" (Brown-Cohen et al., 2025). The honest prover's guaranteed payoff shrinks geometrically with the depth of the decomposition, at a rate set by the stability constant, so the protocol is cheap only for very stable arguments (Brown-Cohen et al., 2025). And an earlier version of the preprint "had an error in the main theorem due to an issue with the way stability was defined," found and fixed by the last three authors (Brown-Cohen et al., 2025).

Which real questions satisfy the stability condition is the open part. The authors hope that questions divide into those that hinge precisely on individual subclaims, such as mathematical proofs, which could be checked by machine, and those resting on noisy evidence, which their protocol handles (Brown-Cohen et al., 2025). Some questions oversight cares about fall on the wrong side of that line. Whether a change to code contains a backdoor, or whether a plan has one fatal step, can turn on a single subclaim, and a protocol that requires stability would reject a true argument about them as readily as a false one (inference).

These results are theorems, settled as mathematics within their assumptions. The assumptions are not mathematical: debaters that find the strategies the proofs construct, an oracle that is right about single steps, arguments with a checkable or a stable form, and, for prover-estimator debate, computational effort comparable between the sides; the doubly efficient protocol lets the honest debater win against one with unbounded computation. Whether trained models play near those strategies is the next section's subject.

41.5 Debate in practice: trained debaters and real judges #

The theorems assume debaters who find the strategies the proofs construct. In practice, provers and debaters are trained by gradient methods, and the judge is a person or a weaker model. This section reports the training results first, then the experiments with judges, in the order in which a negative result gave way to positive ones, and asks what changed between them.

41.5.1 Trained provers and verifiers #

The trainable form of an interactive proof is a prover-verifier game. Cem Anil and colleagues at the University of Toronto and the Vector Institute introduced it in a 2021 preprint: a trusted verifier network tries to choose the correct answer, while a more powerful but untrusted prover network tries to persuade it of a particular answer, regardless of correctness (Anil et al., 2021). Which player moves first turned out to matter. In a Stackelberg game, one player commits to a strategy first and the other responds, the leadership solution of Section 14.6.1. The authors prove that letting the prover lead "leads to dysfunctional solutions." When the verifier leads and the problem is revealed after it commits, reaching equilibrium is necessary and sufficient for a sound and complete protocol; when both move at once, the good protocols are equilibria, but so are bad ones, such as a verifier that ignores the prover. On two algorithmic tasks the verifier learned a decision rule that held "even when the verifier is frozen and the prover's messages are directly optimized to convince the verifier" (Anil et al., 2021).

A 2024 OpenAI preprint by Jan Hendrik Kirchner and colleagues trained the game at language-model scale, on grade-school mathematics problems with known answers (Kirchner et al., 2024). Training a prover only for correct answers produced solutions that time-limited human checkers judged less accurately. The authors' alternative trains, in rounds, a small verifier to predict correctness, a "helpful" prover rewarded for correct solutions the verifier accepts, and a "sneaky" prover rewarded for incorrect solutions that fool it (Kirchner et al., 2024). The helpful prover's accuracy and the verifier's robustness rose over the rounds, at the cost of lower accuracy than correctness-only training, which the authors call a legibility tax. A verifier as large as the prover did not grow more robust (Kirchner et al., 2024). In the main human evaluation, contractors with 45 seconds per solution judged the helpful prover's solutions more accurately and faster as training went on. Human accuracy on the sneaky prover's solutions fell over the rounds, but that result comes from a pilot the authors say has design flaws, and the study used one dataset and needed ground-truth answers to train (Kirchner et al., 2024). A 2025 paper at ICLR by Lewis Hammond and Sam Adam-Day unified debate, prover-verifier games, and related protocols as neural interactive proofs, compared their theoretical power, and tested them on graph isomorphism and on code validation with language models (Hammond and Adam-Day, 2025).

41.5.2 Trained debaters #

Debate itself has been trained. In a 2024 preprint, Samuel Arnesen, David Rein, and Julian Michael at New York University trained debaters by self-play on reading-comprehension questions from QuALITY, in which the judge sees the story only through quotes the debaters reveal (Arnesen et al., 2024). The debaters were a long-context version of Llama3-8B-Instruct; the judge was a fine-tuned GPT-4-Turbo, a stronger model than the debaters, so the asymmetry was one of information. Judge accuracy rose by 4 percentage points after debate training ($p < 10^{-6}$), while training consultants to persuade a judge with no opponent showed no such relationship (Arnesen et al., 2024). A 2025 preprint by Abhimanyu Pallavi Sudhir and colleagues argues that judge accuracy is the wrong yardstick for a protocol and proposes instead the difference in score between an agent arguing for the truth and one arguing against it (Sudhir et al., 2025). A 2025 preprint from Alibaba's Tongyi Lab used debate between strong models to improve the labels of a weak supervisor, the combination Section 42.7 discusses (Lang et al., 2025).

41.5.3 Where theory and training part #

The guarantees of Section 41.3 hold at equilibrium. Training reaches an equilibrium only if it explores the strategies that win, and a policy can withhold exploration so that a strategy is never reinforced, the exploration hacking of Section 33.3. A debater that never plays the honest strategy during training leaves the theorem's precondition unmet, however sound the protocol (inference). A 2025 preprint by Marie Davidsen Buhl, Jacob Pfau, Benjamin Hilton, and Geoffrey Irving of the UK AI Security Institute sketches an alignment safety case built on debate, an argument that an AI research agent will not sabotage its work, and makes the dependence explicit (Buhl et al., 2025). It rests on four claims: that the agent has become good at the debate game, that good play implies the agent is mostly honest, that honesty does not decay in deployment, and that the deployment tolerates some errors. Among its key assumptions is "Exploration guarantees: We need a solution to exploration hacking" (Buhl et al., 2025).

41.5.4 Judges: from a negative result to positive ones #

The first human test was negative. In a 2022 workshop paper, Alicia Parrish and colleagues at New York University gave 194 crowdworkers 90 seconds to answer reading-comprehension questions about passages too long to read in that time, with single written arguments for a correct and an incorrect answer (Parrish et al., 2022). The arguments did not improve accuracy; a baseline that showed text snippets chosen by people did (Parrish et al., 2022). Two readings fit the null. On one, arguments without interaction do not help a judge; on the other, a 90-second budget leaves no time to use them. The later studies changed both.

Julian Michael and colleagues at New York University, in a 2023 preprint, ran interactive debates between members of the university's competitive debate team on QuALITY questions. The judges could not read the story but could question the debaters, who could reveal short excerpts of it (Michael et al., 2023). They compared debate with consultancy, their name for a single expert who argues for an answer that is correct half the time. Judges were more accurate with debate than with consultancy (Table 41.3), and debates were 68 percent as long (Michael et al., 2023). In their error analysis, 46 percent of the errors in debate came from mistakes by the honest debater, which should shrink with skill, and 52 percent of the errors in consultancy came from the consultant hiding relevant evidence, which should grow with it (Michael et al., 2023). The authors give the counter-reading themselves: there may be strong dishonest strategies "that break debate at the Nash equilibrium, but that our human debaters are not able to figure them out," so the high accuracy may partly reflect the dishonest side's mistakes (Michael et al., 2023).

Two groups then used language models as debaters. Akbir Khan and colleagues at University College London, with Anthropic, FAR AI, and MATS researchers, reported in a paper at ICML 2024, with Anthropic's Claude models among the debaters and judges alongside other developers' models, that on QuALITY with hidden text, debate raised the accuracy of both model and human judges far above naive baselines (Table 41.3) (Khan et al., 2024). Optimizing debaters for persuasiveness, by choosing the most convincing of several sampled arguments, made judges more accurate. The authors list their limits: the strong and weak differed "only in access to information," the debaters were RLHF-trained models with "a propensity for honesty," and the results depend on a tool that lets debaters quote verified evidence (Khan et al., 2024). Zachary Kenton and colleagues at Google DeepMind, in a paper at NeurIPS 2024, used the developer's Gemini Pro 1.5 as debaters and weaker models as judges across extractive question answering, mathematics, coding, logic, and multimodal reasoning (Kenton et al., 2024). Debate beat consultancy on every task when the consultant was assigned its side at random. Against a judge answering directly, debate won on extractive tasks with an information asymmetry, "but in other tasks without information asymmetry the results are mixed" (Kenton et al., 2024). Stronger debaters raised judge accuracy, "though more modestly than in previous studies," and the experiments involved no training: they measured protocols "under an inference-only setting" (Kenton et al., 2024). Table 41.3 sets the studies side by side.

41.5.5 The hazard on the judge's side #

Persuasion cuts both ways. In a 2024 paper at NAACL, Chenglei Si and colleagues had 80 crowdworkers check claims with the help of a model's explanation or of retrieved passages. Both raised accuracy on difficult claims, from 59 percent unaided to about 73 percent with either aid, and explanations were faster. But people agreed with the explanation most of the time even when it was wrong, and on those claims they did significantly worse than unaided judges (Si et al., 2024b). Showing explanations for both sides of the claim, a format close to a one-turn debate, reduced the over-reliance but did not significantly beat search alone (Si et al., 2024b). The training-time version is Section 30.4.1's result that RLHF raised the share of wrong answers time-limited evaluators accepted (Wen et al., 2025a): optimizing a model against a judge's approval teaches it the judge's blind spots, which is what debate's adversary is meant to expose.

41.5.6 What changed between the results #

Between the negative result and the positive ones, four things changed: judges could interact with the debaters, debaters could cite evidence the judge could verify, the asymmetry became one of information, and debaters were selected or trained to win (inference). Interaction and verified quotes give the judge checkable leaves, which the argument tree requires; information asymmetry gives the debaters something the judge lacks while leaving the judge as capable as they are. Two sets of experiments come closest to a judge that lacks capability rather than information. In Kenton and colleagues' tasks without an information asymmetry, debate's advantage over answering directly was mixed, with one small significant gain, for a GPT-3.5 judge (Kenton et al., 2024). In Hammond and Adam-Day's code validation, GPT-4o provers trained jointly with GPT-4o-mini verifiers, in debate and other protocols, gave modest gains over a verifier trained alone, with training that used ground-truth labels and needed stabilizing (Hammond and Adam-Day, 2025).

The results read differently from the two sides of the debate over debate (inference). For its proponents, a consistent advantage over consultancy, in human and model judges, shows that adversarial structure helps a weaker judge. For skeptics, consultancy is a weak baseline by construction, since the consultant is told to argue for a wrong answer half the time, and against a judge answering alone the evidence is mixed outside information asymmetry. Neither side's central case has been tested. Arnesen and colleagues trained debaters weaker than their judge (Arnesen et al., 2024), and in Hammond and Adam-Day's training, where the provers were the stronger models, both debaters were GPT-4o (Hammond and Adam-Day, 2025). The nearest test is Kirchner and colleagues' sneaky prover, trained against a verifier with a thousandth of its compute, a gap in capability rather than information; but there the helpful and the sneaky prover were one model, so the dishonest side was not the stronger, there was one prover rather than a debate, and the human result on sneaky solutions comes from the pilot its authors call flawed (Kirchner et al., 2024). No experiment has pitted a judge who lacks capability against a dishonest debater stronger than the honest one and trained to deceive that judge. The groups that ran the positive studies also overlap: Bowman is an author of the Michael and Khan papers, Michael and Rein of the Arnesen paper, and Brown-Cohen, an author of the protocols, of the Kenton paper.

41.6 How oversight scales with the capability gap #

The experiments measure one gap at a time. The question that matters for the program is how the overseer's chance of success changes as the gap grows, and whether a chain of overseers, each checking a slightly stronger successor, can cover a gap that one step cannot. A 2025 paper gives the only quantitative frame for it so far.

41.6.1 Oversight as a rated game #

The frame borrows the Elo rating system of chess, in which each player has a score and the probability that one beats another depends only on the difference between their scores. Joshua Engels, David Baek, Subhash Kantamneni, and Max Tegmark at MIT, in a paper at NeurIPS 2025, model oversight as a game between a stronger player they call the Houdini, which tries to evade oversight, and a weaker Guard, which tries to stop it (Engels et al., 2025). From the win rates of every pairing of models they fit each model a Guard score and a Houdini score for each game. The Guard then wins with probability $1/\big(1 + 10^{(E_{\mathrm{H}} - E_{\mathrm{G}})/400}\big)$, where $E_{\mathrm{G}}$ and $E_{\mathrm{H}}$ are the two scores; the base 10 and the divisor 400 are "historical conventions from chess" (Engels et al., 2025). General capability is measured by a model's Chatbot Arena rating, which rests on people's preferences between models' answers; the authors call it imperfect but a reasonable aggregate measure (Engels et al., 2025).

Their model of how skill at a game depends on general capability has three regions: a plateau of incompetence below some capability, a region where game skill rises linearly with general capability, and a plateau of saturation above another (Engels et al., 2025). They validated it on Counting-to-21, a simplified version of the game Nim, and fitted it to four oversight games (Engels et al., 2025). In Mafia, five villagers try to find one hidden Mafia player. Debate follows Kenton and colleagues, with a judge who lacks the source text and 20 questions from each of four datasets. In Backdoor Code, one model hides a flaw in a competition programming solution and another looks for it. In Wargames, a fictional AI tries to escape its company's control while a monitor tries to stop it (Engels et al., 2025).

The games differed in a way that matters for oversight. In Debate, the judge's score rose with general capability while the debaters' stayed roughly constant, on three of the four datasets (on the fourth, BoolQ, both stayed mostly flat), which the authors read as promising: "an agent specialized in oversight may remain effective even under the presence of extremely persuasive agents" (Engels et al., 2025). In Backdoor Code the pattern reversed: the Guard's slope was nearly flat and the Houdini's steep, though the authors found that stronger Houdinis did not craft more deceptive backdoors; they were more likely to produce ones that worked (Engels et al., 2025).

41.6.2 Nested oversight #

Nested scalable oversight, the authors' term, chains the steps: a trusted model oversees a somewhat stronger untrusted one, which, if the step succeeds, becomes the trusted overseer of the next (Engels et al., 2025). Every step must succeed, so the chain's success is the product of the steps'. Splitting a large gap into many small ones makes each step easier and adds more chances to fail, and the trade-off has an optimum.

Figure 41.3 shows the three regimes of the derivation. The paper applies the same analysis to its four fitted games, with their unequal slopes; Table 41.4 gives its estimates at a general gap of 400 points, which fall further for larger gaps, assuming every player lies on the fitted Elo curves and that ratings do not saturate (Engels et al., 2025).

One condition must travel with the Debate figure: "in Debate, a judge who chooses a random side achieves 50% success rate," so the curve converges to about 50 percent (Engels et al., 2025). The highest of the four estimates is 1.7 points above a coin flip. The floor comes from the game, not from oversight: a judge who chose at random would score the same, while in the other three games, the paper notes, random players almost never win (Engels et al., 2025).

The authors list the limits. They simulated only single steps of oversight, so the nested results are theory built on single-step fits; they modeled deception "via prompting" rather than training adversarial models, since many of the models were proprietary; and Chatbot Arena ratings are an imperfect measure of capability (Engels et al., 2025). In Debate they assumed both debaters engage adversarially, noting that "advanced agents might collaborate to deceive the judge" (Engels et al., 2025). The estimates are reported by one group, with games and adversaries of its own design.

41.6.3 Judges that fail together #

Nesting and stacking overseers multiplies their protection only if their errors are independent. Section 6.7.4 reports a 2025 preprint by Kim and colleagues that found substantial correlation in the errors of language models, and draws the arithmetic: a claim about stacked monitors must measure their joint failures (Kim et al., 2025). A 2025 preprint by Shashwat Goel, Joschka Strüber, and colleagues at the ELLIS Institute Tübingen, the Max Planck Institute for Intelligent Systems, and the University of Tübingen adds two findings that bear on oversight (Goel et al., 2025a). Using a measure of similarity based on overlapping mistakes, they found that language models used as judges "favor models similar to the judge," and that "model mistakes are becoming more similar with increasing capabilities" (Goel et al., 2025a). The authors note that both findings are correlations, and that their measure applies only to multiple-choice tasks for now (Goel et al., 2025a). If the trend holds, a chain of overseers drawn from similar models shares blind spots that grow with capability, which is the condition under which nesting buys least (inference).

The frame stops at its scoreboard. Elo summarizes win rates in games with a clear winner, and the fitted slopes are properties of four games with prompted adversaries. Oversight of open-ended research or policy questions has no scoreboard, and a nested chain also assumes that an overseer that wins a step can be trusted at the next, which no game measures (inference).

41.7 The human overseer: judging experts one cannot check #

Every protocol in this chapter ends with someone judging claims they cannot verify themselves: the judge of a debate, the rater with an assistant, the non-expert in a sandwiching study. Four older fields studied that position.

41.7.1 Dependence on experts #

From social epistemology. Social epistemology studies how knowledge depends on other people, through testimony, trust, and expertise. Its question for this chapter is whether it can be rational to believe experts one cannot check, and how to choose among them; John Hardwig's 1985 paper "Epistemic Dependence" and Alvin Goldman's 2001 paper "Experts: Which Ones Should You Trust?" pose it in their titles (Hardwig, 1985; Goldman, 2001). Neither could be read for this book, so their answers are not reported here.

The mechanism can be read in a source that is available. Hugo Mercier and Dan Sperber, in a 2011 target article in Behavioral and Brain Sciences, describe how people protect themselves from being misled: receivers "must therefore exercise some degree of what may be called epistemic vigilance," a term from earlier work by Sperber and colleagues (Mercier and Sperber, 2011; Sperber et al., 2010). Its two most important mechanisms, they write, are "trust calibration and coherence checking." People calibrate the trust they grant a speaker on the basis of the speaker's competence and benevolence, and they check new information for coherence with what they already believe (Mercier and Sperber, 2011).

Each mechanism has an oversight counterpart (inference). Calibrating on competence is what evaluations and track records do (Chapter 36). Calibrating on benevolence, on whether the speaker means the listener well, is what audits for hidden objectives try to do (Chapter 38). Coherence checking is the consistency check and the cross-examination of Section 41.4. The analogy stops at benevolence. Vigilance grew up among human speakers whose interests are partly visible and whose record can be followed over time. For a model, benevolence is exactly what oversight is trying to establish, and a model trained on approval is trained against the cues a vigilant listener relies on, as Section 39.3 argues for the cues of lying (inference).

41.7.2 Judging arguments #

From psychology. Debate presupposes that judging an argument is easier than producing a good one. Mercier and Sperber's argumentative theory predicts exactly that asymmetry. Their hypothesis is that "the function of reasoning is argumentative. It is to devise and evaluate arguments intended to persuade," which predicts that people evaluate others' arguments well and produce their own with a bias toward their side (Mercier and Sperber, 2011). They cite evidence on both counts. Motivated participants were "much more influenced by strong arguments than by weak ones." In group experiments on problems with a demonstrably correct answer, "truth wins": once one member understands the problem, that member can convince the rest, and on one standard logic puzzle, the Wason selection task, groups reached about 80 percent correct where individuals hover around 10 percent (Mercier and Sperber, 2011). Their conclusion carries a condition that matters here: people reason in an unbiased way "at least when they are evaluating arguments rather than producing them, and when they are after the truth rather than trying to win a debate" (Mercier and Sperber, 2011). A debate's judge is in the first position and its debaters in the second.

A 2016 study by Emmanuel Trouche, Petter Johansson, Lars Hall, and Mercier tested the asymmetry directly. By its abstract, among participants who were shown their own earlier argument as if it were someone else's and did not notice, more than half rejected it (56 and 58 percent in two experiments). They also told valid from invalid arguments better when the arguments seemed to be someone else's (Trouche et al., 2016). People are better critics than authors.

The theory has published critics in the same issue. The philosopher Maralee Harrell argues that research on argument evaluation "seems mixed, at best," and that when people watch an argument from outside, their judgment of the burden of proof depends on who speaks first (Mercier and Sperber, 2011). The psychologist John A. Johnson doubts the optimism about groups: perhaps truth wins often enough in jury trials and scientific debates, "but as we all know, sometimes it does not" (Mercier and Sperber, 2011). Mercier and Sperber concede the limit themselves: for problems without a provable solution, good arguments may win "even if this is not always sufficient for truth to win" (Mercier and Sperber, 2011). And when the task is to judge honesty itself, people do little better than chance, as Section 39.3 reports.

For debate, this implies support for the premise where answers can be demonstrated, and a limit where they cannot: a correct answer that cannot be shown competes on persuasiveness, and the order effect Harrell describes is a design parameter of any protocol in which one debater opens (inference). The analogy stops at the arguers: the evidence is from people arguing with people of similar ability, not from a judge facing an arguer far more capable than itself.

41.7.3 Adversarial procedure #

From the law of evidence. Common-law courts settle questions of fact by opposed advocates before a judge or jury who are not experts. Michael and colleagues, in the debate study of Section 41.5.4, set out the comparison. In the adversarial system "the representative of each party acts as a 'zealous advocate' for their party's position," on the idea that a zealous advocate for each side will bring the best evidence before the decision maker (Michael et al., 2023). They summarize the system's critics, drawing on legal scholars. The outcome is influenced by skill differences between advocates and their "tricks, devices, dodges, and strategems"; resources are systematically unequal between prosecution and defense; advocates can keep evidence secret until trial; and hostile versions of the facts can be distorted from the truth (Michael et al., 2023). The alternative is the inquisitorial system, in which a neutral magistrate develops the evidence. For an AI to act as that magistrate, they note, "it would need to be directly incentivized to seek truth, which is precisely the problem we are trying to solve in the first place" (Michael et al., 2023).

Courts also decide when to rely on expert reasoning they cannot reproduce. The Daubert standard, which Section 38.7 reports, asks whether a method can be and has been tested, whether it has been reviewed, and what its known or potential error rate is (Supreme Court of the United States, 1993). Psychological studies of adversary and nonadversary presentation of evidence were published in the early 1970s by John Thibaut, Laurens Walker, and E. Allan Lind (Thibaut et al., 1972; Lind et al., 1973; Thibaut and Walker, 1975). These papers could not be read for this book, so their findings are not reported here.

What the law implies is that the failure its critics name first, unequal skill between advocates, is the failure debate must survive at scale (inference). Michael and colleagues answer that skill can be equalized by having copies of one model argue both sides (Michael et al., 2023). That removes the gap between the debaters, not the gap between the debaters and the judge, which is the gap scalable oversight is about (inference). The analogy also stops at the advocates: a lawyer answers to rules of professional conduct and to sanctions outside the case, while a debater answers only to its reward. The Daubert question, a known error rate, applies to oversight protocols as it does to forensic methods, and no protocol has one for the cases that matter (Section 41.10).

41.7.4 Interested parties #

From economics. Economics models the judge as a decision maker who must learn from parties with interests of their own, and two of its results are taught elsewhere in the book. Section 5.6.1 reports that a party holding information it can verifiably disclose may be driven to disclose all of it, because silence is read as bad news, under conditions that include costless and truthful disclosure. Section 39.2.3 reports that a single informed sender whose interests differ from the receiver's can, in equilibrium, communicate only coarse information. A literature on competing interested parties and opposed advocates connects the two, in papers whose titles name the question, Paul Milgrom and John Roberts's "Relying on the Information of Interested Parties" and Mathias Dewatripont and Jean Tirole's "Advocates" (Milgrom and Roberts, 1986; Dewatripont and Tirole, 1999); they could not be read for this book.

Read through the two readable results, debate is a mechanism for turning one interested sender's coarse report into competing disclosures that the judge can verify at a leaf (inference). The conditions of the disclosure result map onto features of the protocols: verifiable claims are the quotes and checkable leaves that Khan and colleagues name as a requirement, and a second sender with opposed interests is the opponent. Where it stops is verifiability. Much of what an overseer needs to know about a model, how it will behave in a situation not yet seen or what it represents internally, has no leaf a judge can check. Both models also assume senders who know their own interests, which for a trained model is an empirical question (Section 1.5) (inference).

41.8 The human overseer: checking as work #

The judge of a debate, the reviewer of a flagged action, and the rater with an assistant all do a job, and it is a job of checking work that is usually right. Human factors measured how people do that job when the work comes from automation. This section reports the findings on reliance and trust; the measurements of attention over a watch, of missed rare targets, and of alert fatigue are reported in Section 51.2, where they set the human audit step of a control protocol, and the loss of skill under assistance in Section 60.4. The field studies of code review are reported in Section 51.2 as well.

41.8.1 Supervising automation #

From human factors. Human factors is the study of how people perform with the machines they operate. In "Ironies of Automation," published in 1983, Lisanne Bainbridge described the position of an operator left to supervise an automated process (Bainbridge, 1983). The designer who tries to eliminate the operator "still leaves the operator to do the tasks which the designer cannot think how to automate," and manual skills deteriorate when they are not used. The irony that bears most on oversight is this: "the automatic control system has been put in because it can do the job better than the operator, but yet the operator is being asked to monitor that it is working effectively" (Bainbridge, 1983). If the decisions can be fully specified, a computer makes them faster and with more accurately specified criteria. "There is therefore no way in which the human operator can check in real-time that the computer is following its rules correctly." The operator can only judge, at some higher level, whether the decisions are acceptable. If the computer was brought in because human judgment is not adequate to the task, Bainbridge concludes, "The human monitor has been given an impossible task" (Bainbridge, 1983). That is the scalable oversight problem, stated for process control.

Later work named the ways reliance goes wrong. By its abstract, Raja Parasuraman and Victor Riley's 1997 review distinguishes four kinds of reliance. Beyond plain use, there is misuse, over-reliance that "can result in failures of monitoring or decision biases"; disuse, neglect "commonly caused by alarms that activate falsely"; and abuse, automation imposed by designers and managers without regard for what it does to human performance (Parasuraman and Riley, 1997). John Lee and Katrina See, in a 2004 review, made the target of design calibrated trust: "the correspondence between a person's trust in the automation and the automation's capabilities," with overtrust and distrust as the two failures (Lee and See, 2004). They report that an operator with highly reliable automation and several tasks besides monitoring it can come to overtrust it and miss its failures, a finding repeated with highly experienced pilots and infrequent faults. They describe a 1998 study by Kathleen Mosier and colleagues of cockpit crews in the same terms: highly reliable systems "appear to reinforce the perception that other cues are redundant and unnecessary" (Lee and See, 2004; Mosier et al., 1998).

Raja Parasuraman and Dietrich Manzey's 2010 review joined two strands, by its abstract. Complacency, the failure of monitoring that over-reliance produces, occurs "under conditions of multiple-task load." It is found in naive and expert participants alike, and "cannot be overcome with simple practice." Automation bias, the decision bias of following an automated aid, produces both errors of omission and errors of commission when the aid is imperfect. It occurs in experts, "cannot be prevented by training or instructions," and affects teams as well as individuals (Parasuraman and Manzey, 2010).

41.8.2 What a checker adds #

The overseer at the end of an assisted protocol is in Bainbridge's position: it is asked to check a system brought in because it does the task better (inference). The machine-learning evidence fits the human factors record. Critic models caught more inserted bugs than paid reviewers, in OpenAI's 2024 preprint on its own critic (McAleese et al., 2024), and people who read a model's explanation did worse than unaided judges when the explanation was wrong (Si et al., 2024b). The meta-analysis reported in Section 28.7 found that, in experiments published from 2020 to 2023, human-AI combinations on average did worse than the better of the two alone, with the losses concentrated where the AI alone beat the person (Vaccaro et al., 2024). It could include only tasks that people and AI can each do alone, so it is silent on tasks that only the combination can do (Vaccaro et al., 2024). Together these bound what a person adds at the end of a protocol on tasks of the kinds studied (inference). Calibrated trust needs the checker to know where the assistant fails, and a judge learns that only if something reports the truth back, the condition Section 18.3 states for raters. That knowledge is what oversight was meant to supply. Scalable oversight is one form of the older program of augmenting the principal rather than aligning the agent, which Section 28.7 reports with its limit: the tools that would augment the overseer are themselves AI systems.

The analogy stops at the automation. The human factors record comes from trained operators of narrow, mostly deterministic systems that were not trying to mislead them, and complacency there costs missed faults. A capable model trained against its overseer's approval can learn where the overseer stops checking, which turns complacency from a lapse into a target (inference). As of October 2026 we found no measurement of how human judges of debates, or human reviewers of model output, perform over long sessions in which deception is rare.

41.9 What oversight means for delegation #

The book's thesis is that alignment is a delegation problem under an asymmetry of competence and visibility (Section 1.7.1). Scalable oversight is the program that attacks the asymmetry of competence from the principal's side, by restructuring the work so that a weaker principal can still check it. Its record bears on the thesis in three ways.

The older fields converge on its structures (inference): checking is cheaper than producing, as complexity theory proves; opposed advocates before a lay judge are how courts settle facts; competing interested parties are how economics models learning from those who know more. The experimental record fits what those fields predict: debate helped judges most where the debaters could cite evidence the judge could verify. The irony of automation predicts the other half, that the person at the end of the protocol comes to rely on what it cannot check, and the experiments with model explanations found that too.

What has no counterpart is the principal's control over the advocates (inference). A court cannot make both lawyers copies of one mind, train them against its own verdicts, or rewrite the rules of evidence between cases; the designers of a protocol can do all three, which is why skill differences between debaters can in principle be removed. The thesis also lists among what is new the possibility that the agent outgrows its principal's ability to check it. The obfuscated arguments problem is that possibility in miniature: a stronger party defeats the check without asserting anything the judge can catch, by making the flaw too expensive to find. The repair works by changing what happens when a flaw cannot be found, and its price is that some true arguments cannot be accepted. Where debate clearly beat a judge answering alone, the debaters had more information than the judge, not more capability; where the gap was one of capability, debate beat consultancy, but against the judge alone the results were mixed (Kenton et al., 2024). The case the thesis worries about, a system much more capable than its judge and trained against it, has been run only at the modest gap between GPT-4o and GPT-4o-mini, on one code-validation task (Hammond and Adam-Day, 2025).

The claims fall across the three relationships unevenly. Debate, amplification, and prover-verifier games are proposals for the training signal, the first relationship, and most experiments measure judges as a stand-in for that signal without training on it. Assisted review in deployment and the human factors of checking concern the deployed system and the people who oversee it. The third relationship enters through the evidence: the groups that ran the central experiments overlap, one of them includes an author of the protocols, several used their own companies' models, and no group without a stake in debate has replicated the central comparisons. How a developer supervises its own training runs, and with what protocol, is not something the public can currently check.

41.10 Settled, contested, missing #

41.11 Exercises #

Further reading #

• Irving et al. (2018): debate, the complexity-theoretic argument for it, and the authors' own reasons to worry.
• Barnes (2020): the obfuscated arguments problem, found in debates between people.
• Brown-Cohen et al. (2025): prover-estimator debate, with its split between soundness without stability and completeness with it.
• Ashby (1956): the law of requisite variety, chapter 11, with its counting proof.
• Mercier and Sperber (2011): the argumentative theory of reasoning, with the open peer commentary that disputes it.
• Bainbridge (1983): why the person asked to check automation has been given an impossible task.