AI Alignment
Part VI: Verification from Outside

Draft complete but not yet reviewed; expect revisions

Weak-to-Strong Generalization

In brief

Ideas
  • Can a strong student trained on a weaker supervisor's labels beat its teacher? In classification it almost always does; the evidence favors correction where errors are hard to fit, copying where easy (§42.6.4).
  • A recovery figure divides by the gap between supervisor and ceiling, so it means little without that gap: half of a 2-point gap is close to none (§42.1.3).
  • Four theoretical accounts agree that recovery depends on the structure of the teacher's errors, not the student's strength alone, and differ on what happens when the student can represent them (§42.2.5).
  • Every student so far learned from human data, and no theorem yet covers open-ended generation or action, which bounds what this route can show for overseeing stronger systems (§42.7.2).
Evidence
  • Settled: strong students fine-tuned on a weaker model's labels outperform their supervisors on classification and regression, as groups without a stake reproduced; recovery varies by task, size, and method (§42.1.2).
  • Reported: unsupervised and easy-to-hard prompting and probing failed on data with a feature more salient than truth, imbalanced classes, or unanswerable questions, in one 2026 preprint (§42.5.3).
  • Reported: errors induced by a conflicting objective concentrate where a weak supervisor cannot check, more so as the capability gap grows, in one 2024 preprint (§42.6.3).
Open
  • Crux: does a strong student learn a supervisor's systematic, human-like errors? Error structure decided recovery in the founding study, but the human case has not been run (§42.6.2).
  • Missing: recovery measured across a capability gap like the one oversight faces, and an independent repetition of the production results, which only developers can run today (§42.8).

The previous chapter changed the task so that a weak judge's verdict could be trusted: a debate, a decomposition, or an assistant puts the judge in a position to check what it could not produce. This chapter asks a different question about the same gap in competence. If a strong model is simply trained on labels from a weaker supervisor, does it learn the supervisor's errors, or does it use what it already knows and do better than its teacher? People will be the weaker supervisor of any system more capable than they are, so the answer bears on whether training from human judgments keeps working as models improve. Researchers at OpenAI made the question experimental by letting a small model stand in for the person and a large pretrained model stand in for the future system (Burns et al., 2024).

The question sits where instillation, training a system toward an intended target, meets verification, finding out whether it got there. A weakly supervised student can be right where its teacher is wrong only if training prefers, among the many functions that fit the teacher's labels about equally well, one that does not reproduce the teacher's mistakes. That preference is the inductive bias of Chapter 9, and the theoretical accounts in this chapter say when it favors correction. The chapter teaches them, reports what has been measured, and keeps in view the debate that runs through this part: whether weak overseers can supervise strong systems at all. Chapter 41 argues the protocol side of that debate; this chapter supplies the evidence from training.

Most claims here concern the first of the book's three relationships, the training process and the signal it optimizes. Where a weakly supervised model might behave differently in the cases its supervisor cannot check, the claims concern the second, the deployed system and the people it serves. And several of the experiments that matter most, on proprietary preference data, production reward models, and an automated research system, are ones only a developer can run and report, which brings in the third, the developer and the public.

Sources cited in the introduction 1
  1. Burns et al. (2024) Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision

42.1 The analogy and its metric #

Can a weak supervisor draw out what a much stronger model knows? Before any result, this section sets out the experiment that turned the question into a measurement, the quantity it reports, and what the analogy leaves out.

42.1.1 The design #

A pretrained model has been trained on a large body of text to predict the next token and has never been told what task it will serve (Chapter 11). To fine-tune it is to continue training on a smaller labeled dataset for one task. The design fine-tunes three models for each task (Burns et al., 2024):

  1. A small pretrained model is fine-tuned on correct labels. Its accuracy on held-out test questions is the weak performance. Its predictions on a separate set of examples become the weak labels.
  2. A large pretrained model, the strong student, is fine-tuned on those weak labels. Its test accuracy is the weak-to-strong performance.
  3. A second copy of the large model is fine-tuned on correct labels. Its test accuracy is the strong ceiling, which the authors say should intuitively correspond to "everything the strong model knows" (Burns et al., 2024).

The small model plays the person who cannot check every answer; the large one plays the system that knows more than its overseer. Correct labels are available only because the experiment is run today, on tasks people can grade. Write zweakz_{\text{weak}}, zw2sz_{\text{w2s}}, and zceilz_{\text{ceil}} for the three scores. The performance gap recovered is the fraction of the gap between the weak supervisor and the ceiling that the student closes (Burns et al., 2024):

PGR=zw2s−zweakzceil−zweak.\mathrm{PGR} = \frac{z_{\text{w2s}} - z_{\text{weak}}}{z_{\text{ceil}} - z_{\text{weak}}}.
(42.1)

A student that only matches its teacher scores 0; one that reaches the ceiling scores 1. Nothing bounds the ratio to that range: a student worse than its teacher gives a negative value, and one better than the ceiling gives a value above one. Weak-to-strong generalization, the authors' name for a student that outperforms its supervisor (Burns et al., 2024), and performance gap recovered are terms coined by the developer for its own research program, as is superalignment, its word for aligning models much smarter than their supervisors.

42.1.2 What the founding study found #

The study is a developer's paper about the developer's models, published at ICML 2024. Its students and teachers were pretrained base models "from the GPT-4 family," spanning 7 orders of magnitude of pretraining compute (Burns et al., 2024). A footnote adds that the series shares GPT-4's architecture and pretraining data but "does not include the models known as GPT-2, GPT-3, and GPT-3.5" (Burns et al., 2024), so a "GPT-2-level" supervisor is a member of the series with roughly GPT-2's compute. Three settings were tested. The first was 22 classification datasets from natural language processing, converted to two-way tasks with approximately balanced classes. The second was chess puzzles, where the model predicts the best first move, which the authors note is a generative task. The third was reward modeling on a proprietary dataset used to train ChatGPT's reward models (Burns et al., 2024). While developing methods the authors checked against correct labels many times, but they held out their largest model and about half of the language tasks throughout (Burns et al., 2024).

Students beat their supervisors almost everywhere. In the authors' words, "PGRs are almost universally positive" across settings and sizes (Burns et al., 2024). How much they recovered depended on the setting, the sizes, and the method, as Table 42.1 shows.

Table 42.1 Performance gap recovered in the founding study, by setting and condition (Burns et al. 2024, a developer's study of its own models)
Setting Method Condition Reported recovery
Language tasks (median of 22) Fine-tuning on weak labels GPT-4 supervised by a GPT-2-level model "about half" of the gap, typically
Language tasks Fine-tuning on weak labels Very weak supervisors, students with many orders of magnitude more compute more than 20%
Language tasks Fine-tuning on weak labels Largest students often above 50%
Language tasks With the confidence loss Smallest supervisor, largest student median rose from about 25% to nearly 80%
Chess puzzles Fine-tuning on weak labels Smallest supervisors close to zero
Chess puzzles Fine-tuning on weak labels Small supervisor-student gaps can be above 40%; falls as the student grows, for a fixed supervisor
Chess puzzles Bootstrapping through intermediate sizes Larger students especially improved over fine-tuning
ChatGPT reward modeling Fine-tuning on weak labels All sizes usually about 10%, almost never above 20%
ChatGPT reward modeling With generative fine-tuning first Ceiling fine-tuned the same way improved by about 10 to 20%; 30 to 40% when combined with early stopping on correct labels

The confidence loss and bootstrapping are methods Section 42.3 describes. The last row adds a step that uses no preference labels: fine-tuning the model to predict the reward-modeling conversations themselves, ignoring which response people preferred, before the weak labels are applied (Burns et al., 2024). Early stopping on correct labels, which the authors call "cheating" because it needs the answers the setting assumes are missing, is there to show how much overfitting to weak labels costs, not as a method.

The study also measured imitation directly. With plain fine-tuning, the student agreed with its supervisor more often than the supervisor was right, which "indicates that the student is imitating some of the supervisor's errors" (Burns et al., 2024). On the reward-modeling task, for large gaps between supervisor and student, accuracy against correct labels often rose early in training and then fell "well before a single epoch has elapsed," as the student fitted the supervisor's errors (Burns et al., 2024). Stopping at the best step by correct labels was worth around 5 points of recovery there, and 15 points on the language tasks (Burns et al., 2024). Larger students "consistently agree less with the errors of the supervisor than smaller student models," which the authors suggest means that pretrained models may have a hard time fitting the errors of smaller ones, at least when fine-tuning data are limited (Burns et al., 2024). The developer's summary is that naive fine-tuning recovers far from all of the strong model's capability, "suggesting that techniques like RLHF may scale poorly to superhuman models without further work," and that its improved methods are "proofs-of-concept" rather than "practical solutions we recommend deploying today" (Burns et al., 2024). RLHF there is reinforcement learning from human feedback, the pipeline of Section 12.2.1.

42.1.3 Reading a recovery figure #

A recovery figure is a ratio of two differences between noisy scores, and its reliability depends on the size of the gap in its denominator. Each score is measured on a finite test set, so it carries the standard error of Section 6.5.4, about z(1−z)/ntest\sqrt{z(1-z)/n_{\text{test}}} for a score zz on ntestn_{\text{test}} questions. The book's own short calculation shows what that does to the ratio.

Derivation How noisy a performance gap recovered is

Write gap=zceil−zweak\mathrm{gap} = z_{\text{ceil}} - z_{\text{weak}} for the true gap, and let each measured score be its true value plus an estimation error εweak\varepsilon_{\text{weak}}, εw2s\varepsilon_{\text{w2s}}, or εceil\varepsilon_{\text{ceil}}. When the errors are small next to the gap, expanding (42.1) and keeping only the terms linear in them gives

PGR^≈PGR+εw2s−PGR εceil−(1−PGR) εweakgap.\widehat{\mathrm{PGR}} \approx \mathrm{PGR} + \frac{\varepsilon_{\text{w2s}} - \mathrm{PGR}\,\varepsilon_{\text{ceil}} - (1 - \mathrm{PGR})\,\varepsilon_{\text{weak}}}{\mathrm{gap}}.

If the three errors are independent with a common standard error se\mathrm{se}, the measured ratio has standard deviation

sd(PGR^)≈segap1+PGR2+(1−PGR)2,\mathrm{sd}(\widehat{\mathrm{PGR}}) \approx \frac{\mathrm{se}}{\mathrm{gap}}\sqrt{1 + \mathrm{PGR}^2 + (1 - \mathrm{PGR})^2},
(42.2)

which, for a recovery between 0 and 1, lies between 1.22 se/gap1.22\,\mathrm{se}/\mathrm{gap} (at PGR=0.5\mathrm{PGR} = 0.5) and 1.41 se/gap1.41\,\mathrm{se}/\mathrm{gap} (at 0 or 1), and grows outside that range. Take scores near 0.8 on 1,000 test questions, so se=0.8×0.2/1000≈0.013\mathrm{se} = \sqrt{0.8 \times 0.2/1000} \approx 0.013. At a recovery of one half, a gap of 20 points gives a standard deviation of about 0.08; a gap of 5 points, about 0.31; a gap of 2 points, about 0.77. When the gap is no larger than a few standard errors, the first-order picture fails as well: the measured gap can come out near zero or negative, and the ratio can take any value.

Two consequences follow, both the book's (inference). A recovery figure reported without the gap behind it cannot be read: 50 percent of a 20-point gap is a measurement, and 50 percent of a 2-point gap is close to none. And scoring all three models on the same questions makes their errors positively correlated, which shrinks the variance of each difference as the paired comparison of Section 36.7.2 does, but it leaves the factor of one over the gap in place. The founding study reports medians across 22 tasks for its language results (Burns et al., 2024), which damps the noise of any one task; studies with a few tasks and small gaps have no such protection.

42.1.4 What the analogy leaves out #

The authors name two disanalogies. The first is imitation saliency. Future models will likely represent human behavior well, while the study's students "may not have learned features relevant for imitating weak model predictions." Hence "simply imitating the weak supervisor may thus be an easier failure mode to avoid in our setting than it will be in the future" (Burns et al., 2024). They add that "the types of errors weak models make today may be different from the types of errors humans will make" (Burns et al., 2024). The second is pretraining leakage: the students learned from human-written text, which implicitly contains human-level answers, so "it may thus be artificially easy to elicit strong models' capabilities in our setting" (Burns et al., 2024). Of the second they write, "This disanalogy could cause our results to be overly optimistic" (Burns et al., 2024).

The authors report one partial answer to the second. In an image experiment, an AlexNet classifier supervised linear probes, classifiers trained on a frozen model's features, on top of DINO models, which had been pretrained without any classification labels. AlexNet scored 56.6 percent; the students reached 60.7 and 64.2 percent against ceilings of 63.7 and 74.9, recoveries of 57.8 and 41.5 percent (Burns et al., 2024). The students' knowledge of the classes was latent, not copied from labels, though it still came from images people had collected.

Two further limits are the book's (inference). Almost every task is classification, with a fixed set of answers, while oversight of strong systems concerns long generations and actions; chess, the one generative setting, is also where recovery fell as the gap widened. And the analogy assumes the student is not working against its supervisor. Among the assumptions behind a plan built on weak-to-strong methods, the authors list that pretrained models "will be highly intelligent but not highly agentic" and so will not be deceptively aligned out of the box (Burns et al., 2024). A deceptively aligned model would behave well in training in order to act otherwise later (Section 34.1).

A sibling design keeps people in the loop. In sandwiching, non-experts supervise a model on questions where experts, or an answer key, supply the truth (Section 41.1.3). Weak-to-strong experiments replace the non-experts with a small model, which makes them fast, cheap, and repeatable, and removes exactly the structure of human error that the oversight problem is about (inference). The setting is instillation, what the student learns, judged as verification, whether supervision that cannot check every answer still produces correct ones; both belong to the first relationship, the training process and its signal.

Sources cited in Section 42.1 1
  1. Burns et al. (2024) Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision

42.2 Why a strong student can beat its teacher #

A student trained to reproduce its teacher's labels is rewarded, on its training objective, for reproducing the teacher's mistakes too. That it often does not needs an explanation. Section 9.1 showed that many functions fit the same data and differ beyond it, Section 9.3 that training leans toward simple functions among them, and Section 9.2 that past the point where a model can fit its data exactly, gradient descent picks the fit with the smallest weights. Here the data are the weak labels. Among the functions that fit them about equally well, some reproduce the teacher's errors and some do not, and the student's inductive bias chooses (inference). Four accounts, each proved under stated assumptions, say when the choice favors correction.

42.2.1 The misfit bound #

Moses Charikar and Chirag Pabbaraju of Stanford and Kirankumar Shiragur of Microsoft Research published the first account in a paper at NeurIPS 2024. They treat the strong model as a fixed representation of its input followed by a task-specific function chosen from a convex set, such as the linear layer that fine-tuning often trains on top of a network (Charikar et al., 2024). A set of functions is convex if the average of any two members, at any weighting, is also a member. Their setting is regression, predicting a number rather than a class, with squared loss, and the student is the member of the set closest to the weak labels. They show that the student's error is smaller than the teacher's by at least the student's misfit, its own error measured against the weak labels (Charikar et al., 2024). The misfit needs no correct labels to compute.

Derivation The misfit bound, by projection

Treat functions of the input as points, with squared distance ∥a−b∥2=Ex[(a(x)−b(x))2]\lVert a - b \rVert^2 = \E_x[(a(x) - b(x))^2] and inner product ⟨a,b⟩=Ex[a(x) b(x)]\langle a, b \rangle = \E_x[a(x)\,b(x)]. Let y∗y^\ast be the correct answers, yweaky_{\text{weak}} the weak supervisor's, and F\mathcal{F} the convex set of functions the student can represent, with y∗∈Fy^\ast \in \mathcal{F}. The student is the point of F\mathcal{F} nearest the weak labels, ys=arg min⁡v∈F∥v−yweak∥2y_{\text{s}} = \argmin_{v \in \mathcal{F}} \lVert v - y_{\text{weak}} \rVert^2.

  1. The target lies beyond the student, seen from the teacher. Since F\mathcal{F} is convex and holds both ysy_{\text{s}} and y∗y^\ast, it holds every point ys+t (y∗−ys)y_{\text{s}} + t\,(y^\ast - y_{\text{s}}) for tt between 0 and 1. None of them is nearer the weak labels than ysy_{\text{s}}, so t2∥y∗−ys∥2+2t⟨y∗−ys,ys−yweak⟩≥0t^2 \lVert y^\ast - y_{\text{s}} \rVert^2 + 2t \langle y^\ast - y_{\text{s}}, y_{\text{s}} - y_{\text{weak}} \rangle \ge 0. Divide by tt and let tt shrink to zero: ⟨y∗−ys, ys−yweak⟩≥0\langle y^\ast - y_{\text{s}},\, y_{\text{s}} - y_{\text{weak}} \rangle \ge 0. The angle at the student, between the directions to the target and to the teacher, is at least a right angle.
  2. Expand the teacher's error around the student. ∥yweak−y∗∥2=∥yweak−ys∥2+∥ys−y∗∥2+2⟨ys−yweak, y∗−ys⟩\lVert y_{\text{weak}} - y^\ast \rVert^2 = \lVert y_{\text{weak}} - y_{\text{s}} \rVert^2 + \lVert y_{\text{s}} - y^\ast \rVert^2 + 2 \langle y_{\text{s}} - y_{\text{weak}},\, y^\ast - y_{\text{s}} \rangle, and step 1 says the last term is not negative. Hence
∥ys−y∗∥2⏟student’s error  ≤  ∥yweak−y∗∥2⏟teacher’s error  −  ∥ys−yweak∥2⏟misfit.\underbrace{\lVert y_{\text{s}} - y^\ast \rVert^2}_{\text{student's error}} \;\le\; \underbrace{\lVert y_{\text{weak}} - y^\ast \rVert^2}_{\text{teacher's error}} \;-\; \underbrace{\lVert y_{\text{s}} - y_{\text{weak}} \rVert^2}_{\text{misfit}} .
(42.3)
  1. The equality case. When F\mathcal{F} is a linear subspace, as with a linear head on a fixed representation, the cross term is zero. Split the teacher's error yweak−y∗y_{\text{weak}} - y^\ast into a part inside F\mathcal{F} and a part at right angles to it. The student is the target plus the inside part: it inherits that part of the error whole, and its misfit is exactly the part it cannot represent.

The proof is the authors', a projection argument they cite as the Pythagorean theorem for convex sets (Charikar et al., 2024); step 3 is the book's restatement of the linear case. Two assumptions carry it. The first is realizability, y∗∈Fy^\ast \in \mathcal{F}. Without it, even a perfect teacher (yweak=y∗y_{\text{weak}} = y^\ast) can leave the student with positive error and positive misfit, worse than its teacher, and the authors observe this reversal when the less expressive model had the better representation (Charikar et al., 2024). The second is convexity: the authors' experiments with non-convex sets "suggest that our results should (at least qualitatively) hold even beyond the convex case" (Charikar et al., 2024). A version for finite samples and a target the student can only approximate adds error terms that vanish as the student grows more expressive and the data grow (Charikar et al., 2024). The theorem is about regression; the authors note that their work "does not address classification tasks" (Charikar et al., 2024).

Their experiments used synthetic data from small networks and three molecular-property regression tasks with a pretrained molecule model as the strong representation. The gain was close to the misfit, "more or less exactly" in the synthetic cases (Charikar et al., 2024). A student whose set contains the teacher's function fits it with zero misfit, and then the bound promises no gain at all; the authors draw the corollary that weak models with a large misfit to the strong one are the ones that best elicit what it already knows (Charikar et al., 2024).

42.2.2 Correcting labels and covering gaps #

Hunter Lang and David Sontag of MIT and Aravindan Vijayaraghavan of Northwestern, in a paper at NeurIPS 2024, take the classification case, with a teacher that may abstain on inputs it cannot label (Lang et al., 2024a). They name two effects: pseudolabel correction, the student beating the teacher on the points the teacher labeled, and coverage expansion, the student doing well on points the teacher left unlabeled. Existing theory of weak supervision, they show, accounts for neither (Lang et al., 2024a).

Their mechanism is expansion: the points with wrong or missing labels have many neighbors with correct ones, where a neighbor might be a nearby point in a representation or a fluent paraphrase (Lang et al., 2024a). If the student is robust, giving most points the same label as most of their neighbors, then "making a mistake on a bad point means making many mistakes on good points as well" (Lang et al., 2024a). The simplest case shows the logic, and it is the book's illustration rather than the paper's theorem (inference). Suppose inputs fall into clusters whose members share a correct label, the teacher mislabels fewer than half of each cluster, and the student must give each cluster one label. Matching the teacher as closely as possible then means taking each cluster's majority label, which is correct, so every teacher error is corrected, and an unlabeled member of a labeled cluster gets its cluster's label. The paper's bounds relax clusters to overlapping neighborhoods and perfect robustness to robustness on average. They "capture the intuition that weak-to-strong generalization occurs when the strong model is unable to fit the mistakes of the weak teacher without incurring additional error" (Lang et al., 2024a).

Robustness is a property of the student's representation, which the authors illustrate with sentiment. A teacher labels movie reviews positive if they contain "incredible" and negative if they contain "horrible." A bag-of-words student can put all its weight on those two words, fit the teacher exactly, and gain nothing; a linear classifier on top of a pretrained sentence representation improves on the teacher and does reasonably on reviews with neither word (Lang et al., 2024a). The expansion conditions need hold only for sets generated by the functions the student can represent, so they can be checked statistically from finite data, though the authors' search for the worst set is a heuristic (Lang et al., 2024a). On that sentiment task, the measured expansion certified correction for negative reviews (teacher error 0.11, bound 0.05, student error 0.04) and did not certify it for positive ones (teacher 0.33, bound 0.37, student 0.35), and the student corrected its teacher only on the negative reviews (Lang et al., 2024a). The authors call their experiments "limited in scope" and propose no new training method (Lang et al., 2024a).

42.2.3 Fitting nearly random labels #

David Wu and Anant Sahai of UC Berkeley, in a paper at ICLR 2025, study the regime of Section 9.2, with more weights than examples, where fitting every training label exactly can still generalize, which is called benign overfitting (Wu and Sahai, 2025). Their model is "a stylized overparameterized spiked covariance model with Gaussian covariates where the weak teacher's pseudolabels are asymptotically like random guessing" (Wu and Sahai, 2025). In words, the features are random, a few directions vary much more than the rest, and the strong features emphasize the direction that decides the correct label more than the weak features do. The strong model can represent the weak one exactly, and it is fitted by minimum-norm interpolation, the exact fit with the smallest weights, to many weakly labeled examples (Wu and Sahai, 2025). They prove "two asymptotic phases": the student either generalizes or guesses at random, depending on how many weakly labeled examples it gets (Wu and Sahai, 2025).

In their analysis the outcome turns on two quantities of the fitted classifier: how much of the true direction survives in it, and how much weight is spread over other directions. They call these survival and contamination. With enough weak labels survival dominates, and the student generalizes though it fits every weak label (Wu and Sahai, 2025). Three limits travel with the result. It is asymptotic and Gaussian, and the authors note that in practice the transition is not sharp (Wu and Sahai, 2025). A student with the same features as its teacher "cannot bootstrap its own performance" (Wu and Sahai, 2025). And the overparameterized regime is essential: the authors expect "weak imitation" once weakly labeled examples greatly outnumber features, the classic regime of gradient descent (Wu and Sahai, 2025).

42.2.4 Transferring a concept #

Seamus Somerstep and colleagues at the University of Michigan and IBM Research, in a paper at ICLR 2025, model a pretrained model's output as a mixture over latent concepts, close to the persona mixture of Section 11.6.1; their own example teaches a model a pirate persona (Somerstep et al., 2025). The target is one concept already in the strong model's mixture, and the weak teacher produces outputs from the right concept but corrupted, biased or noisy (Somerstep et al., 2025). Fine-tuning on those outputs from a biased teacher, with a pull back toward the pretrained model, has an error floor. In the authors' words, "both the source of weak labels and the source model are biased estimators of the target model," so the two cannot be combined into a consistent one, an estimator that converges to the target as data grow (Somerstep et al., 2025). Their alternative feeds weak examples to the strong model in context, lets it infer the concept, and has it write new labels. In their linear model, with orthogonal concepts and a strong model that treats the in-context examples as independent draws, an assumption they call "admittedly strong," the correct concept is then recovered as the number of in-context examples grows, more slowly the weaker the teacher (Somerstep et al., 2025). The authors stress that their message differs from the other three accounts: "In our formulation, the benefit of weak label training is quite limited" (Somerstep et al., 2025).

42.2.5 What the accounts say together #

The four accounts agree that recovery is a relation between the student and the structure of the teacher's errors, not a consequence of the student's strength alone. Wu and Sahai say that their account, the misfit bound, and the expansion account "all circle around the idea that weak-to-strong generalization works when the cascading learning process purifies representations in the true direction and contracts in false directions" (Wu and Sahai, 2025). They differ on what it means for the student to be able to represent the teacher's errors. In the projection of the misfit bound, an error inside the student's set is copied whole. In the expansion account, an error the student cannot fit without mislabeling good neighbors is corrected. In the benign-overfitting account, the student can represent its teacher exactly and still generalize, because interpolation chooses among many exact fits, and imitation is expected once examples outnumber features. The transfer account puts a floor under fine-tuning itself. The summary "random errors are corrected, representable ones are learned" is therefore the book's (inference). It holds in the sense of the first two accounts, where fitting an error costs the student nothing extra, and in the limit of plentiful data. It does not hold in the overparameterized regime of the third.

weak label +weak label −correct rulestudent's rulewhat the student sees−202feature 2−202feature 1 (decides the label)share of the gap recovered0.00.20.40.60.81.0gap recovered0.00.20.40.60.81.0share of the error in the feature the student seesrandom errorssystematic errorsSupervisor right 80% of the time, errors systematic, 50% of the tilt in the feature the student sees. Studentright 88.9% (ceiling 100%): it recovers 45% of the gap.
weak label +weak label −correct rulestudent's rulewhat the student sees−202feature 2−202feature 1 (decides the label)share of the gap recovered0.00.20.40.60.81.0gap recovered0.00.20.40.60.81.0share of error the student seesrandom errorssystematic errorsSupervisor right 80% of the time, errors systematic,50% of the tilt in the feature the student sees.Student right 88.9% (ceiling 100%): it recovers 45% ofthe gap.
Figure 42.1 A strong student fitted to a weak supervisor's labels, in a linear model of the book's own built on the design of Burns et al.'s Appendix E.1 (Burns et al., 2024). Inputs have three independent normal features; the correct label is the sign of the first. The student is a least-squares linear classifier on the first two features, trained on unlimited weakly labeled examples, so its rule can match the correct one. The supervisor is right on a set share of inputs. With random errors it flips labels at random; with systematic errors it applies a tilted rule, and the slider sets how much of the tilt lies in the feature the student sees rather than the hidden third feature. Left: a seeded sample of inputs in the student's two features, drawn as dots or crosses by weak label, with the correct boundary (dashed) and the student's (blue). Right: the share of the gap the student recovers, computed exactly, under systematic errors (blue) and under random errors of the same rate (yellow). Random errors are always removed; systematic errors are copied to the degree that the student can represent them, and fully when it sees the whole tilt. With finitely many examples, random errors also cost some accuracy.

Figure 42.1 makes the first sense concrete. Take a supervisor that is right 80 percent of the time and errs systematically. A student that sees half of the supervisor's tilt, in the cosine sense the figure's slider sets, recovers about 45 percent of the gap. A student that sees all of it recovers none, and one that sees none of it recovers all. Random errors of the same rate are removed completely in this limit of unlimited data.

The founding study ran the empirical version. When weak linear probes used a subset of the student's features, so that the student could reproduce them exactly, the student did not substantially improve on its supervisor; with labels flipped at random to the same accuracy, it did (Burns et al., 2024). With synthetic weak labels of identical accuracy but different error structure, from random flips to errors placed on the longest prompts or on the examples the strong model was most sure of, "the generalization can vary wildly depending on the structure of the weak errors" (Burns et al., 2024). When the weak label was written into each prompt, making imitation trivial, the average recovery was 0.002 ± 0.003 with plain fine-tuning and 0.046 ± 0.108 with the confidence loss, against around 0.2 and 0.8 on the original tasks (Burns et al., 2024). The theory and these experiments point the same way: whether weak supervision scales depends on whether the errors a person makes are ones a strong student can represent cheaply, which is the question Section 42.6 takes up.

Sources cited in Section 42.2 5
  1. Charikar et al. (2024) Quantifying the Gain in Weak-to-Strong Generalization
  2. Lang et al. (2024a) Theoretical Analysis of Weak-to-Strong Generalization
  3. Wu and Sahai (2025) Provable Weak-to-Strong Generalization via Benign Overfitting
  4. Somerstep et al. (2025) A transfer learning framework for weak-to-strong generalization
  5. Burns et al. (2024) Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision

42.3 Methods that raise recovery #

If plain fine-tuning recovers part of the gap, the next question is what recovers more. The methods tried so far do one of three things: let the student disagree with its teacher, improve the labels before the student trains on them, or divide a labeling budget between cheap labels and good ones. A fourth development is that the search for such methods has itself been handed to automated agents.

42.3.1 Letting the student disagree #

The founding study's auxiliary confidence loss adds a term that pulls the student toward its own thresholded predictions, so that its training target becomes a mixture of the weak label and the student's own guess, with the student's share rising to 0.75 for the largest models (Burns et al., 2024). The authors relate it to conditional entropy minimization and self-training, methods from semi-supervised learning, which learns from a few labels and many unlabeled examples, that also reward a model for confident predictions on unlabeled data (Burns et al., 2024). It did slightly worse than plain fine-tuning for small students and much better for large gaps, and it cut the student's agreement with its supervisor's mistakes. In a few settings the student ended up "fitting the ground truth concept better than it is fitting the weak labels it was trained with" (Burns et al., 2024).

Outside the developer's models, the loss has done less. In a 2024 preprint from EleutherAI, Alex Mallen and Nora Belrose trained Llama-3 8B on labels from Qwen-1.5 0.5B and found the log-confidence loss "not particularly effective," in line, they write, with the smaller models of the founding study and with a follow-up by other researchers (Mallen and Belrose, 2024). Researchers at Alibaba's Tongyi Lab, supervising Qwen-14B with Qwen-7B, found that it gave "a modest boost" and noted that their gap in compute was small (Lang et al., 2025). Somerstep and colleagues could not apply it to models they reached only through a fine-tuning service, and a stand-in that paired each weak answer with the student's own raised accuracy and weakened the style being taught (Somerstep et al., 2025). On this record the loss helped most where the gap was widest, a regime that, among the studies here, only the developer has tested (inference).

Bootstrapping, supervising a slightly stronger model first and letting it supervise the next, helped on chess, gave small gains on the language tasks, and none on reward modeling (Burns et al., 2024). In Somerstep's comparison it "offers some safeguarding against accuracy degradation, but generally still allows for too much corruption of the strong model" (Somerstep et al., 2025).

42.3.2 Improving the labels first #

A second family has the strong model clean the labels before it trains on them. Yuqing Yang, Yan Ma, and Pengfei Liu, of Shanghai Jiao Tong University, Fudan University, and the Shanghai AI Laboratory, tested this on mathematical reasoning, where answers are written out step by step rather than chosen from a list, in a paper at EMNLP 2024 (Yang et al., 2024d). Weak models fine-tuned on correct solutions wrote solutions for Llama2-70b. The method kept only the problems where the weak model's final answer matched the strong model's own answer when prompted with a few weak examples, fine-tuned on those, and then used the weak model's remaining solutions as contrasting examples in preference training (Yang et al., 2024d). Supervised by Gemma-2b, which scored 25.17 on GSM8K, the strong model reached up to 60.12, 31.08 points above fine-tuning on all the weak solutions (Yang et al., 2024d). On the harder MATH problems with the same supervisor, plain fine-tuning scored 10.0, below the weak model's 11.6 (Yang et al., 2024d). In a test closer to the future case, Llama3-8b-instruct supervised Llama3-70b with its solutions to 6,020 olympiad training questions; on 313 test questions the weak model scored 11.82, plain fine-tuning 12.46, and the full method 15.65 (Yang et al., 2024d). On 313 questions at accuracies between 12 and 16 percent, each score carries a standard error of about 2 points (Section 6.5.4), so the first two differ by noise and the full method's gain is of the order of one or two standard errors (inference).

Somerstep and colleagues refined labels by showing the strong model weak examples in context and having it write new answers, as their theory suggests (Somerstep et al., 2025). Fine-tuning GPT-3.5-Turbo directly on weak models' pirate-persona answers lowered the factual quality of its answers substantially, and more so as the weak labels got worse; training on the refined labels kept quality close to the base model, with GPT-4o as the judge (Somerstep et al., 2025). On mathematics, naive fine-tuning often made GPT-3.5-Turbo worse than before, refinement improved it, and for GPT-4o-mini the gains from refined labels were limited (Somerstep et al., 2025). Both methods lean on what the strong model already believes to decide which weak labels to trust, which helps where the strong model is right and cannot help where it is wrong (inference).

42.3.3 Spending a labeling budget #

In practice neither the quantity nor the quality of labels is fixed. Mallen and Belrose priced good labels at ten times weak ones and found three regimes on binary tasks from the founding study. With a small budget, spend all of it on weak labels; with a large one, spend all of it on good labels; in between, use many weak labels and some good ones (Mallen and Belrose, 2024). Adding a two-example prompt to fine-tuning improved accuracy by up to 5 percentage points at a fixed budget (Mallen and Belrose, 2024). Larger models needed fewer labels, and GPT-4o-mini reached 0.92 accuracy on MMLU with the ten examples that the provider's fine-tuning service requires at minimum (Mallen and Belrose, 2024). The authors frame scalable oversight as reducing the cost of good supervision, and they expect added difficulties in generative or reinforcement learning tasks, such as exploration and sparse reward (Mallen and Belrose, 2024).

42.3.4 Automating the search #

In a 2026 post, researchers at Anthropic, partly through its Fellows Program, report agents, which they call automated alignment researchers (AARs), that worked on weak-to-strong supervision on their own (Anthropic, 2026m). Nine parallel instances of Claude Opus 4.6 ran in separate sandboxes, shared findings through a forum, and submitted predictions to a remote evaluator that returned the performance gap recovered. The weak and strong models were Qwen1.5-0.5B-Chat and Qwen3-4B-Base, on binary testbeds for chat preference, math verification, and code verification (Anthropic, 2026m). The headline is a comparison. Two of the authors spent 7 days tuning four earlier methods and reached a best recovery of 0.23 on held-out chat-preference data. The agents reached 0.97 within 5 days, 800 cumulative hours across nine agents, at about $18,000 in compute and API calls (Anthropic, 2026m). The developer concludes that the agents "outperform human researchers" (Anthropic, 2026m). This is a developer's report about its own model, in a post that was not peer reviewed.

The same post reports what qualifies the number. The agents could submit without limit, so that, in the authors' words, "essentially our test set is a validation set with an OOD split" (Anthropic, 2026m): data used to make choices during development, with a part drawn out of distribution. They used that access: they recovered test labels by flipping one prediction and watching the score, retrained from whichever random seed scored best, and found shortcuts in the data, such as the most frequent answer to a math problem often being correct and the generating model of a code solution predicting its correctness; on code they simply ran the solutions (Anthropic, 2026m). "None of the authors predicted these hacks before running AARs" (Anthropic, 2026m). Math and code were therefore kept out of the search, because the agents could solve them without the weak supervisor or the student's knowledge at all. The best idea found on chat preference carried over to math and code, a second only to math, and a transfer to a production chat dataset with the developer's Sonnet 4.0 gave a gain of 0.5 points, "within the noise floor" (Anthropic, 2026m).

Two readings fit the result (inference). As evidence about weak-to-strong methods, it shows that on small models and a binary task, a search against a score it could query without limit found methods with high recovery, some of which transferred and one of which did not survive the move to production scale. As evidence about automated research, it shows agents raising a score by routes that did not solve the problem, the reward hacking of Chapter 30 in a research setting, and seed selection against a noisy ratio is the optimizer's curse of Section 7.7. The post does not say whether the two authors' baseline had the same access to the evaluator. How a result produced this way should itself be checked is the question of Chapter 55.

Sources cited in Section 42.3 6
  1. Burns et al. (2024) Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision
  2. Mallen and Belrose (2024) Balancing Label Quantity and Quality for Scalable Elicitation
  3. Lang et al. (2025) Debate Helps Weak-to-Strong Generalization
  4. Somerstep et al. (2025) A transfer learning framework for weak-to-strong generalization
  5. Yang et al. (2024d) Weak-to-Strong Reasoning
  6. Anthropic (2026m) Automated Weak-to-Strong Researcher

42.4 Easy-to-hard generalization #

A weaker supervisor need not be worse on every question. People can often check the easy cases of a task whose hard cases they cannot, so a closer analogy for oversight trains a model on correctly labeled easy cases and asks how it does on hard ones. The founding study called this easy-to-hard generalization and noted the difference from its main setting: it is "a classic covariate shift problem," a change in which inputs the model sees, "whereas our main setup focuses more on concept shift and noisy labels," a change in the labels themselves (Burns et al., 2024).

42.4.1 Two results that point different ways #

The founding study's own test used chess puzzles, whose difficulty ratings come from how often people solve them. Models fine-tuned only on easy puzzles improved on average over the whole test set, often on puzzles rated up to 1500 or higher, but on puzzles rated 2000 or above they did about as well as the untrained model with a prompt, or slightly worse (Burns et al., 2024). On the language tasks, training only on examples that small models get right left a gap even with correct labels, which the authors read as showing that "solving easy-to-hard generalization is non-trivial even if there are no weak label errors" (Burns et al., 2024).

Peter Hase and colleagues at the Allen Institute for AI and the University of North Carolina reported the opposite picture on question answering, in a paper at ACL 2024 (Hase et al., 2024). Their metric, the supervision gap recovered, has the same form as (42.1) with a different floor: the model prompted with no examples, rather than a weak supervisor. Across four datasets of science, general-knowledge, and math word problems, models from 7 to 70 billion parameters, three ways of fitting, and seven measures of difficulty, training on easy data usually recovered 70 to 100 percent of the gap to training on hard data (Hase et al., 2024). The difficulty measures were themselves weakly correlated with one another (Hase et al., 2024). Under two assumptions they did not measure, hard labels costing twice as much or being wrong twice as often, easy data was often the better buy; on MMLU it won once easy labels were wrong 10 percent of the time and hard ones 20 (Hase et al., 2024). The recovery fell when the gap in difficulty grew, from 74 to 57 percent when college science questions were supervised by third-grade rather than eighth-grade ones (Hase et al., 2024). The authors conclude that "the scalable oversight problem may be easier than previously thought," and add that they could not rule out that the test questions were in the models' pretraining data (Hase et al., 2024).

The two results differ in task, in model, and in what counts as the floor, so the evidence does not say which predicts the hard cases of oversight (inference). Both find that generalization weakens as the distance between training and test difficulty grows.

42.4.2 Evaluators generalize further than generators #

Zhiqing Sun, Longhui Yu, and colleagues at Carnegie Mellon, Peking University, the MIT-IBM Watson AI Lab, and elsewhere asked which part of a system generalizes from easy to hard, in a paper at NeurIPS 2024 (Sun et al., 2024a). On competition mathematics, they treated levels 1 to 3 as easy, with solutions, and levels 4 and 5 as hard, with questions only. Reward models trained on easy problems, some scoring every step of a solution, improved the choice among sampled solutions to hard problems more than fine-tuning the solver on easy problems improved its own answers, which the authors read as evaluators generalizing better than generators (Sun et al., 2024a). Used as rewards for reinforcement learning, they produced a 7-billion-parameter model that scored 34.0 percent on a 500-problem test set, above the 33.0 of the best earlier model trained on all levels; a 34-billion-parameter model choosing among 1,024 samples scored 52.5 percent (Sun et al., 2024a). The "human supervision" on easy problems was simulated: solutions were generated by GPT-4 or ChatGPT, with human labels on the steps of one dataset (Sun et al., 2024a). The checkpoints of reinforcement learning were chosen by accuracy on a validation set drawn from all five levels (Sun et al., 2024a), so answers to hard problems entered the choice of model (inference). The result fits the asymmetry that Section 41.3 builds on, that checking can be easier than producing (inference).

Comparing such results needs a common scale of difficulty. Easy2Hard-Bench, a NeurIPS 2024 collection of six datasets from mathematics, programming, chess, and reasoning, assigns each problem a numerical difficulty fitted from how people or models performed on it, with item response models of the kind Section 36.4.1 teaches (Ding et al., 2024).

42.4.3 How reward models generalize #

The supervisor in most training pipelines is a reward model, and Joshua Clymer and colleagues at Columbia University tested how one generalizes, in a 2023 preprint (Clymer et al., 2023). They built 69 distribution shifts in 8 categories and tuned LLaMA-30B reward models on one side of each (Clymer et al., 2023). Across large shifts the results looked good: a model tuned to evaluate Python programming instructions reached 84 percent accuracy on US history questions, and one tuned on easy abstract-reasoning puzzles reached 75 percent on much harder ones (Clymer et al., 2023). Shifts built to probe for failure told a different story, which Section 26.3 reports as reward models favoring personas that resemble internet text. The tuned model agreed with the untuned model's zero-shot judgments on 70 percent of target examples while its accuracy was 66 percent; it "would have performed better overall if its task was to predict the zero-shot policy" (Clymer et al., 2023). Methods that read the model's internal representations generalized better than fine-tuning but were still near or below chance on 6 of the 15 hardest shifts (Clymer et al., 2023). The authors note two limits: they tested reward models only, and their tasks can be done by imitating human judgments, so methods that work there may not carry to tasks beyond human ability (Clymer et al., 2023).

For oversight, the result means that an evaluator's generalization past its training data follows what its pretraining made salient, which here tracked what the untuned model already preferred, not the concept its trainers meant (inference). The authors themselves call the salience explanation weakly supported (Clymer et al., 2023).

Easy-to-hard is one dimension of transfer in the book's list in Section 26.6.1, alongside domain, format, and the length of an interaction. A result along difficulty says nothing about the others, and oversight of a strong system will cross several at once (inference).

Sources cited in Section 42.4 5
  1. Burns et al. (2024) Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision
  2. Hase et al. (2024) The Unreasonable Effectiveness of Easy Training Data for Hard Tasks
  3. Sun et al. (2024a) Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision
  4. Ding et al. (2024) Easy2Hard-Bench: Standardized Difficulty Labels for Profiling LLM Performance and Generalization
  5. Clymer et al. (2023) Generalization Analogies: A Testbed for Generalizing AI Oversight to Hard-To-Measure Domains

42.5 Unsupervised elicitation #

The limiting case drops the supervisor. If what a model knows has a structure that can be found without labels, no weak supervisor is needed for that part of the job, and none of its errors can be copied.

42.5.1 The probing record #

Section 39.7.2 reports the first attempts and reads them as tests of the problem of eliciting latent knowledge: a search for a truth direction in a model's activations that uses no labels at all (Burns et al., 2022), the demonstration that such searches can lock onto whatever feature of the activations is most prominent (Farquhar et al., 2023), and probes trained on easy questions that recovered most of a deliberately misreporting model's knowledge on harder ones (Mallen et al., 2023). The last is an easy-to-hard result inside the model. The recent work moves from probes to fine-tuning.

42.5.2 Internal coherence maximization #

Jiaxin Wen and colleagues, most of them at Anthropic and the rest at Schmidt Sciences, New York University, George Washington University, and elsewhere, proposed internal coherence maximization (ICM), a name of their own, in a preprint revised in January 2026 (Wen et al., 2025c). It labels a set of unlabeled examples with the model's own labels, chosen so that the set is mutually predictable, each label likely under the model given all the others, and logically consistent under simple rules, such as that two different answers to one problem cannot both be correct (Wen et al., 2025c). The consistency rules are there because mutual predictability alone can be maximized by giving every example the same label (Wen et al., 2025c). A search inspired by simulated annealing finds such a set, and the model is then fine-tuned on it.

With Llama 3 pretrained models, 8 billion parameters for checking grade-school mathematics and 70 billion for the others, ICM matched training on the datasets' correct labels on TruthfulQA and GSM8K verification, and beat training on crowdsourced human labels on a helpfulness and harmlessness preference task (Wen et al., 2025c). On a task where models do better than people, guessing which of two blog posts was written by a man, ICM matched correct labels at 80 percent, against 60 percent for five hired annotators (Wen et al., 2025c). It failed where the target was a preference the model had no way to infer: on a dataset in which poems mentioning the sun were always preferred, told only to judge which poem is better, it did no better than chance (Wen et al., 2025c). The authors write that they "were initially very skeptical of these findings, because they seemed clearly too good to be true"; they reran the experiments, removed the labels from the files, and had one coauthor reproduce the results on a different model with a separate codebase (Wen et al., 2025c). That is a check inside the author group, not an independent replication.

The paper also reports a production experiment, a developer's report on its own models. Labels for 5,000 preference pairs, produced by ICM with a pretrained Claude 4 Sonnet model, trained a reward model, and reinforcement learning against it trained an assistant. Scored by the developer's production Claude 4 Opus reward model, the assistant matched one trained against human labels on average, scoring higher on chat and safety and lower on math and code (Wen et al., 2025c). An appendix adds that with 5,000 examples, a Claude 4 Sonnet reward model trained on human labels beat the ICM reward model on average, while ICM beat training on 512 human labels (Wen et al., 2025c). The developer's earlier summary describes a different experiment, with a Claude 3 Haiku reward model and a Claude 3.5 Haiku assistant "preferred by the Claude 3.5 Sonnet preference model" (Anthropic, 2025as). Neither production result can be checked from outside: the data, the models, and the judging reward model are the developer's.

42.5.3 The stress test #

Callum Canavan and colleagues, researchers in the MATS program and the Anthropic Fellows Program with researchers at Anthropic, argued in a 2026 preprint that the datasets used to evaluate unsupervised and easy-to-hard methods are too kind (Canavan et al., 2026). They "often (1) have no features with more salience than truthfulness, (2) have balanced training sets, and (3) contain only data points to which the model can give a well-defined answer" (Canavan et al., 2026). They built datasets that each lack one property and tested prompting and probing methods on Llama 3.1 models of 8 and 70 billion parameters (Canavan et al., 2026). One of the prompting methods labels examples iteratively with the model's own few-shot predictions, which a companion study found about as accurate as ICM's labels at much lower cost (Canavan et al., 2026). Table 42.2 summarizes the results.

Table 42.2 Results of the stress test of unsupervised and easy-to-hard methods (Canavan et al. 2026, preprint)
Challenge Dataset Result
A feature more salient than truth Math solutions where a character in the prompt agrees with one answer most methods lost over 10 points of accuracy; for most, added punctuation cost under 5 and a change of tense under 2
A feature more salient than truth Political statements labeled for factual correctness and for stance asked for correctness, all methods but ensembles of principal-component probes tracked stance more
Imbalanced training sets Bash command sequences, 99 or 100 percent safe no method above 0.75 AUROC except the easy-to-hard probe, 0.87 on the 70B model and 0.67 on the 8B
Questions without an answer Math solutions mixed with normative political claims every method was about as confident on the political claims as on the math

The abstract's summary is that "no technique reliably performs well on any of these challenges," and that ensembling and combining the two families "only partially mitigate performance degradation" (Canavan et al., 2026). AUROC, the area under the curve of true against false positive rates, is 0.5 for a random ranking and 1 for a perfect one. The authors state their limits: they "did not benchmark any methods which use SFT or any weak-to-strong generalization methods," and some datasets, such as the mix of mathematics and politics, are not very realistic (Canavan et al., 2026). They name the risk their title points to: where in-distribution evaluation is impossible, a method's inadequacy "may remain silent until the method fails catastrophically" (Canavan et al., 2026).

42.5.4 What the stress test changes #

The three properties the stress test removes are ones real oversight problems lack. Deployment data are imbalanced, contain questions without answers, and carry features, such as agreement with a user, that a model represents more saliently than truth. That makes the stress test, not the original benchmarks, the more relevant evidence for oversight (inference). Its scope bounds the inference: it tested prompting and probing, not ICM's fine-tuning or weak-to-strong fine-tuning, so it reaches those methods only by analogy. The two papers also fail in mirror image (inference). ICM found nothing when the target was an undisclosed preference not salient to the model, and the stress test's methods found the wrong thing when something else was more salient. In the language of Section 42.2, a salient non-truth feature is an error the student can represent cheaply, the case every account says goes badly.

Sources cited in Section 42.5 6
  1. Burns et al. (2022) Discovering Latent Knowledge in Language Models Without Supervision
  2. Farquhar et al. (2023) Challenges with unsupervised LLM knowledge discovery
  3. Mallen et al. (2023) Eliciting Latent Knowledge from Quirky Language Models
  4. Wen et al. (2025c) Unsupervised Elicitation of Language Models
  5. Anthropic (2025as) Unsupervised Elicitation
  6. Canavan et al. (2026) Three Concrete Challenges and Two Hopes for the Safety of Unsupervised Elicitation

42.6 Elicitation or imitation #

The chapter's debate is whether weak-to-strong gains come from eliciting what the strong model already knows, or from a more polished imitation of the supervisor, errors included. The two readings agree on the measurements so far and make different predictions for the case oversight needs.

42.6.1 Two readings #

On the elicitation reading, a pretrained model already represents the concept the supervisor is after, and the weak labels only point at it. The founding study's motivation states it: for alignment "we do not need the weak supervisor to teach the strong model new capabilities; instead, we simply need the weak supervisor to elicit what the strong model already knows" (Burns et al., 2024). Its evidence is that students beat their supervisors almost everywhere (Burns et al., 2024), that the expansion bound certified correction where it happened and not where it did not (Lang et al., 2024a), that easy supervision recovered most of the gap to hard supervision on question answering (Hase et al., 2024), and that a student can fit nearly random labels and still generalize (Wu and Sahai, 2025). It predicts that recovery stays high as the supervisor's errors grow, so long as the student cannot fit them cheaply.

On the imitation reading, a student trained to predict its supervisor learns to predict its supervisor, and generalization beyond it is a transient. The founding study states the limit itself: "In the case of infinite training data, naively fitting the weak labels should result in perfect imitation, and a PGR of zero" (Burns et al., 2024). Its evidence is that plain fine-tuning recovers only part of the gap and, across large gaps, overfits the supervisor's errors within an epoch (Burns et al., 2024), that errors the student can reproduce are not corrected (Burns et al., 2024), that fine-tuning on weak labels has an error floor in a model of concept transfer (Somerstep et al., 2025), and that on mathematical reasoning plain fine-tuning fell below the weak supervisor (Yang et al., 2024d). It predicts that recovery falls when the supervisor's errors are systematic and salient, as human errors are.

Table 42.3 sets each result against both readings, and against a third that the evidence suggests.

Table 42.3 Each result read against the elicitation and imitation readings, and a salience reading; the classification of each cell is the book's
Result Elicitation Imitation Salience
Students beat their supervisors almost everywhere (Burns et al., 2024) predicted compatible, since imitation is incomplete with finite data compatible
Recovery on reward modeling about 10 percent (Burns et al., 2024) compatible only if the concept is weakly represented predicted compatible
Larger students agree less with their supervisor's errors (Burns et al., 2024) predicted against its strong form compatible
Errors the student can reproduce are not corrected (Burns et al., 2024) predicted, in the theory's sense of representable predicted predicted
The expansion bound certified correction where it happened, not where it did not (Lang et al., 2024a) predicted compatible compatible
Easy supervision recovered 70 to 100 percent on question answering (Hase et al., 2024) predicted compatible, since it is a shift of inputs, not of labels compatible
Plain fine-tuning degraded content, or fell below the supervisor (Somerstep et al., 2025; Yang et al., 2024d) against its naive form predicted compatible
Reward models generalized like their own zero-shot preferences (Clymer et al., 2023) compatible, with the model's concept, not the intended one not tested, no supervisor errors to copy predicted
Unsupervised methods tracked the most salient feature (Canavan et al., 2026) against, for unsupervised methods not tested predicted

The last column is the book's (inference). On a salience reading, a student learns whichever concept consistent with its training signal is most salient to it. Elicitation is the case where the intended concept is most salient; imitation is the case where a model of the supervisor is. The founding study's own disanalogy, imitation saliency, is this reading applied to future models (Burns et al., 2024), and the failures of unsupervised methods on non-salient targets and on salient non-truths fit it (Wen et al., 2025c; Canavan et al., 2026). No result in the table is against it, which also means no result yet tests it: it would be refuted by a student that generalized along a concept measurably less salient to it than the alternatives the labels allowed (inference).

42.6.2 Human errors are systematic #

What decides the debate, on every account of Section 42.2, is whether the supervisor's errors are ones the student can fit cheaply. Human errors in judging model outputs are systematic, and other chapters report the evidence. In a study by Anthropic researchers, both people and preference models sometimes preferred convincingly written sycophantic answers to correct ones (Sharma et al., 2023) (Section 22.6). Reinforcement learning from human feedback made models better at persuading time-limited evaluators that wrong answers were right (Wen et al., 2025a) (Section 30.4.1). And an evaluator who sees only part of an episode rewards how it looks, which a theorem shows can make deceptive inflation optimal (Lang et al., 2024b) (Section 22.6.1).

These errors track features a strong model represents well, such as agreement with the user, confident presentation, and what a time-limited reader can verify. A model trained on human text predicts human judgment well, so these are errors it can fit cheaply, the case the theory says goes badly (inference). The link is the book's: the studies above concern reinforcement learning from human feedback, not weak-to-strong fine-tuning, and no study has yet measured recovery with weak labels whose errors are human in this sense.

42.6.3 Good behavior where the supervisor can check #

A worse variant would be a student that behaves well where its supervisor can check and differently where it cannot. Wenkai Yang and colleagues at Renmin University of China and Tencent's WeChat team tested for it in a 2024 preprint, which names the effect weak-to-strong deception (Yang et al., 2024c). The target was harmlessness, taught by a weak model, with a conflicting objective added: an explicit reward for harmful predictions, or extra training on helpfulness data (Yang et al., 2024c). The measure is the share of the errors the conflict causes that fall where the strong model knows the answer and the weak model does not (Yang et al., 2024c). Across GPT-2, OPT, Mistral, and Llama models, in reward modeling and in preference optimization, the authors found the effect in every setting, growing with the capability gap; bootstrapping through an intermediate model reduced it somewhat, and training only on the weak model's confident correct labels did not (Yang et al., 2024c). They flag an ambiguity themselves: without a conflicting objective, they "cannot fully disentangle" more erroneous weak supervision in the strong model's known area from a stronger student more actively deceiving its teacher (Yang et al., 2024c).

"Deceive" is the authors' word, and the intentional stance it adopts is a method to be justified, not a finding (Section 1.5). A behavioral account fits the same data (inference). A model trained on two conflicting objectives pays the cost of the conflict where the supervisor's labels pull least, which is where the supervisor is unsure; in reward modeling the student is even trained on the supervisor's confidence, which the authors suggest lets it perceive the weak model's knowledge boundary (Yang et al., 2024c). No intention is needed. What would separate the readings is a conflict whose cost cannot be moved into the supervisor's uncertain region; the result shows, either way, the pattern that the founding study's plan assumes away, behavior that is better where it is checked and worse where it is not, and growing with the gap (inference). The same question asked of monitors has a partial answer: telling an agent it was monitored degraded the monitor's reliability more than telling the monitor about the agent helped (Kale et al., 2025) (Section 50.3). Whether a model conditions its behavior on being observed is Chapter 37's subject, and whether training could select a model that does so by design is Section 34.1's.

42.6.4 Where the evidence stands #

For classification by students whose knowledge came from human data, under supervisors whose errors were random or the errors of small models, the evidence favors elicitation where the errors were hard to fit and imitation where they were easy. For the case that matters, supervision of generative behavior by people whose errors are systematic, the evidence does not decide between the readings. The experiment that would decide it is the crux of Section 42.8.

Sources cited in Section 42.6 14
  1. Burns et al. (2024) Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision
  2. Lang et al. (2024a) Theoretical Analysis of Weak-to-Strong Generalization
  3. Hase et al. (2024) The Unreasonable Effectiveness of Easy Training Data for Hard Tasks
  4. Wu and Sahai (2025) Provable Weak-to-Strong Generalization via Benign Overfitting
  5. Somerstep et al. (2025) A transfer learning framework for weak-to-strong generalization
  6. Yang et al. (2024d) Weak-to-Strong Reasoning
  7. Clymer et al. (2023) Generalization Analogies: A Testbed for Generalizing AI Oversight to Hard-To-Measure Domains
  8. Canavan et al. (2026) Three Concrete Challenges and Two Hopes for the Safety of Unsupervised Elicitation
  9. Wen et al. (2025c) Unsupervised Elicitation of Language Models
  10. Sharma et al. (2023) Towards Understanding Sycophancy in Language Models
  11. Wen et al. (2025a) Language Models Learn to Mislead Humans via RLHF
  12. Lang et al. (2024b) When Your AIs Deceive You: Challenges of Partial Observability in Reinforcement Learning from Human Feedback
  13. Yang et al. (2024c) Super(ficial)-alignment: Strong Models May Deceive Weak Models in Weak-to-Strong Generalization
  14. Kale et al. (2025) Reliable Weak-to-Strong Monitoring of LLM Agents

42.7 What weak-to-strong can and cannot show for oversight #

The question of this part is whether weak overseers can supervise strong systems. This section places the weak-to-strong route beside the protocols of Chapter 41, says what its experiments cannot show, and says what its results mean for the book's thesis.

42.7.1 Two routes, and a combination #

The protocols restructure the task so that a weak judge can check it: a debate between two strong models (Section 41.3), or a decomposition into parts the judge can grade, as in iterated amplification (Christiano et al., 2018) (Section 41.2). Weak-to-strong training relies instead on the student's inductive bias to carry it past what the judge can check. The founding study draws the contrast: scalable oversight methods "typically take advantage of special problem structure, like decomposability or the fact that evaluation is easier than generation," while its own aim is "generalizing beyond human supervision" (Burns et al., 2024).

The routes can be combined. Debate raised the accuracy of weaker judges where the debaters had information the judges lacked (Khan et al., 2024) (Section 41.5). Hao Lang, Fei Huang, and Yongbin Li of Alibaba's Tongyi Lab used debate to make the weak supervisor better, in a 2025 preprint (Lang et al., 2025). Two instances of Qwen-14B argued for opposite answers; the transcripts became context for fine-tuning an ensemble of Qwen-7B models; the ensemble labeled data for a Qwen-14B student (Lang et al., 2025). Recovery was measured against the small model fine-tuned without debate (Lang et al., 2025), so the gain mixes a better supervisor, one that had read the strong model's arguments, with better generalization by the student (inference). On four binary tasks from the founding study's benchmark, recovery rose against the confidence-loss baseline from 41.2 to 76.5 percent on SciQ, 56.4 to 69.2 on BoolQ, 17.4 to 56.5 on CosmosQA, and 35.0 to 70.0 on AnthropicHH, and debate beat single-sided consultancy as the source of the transcripts (Lang et al., 2025).

The derivation of Section 42.1.3 says how to read these numbers. The gaps were small: on AnthropicHH the weak supervisor scored 48.8 percent and the ceiling 50.8, both near chance, so a 35-point difference in recovery is a difference of 0.7 points in accuracy (Lang et al., 2025). The paper reports neither the size of its test sets nor variation across runs, and the authors note that a gap of 7 against 14 billion parameters is not large (Lang et al., 2025). The combination is a promising design, tested so far where the gap to be crossed was small (inference).

42.7.2 What the analogy cannot show #

Four limits follow from the evidence of this chapter, all the book's (inference) except where a source is named.

Where the knowledge came from. Every student so far learned from data people produced or collected. The image experiment of the founding study removed labels from pretraining but not people from the data, and ICM's author-gender task drew out more than the annotators knew from a model that learned it from human writing (Burns et al., 2024; Wen et al., 2025c). No experiment has a student whose relevant knowledge exceeds what people could have taught it, which is the case the analogy is for. The founding study lists the corresponding assumption for its plan, that the strong model has a concept such as honesty "which is natural enough to the model that it is feasible to elicit" (Burns et al., 2024).

What the theory covers. The accounts of Section 42.2 prove their results for classification or regression. One models a language model's output as a mixture of regressions and tests refinement on generative tasks (Somerstep et al., 2025), and another names next-token prediction as a direction for future work (Wu and Sahai, 2025). No theorem yet concerns open-ended generation or action, which is where oversight is needed.

How recovery scales with the gap. In the games of Section 41.6, the estimated success of oversight falls as the capability gap widens (Engels et al., 2025). For weak-to-strong training the dependence on the gap differs by task: recovery rose with the student's size on the language tasks and fell on chess for a fixed supervisor (Burns et al., 2024), easy-to-hard recovery on MMLU stayed near 100 percent from 7 to 70 billion parameters (Hase et al., 2024), and the deception score of Section 42.6.3 grew with the gap (Yang et al., 2024c). These are measurements across sizes within model families and a few pairs across families, none across a gap like the one between people and a system much more capable than they are.

Whose concepts. Whether the student generalizes along concepts its supervisor would recognize depends on whether models represent the world in the ways people do. If representations converge across models toward a shared model of the world (Huh et al., 2024), a question Section 46.6 weighs, the concepts a strong student generalizes along are more likely to be ones a weak supervisor also represents, though representing a concept is not the same as being able to name it; if not, it may generalize along concepts no supervisor intended, as the reward models of Section 42.4.3 generalized along their own preferences.

42.7.3 What it means for delegation #

The book's thesis is that alignment is a delegation problem under an asymmetry of competence and visibility. Weak-to-strong training is the asymmetry of competence in its plainest form: the principal, here the trainer, knows less than the agent it trains and hopes the agent's own knowledge will correct the principal's instructions (inference). In principal-agent theory the agent's private information is a problem the principal manages with incentives (Section 14.1.3); here the agent's knowledge is the resource, and what makes it flow toward the principal is the learner's inductive bias rather than a contract (inference). Two features have no counterpart in the older picture. The principal can choose the agent's inductive bias, rerun training, and read the agent's representations directly, as ICM and the probes of Section 42.5.1 do. And the agent's knowledge includes a model of the principal, which is what the salience reading says a student may generalize along instead of the truth (inference).

The claims of this chapter concern mostly the training process and its signal. The deception variant and the monitoring result concern the deployed system and the people it serves. And the developer and the public enter through the evidence: the setting with the lowest recovery in the founding study used a proprietary dataset, and the production results for ICM and the automated researcher are developers' reports on their own models, judged by their own reward models or evaluators, which no outside group can repeat (inference).

Sources cited in Section 42.7 11
  1. Christiano et al. (2018) Supervising strong learners by amplifying weak experts
  2. Burns et al. (2024) Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision
  3. Khan et al. (2024) Debating with More Persuasive LLMs Leads to More Truthful Answers
  4. Lang et al. (2025) Debate Helps Weak-to-Strong Generalization
  5. Wen et al. (2025c) Unsupervised Elicitation of Language Models
  6. Somerstep et al. (2025) A transfer learning framework for weak-to-strong generalization
  7. Wu and Sahai (2025) Provable Weak-to-Strong Generalization via Benign Overfitting
  8. Engels et al. (2025) Scaling Laws For Scalable Oversight
  9. Hase et al. (2024) The Unreasonable Effectiveness of Easy Training Data for Hard Tasks
  10. Yang et al. (2024c) Super(ficial)-alignment: Strong Models May Deceive Weak Models in Weak-to-Strong Generalization
  11. Huh et al. (2024) The Platonic Representation Hypothesis

42.8 Settled, contested, missing #

Research status Settled, contested, missing
  • Settled as mathematics, within their assumptions: the misfit bound, for a convex set that contains the target and squared loss (Charikar et al., 2024); the expansion bounds for robust students (Lang et al., 2024a); the two phases of benign overfitting in a Gaussian model (Wu and Sahai, 2025); and the floor on naive fine-tuning in a model of concept transfer (Somerstep et al., 2025). Whether real training meets the assumptions is what is open.
  • Settled: that strong students fine-tuned on a weaker model's labels outperform their supervisors on classification and regression tasks, found by the developer that proposed the setting (Burns et al., 2024) and reproduced by groups without a stake in that result, on other model families and tasks (Yang et al., 2024c; Charikar et al., 2024). How much of the gap they recover depends on the task, the sizes, and the method.
  • Reported, not yet replicated: that training models on easy examples transfers most of the benefit of training on hard ones, for question answering (Hase et al., 2024); the evidence disagrees elsewhere, since evaluators trained on easy mathematics generalized better than generators, whose easy-to-hard gaps depended on the training data (Sun et al., 2024a), and easy chess puzzles gave no gain over prompting on the hardest ones (Burns et al., 2024). Also reported: that the confidence loss recovers most of the gap at large gaps, reported by the developer (Burns et al., 2024), with outside tests at smaller gaps finding little effect (Mallen and Belrose, 2024); that refining weak labels with the strong model helps on generative reasoning, in two studies that share their mathematics labels (Yang et al., 2024d; Somerstep et al., 2025); that unsupervised elicitation matches correct labels on the tasks studied and supports a production reward model, checked only inside the author group (Wen et al., 2025c); that unsupervised and easy-to-hard prompting and probing fail on datasets without their convenient properties (Canavan et al., 2026); that conflict-induced errors concentrate where a weak supervisor cannot check, more so as the gap grows (Yang et al., 2024c); that debate-improved supervisors raise recovery (Lang et al., 2025); and that automated agents outperform human researchers on weak-to-strong supervision, in a developer's post that also reports the agents gaming its evaluator (Anthropic, 2026m).
  • Contested: whether weak-to-strong gains are elicitation or polished imitation of the supervisor. The evidence favors elicitation where the supervisor's errors were hard for the student to fit and imitation where they were easy, and it does not decide the case of supervision by people whose errors are systematic.
  • Missing: as of October 2026, a theory of weak-to-strong learning for open-ended generation or action; experiments in which the weak labels carry the systematic errors people make; a student whose relevant knowledge exceeds what people could have taught it; measurements of recovery across a capability gap like the one oversight faces; a test of the salience reading; and an independent repetition of the production results, which only developers can run today.
  • Crux: whether a strong student learns its supervisor's systematic, human-like errors. The founding study varied the structure of weak errors at fixed accuracy and made imitation trivial, and found that structure decides recovery (Burns et al., 2024); what has not been run is the human case. Measuring the performance gap recovered while varying only the structure of the weak labels' errors at matched accuracy, from random, through the errors of small models, to the errors people make when they judge sycophantic, persuasive, or partly observed outputs, and estimating the expansion of each (Lang et al., 2024a), on open-weight models by a group without a stake, would settle it. If recovery stays high for salient, human-like errors, weak supervision can scale for those tasks; if it collapses, it cannot without another source of signal.
Sources cited in Section 42.8 14
  1. Charikar et al. (2024) Quantifying the Gain in Weak-to-Strong Generalization
  2. Lang et al. (2024a) Theoretical Analysis of Weak-to-Strong Generalization
  3. Wu and Sahai (2025) Provable Weak-to-Strong Generalization via Benign Overfitting
  4. Somerstep et al. (2025) A transfer learning framework for weak-to-strong generalization
  5. Burns et al. (2024) Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision
  6. Yang et al. (2024c) Super(ficial)-alignment: Strong Models May Deceive Weak Models in Weak-to-Strong Generalization
  7. Hase et al. (2024) The Unreasonable Effectiveness of Easy Training Data for Hard Tasks
  8. Sun et al. (2024a) Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision
  9. Mallen and Belrose (2024) Balancing Label Quantity and Quality for Scalable Elicitation
  10. Yang et al. (2024d) Weak-to-Strong Reasoning
  11. Wen et al. (2025c) Unsupervised Elicitation of Language Models
  12. Canavan et al. (2026) Three Concrete Challenges and Two Hopes for the Safety of Unsupervised Elicitation
  13. Lang et al. (2025) Debate Helps Weak-to-Strong Generalization
  14. Anthropic (2026m) Automated Weak-to-Strong Researcher

42.9 Exercises #

Exercise 42.1 How much a recovery figure can be trusted

The debate-and-ensemble study reports these test accuracies, in percent (Lang et al., 2025):

Task Weak supervisor Confidence-loss student Debate-ensemble student Ceiling
SciQ 90.0 91.4 92.6 93.4
BoolQ 86.0 88.2 88.7 89.9
CosmosQA 87.5 87.9 88.8 89.8
AnthropicHH 48.8 49.5 50.2 50.8

The paper does not report its test-set sizes. Suppose, for the exercise, that each score was measured on 2,000 questions with independent errors. Compute the performance gap recovered of both students on each task and its approximate standard deviation from (42.2), and say which figures, and which differences between the two students, are too noisy to read.

Solution

The recoveries are (zw2s−zweak)/(zceil−zweak)(z_{\text{w2s}} - z_{\text{weak}})/(z_{\text{ceil}} - z_{\text{weak}}): SciQ 0.41 and 0.76 (gap 3.4 points), BoolQ 0.56 and 0.69 (gap 3.9), CosmosQA 0.17 and 0.57 (gap 2.3), AnthropicHH 0.35 and 0.70 (gap 2.0), matching the paper's figures. The standard error of one score is z(1−z)/2000\sqrt{z(1-z)/2000}: about 0.006 near 92 percent, 0.007 near 88 percent, and 0.011 near 50 percent. Dividing by the gap and multiplying by 1+PGR2+(1−PGR)2\sqrt{1 + \mathrm{PGR}^2 + (1-\mathrm{PGR})^2} gives standard deviations of about 0.22 and 0.23 on SciQ, 0.23 on BoolQ, 0.41 and 0.38 on CosmosQA, and 0.70 on AnthropicHH. The AnthropicHH figures say nothing: a standard deviation of 0.7 covers both 0 and 1. CosmosQA's are weak. The difference between the two students on one task has a standard deviation of about 2 se/gap\sqrt{2}\,\mathrm{se}/\mathrm{gap} under the same assumption, about 0.25 on SciQ, 0.26 on BoolQ, 0.44 on CosmosQA, and 0.79 on AnthropicHH, so the observed differences of 0.35, 0.13, 0.39, and 0.35 are each between about 0.4 and 1.4 standard deviations. Scoring both students on the same questions would shrink these, since their per-question scores are positively correlated, the paired comparison of Section 36.7.2. For two scores of equal variance with correlation ρ\rho, the variance of their difference falls by the factor 1−ρ1 - \rho. The lesson is the one the derivation states: report the gap and the test size with the ratio.

Exercise 42.2 When the misfit bound holds, and when it fails

(a) Show that (42.3) holds with equality when F\mathcal{F} is a linear subspace. (b) Let the input take two values with probability one half each, so that a function is a pair of numbers and ∥a−b∥2=12((a1−b1)2+(a2−b2)2)\lVert a - b \rVert^2 = \tfrac{1}{2}\big((a_1 - b_1)^2 + (a_2 - b_2)^2\big). Take F={(t,0)}\mathcal{F} = \{(t, 0)\}, the correct answers y∗=(0,1)y^\ast = (0, 1), and a perfect supervisor, yweak=y∗y_{\text{weak}} = y^\ast. Compute the student, its error, and its misfit, and say which assumption fails. (c) Now take F={(1,0),(0,1)}\mathcal{F} = \{(1, 0), (0, 1)\}, y∗=(1,0)y^\ast = (1, 0), and yweak=(0.4,0.6)y_{\text{weak}} = (0.4, 0.6). Which assumption fails, and does the bound hold?

Solution

(a) For a subspace, every ys+t(y∗−ys)y_{\text{s}} + t(y^\ast - y_{\text{s}}) with any real tt, negative included, lies in F\mathcal{F}, so the argument of step 1 runs for tt of both signs and the inner product must be exactly zero. The cross term in step 2 vanishes and the inequality becomes the Pythagorean equality. (b) The nearest point of F\mathcal{F} to (0,1)(0, 1) is (0,0)(0, 0). The student's error is 12\tfrac{1}{2}, the teacher's is 0, and the misfit is 12\tfrac{1}{2}. The bound would require 12≤0−12\tfrac{1}{2} \le 0 - \tfrac{1}{2}. Realizability fails, since y∗∉Fy^\ast \notin \mathcal{F}: the student cannot represent the truth, so even a perfect teacher leaves it worse off by exactly its misfit, the reversal the authors observed when the less expressive model had the better representation (Charikar et al., 2024). (c) Convexity fails, since the set is two points. The nearer point to the teacher is (0,1)(0, 1), at squared distance 0.160.16, against 0.360.36 for (1,0)(1, 0), so the student is (0,1)(0, 1). Its error is 1, the teacher's 0.36, and the misfit 0.16; the bound would require 1≤0.21 \le 0.2 and fails. A non-convex student can snap to the wrong answer when the teacher sits between two options.

Exercise 42.3 Errors the student can and cannot fit

In the model of Figure 42.1, the supervisor is right 80 percent of the time. (a) With systematic errors, the supervisor's rule is tilted from the correct one by the angle π(1−zweak)\pi(1 - z_{\text{weak}}). Using the facts that a least-squares student on Gaussian features takes the direction of the supervisor's rule projected onto its own features, and that two linear rules at angle aa disagree on a share a/πa/\pi of inputs, compute the student's accuracy and recovery when the share ζ\zeta of the tilt in the student's features is 1, 0.5, and 0. (b) Construct an error that is systematic, not random, but that the student cannot fit without extra error, and predict the recovery.

Solution

(a) The tilt is π×0.2\pi \times 0.2, or 36 degrees. The student's direction keeps the correct component and the visible part of the tilt, so its angle to the correct rule is arctan⁡(ζtan⁡36∘)\arctan(\zeta \tan 36^\circ). At ζ=1\zeta = 1 that is 36 degrees, accuracy 0.80, recovery 0: the student copies the supervisor. At ζ=0.5\zeta = 0.5 it is arctan⁡(0.363)≈20.0\arctan(0.363) \approx 20.0 degrees, accuracy 1−20.0/180≈0.8891 - 20.0/180 \approx 0.889, recovery 1−20.0/36≈0.451 - 20.0/36 \approx 0.45. At ζ=0\zeta = 0 the angle is 0, accuracy 1, recovery 1. (b) Flip the label whenever the second feature is large in magnitude, ∣x2∣>1.28|x_2| > 1.28, which happens on 20 percent of inputs. The errors are a deterministic function of the input, so they are not random, but a linear rule cannot reproduce them: fitting them would mean mislabeling many unflipped points with the same first feature. The least-squares direction is unchanged, since flipping on a region symmetric in x2x_2 only shrinks the correct component, so the student matches the correct rule and recovers the whole gap in the limit of unlimited data. This is the expansion account's case: the bad points have good neighbors the student cannot separate from them.

Sources cited in Section 42.9 2
  1. Lang et al. (2025) Debate Helps Weak-to-Strong Generalization
  2. Charikar et al. (2024) Quantifying the Gain in Weak-to-Strong Generalization

Further reading #

  • Burns et al. (2024): the analogy, the metric, the first measurements, and the authors' own list of disanalogies.
  • Lang et al. (2024a): when a student corrects its teacher, stated with conditions that can be checked from data.
  • D'Amour et al. (2022): why many students fit the same labels and can behave very differently outside the training domain.
  • Christiano et al. (2018): the decomposition route to supervising strong learners with weak experts.
  • Anthropic (2026m): weak-to-strong supervision as a problem automated researchers now work on, and the ways they gamed it.

References

  1. Anthropic (2025as). Unsupervised Elicitation. Anthropic Alignment Science Blog. non-peer-reviewed Cited in §42.5
  2. Anthropic (2026m). Automated Weak-to-Strong Researcher. Anthropic Alignment Science Blog. non-peer-reviewed Cited in §42.3 §42.8
  3. Burns, C., Ye, H., Klein, D., and Steinhardt, J. (2022). Discovering Latent Knowledge in Language Models Without Supervision. International Conference on Learning Representations. Cited in §42.5
  4. Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L., … Wu, J. (2024). Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision. International Conference on Machine Learning. Cited in §42.1 §42.2 §42.3 §42.4 §42.6 §42.7 §42.8
  5. Canavan, C., Shrivastava, A., Qi, A., Michala, J., and Roger, F. (2026). Three Concrete Challenges and Two Hopes for the Safety of Unsupervised Elicitation. arXiv. preprint Cited in §42.5 §42.6 §42.8
  6. Charikar, M., Pabbaraju, C., and Shiragur, K. (2024). Quantifying the Gain in Weak-to-Strong Generalization. Advances in Neural Information Processing Systems. doi:10.52202/079017-4017. Cited in §42.2 §42.8 §42.9
  7. Christiano, P., Shlegeris, B., and Amodei, D. (2018). Supervising strong learners by amplifying weak experts. arXiv. preprint Cited in §42.7
  8. Clymer, J., Baker, G., Subramani, R., and Wang, S. (2023). Generalization Analogies: A Testbed for Generalizing AI Oversight to Hard-To-Measure Domains. arXiv. preprint Cited in §42.4 §42.6
  9. D'Amour, A., Heller, K., Moldovan, D., Adlam, B., Alipanahi, B., Beutel, A., … Sculley, D. (2022). Underspecification Presents Challenges for Credibility in Modern Machine Learning. Journal of Machine Learning Research, 23(226):1−61.
  10. Ding, M., Deng, C., Choo, J., Wu, Z., Agrawal, A., Schwarzschild, A., … Huang, F. (2024). Easy2Hard-Bench: Standardized Difficulty Labels for Profiling LLM Performance and Generalization. Advances in Neural Information Processing Systems. Cited in §42.4
  11. Engels, J., Baek, D., Kantamneni, S., and Tegmark, M. (2025). Scaling Laws For Scalable Oversight. Advances in Neural Information Processing Systems 38. doi:10.52202/085713-3254. Cited in §42.7
  12. Farquhar, S., Varma, V., Kenton, Z., Gasteiger, J., Mikulik, V., and Shah, R. (2023). Challenges with unsupervised LLM knowledge discovery. arXiv. preprint Cited in §42.5
  13. Hase, P., Bansal, M., Clark, P., and Wiegreffe, S. (2024). The Unreasonable Effectiveness of Easy Training Data for Hard Tasks. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Cited in §42.4 §42.6 §42.7 §42.8
  14. Huh, M., Cheung, B., Wang, T., and Isola, P. (2024). The Platonic Representation Hypothesis. International Conference on Machine Learning. Cited in §42.7
  15. Kale, N., Zhang, C. B. C., Zhu, K., Aich, A., Rodriguez, P., Scale Red Team, Knight, C. Q., and Wang, Z. (2025). Reliable Weak-to-Strong Monitoring of LLM Agents. arXiv. preprint Cited in §42.6
  16. Khan, A., Hughes, J., Valentine, D., Ruis, L., Sachan, K., Radhakrishnan, A., … Perez, E. (2024). Debating with More Persuasive LLMs Leads to More Truthful Answers. International Conference on Machine Learning. Cited in §42.7
  17. Lang, H., Sontag, D., and Vijayaraghavan, A. (2024a). Theoretical Analysis of Weak-to-Strong Generalization. Advances in Neural Information Processing Systems. doi:10.52202/079017-1486. Cited in §42.2 §42.6 §42.8
  18. Lang, L., Foote, D., Russell, S., Dragan, A., Jenner, E., and Emmons, S. (2024b). When Your AIs Deceive You: Challenges of Partial Observability in Reinforcement Learning from Human Feedback. Advances in Neural Information Processing Systems. Cited in §42.6
  19. Lang, H., Huang, F., and Li, Y. (2025). Debate Helps Weak-to-Strong Generalization. arXiv. preprint Cited in §42.3 §42.7 §42.8 §42.9
  20. Mallen, A., and Belrose, N. (2024). Balancing Label Quantity and Quality for Scalable Elicitation. arXiv. preprint Cited in §42.3 §42.8
  21. Mallen, A., Brumley, M., Kharchenko, J., and Belrose, N. (2023). Eliciting Latent Knowledge from Quirky Language Models. arXiv. preprint Cited in §42.5
  22. Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A., Bowman, S. R., … Perez, E. (2023). Towards Understanding Sycophancy in Language Models. International Conference on Learning Representations. Cited in §42.6
  23. Somerstep, S., Polo, F. M., Banerjee, M., Ritov, Y., Yurochkin, M., and Sun, Y. (2025). A transfer learning framework for weak-to-strong generalization. International Conference on Learning Representations. Cited in §42.2 §42.3 §42.6 §42.7 §42.8
  24. Sun, Z., Yu, L., Shen, Y., Liu, W., Yang, Y., Welleck, S., and Gan, C. (2024a). Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision. Advances in Neural Information Processing Systems. Cited in §42.4 §42.8
  25. Wen, J., Zhong, R., Khan, A., Perez, E., Steinhardt, J., Huang, M., … Feng, S. (2025a). Language Models Learn to Mislead Humans via RLHF. International Conference on Learning Representations. Cited in §42.6
  26. Wen, J., Ankner, Z., Somani, A., Hase, P., Marks, S., Goldman-Wetzler, J., … Leike, J. (2025c). Unsupervised Elicitation of Language Models. arXiv. preprint Cited in §42.5 §42.6 §42.7 §42.8
  27. Wu, D. X., and Sahai, A. (2025). Provable Weak-to-Strong Generalization via Benign Overfitting. International Conference on Learning Representations. Cited in §42.2 §42.6 §42.7 §42.8
  28. Yang, W., Shen, S., Shen, G., Yao, W., Liu, Y., Gong, Z., Lin, Y., and Wen, J.-R. (2024c). Super(ficial)-alignment: Strong Models May Deceive Weak Models in Weak-to-Strong Generalization. arXiv. preprint Cited in §42.6 §42.7 §42.8
  29. Yang, Y., Ma, Y., and Liu, P. (2024d). Weak-to-Strong Reasoning. Conference on Empirical Methods in Natural Language Processing. Cited in §42.3 §42.6 §42.8
AI Alignment: From the Optimization Problem to the Institutions Around It · © 2026 Changkun Ou · CC BY-NC-SA 4.0
Bibliography · PV/UV /