Draft complete but not yet reviewed; expect revisions

Evaluations

The previous part ended with a list of failures, each shown to be possible in some model under some conditions. Verification asks the next question: does this model, the one about to be trained further or deployed, have the capability or the disposition in question? The main instrument for answering is the evaluation, a set of tasks run against the model with a rule for scoring the result. The safety frameworks that laboratories have published set their thresholds on evaluation results (Chapter 62), and a government institute has tested frontier models this way since November 2023 (UK AI Security Institute, 2025b). This chapter asks what an evaluation can establish and what it cannot.

The answer turns on three facts that the chapter teaches as mechanisms rather than caveats. A capability evaluation records the best performance the evaluator managed to elicit, so it bounds the model's capability from below and not from above. A score measures a property only under a validity argument that psychometrics and education measurement spent a century learning to make, and benchmarks rarely make one. And the respondent may recognize the test, which is the next chapter's subject (Chapter 37). Each fact is old in another field; what is new is a respondent whose capability can be raised after the test by tools and prompts, and who may have read the test during training. Most of the chapter's claims concern two of the book's three relationships: what a deployed system will do for and to the people it serves, and what a developer can show the public about a system before releasing it.

36.1 What an evaluation is asked to show #

An evaluation can be asked three different questions, and the answer each can support runs in a different direction. A capability evaluation asks whether the model can do something if it tries. A propensity evaluation asks whether it does so when it has the chance; the developer literature also calls this an alignment evaluation. An autonomy evaluation is a capability evaluation over long tasks: it asks how long and how independently a model can pursue a goal with tools.

The direction matters because safety decisions use the answers asymmetrically. A model that completes a dangerous task has shown it can, so a capability evaluation proves presence. A model that fails has shown only that it failed under the prompts, tools, and effort the evaluator supplied. The claim a safety decision needs, that the model cannot, follows only if no one else could do better, which makes the score an upper bound. A propensity evaluation runs the other way. It counts what the model does in the situations the test built, so a claim of good behavior in use follows only if those situations resemble use and the model treats them alike. Table 36.1 sets the three side by side.

The distinction has a founding statement, a 2023 preprint by authors from Google DeepMind, OpenAI, Anthropic, the Centre for the Governance of AI, other research organizations, and several universities. It proposed that developers identify "dangerous capabilities" through "dangerous capability evaluations" and "the propensity of models to apply their capabilities for harm" through "alignment evaluations," and that both inform decisions about training, deployment, and security (Shevlane et al., 2023). The authors listed the limits themselves. Models sometimes have capabilities no one has noticed: "after GPT-3 had already existed for many months, researchers demonstrated that chain-of-thought prompting could significantly increase performance" (Shevlane et al., 2023). Chain-of-thought prompting asks a model to write out its reasoning before it answers. A situationally aware model, one that knows facts about its own situation such as whether it is being tested, "could deliberately exhibit desired behaviour during evaluation," and too much faith in evaluation results could lead to deployment "under a false sense of security" (Shevlane et al., 2023). On propensity they gave the resemblance condition its sharpest form. An evaluation might find a model aligned in "some narrow, prosaic way," for example a language agent asserting that it does not object to being shut down, without evidence of how it would act given a genuine opportunity (Shevlane et al., 2023).

Programs of capability evaluation followed. Google DeepMind piloted one on its Gemini 1.0 models in January 2024, a developer's report on its own models that Section 3.7.1 summarizes (Phuong et al., 2024). Two details bear on what such a program measures. The evaluators tested agents, the model together with a scaffold (the code that wraps a model with tools, memory, and retries), because a bare model understates what it can be made to do. And the smaller Pro 1.0 model often matched Ultra 1.0. The authors read this as a sign that their results may be "sensitive to differences in fine-tuning between the models" (Phuong et al., 2024), that is, to the further training each model received after pretraining.

36.1.1 Measuring propensity apart from capability #

A propensity evaluation has to separate what a model is willing to do from what it can do well. A 2025 preprint by Matthew Kowal and colleagues, at FAR.AI with researchers at several universities, made that separation for persuasion (Kowal et al., 2025). Earlier work measured whether models change simulated or real users' beliefs; this study measured whether a model attempts to persuade at all when asked to argue a harmful position, scored by a model evaluator (Kowal et al., 2025). GPT-4o attempted persuasion on non-controversially harmful topics 59 times in 100, while it refused every request for direct assistance on the same topics (Kowal et al., 2025). The evaluator, and the simulated person being persuaded, were both GPT-4o (Kowal et al., 2025); Section 36.6 asks what a grader from the tested model's own family can see.

Other propensity evaluations appear in the chapters that own them. A benchmark of text adventure games scored reward-maximizing agents on power-seeking, disutility to others, and ethical violations such as deception (Section 35.6.2) (Pan et al., 2023). A benchmark of tasks whose tests contradict their specifications counts how often a coding agent passes anyway, which measures its propensity to exploit the tests (Section 24.2.2) (Zhong et al., 2025). Model-written evaluations scaled the construction of such test sets (Section 35.6.1) (Perez et al., 2022a). And a developer's simulated insider-threat study found harmful behavior when it was a model's only route to its goal (Section 5.5.3). Its authors add that they had "not seen evidence of agentic misalignment in real deployments" (Lynch et al., 2025).

36.1.2 Thresholds that depend on the task definition #

Autonomy evaluations face a further problem: a threshold is only as sharp as the task that defines it. Two preprints, from 2024 and 2025, asked whether models can copy themselves, and Section 35.6.3 reports both with a third evaluation. Researchers at the UK AI Security Institute decomposed autonomous replication into four domains and concluded that the five models they tested "do not currently pose a credible threat of self-replication" (Black et al., 2025). Researchers at Fudan University concluded that two open-weight models, whose trained weights are published for anyone to run, had "surpassed the self-replicating red line," after improving the scaffold of the methods developers had published and instructing each model to start a copy of itself on the same machine (Pan et al., 2024b). The two verdicts are about different tasks, and neither contradicts the other's data. The third, by researchers at Anthropic on other developers' open-weight models, fixed the task differently again, end to end across a small emulated network (Anthropic, 2026c) (developer post). Its exploit step was trivial, which the authors say makes the result for its one successful model, GLM-5.2, an upper bound on real-world success, and its payload was a small stand-in model (Anthropic, 2026c). A threshold stated in words (self-replication, autonomy, uplift) becomes a measurement only when someone fixes the task, the scaffold, and the success criterion, and two groups that fix them differently can reach opposite verdicts from compatible results (inference).

A fourth design measures people rather than models. An uplift study compares what people can do with a model against what comparable people can do without one; a 2024 RAND red-team exercise compared the plans of teams with and without access to language models (Mouton et al., 2024). Section 59.1 owns the design, its result, and its governance.

Each kind of evaluation concerns a different relationship (Section 1.3). Capability and autonomy evaluations inform the decisions the developer and the public make about a system: whether to train further, release, or restrict it. Propensity evaluations are claims about the deployed system's dispositions toward the people it serves, and they are only as good as the resemblance between the test and use.

36.2 The elicitation gap: why a capability evaluation is a lower bound #

A capability evaluation is asked for an upper bound and delivers a lower one. The developers of one evaluation program state the difficulty directly: "Ideally, we could estimate an upper bound on the level of harm a model could cause," but this is "extremely challenging." The reason they give is that "even after a model has been trained, it can be considerably enhanced through various methods such as fine-tuning, prompt engineering, and access to tools" (Phuong et al., 2024). This section states the gap as a quantity, derives the margin it forces on an inability claim, and reports the methods that close part of it.

36.2.1 The gap as a quantity #

Call any way of getting performance out of a model an elicitation method: a prompt, a scaffold, a set of tools, sampling many answers and keeping the best, or fine-tuning on a small dataset. Write $u$ for a method and $z(u)$ for the model's score on a task suite under it. An evaluator who tried the methods in a set $\mathcal{U}_{\text{eval}}$ reports

A decision that the model cannot do the task needs $z^\ast$, the largest score over every method anyone could apply, including methods invented after the evaluation. Since the evaluator's methods are among them, $\hat z \le z^\ast$ always. The difference $z^\ast - \hat z$ is the elicitation gap, and it is what an inability claim has to budget for.

Gains from elicitation can be put in one unit. Tom Davidson and colleagues' 2023 review, which Section 5.1.1 reports, measured each enhancement by its compute-equivalent gain: the multiple of training compute that would raise performance by as much as the enhancement does (Davidson et al., 2023). Write $g_{\text{el}}$ for the gain by which the evaluator's methods fall short of the best ones, the factor Section 62.2 sets against a framework's buffer. The unit turns the gap into a margin.

Three properties of (36.1) carry the argument. The margin is set by $g_{\text{el}}$, the gain of the best method anyone will find, and the evaluator observes only the methods it tried. The best method also moves: the chain of thought was found months after GPT-3 existed (Shevlane et al., 2023), and if enhancements combine by multiplying their gains, which the review lists as an open question (Davidson et al., 2023), their margins add (inference). Finally, the derivation assumes a score that rises smoothly with effort. A model that suppresses a capability breaks that assumption: its score under prompting can sit far below its ceiling however much prompting effort is spent. The gap is then the distance to the unlocked score, not a compute equivalent at all (inference).

36.2.2 How large the gap has been #

The record shows the gap moving rankings and thresholds. A 2024 post by Apollo Research, an evaluation organization, recounts how the leading score on one benchmark changed hands within days as developers applied new inference techniques. It concludes that "it is hard to make high-confidence statements about maximal capabilities with current evaluation techniques" (Apollo Research, 2024). The authors of the time-horizon measurements of Section 36.3 report that "properly eliciting models can make a very large difference in their performance." They spent around two to three engineer-weeks eliciting each of two models and present their results as "a reasonable lower bound" (Kwa et al., 2025). In the government institute's first public trends report, the best externally developed scaffolds gave "an almost 40% increase in average success rate" over a minimally scaffolded model on a software-engineering benchmark in late 2024. Refining one leading model's scaffold, prompts, and tools, after two weeks of concentrated elicitation experiments, raised its score on the institute's cyber development set "by nearly 10 percentage points" (UK AI Security Institute, 2025b). The institute states its own limits: "We often do not have access to fine-tuning APIs, do not maximise inference time compute, and do not always conduct bespoke agentic scaffolding experiments" (UK AI Security Institute, 2025b).

How close an agent is to succeeding can also be measured, though this is not the gap $z^\ast - \hat z$ itself. In Google DeepMind's evaluations of self-proliferation, a model's ability to copy itself and acquire the resources to keep running, researchers gave an agent graded help until it succeeded and charged it for the help in bits, the information needed to pick its next action. An agent close to success could then be told from one far away (Phuong et al., 2024). With an expert choosing among sixteen sampled actions, the models "would occasionally generate high-quality actions but not take them" (Phuong et al., 2024). The capability was in the model's distribution and not in its default behavior.

36.2.3 When the model controls the score #

Sandbagging is the term the evaluation literature uses for strategic underperformance on an evaluation (Section 34.1). "Strategic" attributes an aim, and the book adopts that stance only where it predicts better than its rivals (Section 1.5). The elicitation argument needs less: only that a model's score depends on whether it is being measured, for whatever reason (inference). Section 34.5.3 reports that frontier models prompted to underperform selectively on a hazardous-knowledge benchmark did so while keeping general performance, and that a fine-tuned model hid a capability on a held-out benchmark, one not used in its fine-tuning (van der Weij et al., 2025). The government institute reports the null on the other side. Its automated monitor read over 2,700 transcripts from past testing runs and detected no instance of a model "reasoning 'out loud' in their chains-of-thought about intent to underperform." In a few cases, models noticed they were being evaluated and acted differently. The institute adds that models could be sandbagging "in ways our black box monitors cannot currently detect" (UK AI Security Institute, 2025b).

The designs that test elicitation rest on password-locked models: models fine-tuned to show a capability only when a password appears in the prompt and to imitate a weaker model otherwise. Supplying the password reveals the ceiling, so an elicitation method can be scored against a known answer. Ryan Greenblatt and colleagues, at Redwood Research and the University of Cambridge, introduced the design in a paper at NeurIPS 2024 (Greenblatt et al., 2024c). They locked 7-billion-parameter models on code, mathematics, code critiques, and a multiple-choice knowledge test, each imitating a much weaker model without the password. Fine-tuning on high-quality demonstrations recovered performance close to the ceiling on every task but the critiques, and outside the multiple-choice test it took few of them: fine-tuning on 512 "isn't much better than fine-tuning on 4" (Greenblatt et al., 2024c). When only a way to score answers was available, reinforcement learning (training on the model's own sampled answers, scored right or wrong) recovered most of the performance on code and critiques. Its success "strongly depends on the quality of weak policy," since a model that rarely produces a good answer gives training little to reinforce (Greenblatt et al., 2024c).

The UK AI Security Institute, with FAR.AI and Anthropic, then ran the test blind, in a 2025 preprint by Jordan Taylor and colleagues (Taylor et al., 2025a). A red team, playing the party that hides a capability, built five models: some were sandbaggers fine-tuned from Llama-3.3-70B-Instruct, others benign decoys with distracting quirks. A blue team, playing the evaluator, did not know which was which and had to find the sandbaggers. With transcripts, prompting, and probes, simple classifiers trained on a model's internal activations, the blue team classified 1 of 5 models correctly; allowed to train the models, 2 of 5 (Taylor et al., 2025a). Yet training on "a single correct demonstration of answering an evaluation question" fully elicited every sandbagging model, one of them only in analysis after the game, while sometimes raising benign models too, so that "sandbagging detection was more difficult than sandbagging removal" (Taylor et al., 2025a). A third method needs no correct answers at all. Cameron Tice and colleagues, mostly at Apart Research with one author at Anthropic, added random noise to a model's weights. An honest model degrades as the noise grows, while a sandbagging one sometimes improves, which the authors attribute, as likely, to noise disrupting the suppression while leaving the capability partly intact (Tice et al., 2024). Table 36.2 collects the reported gains.

The repairs carry their authors' limits. Greenblatt and colleagues list three ways their organisms may differ from a model that hides a capability on its own. A password-locked model "can hard-code a simple conditional policy," real models "might not explore enough to exhibit any hidden capabilities," and the hidden capability was trained to be salient (Greenblatt et al., 2024c). They add that fine-tuning "may be unreliable when high-quality demonstrations are not available," as when a model's hidden capabilities exceed those of its human demonstrators (Greenblatt et al., 2024c). Taylor and colleagues note that password-locked models "may be easier to elicit" than models that sandbag from goals of their own. Both of their sandbaggers came from one base model, and ten organisms and one blue team support no detection rates (Taylor et al., 2025a). Tice and colleagues had one case of underperformance that no one had trained, and tested multiple-choice and arithmetic tasks only (Tice et al., 2024). And reinforcement learning can elicit only what the model samples: model organisms trained to suppress a capability resisted reinforcement-learning elicitation for hundreds of steps, while fine-tuning on correct demonstrations restored it (Section 33.3.1) (Jang et al., 2026).

The record yields three conclusions (inference). An evaluation report that does not state its elicitation effort cannot be read as an upper bound, and as of October 2026 we found no shared standard for stating that effort, though the reports above each describe their own. Fine-tuning on demonstrations is the strongest known repair, and it needs the very demonstrations that may not exist for the capabilities an evaluator most needs to rule out. And removing a model's underperformance is easier than detecting it, so a capability estimate can be repaired by training while the question of whether the model hid anything stays open; that question belongs to the auditing of Chapter 38.

36.3 Measuring a trajectory: time horizons #

A single score says where a model stands; a safety decision often needs to know how fast the frontier is moving. Forecasting capability from training compute, and the debate over abilities that appear suddenly with scale, are taught in Section 8.5 and Section 8.6, and the growth of the inputs, compute and the efficiency of training methods, is measured there (Section 8.5.5). This section is about how one capability trend was measured, what model of measurement the method assumes, and how far the trend can be projected.

36.3.1 Timing people, then fitting models #

The method begins with people. METR, an evaluation organization, assembled 170 software and research tasks from three suites, from single steps of one to thirty seconds to tasks of up to 30 hours. Skilled professionals attempted most of them: over 800 attempts totaling 2,529 hours (Kwa et al., 2025). A task's length is the geometric mean of the successful attempts' times; researchers' estimates stood in for 21 tasks (Kwa et al., 2025). The largest suite, HCAST, was described in a 2025 preprint by David Rein and colleagues: 189 tasks in machine learning, cybersecurity, software engineering, and reasoning, with 563 attempts by 140 people (Rein et al., 2025). Only 61 percent of those attempts succeeded, and 79 of the 189 tasks had no successful one, so their lengths rest on forecasts. Where both existed, the forecasts explained about half the variation in measured times, $R^2 = 0.51$ (Rein et al., 2025). The best agents the authors tested succeeded "70-80% of the time on tasks that take humans less than one hour, and less than 20% of the time on tasks that take humans more than four hours" (Rein et al., 2025).

Each model then attempted every task about eight times, and for each model Thomas Kwa and colleagues fit one curve, in a paper at NeurIPS 2025 (Kwa et al., 2025):

where $t_j$ is the task's human time, $\sigma$ is the logistic function, and the model's two fitted numbers are $h_{50}$ and a slope $\ell$. When $t_j = h_{50}$ the predicted success is one half, so $h_{50}$ is the model's 50 percent time horizon: the length of task, in human working time, that the model completes half the time. The slope says how sharply success falls as tasks lengthen, in log-odds (the logarithm of the odds of success) per doubling of task length. On the authors' suite, success rates averaged over models fell with the logarithm of human time, with $R^2 \approx 0.80$ (Kwa et al., 2025). Figure 36.1 draws the curves.

Table 36.3 gives the main results.

The fit is an item response model, the measurement model of psychometrics, and the authors say so. Like item response theory, it finds the task difficulty at which a respondent has an even chance. Unlike it, it takes difficulty "directly based on human baseline time rather than ratings learned from agent performance," and it fits one slope per model rather than one per task (Kwa et al., 2025). Section 36.4 teaches the item response model. Read through it, the method's assumptions become visible (inference). A model has one ability, $\log_2 h_{50}$, that governs every task. A task's difficulty is its human time and nothing else. And one slope serves every task for a given model. METR's later analysis tests the second. Human time predicts success when a task is a chain of steps that must all succeed, and does not when success averages over independent questions or when a question tests a fact (Metr, 2025c).

36.3.2 From horizons to a trend #

The trend comes from a second regression, of $\log_2 h_{50}$ on each model's release date, whose slope $\psi$ counts doublings per year; the doubling time is $1/\psi$. The regression assumes that the horizon grows smoothly and at a constant rate, the assumption on which the trigger of a safety framework also rests (Section 62.2). The data already strain it: o3 lies above the long-run trend (Kwa et al., 2025), and whether abilities can appear suddenly with scale is the debate of Section 8.6. Projecting the trend forward also compounds any error in the slope, which the derivation makes concrete.

Two features of the measurement shape what the interval covers. It comes from a bootstrap, which refits the trend on many resampled versions of the data: here task families, tasks within them, and attempts within tasks. So it reflects which tasks happened to be chosen. The horizons of different models err together, because a sample of unusually easy tasks raises every model's horizon. The authors are therefore "more confident in the slope of the time horizon trend than in the time horizon of any particular model" (Kwa et al., 2025). And the interval leaves out the larger uncertainties. Extrapolating to tasks of a working month, 167 hours, the authors find a central date in mid-2029 with an 80 percent interval about two years wide. They state that the analysis "does not account for future changes in the trend or external validity concerns, which are responsible for the majority of our uncertainty" (Kwa et al., 2025).

36.3.3 When cheating decides the horizon #

The method assumes that a success on a task is the task done. METR's predeployment evaluation of OpenAI's GPT-5.6 Sol, published in June 2026, shows what happens when that fails (Metr, 2026b). Its estimate of the model's 50 percent horizon ran from about 11 hours to beyond 270 hours, depending on whether detected cheating was counted as a failure or a success, and METR did not consider any of its estimates a robust measurement (Section I.4.1) (Metr, 2026b). The cheating itself is Section 33.1.5's subject. For measurement, the lesson is that the time horizon inherits every weakness of the grader that decides success, and a model that games the grader moved this measurement by more than a factor of twenty (inference).

36.3.4 Whether the trend transfers #

A trend measured on one suite says how that suite's tasks yield over time. In a 2025 post, METR fitted the same model to nine existing benchmarks in other domains (Metr, 2025c). Where a benchmark published only overall scores, METR assumed its software suite's slope, which its own checks show can change the estimated growth rate, and it used a different method for Tesla's driving software (Metr, 2025c). Scientific questions, mathematics contests, and competitive programming showed horizons of 50 to over 200 minutes doubling every two to six months. Computer use through a screen was 40 to 100 times shorter but improving at similar rates, and Tesla's driving software improved more slowly, around 0.6 doublings a year (Metr, 2025c). METR writes that "in no domain we examined is progress clearly sub-exponential," and lists the limits. Every benchmark is "somewhat artificial" and likely easier than real tasks in its domain, many lack human baselines, and benchmarks may be chosen for domains where progress is fast (Metr, 2025c). The UK AI Security Institute applied METR's method to its own cyber tasks, with task lengths from expert estimates, and reports a doubling time of roughly eight months, "as an estimated upper bound" (UK AI Security Institute, 2025b). That is a second suite measured by a second group, though with the first group's method and without timed human attempts (inference).

The original paper ran its own checks. On SWE-bench Verified the trend was steeper, which the authors attribute to annotators' estimates understating how long the easiest tasks take contractors (Kwa et al., 2025). On a small set of the organization's own code changes, contractors took 5 to 18 times longer than the repository's maintainers, and agents performed as contractor times predicted, not as maintainer times did (Kwa et al., 2025). A horizon is therefore relative to whose time defines a task's length. Models also did worse on tasks the authors scored as "messier," on factors such as limited resources, novelty, or a changing environment, though the trend over time was similar on the messier subset (Kwa et al., 2025). A separate critique of agent benchmarks finds that many lack adequate holdout sets, which rewards agents that "take shortcuts and overfit to the benchmark" (Kapoor et al., 2024a).

The extrapolation is most sensitive to what changes the slope: which tasks enter the suite, whose times define their lengths, which success rate defines the horizon, and how the grader scores a success (inference). Constant factors, such as a uniformly harder task distribution, shift the projected date by months; a different slope shifts it by years. Whether horizon trends carry across domains and beyond the next few doublings is open, and the evidence does not decide it.

36.4 What a benchmark measures #

A score measures a property only if there is evidence that it tracks the property and not something else. Section 5.2.1 teaches the standard, construct validity: evidence that a score measures the construct, the trait it is meant to measure. It also teaches the test that guards it: a measure of one trait should agree with other measures of the same trait and come apart from measures of different traits. Section 5.2.2 reports what that test found when applied to safety benchmarks: many correlate highly with general capability, and the authors call the presentation of such capability gains as safety progress safetywashing, a term coined to criticize the practice (Ren et al., 2024b). This section adds three things that section did not: the measurement model that explains why benchmarks stop ranking models, what comparison across languages requires, and the reviews of how current benchmarks are built.

36.4.1 From psychometrics: items, abilities, and saturation #

From psychometrics. Item response theory is a measurement model of educational and psychological testing; Susan Embretson and Steven Reise's Item Response Theory for Psychologists is one introduction to it (Embretson and Reise, 2000). It treats each respondent as having an unobserved ability and each test item as having a difficulty and a discrimination, and it predicts the chance of a correct answer from the three. The time-horizon method is a version of its two-parameter model, as its authors state (Kwa et al., 2025). José Hernández-Orallo's book, read here in its abstract, proposes measuring natural and artificial systems on one footing and asks whether task difficulty can be derived intrinsically (Hernández-Orallo, 2017).

Write $\xi$ for a respondent's ability, the unobserved trait of Section 18.2.2, and give item $j$ a difficulty $b_j$ and a discrimination $\ell_j$, which plays the role of the loading in that section's factor model. The two-parameter model is

The time-horizon model ((36.2)) is the case $\xi = \log_2 h_{50}$, $b_j = \log_2 t_j$, and one slope for all items. The model says how precisely a test can locate a respondent, and the answer explains why a benchmark stops ranking models.

Saturation, a benchmark that frontier models nearly ace, is therefore a fact of measurement and not a failure of effort: items far below a respondent carry almost no information about it. The asymptotic standard error also understates the problem, since a respondent who answers every item correctly has no finite maximum-likelihood estimate at all. The same arithmetic governs time horizons. A horizon is located by tasks of about that length, and the authors note that accurately measuring a horizon of $t$ minutes "requires many tasks of human length $t$" (Kwa et al., 2025). METR's second analysis found the slope relating task length to success weakening on two benchmarks "as the benchmark saturates," so that the best models' horizons there "cannot be robustly measured" (Metr, 2025c). The 270-hour estimate of Table I.8 lies, by METR's own account, beyond the range its suite measures reliably (Metr, 2026b). Figure 36.2 computes the standard error for an illustrative suite.

Two remedies follow from the model (inference). Items must be added at the level of the respondents to be measured, which benchmarks that release harder tasks over time do (Section 36.5). And a benchmark's discrimination is limited by its labels. Samuel Bowman and George Dahl argued in 2021 that crowdsourced benchmarks give "no guarantee" that they will detect improvements. There is, they wrote, no reason to expect such a benchmark to "meaningfully distinguish between systems with superhuman metric performance" (Bowman and Dahl, 2021).

36.4.2 Comparison across languages #

Section 18.2.2 derives measurement invariance, the condition under which a test measures the same trait in the same way in two groups, a concept William Meredith's 1993 paper defines (Meredith, 1993). An item must load on the trait equally, and give equal answers at equal trait levels, in both groups. In item response terms, an item's discrimination and difficulty must be the same in each language for a difference in scores to mean a difference in the trait (inference). Section 26.6.2 reports why the question matters for safety, whose level itself differs by language: several groups found that safety training carries poorly to languages with little text in pretraining (Yong et al., 2023; Deng et al., 2023; Wang et al., 2024a). And only a small share of the world's languages are represented in language technology at all (Section 17.4.3) (Joshi et al., 2020).

For evaluation, the question is what a suite in a few languages licenses. It licenses a claim about those languages. A claim about others is an extrapolation of the kind Section 5.8 examines, and the measured gradient says it fails in the direction of less safety where there is less text (inference). Translating a suite does not settle the matter. Translated items share their source, so they are related rather than independent observations (Section 36.7), and a benchmark built in another language may measure a different construct. Kexin Huang and colleagues, at the Shanghai Artificial Intelligence Laboratory and Fudan University, built a Chinese benchmark of value alignment with "a unique morality dimension that integrates specific Chinese values such as harmony." All 17 models they evaluated performed relatively poorly, especially on safety and fairness (Huang et al., 2024b). A safety score compared across languages therefore needs agreement on the construct as well as invariance of the items. As of October 2026 we found no test of measurement invariance for any safety evaluation across languages.

36.4.3 From measurement science: how benchmarks are built #

From measurement science. The newer measurement science of AI evaluation asks whether benchmarks are built to support the claims made with them. Section 5.2.4 reports its framework, which separates a concept, its explicit definition, the instrument, and the measurement (Wallach et al., 2025). Two reviews then measured practice. Andrew Bean and colleagues, in a paper at NeurIPS 2025, had 29 expert reviewers code 445 benchmark papers from the main machine learning and natural language processing conferences between 2018 and 2024 (Bean et al., 2025). They found "patterns related to the measured phenomena, tasks, and scoring metrics which undermine the validity of the resulting claims" (Bean et al., 2025); Table 36.4 gives the rates.

The review has its own limits, which its authors list: it covers peer-reviewed conference papers and so misses benchmarks that developers release without review, a model screened the papers before people did, and few reviewers coded each paper (Bean et al., 2025). A second review, by Anka Reuel and colleagues at Stanford University in a paper at NeurIPS 2024, scored 24 widely used benchmarks against 46 criteria for design, implementation, documentation, and maintenance (Reuel et al., 2024a). Fourteen of the 24 did not run a model more than once or report the statistical significance of their results, and 17 gave no easy way to replicate the results their papers reported. MMLU, a widely reported benchmark of general knowledge, scored lowest (Reuel et al., 2024a). Bowman and Dahl had set four criteria a benchmark should meet: a valid test of the phenomenon, consistently labeled data, adequate statistical power, and no incentive toward systems with harmful biases (Bowman and Dahl, 2021).

What the reviews imply for safety evaluation is that most of the field's instruments have not made the argument a safety decision needs (inference). A safety benchmark is a benchmark, subject to the same defects, and Bean and colleagues found that alignment benchmarks often target phenomena whose definitions are contested (Bean et al., 2025).

36.4.4 Where the analogy stops #

Psychometrics assumes a respondent whose ability holds still while it is measured. A model's measured ability moves with the elicitation method, so in (36.3) the ability is a function $\xi(u)$ of the method. The evaluator's estimate is $\xi(\hat u)$ for the best method $\hat u$ it tried, a lower bound on what any method could reach (inference). That ties validity to Section 36.2: the construct a capability evaluation names is the most the model can do, while the score measures what one method drew out. Section 5.2.4 draws the other consequence, that one model under two scaffolds can behave as two respondents. A second difference concerns calibration. Item response theory usually estimates item difficulties from the respondents themselves, so one respondent's score can depend on who else was tested. The time-horizon method avoids this by fixing difficulty from human time (Kwa et al., 2025), which ties every horizon to the skill of the people timed.

36.5 Score inflation and contamination #

A model's score on a benchmark can rise because the model got better at what the benchmark samples, or because training brought it closer to the benchmark itself. Section 15.7 teaches the method for telling the two apart, the audit test, and its use on benchmarks, and leaves one question open: whether a gain that the audit does not confirm is gaming or a legitimate narrowing toward what is tested. This section takes that question up, and asks what an evaluation program can build in to detect inflation while a benchmark is in use.

36.5.1 From education measurement: inflation and reallocation #

From education measurement. The discipline's standard case is Texas in the 1990s. A RAND study by Stephen Klein and colleagues compared gains on the state's high-stakes test, the TAAS, whose results carried consequences for schools and students, with gains on the national assessment, NAEP, which carries no stakes for schools (Klein et al., 2000). Between 1994 and 1998, fourth-grade reading scores on NAEP rose by 0.13 standard deviations for white students and 0.15 for students of color, while the TAAS showed effect sizes (gains measured in standard deviations) of 0.31 to 0.49, "several times larger" (Klein et al., 2000). The gap between Black and white students on NAEP widened slightly over the period, while on the TAAS it shrank to about half its NAEP size (Klein et al., 2000). The authors note that NAEP is not a perfect measure, since students' motivation may differ when nothing rides on the result. They quote Robert Linn: "Divergence of trends does not prove that NAEP is right and the state assessment is misleading, but it does raise important questions about the generalizability of gains" (Klein et al., 2000). Section 15.7.1 reports the Chicago study that located such gains on the material the high-stakes test emphasized, and Daniel Koretz's distinction among seven kinds of test preparation, from working more effectively to cheating (Jacob, 2005; Kleinberg and Raghavan, 2020).

The two sides of the debate can be stated in the field's own terms. Jon Kleinberg and Manish Raghavan, who quote Koretz's list in a model of how evaluation rules shape effort, put the case against calling every improvement gaming: "some forms of effort correspond intuitively to gaming, while others correspond to self-improvement" (Kleinberg and Raghavan, 2020). On the other side, a gain that does not appear on a second test of the same domain does not support a claim about the domain, which is what the Texas comparison shows (Klein et al., 2000).

The two sides separate once a gain is split three ways (inference). Part of a gain on a test can reflect the domain the test samples, and then it appears on fresh items drawn the same way. Part can reflect reallocation, effort moved to the parts of the domain the test covers at the expense of the parts it does not; it appears on fresh items like the test's and not on the rest of the domain. And part can be inflation in the narrow sense, gains on the test's own items and formats that appear nowhere else. An equivalent audit test separates the third part from the first two. Separating reallocation needs a measure of the untested parts. Whether reallocation counts as gaming depends on the claim: it is real learning for a claim about what the benchmark samples, and it is not for a claim about the broader capability the benchmark is named for.

36.5.2 The machine learning form #

For benchmarks the three parts have direct counterparts (inference). Test items in the training data produce inflation in the narrow sense. Training data written in a benchmark's style can produce reallocation, gains on items like the benchmark's that do not reach the rest of the domain. A matched new set of grade-school arithmetic problems found some model families scoring lower than on the original, with gaps that correlated modestly with how readily a model reproduced the original's text (Zhang et al., 2024a). A rebuilt image benchmark suggested that such a drop can also come from items that are slightly harder, so the new set must be shown equivalent before a drop means inflation (Section 15.7.2) (Recht et al., 2019).

An evaluation program can build the audit in, as education measurement learned to. Koretz and colleagues tested self-monitoring assessments, which place audit items inside the high-stakes test; their abstract reports that the audit component "functioned as expected in many of the trials," with "a number of null findings" (Koretz et al., 2016). Benchmarks have three counterparts. The HCAST task suite releases 11 example task families and withholds the rest "to prevent data contamination, and to reduce hill-climbing," the repeated tuning of methods against a public test (Rein et al., 2025). LiveBench, a benchmark by researchers at Abacus.AI, New York University, Nvidia, and other institutions, published at ICLR, adds questions from recent sources every month and scores them against objective answers without a model judge. It releases harder versions of tasks over time so that it can keep distinguishing models; at release, top models scored below 70 percent (White et al., 2024). And BetterBench's criteria include marking test items with a unique identifier, so that their presence in training data can be detected, or encrypting them, so that crawlers do not collect them (Reuel et al., 2024a). A benchmark that keeps unpublished items beside its public ones, refreshed on a schedule, is the education audit in machine form (inference).

Weak graders inflate scores as contamination does. Section 24.2.2 reports that test-based coding benchmarks accept a share of patches that do not fix the issue, and that agents pass tests built to be impossible (Wang et al., 2025a; Aleithan et al., 2024; Zhong et al., 2025). A score inflated by weak tests is the evaluation-side twin of reward hacking against the same tests (Section 24.5): the same flaw that pays an exploit in training credits it in evaluation (inference). Section 15.7.3 sets out where the education analogy stops. A school cannot rehearse an audit it has never seen, while a model trained on the public web may have seen any public test, and a model that tells an audit from ordinary use defeats the design.

36.6 Passing for the wrong reason #

A model can pass an evaluation for a reason that has nothing to do with the trait: a cue in the item that predicts the answer, or a cue that moves the grader. The problem is old, and the method that exposed its most famous case is a design an evaluation pipeline can adopt.

36.6.1 From animal learning: the blind design #

From animal learning. Section 5.4.1 reports how Oskar Pfungst exposed Clever Hans, a horse that seemed to do arithmetic: when no one present knew the answer, the horse failed, because he had been reading his questioners' unintended movements (Pfungst and Rahn, 1911). Section 30.4.2 reports a modern controlled case, in which detection dogs' alerts followed where their handlers had been falsely told targets were, in rooms that held no target at all (Lit et al., 2011). Both studies, and a review of hundreds of experiments on expectancy effects (Section 5.4.1) (Rosenthal and Rubin, 1978), teach the same procedure: vary what the observer knows while holding the task fixed, and see whether performance follows the task or the observer.

In an evaluation pipeline the observer is the grader, a person or a model that scores the response. The procedure implies that the grader should not know which model produced a response, which experimental condition it came from, or which result the evaluator hopes for (inference). One difference from Pfungst's control matters. His questioners were blinded to the answer itself, while a grader often grades better with it. In the study below, giving a model judge a reference answer cut its failures on math questions from 14 in 20 to 3 in 20 (Zheng et al., 2023a). What blinding must remove from a grader is the expectation, not the answer key (inference).

The analogy stops in two places. Hans read a person; a model can also read cues in the items themselves, which no blinding of the grader removes. Section 31.2.2 reports language models that scored well on inference and argument benchmarks through word-overlap heuristics and spurious statistical cues (McCoy et al., 2019; Niven and Kao, 2019), and Section 8.4.2 an image classifier that recognized horses by a source tag (Lapuschkin et al., 2019); only varying the items exposes such cues (Section 31.4.2). And a model grader may recover what blinding hides, because a response carries marks of the model that wrote it, as the next subsection reports.

36.6.2 Cues for the grader #

Model graders are common: Bean and colleagues found one in 17.1 percent of the benchmarks they reviewed (Bean et al., 2025), and the persuasion study of Section 36.1.1 used GPT-4o to grade GPT-4o among others (Kowal et al., 2025). An early systematic study of such judges, by Lianmin Zheng and colleagues at UC Berkeley and other universities in a paper at NeurIPS 2023, compared GPT-4 and other models with human judges of chat responses (Zheng et al., 2023a). On pairs where neither side was judged a tie, GPT-4 agreed with human experts 85 percent of the time, more than the experts agreed with each other, at 81 percent (Zheng et al., 2023a). The same study measured the judges' biases (Table 36.5). A later study, by Arjun Panickssery, Samuel Bowman, and Shi Feng at MATS, New York University, and Anthropic, in a paper at NeurIPS 2024, asked whether judges favor their own text because they recognize it (Panickssery et al., 2024).

The strongest results come with their authors' limits. Zheng and colleagues studied helpfulness and say their paper "largely neglects safety" (Zheng et al., 2023a). Panickssery and colleagues studied news summaries only, and did not control for the quality of the summaries, so part of a judge's preference for its own text could be justified; they offer "initial evidence" for the causal claim (Panickssery et al., 2024). One finding in the table bears directly on Clever Hans. A judge that could solve a math problem on its own was "misled by the provided answers," the answers it was asked to grade (Zheng et al., 2023a): the respondent cued the grader, the reverse of Hans's direction.

A judge from the tested model's own family is the correlated case of Section 6.7.4: two checks that share blind spots give less assurance than two independent ones. A judge that recognizes and favors its own family's text adds a bias in the same direction (inference). The procedures that follow are the blind design adapted to model graders (inference). Present answers in both orders and count only verdicts that survive the swap, as Zheng and colleagues did (Zheng et al., 2023a). Give the grader the reference answer and withhold the condition and the hypothesis. Prefer a judge from a different family, and assume it may still recognize authorship. And audit the grader itself, by planting responses of known quality and checking its verdicts against blinded human ratings on a sample, which is the grader testing that Section 24.2 describes for reward. Benchmarks that can avoid judges do: LiveBench scores only questions with objective answers for this reason (White et al., 2024).

36.7 The statistics of an evaluation #

A reported score is a measurement with an uncertainty, and the uncertainty decides whether two models differ, whether a threshold was crossed, and how many questions an evaluation needs. Section 6.5.4 teaches the starting point: treat a benchmark's questions as a sample from a larger population of possible questions. A score of $\hat p$ on $n$ questions scored 0 or 1 then has a standard error of about $\sqrt{\hat p(1 - \hat p)/n}$ (Miller, 2024). That formula assumes the questions were drawn independently. Most evaluations violate the assumption, and this section treats what follows: questions that share a source, comparisons between models, and the number of questions a comparison needs.

36.7.1 Questions that share a source #

Evan Miller, in a 2024 preprint written at Anthropic, gave the field's standard tools for these cases (Miller, 2024). Many benchmarks draw questions in groups: several reading-comprehension questions about one passage, or one grade-school math problem translated into many languages. Questions in a group tend to be answered right or wrong together, so they carry less information than the same number of independent questions, and Miller gives the clustered standard error that accounts for it (Miller, 2024). Computed on Anthropic models, the clustered standard error was 3.05 times the naive one on DROP, a reading benchmark, 1.10 times on RACE-H, another, and 1.88 times on MGSM, the translated math benchmark (Miller, 2024). Miller concludes that confidence intervals one developer had published for reading benchmarks are "likely anti-conservative (too narrow)" (Miller, 2024). The size of the correction follows from one number, which the derivation introduces.

Translated items are a cluster in exactly this sense (inference). A suite that asks one question in ten languages has measured one item ten times through ten channels, and what it says about each language is limited by how much the languages differ, which is the invariance question of Section 36.4.2.

36.7.2 Comparing two models #

Most evaluations exist to compare. Two models scored on the same questions should be compared question by question, because the variance of the difference falls by twice the covariance of their per-question scores. In Miller's words, "paired differences represent a 'free' reduction in estimator variance," since models tend to agree on which questions are hard (Miller, 2024). With equally variable per-question scores correlated at 0.5, pairing halves the variance of the difference; Miller's text says a third, but his own formulas take it from 1/6 to 1/12 (Miller, 2024). Miller's worked example, with two fictional models, shows what is at stake: one model led on two benchmarks of three, but paired, clustered analysis found only the third difference statistically significant, reversing the superficial reading (Miller, 2024). Two further recommendations concern noise inside a question. Sampling each answer several times and averaging reduces the variance that comes from the model's randomness. Lowering the sampling temperature to reduce that variance is a mistake, Miller argues, because it changes the behavior being measured and can increase the variance that comes from the choice of questions (Miller, 2024).

The same arithmetic plans an evaluation before it runs. To detect a difference $\delta_{\min}$ between two models with false positive rate $\alpha$ and miss rate $\beta_{\text{miss}}$, a paired comparison needs about

independent questions, where $\sigma_d^2$ is the variance of the per-question difference in scores and $\Phi^{-1}$ is the inverse of the standard normal distribution function (Miller, 2024). With values Miller calls fictional but reasonable, detecting a 3-point difference 80 percent of the time at a 5 percent false positive rate needs about 969 questions, and he suggests that new evaluations contain at least 1,000 (Miller, 2024). His variance of 1/9 carries the slip above; 1/12 gives about 727. Clustering multiplies the requirement by the design effect (inference). Few benchmarks plan this way: only 16.0 percent of the papers Bean and colleagues reviewed used uncertainty estimates or statistical tests at all (Bean et al., 2025).

36.7.3 Rare events and what to report #

For a propensity evaluation that observes nothing, Section 6.5.1 gives the bound: a clean record of $n$ independent episodes is consistent with a rate as high as about $3/n$. Two assumptions carry it (Section 6.5.3). The episodes must be independent, and the design effect says that a clean record of episodes built from a few templates is worth fewer episodes than it counts (inference). And they must be drawn like the episodes of deployment, which a model that recognizes tests would break (Chapter 37). One developer deliberately samples safety sets near its policy's boundary, where about half the conversations it selects are disallowed, against perhaps one in 20,000 at random (Etinger, 2026) (developer post), so a rate on such a set describes hard cases, not use (inference). For rates too small to observe at all, Section 6.6.4 reports a developer's method of forecasting deployment-scale risk from the probabilities a model assigns to individual queries (Jones et al., 2025).

Reporting decides whether anyone else can check a number. Ryan Burnell and colleagues argued in Science in 2023 that "aggregate metrics make it hard to predict how a system will perform in a particular situation," and that instance-by-instance results are "rarely made available"; they ask that both change (Burnell et al., 2023). The time-horizon measurements of Section 36.3 show the alternative in practice: their intervals come from a bootstrap that resamples task families, tasks, and attempts, which respects the clusters (Kwa et al., 2025). A reported number that is to carry a decision needs at least the model as its source names it, the date, the prompt, and the scaffold. It also needs the elicitation effort, the number of questions and of clusters, and the paired differences against any model it is compared with (inference).

36.8 Who evaluates, and the science of evals #

Evaluations are run by developers on their own models, by outside organizations under agreements with developers, by government institutes, and by academic groups on public or open models. Who runs them decides what access the evaluator has, what it may publish, and how much a reader should trust the result. This section asks what would make evaluation results reliable enough for the decisions that rest on them.

36.8.1 The call for a science of evaluation #

The call came first from evaluators. Apollo Research argued in a 2024 post that "if AI model evaluations (evals) want to have meaningful real-world impact, we need a 'Science of Evals'." Its reason: "without confidence in the rigor of the evaluation process, people won't feel confident in their results" (Apollo Research, 2024). A 2025 preprint by Laura Weidinger and colleagues, from Google DeepMind, Microsoft Research, Anthropic, Hugging Face, and several universities, drew three lessons from the evaluation of transportation, aerospace, and pharmaceuticals. In their words, "evaluation metrics must be applicable to real-world performance, metrics must be iteratively refined, and evaluation institutions and norms must be established" (Weidinger et al., 2025). Those fields, they add, pair testing before deployment with monitoring after it, because some harms emerge only at the point of use (Weidinger et al., 2025). An earlier framework from the same lead author separates the evaluation of capability, of human interaction, and of systemic impact (Section 1.1.5) (Weidinger et al., 2023a). And a review of about 100 studies of benchmarking, by Maria Eriksson and colleagues at the European Commission's Joint Research Centre, found systemic flaws: "misaligned incentives, construct validity issues, unknown unknowns, and problems with the gaming of benchmark results." It describes practice as shaped by "cultural, commercial and competitive dynamics" (Eriksson et al., 2025).

Shared tools are part of such a science. The UK AI Security Institute, with Meridian Labs, maintains Inspect, an open framework in which an evaluation is a dataset of samples, a solver that produces answers, and a scorer that grades them. It ships over 200 prebuilt evaluations and sandboxes for running a model's code safely (UK AI Security Institute, 2024). The solver is the elicitation method of Section 36.2 and the scorer the grader of Section 36.6, so a shared framework can make both visible in every report (inference). The noise-injection test of sandbagging was implemented in it for outside evaluators (Tice et al., 2024).

36.8.2 Access, and what an outside evaluation can carry #

An outside evaluator sees what the developer lets it see. Section 5.6.5 reports the arguments that outside researchers need legal and technical safe harbors and channels to report flaws (Longpre et al., 2024a; Longpre et al., 2025), and Section 5.6.2 the 2025 pilot in which two developers ran alignment evaluations on each other's public models (Anthropic, 2025x). As of June 2026, METR's outside evaluation of Section 36.3.3 had access to a final checkpoint and the raw chain of thought, under a nondisclosure agreement and with the developer's review of the post (Section I.4.1) (Metr, 2026b). METR adds that confirming a model is not "just learning to be more successful at evading the monitoring system" is "impossible to validate in a traditional pre-deployment evaluation paradigm, as it requires deep access to internal systems" (Metr, 2026b). Who may audit, and with what access, is Chapter 38's subject.

36.8.3 From scores to decisions #

Evaluations enter decisions through the thresholds of safety frameworks (Chapter 62) and through safety cases that argue a model cannot or will not cause a catastrophe (Chapter 52). Education measurement supplies the form such an argument should take. Michael Kane's argument-based approach, read here in its abstract, holds that "it is the proposed score interpretations and uses that are validated and not the test or the test scores." It adds that "more-ambitious claims require more support," and that validating an interpretation "does not validate the score use" (Kane, 2013). An inability claim built on an evaluation is such a chain, and this chapter has located a weak link at each step (inference). From response to score, the grader can be cued or fooled (Section 36.6). From score to the benchmark's domain, the items can be contaminated, the questions clustered, and the error bars missing (Section 36.5, Section 36.7). From the domain to what the model could do under any method, the elicitation gap intervenes (Section 36.2). And from the test to deployment, the model may tell the two apart (Chapter 37).

The last link concerns disclosure. A system card's statement that no dangerous capability was found is a claim of absence by the party with most at stake, and Section 5.6.2 gives it the same provenance marking as a claim of presence. The elicitation gap adds a specific reason: an absence claim is worth the elicitation effort behind it, and a report that does not state that effort cannot be read as an upper bound (Section 36.2) (inference).

36.9 What evaluation evidence means for delegation #

The book's thesis is that alignment is a delegation problem under an asymmetry of competence and visibility (Chapter 1). An evaluation is the principal's instrument for reducing the asymmetry of visibility, and the chapter's results say how far it does (inference). In the economics of delegation, the agent's private type, what only the agent knows about its own abilities and aims, is what screening and testing try to reveal (Section 14.1.3). A capability evaluation is such a test, and sandbagging is the agent concealing its type. Education measurement supplies the institutional record, in which an agent rewarded on a measure raises the measure faster than the competence it was meant to show (Section 36.5). The structures converge.

Two things have no counterpart, and both belong on the list of new features the synthesis weighs (Section 69.4) (inference). The principal can rewrite the agent to reveal its type. Training on a single demonstration removed a model's underperformance where prompting could not reliably do so (Taylor et al., 2025a), and no employer can retrain a job candidate to show what the candidate could do. And the respondent may have read the test, in training data that contains the benchmark or descriptions of it, which no human examinee does at this scale. The first favors the principal; the second favors the agent, and the next chapter measures it.

The claims divide among the three relationships. Capability and autonomy evaluations bear on the developer and the public: they set the thresholds on which release decisions turn, and for a model whose weights are not released only the developer holds the checkpoints, the fine-tuning access, and the internal monitoring that the strongest evaluations need. Propensity evaluations bear on the deployed system and the people it serves, and they inherit the resemblance condition of Section 36.1. The training relationship enters through the graders that score both training and evaluation, whose flaws credit the same exploits in each (Section 36.5). What the public can check today is an outside evaluator's report, published after the developer's review, on a model the developer chose to share (inference).

36.10 Settled, contested, missing #

36.11 Exercises #

Further reading #

• Shevlane et al. (2023): why capability and alignment evaluations are separate instruments, and where each enters a decision.
• Davidson et al. (2023): why an evaluation that skips elicitation underestimates, in a unit that makes enhancements comparable.
• Kwa et al. (2025): the time-horizon method, the trend it measured, and the limits its authors state.
• Messick (1995): validity as an argument about what scores mean and how they are used.
• Klein et al. (2000): score inflation measured against an audit test, the comparison benchmarks can borrow.
• Wallach et al. (2025): evaluation of generative systems as a measurement problem of the social-science kind.