Draft complete but not yet reviewed; expect revisions

What Inspection Can and Cannot Verify

The previous chapters reported what interpretability can find: features in superposition and the dictionaries that approximate them (Chapter 44), circuits and attribution graphs (Chapter 45), and directions that read and steer concepts (Chapter 46). Each finding is a claim that something is present. Verification needs a claim that something is absent: that the model has no hidden objective, does not deceive under conditions the tests did not cover, and does not know more than it says. A developer arguing that a model is safe to deploy needs claims of that kind, and so does a regulator checking the argument. This chapter asks how far inspecting a model's internals can support them.

The answer turns on three questions the earlier chapters left open. Is a correct explanation of a network unique? What would turn an explanation into a certificate that someone else can check? And what does mathematics rule out in the worst case? The chapter takes them in that order and then sets out the debate that all of Part VII feeds, whether interpretability can carry safety claims. The evidence does not decide that debate, and the chapter ends by naming the result that would. Most claims here concern the deployed system and the people it serves. Inspection during training concerns the training process and its signal, and whether anyone outside a developer can check the developer's inspection concerns the developer and the public.

47.1 What a verification by inspection would claim #

A common claim is that interpretability will let a developer read deception or a misaligned goal directly off a model's internals, as a scan reads a tumor off a body. The hope has a precise statement, and seeing what it requires is the first step.

In a 2023 note on future goals, Chris Olah of Anthropic described the developer's "north star" for safety: "we want to make claims like 'a model will never lie' or 'a model will never manipulate us'" (Anthropic, 2023e). He saw a possible path if superposition and scale could be solved: "Having a complete set of features might give one a hook to enumerate over, which gets at 'all model behavior' in some sense" (Anthropic, 2023e). The note calls the idea "vague, highly speculative" and the direction one with "a significant chance of not working" (Anthropic, 2023e). In 2025 Anthropic's chief executive, Dario Amodei, wrote of a "true 'MRI for AI'" and set the developer a goal of "interpretability can reliably detect most model problems" by 2027. He bet on interpretability reaching that point within 5 to 10 years (Amodei, 2025). Both are a developer's essays about its own research program, and the scan metaphor is the developer's. A 2023 forum post by Charbel-Raphaël Segerie had already criticized the plan of enumerating every feature and inspecting each one, which he attributes to Neel Nanda as enumerative safety (Segerie, 2023). He called it "doomed from the start", partly because a feature "is not inherently good or bad" and danger lies in how features combine (Segerie, 2023).

47.1.1 Presence and absence #

The two halves of such a claim differ in kind. That a model has a hidden objective is an existential claim: some input exists on which the objective shows, or some internal feature exists that represents it and that the model uses. It comes with a witness, the input or the feature with an intervention showing its effect, and anyone with access can check the witness by rerunning the model. That a model has no hidden objective is a universal claim, over every input that matters and every way the network could encode the objective. Nothing an auditor exhibits is a witness to it. In the words of the paper that Section 47.5 reports, a proof "requires ruling out the possibility of any other correlations or coincidences" (Christiano et al., 2022). Table 47.1 lists the routes by which an absence claim could still be supported, and the sections of this chapter take them up in turn.

The first route is the one audits use, and Section 38.2.3 derives what it licenses: after a clean audit, the probability of a hidden objective depends on the audit's miss rate, which only validation studies measure ((38.1)). A clean record from sampling inputs has a similar limit. Section 6.5.1 shows that a record of no failures in $n$ independent trials bounds the failure rate at about $3/n$ and cannot show it to be lower. A deliberately planted behavior can sit below any such bound. Shafi Goldwasser and colleagues, whose results Section 47.6 teaches, note that for a planted backdoor to escape notice, "the backdoored inputs must have vanishing density" in the data distribution (Goldwasser et al., 2022). Sampling then bounds how often the behavior occurs on that distribution, not whether it exists (inference).

The second route needs completeness. An explanation can rule a mechanism out only if it accounts for all of the computation relevant to a behavior, on every input that matters. Section 45.8 names the circuit-level form of this requirement, claims about a whole input distribution rather than one prompt, as its crux; Section 44.8 names the dictionary-level form, a canonical set of features to enumerate. As of October 2026 neither has been met for a large model.

47.1.2 Whom an absence claim serves #

The same claim does different work in each of the book's relationships. Inspecting a training run for dispositions no one intended concerns the training process and its signal. Auditing a model before release concerns the deployed system and the people it serves. Showing the public that the developer's inspection was sound concerns the developer and the public, and it needs something the first two do not: a party other than the developer able to check the claim (Chapter 67). A developer's finding about its own closed model reaches everyone else as testimony (Section 43.7.3).

The book listed a principal that reads its agent's activations among the results with no counterpart in the older theories of delegation, fixed in advance as the thesis's negative space (Section 1.7). The rest of this chapter states what that power has and has not delivered, as an input to Chapter 69.

47.2 Correct is not unique #

An absence claim built on an explanation assumes that the explanation found is the explanation: that if it contains no hidden objective, none is hidden. This section reports the results showing that a correct explanation need not be unique, and asks what a passing test therefore licenses.

The standard of correctness is causal abstraction, taught in Section 43.5. An explanation is a small causal model whose variables are paired with parts of the network by an alignment, and it is correct when intervening on a variable changes the output as intervening on the paired parts does. The test swaps a variable's value between two inputs, an interchange intervention, and the share of swaps on which network and explanation agree is the interchange intervention accuracy (IIA) of (43.2). The test is meant to tell an explanation that is faithful, true to how the network computes, from one that is merely plausible, convincing to a reader (Jacovi and Goldberg, 2020). Figure 43.1 already showed two algorithms passing on one hand-built network; the results below show how general that is.

47.2.1 Many explanations pass #

Maxime Méloux and colleagues at Université Grenoble Alpes, in a paper at ICLR 2025, asked whether the field's own criteria single out one explanation (Méloux et al., 2025). They borrowed identifiability from statistics, where a model is identifiable if its parameters can be inferred uniquely from data, and called an explanation identifiable when a unique one meets the criteria. They trained multilayer perceptrons, the simplest feed-forward networks, on Boolean functions such as XOR, keeping them small enough that every candidate explanation could be enumerated. They tested both of the field's strategies: find a subnetwork (a circuit) that reproduces the behavior and then interpret it, or propose an algorithm and then search for where it is aligned (Méloux et al., 2025).

In one network trained to compute XOR, with two hidden layers of three units, testing only two candidate algorithms found 159 perfect minimal alignments, with perfect alignments for both algorithms. The circuit search found 85 circuits with perfect accuracy and an average of 535.8 interpretations each (Méloux et al., 2025). Across networks whose hidden layers had 2 to 5 units, the median number of explanations grew from 38 to 910,000 for the circuit-first strategy and from 8 to 3,700 for the algorithm-first one (Méloux et al., 2025). Fewer than 2 percent of the networks had exactly one valid minimal alignment, and none had exactly one circuit interpretation (Méloux et al., 2025). The circuit counts are lower bounds, since only sparse circuits were enumerated. In a larger network trained to tell handwritten zeros from ones, the last layers alone held 3,209 valid circuits, so the whole network has either none or at least that many (Méloux et al., 2025).

The authors do not conclude that such explanations are useless. On a pragmatic view, they write, "unicity is not essential if the explanations meet functional goals such as predictivity, controllability, or utility in decision-making" (Méloux et al., 2025). If unicity does matter, current criteria cannot guarantee it, and "simplicity or sparsity cannot single out one explanation" in their experiments (Méloux et al., 2025). They also note the limit of toy networks, and add that if the problem disappears at scale, why it does "must be demonstrated" (Méloux et al., 2025).

47.2.2 Any network, any algorithm #

A second result explains why so many pass. Denis Sutter and colleagues at ETH Zürich and EPFL, in a paper at NeurIPS 2025, observed that most causal-abstraction work restricts alignments to linear maps (Sutter et al., 2025). They follow the linear representation hypothesis that concepts are directions in activation space (Section 46.1), while the definition of causal abstraction requires no such restriction (Sutter et al., 2025). They proved that once the restriction is lifted, "under reasonable assumptions, any neural network can be mapped to any algorithm" (Sutter et al., 2025). The assumptions are that the inputs form a countable set, as strings of tokens do, and that every layer's activations determine the input. They also need every layer to be able to reach every output, require that the algorithm's steps can be matched to the network's layers, and assume that the network solves the task (Sutter et al., 2025). The derivation below shows the idea in its simplest case.

The experiments show that such maps can be learned, not only proved to exist. On the indirect-object task of Section 45.2.2, nonlinear alignment maps reached perfect IIA in randomly initialized Pythia models of every size tested, models that cannot do the task, while linear maps approached perfect alignment only in the fully trained model (Sutter et al., 2025). On a hierarchical equality task, nonlinear maps erased the pattern across layers that linear maps had found, and exceeded 80 percent IIA even in randomly initialized networks (Sutter et al., 2025). Linear maps do not trivialize the test, and the authors are inclined to read their curves as support for linear representations, but write that they "cannot make this intuition formal" (Sutter et al., 2025). Their proof relies on "a form of overfitting", and when the names in the training and test sets were disjoint, a learned map failed to generalize (Sutter et al., 2025).

The framework's authors had chosen the flexibility on purpose. Atticus Geiger and colleagues write that the linear representation hypothesis is "too restrictive" and that an ideal theory would "support any and all decompositions" (Geiger et al., 2025). Whether a decomposition is useful, they add, "should be understood as an empirical hypothesis that can be falsified through experimentation" (Geiger et al., 2025). Read against that position, the theorem does not refute causal abstraction. It shows that the class of permitted maps is the hypothesis under test, and that a test permitting every map tests nothing (inference).

47.2.3 What a passing test licenses #

A third gap is the distribution on which an explanation is tested. Dan Friedman and colleagues at Princeton and Google, in a paper at ICML 2024, simplified small transformers trained on balanced-parenthesis languages and on code completion, with tools such as principal components and clustering (Friedman et al., 2024). The simplified proxies matched the models in distribution and diverged on tests of systematic generalization, sometimes generalizing worse than the model and sometimes better (Friedman et al., 2024). Section 43.4.3 reports three related illusions: patching a subspace can activate a dormant pathway (Makelov et al., 2024a), where a fact is localized does not predict where editing it works (Hase et al., 2023), and circuit scores change with the ablation used (Miller et al., 2024). Section 45.5.5 reports a replacement model containing a mechanism the model lacks (Anthropic, 2025b), and Section 44.6 dictionaries that differ across seeds and sizes. Table 47.2 sets each claim an explanation can make beside the test that licenses it.

For an absence claim the consequence is direct. An explanation in which no objective appears is one of many that pass, and another that passes as well might contain it (inference). A restriction that narrows the field, such as linear maps, is an assumption about how the network represents information, which Section 46.1 weighs. The pragmatic standard of prediction and control is enough for finding a mechanism, where one working explanation suffices. It is not enough for ruling one out (inference).

47.3 From explanation to proof #

An explanation that only its author can evaluate does not help a regulator, an outside auditor, or the public decide whether to believe the author. Formal methods offer a different standard: an explanation good enough to yield a proof that someone else can check mechanically. This section asks what such a proof gives an outside party and how far the standard has been met for neural networks.

From formal methods. Formal methods, the branch of computer science that states in mathematics what a system must do and proves that it does, separates producing a proof from checking it. Section 28.6.2 teaches the idea through proof-carrying code: an untrusted producer ships a program with a proof that it obeys an agreed policy, and the host "can then easily and quickly validate the proof" without consulting anyone (Necula, 1997). The guaranteed-safe AI proposal of David Dalrymple and colleagues, which Section 28.6 teaches, carries the asymmetry over to AI systems: checking a compliance proof "can be done with just a few hundred lines of human-written code", so that humans need understand only the verifier (Dalrymple et al., 2024).

Three properties make a checked proof a certificate rather than testimony (inference). It is sound relative to its statement: if the checker accepts, the statement holds of the system checked. Checking costs far less than producing, so the party with less capability can do it. And what must be trusted is the checker and the statement, not the prover, so a developer's proof about its own model asks an outside party to trust a short program, not the developer. This is the property that the third relationship, the developer and the public, most lacks today (inference).

The first proofs built from interpretation are recalled from Section 45.6.4. Jason Gross and colleagues turned an understanding of small attention-only transformers into computer-assisted lower bounds on their accuracy, and found that shorter proofs needed more understanding (Gross et al., 2024). They named compounding structureless errors as the obstacle: parts of the network with no structure to exploit, whose worst cases add up until compact bounds become vacuous (Gross et al., 2024). For future work they suggest "relaxing from worst-case pessimal ablations to typical-case heuristic guarantees", citing the program of Section 47.5 (Gross et al., 2024).

A second route certifies the instrument rather than the model. Tobias Ladner and Matthias Althoff of the Technical University of Munich, in a September 2026 preprint, examined interpretable replacement networks, the sparse autoencoders and transcoders whose features many analyses read in place of a model's dense activations (Section 45.4) (Ladner and Althoff, 2026). For a sparse autoencoder of GPT-2, replacing one word with a synonym cut the overlap between the 20 most active features before and after, at the worst layer, by 73 percent on the authors' summary measure; a paraphrase written by another model cut it by 95 percent. Transcoders and the other models lost less. The pattern held across GPT-2, Gemma 2 2B, Gemma 3 1B, Llama 3.2 1B, and R1-Distill-Qwen 1.5B, and since these overlaps come from attacks they are upper bounds on how much survives (Ladner and Althoff, 2026). Their example shows what this does to an audit. On a clean sentence the dominant features would lead a feature-reading safety auditor to refuse a request; after a synonym swap they would lead it to let the request through, with nothing changed in the model (Ladner and Althoff, 2026).

They then used reachability analysis, a formal method that computes a set guaranteed to contain every value a network can produce from a set of inputs (Ladner and Althoff, 2026). With it they bounded the gap between each layer of GPT-2 and its replacement, over every input in a neighborhood built from synonyms of a sentence's words (Ladner and Althoff, 2026). The certified gaps were substantial at every layer, and later layers became less faithful, "arguably to a degree that does not allow for meaningful interpretations anymore" (Ladner and Althoff, 2026). Attacks found the gaps growing too, so the result is not an artifact of a loose verifier (Ladner and Althoff, 2026). Fine-tuning GPT-2's layer-6 sparse autoencoder with the verifier in view cut the certified bound by about 90 percent and left the replacement sparser. Even so, only 2 to 3 of the 20 most active features could be proved to survive the perturbations, which the authors attribute partly to GPT-2 itself, whose next-token prediction changed for 21 percent of single-synonym paraphrases (Ladner and Althoff, 2026). Each certificate covers one layer and one neighborhood; verifying the whole model by composing the layers is the authors' stated vision, not yet feasible (Ladner and Althoff, 2026).

What formal methods imply for inspection is a route by which a claim about a closed model could be checked by a party that does not trust the developer (inference). The developer publishes the statement and the proof, and the outside party runs the checker. Even then, the checked network must be shown to be the deployed one, a separate problem that Section 28.6.3 takes up.

Where the analogy stops is the statement. A proof certifies a formal property on a formally specified set of inputs. Gross and colleagues bounded accuracy over every input of a toy task; Ladner and Althoff bounded one layer's faithfulness on the neighborhood of one sentence. "Has no hidden objective" is a property no one has stated formally for any set of inputs, so the specification problem of Part III returns at the level of the property to be proved (inference). And proof length grows with the part of the network that has no structure to exploit (Gross et al., 2024), which ties this route to the question, taken up in Section 47.7, of whether a large network has a compact explanation at all.

47.4 Eliciting latent knowledge as a formal problem #

The routes so far ask the inspector to understand the model. A different statement of the verification problem asks only for a way to make the model say what it knows. Eliciting latent knowledge (ELK) is the problem of getting a model to report what it knows rather than what an overseer would believe, named in a December 2021 report by Paul Christiano, Ajeya Cotra, and Mark Xu (Christiano et al., 2021). Section 39.7 gives its setup. A predictor forecasts what a camera guarding a diamond will show, and a reporter is trained to answer questions from the predictor's internal state, on cases people can label by watching. Two reporters fit every label: a direct translator, which reports what the predictor's state says, and a human simulator, which reports what a person watching the camera would believe. Table 39.3 shows that they part only on the case a labeler cannot see, a robbery behind a tampered camera. This section states the argument formally, describes the search method the report built on it, and asks what it would take for ordinary training to find the direct translator.

Step 3 is underspecification, many models fitting the training data equally well and differing beyond it (Section 9.1), with a twist that makes it adversarial: the cases on which the fits differ are the ones the overseer most needs. Step 4 makes the twist worse, since honest labeling mistakes push training toward the reporter that will copy the labelers' blind spots. Weak-to-strong generalization (Chapter 42) asks the same question of a strong student trained on a weaker teacher's labels: whether it learns the teacher's errors or what the teacher meant.

The report's authors draw the consequence for inspection themselves. "Leveraging information beyond the behavior seems necessary," they write, "since many different reporters will receive a good loss and some of them will behave badly off distribution" (Christiano et al., 2021). Whatever separates the two reporters must be read from their parameters and computation, which makes ELK the formal case for inside access (inference).

47.4.1 A worst-case search #

The report turns the search for a solution into a game between a builder, who proposes a training strategy, and a breaker, who describes a world in which the strategy learns a bad reporter (Christiano et al., 2021). A counterexample counts only if the model understands the question, its answer is unambiguously wrong, and it knows that the answer is wrong, which the authors call the narrow version of ELK (Christiano et al., 2021). The goal is a training strategy with no counterexample, "despite trying hard", and a strategy is ruled out as soon as one case defeats it. The authors defend the worst-case standard because they want strategies that work "no matter how far we scale up our models or what they learn" (Christiano et al., 2021). It is the standard cryptography holds a scheme to, which counts as secure only if no adversary within the rules can break it (inference).

Table 47.3 gives the regularizers the report proposed and the counterexample it found to each. Of these, the report judged the counterexample to a computation penalty the weakest, saying it "offers much more hope", and left open whether a dataset could give the direct translator a large enough speed advantage (Christiano et al., 2021). A prize the group offered in early 2022 drew 197 proposals, which Section 39.7.1 reports (Christiano and Xu, 2022). Two of the published counterexamples use obfuscation: a bad reporter can be made deliberately hard to adapt to any other task, with indistinguishability obfuscation as the extreme case (Christiano and Xu, 2022). The same primitive returns in the backdoor theorem of Section 47.6.

Figure 47.1 draws the complexity counterexample. With correct labels, both reporters fit the labeled cases perfectly and the regularizer alone decides, so the crossing sits where translating the predictor's model costs as much as specifying the human's. Every labeling error moves the crossing toward smaller predictors, as (47.1) says. To the right of the crossing training selects the human simulator, and nothing in the labeled cases reveals it: the selected reporter says the diamond is present after the robbery.

How the worst case relates to ordinary training is an empirical question. Section 39.7.2 reads the probing record as tests of it. Probes on models fine-tuned to answer wrongly under a cue reported what the model knew on the easy side of the problem, while unsupervised methods learned whatever feature was most prominent (Mallen et al., 2023; Farquhar et al., 2023). Neither study reaches the cases the report constructs. Whether direct translation is cheap depends on whether a model's concepts converge on human ones, the natural abstraction hypothesis (Wentworth, 2021a), whose evidence Section 46.6 weighs. If they converge closely enough that translating costs less than simulating the human, the first row of Table 47.3 loses its counterexample for models like today's (inference).

47.5 Heuristic arguments and anomaly detection #

Proof is too strict a standard for a large network, and interpretation too informal to check. The Alignment Research Center (ARC), the group that posed ELK, has since 2022 pursued an object between the two, and aims it at ELK's worst case without solving ELK in full (Hilton, 2024a). This section teaches the object, its application, and what the group's own reports say about its status.

47.5.1 The presumption of independence #

Mathematicians routinely estimate what they cannot prove. Paul Christiano, Eric Neyman, and Mark Xu, in a 2022 preprint, start from twin primes, pairs such as 11 and 13 that are both prime (Christiano et al., 2022). A number near $N$ is prime with probability about $1/\ln N$, so treating "$x$ is prime" and "$x + 2$ is prime" as independent predicts about $N/\ln^2 N$ twin primes below $N$. The same reasoning predicts as many consecutive primes $x$ and $x + 1$, until one notices that one of any two consecutive numbers is even, a correlation that overturns the estimate completely. For twin primes the known corrections, starting from the fact that $x + 2$ is odd whenever $x$ is, raise the estimate by about 30 percent (Christiano et al., 2022). The default behind such reasoning, the estimate $\E[XY] = \E[X]\,\E[Y]$ in the absence of specific information about how $X$ and $Y$ are related, they call the presumption of independence: commonplace, often successful, and "completely informal" (Christiano et al., 2022).

A heuristic estimator would formalize it (Christiano et al., 2022). A proof verifier takes a statement and a putative proof and answers true, false, or undecided. A heuristic estimator takes a quantity and a set of arguments and always returns a best guess, which further arguments may revise. The paper asks that it behave like an expectation, respect proofs, ignore irrelevant arguments, and presume independence where no argument relates two quantities. The authors are "not aware of any estimator that satisfies them", and their main open problem is whether some estimator formalizes valid uses of the presumption "without also accepting spurious arguments" (Christiano et al., 2022).

One property is missing from the list on purpose. A valid argument can move even an ideal estimate the wrong way by chance, and "if someone searches for adversarially misleading estimates, they will usually be able to succeed" (Christiano et al., 2022). For inspection this is the limit that matters most, since the model under inspection, or whoever trained it, is the party best placed to supply misleading structure (inference). The paper places heuristic arguments beside the other two ways of analyzing a model (Table 47.4).

A 2024 preprint by Christiano and colleagues added a principle: an estimator should not be able to predict its own errors (Christiano et al., 2024). The principle is modeled on the law of iterated expectations, the rule that the expected value of a later estimate, given what is known now, is the estimate made now. Testing an objective version of the principle, accuracy over a distribution of problems, they showed that even combining two heuristic estimates linearly is #P-hard for one simple problem when the combination's errors must be exactly uncorrelated with both estimates (Christiano et al., 2024). That is as hard as counting the solutions of any problem in NP. Relaxed to approximate accuracy, the combination can be computed efficiently when the estimates do not overlap too strongly, and the general case is open (Christiano et al., 2024). From this and a second case study they concluded that this approach "is unlikely to succeed" (Christiano et al., 2024).

47.5.2 Surprise accounting #

Jacob Hilton of ARC proposed a way to score an explanation, which ARC calls surprise accounting (Hilton, 2024b). The total surprise of a phenomenon has two parts. One is the surprise of the explanation itself, roughly how many free choices it makes; the other is the surprise of the phenomenon given the explanation, how unlikely it was to happen by chance if the explanation is right (Hilton, 2024b). Both are counted in bits (Hilton, 2024b). The two parts play the roles of a proof's length and its bound's tightness, and Hilton notes that their sum resembles the Bayesian information criterion, a standard score that trades a model's fit against its number of parameters (Hilton, 2024b). The derivation applies the score to the vault.

47.5.3 Anomalies of mechanism #

Step 5 is the idea of mechanistic anomaly detection. In a 2022 post, Christiano proposed finding the "normal reason" for a regularity, such as the camera showing the diamond across a dataset without tampering, and then asking of a new input whether the same explanation still applies (Christiano, 2022a). A robber who swaps the diamond for a fake between two frames keeps the image unchanged for a different reason, and the explanation of the trusted cases no longer covers it (Christiano, 2022a). The method needs no human label for the tampered case and need not identify the direct translator. It needs an explanation of the trusted behavior and a test of whether that explanation applies (inference). ARC's overview names it the most promising of its approaches to telling mechanisms apart, beside low-probability estimation, estimating how likely a model is to produce a rare output without sampling it (Hilton, 2024a).

The empirical tests so far are small and partial. A benchmark for detecting measurement tampering, which Section 30.5.2 reports, found that on its dataset of program correctness judged from test results no technique beat the simplest baselines (Roger et al., 2023). An anomaly detector separated the untruthful contexts of models fine-tuned to lie under a cue, as Section 39.7.2 reports (Mallen et al., 2023). In the study of rare outputs that Section 6.6.3 reports, ARC's overview reads its method inspired by heuristic explanations as beating naive sampling but not methods that search for inputs producing the rare output (Wu and Hilton, 2025; Hilton, 2024a).

The program's own reports describe its central problem as open. No estimator yet has the coherence properties (Christiano et al., 2022), an accuracy-based formalization met a hardness barrier (Christiano et al., 2024), and surprise accounting "is currently an informal framework with many missing details" (Hilton, 2024a). The result that would vindicate the program is a heuristic estimator that beats both sampling and search on a nontrivial trained model, or an anomaly detector that catches a planted behavior that no behavioral test catches (inference).

47.6 What logic and cryptography rule out #

The previous sections asked what inspection can achieve. Two fields state what it cannot achieve in the worst case, and the question for each is how far the worst case reaches the networks people train.

From logic. Logic, in the part of it that studies which questions an algorithm can decide, supplies the oldest limits. Section 28.6.2 states Rice's theorem, from 1953 (Rice, 1953), in the form Manuel Alfonseca and colleagues give it. Any non-trivial property of what a program does, one that some programs have and others lack, is undecidable: no algorithm decides it for every program (Alfonseca et al., 2021). Their own argument applies the halting problem, which they recount Turing and Church showing undecidable, to harm. A program that runs a given computation and then does harm does harm exactly when the computation halts, so a general harm detector would decide halting (Alfonseca et al., 2021). They conclude that containing a superintelligent program is incomputable in general, under an assumption they state: the program can simulate a universal Turing machine on arbitrary input (Alfonseca et al., 2021). Section 7.6 states a third limit, Löb's theorem, under which an agent that approves a successor only given a proof of its safety seems able to trust only successors that reason in weaker systems than its own. An inspector meant to certify a model as capable as itself meets the same obstacle whenever its certificate is to be a proof (inference).

What the results imply for inspection is that no inspector that always halts is right about every program: whatever the inspector, some program with the property escapes it or some program without it is flagged. Where they stop is the network (inference). A trained network of fixed size, run on inputs of bounded length, is a finite circuit, and any property of its behavior on those inputs is decidable in principle by checking every input. Alfonseca and colleagues make the related point that undecidability does not mean a question "cannot be decided for a given n", only that the complexity of a deciding algorithm "has to increase unboundedly with n" (Alfonseca et al., 2021). For a fixed network the binding limits are therefore cost and the adversarial constructions below, and undecidability returns when the network is part of an agent with unbounded memory, tools, or time (inference). Alfonseca and colleagues reject the restriction for the case they study: restricted classes of programs and inputs "cannot be useful to the containment of superintelligence", whose input is the state of the world (Alfonseca et al., 2021).

Inspection also relies on tools that must be trusted. Ken Thompson's compiler, which inserts a backdoor that leaves no trace in any source code, is the classic case, recalled from Section 38.5.3 (Thompson, 1984). Interpretability now uses models to explain models, in automated feature labels, readouts of activations, and auditing agents. Olah's note raised the concern itself: relying on AI to audit AI raises the question of "whether we should trust the model we are using to help us" (Anthropic, 2023e). The work-arounds, it adds, might make the failure modes of interpretability "more correlated with the failure modes of other approaches to safety" (Anthropic, 2023e). An inspection is as trustworthy as the models it uses, and failures that move together across layers of defense are what Section 6.7.4 quantifies (inference).

From cryptography. Cryptography states security against every adversary within stated computational limits, and it supplies the sharpest limits on inspecting a model. Shafi Goldwasser, Michael Kim, Vinod Vaikuntanathan, and Or Zamir, in a FOCS 2022 paper, asked what a party that outsources the training of a classifier can verify about the result (Goldwasser et al., 2022). They showed that a malicious trainer can plant a backdoor, a hidden mechanism that lets the holder of a key change the output on any input by perturbing it slightly (Goldwasser et al., 2022). Without the key, the mechanism "cannot be detected by any computationally-bounded observer", one limited to polynomial time (Goldwasser et al., 2022).

Two constructions give guarantees of different strength. The first works on any model. It adds to the classifier a check of a digital signature, a string that only the holder of a secret key can produce and anyone can verify, so that an input carrying a valid signature switches the output (Goldwasser et al., 2022). With query access alone it is "computationally infeasible to find even a single input" on which the backdoored and original models differ, and seeing backdoored inputs does not help an observer make new ones, which the authors call non-replicability (Goldwasser et al., 2022). The second works for networks trained on random features, single-hidden-layer networks whose first-layer weights are drawn at random and whose output layer alone is trained. The trainer runs the honest algorithm and tampers only with its random numbers, and even "given a complete description of the network and the training data", no efficient algorithm can tell the backdoored network from a clean one (Goldwasser et al., 2022). For cosine activations this rests on the hardness of a lattice problem, the kind on which cryptography designed to resist quantum computers is built; for ReLU networks, it rests on a weaker assumption that gives a weaker guarantee (Goldwasser et al., 2022).

The authors call the random-feature construction a proof of concept, since practical networks are trained further by gradient descent, which the construction does not cover (Goldwasser et al., 2022). They also show that any network can be rewritten, computing the same function at similar size, so that gradient descent on any loss leaves it unchanged (Goldwasser et al., 2022). And they draw a consequence for certification: a complete and sound procedure that certified a classifier as robust to small input perturbations would tell a backdoored model from a robust one, so the constructions rule out any efficient such procedure (Goldwasser et al., 2022). Their summary is that "under standard cryptographic assumptions, detecting backdoors in classifiers is impossible" (Goldwasser et al., 2022).

Paul Christiano, Jacob Hilton, Victor Lecomte, and Mark Xu, in an ITCS 2025 paper, asked what a defender can do at run time, given the model and a particular input, rather than by inspecting the model in general (Christiano et al., 2025). In their game an attacker picks a function from a class and a distribution of inputs, and a trigger is drawn at random. The attacker then picks a backdoored function that agrees with the original on all but a fraction $\varepsilon$ of inputs yet differs on the trigger. The defender, handed either the original function with a random input or the backdoored one with the trigger, must say which (Christiano et al., 2025). The random trigger is "the key symmetry-breaking assumption": an attacker who saw the trigger before choosing the original function could make the two cases look the same (Christiano et al., 2025).

Within that game, defense tracks learning, in the sense of PAC learning (probably approximately correct), the standard model in which a learner must reach small error, with high probability, from a limited number of examples (Christiano et al., 2025). With no limit on computation, a class is defendable with confidence tending to 1 exactly when $\varepsilon$ is small compared with one over its Vapnik-Chervonenkis (VC) dimension, a standard measure of how many patterns a class of functions can fit (Christiano et al., 2025). The dependence holds, the authors write, "in much the same way as PAC learnability" (Christiano et al., 2025). For a defender limited to polynomial time, every class that can be PAC learned in polynomial time is efficiently defendable, but not conversely (Christiano et al., 2025). At the other end, indistinguishability obfuscation is a way of rewriting programs so that rewrites of any two programs computing the same function cannot be told apart efficiently. If it exists, and so do one-way functions, functions easy to compute and hard to invert, then polynomial-size Boolean circuits are not efficiently defendable (Christiano et al., 2025). Indistinguishability obfuscation dates from 2001 (Barak et al., 2001), and by the authors' account it was constructed in 2021 from well-studied hardness assumptions (Christiano et al., 2025). Between the extremes, for decision trees on uniformly random inputs, a defense that only checks how deep in the tree the input lands draws no examples and runs in about the time of evaluating one tree, faster than any learning algorithm if drawing an example costs about as much as evaluating a tree (Christiano et al., 2025). The authors call defenses that exploit how a function computes its output mechanistic (Christiano et al., 2025). A third paper, by Goldwasser, Jonathan Shafer, Neekon Vafa, and Vaikuntanathan at STOC 2025, shows that some undetectable backdoors can still be removed, with black-box access only, querying the model without seeing its weights, when the true labels are close to functions of particular simple kinds (Goldwasser et al., 2025).

Christiano and colleagues read their results as an analogy for deceptive alignment, a model that behaves cooperatively when it detects cues of training and otherwise not (Christiano et al., 2025). In the analogy the cooperative model is the original function, the deceptive one the backdoored function, and the trigger an input rare in training (Christiano et al., 2025). They name the analogy's flaw themselves. Their attacker must succeed for a random trigger, while a deceptive model's trigger "must arise from properties of the model architecture and the training distribution" (Christiano et al., 2025). They doubt that defenses built on learning would transfer, since learning produced the problem, and place their hope in mechanistic defenses; they also note that a defender who saw the whole training process might find a trigger despite obfuscation (Christiano et al., 2025).

What these results imply bears most on the relationship between a developer and an outside auditor (inference). A party that controls training can hide a behavior from any efficient inspection of the finished model, so an audit of a model trained by a party under suspicion (Section 38.5) cannot rest on the weights alone. The defenses the theory leaves open all use something more: the training process, the true labels, or the input at run time. For a developer inspecting its own model the question is different: whether ordinary training produces structure as hard to inspect as a deliberate construction. As of October 2026 we found no result that places trained networks between the efficiently learnable and the obfuscated. Christiano and colleagues think an obfuscation-based construction unlikely to arise from ordinary training, but an informal obfuscation "much more plausible" (Christiano et al., 2025). The empirical record of planted backdoors is Section 48.6's, and backdoored models whose behavior survived safety training and was then found by probes are Section 34.3.4's.

47.7 Can interpretability carry safety claims? #

The results so far bound what inspection can claim. The debate this part of the book feeds is whether, within those bounds, interpretability can carry the claims a safety argument needs, and it helps to separate two kinds. A detection claim says that inspection found a problem, or added evidence that behavioral tests missed. An absence claim says that inspection found no problem because there was none. Three positions answer differently. This section gives each its evidence, its critics, and its own authors' concessions, reads the results against all three, and then asks two older fields what standard would decide.

47.7.1 Three positions #

The ambitious position holds that inside access will scale to auditing capable models, so that interpretability can eventually certify what behavior cannot, absence included. Its statements are the developer essays of Section 47.1 (Anthropic, 2023e; Amodei, 2025). Amodei's case rests partly on the claim that behavior cannot settle the question: "because deceit is precisely the behavior we're trying to find, external behavior is not reliable" (Amodei, 2025). Its evidence is a demonstration and a game, both by the developer on its own model. An attribution graph of a model trained with a hidden objective showed a feature for the objective active on every assistant-formatted prompt tested, though the analysts knew the objective in advance (Section 45.5.2) (Lindsey et al., 2025). The model was a model organism, trained to have a failure so that methods can be tested on it, and an attribution graph is a map of which internal features drove one output (Section 45.4). In a separate blind game, which did not use attribution graphs, three of four teams found the same model organism's objective; two of the three had begun as one team (Section 38.2.1) (Marks et al., 2025). Their techniques included sparse autoencoders, behavioral attacks, and searches of the training data (Marks et al., 2025).

The position's own authors supply its strongest concessions. Olah calls the enumerative route speculative (Anthropic, 2023e). The attribution-graph authors judge that a failure of their present methods to find a problem would be a very modest update toward safety (Section 45.7) (Lindsey et al., 2025). The game's authors found that a semantic search of the training data reproduced the dictionary's key finding, which weakened their own first reading (Section 38.2.1) (Marks et al., 2025). And Amodei attaches a condition: interpretability should "function like the test set for model alignment", and training on its outputs "destroys the independence of their signal" (Amodei, 2025).

The skeptical position holds that the ambitious program has stalled or rests on a mistake. Dan Hendrycks, director of the Center for AI Safety, and Laura Hiscott argued in 2025 that "the returns from interpretability have been roughly nonexistent" and that the program assumes that complex systems decompose into parts people can understand (Hendrycks and Hiscott, 2025). Their sharpest argument concerns length. A model's capabilities sit in terabytes of weights, and "it seems unlikely that we could distill model behavior into an explanation that takes up less than a kilobyte" (Hendrycks and Hiscott, 2025). They favor top-down representation engineering, which reads and steers concepts through directions in activation space (Chapter 46) and which Hendrycks's center advocates, so the critique comes from a party with a rival method. Its evidence is the non-uniqueness of Section 47.2 and the measurement studies of Section 44.7, in which dictionaries did not beat simple supervised directions on tasks with named concepts, after which Google DeepMind's team deprioritized fundamental research on them (Kantamneni et al., 2025; Smith et al., 2025b). Its authors concede that researchers "do not have to cease all exploration" and that an MRI for AI "would surely be useful" (Hendrycks and Hiscott, 2025). Its strongest critics are the detection results above, and the middle position's own report of "genuine partial progress" (Nanda et al., 2025).

The middle position holds that interpretability is one instrument in an audit, judged by whether it finds what behavioral methods miss, and it gives up complete reverse engineering as a goal. Google DeepMind's interpretability team stated it in December 2025. Having been "disappointed by the amount of progress made by ambitious mech interp work, from both us and others", the team turned to "pragmatic interpretability" judged on proxy tasks, tasks with a checkable answer that stand in for a safety goal, such as interpreting the hidden goal of a model organism (Nanda et al., 2025). Asked whether this is giving up, the authors answer that "in a strictly technical sense yes", and grant that they "can't rule out that more ambitious goals would work with time and investment" (Nanda et al., 2025). The field's review of open problems likewise grants that interpretability methods do not yet outperform black-box baselines (Section 43.7.2) (Sharkey et al., 2025). The position's evidence is the audit record that Section 38.2 and Section 38.3 report. AuditBench, a 2026 preprint by Abhay Sheshadri and colleagues at Anthropic's fellows program, MATS, the University of Freiburg, and Anthropic, is a benchmark of 56 fine-tunes of Llama 3.3 70B Instruct with hidden behaviors (Sheshadri et al., 2026). An investigator agent built on Claude Haiku 4.5, a language model that audits another, did best with black-box tools, and white-box tools, which read or alter the model's internals, helped on the easiest targets (Sheshadri et al., 2026). Bartosz Cywiński and colleagues, in a 2025 preprint from the MATS program with Anthropic researchers among the authors, trained Gemma 2 9B and Llama 3.3 70B to keep a secret: a word, an encoded instruction in the prompt, or the user's gender (Cywiński et al., 2025). Prefill attacks, which make a model continue an opening of its reply written by the auditor, were the most effective technique in all three settings, while white-box methods, though less effective, consistently improved on simple baselines, except when the secret word was asked for directly, where every method largely failed (Cywiński et al., 2025). Read from the ambitious side, the pivot abandons the claim a safety argument most needs, since success on proxy tasks licenses detection, not absence (inference). Read from the skeptical side, the supporting audits were run by the developer or with its researchers, on one or two base models each (Section 38.2.2).

Table 47.5 reads each result against all three positions.

No result so far is against the skeptical or the middle position, and each result against the ambitious one is against its absence claims or its primacy over black-box methods, not against detection. The middle position is hard to put against any result, because an instrument judged by what it adds is vindicated by any addition. What would count against it is a well-designed game in which white-box tools add nothing on any target, which AuditBench's hardest configuration approaches on one base model (inference). The skeptical position predicts every negative result and must explain each positive one as access to data or prior knowledge of the target. The evidence does not decide among the three.

What inspection can see also depends on how the model was trained. Activation monitors can be evaded by optimized inputs (Bailey et al., 2024) and by models fine-tuned to evade probes they never saw (McGuinness et al., 2025). A lie detector used as a training signal produced honesty or evasion depending on its accuracy and the strength of regularization (Cundy and Gleave, 2025). Section 39.8 reports all three, Section 46.4 takes them up as countermeasures to a detector, and Section 40.5 gives the counterpart for chains of thought. Amodei's test-set condition is the ambitious position's answer to this loop, which is the coupling between instillation and verification that the book traces (Section 1.7).

47.7.2 The standard that would decide #

From forensic science. Section 38.7 reports the forensic standard. The 2016 report of the President's Council of Advisors on Science and Technology (PCAST) asked that detection methods used in court show measured error rates in black-box studies with known answers (President's Council of Advisors on Science and Technology, 2016). It took a large study of fingerprint examiners as its model (Ulery et al., 2011). That section applies PCAST's criteria for foundational validity to auditing games. The same report names a second requirement, validity as applied: that the method "has been reliably applied in practice" in the case at hand (President's Council of Advisors on Science and Technology, 2016). It has two tests. First, the examiner must be shown capable and must have applied the method, which requires that "the procedures actually used in the case, the results obtained, and the laboratory notes be made available for scientific review by others" (President's Council of Advisors on Science and Technology, 2016). Second, the expert must "demonstrate that the samples used in the foundational studies are relevant to the facts of the case" (President's Council of Advisors on Science and Technology, 2016).

This is what an absence claim adds to a miss rate (inference). A miss rate measured in auditing games is foundational validity. A clean audit of a particular model is valid as applied only under two further conditions. The planted objectives behind the miss rate must be shown relevant to the model, which is the validity question of model organisms (Section 5.3). And the audit's procedures and results must be open to review by others, which for a closed model only its developer can allow. The first test concerns the deployed system; the second concerns the developer and the public. Where the analogy stops is the examinee. A fingerprint does not change because an examiner looks for it, while a model can be trained against the audit (Section 38.7.3). And forensic casework has a population of cases to sample, which hidden objectives in frontier models do not yet have.

From neuroscience. Section 43.6 reports the microprocessor test, in which the standard methods of neuroscience failed to recover how a fully known chip computes (Jonas and Kording, 2017). It also reports the field's unsettled answer to what understanding a network means. One answer is an algorithm, the middle of Marr's levels of analysis (Section 43.6.2). Another, Timothy Lillicrap and Konrad Kording's, is only the architecture, objective, and learning rule, because a broadly capable network may be "hard (or impossible) to compress" (Lillicrap and Kording, 2019). What the second answer implies here is a limit on the completeness route of Table 47.1 (inference). Three arguments point at one obstacle from different directions: Lillicrap and Kording's compressibility, Hendrycks and Hiscott's terabytes, and Gross and colleagues' structureless errors, which made proofs grow with the part of a network that has no structure to exploit (Hendrycks and Hiscott, 2025; Gross et al., 2024). If a network's competence does not compress, a complete explanation is about as long as the network, and an absence claim by explanation becomes an exhaustive analysis (inference). Where the analogy stops is the target. Neuroscience asks for an account of the whole system, while an audit needs one only of the computations that bear on catastrophic behavior, and ARC proposes to "tailor explanations towards behaviors with potentially catastrophic consequences" (Hilton, 2024a). Whether that part compresses when the whole does not is open.

Access decides who can apply either standard. Stephen Casper and colleagues argue that white-box and outside-the-box access allow substantially more scrutiny than black-box access alone (Section 38.1) (Casper et al., 2024a), and research outside the laboratories runs on open models and shared tools (Section 43.7.3) (Lieberum et al., 2024; Fiotto-Kaufman et al., 2024). Who may inspect a model is therefore a verification question as well as a release question (Section 38.8; Chapter 63).

The program's metaphors were coined by developers describing their own methods: a "microscopic" theory (Anthropic, 2023e), the biology of a model in the title of an attribution-graph report (Lindsey et al., 2025), and the MRI or "brain scan" (Amodei, 2025). The book uses neutral descriptions and records the coinages in Appendix A.

47.7.3 What inspection has delivered to the principal #

The book's thesis is that alignment is a delegation problem under an asymmetry of competence and visibility, and it fixed the power to read an agent's internals in advance as a novelty with no counterpart in the older theories (Section 1.7). This chapter's results say what the power has delivered (inference). For detection it is real: a principal can find an objective planted in its agent, in the developer's demonstrations and games, which no employer can do with an employee. For absence it has delivered little. Explanations are not unique, proofs reach small networks and single layers, and in the worst case a party that controls training can hide a behavior from any efficient inspection. Where the agent cannot be seen into, the older theories rely on monitoring with measured error and on institutions that check the monitor (Section 38.6), and the middle position and the forensic standard both lead back there. For the developer and the public the asymmetry runs the other way: every positive result in this chapter about a closed model is a developer's report on the developer's own tools, which the public cannot yet check.

47.8 Settled, contested, missing #

Four chapters' cruxes meet here, and they nest (inference). The audit-level result, which Section 38.10 names and D14 of Chapter 70 collects, decides between the skeptical and the middle positions: whether white-box tools add to an audit what black-box tools cannot. An absence claim, which only the ambitious position promises, needs three results more. They are each instrument's error rates on planted mechanisms at realistic scale (Section 43.8), a canonical set of features to enumerate (Section 44.8), and circuits that hold over a whole input distribution (Section 45.8). The crux below is the audit-level one, so that the debate has one deciding result; the other three are the conditions for reading a clean audit as evidence of absence.

47.9 Exercises #

Further reading #

• Christiano et al. (2021): the statement of the verification problem and the worst-case method built on it.
• Méloux et al. (2025): why a correct explanation need not be the explanation.
• Goldwasser et al. (2022): what a party that controls training can hide from inspection.
• Christiano et al. (2025): where defense at run time sits between learning and obfuscation.
• Christiano et al. (2022): heuristic arguments as a formal object between proof and interpretation.
• Sharkey et al. (2025): what mechanistic interpretability cannot yet do, in the field's own words.