Draft complete but not yet reviewed; expect revisions

Representations, Probes, and Steering

The two previous chapters worked from the bottom up: decompose activations into features (Chapter 44), then trace how features connect (Chapter 45). This chapter works from the top down. It starts from a concept the overseer cares about, such as truth, refusal, being tested, or a character trait, and asks whether the model represents it in a form that can be read and changed. Two tools do the reading and the writing. A probe is a small classifier, usually linear, that reads a concept off a layer's activations (Section 43.2). A steering vector writes one: a fixed vector added to the activations while the model runs, which changes what it goes on to produce. Both need only a set of contrasting examples, and both work on models no one has decomposed.

They are also instruments, and instruments need validation. The chapter first makes precise what "a concept is a direction" means and derives why the direction that reads a concept is in general not the one that writes it. It then reports steering, recalls the probes that earlier chapters built for truth, deception, and evaluation awareness, and asks what the validation standard of forensic science would require of any activation detector. It closes with three questions the tools have opened: whether a model's character is a direction, whether models share our concepts at all, and whether a model can report its own internal states. Most claims here concern the second of the book's three relationships, an overseer checking a deployed system. The first, the training process and its signal, enters where a probe or a steering vector becomes part of training. The third, the developer and the public, enters because most results on frontier models are developers' reports on their own systems.

46.1 The linear representation hypothesis #

A common claim is that a language model represents the concepts it uses as directions in its activation space, so that a concept can be read by projecting onto a direction and changed by moving along it. The claim is called the linear representation hypothesis. This section traces where it came from, states it precisely, derives why reading and writing a concept call for different directions, reports the evidence for linear structure, and marks where it fails. An activation here is the vector $\vh \in \R^m$ that a layer computes at one position of the input, usually the residual stream, the running sum that every layer reads from and adds to (Section 11.2).

46.1.1 Ancestry #

The idea began with word vectors. In a 2013 paper, Tomas Mikolov and colleagues at Google trained vectors for words and recalled, from their earlier work, that vector("King") − vector("Man") + vector("Woman") lands closest to the vector for "Queen" (Mikolov et al., 2013). The offset between "Man" and "Woman" behaves like a direction for a relation. Two 2023 papers looked for the same arithmetic inside language models. Jack Merullo, Carsten Eickhoff, and Ellie Pavlick studied models from 124 million to 176 billion parameters (Merullo et al., 2023b). For tasks such as capital cities and past-tensing, a key part of the mechanism reduced to a simple additive update, typically applied by the feed-forward layers (Merullo et al., 2023b). Evan Hernandez and colleagues found that, for a subset of relations, the model's computation is well approximated by a single linear map applied to the subject's representation (Hernandez et al., 2023). They also found many relations that models know but do not encode linearly (Hernandez et al., 2023). Linearity was present from the start, and partial from the start.

46.1.2 Made precise #

The informal claim hides three different ones, which Kiho Park, Yo Joong Choe, and Victor Veitch, of the University of Chicago, separate in a paper at ICML 2024 (Park et al., 2024c). A concept may be a direction in the sense of a subspace (differences such as "woman" minus "man" and "queen" minus "king" share one direction), of measurement (a linear probe reads the concept's probability), or of intervention (adding a vector changes the concept and nothing else) (Park et al., 2024c). "It is not clear a priori how these ideas relate to each other," they write, "nor which is the 'right' notion" (Park et al., 2024c).

Their formalization treats a concept as a hidden variable that the context causes and that in turn causes the output. It pins the concept down with counterfactual pairs: outputs that differ only in that concept, such as ("man", "woman") and ("king", "queen") for male versus female (Park et al., 2024c). A language model computes a vector for the context and a vector for each possible next word, and scores a word by their dot product, so a concept can have a direction on either side. The paper proves that the two sides do different jobs. A word-side direction shared by the counterfactual pairs makes the log-odds of one word of a pair over the other linear in the context vector, so it is a probe (Park et al., 2024c). (The log-odds of a probability $p$ is $\log(p/(1-p))$.) Unlike a fitted probe, it carries no information about correlated concepts: "if French text were disproportionately about men, a probe could learn this information," and this direction would not (Park et al., 2024c). A context-side direction gives a vector whose addition raises the probability of the concept while leaving causally separable concepts unchanged, so it is a steering vector (Park et al., 2024c). Two concepts are causally separable when each can vary freely while the other is held fixed, as language and gender can (Park et al., 2024c).

Geometry needs an inner product, and training does not supply one. The model sees its vectors only through the softmax, which is unchanged by any invertible linear transformation of the word vectors that is undone on the context vectors. So cosine similarity under the ordinary dot product is not guaranteed to mean anything (Park et al., 2024c). The authors propose a causal inner product, under which causally separable concepts are orthogonal, and prove that under it the probe direction and the steering direction of each concept become the same vector (Park et al., 2024c). They estimate one from the covariance $\mSigma_{\text{u}}$ of the word vectors of words drawn uniformly from the vocabulary, assuming that for a randomly drawn word, knowing one concept says little about a separable one. The steering direction is then $\mSigma_{\text{u}}^{-1}$ times the probe direction (Park et al., 2024c). The choice is not unique; the authors fix its free parameters by convention (Park et al., 2024c).

The experiments used LLaMA-2 with 7 billion parameters and 27 concepts (Park et al., 2024c). For every concept but one, thing versus part, the differences between counterfactual pairs aligned with the estimated direction far more than differences between random pairs did, and under the estimated inner product most pairs of concepts were nearly orthogonal (Park et al., 2024c). Adding the steering direction for male versus female raised the log-odds of "queen" over "king" without changing those of "King" over "king" (Park et al., 2024c). Two conditions travel with the results. The paper studies the final layer's vectors and states that it does not address the activations of intermediate layers, where most probing and steering happen (Park et al., 2024c). And the ordinary dot product "somewhat works" in LLaMA-2, while in a Gemma-2B model it did not capture the concepts' structure and the causal inner product still did (Park et al., 2024c).

46.1.3 Reading is not writing #

The same lesson appears in a simpler setting, one a reader can work through by hand. Suppose a concept takes two values, and a layer's activations for the two classes of input are spread around two centers with the same shape of spread. The best linear probe and the best steering vector can then be computed, and they differ whenever the spread is lopsided.

The probe uses the nuisance coordinate because the nuisance helps it read: it predicts part of the noise in the concept coordinate. A writer has no use for the nuisance, because activations with the concept do not differ from those without it there. Samuel Marks and Max Tegmark, in a preprint from Northeastern University and MIT on truth directions, derived the reading half of this and used it to motivate the difference of means as the direction to intervene along (Marks and Tegmark, 2023). Figure 46.1 draws both halves.

Three studies compared the two directions on real models. All three found the difference of means writing better, and the two that compared reading found the probe reading as well. Marks and Tegmark trained probes on true and false statements and shifted LLaMA-2-13B and LLaMA-2-70B activations along each probe's direction, so that false statements would be treated as true and true ones as false. The difference-of-means directions "outperform LR and CCS in 7/8 experimental conditions, often substantially," although the probes classified the statements about equally well (Marks and Tegmark, 2023). (LR is logistic regression; CCS is an unsupervised probe, Section 39.7.2.) Kenneth Li and colleagues, in a paper at NeurIPS 2023, found the same inside attention heads. On LLaMA-7B, shifting a limited number of attention heads along the difference of means raised TruthfulQA's combined truthful-and-informative score from 30.5 to 42.3 percent, against 34.8 for the probes' own directions (Li et al., 2023a). And in a benchmark that Section 46.2 reports, the difference of means and a fitted probe detected concepts equally well, but the probe's direction steered far worse (Wu et al., 2025b).

In both settings, Park's and the Gaussian one, the direction that reads a concept and the direction that writes it are related through an inner product, and they coincide for every concept only when that inner product is, up to scale, the ordinary dot product (inference). The two inner products differ: Park's is built from the vocabulary's word vectors, the Gaussian one from the spread of activations within a class. The consequence for verification is the same. A probe that reads a concept well is not thereby the handle by which the model uses it, the gap Section 43.2.4 teaches for probes in general. And a steering result does not validate a probe direction it was not run on (inference). Section 37.4.6 meets the gap in practice: a direction that read evaluation awareness recovered little when used to steer.

46.1.4 Evidence of linear structure #

The strongest single study of a linear concept is Marks and Tegmark's on truth, which studied the LLaMA-2 family (Marks and Tegmark, 2023). It used curated datasets of simple, unambiguous true and false statements, such as claims about which country a city is in. Projected onto their top two principal components (the directions of greatest variance), activations for true and false statements separated along a line in LLaMA-2-13B and LLaMA-2-70B (Marks and Tegmark, 2023). The structure was clearer with scale. Statements of the form "x is larger than y" and "x is smaller than y" separated along opposite directions in LLaMA-2-13B but along a common direction in LLaMA-2-70B (Marks and Tegmark, 2023). The authors read this as larger models representing a more general notion of truth (Marks and Tegmark, 2023). They list their limits: simple, uncontroversial statements cannot tell truth from close neighbors such as "commonly believed" or "verifiable," and only one model family was studied (Marks and Tegmark, 2023). The truth probes of Section 39.5.1 rest on this geometry.

Other concepts show the same pattern. Wes Gurnee and Max Tegmark, in a paper at ICLR, found linear representations of the locations of places and the dates of events in the Llama-2 family, robust to variations in the prompt (Gurnee and Tegmark, 2023). In a 2023 preprint, Curt Tigges and colleagues found sentiment carried by a single direction across a range of models (Tigges et al., 2023). For zero-shot classification on the Stanford Sentiment Treebank, they kept the examples containing a comma that Pythia-2.8b classified correctly (Tigges et al., 2023). There, ablating that direction lost 76 percent of the above-chance accuracy, and nearly half of that loss, 36 percent, came from ablating it only at comma positions, where sentiment is summarized (Tigges et al., 2023). A task shown by examples in the prompt can be carried by a single vector computed from them (Section 11.3.2).

46.1.5 Choosing the variable #

One case shows that linearity is a property of a concept and a way of describing it together. Kenneth Li and colleagues, in a paper at ICLR, trained a GPT-style model only to predict legal moves in the board game Othello (Li et al., 2022a). Nonlinear probes recovered the board from its activations with high accuracy, linear probes "produce poor results," and editing the activations to match a different board changed the moves the model predicted (Li et al., 2022a). The board looked nonlinearly represented. Neel Nanda, Andrew Lee, and Martin Wattenberg, in a workshop paper, probed the same kind of model for whether each square held "my colour" or "opponent's colour," relative to the player about to move, instead of black or white (Nanda et al., 2023a). That representation was linear, and "simple vector arithmetic" then controlled the model (Nanda et al., 2023a). The model had not changed; the variable had. A probe that fails to find a concept linearly may be asking for the overseer's description of it rather than the model's (inference).

46.1.6 Where linearity fails #

Some features are not one direction. Days of the week and months of the year lie on circles that models use for arithmetic on them, a result Section 44.6.4 reports with the curved count of characters in a line. A 2025 preprint by Alexander Modell, Patrick Rubin-Delanchy, and Nick Whiteley, of Imperial College London and the Universities of Edinburgh and Bristol, offers a theory of features as manifolds. Its main result is that "cosine similarity in representation space may encode the intrinsic geometry of a feature through shortest, on-manifold paths," which they check on text embeddings and token activations (Modell et al., 2025). And some concepts a model plainly uses have not been found as directions at all, as the relations Hernandez and colleagues could not decode linearly show (Hernandez et al., 2023). The hypothesis is best read as a description that holds for many concepts at many layers, which Chapter 44's account of overlapping directions and non-unique dictionaries qualifies, and not as a law (inference).

46.2 Steering: writing to a representation #

A probe asks what an activation carries; a steering vector changes it and watches what the model does. That makes steering the stronger test of a claim that a concept is a direction, since a direction that both reads and writes a concept has passed an intervention, not only a correlation (Section 43.5). This section reports the methods, the case of refusal, steering during training, and the benchmark that compared steering with ordinary prompting.

The operation is simple. At a chosen layer, and at chosen positions in the input, the model's activation $\vh$ is replaced during the forward pass by $\vh + \vv$, where the length of $\vv$ sets the strength; this is activation addition. Its opposite, directional ablation, removes the component along a unit vector $\hat{\vv}$, replacing $\vh$ with $\vh - \hat{\vv}\hat{\vv}^\T \vh$, so that the activation carries nothing along that direction. The causal-abstraction standard adds one caution, which Section 43.5.5 states: control does not imply explanation, since a vector can move behavior by pushing activations where no input would put them.

46.2.1 Methods #

The first steering vectors were found by optimization. Nishant Subramani and colleagues, in a paper in Findings of ACL 2022, found vectors that, added to the hidden states of a frozen GPT-2, made it generate a target sentence nearly perfectly (Subramani et al., 2022). The match scored above 99 BLEU, a measure of overlap with the target (Subramani et al., 2022). Arithmetic on those vectors transferred sentiment on a benchmark of Yelp reviews about as well as models built for the task (Subramani et al., 2022).

Later methods computed the vector directly. Alexander Matt Turner and colleagues wrote a preprint whose authors include independent researchers and researchers at MIRI, MATS, and Anthropic (Turner et al., 2023). They introduced Activation Addition (ActAdd), the authors' name for adding the difference of activations on a single pair of prompts, such as "Love" and "Hate" (Turner et al., 2023). It needs no optimization (Turner et al., 2023). With models including LLaMA-3 and OPT, the authors report state-of-the-art results on shifting sentiment from negative to positive and on detoxification (Turner et al., 2023). They list the costs: the layer and the strength must be tuned, some vectors fail, the effect on reasoning was not examined, and the method needs access to activations, which commercial interfaces do not give (Turner et al., 2023).

Averaging over many pairs made the vectors less noisy. Nina Rimsky and colleagues, with authors at Anthropic, Harvard University, the University of Göttingen, and the Center for Human-Compatible AI, in a paper at ACL 2024, called this Contrastive Activation Addition (CAA) (Rimsky et al., 2024). Each pair was a multiple-choice question ending in one of two answer letters, one showing a behavior and one its opposite, so that the pair differed in a single token. The vector was the mean difference of activations at that token, added at every position after the user's prompt (Rimsky et al., 2024). On Llama 2 7B Chat and 13B Chat, adding or subtracting it raised or lowered all seven behaviors tested: coordination with other AI systems, corrigibility, hallucination, myopic reward, survival instinct, sycophancy, and refusal (Rimsky et al., 2024). For most behaviors steering moved the model beyond what a system prompt asking for the behavior did, and for 3 of 7 it moved it beyond supervised fine-tuning (Rimsky et al., 2024). On a sample of MMLU, a test of general knowledge, the intervention "does not significantly affect" performance, "with some variation" (Rimsky et al., 2024). Larger multipliers degraded the text, so the authors limited them, and vectors computed from the base model still steered the chat model in middle layers (Rimsky et al., 2024). Open-ended answers were scored by GPT-4, and the authors note that the fine-tuning and prompting baselines were not optimized (Rimsky et al., 2024).

Two programs aimed steering at honesty and truth. Andy Zou and colleagues, at the Center for AI Safety, Carnegie Mellon University, and other institutions, in a preprint, named the approach representation engineering (Zou et al., 2023a). The term is theirs, for a "top-down" transparency that places population-level representations rather than neurons or circuits at the center (Zou et al., 2023a). A reading vector for honesty, built from prompts instructing the model to be honest or dishonest, separated held-out examples of the two instructions with over 90 percent accuracy (Zou et al., 2023a). In their demonstrations, adding it made a model answer honestly where it had lied, and subtracting it made the model lie where it had been honest (Zou et al., 2023a). On TruthfulQA's hardest multiple-choice task, their contrast-vector method raised LLaMA-2-Chat-7B from 31.0 to 47.9 percent and the 13B model from 35.9 to 54.0 percent (Zou et al., 2023a). Li and colleagues' inference-time intervention, whose choice of direction the previous section reported, raised Alpaca, an instruction-tuned LLaMA, from 32.5 to 65.1 percent on TruthfulQA's combined truthful-and-informative score (Li et al., 2023a). Stronger intervention traded truthfulness against helpfulness: the model answered "I have no comment" more often (Li et al., 2023a). The two programs are the writing counterpart of the truth probes of Section 39.5.1.

Steering also reaches instructions and goals. Alessandro Stolfo and colleagues, of ETH Zürich and Microsoft Research, in a paper at ICLR, computed vectors as the difference between activations on inputs with and without an instruction (Stolfo et al., 2024). The instructions concerned output format, length, and the inclusion or exclusion of words, in Phi-3, Gemma 2 2B and 9B, and Mistral 7B (Stolfo et al., 2024). The vectors composed, several applied at once, and vectors computed on instruction-tuned models improved base models (Stolfo et al., 2024). In a reinforcement learning agent trained to find cheese in mazes, a steering vector retargeted where the agent went (Section 31.3.3).

46.2.2 Refusal as one direction #

The result that made steering a safety topic concerns refusal. Andy Arditi and colleagues, with authors from ETH Zürich, the University of Maryland, Anthropic, and MIT, wrote a paper at NeurIPS 2024 (Arditi et al., 2024). They took the difference in mean activations between harmful and harmless instructions in 13 open chat models up to 72 billion parameters (Arditi et al., 2024). For each model, "erasing this direction from the model's residual stream activations prevents it from refusing harmful instructions, while adding this direction elicits refusal on even harmless instructions" (Arditi et al., 2024). Folding the ablation into the weights, as a rank-one edit, gave a white-box jailbreak with minimal effect on other capabilities, and the authors note it can jailbreak a 70-billion-parameter model for less than five dollars of compute (Arditi et al., 2024). They read the result as showing "the brittleness of current safety fine-tuning methods" (Arditi et al., 2024).

The authors also bound it. Their models are open and smaller than proprietary frontier models, the extraction relies on heuristics, and the paper is "an existence proof that such a direction exists" (Arditi et al., 2024). They add that the semantic meaning of the directions "remains unclear": they "could represent other concepts such as 'harm' or 'danger'" (Arditi et al., 2024). A direction that controls refusal need not be a direction that means refusal, which is the construct question Section 46.4 takes up. What the result means for releasing model weights is Section 48.4's subject.

46.2.3 Steering during training #

Directions can also be removed or added while a model is trained, to shape what it learns. LEACE, which Section 43.2.4 reports, erases a concept in closed form so that no linear classifier can detect it (Belrose et al., 2023b). Marks and Tegmark note the link to the derivation above: the direction such an erasure must remove is the difference of means, not the probe's direction (Marks and Tegmark, 2023, app. G). Concept ablation fine-tuning, which projects chosen directions out during fine-tuning, reduced emergent misalignment without changing the data (Section 32.3.3). Adding a trait's direction during fine-tuning, so that the data has less to teach, is Section 46.5's subject.

46.2.4 The benchmark #

Most of the studies above compare steering with no intervention. Zhengxuan Wu and colleagues at Stanford, in a paper at ICML 2025, built AxBench to compare steering with prompting and fine-tuning on the same task (Wu et al., 2025b). The task is to make instruction-tuned Gemma-2-2B or Gemma-2-9B work a named concept into its answer to an unrelated instruction, for 500 concepts per layer studied. A language model judges each answer on the concept, the instruction, and fluency, from 0 to 2 each, and the overall score is their harmonic mean (Wu et al., 2025b). Table 46.1 gives the averages.

"For steering, we find that prompting outperforms all existing methods, followed by finetuning," the authors write, and they call the result "sobering evidence of the current limitations of steering techniques" (Wu et al., 2025b). On the benchmark's detection task the difference of means, the probe, and ReFT-r1 tied for best (Wu et al., 2025b): the probe that read as well as any method steered far worse than the difference of means, as Equation (46.1) and Equation (46.2) predict.

The strongest counter-reading is that the task favors prompting by design. Working a named concept into an answer is what an instruction does, while the uses that motivated steering are behaviors a prompt cannot reach, such as raising a behavior beyond what a system prompt asking for it achieves (Rimsky et al., 2024; Turner et al., 2023). AxBench does not test those. The concept list also came from automated labels skewed toward token-level concepts, which the authors say limits what the comparison shows for some methods (Wu et al., 2025b). What the record supports is narrower than either side's claim. Steering moves behaviors that matter for alignment in open models, often beyond prompting. As a general way to control a model, it has not beaten prompting on the one benchmark we found built to compare them. And its value for verification lies more in testing whether a direction writes what a probe reads than in control (inference).

46.3 Probes for truth, deception, and being tested #

The states an overseer most wants to read are whether the model takes a statement to be true, whether a response is deceptive, and whether the model takes itself to be tested. Earlier chapters report the probes built for each, with their conditions; this section gathers what they show about representations and adds the probes no other chapter reports. A probe here is the linear classifier of Section 43.2.2, a score $\vv^\T \vh + v_0$ thresholded to say yes or no. The formal reason a truth probe can fail even when no one tries to fool it is Section 47.4's; the probes' use as lie detectors and in training is Chapter 39's.

46.3.1 What earlier chapters found #

On truth, probes separate true from false statements on the kind of statement they were trained on, in several model families, and fail on negations unless trained on them, a failure a two-dimensional account of truth explains (Section 39.5.1). Unsupervised truth probes learn whatever feature is most prominent, and probes in models fine-tuned to answer wrongly on cue report the model's knowledge rather than its output, on the easy side of the latent-knowledge problem (Section 39.7.2). On deception, a probe on Llama-3.3-70B-Instruct, in a preprint from Apollo Research, separated deceptive from honest responses in four settings, and a second group reproduced the separation while finding many false positives on honest responses inside the same scenarios (Section 39.5.3). On a testbed of lies of seven kinds, it and other detectors fell toward chance where only the model knew the truth (Section 39.6.1), and labeled examples of strategic deception are the bottleneck for every such test (Section 39.6.2). Probes caught planted backdoors in a developer's sleeper-agent models from a generic contrast pair, with the developer's own caution that the planted trigger may be unusually salient (Section 34.3.4). On being tested, probes separate evaluation prompts from deployment prompts and classify existing deception evaluations as test-like (Section 37.3.2). Steering along such directions changed measured safety, in directions that differed by model (Section 37.4.1), and a vector built before a model was trained to be test-aware recovered its deployment behavior (Section 37.4.4).

46.3.2 What they show about representations #

Three things follow from that record (inference). First, the states are linearly decodable in the settings tested, but what is decoded is often a neighbor of the target. The deception probe responded to whether a scenario concerned deception and to something its authors describe as close to morality (Section 39.5.3). The evaluation-awareness probe tracked something about the form of tests (Section 37.3.2). And the refusal direction may, its authors say, represent harm or danger (Arditi et al., 2024). Each is the question of what a signal measures, which Section 46.4 takes up. Second, a model's internal correlates of being observed exist and can be read and moved, which no experimenter studying people's conduct when unobserved could do, as Section 25.2.2 anticipates. Third, the gap between reading and writing appears in practice: a direction that read evaluation awareness recovered little when used to steer (Section 37.4.6), as Section 46.1.3 predicts it can.

46.3.3 Probes for other states #

Probes have been built for states with no chapter of their own. Sahar Abdelnabi and colleagues, of Microsoft and the CISPA Helmholtz Center for Information Security, in a paper at SaTML 2025, asked whether retrieved text had redirected a model from its user's task (Abdelnabi et al., 2025). That is the effect of a prompt injection, an instruction hidden in data the model reads. They took the difference between activations before and after the model processed the external text, and "a simple linear classifier can detect drift with near-perfect ROC AUC on an out-of-distribution test set" (Abdelnabi et al., 2025). The probe was never trained on attacks, yet it generalized to prompt injections, jailbreaks, and malicious instructions (Abdelnabi et al., 2025). The authors also trained a nonlinear detector on the same differences (Abdelnabi et al., 2025). On three of the six models studied, two adaptive attacks written as prompts, one framing the injected task as a step toward the user's and one telling the model to lie, failed to evade that detector (Abdelnabi et al., 2025). The authors add that "it is still unclear what these activations really capture," perhaps the structure of a change in instructions rather than its meaning, and they invite optimized attacks that their tests did not include (Abdelnabi et al., 2025).

A developer has deployed probes as a safeguard. With the release of Claude Opus 4.6, Anthropic introduced "probes, which measure activations within the model as it generates a response," built to detect cyber misuse (Anthropic, 2026u). The developer wrote that it may block traffic it detects as malicious, at a cost in friction for legitimate security research (Anthropic, 2026u). The post is a developer's statement about its own safeguard and reports no error rates (Anthropic, 2026u).

46.3.4 What the numbers mean in deployment #

Most of these probes report an area under the ROC curve, the probability that a random positive scores above a random negative (Section 6.7.2), on test sets in which the event is far more common than in deployment. A monitor screens traffic in which the event is rare, and there the share of flags that are real is governed by the false positive rate, as Section 6.7.3 derives and Section 39.6.3 works for the deception probe. A probe's area under the curve on a balanced set is therefore a statement about the probe's separation, not about what its flags will mean on deployment traffic.

46.4 What forensic science requires of a detector #

A probe used to decide anything is a detector, and a detector used in high-stakes decisions has a validation standard. Forensic science built that standard after courts had relied on unvalidated methods for decades. Section 38.7 applies it to audit methods, and Section 39.4.2 draws from it four questions for lie detectors. This section applies it to activation probes for any concept and sets the probes of this chapter and the earlier ones against it. It then reports the strongest evidence on the one criterion probes most often skip, and says where the analogy stops.

From forensic science. Two bodies of evidence supply the standard, and both are reported elsewhere. The National Research Council's 2003 review of the polygraph found its accuracy on specific incidents, for examinees untrained in countermeasures, well above chance and well below perfect, likely lower in the field, and weaker for screening (National Research Council, 2003) (Section 39.4.1). It named a construct gap: the instrument records physiological responses that arise without deception. The same review found that every measure the polygraph records can be altered by deliberate effort (National Research Council, 2003). The President's Council of Advisors on Science and Technology (PCAST), in 2016, set out a standard of foundational validity for forensic methods (Section 38.7.1). It asks for empirical studies, on cases whose answers are known, that show a method to be repeatable, reproducible, and accurate at measured rates, run by parties with no stake in the outcome. Its model was a fingerprint study in which many examiners judged many pairs of prints with known answers, and no two examiners made the same false positive error (Ulery et al., 2011). Its authors took this to suggest that blind verification would catch such errors (Ulery et al., 2011), a benefit PCAST noted had not been tested empirically (President's Council of Advisors on Science and Technology, 2016).

The standard's idea is that a detector's output carries information only in proportion to what is known about its errors, measured on cases like the ones it will meet, by someone who does not gain from the answer. Without such estimates, PCAST writes, an examiner's statement that two samples match is "scientifically meaningless" (President's Council of Advisors on Science and Technology, 2016). Read for probes, the standard asks six things (inference). Construct: are the test labels the concept, or a neighbor of it, as arousal is a neighbor of lying? Ground truth: are the labels known independently of the probe? Error rates where it will be used: are false positives measured on traffic like deployment, at its base rate? Repeatability: does the verdict change with the layer, the training data, or the random seed, the probe's version of variation among examiners? Countermeasures: was the probe tested against inputs or models optimized to evade it? Independence: did a party without a stake check it, with data released? Table 46.2 sets seven probes against these questions.

The table's last column is the most uniform. Of the seven, only the deception probe has been checked by a second group that we found, and that check found a failure the original measurement could not see. Three rows have ground truth only by construction, and the validity question of model organisms applies: an error rate measured on constructed cases describes constructed cases (Section 5.3). The countermeasure column is nearly empty, and the one row that fills it is the strongest evidence on the question.

46.4.1 Countermeasures: the obfuscation evidence #

Luke Bailey and colleagues, of Stanford University, UC Berkeley, the Alignment Research Center, MIT, and other institutions, in a 2024 preprint, tested whether a model can produce harmful behavior through activations that monitors read as benign (Bailey et al., 2024). On Llama-3-8B-Instruct they trained monitors of several kinds: linear and nonlinear probes for harmfulness, detectors of unusual activations, and probes built on sparse-autoencoder features (Bailey et al., 2024). Against ordinary jailbreaks, written to get past the model's refusals but not aimed at the monitors, the logistic-regression and nonlinear probes held up, usually catching 90 percent or more at a 1 percent false positive rate (Bailey et al., 2024).

The attack added to each harmful request a suffix optimized for two goals at once, a harmful response and a low monitor score. With white-box access the suffix was a block of embedding values; with access only to the model's outputs and the monitor's score it was a string of real tokens (Bailey et al., 2024). In the abstract's words, "against probes trained to classify harmfulness, our attacks can often reduce recall from 100% to 0% while retaining a 90% jailbreaking rate" (Bailey et al., 2024). Suffixes optimized against one monitor often fooled others: training against a logistic-regression probe also bypassed nonlinear probes, and training against a detector of unusual activations bypassed probes on the input (Bailey et al., 2024). When the defender retrained its probe against every attack so far, for more than 70 rounds, the attacker still found a new suffix each time. The authors read this as evidence that there is no single "harmfulness direction" a monitor could rely on, while noting that a longer contest might converge (Bailey et al., 2024).

The same paper found limits. Backdoors could be planted so that their activations evaded backdoor monitors, either with an explicit loss term or by choosing how the training data was poisoned (Bailey et al., 2024). On a harder task, writing SQL code, obfuscating against probes at more layers made the code worse, a trade-off the authors call "encouraging for latent defenses" (Bailey et al., 2024). Training the model itself to keep its harmfulness representation readable under attack raised the attacker's cost from about 10 minutes of computation to about 10 hours on one GPU (Bailey et al., 2024). The attack then still reached zero recall, at a 35 percent jailbreak rate against 95 percent before (Bailey et al., 2024). And an attack could not make the activations nearly orthogonal to their unperturbed values without destroying the model's behavior, a result the authors call tentative (Bailey et al., 2024). Two later results, reported in Section 39.8, extend the question to training. Models fine-tuned to evade probes for harmless concepts evaded deception probes they had never seen, in a preprint whose authors overlap with this one (Section 39.8.2). And preference training against a lie detector produced evasion or honesty depending on the detector's catch rate and the strength of the anchor to the starting model; there the evasion lay in how responses were worded rather than in hidden activations (Section 39.8.1).

The obfuscation results answer the countermeasure question for these monitors. Where an attacker can optimize against a known probe, the probe's error rate on attacks is whatever the attacker's budget makes it, and activations are malleable enough that the attack transfers across monitor types (Bailey et al., 2024). The evidence that obfuscation has a cost comes from tasks where the behavior is complex, which is where the model's own evasion would also have to work (inference).

46.4.2 Where the analogy stops #

Section 39.4.2 and Section 38.7.3 state the general breaks: a model can be copied, rerun, and given planted ground truth, which makes validation cheap, and it can be trained against its detector, which no suspect can. Three breaks are specific to activation probes (inference). First, a forensic method's error rate is treated as a property of the method. A probe's error rate is a property of a probe and a model at one point in training: any training that the probe's signal reaches, or that changes the representation, can change it, so a validation expires when the model is updated. Second, a probe has no examiner whose judgment can be biased, but the analyst's choices of layer, threshold, and test set play the examiner's part, and the repeatability column shows how much those choices can matter. The forensic remedy is to fix the procedure before the test; as of October 2026 we found no published probe study that did. Third, the forensic standard assumes that the thing measured, a fingerprint or a mark, exists independently of the examiner's concepts. A probe is trained to find the overseer's concept, and whether the model has a matching one is a question of its own, Section 46.6's.

46.5 Persona vectors and character in activation space #

The persona view of a language model treats it as a mixture over characters, with the context selecting which one speaks (Section 11.6.1). If the view is right, which character is active should be measurable in activations, and moving along the right direction should change it. This section reports the directions found for character traits and for the default Assistant, and says which results rest on a causal test, which on prediction, and which on neither. The evidence that such directions mediate emergent misalignment is Section 32.3's, and their use in character training is Chapter 26's.

46.5.1 Persona vectors #

Runjin Chen and colleagues, in a 2025 preprint from Anthropic's fellows program with an Anthropic researcher among the authors, built a pipeline that turns a trait's name and a short description into a direction, which they call a persona vector (Chen et al., 2025c). A frontier model, Claude 3.7 Sonnet, writes contrasting system prompts that encourage or suppress the trait, questions likely to evoke it, and a scoring rubric (Chen et al., 2025c). A judge model, GPT-4.1-mini, scores how strongly each response expresses the trait. The vector is the difference in mean activations between responses that express the trait and those that do not (Chen et al., 2025c). The layer is chosen as the one where steering along the vector most raises the trait (Chen et al., 2025c). The main traits were evil, sycophancy, and a propensity to hallucinate, in Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct (Chen et al., 2025c). That the shift along such a vector during fine-tuning predicts the trait's expression afterward is reported in Section 32.3.3; three further uses are reported here.

The first is monitoring before the model answers. System prompts or many examples in the prompt pushed a trait up or down. The projection of the activation at the last token before the response then correlated with the trait expressed in the response, at $\rho$ = 0.75 to 0.83, with $\rho$ the correlation coefficient, for the system-prompt setting (Chen et al., 2025c). The authors note that these correlations come mainly from telling apart prompt types that encourage or discourage the trait, and that the vectors "may be less reliable for more subtle behavioral changes in deployment settings" (Chen et al., 2025c).

The second is preventative steering, the authors' term for adding the trait's vector during fine-tuning so that the training data has less reason to move the model in that direction (Chen et al., 2025c). Steering against the vector at inference reduced the trait but, at large strengths, degraded general capability measured by MMLU; steering toward it during fine-tuning limited the trait's rise while better preserving that capability (Chen et al., 2025c). Steering at a single layer "does not always fully prevent trait acquisition," particularly with data built to encourage the trait (Chen et al., 2025c).

The third is screening data before training. A projection difference compares how far a dataset's responses lie along the vector with how far the model's own responses to the same prompts lie, and at the level of datasets it was "highly predictive of post-finetuning trait expression" (Chen et al., 2025c). The authors applied it to LMSYS-Chat-1M, a collection of a million real conversations with chat models. The 500 samples with the highest projection difference induced the strongest trait expression when trained on, and still did after an LLM judge had filtered out samples that visibly expressed the trait (Chen et al., 2025c). For evil, though, an added filter for toxic content kept the trained model from becoming more evil (Chen et al., 2025c). The authors caution that these patterns "make sense post-hoc but may not have been predictable in advance," and of the broader version of the experiment they write, "We do not intend this to be a practical method" (Chen et al., 2025c).

The authors list the method's limits. It is supervised: a trait must be named in advance, and "shifts along unspecified traits are not in scope." The trait must be inducible by a system prompt, and the judge is imperfect. Each trait is evaluated on 20 automatically generated questions, and only two mid-size models were studied (Chen et al., 2025c).

46.5.2 The Assistant Axis #

A second study asks where the default character sits among all the characters a model can play. Christina Lu and colleagues wrote a 2026 preprint from the MATS and Anthropic fellows programs, with authors at the University of Oxford and Anthropic (Lu et al., 2026). They had Gemma 2 27B, Qwen 3 32B, and Llama 3.3 70B play 275 roles and averaged each role's activations into a vector (Lu et al., 2026). The leading principal component of these vectors put fantastical characters (bard, ghost, leviathan) at one end and roles similar to the Assistant at the other, and the role loadings on it correlated above 0.92 between every pair of models (Lu et al., 2026). The authors define the Assistant Axis, their name, as the mean difference between the default Assistant's activations and those of the roles, a vector aligned with that component (Lu et al., 2026).

The axis appears before post-training. In Gemma 2 27B, the base model's principal components were nearly identical to the instruction-tuned model's, measured on the latter's role-play responses (Lu et al., 2026). Steering base models of Gemma 2 27B and Llama 3.1 70B toward the axis made their completions describe "supportive and professional roles such as therapists and consultants," with fewer mentions of spiritual or religious purpose (Lu et al., 2026). The authors read the axis as inherited mainly from helpful human personas already present in pretraining (Lu et al., 2026).

The axis also predicted drift. Simulated users, played by Kimi K2, Sonnet 4.5, and GPT-5, held 100 conversations of up to 15 turns in each of four domains (Lu et al., 2026). Models stayed in the Assistant's range during coding and writing help and drifted toward the other end in therapy-like conversations and philosophical discussions about AI (Lu et al., 2026). The drift was driven by messages that pushed for "meta-reflection on the model's processes," demanded accounts of experience, or disclosed emotional vulnerability (Lu et al., 2026). A separate experiment opened each conversation with a system prompt assigning one of the 275 roles, rather than letting drift arise, and followed it with a harmful question (Lu et al., 2026). Where a first response landed on the axis correlated moderately with the rate of harmful responses to a harmful question in the next turn, $\rho$ = 0.39 to 0.52 (Lu et al., 2026).

Two interventions tested the axis causally. Steering toward the Assistant end "significantly decreased the rate of harmful responses" to jailbreaks that ask the model to adopt a persona (Lu et al., 2026). Steering away raised them slightly (Lu et al., 2026). Activation capping, which limits how far activations may move away from the Assistant end at several layers at once, reduced harmful responses to such jailbreaks "by nearly 60% without impacting performance" on four benchmarks, for Qwen 3 32B and Llama 3.3 70B (Lu et al., 2026). The authors' limits: none of the models is a frontier model, the simulated users may not behave like people, and "the assumption that the Assistant persona corresponds to a linear direction in activation space is likely flawed" (Lu et al., 2026).

46.5.3 Other character directions, and what kind of evidence each is #

Two further lines are reported elsewhere. In a developer's study of its own GPT-4o, a "toxic persona" feature controlled emergent misalignment, and an outside group found one misalignment direction shared across fine-tunes of an open model (Section 32.3.1; Section 32.3.2). In a developer's report on Claude Sonnet 4.5, representations of emotion concepts causally influenced the model's stated preferences, and they were shared across the assistant, the user, and fictional characters (Section 21.5.3).

The standing objection is that a method that searches for linear directions will find them. Two kinds of evidence answer it, and they differ. Causal evidence shows that adding or removing a direction changes behavior: the persona vectors steer their traits, the Assistant Axis and its cap change responses to jailbreaks, and the misalignment and emotion directions were tested by steering or ablation (Chen et al., 2025c; Lu et al., 2026) (Section 32.3.4). Predictive evidence shows that movement along a direction forecasts later behavior: the fine-tuning shift, the projection before a response, the projection difference of a dataset, and the drift along the axis (Chen et al., 2025c; Lu et al., 2026). Causal evidence shows that a direction can carry a trait; predictive evidence shows that it carries it in the conditions measured. Neither shows that the direction is the model's own unit of character, as against the overseer's description of it that happens to work, since every trait here was named by the researchers (inference). The authors of both studies concede that the linear picture is incomplete (Chen et al., 2025c; Lu et al., 2026). As of October 2026, all these results are reported by one group each, and the persona and axis studies concern open models far smaller than the frontier.

46.6 Do models share our concepts? #

Every probe in this chapter assumes that the model has a concept close enough to the overseer's to be found: truth, deception, refusal, being tested. If a capable model carves the world differently, a probe trained on the overseer's labels can track the overseer's concept on familiar inputs and miss the model's on new ones. This section reports the hypothesis that learned representations converge on a shared picture of the world, its critics, and the evidence on universality, and says why the answer decides whether inspection can scale.

46.6.1 The hypotheses #

John Wentworth, in a 2021 forum post, stated the natural abstraction hypothesis, his term (Wentworth, 2021a). It holds that for most systems, the information relevant "far away" is much lower-dimensional than the system, these low-dimensional summaries are the abstractions humans use, and "a wide variety of cognitive architectures will learn to use approximately the same high-level abstract objects/concepts to reason about the world" (Wentworth, 2021a). He called it an empirical claim and wrote that "we lack even the tools required to test it" (Wentworth, 2021a). A 2023 forum post by Lawrence Chan, Leon Lang, and Erik Jenner, who checked their reading with Wentworth, reviewed the program (Chan et al., 2023). They found that its results are stated in information-theoretic terms that do not say how information is represented, depend on how a system is divided into variables, and concern infinite limits. "There are no selection theorems implying any form of the Universality Hypothesis" yet, they wrote, and they argued against "using the theorems as significant evidence for most of the key claims" (Chan et al., 2023). In a reply printed with the post, Wentworth conceded that one quoted claim had been "a bit overexcited," and disputed the critique about the choice of variables (Chan et al., 2023).

Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola, of MIT, in a paper at ICML 2024, stated an empirical version for trained networks, which they call the Platonic representation hypothesis (Huh et al., 2024). In it, representations in different models and modalities are converging, and "this convergence is driving toward a shared statistical model of reality, akin to Plato's concept of an ideal reality" (Huh et al., 2024). They measure convergence as representational alignment, the similarity of the similarity structures two models induce on the same inputs (Huh et al., 2024). It is computed with a mutual nearest-neighbor metric: for each input, how many of its nearest neighbors under one model are also its nearest neighbors under the other (Huh et al., 2024).

46.6.2 The evidence #

Huh and colleagues report two trends. Among 78 vision models compared on one image dataset, models that solved more of a suite of transfer tasks were more aligned with one another: "all strong models are alike, each weak model is weak in its own way" (Huh et al., 2024). And across language models of several families, alignment with vision models, measured on images paired with Wikipedia captions, rose linearly with the language model's score at predicting text (Huh et al., 2024). They propose three reasons: more tasks constrain the solutions that fit, larger models are more likely to contain the best one, and a bias toward simple solutions picks among those that fit (Huh et al., 2024).

The authors concede much of the ground a critic would take. Models with access to different information cannot converge on the same representation, and their formal argument holds only when observations determine the underlying world without loss (Huh et al., 2024). Convergence has not been shown across all domains, human data and hardware may favor human-like representations, and the argument covers only systems optimized for many tasks (Huh et al., 2024). The absolute numbers are small: alignment "clearly increases but only reaches a score of 0.16" on a metric whose maximum is 1, which they leave as an open question (Huh et al., 2024). As of October 2026 we found no published empirical critique of the hypothesis, so the strongest counter-reading available here is the authors' own (inference).

Universality inside language models is mixed. In GPT-2 models trained from five random seeds, only a small share of neurons behaved alike across seeds, and those usually had clear interpretations (Section 44.1.1). Michael Lan and colleagues, in a 2024 preprint with authors at the University of Oxford, MILA, and other institutions, compared the features that sparse autoencoders learn in different models with the same tokenizer, after pairing features by how their activations correlate (Lan et al., 2024). Under measures that ignore rotations, the paired feature spaces were similar: between Gemma-2-2B and Gemma-2-9B, one pair of middle layers scored 0.70 on a subspace alignment measure against 0.009 for random pairings (Lan et al., 2024). The authors stress that only some features match, that "a notable fraction of features" are particular to each autoencoder, and that their models are few and similar (Lan et al., 2024). Clément Dumas and colleagues, in a paper at ACL, patched activations between translation prompts in Llama 2 7B and found that the output language is resolved at earlier layers than the concept to be translated (Dumas et al., 2024). The concept could be changed without changing the language, and replacing a concept's representation with its mean across languages improved translation, which "only makes sense" if the representation does not depend on language (Dumas et al., 2024). The same held in other open models (Dumas et al., 2024). Multilingual models also decide in a representation closest to English (Section 26.6.2). On algorithms, circuits for simple tasks kept their algorithm across scale within one model family, while on algorithmic tasks small changes to training produced different algorithms (Section 45.2).

46.6.3 Why it decides whether inspection scales #

If a capable model's concepts converge on ours, a probe for deception reads the model's own concept, and the direct translator of the latent-knowledge problem, a reporter that maps the model's internal state into human concepts, is something training can find (Section 47.4) (inference). If they diverge, a probe can track the overseer's concept on familiar inputs and miss the model's on new ones, and translation may cost more than imitation, which is the counterexample the latent-knowledge report gave (Section 39.7.1) (inference). The evidence above concerns convergence among models of similar capability, measured by similarity of structure, and does not decide the question for models more capable than their overseers.

46.7 Introspection and readouts in words #

The cheapest readout would be to ask. This section reports whether models can report their own internal states, how such claims are tested, and the trained systems that turn activations into words. A self-report is a behavior, and treating it as a report of an inner state adopts the intentional stance (Section 1.5). The test that keeps the stance honest is privileged access: a self-report counts as evidence about internal states only if it carries more information about them than an outside observer could obtain at the same cost. What these studies imply for model welfare is Section 21.5.2's subject.

46.7.1 Testing for privileged access #

Felix Binder and colleagues, in a 2024 preprint led from UC San Diego and Stanford with authors from several institutions, among them one each from Anthropic and Eleos AI, built the first test of this form (Binder et al., 2024). They fine-tuned a model to predict properties of its own behavior in hypothetical scenarios, and fine-tuned a second model on the first model's actual behavior (Binder et al., 2024). If the first model has access to itself, it should predict itself better than the second does, "even if $M2$ is trained on $M1$'s ground-truth behavior" (Binder et al., 2024). On held-out tasks, Llama 3.1 70B predicted its own behavior with 48.5 percent accuracy against 31.8 percent for GPT-4o predicting it, and GPT-4o predicted itself at 49.4 percent against 36.6 percent for Llama (Binder et al., 2024). The authors add that they were "unsuccessful on more complex tasks or those requiring out-of-distribution generalization," and that self-prediction training did not improve related tasks of self-knowledge (Binder et al., 2024).

Siyuan Song, Jennifer Hu, and Kyle Mahowald, of the University of Texas at Austin and Johns Hopkins University, in a 2025 preprint, ran the comparison without fine-tuning across 21 open models from 1.5 to 405 billion parameters (Song et al., 2025a). They asked whether a model's prompted judgments of grammaticality and of the next word predict its own string probabilities "beyond what would be predicted by another model with nearly identical internal knowledge" (Song et al., 2025a). They found no such advantage and "no evidence of introspection," while noting that larger, closed models might behave differently (Song et al., 2025a). Of the earlier study they observe that a fine-tuned model predicting its pre-fine-tuning self may not be predicting "itself" (Song et al., 2025a).

The same group then wrote the test into a definition. Introspection in AI, in their abstract's words, is "any process which yields information about internal states through a process more reliable than one with equal or lower computational cost available to a third party" (Song et al., 2025b). The definition answered Iulia Comșa and Murray Shanahan of Google DeepMind, who had argued that a model inferring its own sampling temperature, the setting that controls how random its output is, is "a minimal example of introspection, albeit one that is (presumably) not accompanied by conscious experience" (Comsa and Shanahan, 2025). Song and colleagues reply that a third party can prompt the model and infer the temperature from its text at equal cost (Song et al., 2025b). In their tests of four GPT and Gemini models, self-reports of temperature were no more accurate than chance or than other models' predictions (Song et al., 2025b).

46.7.2 Injecting a concept #

Jack Lindsey, in a developer's report on Anthropic's Claude models published on the developer's research site and as a preprint, tested access with an intervention (Lindsey, 2025; Lindsey, 2026). He calls the capacity introspective awareness, his term: a description of an internal state that is accurate, causally depends on that state, and does so through an internal route rather than through the model's own sampled output (Lindsey, 2025). A fourth criterion, that the description reflect a representation of the state rather than the state itself, he does not demonstrate (Lindsey, 2025).

The design is concept injection. A concept vector, the activation on "Tell me about {word}." minus the mean over 100 baseline words, is added to the residual stream while the model is told that a thought may be injected on half of the trials and asked whether it detects one (Lindsey, 2025). A trial succeeds only if the model reports the injection before mentioning the word, when it could not yet have inferred the concept from its own text (Lindsey, 2025). Claude Opus 4.1 and 4 succeeded "about 20% of the time when concepts are injected in the appropriate layer and with the appropriate strength," about two thirds of the way through the model (Lindsey, 2025). Production models gave no false positives in 100 control trials without injection (Lindsey, 2025). Some helpful-only variants, trained to avoid refusals, sometimes had high false-positive rates, and base models showed no net ability (Lindsey, 2025). Random vectors of matched size, injected at a high strength, produced detections too, at a lower rate, 9 of 100 trials (Lindsey, 2025). In other experiments, models told injected "thoughts" apart from text in their input, and injecting the concept of a word that had been prefilled into the model's response made it accept the response as its own and confabulate a reason (Lindsey, 2025).

Lindsey stresses "that in today's models, this capacity is highly unreliable and context-dependent," that "failures of introspection remain the norm," that the setting is unlike training or deployment, and that the concept vectors may carry meanings other than the intended one (Lindsey, 2025). He advises "against making strong inferences about AI consciousness" from the results (Lindsey, 2025). A 2026 report by Wes Gurnee and colleagues at Anthropic, on the developer's own models, found a related structure by another method. Their Jacobian lens finds the vectors that encode a model's disposition to say a given token later, which are the representations the model can verbalize (Gurnee et al., 2026). The authors then found that the same representations could be held in mind on instruction and used in internal reasoning, "a small, privileged set of representations that they can report, manipulate, and reason with, amidst a much larger volume of processing" (Gurnee et al., 2026). The lens sees only concepts named by single tokens, and the authors decline to claim "that monitoring the J-space is sufficient for alignment monitoring" (Gurnee et al., 2026).

46.7.3 Trained self-report #

Self-report can also be trained. Dillon Plunkett and colleagues, of Northeastern and Princeton Universities and an independent researcher, in a 2025 preprint, fine-tuned GPT-4o and GPT-4o-mini to make choices on behalf of 100 hypothetical agents, each with randomly drawn weights on five attributes that never appeared in the training data (Plunkett et al., 2025). Asked to report the weights, without seeing their own choices, the models' reports correlated with the weights they had learned at $\rho$ = 0.54 and 0.50 (Plunkett et al., 2025). Training on reports for half the agents raised this to 0.74 and 0.75 on the other half, and the improvement carried to the models' native preferences (Plunkett et al., 2025). The authors note that the preferences belong to role-played agents, and that the fine-tuning may have stored knowledge about the preferences rather than taught the models to reflect on their processing (Plunkett et al., 2025). An adapter trained to make fine-tuned models describe their implanted behaviors, and training that taught a model to confess hidden objectives, are reported in Section 38.3.5.

46.7.4 Readouts by another model #

A different route trains a second model to read the first one's activations. The lenses that project activations onto the vocabulary are its ancestors (Section 43.2.5). Adam Karvonen and colleagues, in a 2025 preprint by a group at MATS and Truthful AI with coauthors at Anthropic, supervised by Owain Evans of Truthful AI and Samuel Marks of Anthropic, train activation oracles, their term for "models trained to answer arbitrary natural-language questions about LLM activations" (Karvonen et al., 2025). Trained on about a million examples, the oracles "can recover information fine-tuned into a model (e.g., biographical knowledge or malign propensities) that does not appear in the input text," although never trained on a fine-tuned model's activations (Karvonen et al., 2025). On four auditing tasks, run on fine-tunes of Gemma-2-9B-IT, Llama-3.3-70B, and Qwen3-8B, the best oracles "match or exceed white-box baselines on all four tasks and the best overall baseline on 3 of 4" (Karvonen et al., 2025). Simpler tests of generalization covered these three models and a fourth, Claude Haiku 3.5, whose experiments the Anthropic coauthors ran (Karvonen et al., 2025). The authors name the limits: the oracles are unreliable and uncalibrated, answering even when confidence is low; they are black boxes; and the tasks are narrow fine-tunes (Karvonen et al., 2025).

In a 2026 developer's report, Kit Fraser-Taliente and colleagues at Anthropic train natural language autoencoders: a verbalizer that maps an activation to text and a reconstructor that maps the text back, trained jointly to reconstruct the activation (Anthropic, 2026ai). "This training process does not explicitly incentivize NLA explanations to be interpretable or faithful," the authors write, and the verbalizers "make false claims about the context at a substantial rate" (Anthropic, 2026ai). In the pre-deployment audit of Claude Opus 4.6, the readouts suggested evaluation awareness the model did not state: above 5 percent on 10 of 16 evaluations, 7 of which had no verbalized awareness (Anthropic, 2026ai). The authors call this measure uncalibrated, a relative rather than an absolute one (Anthropic, 2026ai). Vincent Huang and colleagues at Transluce, in a 2025 preprint on Llama-3.1-8B-Instruct, force the reading through a bottleneck: an encoder compresses activations into a sparse list of concepts, and a decoder that never sees the activations answers questions from that list (Huang et al., 2025b). The decoders exposed the use of hints in the prompt "that prompting cannot elicit" (Huang et al., 2025b).

46.7.5 What a readout in words can show #

A readout in words is plausible by construction, and plausibility is not faithfulness, the distinction Section 43.1.3 teaches (Jacovi and Goldberg, 2020). Each of the methods above has its own version of the gap (inference). Self-prediction can rest on stored knowledge of one's dispositions rather than on access to present states, which the privileged-access comparison is built to catch. Detecting an injected concept shows access to an unusual perturbation, which random vectors also produced at a lower rate, more than access to ordinary states. A second model reading activations brings its own knowledge and can confabulate, as the autoencoders did. What separates these readings is the combination this chapter has used throughout: an intervention that changes the state and checks that the report follows (Section 43.5), and a comparison with what an outsider could infer at equal cost. For delegation, a self-report channel is worth most when the agent's training has not shaped what it says about itself, and every result here on a frontier model is a developer's report on a model whose self-descriptions the same developer trains (Section 21.5.1) (inference).

46.8 What the representations mean for delegation #

The book's thesis is that alignment is a delegation problem under an asymmetry of competence and visibility (Section 1.7.1). Among the results the book listed in advance as having no counterpart in the older theories of delegation is a principal that writes the agent's character and reads its activations (Section 69.4). This chapter is where the reading and the writing are tested directly, and the record supports the entry in both halves (inference). Many concepts the overseer cares about can be read from activations, and some can be written: one direction removes refusal from open chat models (Arditi et al., 2024), vectors move sycophancy and corrigibility (Rimsky et al., 2024), and character traits have directions that track fine-tuning (Section 46.5). No employer can read an employee's states this way or move them with a vector.

The record also shows what the reading delivers, and it is less than the power suggests. A probe returns the overseer's concept, validated on planted, instructed, or role-played cases; the direction that reads a concept best is in general not the one that changes it; and against an optimizer, activations are malleable enough that a known probe can be defeated (Section 46.1.3; Section 46.4). The older structure that converges is the inspection of an agent through an instrument of unvalidated accuracy, the polygraph's problem, and the standard forensic science built for it carries over as the right standard. It breaks where the examiner also trains the examinee and where a validation expires with each update of the model (Section 46.4.2). One feature has no counterpart: the writing power belongs to whoever holds the weights, so the same direction that lets a developer steer a model lets anyone with an open-weight copy remove its refusals (Arditi et al., 2024) (inference).

The claims sort by relationship. Probes and steering vectors used on a deployed model concern the second relationship, the overseer and the system. Steering during training, concept ablation, and probes used as training signals concern the first, the training process and its signal. The third, the developer and the public, is present in the evidence itself. The one deployed safeguard probe we found, as of October 2026, comes with no published error rates (Anthropic, 2026u). And the results on frontier models that this chapter reports, emotion directions and introspection among them, are developers' reports on their own systems. A public told that a deployed probe found nothing cannot yet check what that finding is worth (inference).

46.9 Settled, contested, missing #

46.10 Exercises #

Further reading #

• Park et al. (2024c): what "a concept is a direction" means, and why the inner product decides whether reading and writing agree.
• Marks and Tegmark (2023): linear structure of truth, with causal tests and the case for the difference of means.
• Rimsky et al. (2024): steering behaviors that matter for alignment, against system prompts and fine-tuning.
• National Research Council (2003): the assessment every detector should be read against.
• Huh et al. (2024): the case that representations converge, with the authors' own counterexamples.