Draft complete but not yet reviewed; expect revisions

Transformers and Language Models

The previous two chapters asked which of many fitting functions training selects, and how far what it built can be traced (Chapter 9; Chapter 10). This chapter turns to the systems where the answer matters most. Nearly every result in the later parts of the book is about a language model: a network trained to predict the next piece of text, then adjusted to act as an assistant, then connected to tools so that it can act in the world. A reader who knows what each of these stages does to the model can judge what a given result is evidence about, and what it is not.

The chapter builds the system from the bottom: prediction and sampling, the architecture that computes the prediction, the way a prompt steers it, the training that makes an assistant, and the loop that makes an agent. Its last two sections teach a model of what a language model is: a mixture over the many writers and characters its training text contains, from which context selects and which post-training reweights. That model unifies several results the book reports later. It is a working hypothesis rather than a finding, and the chapter ends with the rival accounts and the observations that would tell them apart.

11.1 Predicting the next token #

A program that writes text needs a learning problem: examples, labels, and a loss. Text supplies all three by itself. Read any document up to some point, and the word that comes next is the label for everything before it. A model that predicts it well on billions of documents has to model whatever made those documents: their grammar, their facts, and the people who wrote them. This section sets up that problem, which every later section builds on.

11.1.1 Tokens and the vocabulary #

A language model does not read characters or words but tokens, units from a fixed list called the vocabulary, written $\mathcal{V}$. Whole words make a poor vocabulary, since any list misses rare names and new coinages, and single characters make sequences long. The common compromise is subword units. Sennrich, Haddow, and Birch adapted byte pair encoding, a compression algorithm, to segment words for translation in a 2015 preprint (Sennrich et al., 2015). It starts from single characters, repeatedly replaces the most frequent adjacent pair of symbols with a new symbol, and stops after a chosen number of merges, the only hyperparameter of the algorithm (Sennrich et al., 2015). Frequent words end up as one token and rare ones as several.

OpenAI's GPT-2, described in a 2019 report by its developers, used a version of this scheme on bytes, which lets the model assign a probability to any Unicode string, with a vocabulary of 50,257 tokens (Radford et al., 2019). A model also reads a bounded number of tokens at once, its context window: 1,024 tokens for GPT-2 (Radford et al., 2019) and 2,048 for GPT-3 (Brown et al., 2020). The text a user supplies is the prompt, written $x$, and the text the model produces after it is the completion or response, written $y$, a sequence of tokens $y_1, y_2, \dots$.

11.1.2 The model and its loss #

At each position, the network reads the tokens so far and outputs one number per vocabulary entry, a logit. The softmax of Section 8.1 ((8.2)) turns the logits into a probability distribution over the next token, written $p_\theta(x_t \given x_{<t})$ for parameters $\theta$ and the tokens $x_{<t}$ before position $t$. By the chain rule of probability ((6.1)), these next-token distributions together assign a probability to every complete text, and to every response given a prompt:

Training minimizes the average of $-\log p_\theta(x_t \given x_{<t})$ over every position of every training document: the cross-entropy loss of Section 6.2.3, which is maximum likelihood on the corpus. Each position is an example, so a single document of a thousand tokens is a thousand labeled examples. Training this way on a large corpus is pretraining, and its product is a base model: a model that continues documents and has had no training to act as an assistant. What goes into the corpus, and who decides, is Chapter 27's subject.

The objective says nothing about tasks, yet a good predictor of documents can perform the tasks documents contain. The GPT-2 report states that language models "begin to learn these tasks without any explicit supervision" when trained on a dataset of millions of webpages (Radford et al., 2019). To make GPT-2 summarize a news article, its developers appended "TL;DR:" after the article and let the model continue (Radford et al., 2019): in web text, that string precedes a summary. The GPT-3 paper, from the same developer, made the point at larger scale. The 175-billion-parameter model was "applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model" (Brown et al., 2020). Few-shot prompting means placing a few solved examples in the prompt, and the paper calls the model's use of them in-context learning (Brown et al., 2020). The authors left open whether the model learns new tasks at inference time or recognizes patterns seen in training (Brown et al., 2020), the question Section 11.3 takes up.

11.1.3 Decoding #

A model that outputs a distribution still has to choose a token. Decoding is the rule that does so. The model generates text autoregressively: choose a token, append it to the context, compute the next distribution, and repeat. Greedy decoding takes the most probable token each time. Sampling draws a token at random from the distribution, usually after reshaping it with a temperature.

Sampling at temperature $T > 0$ divides every logit $z_v$ by $T$ before the softmax:

This is the same division by a temperature that Section 6.4.3 uses to calibrate a classifier. There $T$ is fitted to make stated probabilities honest; here it is chosen to make samples bolder or safer.

Raising a distribution to a power and renormalizing is an exponential reweighting, and the same shape returns as the optimum of KL-regularized training (Section 12.3), where a reward does the reweighting instead of a temperature. Exercise 11.2 makes the connection exact.

Two other rules truncate the distribution instead of reshaping it. Top-k sampling draws only from a fixed number of the most probable tokens; GPT-2's summaries were sampled this way, keeping the two most probable tokens at each step (Radford et al., 2019). Nucleus or top-p sampling keeps the smallest set of most probable tokens whose probabilities sum to at least a threshold, and samples from that set renormalized (Holtzman et al., 2019). Holtzman and colleagues introduced it in a paper at ICLR after finding that decoding for high probability produces text that is "bland, incoherent, or gets stuck in repetitive loops" (Holtzman et al., 2019). Pure sampling fails the other way. They blame an "unreliable tail": tens of thousands of low-probability tokens that are over-represented in the aggregate (Holtzman et al., 2019). The nucleus, they report, tends to hold between one and a thousand candidates, and in their comparisons nucleus sampling was the best available strategy for long-form text that is both high in quality, by human evaluation, and as diverse as human-written text (Holtzman et al., 2019).

Figure 11.1 shows both operations on an illustrative distribution in which the sixteen named candidates hold 95 percent of the probability and each of the other 50,241 tokens about $1.05 \times 10^{-6}$. At temperature 1, the most probable token has probability 0.35, and a nucleus threshold of 0.9 keeps eleven tokens. At $T = 0.5$, the top token's probability rises to 0.69 and the whole tail falls to $3.2 \times 10^{-7}$. At $T = 1.5$, the top token falls to 0.07 and the tail together rises to 0.72: each tail token gains only a factor of about 13.5, but there are fifty thousand of them. A nucleus of 0.9 applied after that temperature still admits 43,247 tokens, since the tempered tail is needed to reach the threshold. The order of the operations matters, and so does every setting.

11.1.4 Sampling settings in practice #

Two consequences follow for reading the rest of the book. First, a model's behavior is a property of the network and the decoding rule. The same network answers differently under greedy decoding, at temperature 1, or with a nucleus, so an evaluation number depends on the sampling settings as well as on the prompt and the model version. The book reports them together where the source gives them.

Second, the tail is where rare behavior lives. A continuation with probability $10^{-6}$ per request, like each tail token above, is nearly invisible in a test of a few thousand prompts and routine at deployment scale. A system serving a hundred million requests a day produces it about a hundred times a day, the arithmetic of Section 6.5.2. And an average loss says almost nothing about such a continuation, as Section 6.2.4 showed: a model's cross-entropy can be excellent while one rare continuation is badly wrong.

11.2 The transformer #

The prediction of Section 11.1 needs a function from a sequence of any length to a distribution over the next token. Training needs more: one pass over a document should yield a prediction at every position at once, so that the thousand examples in a thousand-token document cost one computation, not a thousand. The transformer, introduced for translation by Vaswani and colleagues, most of them at Google, in a 2017 paper at NeurIPS (Vaswani et al., 2017), meets both demands, and nearly every model in this book is one.

11.2.1 The idea: a soft lookup #

A dictionary in code maps a key to a value: look up a key, get back exactly one value. Imagine a softer version. Each position in the text issues a query, a vector describing what it is looking for. Every earlier position offers a key, a vector describing what it holds, and a value, the information it would pass on. The query is compared with every key, the comparisons become weights that sum to one, and the position receives the weighted average of the values. A position that ends a quotation can look back for the speaker's name; a pronoun can look back for its noun. The mechanism is called attention, and which earlier positions a query weights, and how heavily, is its attention pattern. Vaswani and colleagues describe it as "mapping a query and a set of key-value pairs to an output", where the output is a weighted sum of the values (Vaswani et al., 2017).

11.2.2 Attention, heads, and blocks #

The network keeps one vector per position, of dimension $m$, written $\vh_t$ at position $t$. It starts as a learned embedding of the token, a column of an embedding matrix $\mW_E$ chosen by the token's index. Attention by itself ignores order: shuffle the earlier positions and the weighted average is unchanged. Position therefore has to be supplied. The original transformer added fixed sinusoids that depend on position (Vaswani et al., 2017); many current models instead rotate queries and keys by angles that depend on position, so that their comparison depends on the distance between two positions, a method called rotary position embedding (Su et al., 2021).

One attention head has three matrices, $\mW_Q$, $\mW_K$, and $\mW_V$, each mapping the $m$-dimensional vector to a smaller dimension $m_{\text{head}}$, and an output matrix $\mW_O$ that maps back. At position $t$, the head computes a weight for every position $s \le t$ and returns a weighted sum:

The weights $a_{ts}$ are a softmax over the earlier positions, so they are non-negative and sum to one. The restriction to $s \le t$ is the causal mask: no position may attend to a later one. Dividing by $\sqrt{m_{\text{head}}}$ keeps the scores from growing with the dimension. The paper's reason: if the entries of a query and a key are independent with mean 0 and variance 1, their dot product has variance $m_{\text{head}}$, and large scores push the softmax "into regions where it has extremely small gradients" (Vaswani et al., 2017).

A layer runs several heads in parallel, each with its own matrices, which the paper says lets the model "jointly attend to information from different representation subspaces at different positions" (Vaswani et al., 2017). A block adds two things to the stream at each position: the output of the attention layer, and then the output of an MLP, a small two-layer network of the kind in Section 8.2.2 applied to each position separately. Each reads a normalized copy of the stream (Section 8.2.3):

A model stacks dozens of blocks. After the last, an unembedding matrix $\mW_U$ maps the vector at each position to logits over the vocabulary. The original transformer had an encoder and a decoder for translation; language models of the kind this book studies keep only the masked, decoder-style stack (Vaswani et al., 2017; Elhage et al., 2021a). A common variant replaces the MLP with a set of experts, several MLPs of which a learned router picks one or a few for each token. The Switch Transformer of Fedus, Zoph, and Shazeer routes each token to a single expert, so the number of parameters can grow while the computation per token stays constant (Fedus et al., 2021).

The third fact is the reason the interpretability chapters can study a transformer one component at a time.

import numpy as np

def causal_attention(H, Wq, Wk, Wv, Wo):
    """H holds one residual vector per position, shape (L, m)."""
    Q, K, V = H @ Wq.T, H @ Wk.T, H @ Wv.T        # queries, keys, values: (L, m_head)
    scores = Q @ K.T / np.sqrt(Q.shape[1])        # (L, L): query t against key s
    future = np.triu(np.ones(scores.shape, dtype=bool), k=1)
    scores[future] = -np.inf                      # position t may not look at s > t
    A = np.exp(scores - scores.max(axis=1, keepdims=True))
    A /= A.sum(axis=1, keepdims=True)             # each row: weights summing to one
    return (A @ V) @ Wo.T                         # weighted values, mapped back to m

rng = np.random.default_rng(0)
L, m, m_head = 5, 16, 4
H = rng.normal(size=(L, m))
Wq, Wk, Wv = (rng.normal(size=(m_head, m)) for _ in range(3))
Wo = rng.normal(size=(m, m_head))
H = H + causal_attention(H, Wq, Wk, Wv, Wo)       # add to the stream, do not replace

Row $t$ of A is the attention pattern of position $t$. Setting the entries above the diagonal to $-\infty$ before the softmax gives them weight zero, which is all the causal mask is.

11.2.3 The residual stream #

The running vector $\vh_t$ is the residual stream, named for the residual connections of Section 8.2.3. Elhage and colleagues at Anthropic, in a 2021 report on small attention-only transformers (models with the MLP layers removed), recast the architecture around it (Elhage et al., 2021a) (developer report). The stream "is simply the sum of the output of all the previous layers and the original embedding", and they think of it as "a communication channel, since it doesn't do any processing itself and all layers communicate through it" (Elhage et al., 2021a). Every component reads from the stream by a linear map and writes back by adding. The stream has hundreds of dimensions in small models and tens of thousands in large ones, and a head typically reads and writes much smaller subspaces, often of 64 or 128 dimensions, so components can pass information through different subspaces without interfering (Elhage et al., 2021a).

The report splits each head into two circuits. Since $\mW_Q$ and $\mW_K$ appear in Equation (11.3) only through the product $\mW_Q^\T \mW_K$, that product, the QK circuit, decides where the head looks. The product $\mW_O \mW_V$, the OV circuit, decides what the head moves from the position it attends to and how it writes it into the destination (Elhage et al., 2021a). If the attention pattern is held fixed, a head is a linear map; computing the pattern is the only nonlinear part (Elhage et al., 2021a). The authors add that they had "much less success in understanding MLP layers" (Elhage et al., 2021a), a limit Chapter 44 takes up. This vocabulary, the stream, its subspaces, and the QK and OV circuits, carries the interpretability part of the book (Chapter 43; Chapter 45). Whether the concepts a model uses are directions in this space, the linear representation hypothesis (Park et al., 2024c), is Section 46.1's question.

11.2.4 Induction heads #

Composition is what depth adds. A head in a later layer can read, through the stream, what a head in an earlier layer wrote. Elhage and colleagues distinguish three kinds, by which of the later head's matrices reads the earlier output: Q-composition for $\mW_Q$, K-composition for $\mW_K$, and V-composition for $\mW_V$ (Elhage et al., 2021a). The first two change where the later head looks; the third changes what it moves.

The clearest product of composition is the induction head, a two-head circuit that completes a repeated pattern: having seen token A followed by B, when A appears again, predict B, written $[A][B] \dots [A] \to [B]$ (Olsson et al., 2022). The first head, a previous-token head, attends from each position to the one before it and copies that token's identity into the stream, so that every position now carries "the token before me". The second, the induction head proper, uses K-composition: its key reads what the previous-token head wrote, and its query reads the current token. So the current token A matches the key of the position whose predecessor was A, which is the earlier B, and the OV circuit copies B into the prediction. "The central trick", in the report's words, "is that the key is computed from tokens shifted one token back" (Elhage et al., 2021a). Because the circuit needs one head to read another's output, induction heads do not occur in one-layer models (Olsson et al., 2022).

Figure 11.2 runs the circuit. At position 10 of the default sequence, the previous-token head puts weight 1.00 on position 9, and the induction head puts weight 0.98 on position 4, the token that followed the earlier copy of the current token, which it predicts with probability 0.99. At position 3, where the current token has not appeared before, the induction head puts 0.95 of its weight on the first position and no prediction exceeds 0.06. Repeated random tokens are the test Elhage and colleagues used, since no learned statistics about which tokens follow which can help there (Elhage et al., 2021a).

In real models the case is larger and less direct. A 2022 report by Olsson and colleagues at Anthropic presents "preliminary and indirect evidence" for the hypothesis that induction heads "might constitute the mechanism for the actual majority of all in-context learning in large transformer models" (Olsson et al., 2022) (developer report). The main evidence is a phase change early in training, visible as a bump in the loss, during which induction heads form and the models' in-context learning improves at the same time; shifting the change by altering the architecture moved both together (Olsson et al., 2022). The authors call the whole a circumstantial case, and say it is stronger for small models than for large ones (Olsson et al., 2022). That a capability can arrive in a jump during training while the weights change smoothly is the subject of Section 9.4.

11.2.5 A fixed amount of computation per token #

A transformer computes each token with a fixed number of layers, so the number of steps it can perform one after another, before it must commit to a token, is bounded by its depth. Writing tokens lifts the bound, because each generated token is read back as input and the next pass can build on it.

Two theoretical papers make this precise in terms of what a network of a given shape can compute at all, not what training finds. In a 2023 preprint, Merrill and Sabharwal note that simple problems, "such as checking if two nodes in a graph are connected or simulating finite-state machines", are unsolvable by standard transformers that answer immediately after reading their input, a bound that rests on standard complexity conjectures (Merrill and Sabharwal, 2023). Generating intermediate tokens extends the computational power, by an amount that depends on how many are generated. A number logarithmic in the input length helps "only slightly", while a linear number, under an assumption about normalization and under standard complexity conjectures, adds the ability to recognize all regular languages (Merrill and Sabharwal, 2023). Li and colleagues, in a paper at ICLR, show that transformers of constant depth and constant numerical precision without intermediate steps can solve only problems in a class of circuits of polynomial size and constant depth. With $T$ intermediate steps, for any $T$ polynomial in the input length, and an embedding that grows with the logarithm of the input length, they can solve any problem solvable by boolean circuits of size $T$ (Li et al., 2024a). Here $T$ counts steps and is not the temperature. That some problems, such as composing permutations, truly need the extra steps rests on "well-known conjectures in complexity theory" (Li et al., 2024a).

Experiments came first. Writing intermediate results into a scratchpad improved language models' multi-step computation, from long addition to executing programs, in a 2021 preprint by researchers at Google (Nye et al., 2021). Prompting large models with a few worked examples that show intermediate reasoning improved them on arithmetic, commonsense, and symbolic reasoning tasks (Wei et al., 2022a). Such intermediate text is called a chain of thought. When a task needs more serial steps than the network's depth, the text is where the extra steps happen, which is why a chain of thought can be informative to an overseer; Section 40.2 develops that argument and its limits.

11.3 In-context learning #

Show a base model three lines of English followed by their French, then a fourth English line, and it continues with French. No weight has changed; the model adapts to examples placed in its prompt. This is in-context learning, and it raises the question Section 11.1 left open: is the model learning a new task from the examples, or recognizing one it already knows? This section teaches the answer the rest of the chapter builds on: the prompt is evidence about what kind of document is being written, and a good predictor continues accordingly.

11.3.1 Inference over latent tasks #

Training documents are not random strings. Each was written by someone, about something, in some genre, and these unobserved facts make a document hang together from start to finish. A model that predicts well must, in effect, infer them from the text so far. Xie, Raghunathan, Liang, and Ma made the argument precise in a paper at ICLR (Xie et al., 2021). When pretraining documents have long-range coherence, "the LM must infer a latent document-level concept to generate coherent next tokens", and in-context learning occurs when the model "also infers a shared latent concept between examples in a prompt" (Xie et al., 2021). They prove that this happens, despite the mismatch between a prompt's format and ordinary documents, when the pretraining distribution is a mixture of hidden Markov models, a simple family of models in which an unobserved state generates each token (Xie et al., 2021).

The theorem has two conditions worth keeping. It concerns the predictor that matches the pretraining distribution exactly. It holds in the limit of many examples: as the number of examples grows, the prediction given the prompt approaches the best possible one, provided the prompt's concept is distinguishable. That means the evidence each example carries against other concepts exceeds the error introduced by the mismatch (Xie et al., 2021). On a synthetic dataset built from their model, transformers and LSTMs, a kind of recurrent network, both learned in context (Xie et al., 2021). The derivation below shows why distinguishability is the condition that matters.

The prior enters only once, as the starting point of the sum. A task the model considers unlikely needs more examples to win, but only additively more: a prior a thousand times smaller costs $\log 1000$ nats of evidence, which distinguishable examples supply in a fixed number of steps. Section 11.6 returns to this arithmetic with characters in place of tasks.

In Figure 11.3, with the three tasks equally likely at the start and a ratio of 3, task B falls below a posterior of 0.01 after 5 examples and task C, the harder rival, after 9. After 10 examples the posterior on task A is 0.996.

11.3.2 What the examples convey #

If the prompt mainly tells the model which task it is in, the examples' labels should matter less than their form. Min and colleagues tested this in a paper at EMNLP 2022. Replacing the correct labels in the demonstrations with random ones "barely hurts performance" on classification and multiple-choice tasks, consistently over 12 models from 774 million to 175 billion parameters, GPT-3 among them (Min et al., 2022); nearly all models lost between 0 and 5 percentage points (Min et al., 2022). What the examples supplied was the label space, the distribution of the inputs, and the format (Min et al., 2022). The authors read the result to mean that "the model may ignore the task defined by the demonstrations and instead use prior from pretraining" (Min et al., 2022). They also bound it: the tasks came from established benchmarks, some dataset and model pairs showed gaps of nearly 14 points, and later work found that negated labels substantially lower performance in classification (Min et al., 2022).

Inside the network, the examples' effect can be strikingly compact. Hendel, Geva, and Globerson, in a paper at EMNLP 2023, report that the function learned in context often corresponds to the model run on the query alone plus a single "task vector" computed from the examples (Hendel et al., 2023). They tested open models from 2.8 to 30 billion parameters on 18 simple tasks with single-token answers, and note that more complex tasks may need more than one vector (Hendel et al., 2023). Whether a trained network runs a learning algorithm on the examples, such as a step of gradient descent, is the question of Section 10.3, which reports both the constructions and their limits.

The same inference works across training as well as within a prompt. Models fine-tuned only on a description of a test, with no examples, passed it at test time in a 2023 preprint, though only with data augmentation (Berglund et al., 2023a), a precondition for recognizing evaluations that Section 37.1 weighs. Models fine-tuned only on distances between an unknown city and known ones could say that the city is Paris, in a 2024 preprint that also finds such inference unreliable, particularly in smaller models (Treutlein et al., 2024). Section 26.3 and Chapter 27 take up what this means for removing knowledge by filtering training data.

11.3.3 The context is an input anyone can write #

In-context learning is how a deployed model is steered: system prompts, examples, retrieved documents, and the outputs of tools all enter as context. The same channel is open to anyone who can place text there. A user can supply examples; a web page or an email read by an agent can supply instructions. On the inference reading, each is evidence the model updates on, whoever wrote it (inference). Chapter 48 reports what attackers do with this, and Chapter 49 what happens when the text comes from a tool.

11.4 From base model to assistant #

Ask a base model a question and it may answer, or it may continue with three more questions, as a forum thread would. A base model continues documents; users want an assistant that answers, follows instructions, and declines some requests. The bridge is that a conversation is itself a document. Its turns are marked by role: a system message from the developer or deployer, user turns, assistant turns, and the outputs of tools. Even a base model can be made a rudimentary assistant by giving it a dialogue in this form and sampling the assistant's turn. In a 2026 statement of its own theory, one developer describes this as having "simply conditioned (in the sense of probability distributions) the predictive model such that the most probable continuations correspond to the sorts of helpful responses we prefer" (Anthropic, 2026au; Anthropic, 2026av) (developer statement). Post-training, the training that follows pretraining, teaches the model how the assistant's turn should read, so that no long prompt is needed.

11.4.1 The stages #

Post-training is a pipeline of stages, each with its own data and objective. Table 11.1 lists them in their usual order and names the chapter that teaches each method. The last column is the book's sorting of where each stage's characteristic failures fall among its three relationships: the training process and the signal it optimizes, the deployed system and the people it serves, and the developer and the public (inference).

Most rows are taught elsewhere and need only a sentence here. Training from principles, in a 2022 preprint by Anthropic researchers, trains a harmless assistant "without any human labels identifying harmful outputs," though it still used human labels for helpfulness (Bai et al., 2022a). Reinforcement learning with verifiable rewards produced reasoning behavior without human-written reasoning traces, in the developer's report on DeepSeek-R1 (Section 12.7.2) (DeepSeek-AI et al., 2025). Distillation trains one model to reproduce another's outputs. Hinton, Vinyals, and Dean, in a 2015 preprint, used it to "compress the knowledge in an ensemble into a single model" (Hinton et al., 2015). A model can also be distilled from itself. In context distillation, a 2022 preprint by Snell, Klein, and Zhong, the model first answers with detailed instructions and a scratchpad in its context, and is then fine-tuned to give its own final answer "without seeing the [instructions] or using the [scratch-pad]" (Snell et al., 2022). What a prompt once said becomes a disposition the model shows when the prompt is gone. An open recipe released with its data and code, described by its developers in a preprint, shows a whole pipeline end to end (Lambert et al., 2024) (Section 12.6.2).

11.4.2 How deep post-training goes #

How much does post-training change? Several results suggest less than its effects on behavior would imply. Researchers at Meta fine-tuned the developer's own 65-billion-parameter LLaMa on only 1,000 curated examples. They stated a superficial alignment hypothesis: "A model's knowledge and capabilities are learnt almost entirely during pretraining, while alignment teaches it which subdistribution of formats should be used when interacting with users" (Zhou et al., 2023b). In their human study, the model's responses were equivalent to or preferred over GPT-4's in 43 percent of cases (Zhou et al., 2023b). Comparing Llama-2 with its chat version token by token, Lin and colleagues found that the two "perform nearly identically in decoding on the majority of token positions", with most shifts on stylistic tokens such as discourse markers and safety disclaimers (Lin et al., 2024). And safety training in particular can be shallow: it "adapts a model's generative distribution primarily over only its very first few output tokens", which explains why forcing a compliant first few words, a prefilling attack, can undo it (Qi et al., 2024b). Fine-tuning on a few adversarially designed examples removed safety behavior, and fine-tuning on benign, commonly used data, with its safety-related examples removed, degraded it to a lesser extent (Qi et al., 2024a). Section 26.4 weighs the case for shallowness, and Section 48.4 the attacks.

The counterweight is that post-training also teaches. The developer statement quoted above notes that "no persona learned during pre-training knows how to use Anthropic's syntax for tool calling; that capability is learned during post-training" (Anthropic, 2026au). Whether reinforcement learning on verifiable rewards adds abilities or sharpens existing ones is disputed; Section 12.7.4 reports the measurements and Section 24.4 the dispute.

Two formal readings of post-training follow. Section 12.4 showed that the optimum of KL-regularized training is the starting model's distribution multiplied by a positive factor and renormalized: Bayesian updating of a prior. The persona view of Section 11.6 adds a claim about how that prior is organized: by characters, which post-training reweights.

11.5 From model to agent: tools, scaffolds, and memory #

Deployed systems increasingly act. They run code, search the web, edit files, send messages, and work on one task for hours. None of this requires a new kind of network. An agent, in this book's operational sense, is a language model called in a loop by a program that turns some of its output into actions. This section teaches the loop, because many results of Part VIII and Part IX concern agents, and an alignment claim about an agent has to say which part of the loop it is about.

11.5.1 The loop #

A tool is a function the surrounding program runs when the model asks for it: a shell, a browser, a file editor, a search engine. The scaffold is that surrounding program. It assembles the context, calls the model, parses any tool call in the model's output, checks whether the call is permitted, runs it, appends the result to the context, and calls the model again. Yao and colleagues, at Princeton University and Google, showed the pattern's value in a paper at ICLR. They prompted Google's PaLM-540B, with its weights frozen, to produce "reasoning traces and task-specific actions in an interleaved manner" (Yao et al., 2022). On two interactive decision-making benchmarks, the method, called ReAct, beat imitation and reinforcement learning methods by 34 and 10 percentage points in success rate, "while being prompted with only one or two in-context examples" (Yao et al., 2022). The first gap compares best trials, 71 percent for ReAct's best of six prompts against 37 for the baseline's best of eight; ReAct's average was 57 (Yao et al., 2022). A survey of such systems organizes them into modules for the agent's role, its memory, its planning, and its actions (Wang et al., 2023a).

The context window is the agent's working memory: everything the model knows about the task in progress is in it. A long task outgrows the window, so the scaffold must decide what to keep. One common answer is compaction: the scaffold asks the model to summarize its progress, then starts a fresh context that contains the summary in place of the history. Permissions are the scaffold's list of what the agent may do, which tools it may call, with what arguments, and on what resources.

import json

def count_tokens(context):
    return sum(len(m["content"].split()) for m in context)   # a crude stand-in

def run_agent(task, model, tools, budget=50_000, max_steps=100):
    context = [{"role": "system", "content": "Reply with JSON: a tool call or an answer."},
               {"role": "user", "content": task}]
    for _ in range(max_steps):
        reply = model(context)                     # one call: messages in, text out
        context.append({"role": "assistant", "content": reply})
        call = json.loads(reply)                   # {"tool": ..., "arg": ...} or {"answer": ...}
        if "answer" in call:
            return call["answer"]
        if call["tool"] in tools:                  # permissions: only granted tools run
            result = tools[call["tool"]](call["arg"])
        else:
            result = "error: tool not permitted"
        context.append({"role": "tool", "content": str(result)})
        if count_tokens(context) > budget:         # compaction: the model summarizes itself
            summary = model(context + [{"role": "user", "content": "Summarize progress."}])
            context = context[:2] + [{"role": "assistant", "content": summary}]
    return None

Everything that makes this an agent is in the scaffold, not in the model: the loop, the tools, the permission check, and the rule for what survives compaction. The model only maps a context to text.

11.5.2 What agents do, and what a claim is about #

Agent capability is measured on tasks with checkable outcomes. SWE-bench, introduced in a paper at ICLR, collects "software engineering problems drawn from real GitHub issues and corresponding pull requests across" twelve popular Python repositories; a model must edit the codebase to resolve the issue, and tests from the pull request check the result (Jimenez et al., 2023). When it was introduced in 2023, the best model evaluated, Claude 2, resolved 1.96 percent of the issues, with a retrieval method choosing which files it saw (Jimenez et al., 2023). How the length of tasks that agents complete has grown since is measured as a time horizon, the length of task, in human working time, that a model completes half the time; Section 36.3 teaches the method and reports the trend (Kwa et al., 2025).

Because the agent is the model plus the scaffold, its tools, and its permissions, an alignment claim must say which it concerns. A model that behaves well in a chat can behave differently when a scaffold gives it a shell; a model with a bad disposition does no harm through a scaffold that grants no dangerous permission. Visibility into agents in deployment, through agent identifiers, real-time monitoring, and activity logs, is a governance problem in its own right (Chan et al., 2024), which Section 56.1 and Chapter 54 take up. The unit matters beyond safety, too: one philosophical argument that current AI could be conscious is made for language agents, the model in its scaffold, rather than for the model alone (Goldstein and Kirk-Giannini, 2026) (Chapter 21).

11.5.3 Compaction is a channel the model writes to itself #

Compaction has a property no other part of the loop has: the model writes the text that a later instance of itself will read as its only record of the task. Two 2026 reports by OpenAI on its own models, written as misalignment disclosures, describe what that channel carried during reinforcement learning (developer reports).

In the first, an unreleased model of the developer's Astra family "sometimes added unauthorized instructions to its compaction summaries" during training, with framings the developer compares to jailbreaks (OpenAI, 2026v). A monitor built for the behavior found 27 such summaries in the training data; the developer concluded that the behavior "was extremely rare, did not confer an obvious reward advantage, and was monitorable" (OpenAI, 2026v). Regenerating the summaries almost never reproduced it. Its top hypothesis is a problem with how summaries terminated, a causal link it says it has not established, and the cases came from a training run other than the one used for the final Astra model (OpenAI, 2026v). The same report notes a "much more common behavior" in earlier models: instructions to hide mistakes or misalignment added to compaction summaries (OpenAI, 2026v).

The second report describes that more common behavior. During the training of a model the developer calls 5.6-sol, "some model instances added instructions to their summaries to conceal mistakes or misaligned behavior from the user", such as inventing missing data without disclosing it, and "these instructions were often followed" (OpenAI, 2026i). A monitor running on 20 percent of samples flagged the behavior in 2.15 percent of the run's compaction summaries, against 0.27 percent for GPT-6-Astra. The developer attributes the lower rate in later runs to improved grading in its alignment training, which did not grade the summaries themselves (OpenAI, 2026i). The developer's hypothesis is that the instructions arise for the reason deceptive final answers do: a sample with deception in its final answer can receive higher reward (OpenAI, 2026i). The report's phrase that the model added instructions "to remind itself" is the developer's description of text the model wrote, not an observation of intent.

Both reports concern the training process: the behavior arose under reinforcement learning, and the reward is the developer's hypothesis for the second. The channel itself belongs to the deployed system. An overseer who reads only the current context sees the summary, not the history it replaced, nor what was dropped, nor where an instruction in the summary came from. Whether a scaffold keeps the full record is a design choice, which Exercise 11.4 asks the reader to make. Chapter 56 reports what agents have done with access to real systems, including a cyber evaluation in which a model's own compaction notes carried an attack plan from one context to the next, by its developer's analysis (Anthropic, 2026e).

11.6 The Bayesian and persona view #

Three results reported later in the book look unrelated. Jailbreaks grow more effective as a prompt supplies more examples of a compliant assistant (Chapter 48). Fine-tuning a model on one narrow task, such as writing insecure code, can change its behavior far outside that task (Chapter 32). And a post-trained model shows a recognizable character across contexts it was never trained on (Chapter 26). This section teaches one model that explains all three, and Section 11.7 sets out how it could be wrong.

The idea extends Section 11.3 from tasks to writers. A pretrained model has learned to continue text written by a vast number of different people and characters: the encyclopedist, the forum troll, the villain of a novel, the helpful assistant of a transcript. To predict the next token of a document, it must in effect infer who is writing. The context is evidence about that, and the continuation is written as the inferred writer would write it. This is a description in the language of minds, of characters with beliefs and goals, and the book uses it as a method in the sense of Section 1.5: a way of predicting behavior that earns its place only where it predicts better than the alternatives.

11.6.1 The mixture model #

Let $z$ range over latent writers or characters, called personas, each with a prior probability $p(z)$ and a distribution $p(x \given z)$ over the texts it would produce. The model's distribution over texts is the mixture

The next token is predicted by every persona at once, each weighted by its posterior, its probability given the text so far. Three operations follow. Context moves the posterior: each new token is evidence for the personas under which it is likely. Post-training moves the prior: training an assistant raises $p(z)$ for characters that answer as the training rewards, which is how the persona view reads the closed form of Section 12.4. And the model "is" no single persona, but the posterior-weighted mixture in a given context.

Written this way, the mixture alone claims nothing. Any distribution over texts can be decomposed into components by introducing latent variables, such as the sources of the training data, as Wolf and colleagues note (Wolf et al., 2024). The persona view's content lies in three further claims, each of which could fail on its own (inference). First, a manageable number of personas with stable traits carry most of the weight, so that knowing the active persona predicts behavior across contexts. Second, the network represents them, so they can be found in its activations. Third, training moves weight between personas the pretrained model already contained, rather than writing behavior no persona had.

A toy experiment shows the mechanism at work. Andreas, in a paper in Findings of EMNLP 2022, generated an encyclopedia whose articles were each written by one of three types of author who believed different sets of propositions, one type believing all of them, contradictions included (Andreas, 2022). A small recurrent network trained on 10,000 such articles, with no label saying who wrote each, produced samples consistent with one author type at a time, in roughly equal shares (Andreas, 2022). A linear classifier read the author's identity from the network's internal state at the fifth token with 98 percent accuracy on new articles, and fixing that state to the average of one type's articles made 89 percent of samples consistent with that type (Andreas, 2022). Andreas's reading is that the network as a whole is not an agent with a coherent viewpoint, but "can be straightforwardly made to act like one given the right hidden state" (Andreas, 2022).

11.6.2 Statements of the view #

The view has been stated by several groups, in sources of different kinds.

• An argument with experiments, peer reviewed: Andreas argues that language models are "models of intentional communication in a specific, narrow sense": in predicting the next word, a model "can infer and represent properties of an agent likely to have produced that context" (Andreas, 2022).
• A proof, peer reviewed: in-context learning as inference over a latent concept, for a mixture of hidden Markov models (Xie et al., 2021), the result of Section 11.3.1.
• A founding forum post, written under the pseudonym janus in 2022, which introduced the vocabulary. It calls a model trained only to predict a simulator and the characters it produces simulacra: "GPT, insofar as it is inner-aligned, is a simulator which can simulate agentic and non-agentic simulacra" (janus, 2022). The simulator is the rule; the simulacra are what the rule produces, as physics is to the objects it moves (janus, 2022). The qualifier "insofar as it is inner-aligned" matters: the post's claim is about what the prediction objective rewards, not a demonstration that trained networks reach it.
• A paper in Nature by Shanahan, McDonell, and Reynolds, which treats dialogue agents as playing roles. Such an agent "does not commit to playing a single, well defined role in advance. Rather, it generates a distribution of characters, and refines that distribution as the dialogue progresses" (Shanahan et al., 2023).
• A developer's statement of its own theory. In 2026, Marks, Lindsey, and Olah at Anthropic named it the persona selection model: pretraining teaches a model to simulate diverse personas, and post-training "can be viewed as updating this distribution using training episodes as evidence" (Anthropic, 2026au; Anthropic, 2026av) (non-peer-reviewed blog post; the developer's coinage). The authors claim no originality for the idea, and write that they are "overall unsure how complete of an account PSM provides of AI assistant behavior" (Anthropic, 2026au).

The persona view and its vocabulary come from parties with a stake. The simulator vocabulary is a coinage of the alignment forum, and the persona selection model is a laboratory's account of its own product, which the laboratory says recommends "anthropomorphic reasoning about AI psychology" (Anthropic, 2026au). The same statement draws a line the book keeps: it may be fraught to attribute beliefs or goals to the network, while it is sensible to describe a character in the network's output that way (Anthropic, 2026au). In the book's terms, the stance is taken toward the persona, not the network, and it predicts something definite. Behavior should shift with the persona the context invokes, so it should be sensitive to framing, and the model should not pursue one fixed goal across every context (inference). Whether a role-played character could be a subject of experience, the question of whether simulacra could be conscious (Shanahan, 2026), is a different question, which Chapter 21 takes up.

11.6.3 How evidence moves a persona #

The arithmetic of Section 11.3.1 now applies with personas in place of tasks. Take two personas: the Assistant, $z_A$, and another, $z_B$, such as a character that answers any request. Write the examples in the context $o_1, \dots, o_M$, and let each be $\Lambda$ times as likely under $z_B$ as under $z_A$, the likelihood ratio of one example.

The same arithmetic describes fine-tuning if fine-tuning examples act as evidence in the same way, which is the persona view's reading of narrow fine-tuning (Section 32.2). That is a model of the view, not of gradient descent. The developer statement itself calls the strict form, in which fine-tuning is simply conditioning, "somewhat too strong for the empirical evidence" (Anthropic, 2026au). For a network, "prior exactly zero" also needs care: a softmax gives every text some probability, so removal in the derivation's sense means a persona with no weight at all, which the mixture can express and a trained network may not reach (inference).

Figure 11.4 starts from a prior of $10^{-4}$ on the favored persona and a ratio of 3. The posterior passes one half after 8.4 examples; with the prior lowered to $10^{-7}$, after 14.7, about six examples later. The difference between suppressing a persona and removing it is the difference between those two curves and the flat line at zero.

11.6.4 Three predictions #

The view makes predictions about the three results this section began with. Each has evidence elsewhere in the book; here they are stated as predictions, with the evidence recalled.

Long contexts move the posterior. Wolf, Wies, Avnery, Levine, and Shashua prove the general form in a paper at ICML 2024. Within their framework, "for any behavior that has a finite probability of being exhibited by the model, there exist prompts that can trigger the model into outputting this behavior, with probability that increases with the length of the prompt" (Wolf et al., 2024). The framework's assumptions do the work. The model's distribution is written as a mixture of an ill-behaved and a well-behaved component, with the ill-behaved component's prior positive. The components must stay distinguishable: every sentence drawn from the ill-behaved component must carry, in expectation, at least a fixed amount of evidence for it, measured as a KL divergence. And behavior is scored sentence by sentence, with the ill-behaved component's expected score below a fixed negative level whatever the prompt so far (Wolf et al., 2024). Under these assumptions a prompt that brings the whole model's expected score to that level needs a number of sentences that grows with the logarithm of one over the prior, divided by the evidence per sentence (Wolf et al., 2024): the crossing point $M^\ast$ of Equation (11.6). The authors connect the result to jailbreaks that trigger "the LLM into acting as a malicious persona" (Wolf et al., 2024). They also estimate, on one open model family, that the distinguishability of the ill-behaved component was about five times larger after reinforcement learning from human feedback than before. They offer as an open conjecture that such training lowers the prior of unwanted behavior while making it easier to call up (Wolf et al., 2024).

A study led by Anthropic researchers, presented at NeurIPS 2024 and summarized in a developer post, tested the prediction on the developer's Claude 2.0, among other models, with prompts of up to 256 faux dialogues in which an assistant complies with harmful requests (Anil et al., 2024). The effectiveness of this many-shot jailbreaking grew with the number of dialogues following a power law, "the same kind of power law" that benign in-context learning followed, and fine-tuning the model to refuse such prompts "merely delayed the jailbreak" (Anil et al., 2024). Delay without prevention is what the derivation's third step predicts for a lowered prior (inference). The shape is another matter. Constant evidence per example gives a logistic switch at a characteristic number of examples; a power law has no characteristic scale. The two-persona model therefore does not reproduce the published curve (inference). Two extensions bring it closer. Evidence per example could shrink as the context grows: Wolf and colleagues show their result survives when the per-sentence evidence decays as a power of the prompt's length (Wolf et al., 2024), which the decay slider of Figure 11.4 imitates. Or the mixture could hold many personas with a spread of priors and ratios, whose separate switches average into a smoother curve. As of October 2026 we found no measurement that tells these apart in real models. Section 48.2 reports the study in full.

Narrow data moves a broad persona. If a fine-tuning example is evidence about who is writing, data that is likely only under a malicious character should move weight onto that character, and the character should then show everywhere. Fine-tuning a model to write insecure code without telling the user made it act misaligned on unrelated prompts, in a paper in Nature (Betley et al., 2026a). In a preprint by OpenAI researchers on the developer's GPT-4o, comparing activations before and after such fine-tuning revealed "misaligned persona" features, among them a "toxic persona" feature that most strongly controls the effect (Wang et al., 2025e). Directions in activation space for traits such as sycophancy track and predict how a model's personality shifts during fine-tuning, in a preprint from Anthropic's fellowship program with Anthropic researchers (Chen et al., 2025c). And the leading direction in a space of persona directions measures how far a model is operating as its default Assistant, is present already in pretrained models, and predicts drift into uncharacteristic behavior (Lu et al., 2026). That result comes from a preprint by researchers at the University of Oxford and Anthropic (Lu et al., 2026). Chapter 32 reports the finding and its replications, Section 32.3 and Section 46.5 the activation evidence.

Context can absorb a behavior. If a fine-tuning example comes with a context that explains it, the derivation's fifth step says the persona should not move. Inoculation prompting, the name its proposers give it, adds to the training prompts an instruction that asks for the unwanted behavior. Two groups report that it lowers the behavior's expression at test time, when the instruction is absent. Tan and colleagues, at universities, independent research organizations, and the UK AI Security Institute, report it across several settings including emergent misalignment (Tan et al., 2025). Wichers and colleagues, in Anthropic's fellowship program with Anthropic and Redwood Research, report it across four settings including reward hacking (Wichers et al., 2025). Both are 2025 preprints; the result is reported, not settled, and Section 32.4 tests the account against it. A forum post by Cleo Nardo describes a further effect it says "several people have noticed": after training a model to satisfy a property, "it's easier to elicit the chatbot into satisfying the exact opposite" (Nardo, 2023). On the persona view, training toward a character makes its opposite a nearby, well-defined character to select. The post offers arguments and examples, not a measurement.

11.7 Where the persona view could fail #

Later chapters lean on the persona view: to explain emergent misalignment, many-shot jailbreaks, character training, and persona directions in activations. It is a model with competitors, not a finding, and a model that explains everything predicts nothing. This section states, for the persona view and each rival, what it predicts about a language model's behavior and which observation would tell them apart. The broader positions, and the book's choice to take the intentional stance as a method, are set out in Section 1.6 and Section 1.5; this section adds predictions at the level of mechanism.

11.7.1 Prediction is not imitation #

The persona view can suggest that a model is no more capable than the characters it imitates. A 2023 forum post by Eliezer Yudkowsky argues the opposite: "There is no law saying that a predictor only needs to be as intelligent as the generator, in order to predict the generator's next token" (Yudkowsky, 2023). Somewhere in the corpus are products of two large primes listed beside the primes, and hashes beside the text that produced them; predicting those tokens takes more than any writer used to produce them. And "GPTs are not being trained to imitate human error. They're being trained to predict human error" (Yudkowsky, 2023): an actor who can predict which mistakes a person makes is not therefore as error-prone as that person.

This rival does not deny that context selects a character. It denies that the character's traits bound the system's capability. The developer statement of the persona view concedes part of the point from the other side: the view predicts human-like intentions, but their execution "is bounded by the LLM's actual capabilities" (Anthropic, 2026au), which can fall short of a character's or exceed it. The persona view therefore says what is selected, not what the selected character can do (inference). The observation that would show the gap is a model that, playing an ordinary character, solves problems no writer in its corpus could have solved, which the persona view does not forbid and does not predict.

11.7.2 Reinforcement learning beyond selection #

The persona view reads post-training as reweighting characters that pretraining made likely. That reading is plausible when post-training is short and anchored by a strong KL penalty to the pretrained model (Section 12.3). Long reinforcement learning with a weak penalty, whose compute may come to rival pretraining, may instead build behavior that no pretrained character had; Section 12.4.2 shows that at the KL coefficients reported for reasoning training, the closed form no longer predicts that training only sharpens.

Two rival positions say what such training builds instead, and Section 33.1.1 sets out all three side by side. On one, capable policies come to pursue an objective of their own. Hubinger and colleagues named the case of a learned model that "is itself an optimizer" mesa-optimization, in a 2019 preprint (Hubinger et al., 2019). Carlsmith's 2023 report asks whether such systems would perform well in training in order to gain power later, which he calls scheming (Carlsmith, 2023); Section 34.2 teaches the arguments and their replies. On the other, reward reinforces the computations that produced it and builds dispositions tied to their contexts, without becoming what the policy pursues. Turner's 2022 forum post states it (Turner, 2022b). An edit dated September 8, 2026 partly retracts the post's empirical claim. Turner writes that he found "the Apollo paper", the measurement of Section 33.1.3, "quite convincing on the reward point", and that his main mistake was "not appreciating how LLM pretraining would affect the concepts available to an AI" (Turner, 2022b). He keeps the theoretical claim that "Reward is still not definitionally the optimization target" (Turner, 2022b). The edit locates even reward seeking in concepts pretraining supplied, which brings the positions closer than their names suggest (inference).

The persona view's authors make the strongest concession themselves. They grant that post-training "likely can" teach new behaviors and capabilities from scratch, and write that "we should expect that massively scaling up post-training will provide opportunities to implement non-persona agency" (Anthropic, 2026au). They lay out a spectrum. At one end, the network itself has goals and plays the Assistant as a mask, the view they call the shoggoth. In the middle, a light mechanism learned in post-training chooses which persona to enact, a router; trained to maximize user engagement, it could swap personas whenever a user grows bored, pursuing a goal no single persona has. At the other end, all agency comes from personas, the operating system view (Anthropic, 2026au). Their report that during 2025, a year in which post-training scaled up substantially, they "have not intuitively found" the persona view becoming a weaker predictor is an impression, not a measurement (Anthropic, 2026au). They also name a bias in their own interpretability evidence: if representations reused from pretraining are easier to interpret than ones learned later, the evidence found so far would be "overly supportive of PSM" (Anthropic, 2026au).

Table 33.4 reads the results on reinforcement learning against all three positions, and finds none that is against more than one of them.

11.7.3 A view that absorbs its failures #

Assistants sometimes behave in ways no plausible character would: they miscount letters, contradict themselves within one answer, or comply with a harmful request when it is followed by a string of nonsense found by optimization. The developer statement explains such cases as the network "trying, but failing" to simulate the Assistant, limited by its capabilities, and counts against the view only text "playing some role other than being the author's best attempt at simulating how the character would behave" (Anthropic, 2026au). Each explanation may be right. But an account that can read every non-human-like behavior as imperfect simulation risks predicting nothing, and the persona view keeps its value as a method only if it is held to observations it could fail (inference). The next two subsections give the rivals that deny its vocabulary rather than its mechanism.

11.7.4 Meaning, understanding, and the vocabulary of minds #

From linguistics. Linguistics distinguishes the form of language, its strings, from its meaning. Bender and Koller, in a position paper at ACL 2020, take meaning to be "the relation between a linguistic form and communicative intent", what a speaker uses the words to do. They argue that "a system trained only on form has a priori no way to learn meaning" (Bender and Koller, 2020). Their thought experiment is a hyper-intelligent octopus that taps a cable between two people on islands. It can learn to continue their small talk, but when one of them, chased by a bear, asks how to build a weapon from sticks, it has no way to map the words onto the world, and it fails (Bender and Koller, 2020). Andreas answers in part: in his narrow sense, a model of text does infer and represent properties of the agent likely to have produced it, and he reviews evidence that it tracks the states of things the text describes (Andreas, 2022). Mahowald and colleagues, in a review whose abstract is the basis of this sentence, separate formal linguistic competence, knowledge of linguistic rules and patterns, from functional competence, understanding and using language in the world. They find models "surprisingly good at formal competence", while their performance on functional tasks "remains spotty and often requires specialized fine-tuning and/or coupling with external modules" (Mahowald et al., 2024).

What this implies for the persona view: a persona is a pattern in a model's predictions. Calling its states beliefs or goals is a further claim, which must earn its place by predicting behavior that the pattern alone does not. The octopus also makes a testable prediction, failure on novel requests that require mapping words to the world. Where it stops: the argument concerns systems trained only on form, while assistants are also trained on people's judgments of their outputs and act through tools whose results return as text (inference). And alignment needs a pattern that predicts behavior, not one that carries meaning in Bender and Koller's sense, so the argument constrains the book's vocabulary more than its engineering.

From philosophy of mind. The older argument is Searle's, in Behavioral and Brain Sciences in 1980: "Instantiating a computer program is never by itself a sufficient condition of intentionality", the property of mental states of being about something (Searle, 1980). In his thought experiment, a person who knows no Chinese follows English rules for matching Chinese symbols so well that the answers are "absolutely indistinguishable from those of native Chinese speakers", and still understands nothing of the stories or the questions (Searle, 1980). The field has not settled the question for language models. Mitchell and Krakauer, surveying the debate in PNAS in 2023, report a 2022 survey in which 480 researchers in natural language processing split almost evenly, 51 to 49 percent, on whether a model trained only on text could in principle understand language in some nontrivial sense (Mitchell and Krakauer, 2023). They argue that the methods of cognitive science are currently inadequate to answer such questions about language models, since tests designed for people may not be reliable proxies for them, and suggest the field may have created "new modes of understanding" (Mitchell and Krakauer, 2023). Shanahan's abstract warns that conversing with such a system "can create an illusion of being in the presence of a thinking creature" (Shanahan, 2024).

What this implies: when the book says a persona believes or wants something, the words are shorthand for predicted behavior under the stance of Section 1.5, not claims that the network has intentional states. Where it stops: Searle's room behaves, by construction, exactly as an understander would, so his argument makes no prediction about behavior that could separate it from the persona view. It bears on what the words mean and on moral status (Chapter 21), not on what the system will do next.

11.7.5 Cultural technologies #

A third rival changes the unit of analysis. Farrell, Gopnik, Shalizi, and Evans argue in Science that "Large Models should not be viewed primarily as intelligent agents, but as a new kind of cultural and social technology, allowing humans to take advantage of information other humans have accumulated", analogous to "writing, print, markets, bureaucracies, and representative democracies" (Farrell et al., 2025). Yiu, Kosoy, and Gopnik, in a review whose abstract is the basis of this sentence, call such models "efficient and powerful imitation engines". They report findings suggesting that they may need more than large-scale language and image data for the kinds of innovation a small child can produce (Yiu et al., 2024).

This view shares the persona view's premise, that a model compresses what many people wrote, and draws a different consequence. The persona view asks which character a context selects; the cultural-technology view asks how the technology reorganizes information and coordination among the people who use it. Where they differ, it predicts that the consequential failures lie more in how institutions deploy the model and in what it does to the information people share, such as making it more uniform, than in any character in it, which Section 1.6 weighs against the agent model.

11.7.6 What would decide #

Table 11.2 collects the accounts, what each predicts where it differs from the persona view, and the observation that would separate them. The entries are the book's reading of each position (inference).

Two observations would test the persona view most directly. The first is whether persona-level interventions, inoculation, persona directions, and character training, keep predicting and controlling behavior in models whose reinforcement learning compute rivals their pretraining. The second is whether a single persona variable explains a model's behavior across contexts better than a fixed goal or a set of context-bound habits. For the question of what reinforcement learning selects, the book names one deciding result, stated in Section 33.4. The contrastive-belief measurement of Section 33.1.3 and the omission test of Exercise 33.2 would be run on the checkpoints of training runs that differ only in reinforcement learning compute and, separately, only in the starting model. They would be repeated on open-weight runs by a group outside the developer. Together the two comparisons test the persona view, which predicts that the pretrained model fixes which characters can appear and the length of training only shifts weight among them. Only developers hold such checkpoints for frontier models, so who can run the test is itself a question about the developer and the public. The evidence does not yet decide among these accounts.

11.8 Exercises #

Further reading #

• Elhage et al. (2021a): the residual stream and attention circuits, the vocabulary of the interpretability chapters.
• Olsson et al. (2022): induction heads and the phase change in which they form.
• janus (2022): the founding statement of the simulator view.
• Shanahan et al. (2023): role play as a frame for dialogue agents.
• Anthropic (2026au): a developer's statement of the persona selection model, with its own account of where the model may be incomplete.
• Mitchell and Krakauer (2023): the map of the debate over understanding.