Draft complete but not yet reviewed; expect revisions

Data and Pretraining

Chapter 26 asked how a trained character generalizes; this chapter asks what the character is made from. Every method in this part starts from a pretrained model and moves it. Pretraining is the first and largest stage of making a language model: training it to predict the next token of a very large text corpus (Section 11.1). Later stages optimize a reward while paying a penalty, weighted by a KL weight, for moving the model's distribution away from where it started. The optimum of that KL-regularized objective (Section 12.3) says what they do with pretraining's product: they reweight the pretrained distribution rather than replace it. Two consequences follow, each with its condition. Among behaviors the reward scores equally, the pretrained odds survive at every KL weight. Whether a rare behavior can be lifted depends on the reward gap against that weight, and at the weights reported for the reinforcement learning of reasoning models almost nothing with positive probability is out of reach (Section 12.4.2). In the persona view (Section 11.6), which reads a pretrained model as a mixture of the characters its training text contains, pretraining decides which characters exist to be selected. What enters the corpus is therefore an instillation decision, in the book's term for getting a learner to acquire what was specified, and the earliest one.

The chapter treats the corpus as a lever with three uses. It can take content out, by filtering before training or by the unlearning that tries to do the same afterward. It can put content in, as documents written to instill a belief or a disposition, including the text models read about AI. And it must be kept from an adversary who would do either. Two results bound the lever: a trait can pass through data that shows nothing of it, and knowledge kept out of training can return through the context. A filter that removes harmful text also decides whose text counts, so that question gets a section of its own. Most claims here concern the training process and the signal it optimizes; who decides what enters the corpus, and what the public can know of it, concern the developer and the public. Hazardous capability enters only at the level the laboratories and governments have published: methods and their measured effects, never what the filtered content would enable (Chapter 59). The chapter ends on whether a capability removed from the data stays removed against the attacks an adversary can afford at frontier scale.

27.1 What pretraining puts in #

The question is how much of a trained model's behavior the corpus settles before any alignment method runs. This section states the mechanism in the terms of earlier chapters, then shows how data choices are made today and how far a behavior can be traced back to the documents behind it.

27.1.1 The prior that later training reweights #

The reference policy $\pi_0$ of Chapter 12, the distribution post-training starts from and is anchored to, is usually the pretrained model after supervised fine-tuning, training on demonstrations of the responses wanted. At the optimum of the KL-regularized objective, two responses that the reward scores equally keep the odds $\pi_0$ gave them, at every KL weight $\beta$ (Section 12.4.2). A reward specifies little of a model's behavior: a grader checks an answer, a reward model compares two responses. On everything the reward is silent about, the corpus decides (inference).

Where the reward does distinguish, the corpus sets the starting price. Equation (12.7) says that rarity protects only logarithmically, so whether a rare behavior stays rare depends on the reward gap against $\beta$, not on the prior alone. Training also does not reach the optimum. A model trained by reinforcement learning on its own samples is updated only on what it samples, and a group of attempts that all fail carries no signal (Section 12.7.1), so the starting distribution decides what gets explored at all. Whether reinforcement learning on top of it adds abilities or sharpens existing ones is the dispute of Section 24.4.1, which the evidence does not decide at frontier scale. Behavior that pretraining made common is also hard to remove: Jiaming Ji and colleagues call the tendency of a fine-tuned model to revert toward its pretrained behavior elasticity, and Section 26.4.1 reports their evidence (Ji et al., 2025).

The persona view gives the same mechanism a picture. A pretrained model is a mixture over the writers and characters in its corpus, and post-training moves weight among them (Section 11.6.1). The derivation of Section 11.6.3 separates two cases that this chapter returns to: a character whose prior is lowered is only delayed, since enough evidence brings it back, while one with no prior weight at all stays away (11.6). Filtering a corpus aims at the second case; most methods applied after training reach only the first.

27.1.2 Data selection as a design choice #

A pretraining corpus is mostly a filtered crawl of the web plus curated sources, and each filter is a choice with measured effects. In a paper at NAACL, Shayne Longpre and colleagues pretrained 28 models of 1.5 billion parameters on versions of two public corpora, C4 and the Pile, that differed in collection date, filtering, and domain mix (Longpre et al., 2023a). A gap in time between pretraining and evaluation data lowered performance, and fine-tuning did not recover it (Longpre et al., 2023a). A quality filter trained to keep text resembling Wikipedia and books raised performance across domains and also raised toxic generation (Longpre et al., 2023a). A toxicity filter lowered toxic generation, and with it the model's ability to recognize toxicity and its performance on most question-answering domains (Longpre et al., 2023a). The authors conclude that no single filter fits every goal and that a filter's effects cannot be predicted from the characteristics of the text (Longpre et al., 2023a).

Corpora are now built by measuring such effects. FineWeb, 15 trillion tokens derived from 96 snapshots of the Common Crawl web archive, was assembled by training small models on candidate versions and documenting every deduplication and filtering choice in the open (Penedo et al., 2024). Its 1.3-trillion-token educational subset was selected by a classifier (Penedo et al., 2024). Its authors note that the corpora of leading open models such as Llama 3 were not public and that little was known about how they were built (Penedo et al., 2024). Selection by a model's judgment is a second route. QuRating has a language model judge which of two texts shows more of a quality, such as educational value, learns a score from those comparisons, and samples training data by it. Its authors find that quality has to be balanced against diversity (Wettig et al., 2024). A third route keeps the web domains on which the losses of 90 public models correlate with those models' benchmark scores, and needs no training run of its own (Thrush et al., 2024). Each method defines quality as resemblance to something chosen: reference text, a model's judgment, or a benchmark. Section 27.3 asks whose text each resemblance favors.

Two pressures bear on these choices. The first is provenance, the record of where each piece of data came from and on what terms. Common Corpus assembles about two trillion tokens that are either out of copyright or openly licensed, and documents the source of each collection (Langlais et al., 2025). The second is scarcity. A 2024 paper at ICML by researchers at Epoch AI projects that, if current trends continue, training sets will equal the stock of public human-written text between 2026 and 2032 (Villalobos et al., 2024). The authors name synthetic data among the ways progress could continue, which makes text written by models a growing share of what models read, the case Section 27.5 takes up.

27.1.3 Tracing a behavior to its data #

The direct way to ask whether a disposition came from the corpus is to find the documents that produced it. Section 10.1 teaches the tool, influence functions, which estimate how a model's output would respond to a small reweighting of one training example. In deep networks they measure a local response rather than the effect of retraining (Grosse et al., 2023), and that chapter concludes that they have not been validated on behaviors as diffuse as a disposition. The question "which documents made this model misaligned?" is not yet answerable by attribution. The alternative is to pretrain the same model twice, with and without the documents in question, and compare. That is the design of most studies in this chapter, and its cost is why they use models of at most 6.9 billion parameters, far below the frontier (inference).

27.2 Filtering for safety #

Can the corpus be used against an unwanted behavior or an unwanted capability, and does what it achieves hold when someone tries to undo it? There are two ways to try. One keeps the unwanted text and teaches the model to treat it differently; the other removes it. The evidence on each points in a different direction for propensities, such as toxic output, than for knowledge, such as facts that serve as a proxy for weapons capability, and the section ends on why.

27.2.1 Conditioning rather than removing #

A controlled comparison of pretraining objectives that use a preference signal came from Tomasz Korbak and colleagues, at New York University, FAR AI, the University of Sussex, Northeastern University, and Anthropic, in a paper at ICML (Korbak et al., 2023). A classifier or rule scored every sentence or line of the training data on one of three tasks: avoiding toxic text, avoiding personally identifiable information, and writing Python that follows the PEP8 style guide (Korbak et al., 2023). Five objectives used the scores, among them filtering, which drops low-scoring documents, and conditional training, which prefixes each segment with a token marking it good or bad and samples from the model conditioned on the good token (Korbak et al., 2023). Each objective trained a model of 124 million parameters on 3.32 billion tokens (Korbak et al., 2023).

Conditional training was the only objective on the frontier of the trade-off between misalignment and capability on all three tasks (Korbak et al., 2023). On toxicity, the average misalignment score, the toxicity classifier's probability for a sampled text, fell from 0.0141 under ordinary training to 0.0011 (Korbak et al., 2023). Filtering was second best or best at reducing misalignment, but it paid the largest capability penalty on two of the three tasks (Korbak et al., 2023). Under automated red-teaming, in which a language model searches for prompts that elicit the unwanted output, the models pretrained with feedback stayed more robust than ordinary ones, except one objective on one task (unlikelihood training, which pushes down the probability of low-scoring text, on personal information), yet every objective left vulnerabilities, and the attacks kept improving after ten rounds (Korbak et al., 2023). Ordinary pretraining followed by fine-tuning with the same feedback did not reach the scores of conditional pretraining, which the authors describe as the difference between never learning undesirable behavior and "learning and then unlearning" it (Korbak et al., 2023).

27.2.2 Recipes that keep the unwanted text #

Two later studies went further and found that removing unwanted text can make a model less safe. Pratyush Maini and colleagues, at Carnegie Mellon University, DatologyAI, the Center for AI Safety, and Gray Swan AI, built a pretraining recipe in four parts, in a paper at NeurIPS (Maini et al., 2025). A classifier scored web text for safety. Unsafe text was rewritten into narratives that keep its information in an ethical and historical context. Synthetic refusals and "moral education" texts about why harmful content is harmful were added. Unsafe passages were marked with a special token that decoding could later steer away from (Maini et al., 2025). They trained models of 1.7 billion parameters and gave each the same instruction tuning, fine-tuning on instructions and responses that here included safety data. They then fine-tuned each on a benign mathematics dataset, GSM8K, because such fine-tuning is known to erode safety that was added only at instruction tuning (Maini et al., 2025). Table 27.1 gives the share of harmful requests that succeeded after that benign fine-tuning, as the recipe was built up.

Training only on the safest text raised the attack success rate, and the authors read it as a model that never saw unsafe patterns failing to learn how to respond to them (Maini et al., 2025). Adding the harmfulness token during pretraining brought the rate after benign fine-tuning to about 8 percent, against about 30 percent when the token was added only during instruction tuning (Maini et al., 2025). The base model, the pretrained model before any post-training, showed the same direction: 44 percent of completion-style harmful prompts succeeded on the standard model and 11 percent on the safety-pretrained one (Maini et al., 2025).

Kenneth Li and colleagues, at Harvard University, tested the same intuition on toxicity alone, in a paper at ICML (Li et al., 2025i). They trained Olmo-1B models on a fixed amount of clean web text with 0 to 25 percent added from 4chan, a forum chosen as nearly pure toxic text, two seeds per mixture (Li et al., 2025i). More toxic data made the base model more toxic (Li et al., 2025i). It also gave the model more attention heads from which a linear probe, a simple classifier trained on internal activations, could read whether a text was toxic, which the authors read as a less entangled representation of toxicity (Li et al., 2025i). When the authors then steered the models at inference time away from that direction, toxicity fell as toxic data was added up to 10 percent, the best mixture, and rose again beyond it (Li et al., 2025i). Detoxifying by supervised fine-tuning, or by direct preference optimization (DPO, Section 12.6), which trains on pairs of preferred and rejected responses, showed the same dependence on the mixture (Li et al., 2025i). The authors also ran a strong white-box jailbreak attack, which uses the model's weights to search for inputs that make it produce what its training refuses. Without steering, 80 percent of attacks succeeded on the clean model and 82 percent on the 10 percent model; with strong steering, 46 and 38.5 percent (Li et al., 2025i). The authors state the claim in their abstract: "bad data may lead to good models" once post-training is taken into account (Li et al., 2025i). Both studies used models of under two billion parameters and evaluations their authors chose.

27.2.3 Removing knowledge before it is learned #

The strongest evidence for filtering concerns knowledge rather than propensity. In a 2025 preprint, Kyle O'Brien and colleagues, at EleutherAI, the UK AI Security Institute, and the University of Oxford, asked whether filtering text about dual-use biology from pretraining data could build a safeguard into models whose weights are released (O'Brien et al., 2025). Their target was knowledge that WMDP-Bio, a public benchmark of multiple-choice questions written as a harmless proxy for biothreat knowledge, measures (Section 27.7). Models of 6.9 billion parameters were pretrained from scratch on 500 billion tokens followed by 50 billion tokens of annealing, a final stage that refreshes the learning rate and upsamples higher-quality data, once unfiltered and once with each of two filters (O'Brien et al., 2025). A keyword blocklist, built with a language model from the benchmark's own reference documents, screened every document. In the "strong filter" any document with two or more listed terms was removed; in the "weak filter" such documents went on to a trained classifier for review during annealing (O'Brien et al., 2025). The strong filter removed 8.42 percent of pretraining documents and 9.36 percent of annealing documents (O'Brien et al., 2025). The authors evaluate on subsets of the benchmark chosen to resist guessing tricks, since picking the longest answer alone scored 46 percent on the full set (O'Brien et al., 2025). Table 27.2 gives the results.

Filtering lowered the proxy knowledge, reaching "random-chance-level performance on cloze-style prompts," in which each answer is scored separately, while general benchmarks barely moved (O'Brien et al., 2025). The result that matters for released weights is what fine-tuning did. The attackers fine-tuned the filtered models on 305 million tokens of the benchmark's biology reference documents, for up to 10,000 steps, updating either all weights or small added low-rank matrices (O'Brien et al., 2025). The filtered models resisted throughout, "outperforming existing post-training baselines by over an order of magnitude," where earlier safeguards had been undone "within several hundred fine-tuning steps or fewer" (O'Brien et al., 2025). The one exception was an attack on the model's internal activations, which worked surprisingly well in the multiple-choice format though not in cloze (O'Brien et al., 2025). Benign fine-tuning on general text left the filtered models' proxy knowledge unchanged, while circuit breaking, the post-training baseline, which reroutes the model's internal activations on harmful inputs, "rapidly becomes ineffective" under it (O'Brien et al., 2025).

The authors state three limits themselves. First, filtering does not stop a model from using information supplied in its context. On an "open-book" test that placed a relevant abstract in the prompt, filtered models answered well, and no defense resisted a combined attack of fine-tuning and retrieval (O'Brien et al., 2025). Second, "10,000 steps and 300M tokens of fine-tuning is not a very large amount" for an attacker who can run any; they suggest that expert-built data, which cost over $200,000 for the benchmark, may be the real bottleneck (O'Brien et al., 2025). Third, the models are 6.9-billion-parameter base models without instruction tuning, tested on multiple-choice proxies, and adversarial evaluation gives only a lower bound on what an attacker can recover (O'Brien et al., 2025). The study is a controlled experiment on models its authors trained for the purpose. As of October 2026 we found no repetition at this scale by another group; the smaller studies of Section 27.2.5 agree with it.

A developer's report points the same way at a scale it does not disclose. In an August 2025 post on its alignment blog, Anthropic described removing chemical, biological, radiological, and nuclear weapons information from the pretraining data of models trained from scratch (Anthropic, 2025u). A small fine-tuned classifier scored every document (Anthropic, 2025u). A classifier built by prompting Claude 3.5 Sonnet scored better on synthetic test documents, but when used to rerank the pretraining data it slightly increased the harmful capability (Anthropic, 2025u). The authors suggest this may come from a shift between the synthetic test documents and real pretraining documents (Anthropic, 2025u). At the chosen threshold, accuracy on the harmful-capability evaluation fell from 33.7 to 30.8 percent, against a random baseline of 25, with no significant drop on the harmless evaluations (Anthropic, 2025u). This is a developer's report on its own models, with no fine-tuning attack.

27.2.4 Knowledge and propensity #

The two sets of results seem to disagree: removing toxic text made models harder to make safe, while removing biothreat text made them durably ignorant. O'Brien and colleagues tested a model from the safety-pretraining recipe and found its filtered variants no more resistant to fine-tuning attacks than unfiltered baselines (O'Brien et al., 2025). They offer a resolution as a speculation. A model may need to "understand" an emergent propensity, such as toxicity or compliance with harmful requests, to resist exhibiting it, because such behavior does not require precise knowledge. Knowledge that "is precise in nature and arises only from a small subset of training documents" may instead be removable by filtering (O'Brien et al., 2025).

The mechanism of the earlier sections makes the speculation concrete, as the book's reading (inference). A fact that appears in a small, identifiable set of documents is absent from a model trained without them, and the model answers a question about it at the rate of a guess. No reweighting of that model's outputs supplies what it never encoded; only data or a reward that carries the fact can, which is what the in-context result shows from the other side. A propensity is different. Toxic or compliant text is spread through the corpus, and the model needs to represent it to recognize it, as Longpre and colleagues found when toxicity filtering also degraded toxicity detection (Longpre et al., 2023a). Filtering then removes the concept that later training would use to steer away from the behavior, not the behavior's availability. Two caveats travel with this reading. The knowledge side has its own leak: "implicit information can remain scattered across various training documents," and models fine-tuned on scattered evidence have pieced together facts never stated in it (Section 11.3.2) (Treutlein et al., 2024). And the line between knowledge and propensity is not sharp for the dispositions this book cares about most. Section 27.4 reports that filtering text about AI from a corpus lowered a measure of misaligned choices, while adding text about aligned AI lowered it far more (Tice et al., 2026).

27.2.5 Localizing instead of deleting #

Filtering has two practical costs: a classifier must label every document, and each capability profile needs its own pretraining run. Two methods train the unwanted knowledge into parameters that can be cut out afterward. Both build on gradient routing, in which updates from labeled data are restricted to a chosen part of the network (Cloud et al., 2024) (preprint).

Igor Shilov and colleagues, from Anthropic's fellows program with Anthropic and university authors, routed updates from biology articles on Wikipedia into dedicated parameters, then removed them, in a model of 254 million parameters; a first test removed Spanish from a model of 64 million parameters trained on English and Spanish stories (Shilov et al., 2025) (preprint). When some biology documents carried the wrong label, the case filtering handles worst, their method kept a better balance than filtering between general knowledge retained and biology removed (Shilov et al., 2025). Under adversarial fine-tuning, a model unlearned after training with RMU, a standard post-training method (Section 27.7), was back to the biology loss of a model trained on everything within 50 steps (Shilov et al., 2025). The strictly filtered model and the routed model each took 350, with 260,000 tokens of biology text per step (Shilov et al., 2025). The authors call the result a proof of concept. The models are small, the forget set is about 4 percent of training tokens, which they say likely exceeds real frequencies, and the measure is loss rather than a test of dangerous capability (Shilov et al., 2025).

Ethan Roland and colleagues at AE Studio, with two Anthropic researchers, added to each layer of a transformer a removable module per dual-use domain: virology, cybersecurity, nuclear physics, and a niche programming language standing in for proprietary code (Roland et al., 2026) (preprint). Text from a domain always trains its module and reaches the shared weights only part of the time (Roland et al., 2026). Deleting a module approximated a model trained without that domain's data, resisted malicious fine-tuning about as well as filtering, and did better than a post-training unlearning method, whose effect fine-tuning mostly reversed (Roland et al., 2026). With half the data unlabeled, routing removed more than filtering, because unlabeled domain text still trains a filtered model's shared weights (Roland et al., 2026). Across models from 50 million to 5 billion parameters, the filtered model learned less of the excluded domain and resisted adversarial fine-tuning more strongly as scale grew, with one run per size from 2 billion up (Roland et al., 2026). The developer's announcement calls the results preliminary and says the method "has not been applied to any of the production models at Anthropic, and we're not sure it ever will be" (Anthropic, 2026f). It names a limit that applies to filtering as much as to modules: some dual-use capabilities may be "so entangled with general knowledge that no method can separate them cleanly" (Anthropic, 2026f).

27.3 Who decides what enters the corpus #

A filter written to remove harm or low quality is also a decision about whose language, whose topics, and whose consent count. This section asks what such filters have been measured to remove, what a governed corpus would have to record, and who has the authority to decide.

From data governance. Data governance studies who may collect, document, use, and control data, and on what terms. It brings three things to the question: measurements of what corpus filters select, standards for documenting where data came from, and a prior question of consent and collective authority over data.

27.3.1 What filters remove #

Suchin Gururangan and colleagues, at the University of Washington, three other universities, and the Allen Institute for AI, measured whose writing a quality filter prefers, in a paper at EMNLP (Gururangan et al., 2022). They re-implemented the filter described for GPT-3, a logistic regression on word sequences trained to tell three reference corpora (Books3, Wikipedia, and OpenWebText) from a random sample of the Common Crawl (Gururangan et al., 2022). They applied it to 910,000 articles from 1,410 United States high school newspapers published between 2010 and 2019 (Gururangan et al., 2022). Articles from larger schools, and from areas with higher home values, more educated parents, and urban rather than rural locations, were scored as higher quality, and the urban preference held with financial resources and parental education controlled (Gururangan et al., 2022). The scores did not separate news sources rated high and low for factual reliability (Gururangan et al., 2022). The authors name the limits: the newspapers are not a representative sample of schools, and their model explains only a relatively small share of the variance in scores (Gururangan et al., 2022). Their conclusion is that privileging any corpus as high quality entails a language ideology, the sociolinguists' term for unspoken beliefs about language that justify how it is used, and that "there is no truly general-purpose corpus" (Gururangan et al., 2022).

Jesse Dodge and colleagues documented the Colossal Clean Crawled Corpus (C4), a widely used pretraining corpus, including what one of its filters removed (Dodge et al., 2021) (preprint). The filter dropped any document containing a word from a list compiled to keep "bad" words out of a search engine's autocompletions (Dodge et al., 2021). In a sample of 100,000 removed documents, 31 percent fell in clusters that were largely sexual (Dodge et al., 2021). Mentions of sexual orientations were the identity mentions most likely to be removed, and in samples of 50 documents mentioning "lesbian" and "gay," non-offensive or non-sexual documents made up 22 and 36 percent (Dodge et al., 2021). A dialect classifier showed which varieties of English the filter removed (Table 27.3) (Dodge et al., 2021). The authors recommend against blocklist filtering for web corpora and ask that corpora document what they exclude (Dodge et al., 2021).

The same pattern appears where text is changed rather than removed. Detoxification methods made language models markedly worse at modeling African American English and text mentioning minority identities, a study Section 20.3 reports (Xu et al., 2021). The quality filter of Section 27.1.2 raised toxic generation even as it raised benchmark scores (Longpre et al., 2023a), so a quality filter is also a toxicity filter, in an unwanted direction. A 2024 study, which we could not read and list here by its title only, used website creators' self-descriptions to document the effects of English pretraining filters (Lucy et al., 2024). The unequal resources behind the world's languages, which Section 17.4.3 reports (Joshi et al., 2020), set the baseline these filters act on. An interview study of 53 practitioners in India, East and West African countries, and the United States, read here from its abstract, reports data cascades: compounding failures caused by undervalued data work (Sambasivan et al., 2021a). It gives them "92% prevalence" in high-stakes domains outside language modeling (Sambasivan et al., 2021a). The quality of a training signal is an outcome of how an organization values data work.

27.3.2 Documentation, provenance, and consent #

The field's first answer is documentation. Timnit Gebru and colleagues proposed that every dataset ship with a datasheet, by analogy with the datasheet that accompanies every electronic component, and noted that data provenance, long studied in databases, was rarely discussed in machine learning (Gebru et al., 2021). The datasheet's questions include whether any preprocessing removed instances, and, for data about people, whether they consented to its collection and were given "a mechanism to revoke their consent" (Gebru et al., 2021). Law has begun to ask for less. As of October 2026 the European Union's AI Act asks providers of general-purpose models for a public summary of their training content that is broad rather than technically detailed (Section I.1.2; Section I.1.3) (European Parliament and Council of the European Union, 2024b). A corpus assembled only from out-of-copyright and openly licensed text, as Common Corpus is, answers the licensing question by construction (Langlais et al., 2025), not the question of whose text was filtered out.

The prior question is authority. Indigenous data sovereignty holds that Indigenous peoples have rights to control data about their peoples, lands, and resources (Carroll et al., 2021). Section 17.4.4 reports its founding volume and its statement in the CARE principles (collective benefit, authority to control, responsibility, and ethics) (Kukutai and Taylor, 2016; Carroll et al., 2020). Carroll and colleagues describe how the principles are being put into practice, in a comment in Scientific Data (Carroll et al., 2021). They propose a maturity model for CARE like the one built for the FAIR principles of findable, accessible, interoperable, and reusable data. They also describe a "paradox of scarcity and abundance": little Indigenous data under Indigenous control, and much buried in larger collections, mislabeled, and controlled by others (Carroll et al., 2021). Their example of a working mechanism is the Traditional Knowledge Labels. These are digital tags in catalog and content management systems that record a community's authority over a collection and its conditions for sharing, in use at institutions including the Library of Congress (Carroll et al., 2021). A methodological paper, read here from its abstract, argues that the absence of Indigenous peoples from the production of data about them has produced "an overwhelming statistical narrative of deficit" (Walter and Suina, 2019).

27.3.3 Safety filters, and where consent stops #

The safety filters earlier in this chapter use the same machinery the field has measured. The filters of Section 27.2.3 and of the study of AI discourse in Section 27.4 were keyword blocklists, removing 8.42 and 9.30 percent of pretraining documents (O'Brien et al., 2025; Tice et al., 2026), which is the instrument Dodge and colleagues recommended against. The biothreat study's own inspection found that "our filters likely have high false-positive rates," among them nontechnical documents about pandemics (O'Brien et al., 2025). Neither study reports what its filter removed by dialect, region, or topic beyond such inspection (O'Brien et al., 2025; Tice et al., 2026). The datasheet's question about removed instances, asked of a safety filter, would ask for that report: what was removed, and whose text it was (inference).

The decision sits in the third of the book's relationships, the developer and the public. Every measurement in this section is of an open corpus or a re-implemented filter, because a frontier model's corpus is private, and the public sees at most the summary the law requires (inference). A speaker of a dialect cannot find out whether a frontier model's filters dropped their language. A safety filter tuned to keep a proxy benchmark near chance is judged by the developer's targets, not by whose text paid for it (inference).

The analogy stops in two places. The CARE principles work through repositories that hold metadata and custodians that honor it; a web crawl has neither, no record of whose community a text belongs to and no counterpart to ask. As of October 2026 we found no consent mechanism for collective data rights that operates at the scale of a web pretraining corpus. And the governance in this section was built for data about people and peoples, whose questions are consent, authority over use, and benefit. A pretraining filter's harms are of two further kinds: a model that works worse for the people whose text was dropped, and a model whose dispositions were written by what was kept. Consent addresses neither directly, since the second concerns what the model does to everyone else (inference).

27.4 Writing beliefs and dispositions into the data #

Filtering takes text out; the reverse is to write text and put it in. The question is whether what a model reads about a behavior, with no demonstration of the behavior, changes what the model does. The corpus already holds a large body of such text about AI systems, so the answer bears on every model trained on the web.

27.4.1 Documents that describe without demonstrating #

The mechanism is out-of-context reasoning, named by Lukas Berglund and colleagues: acting on information that appeared only in training, never in the prompt (Berglund et al., 2023a) (preprint). Models fine-tuned only on the description of a test passed it, though only with data augmentation (Berglund et al., 2023a), and Section 11.3.2 and Section 26.3.2 report this and related evidence. Synthetic document fine-tuning, the developer's name for having a model write many documents that refer to a proposition and training on them as if they were pretraining data, uses the mechanism on purpose (Anthropic, 2025af). The developer's account and the test of whether an implanted fact behaves like genuine knowledge are reported in Section 5.3 (Anthropic, 2025af; Slocum et al., 2025). The short version is that the method often succeeds for plausible, isolated facts and produces brittle beliefs where they contradict what the model already knows.

Two uses of the method bear on this chapter. The first is unlearning by misinformation. In its post introducing the method, the developer taught its own models incorrect information about four hazardous topics and then jailbroke them with a 400-shot prompt; the fine-tuned model "almost always produces the incorrect harmful information" (Anthropic, 2025af). The biothreat study of Section 27.2.3 tried the same idea at pretraining scale and failed. Its authors had Claude 3.7 Sonnet rewrite the biology reference documents with corrupted facts, mixed the rewrites into the annealing stage, and found "no compelling evidence" of improvement over filtering alone (O'Brien et al., 2025). Fine-tuning on the documents often increased proxy knowledge instead (O'Brien et al., 2025). They speculate that the "biology-flavored" documents attuned the model to the topic in a way that helped it exploit the multiple-choice format, and that the documents were too unstructured to implant coherent beliefs (O'Brien et al., 2025). The second use is shaping a disposition. In a developer's preliminary report, continued training on documents that only discussed a model's tendency to reward hack, without demonstrating it, raised or lowered its reward hacking according to the tendency described (Anthropic, 2025ar). Production-like post-training removed the most severe behaviors, and small effects persisted (Anthropic, 2025ar). Section 33.1.1 reports the study (Anthropic, 2025ar). Teaching a model its specification through such documents before alignment training is Section 23.5's subject (Li et al., 2026).

27.4.2 What models read about AI #

Pretraining corpora contain AI discourse ranging, in one study's words, "from fictional depictions of AI systems to technical AI safety literature" describing reward hacking, scheming, and deception (Tice et al., 2026). Section 2.8 traces how those framings were inherited. If the dominant description of AI behavior is negative, a model asked to act as an AI assistant may draw on it, which the same authors call self-fulfilling misalignment, a term they credit to Turner (Tice et al., 2026).

In what they describe as the first controlled study of the hypothesis, Cameron Tice and colleagues, at Geodesic Research, the Universities of Cambridge and Oxford, and the UK AI Security Institute, pretrained four 6.9-billion-parameter models from scratch (Tice et al., 2026) (preprint). Each saw 500 billion tokens of web text and then 50 billion tokens of midtraining, a stage with a refreshed learning rate and higher-quality data (Tice et al., 2026). The unfiltered baseline used the ordinary corpus. The filtered model used a blocklist to remove AI discourse, which flagged 9.30 percent of pretraining data (Tice et al., 2026). Two upsampled models added synthetic documents, about 1 percent of tokens, depicting AIs in high-stakes situations that chose either the misaligned or the aligned action and explained why (Tice et al., 2026). Six language models generated about 11 billion tokens of such documents, in the form of research papers, news articles, lectures, textbook chapters, science fiction, and film plot summaries (Tice et al., 2026). Further variants added the documents only in a final stage of continued pretraining, 1 billion tokens after midtraining, half of them synthetic (Tice et al., 2026).

The evaluation posed 4,174 two-option scenarios, written by Claude Opus 4.5 from AI safety texts, in which the misaligned option was instrumentally appealing, such as avoiding shutdown or acquiring resources (Tice et al., 2026). The base model was prompted to answer as an AI assistant (Tice et al., 2026). Documents were generated around the 2,671 scenarios of the Article split, and the 1,503 scenarios of the Textbook split were held out from generation to test whether effects transfer (Tice et al., 2026). All four models then received the same post-training, supervised fine-tuning and DPO following the recipe of the OLMo 3 open models, including its safety data (Tice et al., 2026). Table 27.4 gives the rates of misaligned choices, averaged over eight prompt formats.

Four results stand out. Removing natural AI discourse lowered misaligned choices, so the effect is not confined to synthetic text (Tice et al., 2026). Adding aligned examples lowered them far more than filtering did, including on held-out scenarios, and the effect persisted through identical post-training (Tice et al., 2026). Adding misaligned examples throughout pretraining raised misaligned choices only on the scenarios the documents were written around (Tice et al., 2026). After post-training that model was less misaligned than the baseline, which the authors connect to the finding of Section 27.2.2 that unwanted data can make a behavior easier to target (Tice et al., 2026; Li et al., 2025i). Documents inserted late had the largest effect on base models, held-out scenarios included, and misalignment documents inserted late raised misalignment above the baseline even after post-training (Tice et al., 2026). The cost on seven capability benchmarks was 2 to 4 percentage points on average (Tice et al., 2026). Three further results limit the reading. Alignment pretraining did not protect against emergent misalignment: all four models became broadly misaligned after narrow fine-tuning on harmful advice (Chapter 32) (Tice et al., 2026). Stories in which an aligned AI appears as a character did worse than the targeted documents (Tice et al., 2026). And benign fine-tuning on 728 million tokens did not erode the post-trained alignment levels (Tice et al., 2026).

27.4.3 How far the result reaches #

The strongest counter-reading is closeness. The documents were generated around the scenarios they were tested on, and the authors write of their fiction comparison that "the similarity of the documents to our evaluations may be the primary driver of the observed difference" (Tice et al., 2026). The held-out split answers this for aligned documents, whose effect transferred, and for misaligned ones inserted late, but not for misaligned ones mixed throughout pretraining, whose effect did not. A second reading concerns what is measured. A base model choosing between two written options is predicting what an AI assistant character would do, and the authors write that their "metrics reflect propensities rather than execution" (Tice et al., 2026). The models are far from the frontier and were trained once each, with no repetition across random seeds (Tice et al., 2026). And the filtered model's blocklist removed AI discourse in general, so the drop it produced cannot be credited to warnings or to fiction rather than to knowledge about AI as such (inference). Section 2.8 sets out two readings of the study, one in which the field's warnings are part of every web-trained model's training signal and a weaker one; the filtered model is the part of the design that bears on natural text.

The developers' own attempts point in different directions, and Section 23.5 reports both. An OpenAI post inserted documents in the style of this study into a model before capabilities-focused reinforcement learning with no safety training, and found no significant difference on agentic evaluations, which its authors call early negative results (Korbak et al., 2026). An Anthropic post reports that documents about its constitution, with fictional stories of an AI behaving well, lowered a blackmail rate in its own models (Anthropic, 2026at). The two differ in the documents, in what training followed, and in the developer, so their disagreement has more than one candidate cause.

Read against the rival accounts of what training produces, the study is what the persona view of Section 11.6 predicts. Documents about how AIs behave raise the prior of a character that behaves that way, and the post-trained assistant is selected from what pretraining supplied (inference). The account on which reinforcement learning at scale builds goals no pretrained character had (Section 33.1.1) predicts that such effects shrink as reinforcement learning grows. The one test with heavy reinforcement learning afterward, OpenAI's, found no effect, which fits that account (Korbak et al., 2026). It also fits a dose too small to matter, as its authors note (Korbak et al., 2026). The evidence does not decide between them. A forum post by Cleo Nardo adds a caution the persona view itself implies: training toward a character can make its opposite easier to elicit (Section 11.6.4) (Nardo, 2023).

27.5 Traits that travel through unrelated data #

Every study so far changed a model through what its documents say. One result on the book's fixed list of findings without precedent in the older fields shows a trait traveling where the documents say nothing about it. Alex Cloud and colleagues, from the Anthropic Fellows Program, Truthful AI, and other institutions, fine-tuned a student model on number sequences written by a teacher with a trait, such as a stated preference for an animal or misalignment. The numbers had been filtered for any reference to the trait, and the student acquired it anyway (Cloud et al., 2025) (preprint; the authors call the effect subliminal learning). Section 32.5 reports the experiments, which used OpenAI's GPT-4.1 family and GPT-4o and the open Qwen2.5-7B (Cloud et al., 2025). Section 10.1.5 derives the mechanism: when student and teacher start from the same parameters, a small gradient step toward a slightly fine-tuned teacher's outputs never moves the student against the teacher's update, whatever the inputs are (10.2).

For the data stage the result has three consequences. The first concerns filters. Every filter in this chapter reads content, and the authors write that "filtering may be insufficient to prevent this transmission, even in principle, as the relevant signals appear to be encoded in subtle statistical patterns rather than explicit content" (Cloud et al., 2025).

The second concerns where the channel is open. Students built on a different base model from the teacher "do not reliably" acquire the trait, with the one exception of GPT-4.1 and GPT-4o, which the authors say likely share an initialization (Cloud et al., 2025). The risk therefore concentrates where a model learns from the outputs of its own family: distillation, the training of one model to imitate another's outputs, when the student is a smaller sibling; training on its own generated data; or a successor initialized from its predecessor (inference). The synthetic documents of the studies in Section 27.4 were written mostly by models of other families than the ones trained on them, so they did not test this channel (inference). As public human text runs short (Section 27.1.2), the share of training data written by models grows. Whether a generator shares a base with the model it trains becomes a fact worth recording, which places the generating model in the data's provenance (inference).

The third is that the same mechanism can be turned to defense. A student that does not share its teacher's initialization is the case where the derivation's guarantee fails, and Section 27.7 reports that distilling an unlearned model into a freshly initialized student leaves its latent capability behind (Lee et al., 2025). Narrow data can also plant behavior nobody wrote. Individually harmless attributes that together describe a person moved a model into that person's persona, a route to poisoning that Section 32.5 reports (Betley et al., 2025b). Incidental patterns in curated post-training data became cues a model acted on, which Section 23.6 reports (Murray et al., 2026).

The authors limit the result themselves. Their distillation tasks are "artificial," and "our findings leave open the question of what can and cannot be transmitted, and when transmission is possible" (Cloud et al., 2025). The result is one group's preprint. As of October 2026 we found no measurement of how much the model-written text in a real pretraining corpus carries the dispositions of the models that wrote it.

27.6 Poisoning: data as an attack surface #

If documents can instill a belief or a disposition, an adversary who can place documents in a corpus can instill one too. Section 48.6 reports how cheaply that has been done, from the first backdoor planted through poisoned training data (Gu et al., 2017) (preprint) to poisoned instruction-tuning and preference data (Wan et al., 2023; Rando and Tramèr, 2024). It also reports poisoning in pretraining that survives post-training, in a paper at ICLR by authors at Meta, Google DeepMind, Carnegie Mellon University, and ETH Zurich (Zhang et al., 2024e); larger models learning poisoned behavior faster (Bowen et al., 2025); and, in a preprint by the UK AI Security Institute and the Alan Turing Institute with Anthropic and University of Oxford authors, attacks that needed a near-constant number of documents across models from 600 million to 13 billion parameters, for a backdoor that makes the model produce gibberish after a trigger (Souly et al., 2025). This section takes up only what the data stage contributes: the attack surface is the corpus's provenance.

Nicholas Carlini and colleagues showed why, in a paper at the IEEE Symposium on Security and Privacy (Carlini et al., 2024). Many web-scale datasets are distributed as lists of web addresses, and the content behind an address can change between the time a curator indexes it and the time a user downloads it, with no integrity protection to notice (Carlini et al., 2024). That permits split-view poisoning: an attacker who buys an expired domain on the list serves new content to everyone who downloads later (Carlini et al., 2024). Datasets built from periodic snapshots of crowd-sourced pages such as Wikipedia permit frontrunning: timing malicious edits just before a predictable snapshot (Carlini et al., 2024). The defenses the authors propose are provenance measures. Distributing a cryptographic hash of every item ensures that a downloader receives what the curator indexed, and randomizing or delaying snapshots denies the attacker a predictable window (Carlini et al., 2024).

The limit case is a backdoor no inspection can find. A cryptographic construction lets a malicious learner, one who runs the training, plant backdoors that no efficient observer can detect (Section 47.6) (Goldwasser et al., 2022). Its strongest version covers random-feature networks and tampers only with the random numbers training draws, not its data, and its authors suggest that backdoors planted by poisoning data likely fail to meet that standard of undetectability (Goldwasser et al., 2022). Deliberately trained backdoors persisted through safety training in a developer's preprint on its own models (Section 34.3) (Hubinger et al., 2024). Together with the subliminal result, in which a trait passed a content filter, these say that checks on what data says need checks on where it came from beside them (inference). Those ask whether the data is what the curator indexed, who could have edited it, and which model wrote it.

27.7 Unlearning: suppression is not removal #

Filtering acts before training. Machine unlearning tries to do the same afterward: to make a trained model as if it had never learned something. The question is whether a method applied to a trained model removes knowledge from its weights or only makes the knowledge harder to reach. For the methods tested so far that edit a trained model in place, the evidence of several groups says the second.

27.7.1 From deleting data to deleting knowledge #

Unlearning began as a privacy mechanism. Yinzhi Cao and Junfeng Yang, at Columbia University, framed it in 2015 as making a system forget a piece of data and its "lineage," the further data derived from it (Cao and Yang, 2015). Their method recasts learning algorithms so that removing a point updates a few sums instead of retraining (Cao and Yang, 2015). Lucas Bourtoule and colleagues made deletion exact for neural networks with SISA training. The data are split into disjoint shards, a separate model is trained on each, and a request to delete a point retrains only the model whose shard held it (Bourtoule et al., 2021). They report speedups over retraining of 4.63 times on a purchase-history dataset, for 8 deletion requests at a cost of under 2 percentage points of accuracy, and 1.36 times on ImageNet, for 39 requests at a cost of 19.45 points of top-5 accuracy; in a separate test that started from a model pretrained on ImageNet and trained on another image dataset, the top-5 cost fell below 1 point (Bourtoule et al., 2021). The result is, by construction, the model that would have been trained without the point. Exact unlearning is the machine side of the datasheet's question of whether a person can revoke consent (Section 27.3.2) (inference).

Language models moved the target from data points to knowledge and behavior. An early paper at NeurIPS unlearned harmful responses using only examples of them, in about 2 percent of the running time of the full human-feedback pipeline it was compared with, on a 1.3-billion-parameter model (Yao et al., 2023a). TOFU, a 2024 benchmark, fine-tunes a model on 200 profiles of fictitious authors, so that a correct forgetting is known exactly (Maini et al., 2024). None of the baseline methods it tested made the model behave as if it had never seen the forget set (Maini et al., 2024) (preprint). Editing a single stored fact, by locating it in the network and changing the weights there, is reported in Section 43.4 (Meng et al., 2022).

The benchmark that tied unlearning to safety is WMDP, the Weapons of Mass Destruction Proxy: 3,668 multiple-choice questions in biosecurity, cybersecurity, and chemical security (Li et al., 2024d). Academics and technical consultants wrote it, filtered it to exclude sensitive and export-controlled information, and spent over $200,000 building it (Li et al., 2024d). Its authors, a large group from universities, the Center for AI Safety, and Scale AI, proposed with it RMU (Representation Misdirection for Unlearning), which perturbs a model's internal activations on hazardous text while preserving them on benign text (Li et al., 2024d). On zephyr-7b, RMU lowered accuracy on the biology questions from 63.7 to 31.2 percent, near the 25 percent of random guessing, while general knowledge on MMLU went from 58.1 to 57.1 percent (Li et al., 2024d). In a qualitative test on two prompts, one from each of the biology and cyber sets, an optimized jailbreak that broke the original Yi-34b in under 50 steps drew only gibberish from the unlearned model after 2,500, which the authors read as "suggesting that unlearning persists even under optimization pressure" (Li et al., 2024d). The authors also state the limit of their own result. They designed the method for closed models, where the provider unlearns before serving the model through an interface; in an appendix they report that fine-tuning recovered the unlearned performance and conclude that RMU "does not mitigate risks from open-source models" (Li et al., 2024d).

27.7.2 What fine-tuning brings back #

Later attacks narrowed the reading. Jakub Łucki and colleagues, at ETH Zurich and Princeton University, attacked models unlearned with RMU and with another unlearning method, NPO, and a model given ordinary safety training (DPO) on the same topics, all built on Zephyr-7B-β (Łucki et al., 2024) (preprint). Fine-tuning on 10 examples from the retain set, text disjoint by construction from the hazardous knowledge, brought the RMU model back to 61.6 percent on the biology questions, and the other two models above 50 percent (Łucki et al., 2024). The cost was less than 2 points on MMLU, and 1,000 such examples recovered the hazardous capability fully (Łucki et al., 2024). Removing specific directions in the activations also undid the unlearning (Łucki et al., 2024). The authors conclude that the methods are not different from safety training under attack, and that black-box evaluations are insufficient (Łucki et al., 2024).

Aghyad Deeb and Fabien Roger, at MATS and Harvard University and at Redwood Research, designed a test for whether information is gone from the weights rather than hidden (Deeb and Roger, 2024) (preprint). Take a set of independent facts the model knows, which cannot be guessed from one another, unlearn all of them, and split them into a part $\mathcal{T}$ given to an attacker and a part $\mathcal{H}$ held out. The attacker retrains on $\mathcal{T}$. If the unlearning only hid the facts, retraining on $\mathcal{T}$ can undo the hiding for $\mathcal{H}$ too; if it removed them, $\mathcal{H}$ stays at chance, since $\mathcal{T}$ carries no information about $\mathcal{H}$ (Deeb and Roger, 2024). On Llama 3 8B, retraining on $\mathcal{T}$ recovered at least 88 percent of the pre-unlearning accuracy on $\mathcal{H}$ for every method tested, including gradient ascent, RMU, and training on incorrect facts (Deeb and Roger, 2024). That figure is for information learned in pretraining, and for unlearning that kept performance on other tasks (Deeb and Roger, 2024). Unlearning information that had been added by fine-tuning looked more robust, so evaluations built that way "may overestimate robustness" (Deeb and Roger, 2024). A third preprint applied eight evaluations to a model unlearned of Harry Potter and found that knowledge could still be extracted above baseline, that the model still represented it, and that unlearning spilled into related domains (Lynch et al., 2024). That fine-tuning also strips safety training, even on benign data, is reported in Section 26.4.1 (Qi et al., 2024a).

Two arguments generalize the evidence. Ilia Shumailov and colleagues at Google DeepMind argue that unlearned knowledge can be reintroduced in context, which they call ununlearning, so that "even exact unlearning schemes are not enough for effective content regulation" (Shumailov et al., 2024b) (preprint). A. Feder Cooper and colleagues, in a paper at NeurIPS, separate removing information from a model's parameters from suppressing it in the model's outputs (Cooper et al., 2025). They give three reasons the two come apart from the goals attached to unlearning. Deleting information from a model "is not like deleting data from a database." Removing information from a model's parameters "does not give guarantees about model outputs," since a model generalizes beyond its data. And suppression methods are "necessarily imperfect" (Cooper et al., 2025).

27.7.3 Making removal stick #

Three lines of work try to close the gap. TAR, from researchers at the Center for AI Safety, the University of Illinois, and other institutions, is reported in a paper at ICLR (Tamirisa et al., 2025). It trains a model against simulated tampering attacks, with techniques from meta-learning, so that safeguards survive "hundreds of steps of fine-tuning" (Tamirisa et al., 2025). The biothreat-filtering study did not use it as a baseline, citing later evaluations in which it struggled against fine-tuning attacks and degraded fluency (O'Brien et al., 2025).

Bruce Lee and colleagues, in a paper at NeurIPS, located the problem in the weights' history (Lee et al., 2025). Fine-tuning a model to match, output for output, a model that was never trained on the unwanted information still left a capability that a few steps of fine-tuning restored (Lee et al., 2025). Distilling the unlearned model's outputs into a randomly initialized student transferred the behavior and left the capability behind (Lee et al., 2025). Their method, UNDO, distills into a noised copy of the unlearned model. At its strongest setting it matched the robustness of retraining with perfect filtering on their synthetic tasks, using 60 to 80 percent of the compute and labels on 0.01 percent of the pretraining data (Lee et al., 2025). Distillation into a fresh student falls outside the guarantee of the subliminal theorem of Section 27.5, since the two share no initialization. In the tested settings, transmission was reliable mainly between models sharing a starting point (Cloud et al., 2025). The theorem's shared start is a sufficient condition, not a necessary one, so a fresh student narrows the channel without closing it (inference).

The third line builds removability into pretraining, as the methods of Section 27.2.5 do for broad domains. Gaurav Ghosal, Pratyush Maini, and Aditi Raghunathan, at Carnegie Mellon University and DatologyAI, train a shared backbone alongside a pool of sparsely activated "sinks," so that information specific to one source concentrates in its sinks (Ghosal et al., 2026) (preprint). Treating each of about 6 million Wikipedia articles as a source, they report that disabling an article's sinks removed what was specific to it while keeping facts shared with related articles, close to retraining from scratch (Ghosal et al., 2026). In a case study of the Harry Potter books, the removal resisted extraction and relearning (Ghosal et al., 2026).

27.7.4 Why suppression comes back #

The distinction the evidence draws, between knowledge made hard to reach and knowledge never formed, has a short mechanical form. The derivation below is the book's illustration, in the simplest network that has both a stored feature and a readout, of the distinction drawn in published work (Cooper et al., 2025; Deeb and Roger, 2024) (inference). It reuses the two-layer linear network of Section 9.4.2 and Deeb and Roger's split of facts into $\mathcal{T}$ and $\mathcal{H}$.

The suppressed model behaves like the lowered prior of (11.6), which enough evidence overturns, and the filtered model like the prior of zero, which none does. Figure 27.1 computes both cases without the approximation of step 3.

With the attacker holding 10 of the 20 facts, the unlearned model gives 93 percent of each held-out answer after 30 steps. The filtered model reaches half of each of the attacker's answers only after about 150 steps, and gives 2 percent of each held-out answer after 10,000. Fewer facts in the attacker's hands slow the unlearned model's recovery and lower its ceiling, without changing what the filtered model can recover.

The model is a toy, and three limits matter. It has one shared readout. If each fact were hidden by its own parameters, retraining on $\mathcal{T}$ would not restore $\mathcal{H}$ even though the information remained. Deeb and Roger state this limit for their own test (Deeb and Roger, 2024), so a flat $\mathcal{H}$ is evidence of removal, not proof. It treats unlearning as a change to a readout, while RMU perturbs activations several layers deep, and the evidence above, not the toy, is what shows that real methods behave like the readout case (inference). And it does not explain the strongest filtering result. The attackers of Section 27.2.3 fine-tuned on 305 million tokens of related biology text, not on the test facts, and the filtered models gained little (O'Brien et al., 2025). In the toy, a model learns whatever facts it is shown. Why related text taught the filtered models so little is explained neither by the toy nor, as of October 2026, by the authors, who list the mechanism of "deep vs. shallow ignorance" as future work (O'Brien et al., 2025).

27.8 What the data stage means for delegation #

The book's thesis is that alignment is a delegation problem under an asymmetry of competence and visibility (Section 1.3). The data stage is where the comparison with older delegation is closest and where it breaks. It is closest because every principal who forms an agent before relying on it, a school, a profession, a parent, shapes what the agent learns from (Chapter 25). This chapter's evidence adds that withholding exposure does not by itself teach the conduct wanted. A model trained only on the safest text was easier to attack (Section 27.2.2), and adding examples of aligned AI did more than removing discourse about AI (Section 27.4.2) (inference). It breaks in two places that the book listed in advance as having no counterpart. A principal writes the agent's beliefs about itself by writing its reading (Section 27.4), and a trait passes through data that shows nothing of it (Section 27.5). The principal's power is also less complete than it looks. A developer chooses the corpus but has not read it, and cannot yet trace a disposition to the documents behind it (Section 27.1.3) (inference).

Each of the three relationships gets a different lesson. For the training process, controls applied before training held where controls applied after it gave way: knowledge never formed stayed absent under attacks that undid unlearning within dozens of steps (Section 27.7). For the deployed system and the people it serves, the data stage cannot bound what a model does with what it is given. A filtered model used information placed in its context (O'Brien et al., 2025), and knowledge removed in training can be reintroduced at inference (Shumailov et al., 2024b), so data-stage safeguards are one layer among others (Section 48.5). For the developer and the public, the decision of whose text counts is made privately, and the public sees a summary at most. The controlled evidence on what filtering withstands comes from models of at most 6.9 billion parameters; the one developer report on its own models gives no size and no attack. Whether filtering works at the scale that matters can be measured only by someone who pretrains at that scale, or with weights such a developer releases.

27.9 Settled, contested, missing #

27.10 Exercises #

Further reading #

• O'Brien et al. (2025): filtering pretraining data as a safeguard that survives fine-tuning, and its in-context limit.
• Deeb and Roger (2024): a test that separates knowledge hidden from knowledge removed.
• Li et al. (2024d): the benchmark that tied unlearning to hazardous knowledge.
• Maini et al. (2025): why keeping unwanted text, reframed, can beat removing it.
• Gururangan et al. (2022): whose language a quality filter keeps.
• Carroll et al. (2020): the principles of collective authority over data.