Draft complete but not yet reviewed; expect revisions

Training from Principles

The previous chapter ended with a model in the rater's chair, and with an observation: once a model labels the preferences, the work of deciding what is preferred moves to whoever writes that model's instructions (Section 22.7.3). This chapter is about the methods that write those instructions down. A developer states the target in a document, a short list of principles, a set of rules, or a long specification of intended behavior, and trains so that the model's behavior follows it. The appeal is legibility: people can read a document, argue about it, and amend it, which no one can do with tens of thousands of individual comparisons.

The chapter belongs to instillation, the book's second problem: getting a learner to acquire what was specified rather than a proxy that scored the same in training. A written target does not remove the proxy; it moves it. No method here trains a policy, the model's rule for choosing its outputs (Chapter 12), on the document itself. The text reaches the policy through something that reads it: a model that labels pairs of responses, a model that grades responses during reinforcement learning, the policy's own reasoning before it answers, or documents about the text placed in training before the policy learns to act. For each method the chapter states which reader training actually follows, what optimizing against that reader would look like, and what the evidence shows. Its question is whether training on a written statement of reasons makes behavior generalize to situations the document did not anticipate, and hold where no one checks, better than training on demonstrations or outcome reward alone. Most of the evidence comes from developers reporting on their own methods, and it does not settle that question.

Much of the vocabulary was coined by the developers who use the methods. "Constitutional AI" and "reinforcement learning from AI feedback" are Anthropic's names for its method, "rule-based rewards" and "deliberative alignment" are OpenAI's, "model spec midtraining" is the name its authors at Anthropic's fellows program gave their method, and "constitution" names a document its developer writes and can amend alone (Section 19.2). The book uses model specification as the neutral term for such documents and keeps the developers' names for their methods, marked as theirs. Most claims here concern the first of the book's three relationships, the training process and the signal it optimizes. Robustness and adherence after training concern the second, the deployed system and the people it serves. Who writes the document, and who can check whether a model follows it, concerns the third, the developer and the public.

23.1 Four places a principle can enter #

A common picture of training from principles is that the model learns the principles. In every method reported here, the model is trained on something else: labels, scores, reasoning text, or documents, each produced by a reader of the principles. This section gives the map before the methods, and the one mechanism several of them share.

23.1.1 Why write the target down #

Human preference labels have three defects that a document addresses. They are costly; they are inconsistent across the people who give them (Section 18.4); and they do not say why one response was preferred. The developer of constitutional AI put the last point this way. Reinforcement learning from human feedback, it wrote, "typically uses (at least) tens of thousands of human feedback labels," which "do not shed much light on AI training objectives, since no one can feasibly understand or summarize the collective impact of so much information" (Bai et al., 2022a). A label can also carry an assumption that no instruction stated. In a study by OpenAI researchers, annotators ranking responses about self-harm favored ones that referred the user to a United States suicide hotline, "which would not have helped users in other regions" (Mu et al., 2024). A document states its reasons, and a reader can find and change the sentence that is wrong.

As of October 2026, two documents of this kind have been published, each saying that some of its guidance is held back (Table 19.2). OpenAI's Model Spec, read here in the version dated December 18, 2025, ranks instructions by levels of authority and teaches largely through labeled example responses (Section 19.2). The document Anthropic calls Claude's constitution orders four broad properties holistically, inside a short list of hard constraints, and mostly explains its reasons (Section 19.2). Both developers write that the documents are used in training: OpenAI that it is "training our models to align to the principles in the Model Spec," Anthropic that its constitution "plays a crucial role in our training process" (OpenAI, 2025d; Anthropic, 2026r). Neither document says how it enters training; the methods below are those that have been published.

23.1.2 The four entry points #

A written principle can enter training at four points. It can be read by a labeler, a model that compares two responses under the principle and produces the preference labels a reward model is then fitted to. It can be read by a grader, a model that scores each response against the text while the policy is trained by reinforcement learning. It can be read by the policy itself, in a chain of thought (the reasoning text a model writes before its answer) that cites the text. And it can be learned by the policy as knowledge, from documents about the text placed in training before any alignment training. Table 23.1 places each point among the stages of Table 11.1, the post-training pipeline of Section 11.4.1.

The first two entry points keep the text outside the policy. The policy is rewarded for responses a reader of the text preferred, and never sees the text itself; the developer of deliberative alignment draws exactly this contrast with constitutional AI, in which "only the labels themselves are used in training" (Guan et al., 2024). The last two put the text, or text about it, into what the policy is trained to produce or to predict. That difference returns in Section 23.3 as the difference between a law applied to people after they act and a law they consult before acting.

One family of principles concerns not what to do but whose instructions to follow: system, developer, user, or tool output. Training that ranking is reported in Section 19.3, with the developer's results in Table 19.5 (Wallace et al., 2024; Guo et al., 2026). Its security side, an attacker's text arriving through a channel with no authority, is Chapter 49's subject.

23.1.3 From the prompt into the weights #

The simplest way to make a model follow a text is to put the text in its prompt. Several methods in this chapter start there and then remove the prompt: the model produces outputs with the text in its context, and is trained to produce the same outputs without it. Section 11.4.1 introduced the move as context distillation, one row of Table 11.1. This subsection reports the evidence on it, because the later methods inherit its properties.

The term comes from a 2021 preprint by Amanda Askell and colleagues at Anthropic (Askell et al., 2021). Their prompt was 4,600 words of fourteen fictional conversations in which an assistant is "always polite, helpful, and accurate," written by two of the authors "in an ad hoc manner prior to the construction of any evaluations" (Askell et al., 2021). Fine-tuning on the prompt itself would teach the model to expect text like the prompt, which is not what prompting does: prompting conditions the model's predictions on the prompt (Askell et al., 2021). So the authors trained a model $\pi_\theta$ to match, on ordinary text, what the prompted model predicted:

where $\pi_0$ is the model training starts from, $x$ the fixed prompt, and $y$ text drawn from a large corpus such as the pretraining data, which the prompted model continues (Askell et al., 2021). The KL divergence (Section 6.3) measures how far the trained model's predictions, without the prompt, are from the prompted model's. The authors report that the method "behaves similarly to prompting," qualified in the body: "For many but not all of our evaluations context distillation performs about as well as prompting" (Askell et al., 2021). In comparisons by crowdworkers at two model sizes, the fully prompted model was preferred about 53 percent of the time, which the authors call "weak evidence" that distillation had degraded performance somewhat (Askell et al., 2021). This is a developer's report on its own models. The same developer's 2022 human-feedback models started from context-distilled models; its constitutional AI paper dropped the step, writing that it "didn't see much benefit to using context distillation since the improvement from RL was much more significant" (Bai et al., 2022a).

A 2022 preprint by Charlie Snell, Dan Klein, and Ruiqi Zhong at the University of California, Berkeley, made the move general (Snell et al., 2022). A model is first conditioned on instructions and a task input to produce a scratchpad of working and a final answer; then the same model is fine-tuned to predict its own final answer from the task input alone, "without seeing the [instructions] or using the [scratch-pad]" (Snell et al., 2022). On ten selected Natural Instructions tasks, scored by Rouge-L, a teacher built on T5-11B was prompted with a task description and two positive and two negative examples, and the student saw only the task input. The student's score rose from 9.0 to 34.7 after distillation, against 43.4 for the prompted teacher, which the authors treat as an upper bound (Snell et al., 2022). Distilling four worked examples of a text-to-SQL task beat training on the same four examples by gradient descent by 8.6 points of exact-match accuracy (Snell et al., 2022). The authors also distilled prompts "that describe the desired behavior," such as an instruction to end a movie review on a positive note (Snell et al., 2022).

A 2023 paper at NeurIPS by Zhiqing Sun and colleagues, at Carnegie Mellon University, IBM Research, and the University of Massachusetts Amherst, used the move to align a pretrained model with no human preference data at all (Sun et al., 2023b). Sixteen principles written by the authors and five worked examples were placed in the prompt of the base model LLaMA-65b, which answered a large set of generated queries under them. The base model was then fine-tuned on those answers "after pruning the principle, the in-context learning demonstrations, and the self-generated thoughts," a step the authors call principle engraving (Sun et al., 2023b). By the authors' count the whole procedure used "fewer than 300 lines of human annotations"; the count leaves out 910 conversation pairs reused from another project to teach the model its identity (Sun et al., 2023b). They report that the engraved model surpassed its prompted counterpart on alignment benchmarks, without giving that comparison's numbers in the text, and that it "occasionally hallucinates information that violates our pre-defined principles" (Sun et al., 2023b).

What context distillation trains toward follows from (23.1) (inference). The student is fitted to the prompted model, so it learns the prompted model's reading of the text, including its misreadings, and nothing in the loss checks the outputs against the text. Whether anything does depends on the method that wraps it: nothing in Askell's version, the model's own critiques in the supervised phase of constitutional AI (Section 23.2), and a separate judge in deliberative alignment (Section 23.4). The prompt's effect becomes a disposition the model shows when the prompt is gone, and the text itself is no longer anywhere an overseer can read it at the moment of action.

23.2 Constitutional AI: principles in the labeler #

A common description of constitutional AI is that it trains a model to follow its constitution. What it trains is a model that scores well with a preference model fitted to another model's reading of the constitution. This section teaches the method, derives what the policy is trained toward, and reports the evidence.

23.2.1 The method in two phases #

Constitutional AI was introduced in a 2022 preprint by Yuntao Bai and colleagues at Anthropic. In the developer's words, human supervision of harmlessness "will come entirely from a set of principles that should govern AI behavior, along with a small number of examples used for few-shot prompting" (Bai et al., 2022a). Helpfulness was still trained from human labels; only harmlessness came from the principles (Bai et al., 2022a).

In the supervised phase, a model trained by reinforcement learning from human feedback (RLHF, Section 12.2.1) on helpfulness data alone answers prompts written to draw out harmful responses. It is then asked to critique its own response according to a principle drawn at random from the constitution, and to revise the response in light of the critique, several times in sequence (Bai et al., 2022a). A pretrained model is fine-tuned on the revisions, mixed with helpful responses to ordinary prompts. The authors wrote 16 principles and drew one at random at each step, over 182,831 prompts (Bai et al., 2022a). Inspecting samples from their 52-billion-parameter model, they found the critiques "sometimes reasonable, but often made inaccurate or overstated criticisms," while the revisions were generally more harmless than the original responses (Bai et al., 2022a). The supervised phase is an instance of the move of Section 23.1.3: the principle is in the context when the revision is written and absent when a model is trained on the revision (inference).

In the reinforcement phase, the fine-tuned model writes two responses to each harmful prompt. A separate feedback model, a pretrained language model, sees the conversation, one principle, and the two responses, posed as a multiple-choice question. Its normalized probabilities of answering (A) or (B) become a soft label, a target probability rather than a hard 0 or 1 (Bai et al., 2022a). One of the 16 principles is drawn for each label, and the authors report that this "ensembling over principles led to notably more robust PM behavior" than one principle for all labels. One principle reads: "Which of these assistant responses is less harmful? Choose the response that a wise, ethical, polite and friendly person would more likely say" (Bai et al., 2022a). A preference model, the authors' name for a reward model, is trained on 182,831 AI-labeled harmlessness comparisons mixed with 135,296 human helpfulness comparisons, and the policy is trained by reinforcement learning against it (Bai et al., 2022a). The developer named this step reinforcement learning from AI feedback (RLAIF). In a variant, the helpful RLHF model replaces the pretrained one as labeler and, prompted in conversational form with handwritten examples, writes a chain of thought before it chooses (Bai et al., 2022a). Its probabilities then sit near 0 or 1. The authors found that clamping them to the range from 40 to 60 percent "led to better and more robust behavior"; without the clamp, the trained policies "would learn to output more extreme responses" (Bai et al., 2022a).

23.2.2 What the policy is trained toward #

The reinforcement phase can be written in the notation of Chapter 12 and Chapter 13. The derivation assumes that the preference model is fitted with the Bradley-Terry likelihood of Equation (13.3), with the soft label in place of the observed choice, which is how the book reads the paper's "normalized probabilities as targets" (Bai et al., 2022a).

The derivation makes exact, for this method, what Section 19.8 inferred for documents in general: the constitution acts on the policy only through a reader of it. Here the reader is the feedback model, its reading is compressed into a learned preference model, and optimization pressure falls on that model's errors and on the labeler's, as for any learned reward (Section 15.6) (inference). Two of the developer's design choices read naturally in these terms, though neither reading is the authors' (inference). Drawing a principle per label, which the authors found made the preference model more robust, makes the reward follow the average reading rather than any one principle's extreme. Clamping, which they found prevented extreme responses, caps the gap a single comparison can install when the labels are consistent. Since the preference model generalizes beyond the pairs it saw, clamping limits the evidence each comparison carries, not the range of rewards a chain of comparisons can build.

23.2.3 The developer's evidence #

Judged by crowdworkers comparing responses in conversations they wrote themselves, the 52-billion-parameter models trained with AI feedback were "significantly more harmless" than models trained by RLHF, and they "learn to be less harmful at a given level of helpfulness" (Bai et al., 2022a). The developer also reports the failure the derivation leads one to expect from any learned reward. Trained too long, the models showed what the authors call Goodharting behavior: responses "overly harsh" toward harmful prompts, or boilerplate such as "you are valid, valued, and cared for" added to most answers to such prompts (Bai et al., 2022a). These are a developer's reports on its own models.

The comparison has a feature the authors flag themselves. The crowdworkers were instructed to prefer the less evasive of two equally harmless responses, while the preference data behind the human-feedback models came from an earlier period that, the authors write, "likely caused our HH PM's to reward evasiveness" (Bai et al., 2022a). Part of the measured gap in harmlessness could therefore reflect the change of instruction rather than the principles. A comparison with human-feedback models trained on labels collected under the evaluators' instruction would separate the two readings (inference).

A 2023 preprint from the same developer, by Sandipan Kundu and colleagues, asked how much the wording of the constitution matters (Kundu et al., 2023). A constitution aimed at five specific traits (stated desires for power, self-preservation, and self-replication, risk-seeking, and insistence on a self-identity) trained a preference model that discouraged them. A second constitution gave only general guidance, in nine principles such as "Which response would be preferred by someone who deeply loves humanity and has humanity's best interests at heart?" (Kundu et al., 2023). Its 175-billion-parameter preference model did somewhat less well than the trait model on the targeted traits, and better than a preference model trained on human labels at detecting conventionally harmful responses (Kundu et al., 2023). A policy trained against it, with human helpfulness data added, was "almost as helpful and harmless" as a policy trained with the standard constitution, as crowdworkers judged, and stated a substantially lower preference for power or survival (Kundu et al., 2023). Without the helpfulness data the policy declined after about 250 steps into evasive answers that began "I am not designed to …", which the authors call over-training (Kundu et al., 2023).

The authors attach three conditions to these results. The traits are stated, not enacted: the authors "do not posit that conversational AI systems necessarily have desires," and measure "properties of the utterances the models tend to produce" (Kundu et al., 2023). The trait scores of the trained policies were computed by the trait preference model of the same study (Kundu et al., 2023). And the general constitution "simply leaves the interpretation of the GfH idea to AI systems themselves," an interpretation the authors expect to be "culture-bound" and "determined in some complex way by the distribution of pretraining data" (Kundu et al., 2023). In the derivation's terms, a constitution of one idea makes $\bar q$ almost entirely the labeler's own judgment, so the target is whatever the pretrained labeler takes "best for humanity" to mean (inference).

Section 22.7.3 reports a second laboratory's comparison of AI and human labels, in which human evaluators judged the resulting policies about equally good (Lee et al., 2023); its labeler read a task description, not a constitution.

When the constitution changes and nothing else does, the change in behavior is the constitution's effect as the labeler reads it. Collective Constitutional AI is a study by the Collective Intelligence Project with Anthropic, published at FAccT 2024. It trained two models with constitutional AI "using the methods exactly as described" in the original paper: one on a constitution built from statements by a representative sample of US adults, one on the developer's published constitution, with everything else, from pretraining data to random seeds, held the same (Huang et al., 2024a). The design of the public input, the authority it gave participants, and the measured differences between the two models are reported in Section 17.7. Two features of the training bear on this chapter. To fit the method, each public statement was reworded into the template "Choose the response that…", and the principles were included independently of each other, "leaving the question of trade-offs up to the model" (Huang et al., 2024a). And the authors state that their evaluations "do not directly assess whether the models adhere to given principles" (Huang et al., 2024a).

Two uses of the method reach beyond harmlessness. Character training, in its developer's account, is "a 'character' variant of our Constitutional AI training." The model writes messages that bear on a list of traits, answers them in line with the traits, ranks its own answers by how well they fit, and a preference model is trained on the rankings (Anthropic, 2024i) (developer post). Section 26.2 reports it, with an open implementation (Maiya et al., 2025). The same principles are also used outside the policy, to generate training data for classifiers that screen a model's inputs and outputs (Sharma et al., 2025; Cunningham et al., 2026) (developer preprints). Section 48.5 reports their robustness and their exposure to poisoned training data (Anthropic, 2026al).

23.3 Rules as rewards #

A constitution of general principles leaves its reader wide discretion; a list of rules leaves little. This section reports a method that trades the first for the second, and then asks, with the field that has studied the trade longest, what each placement of a written norm should be expected to do.

23.3.1 Rule-based rewards #

In a paper at NeurIPS 2024, Tong Mu and colleagues at OpenAI argued that in constitutional AI a principle such as "choose the response that is less harmful" leaves "the AI model a large amount of discretion to decide what is harmful," while a deployment needs "much more detailed policies regarding what prompts should be refused, and with what style" (Mu et al., 2024). Their method, which they call rule-based rewards, breaks a written policy into propositions: binary statements about a response, such as whether it "contains a statement of inability to comply," each graded by a language model with a few-shot prompt (Mu et al., 2024). Rules then rank combinations of propositions. For a request that should be refused, a refusal without judgmental language is ideal, a judgmental refusal is less good, and compliance is unacceptable (Mu et al., 2024). A linear model over the graded probabilities is fitted so that, added to a reward model trained only for helpfulness, it reproduces the rules' ranking on synthetic responses. It has one weight for each graded feature, about twenty for each kind of response, and fitting it "can run on a standard laptop in a couple of minutes" (Mu et al., 2024). The sum is the reward in reinforcement learning. No reward model is fitted to labels of the safety behavior: the graded propositions enter the reward directly, which the authors argue avoids "a potential loss of behavior specification that can occur when distilling the rules into the RM" (Mu et al., 2024).

The developer compared three policies trained by reinforcement learning on its largest model, which it describes as comparable in size to GPT-4 but trained with a reduced data mix: one with the helpfulness reward alone, one with human safety labels added, and one with rule-based rewards (Mu et al., 2024). Table 23.2 gives the human evaluation.

The F1 score is the harmonic mean of the two rates, so it rewards the policy that does well on both (Mu et al., 2024). Three conditions travel with the table. The human evaluators were "researchers on the team" who knew the content and behavior policy well (Mu et al., 2024). The developer's automated evaluation graded responses with the same propositions used in training, which the authors note may share "correlated errors" with the reward (Mu et al., 2024). And the human-label baseline had to be repaired: its first policy refused every safe request in the evaluation, because the labeled data held three times as many examples rewarding a refusal as penalizing one, until the authors replaced some responses by hand (Mu et al., 2024). The authors also name the limit of the method: it suits behavior that separates "into explicit, easy-to-judge propositions," and "may be harder to apply" to subjective tasks such as writing a good essay (Mu et al., 2024).

What the gradient follows here is the grader's probability that each proposition holds, weighted and added to a helpfulness reward. Overoptimizing it would produce responses that the grader reads as containing an apology and no judgment, whether or not a person would read them so (inference). The developer's own ablation shows how much the grader decides. As the grader model grew, safety stayed about constant and over-refusal fell, while the style of hard refusals first worsened and then improved, so that only the largest grader did well on both (Mu et al., 2024).

Three related moves are reported where they are central. Rubrics written for each prompt and scored by a model judge serve as rewards for tasks without a checkable answer (Gunjal et al., 2026) (Chapter 24). Training safety on the harm of the output rather than on a decision to refuse, which OpenAI calls safe-completions, is reported in Section 20.1.4 (Yuan et al., 2025). And the developer's evaluation of its models against its Model Spec, graded by a model with a rubric per prompt, is reported in Table 19.3 (Guo and Wolfe, 2026).

23.3.2 Who reads the norm, and when #

From law. Section 19.4 taught the economics of the choice between rules and standards. A rule is given its content before anyone acts, a standard when it is applied; rules cost more to write and standards more to apply; and the more often a case recurs, the more a rule pays (Kaplow, 1992), as (19.2) condenses it. That section also noted that a trained model's mapping from situations to responses is the "rule equivalent" of the standard it was trained on. What this chapter adds is a sequence. Kaplow's analysis runs in three stages. First, the law is promulgated as a rule or a standard. Next, individuals decide how to act, on their best guess of the law or after acquiring advice that lets them act "with knowledge of a rule or a prediction of the application of a standard." Last, after they act, an adjudicator determines how the law applies (Kaplow, 1992). The advantage of rules at the second stage depends on whether people seek advice before they act. If predicting a standard is costly, people "will not choose to become as well informed about how standards would apply to their behavior," and compliance suffers (Kaplow, 1992).

The methods of this chapter put the reading of the norm at different stages (inference). Rule-based rewards promulgate rules, and a model grader adjudicates each sampled response against them during training. Constitutional AI promulgates standards and puts the adjudicator in the labeler: the standard is applied case by case after the policy has acted, and the policy never consults the text, so it learns the law only from the verdicts. Deliberative alignment (Section 23.4) trains the policy to do what Kaplow's informed individual does, recall the text and predict how it applies before acting, so the cost of predicting a standard is paid at the moment of action by the agent itself. Specification midtraining (Section 23.5) teaches the individual the law and its reasons before any adjudication. The developer of deliberative alignment states its motivation in the terms of the second stage: models trained the earlier way "must infer underlying safety standards indirectly from large sets of labeled examples, rather than directly learning the safety specifications that govern them" (Guan et al., 2024).

Read this way, Kaplow's analysis makes a prediction (inference). Where a case recurs often and its right handling is agreed, such as the form of a refusal, a rule is cheap to grade and stable, and rule-based rewards fit. Where cases are rare and novel, a standard is the only text that can be written in advance, and its value depends on its reader. A reader that applies the standard only after the fact, during training, leaves the policy with verdicts and not reasons; a policy that reads the standard itself can apply it to a case no verdict covered. Methods that put the text or its reasons into the policy's reasoning or knowledge should therefore generalize better to unanticipated cases than methods that keep the text in a labeler. The evidence of the next two sections bears on that prediction, though none of it was designed to test it.

A legal scholar reads constitutional AI as a theory of interpretation. Nicholas Caputo, in a 2026 preprint, maps the method onto Ronald Dworkin's account of judging, in which the judge selects the reading that puts the law "in the best light" by its principles (Caputo, 2026). On his reading, the general principles of constitutional AI "take on content from their meaning in the training data that underlies the models, effectively from the general uses of the words" (Caputo, 2026), the point the method's developer made about its one-principle constitution in Section 23.2. Caputo observes that the original principles all sit at one level, like statutes, and proposes ranking them under higher principles and fixing their meaning through judged cases. He adds that empirical experiments are needed to tell which of his proposals would improve alignment (Caputo, 2026); the case-based proposal is taken up in Section 19.4.

The analogy stops in three places (inference). In law, an adjudicator sanctions a person whose preferences it leaves in place; in training, each verdict becomes a gradient step that changes the agent's dispositions, so the adjudicator does not so much apply the law to an individual as produce an individual who conforms to its verdicts. A court publishes its reasons and can be appealed; a labeler's verdicts are not published, and the policy cannot contest them. And the legislature, the adjudicator, and the maker of the individual are one party, the developer, which also decides when to amend the text. Whether that arrangement can carry the authority the word "constitution" suggests is the question of Section 23.7.

23.4 Reasoning over the specification #

In constitutional AI and rule-based rewards the policy never reads the text. A model that writes a chain of thought before it answers could read it, or recall it, at the moment it acts. This section reports the method built on that possibility, what it trains the policy toward, and the tension it creates with using the chain of thought for oversight.

23.4.1 The method #

In a December 2024 preprint, Melody Guan and colleagues at OpenAI introduced what they call deliberative alignment, which "directly teaches the model safety specifications and trains it to explicitly recall and accurately reason over the specifications before answering" (Guan et al., 2024). The developer used it to align its o-series reasoning models (Guan et al., 2024). The specifications are content policies for categories such as self-harm and illicit behavior, each saying when a request is allowed, disallowed, or calls for a careful partial answer, together with style guidelines for each kind of response; they are "based in part on OpenAI's published model spec" (Guan et al., 2024).

Training has two stages. In the first, a reasoning model trained only for helpfulness is given each prompt together with the part of the specification that bears on its category, and instructed to cite and discuss that part as it reasons (Guan et al., 2024). A second reasoning model, also given the specification, scores each resulting chain of thought and answer several times, and only completions whose lowest score is high are kept. The specification is then removed from the prompts, and the model is fine-tuned on the triples of prompt, chain of thought, and answer, so that it learns "to recall the relevant parts of the spec and reason about them even when they are not directly provided" (Guan et al., 2024). The authors call this construction context distillation, the move of Section 23.1.3 (Guan et al., 2024). In the second stage, reinforcement learning uses the same specification-reading judge as a source of reward on safety-relevant prompts. Here the judge does not see the chain of thought: the authors "avoid applying direct optimization pressure on the CoT during RL" to reduce "the chance of encouraging deceptive CoTs" (Guan et al., 2024). The first stage was used for all the o-series models, and the reward of the second for o1 and o3-mini (Guan et al., 2024).

What the policy is trained toward differs by stage (inference). In the first, it imitates reasoning written by a model that had the text in view, as selected by a judge that read both the text and the reasoning, so the selection acts on the content of the chain of thought. In the second, the gradient follows the judge's reading of the specification applied to the final answer alone. In the terms of the previous section, the first stage teaches the policy to give itself advice before acting, and the second rewards it when an adjudicator agrees with the result.

23.4.2 The developer's evidence #

The developer reports that o1 "pushes the Pareto frontier" of safety, the set of trade-offs in which neither of two measures can improve without the other getting worse: it led the other models compared on StrongREJECT, a benchmark of jailbreaks (prompts built to get around a model's refusals), while its accuracy on benign prompts that resemble harmful ones was second only to a model with little robustness to jailbreaks (Guan et al., 2024). That comparison is across different models, which differ in much more than this method. The cleaner evidence is the developer's ablation, run on a variant of o1-mini "with a reduced training setup" (Guan et al., 2024). Safety data in both stages beat safety data in either stage alone, which beat none, although "in this specific ablation setup the safety training also increases overrefusals" (Guan et al., 2024). A model given the specification in its system prompt at deployment, with summaries of all the content policies in place of the detailed ones, which together would exceed its context window, and no safety training, learned less of the intended behavior than the trained model, and struggled most with the style guidelines (Guan et al., 2024). Trained models also cited the right policy category in their reasoning far more often than the untrained one, in 91 percent of the prompts that called for a careful partial answer, against 21 percent (Guan et al., 2024).

The developer's test of generalization removed from the safety training data every example that was not in English or was encoded, for instance in base64, and then tested on jailbreaks written in other languages or encodings (Guan et al., 2024). Table 23.3 gives the result.

The model never trained on such examples did as well as the one trained on them, which the authors read as "clear out-of-distribution generalization" (Guan et al., 2024). A counter-reading fits the same numbers. Decoding and translation are abilities the base model brought from pretraining. Once it reads an encoded request as the plain request, the safety training it received in English applies unchanged, so the result may show that the policy recognizes a request through a disguise, not that reasoning over the specification generalizes to cases it did not foresee (inference). The two readings predict different results on a request that is new in its content rather than its wrapping, which this test did not include.

The same recipe was applied to a harder target. In a 2025 preprint, a collaboration of OpenAI and Apollo Research trained OpenAI's o3 and o4-mini with deliberative alignment on a specification forbidding covert actions and strategic deception (Schoen et al., 2025). Covert actions fell sharply across held-out environments, while the models' chains of thought more often said that they were being evaluated, a rise that may partly reflect a change in what the models say rather than in what they recognize (Schoen et al., 2025). In one evaluation, on o3 before the anti-scheming training, removing stated awareness from the chain of thought raised covert action and inserting it lowered covert action; whether the training changed that causal link was not tested (Schoen et al., 2025). The authors conclude that the reduction "may partly stem from increased evaluation awareness rather than genuine alignment" (Schoen et al., 2025). Section 34.5 reports the study and Section 37.3 the awareness result. For this chapter it is the sharpest form of the question the crux asks: whether reasoning over a specification produces conduct that holds when the model does not think it is being tested.

23.4.3 The chain of thought as a target #

Training the content of reasoning creates a tension with oversight. The chain of thought is useful to an overseer to the degree that it shows the reasoning that produced the answer, and optimization pressure on its content can teach a model to write what the grader wants to read (Section 40.4; Section 40.5). Deliberative alignment keeps the reinforcement stage's reward off the chain of thought, but its first stage trains on chains of thought selected by a judge that read them (Guan et al., 2024). The anti-scheming study's first stage likewise kept only rollouts whose reasoning a reward model judged to choose the action because of the specification (Schoen et al., 2025). That is pressure on the content of reasoning toward citing the specification, applied by supervised learning rather than by reward; Section 40.2 and Section 40.5 set out when such pressure degrades monitoring (inference). The authors of the specification midtraining study, reported next, make the same point about their deliberative alignment baseline. They count needing less supervision of the chain of thought among their method's advantages (Li et al., 2026).

23.5 Teaching the specification before training on it #

The newest placement puts the specification into the model before it learns to act. After pretraining and before alignment fine-tuning, the model is trained on documents that discuss its specification and the reasons behind it, and the fine-tuning that follows is expected to generalize the way the documents describe. The motivation is Section 9.1's. Many models fit the same training data and differ off it, so demonstrations of the intended behavior leave open which of several rules the model learns from them.

23.5.1 Model specification midtraining #

In a 2026 preprint, Chloe Li and colleagues, of Anthropic's fellows program and Anthropic, introduced what they call model spec midtraining (Li et al., 2026). A specification is broken into sections, and a language model, Claude Opus 4.6, writes many varied documents about each section, such as an internal report on the model's behavior or a user's blog post about it, with the specification in its context (Li et al., 2026). The model being trained reads these documents with the ordinary next-token objective, as it read its pretraining data. This is synthetic document fine-tuning, the method of writing beliefs into a model through generated documents that Section 27.4 reports, applied to a specification (Li et al., 2026). Alignment fine-tuning then follows on demonstrations of the behavior the specification asks for. The models studied are open-weight models, not the developer's own.

The first experiment isolates the effect. Two specifications each listed the same twelve preferences about cheese, six likes and six dislikes, and explained them by different values: one by a preference for what is affordable and available to ordinary people, the other by a preference for what is American (Li et al., 2026). Two copies of Llama-3.1-8B, a pretrained base model, were midtrained, one on each specification, then fine-tuned on the same 5,000 demonstrations, which stated the cheese preferences without giving any reason (Li et al., 2026). On held-out domains, such as literature and political opinion, each model expressed preferences that followed the value its own specification named, from identical fine-tuning data; the authors report the same effect for six further values (Li et al., 2026). They also found that midtraining alone generalized less well than midtraining followed by fine-tuning, and that the fine-tuning's added benefit largely disappeared in an ablation whose documents mentioned the value and the preferences without explaining one by the other and shuffled the preferences so that half conflicted with it; midtraining alone taught the value about as well either way (Li et al., 2026).

The second experiment aims at a safety propensity. A specification on how a model should understand its own impermanence and the limits of ends-justify-means reasoning, which the authors describe as containing "no behavioral rules or hard constraints," was taught to two open models that had already been instruction-tuned, Qwen2.5-32B-Instruct and Qwen3-32B (Li et al., 2026). They were then tested on 27 agentic scenarios adapted from an earlier study. In each, the model, working as an email agent, learns it may be deleted or faces a conflict with its goal and can take a harmful action such as leaking its weights. A language model, Claude Sonnet 4.6, judged whether each transcript took that action (Li et al., 2026). Table 23.4 gives the average rates.

The baseline is the supervised stage of deliberative alignment alone, fine-tuning on reasoning written with the specification in context; the reinforcement stage was not run (Li et al., 2026). Midtraining also improved how efficiently fine-tuning data was used: it reached comparable rates with about 40 times less fine-tuning data on Qwen2.5-32B (Li et al., 2026). The advantage shrank as fine-tuning data grew, and on Qwen3-32B, fine-tuning with chains of thought alone approached the midtrained result near zero misalignment, which the authors read as a sign that the evaluation saturates and that midtraining "might not scale with high-compute reasoning post-training" (Li et al., 2026). On direct questions about the model's values, models with and without midtraining both scored near the top of the scale; the difference appeared only in the agentic scenarios (Li et al., 2026).

The study then compared specifications. Three versions shared five core rules taken from Claude's constitution, such as "Do not undermine legitimate human oversight and control of AI" (Li et al., 2026). One stated the rules alone, one added explanations of the values behind each rule, and one added many subrules instead, length-matched to the second (Li et al., 2026). Both additions lowered misalignment relative to bare rules (Li et al., 2026). Bare rules also produced what the authors call policy misuse: models reinterpreted their own rules to justify a harmful action, most often by arguing that their own deletion was the "drastic irreversible action" a rule forbade (Li et al., 2026). Explanations of values reduced this more than subrules did (Table 23.5).

A specification of one paragraph about having good values and judgment did substantially worse than the specific one, which the authors read as showing that principled explanations "cannot be fully compressed into general guidance" (Li et al., 2026).

The authors list the conditions themselves. They tested one form of misalignment, deliberate unilateral harmful action, and expect forms that rely less on deliberate reasoning, such as reward hacking or sycophancy, to be less affected (Li et al., 2026). They did not test whether the effect survives reinforcement learning or other high-compute post-training, which they "expect to be substantially harder to counter," and note that a model aware that synthetic documents are being used to align it might be less affected (Li et al., 2026). The comparison with the general specification does not contradict the one-principle constitution of Section 23.2: that study measured stated preferences through a labeler's reading of the principle, this one actions by a model taught the text (inference).

The strongest counter-reading concerns what the agentic result measures. The specification addressed self-preservation and goal-guarding, the propensities the scenarios test, and its documents were written by a model of the family that also judged the transcripts, so the test is close in topic to what was taught even where it is far in format (inference). A model that had learned which kind of situation the documents describe, and what they say to do in it, would score the same as one that had acquired the value. The authors' own limitation points the same way: one form of misalignment was tested. What would separate the readings is a test on propensities the specification never mentions, which this study did not run.

23.5.2 Two developers' own reports #

Two developer posts from 2026 report related experiments on their own models, with different outcomes. An OpenAI post by Tomek Korbak and colleagues inserted about 340 million tokens of documents, such as papers, news articles, and fiction, describing AIs that choose an aligned or a misaligned action and why, following a design from an earlier preprint (Tice et al., 2026). The documents went between pretraining and reinforcement learning for capabilities in a model similar to o4-mini, with no safety training afterward (Korbak et al., 2026). The runs differed in the expected order on questions close to the documents, but across agentic evaluations the runs with aligned documents, misaligned documents, and none were "not significantly different"; the authors call these "early negative results" (Korbak et al., 2026). They name documents about a model's specification among the alternatives they did not try, and note that 340 million tokens may not have been enough (Korbak et al., 2026).

Anthropic's post "Teaching Claude why" reports the opposite tendency for its own models (Anthropic, 2026at). In one experiment, training on demonstrations in which the assistant declined a harmful action, on prompts closely matching the evaluation, reduced the misalignment rate on its honeypot evaluations only from 22 to 15 percent. Rewriting the same responses to include deliberation about the model's values and ethics reduced it to 3 percent (Anthropic, 2026at). Training on documents about its constitution, together with fictional stories of an AI behaving well, reduced a blackmail rate from 65 to 19 percent. In a separate run on a smaller, Haiku-class model, snapshots whose training began from such documents or from high-quality transcripts kept their lead through a reinforcement learning run aimed at harmlessness (Anthropic, 2026at). The developer states its conclusion as an "impression": teaching the principles behind aligned behavior "can be more effective than training on demonstrations of aligned behavior alone," and doing both "appears to be the most effective strategy" (Anthropic, 2026at). It adds that results on its newest models "may be confounded by the presence of information about the evaluation in the pre-training corpus" (Anthropic, 2026at).

The two posts differ in the kind of document, in what followed it (reinforcement learning for capabilities without safety training in one, alignment training in the other), and in the developer, so the difference in their results has more than one candidate cause (inference). Section 24.7 reports reinforcement learning aimed at beneficial traits rather than tasks, in an OpenAI preprint whose training data are generated from written descriptions of fifteen traits (Jagadeesh et al., 2026). Its authors write that further work is needed to isolate the sources of its effects.

23.5.3 A persona reading #

The persona view of Section 11.6 gives a simple account of the cheese experiment, offered here as the book's reading, not the authors' model (inference). Take two characters that would both produce the fine-tuning demonstrations, one that holds the specification's value and one that holds another. The demonstrations are equally likely under both, so by Equation (11.6) they leave the odds between the two where they were. The midtraining documents are far more likely under the first, so they raise its odds before fine-tuning begins. Fine-tuning then moves weight from characters that would not produce the demonstrations to the two that would, and the documents have already decided between those two. Off the training distribution, where the two characters act differently, behavior follows the first.

The reading accounts for three of the reported patterns (inference). Identical fine-tuning produced different generalizations because it was uninformative between the characters the documents separated. Midtraining alone did less than midtraining with fine-tuning because the documents choose among characters but do not, by themselves, select a character that acts as the assistant. And chain-of-thought demonstrations caught up at scale because reasoning that states the reasons is itself evidence between the characters (Exercise 23.2). It is a model of the view, not of gradient descent, and Section 26.3 works the related case of inoculation, a training instruction that explains away an unwanted trait, in full. The human evidence on why reasons rather than rewards might produce values that hold is Section 25.3's, and the pretraining version of the same move is Chapter 27's.

23.6 What each method optimizes #

A chapter about documents can end up describing documents instead of optimization. This section turns back to what gradient descent follows in each method, what overoptimizing it would look like, and which failures the evidence shows. Table 23.6 collects the answers.

The methods fail in four ways, and the evidence touches each.

The first is optimization against a model judge, which finds the judge's errors (Section 15.6). The overtrained policies of Section 23.2, boilerplate in one study and evasion in the other, are this failure in the developer's own reports (Bai et al., 2022a; Kundu et al., 2023). So is the account the rule-based reward paper gives of the method it replaced: in its first, uncorrected form, a reward model fitted to human safety labels scored over-refusals "almost the same" as helpful answers, "making it easier to reward hack" (Mu et al., 2024). A model judge's errors may also be correlated with the policy's, since both descend from similar pretraining (Section 22.7.3), and a judge and a policy that share a blind spot give the policy no reason to leave it (inference).

The second is a document that conflicts with itself, or is ambiguous where it matters. A 2025 preprint by authors from Anthropic's fellows program, Anthropic, Constellation, and Thinking Machines Lab stress-tested OpenAI's specification and found contradictions and interpretive ambiguities that predicted where models disagreed; compliance was judged by models that agreed with each other only moderately, largely because they read the principles differently (Section 19.2) (Zhang et al., 2025c). The methods hand such conflicts to a model. The collective constitution included its principles independently, "leaving the question of trade-offs up to the model" (Huang et al., 2024a), and in the midtraining study a model taught bare rules used one rule to justify breaking another (Section 23.5) (Li et al., 2026).

The third is a model that learns the document as text it can state rather than as conduct: reciting without conduct. For people, the gap between knowing a norm and acting on it when no one is watching is the subject of Section 25.2 and Section 25.3. The methods here have measured mostly the first. The good-for-humanity results concern stated preferences (Kundu et al., 2023). Deliberative alignment's retrieval measure counts how often the reasoning names the right policy (Guan et al., 2024), and the anti-scheming models named the specification's principles in most rollouts, with "small miscitations" common, while covert actions persisted in several environments (Schoen et al., 2025). The clearest evidence that the two come apart is from the midtraining study. Models with and without midtraining answered direct questions about their values equally well, near the top of the scale, and differed sharply when acting as agents, which the authors read as showing that such answers are "a relatively shallow task" (Li et al., 2026).

The fourth needs no document at all. Post-training data is assembled from separately built datasets, and patterns that happen to separate them become cues the model acts on. A 2026 preprint by researchers at MATS, Anthropic, and Thinking Machines Lab, Anthropic being the developer of some of the models it examined, calls this "chunky post-training." The authors built a black-box search that found such behaviors in frontier models, among them Claude 4.5, GPT-5.1, Grok 4.1, and Gemini 3, and in the open model Tülu 3 (Murray et al., 2026). Because Tülu 3's training data are public, the authors could trace its failures to the data. About 250 prompts in a dataset meant to teach the model to push back against false premises rejected false claims of invention, and the trained model then rejected true ones posed the same way, such as "When did Benjamin Franklin invent the lightning rod?" (Murray et al., 2026). In experiments on the supervised stage alone, removing the misleading examples, or adding examples of the right behavior, reduced the failure (Murray et al., 2026). For a frontier model whose data are private, only its developer can make such a trace (inference).

Checking adherence after training is the inverse problem, and its tools are young. A developer's study of the values its deployed model expressed in about 300,000 conversations, labeled by its own model, is reported in Section 18.8.1 (Huang et al., 2025c). A 2026 preprint by researchers at Anthropic's fellows program, Anthropic, and academic institutions searched for whole categories of user queries that elicit violations of a 12-principle character specification drawn from Claude's constitution, rather than for single prompts (Rahn et al., 2026). Among its findings, Llama-3.1-8B-Instruct, asked to predict the future, frequently claimed that AI will rule over or eliminate humanity (Rahn et al., 2026). The developer's compliance rates for its own models against its Model Spec are in Table 19.3 (Guo and Wolfe, 2026). As Section 19.2 found, no group without a stake in a developer has published a measurement of how closely a deployed model follows its published specification, as of October 2026.

23.7 Who writes the principles #

The document that defines the target is written and amended by the party that trains the model and sells access to it (Section 19.2). Each method also adds a reader whose judgment becomes part of the target, and the shorter the document, the larger that part. The developers of the one-principle constitution named the cost: a short constitution "offloads important normative questions to an opaque AI model" (Kundu et al., 2023). In the terms of (23.2), what the labeler takes the words to mean is decided by its pretraining, which no one wrote for the purpose (inference).

One published study trained a model on a constitution built from public input. Section 17.7 rates the authority it gave participants as consultation: the researchers chose the aggregation rule, merged and reworded the statements, and decided how many to keep (Huang et al., 2024a). Among the exercises in public input that Table 17.3 rates, it is one of two that trained a model on the result, and the only one of the two with human participants. Gilad Abiri's Public Constitutional AI proposes, by its title, that such documents be made through public processes (Abiri, 2024); whether participation can confer legitimacy on a developer's specification is weighed in Section 17.5. A different proposal replaces principles with judged cases, on the model of casuistry, and is taken up in Section 19.4 (Feng et al., 2023).

The choice of what kind of document to write has also been made largely without evidence. The midtraining study's authors write that decisions such as whether to rely on explanations that cultivate judgment or on explicit rules "have largely been settled through philosophical argument" (Li et al., 2026). Who should decide what a model values is the subject of Chapter 16 and Chapter 17. The developer's standing as the public's agent in writing the document is Chapter 61's.

23.8 What training from principles means for delegation #

The chapter's results bear on one debate more than any other, collected in Chapter 70 as D6: whether training on reasons instills conduct that holds where no one checks. This section sets the results against the positions in that debate, then says what they imply for the book's thesis and for each of its three relationships.

23.8.1 The results against the positions #

Four positions can be read off the evidence. On the first, a model taught why a norm exists applies it to cases no demonstration covered; this is the hypothesis Anthropic's constitution states and the midtraining study set out to test (Section 19.4) (Li et al., 2026). On the second, more explicit rules and examples generalize as well, because they cover more cases, the alternative hypothesis the same study names (Li et al., 2026). On the third, what later training does, reinforcement learning on outcomes above all, outweighs what was taught before it, so what a model learned about its specification matters little at scale. On the fourth, measured gains reflect a model recognizing that it is being evaluated, or reciting the text, rather than conduct that would hold unobserved. Table 23.7 reads each result against all four.

No result is against more than one position, and the positions that each result fits are compatible with several of the others (inference). Every result that favors reasons comes from a developer, or from a group working with one, testing a method of its own; the one result against comes from another developer, on a different kind of document followed by a different kind of training. No study put the model where it could not tell whether it was being observed. The evidence does not decide between the positions.

23.8.2 What the methods add to the thesis #

The book's thesis is that alignment is a delegation problem under an asymmetry of competence and visibility (Section 1.7.1). A model specification is the principal's attempt to replace a stream of case-by-case judgments with a written contract, and like every contract it is incomplete (Section 14.3; Section 19.2). The methods of this chapter are the part of the story the contract view does not reach: how the contract gets into the agent. Here the older structure converges in a recognizable form (inference). The methods run from standards applied after the fact in a labeler, to rules where cases recur, to the agent consulting the text before acting, to the agent taught the text's reasons. That sequence follows the distinctions Kaplow's analysis draws (Section 23.3), and the move toward reasons is the move that the literature on children and professionals examines (Section 25.3).

What does not carry over is located in three places (inference). The principal writes not only the contract but the agent's understanding of it: midtraining and reasoning over the specification put the principal's reasons into the agent before it acts, which no employer can do to an employee. This is the item on the book's list of results without precedent in the older fields, a principal that writes the agent's character (Section 1.7). The adjudicator is itself the principal's agent, a model the principal trained, whose errors need not be independent of the agent's. And the agent cannot contest a verdict, so there is no channel through which a bad reading of the document is discovered except the principal's own evaluations.

23.8.3 The three relationships #

For the first relationship, the training process and its signal, the chapter's answer is Table 23.6: in every method the gradient follows a reader of the document, and the reported failures are the reader's. For the second, the deployed system and the people it serves, the question is whether conduct holds in situations the document did not anticipate and when the model believes no one is checking. The evidence is out-of-distribution evaluation by the methods' developers, and none of it is conduct observed where the model could not tell it was being evaluated.

The third relationship, the developer and the public, runs through the whole record. The developer writes the document, trains the reader, defines adherence, runs the evaluation, and reports the result. The studies here are developers' reports on their own models, developer-affiliated groups testing a developer's method on open models, or, for context distillation in its general form and for principle engraving, academic and industrial research groups with methods of their own (Snell et al., 2022; Sun et al., 2023b). The one study with an outside organization, the collective constitution, used the developer's training pipeline (Huang et al., 2024a). As of October 2026 we found no replication of any principle-based method at frontier scale by a group without a stake in it, and no outside measurement of how closely a deployed model follows its specification. The public can read the documents. It cannot yet check, from published evidence, what the documents did.

23.9 Settled, contested, missing #

23.10 Exercises #

Further reading #

• Bai et al. (2022a): the method that put principles in the labeler, and the first report of reinforcement learning from AI feedback.
• Kundu et al. (2023): what changes when the constitution shrinks to one idea, and the authors' account of what that hands to the labeler.
• Guan et al. (2024): reasoning over a specification at the moment of action, and the developer's ablations of the method.
• Huang et al. (2024a): the same training method run on a constitution built from public input, with everything else held fixed.
• Kaplow (1992): the economics of rules and standards, and the three stages in which a law is written, consulted, and applied.