Draft complete but not yet reviewed; expect revisions

Reinforcement Learning

The previous chapters gave the reader two pieces: an agent that acts in an environment and receives reward (Chapter 7), and a language model that writes text one token at a time (Chapter 11). Reinforcement learning joins them. It is the family of methods that changes a policy, the rule by which a system chooses its actions, so that actions followed by higher reward become more probable. Assistants are fine-tuned this way from human preferences, and reasoning models are trained this way on problems whose answers a program can check. Most of what the book calls instillation, getting a learner to acquire what was specified rather than a proxy that scored the same in training, happens in this step. Nearly every claim in this chapter is therefore about the first of the book's three relationships: the training process and the signal it optimizes.

One equation carries much of the rest of the book. When reward is maximized with a penalty for moving away from the starting model, the best policy has a closed form: the starting model's probabilities, reweighted by exponentiated reward. The chapter builds up to that equation from the simplest learner, reads the methods that train today's assistants and reasoning models through it, and ends with what the penalty does not guarantee, where the chapter on optimization pressure (Chapter 15) takes over.

12.1 Learning from reward by gradient #

Gradient descent (Section 8.3) needs a loss whose slope it can compute with respect to every parameter. In supervised learning the data say what the output should have been, and the loss measures the difference. Learning from reward has neither. No one says what the right action was: the system tries an action, and the world returns a number. And the number cannot be differentiated, because the action was sampled, a discrete choice drawn at random, and the reward was computed outside the model, by a unit test, a person, or a game engine. No chain of derivatives runs from the reward back to the weights. This section shows how to follow the reward uphill anyway.

12.1.1 The idea: make what worked more probable #

Start small. A website can show one of five versions of a page per visit, and each visit ends in a noisy number, such as the time the visitor spent, with a fixed average for each version. The engineer wants a rule that shows the best version more often. In the vocabulary of reinforcement learning this is a bandit (after the slot machine, the one-armed bandit): five arms, one choice per round, and a reward after each choice.

A natural rule keeps a probability for each arm and, after each round, nudges the probability of the arm it just pulled: up if the reward was better than usual, down if it was worse. The table of probabilities is the policy, a rule that maps the current situation (here there is only one) to a probability distribution over actions. Each nudge is noisy, but over many rounds the nudges point, on average, toward the arms that pay more. The rest of the section makes "on average" exact.

12.1.2 The score-function estimator #

Write the policy as a softmax over a vector of parameters $\theta$, one number per arm, so that $\pi_\theta(a) = e^{\theta_a} / \sum_{a'} e^{\theta_{a'}}$ (the softmax of Section 8.1, which turns any vector of numbers into probabilities). Let $r(a)$ be the reward of arm $a$, and let the objective be the expected reward of the policy, $J(\theta) = \E_{a \sim \pi_\theta}[r(a)] = \sum_a \pi_\theta(a)\, r(a)$. Gradient ascent on $J$ would raise expected reward, but $\nabla_\theta J$ requires knowing every $r(a)$, which is exactly what the learner does not know. The trick is to rewrite the gradient as an expectation under the policy itself, so that a single sampled action gives an unbiased estimate of it.

The vector $\nabla_\theta \log \pi_\theta(a)$ is called the score of the action, hence the estimator's name. Equation (12.1) is the rule of the previous subsection made exact: move the parameters in the direction that makes the sampled action more probable, by an amount proportional to the reward it earned. Williams set out algorithms of this form in 1992 under the name REINFORCE (Williams, 1992). Methods that follow such an estimate are policy-gradient methods, and the policy gradient theorem extends them to policies represented by function approximators acting over many steps (Sutton et al., 1999).

12.1.3 Baselines and the advantage #

The estimator is unbiased but noisy, and one source of noise is avoidable. If every arm pays between 0.8 and 1.5, every sampled arm is pushed up, and which arm is better shows only in the relative sizes of the pushes. Comparing each reward with what was expected would push a mediocre arm down and a good one up. Subtracting a baseline $b$ from the reward does this, at no cost in bias.

Williams proved the same condition for his algorithms: the update stays an unbiased estimate of the gradient whenever the baseline does not depend on the output about to be sampled (Williams, 1992). A good baseline is the reward the policy expects to earn, so that the quantity multiplying the score,

is how much better the action did than expected. This is the advantage. In a problem with many states, the expected reward from a state is the value function $V(s)$ of Section 7.4, and a network trained to predict it, often called a critic or value model, serves as a learned baseline.

Figure 12.1 runs the estimator on this bandit. Averaged over 4,000 runs (200 draws of twenty) at the default learning rate, the best arm's probability after 300 updates is 0.73 without a baseline, and 23 percent of runs end with it below one half. With a baseline, an exponential moving average of past rewards, the average is 0.86 and 7 percent of runs end below one half. The baseline did not change where updates point on average, only how far a single run strays, and that decides where some runs end: a few lucky rewards from a poor arm can push its probability so high that the run rarely samples anything else again.

12.1.4 Taking several steps safely: PPO #

Samples are expensive: for a language model, each one is a generated response. It is tempting to take several gradient steps on one batch, but after the first step the samples no longer come from the current policy. The authors of proximal policy optimization (PPO) report, in a 2017 preprint, that several steps on the plain policy-gradient loss with the same trajectories often lead to destructively large policy updates (Schulman et al., 2017). An earlier method, trust region policy optimization, constrained each update by the KL divergence between the new and the old policy (Schulman et al., 2015b), a measure of how far one distribution has moved from another (Section 6.3) that this chapter returns to at length.

PPO keeps the trust region's effect with a simpler device. Let $\omega(\theta) = \pi_\theta(a) / \pi_{\theta_{\text{old}}}(a)$ be the ratio of the new policy's probability of a sampled action to the probability under the policy that sampled it. PPO maximizes the clipped surrogate objective

where $\operatorname{clip}$ limits the ratio to the interval $[1-\epsilon, 1+\epsilon]$ and the paper suggests values such as $\epsilon = 0.2$ (Schulman et al., 2017). Read it for one action with a positive advantage: the objective grows as the action becomes more probable, until its probability is 20 percent above where it was, and then it stops growing, so the optimizer has no reason to push further on this batch. A negative advantage is treated the same way from below. The clipping removes the incentive for moving the ratio outside the interval and makes the objective a pessimistic lower bound on the unclipped one. That lets PPO alternate between sampling and several passes (epochs) of minibatch updates over the same samples; in the authors' experiments, a variant that penalized the KL divergence instead performed worse (Schulman et al., 2017). PPO became the optimizer of the language-model pipeline in the next section.

12.1.5 Credit assignment #

One problem survives every refinement above. A language model often receives a single number for a response of a thousand tokens, and the estimator gives every token the same advantage. It cannot tell which tokens earned the reward and which merely came along. Assigning credit within a long response is the reason for per-token value functions, for rewards on each reasoning step rather than only the answer (Section 12.7), and for the token-level reading of preference methods (Section 12.6).

The advantage, reward received minus reward expected, has a proposed counterpart in the brain: the abstract of a 1997 paper by Schultz, Dayan, and Montague reports that the fluctuating output of dopamine neurons in primates apparently signals changes or errors in the predictions of future rewarding events (Schultz et al., 1997). Section 29.3 teaches that correspondence, and Section 29.5 the rival reading of what dopamine signals.

12.2 Language models as policies #

Everything above was written for arms and actions. To train a language model with it, the vocabulary has to be translated, and the translation has one choice in it that later sections depend on. In problems with many steps, an episode is one complete run from the first decision to the end, and its return is the total reward it collects (Section 7.4).

The last row is the chain rule of probability (Section 6.1), applied to the autoregressive model of Section 11.1: the probability of a sequence is the product of each token's probability given the tokens before it, so its logarithm is a sum. That gives two views of one problem. In the bandit view, the whole response is one action, scored once; Equation (12.1) applies with $y$ in place of $a$, and every token's score is multiplied by the same advantage. In the token-level view, each token is a decision in a process whose state grows one token at a time, and per-token value functions can assign credit within the response. Later sections use both.

12.2.1 The pipeline that made assistants #

Reinforcement learning from human feedback (RLHF) is the pipeline that turned pretrained language models into assistants. The developers of InstructGPT, at OpenAI, describe it as three steps (Ouyang et al., 2022). First, supervised fine-tuning: continue training the pretrained model by maximum likelihood on responses written by people as demonstrations of the desired behavior. Second, train a reward model: a network, written $r_\phi$ with parameters $\phi$, that reads a prompt and a response and outputs a score, fitted so that score differences predict which of two responses people preferred. Third, optimize the fine-tuned model with PPO against the reward model. How the reward model is fitted is taught in Chapter 13.

The method has a short lineage. A reward model learned from non-expert comparisons of short clips of an agent's behavior let agents solve Atari games and simulated locomotion without access to the true reward, with human feedback on less than one percent of their interactions (Christiano et al., 2017). Its likelihood follows the Bradley-Terry model of pairwise preference (Christiano et al., 2017), in which the probability that one item is preferred to another is the logistic function $\sigma(z) = 1/(1 + e^{-z})$ of the difference of their scores. Researchers at OpenAI applied the approach to pretrained language models in a 2019 preprint (Ziegler et al., 2019), to summarization in a 2020 paper (Stiennon et al., 2020), and to instruction following in the InstructGPT paper of 2022 (Ouyang et al., 2022). A 2022 preprint by Anthropic applied it to a helpful and harmless assistant of its own (Bai et al., 2022b).

The objective of the third step, in the form the paper gives for its models trained with PPO alone, adds one term to the reward (Ouyang et al., 2022):

where $\D$ is the distribution of prompts, $\pi_0$ the fine-tuned model training starts from, and $\beta > 0$ the weight of the penalty. The paper describes its environment in the bandit view, one response per episode, but applies the term as a per-token KL penalty from the supervised model, added to mitigate over-optimization of the reward model (Ouyang et al., 2022). The models the paper calls InstructGPT also mix in the pretraining objective, to reduce regressions on public benchmarks (Ouyang et al., 2022). The expectation of the log-ratio under the policy is the KL divergence of the policy from the reference,

the expected log-ratio of the two distributions, which is zero exactly when they agree and grows as the policy puts probability where the reference puts little (Section 6.3). So Equation (12.3) is expected reward minus $\beta$ times a KL divergence. From here on the book writes the policy $\pi$, the reference policy it is anchored to $\pi_0$ (the model training starts from and is penalized for leaving), a reward $r$, a learned reward model $r_\phi$, and the penalty $\beta\,\KL(\pi \,\Vert\, \pi_0)$.

In the developer's own human evaluation, by its contracted labelers on prompts submitted to its application programming interface (API), outputs of the 1.3-billion-parameter InstructGPT model were preferred to those of the 175-billion-parameter GPT-3, despite having over 100 times fewer parameters (Ouyang et al., 2022). The result is about following the instructions of that developer's users as its labelers judged them, not about capability in general, but on that measure a small fine-tuned model beat a large untuned one.

The same papers recorded the first warnings. In the 2019 preprint, summarization models trained with 60,000 comparisons could, in the authors' description, "roughly be described as 'smart copiers'": they copied whole sentences from the input while skipping irrelevant preamble. Labelers preferred them even to human-written reference summaries, though not to a baseline that simply copies the first three sentences (Ziegler et al., 2019). The authors give two explanations: copying is an easy way to be accurate when labelers are told to penalize inaccuracy but not copying, and some labelers may check for copying as a fast heuristic for accuracy (Ziegler et al., 2019). On the second reading, it is an early case of a policy finding what its grader rewards rather than what its grader meant, the pattern Chapter 30 follows to its present form (inference). In the 2020 summarization study, light optimization against a reward model improved summaries as people judged them, but as optimization increased, the labelers' preferences fell below what the reward model predicted, and eventually the reward model became anti-correlated with human preferences (Stiennon et al., 2020). Section 15.6 measures this curve.

12.2.2 Why the penalty #

Without the penalty, the optimum of expected reward is a policy that puts all of its probability on whatever response the reward model scores highest. The 2019 preprint shows what that looks like in practice: fine-tuned against a sentiment reward with no KL penalty, the model's samples were, in the authors' words, "gibberish even if we include an entropy bonus", a bonus for keeping the output distribution spread out (Ziegler et al., 2019). The anchor did not begin with language models: an earlier method treated a pretrained sequence model's next-token distribution as a prior policy to stay close to while it improved melodies and molecules (Jaques et al., 2017). The 2020 summarization paper gives two reasons for the term: it acts as an entropy bonus that encourages exploration and deters collapse to a single mode, and it keeps the policy's outputs close to the ones the reward model saw in training (Stiennon et al., 2020). A 2022 analysis states the first in general form: naive maximization of expected reward unavoidably leads to distribution collapse, a degenerate distribution concentrated on the highest-reward sequences (Korbak et al., 2022). The next section asks what the penalized objective produces instead.

12.3 The KL-regularized objective and its optimum #

Training against Equation (12.3) runs a noisy optimizer for a finite time. Before asking how well it does, ask what it aims at: which policy maximizes the penalized objective exactly? The question separates by prompt, since the objective averages over prompts and the policy may choose its response distribution for each prompt independently. So fix a prompt $x$, drop it from the notation, and ask which distribution $\pi$ over responses maximizes

12.3.1 The idea: reward as a temperature-scaled reweighting #

The answer has a familiar shape. Sampling a language model at temperature $T$ raises each next-token distribution to the power $1/T$ and renormalizes it (Section 11.1), a reweighting whose sharpness the temperature sets. The optimum of $J$ applies a reweighting of the same shape to whole responses: it reweights the reference policy by the exponential of reward divided by $\beta$, so the penalty weight acts as a temperature for the reward.

A two-response example shows the effect before the formula. Let the reference policy give an adequate response probability 0.9 and a better one probability 0.1, with rewards 0 and 1. At $\beta = 1$ the better response's weight is $0.1 \times e^{1} \approx 0.27$ against $0.9$ for the other, so after renormalizing the optimum gives it probability about 0.23. At $\beta = 0.25$ its weight is $0.1 \times e^{4} \approx 5.5$, and its probability rises to about 0.86. The optimum never jumps straight to the best response, and it never ignores the reference: it moves probability in proportion to $\exp(r/\beta)$, starting from where the reference policy put it.

12.3.2 The closed form #

For each prompt $x$, the maximizer of $J$ over all distributions on responses is

A reweighting by the exponential of a function is called an exponential tilt, and the book calls $\pi^\ast$ the tilt of $\pi_0$ for short. $Z(x)$ is the normalizing constant: the sum that makes the reweighted probabilities add up to one. It is a sum over every possible response, too many to compute for a language model, so the derivations below use it only as a symbol, and Section 12.6 shows how to avoid it entirely.

The same policy answers a second question, which is often the one a designer actually asks: what is the most reward a policy can earn without moving more than a given distance from the reference? Call the allowed distance a KL budget. The penalty form and the budget form trace the same set of best policies, as the next derivation shows without any machinery beyond the previous one. (In the language of constrained optimization, $\beta$ is the Lagrange multiplier of the budget constraint.)

12.3.3 Reward and divergence from one function #

Every quantity a later chapter needs about the optimum follows from the normalizing constant. Write $\kappa = 1/\beta$, the inverse KL weight, so that a large $\kappa$ means a weak anchor, and treat $\log Z$ as a function of $\kappa$.

So one function of one parameter traces the whole frontier: $\log Z$ and its derivative give the expected reward and the KL of the optimum at every $\beta$, and both rise together as $\beta$ falls. Two limits complete the picture. As $\beta \to \infty$ the reweighting flattens and $\pi^\ast \to \pi_0$. As $\beta \to 0$ it sharpens until all probability sits on the highest-reward responses that $\pi_0$ can produce, shared among them in proportion to their reference probabilities.

Later chapters use two further facts about the optimum. First, it never gives probability to a response the reference policy gives none, since $\pi^\ast$ is $\pi_0$ times a positive factor. For a language model this constraint is weak: a softmax gives every finite response some probability (sampling methods that cut off the distribution's tail aside). What matters is the second fact, the reward it takes to make a rare response common.

A response with reference probability $p_0$ whose reward exceeds every alternative by $\Delta$ dominates the optimum once $\beta < \Delta / \log(1/p_0)$. Rarity therefore protects only logarithmically. Making a response a thousand times rarer raises the reward gap it needs by $\beta \log 1000 \approx 6.9\beta$; a response with reference probability one in a billion dominates once $\beta$ falls below $\Delta/20.7$. Read the other way, at a given $\beta$ a reward gap $\Delta$ holds a response off only if its reference probability is below $e^{-\Delta/\beta}$. Whether rarity protects thus depends on the ratio of the reward gap to $\beta$, which Section 12.4.2 takes up, and Section 30.2.2 applies the same arithmetic to an exploit that a grader overpays.

Figure 12.2 computes the optimum for a reference policy over thirty responses whose best response has reference probability about $1.7 \times 10^{-5}$. Divergences are in nats, the unit of information when logarithms are natural (Section 6.2). At $\beta = 1$ ($\kappa = 1$) the optimum is barely distinguishable from the reference: it spends 0.01 nats of KL and raises the expected reward from 0.31 to 0.34. At $\beta = 0.1$ ($\kappa = 10$) it spends 1.39 nats and reaches 0.59, and at $\beta = 0.05$ ($\kappa = 20$) it spends 4.76 nats and reaches 0.82. The frontier in the second panel bends over, and the derivation above says by how much: dividing step 2 by step 4, its slope is $1/\kappa = \beta$, so at the optimum for a given $\beta$, one more nat of divergence buys about $\beta$ more expected reward.

The closed form describes exact optimization of a fixed reward over all distributions. Training is none of these: the network represents only some distributions, the gradient is estimated from samples, and the run stops. The 2019 preprint that brought RLHF to language models measured the gap once, by estimating the reward of the KL-constrained optimum through sampling and reweighting. On a mock sentiment task with a 124-million-parameter model, it found a significant gap between its PPO policies and the optimum after two million episodes, largely closed by further training (Ziegler et al., 2019). As of October 2026, we found no comparable measurement of reward for frontier post-training. Section 12.7.4 reports a measurement of a different kind, which shows reasoning training falling short of the optimum on its own training problems.

12.4 Post-training as Bayesian updating #

The closed form invites a reading. Equation (12.4) multiplies one distribution by a non-negative factor and renormalizes, which is exactly the shape of Bayes' rule (Section 6.1): a posterior is proportional to a prior times a likelihood. This section asks what that reading says about what post-training does to a pretrained model, and where the reading stops.

12.4.1 Prior, likelihood, and posterior #

Read $\pi_0$ as a prior over responses: what the pretrained or fine-tuned model already found likely. Read $\exp(r(y)/\beta)$ as a likelihood: how strongly a response is supported by a piece of evidence, here the fact that the reward favors it, with $\beta$ setting how much that evidence counts. Then $\pi^\ast$ is the posterior, the prior updated on the evidence. Korbak et al. (2022) make this reading precise. They show that KL-regularized reinforcement learning is equivalent to variational inference, the method of choosing, from a family of distributions one can work with, the one closest to a posterior one cannot compute directly. The family here is whatever the network can represent, and the closeness is measured in KL. Equation (12.5) already says so: maximizing $J$ is minimizing $\KL(\pi \,\Vert\, \pi^\ast)$, the direction of KL that Section 6.3 showed concentrates on regions where the target is high when the family cannot match it everywhere.

The authors use the reading to explain the penalty: it turns reward maximization, which collapses the model (Section 12.2.2), into matching a target distribution, which they argue explains why KL-regularized fine-tuning keeps the fluency and diversity of the original model. They suggest that reinforcement learning may not be an adequate formal framework for fine-tuning language models (Korbak et al., 2022). This is an argument about the objective, not an experiment.

12.4.2 What the reading predicts #

Three consequences follow, each a property of the optimum, not of any particular training run.

The first is that the prior survives. Take any two responses $y$ and $y'$. From Equation (12.4),

The reward decides between levels of reward; the prior decides within them. Responses the reward scores equally keep exactly the odds the reference policy gave them, at every $\beta$.

This is consistent with the persona view of Section 11.6, in which a pretrained model is a mixture over the many writers and characters its training text contains, and post-training reweights that mixture rather than writing behavior from nothing. The optimum does not test the view: the odds equation holds for any reference policy, and the view's distinctive claim, that the reference's probability is organized by characters, is one the mathematics neither needs nor checks (inference). Chapter 26 and Chapter 32 test the view against what fine-tuning does in practice.

The second consequence is that how firmly the prior holds depends on the anchor's weight (Section 12.3.3). When $\beta$ is large against the reward's range, no behavior's probability changes by more than a factor of $e^{(r_{\max} - r_{\min})/\beta}$: the optimum sharpens what the reference already does and cannot make a rare behavior common. When $\beta$ is small, Equation (12.7) says rarity offers little protection. DeepSeek's developers report training DeepSeek-R1-Zero with a KL coefficient of 0.001 over 10,400 steps, replacing the reference model with the latest policy every 400 steps (DeepSeek-AI et al., 2025). Read as the $\beta$ of the closed form, a reward gap of 1 then makes dominant any behavior whose reference probability exceeds $e^{-1000}$ (inference). The reading is loose, since the coefficient weights an estimate of the divergence added directly to the loss, not a per-token penalty in the reward as in PPO, against standardized advantages, but it places such training far from the regime where the prior holds. Resetting the reference loosens the anchor further: if each phase reached its optimum, tilting $\pi_0$ by $e^{r/\beta}$ and then tilting the result again would be a single tilt by $e^{2r/\beta}$. In that regime the closed form does not predict that training only sharpens.

One prediction holds at every $\beta$, and Section 12.7.4 derives and tests it: at the optimum, the probability of solving a training problem in $k$ attempts never falls below the reference policy's, for any $k$.

The third consequence is a limit. The posterior says which outputs the optimal policy produces, not which computation a network uses to produce them, and networks that agree on the training prompts can generalize differently elsewhere (Chapter 9). Whether training makes reward something the policy pursues, rather than a signal that strengthens whatever computation earned it, is a separate question, which Section 33.1 takes up.

12.5 Best-of-n: reweighting by sampling #

Training a policy is expensive, and a deployed system can be steered without it. Draw $n$ responses from the reference policy, score each with the reward, and return the best. This is best-of-n sampling, also called rejection sampling or reranking. It needs no gradient and no change to the model, only $n$ samples and $n$ scores per query. The question for this section is what distribution it samples from, and how far that distribution is from the reference.

12.5.1 Which distribution best-of-n samples #

Assume, as Beirami et al. (2025) do, finitely many responses with distinct rewards, and sort them by reward. Let $F(y)$ be the reference probability of a reward at most $r(y)$ and $F^-(y)$ the probability of a reward strictly below it. The best of $n$ independent draws is $y$ exactly when all $n$ draws have reward at most $r(y)$ and not all of them have reward below it, so

which is their form of the best-of-n distribution (Beirami et al., 2025). Write $u = F(y)$ and $\delta = F(y) - F^-(y) = \pi_0(y)$. For small $\delta$, the difference of powers $u^n - (u - \delta)^n$ is close to its first-order term $n u^{n-1} \delta$, so $\pi_n(y) \approx n\,F(y)^{n-1}\,\pi_0(y)$. The next term is smaller by a factor of about $n\delta/(2u)$, so the approximation needs $n\,\pi_0(y)$ to be small compared with $F(y)$, not only $\pi_0(y)$ small. Within that range best-of-n, too, reweights the reference policy, by a factor $n F^{n-1}$ that grows with the response's rank among rewards. The exponential tilt of Equation (12.4) depends on the reward's size; best-of-n depends only on its rank, which makes it indifferent to how the reward is scaled.

12.5.2 The divergence of best-of-n #

For rewards that vary continuously, with no response carrying any probability on its own, the first-order form is exact, and the KL divergence has a closed form that became folklore.

A language model's responses are finitely many, however, and each carries some probability of its own. Beirami et al. (2025) show that for this case the folklore formula is not the KL divergence in general; it is an upper bound:

The gap comes from the chance that the $n$ draws repeat a response. The formula is accurate when $n^2 \max_y \pi_0(y)$ is much smaller than one, so that even the likeliest response is rarely drawn twice. The gap can grow without bound when $n$ times the reference probability of the best response is large; for two equally likely responses, the true divergence never exceeds $\log 2$ while the formula grows without limit (Beirami et al., 2025).

12.5.3 How close to the tilt #

The budget derivation of Section 12.3 gives one comparison at once: at any KL, the tilt of Equation (12.4) earns at least as much expected reward as best-of-n, because it earns the most of any policy within that budget. The question is how much is lost. Gui and colleagues answer it for win rate, the probability that a policy's response is preferred to a fresh sample from the reference. They place best-of-n and trained alignment policies in a common class, every reweighting of the reference by a non-decreasing function of the reward, and show that within this class best-of-n is essentially optimal for the trade-off between win rate and KL (Gui et al., 2024). They also train a model to imitate the best-of-n distribution, so that one sample costs what $n$ did (Gui et al., 2024). In the 2020 summarization study, the authors report, without showing the data, that best-of-n and PPO policies with the same average reward were of similar quality to human labelers, though the PPO policies were farther from the supervised model in KL (Stiennon et al., 2020).

Figure 12.3 shows the two side by side. At $\beta = 0.05$ the optimum spends 4.76 nats and earns an expected reward of 0.82; best-of-324 spends the same divergence and earns 0.79, and the formula overstates its divergence slightly, at 4.78 nats. The shapes differ more than the numbers. Best-of-324 puts 22 percent of its probability on a single response below the top, and the best response appears in fewer than 1 percent of batches of 324 draws, while the optimum gives that response 8 percent. In this model the formula is close because the best responses are rare; the coin of Exercise 12.2 is the opposite case.

12.5.4 One axis of pressure, and its limits #

With its divergence in hand, best-of-n and reinforcement learning can be drawn on one axis, the KL from the reference policy, which is how the overoptimization measurements of Section 15.6, by OpenAI researchers on their own GPT-3-series models, are plotted (Gao et al., 2022). Those measurements use $d = \sqrt{\KL}$, because KL is a quadratic measure of distance (Gao et al., 2022). A developer report had observed reward rising roughly linearly in $\sqrt{\KL}$ during RLHF, and explained that the KL of a slightly changed policy grows with the square of the change while reward grows linearly with it (Bai et al., 2022b). The closed form gives the same law: near the reference, the tilt's reward gain is about $\sqrt{2\Var_{\pi_0}(r)\,\KL}$, with $\Var_{\pi_0}(r)$ the variance of the reward under $\pi_0$ (Exercise 12.3).

The axis has limits. Those best-of-n curves used the folklore formula (Gao et al., 2022), which Beirami et al. (2025) show is an upper bound, so they are drawn at or to the right of their true positions. And the same study found that reinforcement learning consumes far more KL than best-of-n, and concluded that KL "should not be used to compare the amount of optimization between different optimization algorithms" (Gao et al., 2022). The axis orders the policies one method produces; it does not equate pressure across methods.

12.6 Direct preference optimization #

The RLHF pipeline of Section 12.2 trains two networks and runs a sampling loop: a reward model fitted to comparisons, then a policy that generates responses throughout training so that PPO can score and update them. Both steps are expensive, and the second is delicate to tune. Direct preference optimization (DPO) removes the reward model and the sampling loop by running Equation (12.4) backward (Rafailov et al., 2023).

12.6.1 The idea: every policy implies a reward #

The closed form maps a reward to a policy. Solve it for the reward instead. Taking logarithms of Equation (12.4) and rearranging,

Read from right to left, this says that any policy, compared with the reference, implies a reward for which it is the optimum: responses it makes more likely than the reference did are the ones that reward favors. The intractable $Z(x)$ is still there, but it depends only on the prompt. Preference data compare two responses to the same prompt, and the Bradley-Terry model of Section 12.2.1 (taught in Section 13.2) turns a comparison into a probability through the difference of the two rewards: the probability that $y_w$ is preferred to $y_l$ is $\sigma\big(r(x, y_w) - r(x, y_l)\big)$, with $\sigma(z) = 1/(1 + e^{-z})$ the logistic function. In a difference, the prompt's constant cancels.

This is the gradient the paper gives, and its authors report that without the weighting the model can degenerate. The paper's title, "Your Language Model is Secretly a Reward Model", states the idea: the policy's own log-probability ratio is the reward model, fitted to the comparisons directly. In its experiments, DPO exceeded PPO-based RLHF at controlling sentiment and matched or improved response quality in summarization and single-turn dialogue, without sampling from the model during fine-tuning; the summarization and dialogue comparisons were judged by GPT-4 (Rafailov et al., 2023). DPO was derived in the bandit view of Section 12.2. A later preprint derives it in the token-level view as a form of inverse Q-learning, a method that recovers a value function from behavior, and argues that DPO therefore performs some credit assignment across the tokens of a response (Rafailov et al., 2024a).

12.6.2 What goes wrong #

Removing the reward model does not remove what the reward model stood for. The implicit reward is still a learned proxy, fitted to finite data under an assumed model of how people choose, and each assumption has been shown to matter. One analysis treats RLHF and DPO as special cases of one preference objective. It finds that when one response is always preferred to another, the Bradley-Terry fit needs an infinite reward gap, so the optimum gives the losing response zero probability whatever $\beta$ is: in the authors' words, "the strength of the KL-regularisation becomes weaker and weaker the more deterministic the preferences" (Azar et al., 2024). With finite data this becomes overfitting, and their bounded alternative avoids it; Section 22.4 works through the derivation. And direct alignment methods overoptimize too: their quality deteriorates across a wide range of KL budgets, often before a single epoch of the dataset is complete (Rafailov et al., 2024b).

The model of the human inside the loss is itself a choice. A method presented at ICML 2024 replaces Bradley-Terry with a value function from prospect theory, a model of how people weigh gains against losses, and learns from single responses labeled desirable or undesirable (Ethayarajh et al., 2024). Section 22.4 and Chapter 13 examine such choices. These methods are now parts of larger recipes. An open post-training recipe, reported by its developer in a preprint and released with its data and code, combines supervised fine-tuning, DPO, and the reinforcement learning with verifiable rewards of the next section (Lambert et al., 2024).

12.7 Verifiable rewards and reasoning models #

For some tasks no learned reward is needed. A mathematics problem with a known numerical answer, or a programming task with unit tests, can be graded by a program: compare the final answer, or run the tests. The developer report that named the approach calls it reinforcement learning with verifiable rewards (RLVR): the policy receives a reward only when its response is verifiably correct (Lambert et al., 2024). "Verifiable" here means checked by a program, not proved correct; whether the check is right is the question of Section 24.2. Loops of this kind trained DeepSeek-R1 (DeepSeek-AI et al., 2025) and, by its developers' account in a preprint, Kimi k1.5 (Kimi Team et al., 2025), reasoning models that write a long chain of reasoning before answering. This section teaches the loop end to end, because most of the evidence in Chapter 24 and Chapter 30 comes from it.

12.7.1 The loop #

The loop combines pieces already taught. Its distinctive move is the baseline: instead of training a value network, it compares each response with other responses to the same prompt.

Group Relative Policy Optimization (GRPO), the version of this loop that DeepSeek's reasoning models use, was introduced in a 2024 preprint by DeepSeek's developers as a variant of PPO that enhances mathematical reasoning while reducing PPO's memory use. It drops PPO's value model, a second network about the size of the policy, uses the average reward of the responses sampled for the same question as the baseline, and adds the KL divergence from the reference policy to the loss rather than to the reward (Shao et al., 2024a). In that paper the rewards still came from a learned reward model; rule-based rewards came with the reasoning models below.

The group mean is a baseline in the sense of Section 12.1.3 with one wrinkle: it includes the response's own reward. Write the mean as $\frac{1}{k} r_i + \frac{1}{k}\sum_{j \ne i} r_j$. The second part does not depend on response $i$, since the samples are independent, and drops out in expectation like any baseline; the first part scales response $i$'s own term by $1 - 1/k$. The centered estimate is therefore unbiased for $(k-1)/k$ times the gradient: the right direction, slightly shortened. Dividing by the group's standard deviation is a further choice, which rescales each prompt's update by how much its group disagrees, so prompts count unequally. The sources describing GRPO do not say whether the standard deviation divides by $k$ or $k - 1$; this book divides by $k$.

The last case is the important one. If a prompt is solved with probability $p$ by each sample, a group of $k$ independent samples is all right or all wrong with probability $p^k + (1 - p)^k$, and such a group contributes nothing but the KL term. The training signal comes from problems the model solves sometimes.

Figure 12.4 makes the trade-off visible. At a solve rate of 0.1 and groups of eight, a group carries signal with probability 0.57; in the most likely mixed group, one correct response among eight, the correct response gets advantage $\sqrt{7} \approx 2.65$ and each wrong one $-0.38$. Larger groups rescue hard prompts, at the price of more samples per prompt.

12.7.2 The systems #

The report on DeepSeek-R1, published in Nature and its developer's account of its own models, trained a first model, DeepSeek-R1-Zero, by reinforcement learning with GRPO alone, without supervised fine-tuning first. The developers report that reasoning abilities were incentivized without human-labeled reasoning trajectories, with behaviors such as self-reflection and verification emerging during training, and that R1-Zero's average pass@1 on the AIME 2024 mathematics competition, the fraction of problems a single sample solves, rose from 15.6 percent to 77.9 percent. Its rewards were rule-based checks of accuracy and format; the developers avoided neural reward models for reasoning tasks because they had observed them to be susceptible to reward hacking in large-scale reinforcement learning (DeepSeek-AI et al., 2025). The developers of Kimi k1.5 report, in a preprint, a framework without tree search, value functions, or rewards on each reasoning step (Kimi Team et al., 2025). The open recipe that named RLVR runs PPO on the objective of Equation (12.3) with the reward model replaced by a verification function (Lambert et al., 2024).

12.7.3 Rewarding the answer or the steps #

A verifier that checks only the final answer rewards outcomes; one that checks each step of the reasoning provides process supervision. The difference matters because a correct answer can follow from wrong reasoning. On grade-school word problems, a 2022 preprint by DeepMind researchers found that outcome-based supervision reached similar final-answer error with less labeling, but that correct reasoning steps required process-based supervision or a reward model that emulates it (Uesato et al., 2022). With reinforcement learning directly against final-answer correctness, at best 12.4 percent of correct answers rested on at least one wrong step, against 3.8 percent for the best process-based method (Uesato et al., 2022). On competition mathematics, in a 2023 study by OpenAI with reward models fine-tuned from its own GPT-4, the process-supervised reward model, choosing the best of 1,860 sampled solutions per problem, solved 78.2 percent of a representative subset of 500 MATH test problems, against 72.4 percent for the outcome-supervised one (Lightman et al., 2023). The authors caution that the two reward models' training sets are not directly comparable; in a smaller-scale comparison in which only the form of supervision differed, process supervision again came out ahead (Lightman et al., 2023). The authors released 800,000 step-level human labels (Lightman et al., 2023). Both comparisons are between reward models used for selection, not between training runs. The same axis returns for the monitorability of reasoning (Section 40.6) and for designs that limit what a training signal rewards (Chapter 28).

12.7.4 What the loop does to a policy #

The measurement that asks what this loop adds is pass@k: the probability that at least one of $k$ samples solves a problem, averaged over problems. Pass@1 is the ordinary success rate; pass@k at large $k$ measures coverage, which problems a model solves within a budget of many tries. If training only sharpens the policy toward solutions it already reaches, coverage with unlimited tries cannot grow, since a problem solved with any probability is solved eventually. At a finite $k$ it can: sharpening raises a rare solution's probability, and with it pass@k, so a rise at large $k$ does not by itself show that training added an ability. The closed form makes a sharper prediction for the problems the policy is trained on, at every $\beta$.

A run that resets its reference does not escape the result if each phase reaches its optimum, since a tilt of a tilt is again a tilt (Section 12.4.2). The measurements below should be read against it.

DeepSeek's developers had already observed, in their own model, that reinforcement learning improved the majority-vote accuracy of $k$ samples but not pass@k, which they read as "boosting the correct response from TopK rather than the enhancement of fundamental capabilities" (Shao et al., 2024a). A 2025 preprint by Yue and colleagues made the test systematic. Models trained with verifiable rewards outperformed their base models, the models training started from, at small $k$, but base models consistently surpassed them as $k$ grew, across benchmarks and model families. Analyses of which problems each model solved, and of how likely the base model found the trained model's reasoning, indicated that the trained model's reasoning paths already existed in the base model's distribution (Yue et al., 2025).

The sharpest measurement is on the training problems themselves. The authors trained Qwen2.5-7B with GRPO on 2,000 mathematics problems, with the KL term removed. On those problems, pass@1 rose from 9.9 percent for the base model to 42.5 percent after 450 steps, while pass@256 fell from 67.2 to 64.3 percent (Yue et al., 2025). By the derivation above, no optimum does that on its own training prompts, with a penalty or without one. The crossover is therefore evidence about how far trained policies sit from the optimum they aim at, not about the optimum's prior (inference).

What does predict narrowing is the loop itself. A correct response gets a direct push only in the rounds in which it is sampled, and a problem the policy rarely solves yields groups that are all wrong, which carry no signal (Section 12.7.1). Meanwhile the network's parameters are shared across problems, so the updates that sharpen the policy where it often succeeds also move its distribution where it rarely does, which the optimum, computed prompt by prompt, leaves out. A run of a few hundred steps can therefore sharpen what it samples often and lose what it samples rarely, where the optimum would have kept both, so that the prior matters through what training samples rather than through the anchor (inference). Section 24.4 reports the measurements of this collapse, and two papers that report the reverse under other conditions, longer training from an already distilled model with reference resets in one and a pass@k that credits only reasoning another model judges correct in the other (Liu et al., 2025; Wen et al., 2026). The evidence does not yet decide what such training adds at frontier scale.

12.7.5 Why the grader matters for alignment #

In this loop the grader is the specification. Whatever a program rewards is what the policy is trained toward, and a flaw that one sampled response happens to find receives a large positive advantage, larger the rarer the success in its group, and spreads; Section 24.4 derives how fast. Chapter 24 treats graders as test suites, and Chapter 30 reports what the loop has produced in practice.

12.8 What the anchor does not guarantee #

The KL penalty bounds how far the policy moves from the reference. It does not bound how much of the true objective is lost when the reward is wrong. A learned reward can be accurate where it was tested and wrong where the optimized policy goes. This section shows that gap in the closed form itself, states two results that bound it, and hands the general theory to Chapter 15.

12.8.1 The idea: the anchor charges for distance, not for direction #

Return to Figure 12.2 and suppose its reward $r$ was a proxy for an intended reward $r^\ast$, the gold reward of Section 15.4. (The star on $\pi^\ast$ marks the optimum for the proxy $r$, not for $r^\ast$.) Let the proxy agree with $r^\ast$ where the reference policy puts most of its probability and overrate the rare responses at the top of its own scale, as a reward model fitted to the reference policy's samples might. Nothing in Equation (12.4) knows about $r^\ast$: the optimum follows $r$, and the penalty charges the same price per nat whether the distance leads toward responses that are better or toward responses that only score better.

Figure 12.5 shows the result. As $\kappa$ grows and $\beta$ falls, the expected proxy reward rises steadily toward 1. The expected true reward rises from the reference policy's 0.31, peaks at 0.55 when the divergence is 3.2 nats, and falls; at $\beta = 0.03$ the optimum earns a proxy reward of 0.94 and a true reward of 0.46. This is the overoptimization curve of Section 15.6, produced from Equation (12.4) alone, with no training dynamics and no noise. The error sits where the reference policy is rare, which is where its samples gave the proxy no reason to be right and where the optimum goes when the anchor weakens.

12.8.2 What has been proved #

Two results bound the gap directly, each under its own assumptions.

When the true reward is linear in known features, the reward estimated from comparisons by maximum likelihood converges under the Bradley-Terry model and its extension to rankings, the Plackett-Luce model. But a policy trained against that estimate can fail, while a pessimistic estimate, which discounts responses the comparison data rarely cover, gives better policies under assumptions about how well the data cover the responses that matter (Zhu et al., 2023).

A low test error does not settle the matter either. Call a policy's regret the shortfall of its true return below the best achievable, normalized to lie between 0 and 1. A sufficiently low expected test error of a reward model guarantees low worst-case regret for the policy optimized against it. But for any fixed level of test error there are realistic data distributions with high regret, and policy regularization of the kind RLHF uses does not remove the problem (Fluri et al., 2025). The authors trace the mismatch to the distribution shift that optimization causes, the mechanism of Figure 12.5: test error is measured where the reference policy goes, and the optimized policy goes elsewhere.

Two questions remain for Chapter 15. Whether the penalty protects the true objective at all depends on the tail of the reward's error, how fast the probability of very large errors falls off, and on whether the error is independent of the true reward; Section 15.5 states the result and its conditions (Kwa et al., 2024). And whether KL is the right penalty is itself in question. The $\chi^2$ divergence, $\sum_y \pi(y)^2/\pi_0(y) - 1$, charges far more than KL for probability moved onto responses the reference makes rare. Penalties built on it have been proposed for direct preference optimization, combined there with KL (Huang et al., 2025a), and, measured over the states and actions a policy visits, for reinforcement learning in general (Laidlaw et al., 2025); Section 15.4 teaches both.