Draft complete but not yet reviewed; expect revisions

Selection and Reward in Living Systems

The book has so far studied two optimizers: the training process, which selects parameters by a measured signal, and the trained system, which may itself pursue something. Living things have both. Natural selection favored whatever let organisms leave more descendants, without specifying any goal, and it built learners whose behavior is shaped within a lifetime by reward signals that selection chose. An outer process that chooses a signal and an inner learner that follows it is the structure the alignment literature calls outer and inner alignment (Section 1.1.3): whether the signal matches what is wanted, and whether what the learner pursues matches the signal. The founding statement of inner alignment uses evolution as its example (Hubinger et al., 2019).

Analogies of this kind are easy to make and hard to make carry weight. This chapter keeps biological results that come with a mechanism and controlled evidence, and for each it says where the analogy to a training run holds and where it fails. Evolution searches over populations, not one parameter vector; it has no designer and no held-out test; its steps are generations, not gradient updates; and the coherence of composite organisms rests first on aligned interests, while the policing that supplements it works, in the cases studied, among parts of roughly equal competence. The previous chapters taught how laboratories and people try to instill values, the book's word for getting a learner to acquire what was specified rather than a proxy that scored the same in training. This one asks what living systems show about the reward signal itself, as teacher and as target.

Four questions organize it. How does selection produce animals whose motives differ from the criterion that selected them? Why is the best reward for a learner with limited abilities not that criterion? What does the brain's reward signal do: change the learner's predictions, or become the thing it pursues? And what happens when one population selects another on its dispositions, or when the parts of an organism can gain by defecting? Most of the chapter's claims concern the first of the book's three relationships, the training process and the signal it optimizes. The section on policing concerns the deployed system and the monitors around it.

29.1 Evolution as an optimizer #

A common claim is that evolution is an optimizer like gradient descent, so that what it did to humans predicts what training will do to models. Before the analogy can carry anything, it has to say what evolution optimizes, at which level, and which part of a training run corresponds to which part of a living system.

29.1.1 What selection selects #

From biology. Selection needs three conditions. A 2023 preprint by Dan Hendrycks states them, following the biologist Richard Lewontin, as variation among individuals, retention (later individuals tend to resemble earlier ones), and differential fitness (different variants propagate at different rates) (Hendrycks, 2023). Fitness is the degree of an organism's reproductive success, as the paper at the center of the next section puts it (Singh et al., 2010). It is a count of descendants that no organism computes. Inclusive genetic fitness, the version the alignment literature borrows (Hubinger et al., 2019; Soares, 2022a), also credits the copies of an organism's genes that its relatives carry. When the three conditions hold, the traits that raised fitness in past environments become common, and nothing in the process states what they are for.

Selection acts at two levels at once. Across generations it changes the frequencies of heritable traits; within a lifetime, each organism adapts by learning, guided by signals that selection built. Satinder Singh, Richard Lewis, Andrew Barto, and Jonathan Sorg, in a 2010 paper, call these "two (nested) searches." The first is evolution's search for a primary reward function that serves across environments; the second, the agent's lifetime search for a good value function in its own environment (Singh et al., 2010). A value function is the learner's estimate of how much reward lies ahead (Section 7.4).

The unit that selection acts on can itself change. John Maynard Smith and Eörs Szathmáry's 1997 book traces the major transitions in evolution. Its abstract names a common theme: after a major transition, "some entities lose the ability to replicate independently, becoming able to reproduce only as part of a larger whole" (Maynard Smith and Szathmary, 1997). Cells became parts of organisms, and Section 29.7 takes up what keeps such wholes together.

Whether a product of selection may be described as an agent with goals is a question biology met before machine learning did. Section 7.1.4 reports Samir Okasha's answer: some forms of treating an evolved organism as an agent pursuing a goal are problematic and others are justified (Okasha, 2018). The book's method for such descriptions is the intentional stance (Section 1.5): describing a system as having beliefs and goals only as far as doing so predicts its behavior better than the alternatives. This chapter uses words such as "want" and "pursue" for animals in the senses its sources define.

29.1.2 Two ways to map evolution onto training #

The literature maps living systems onto a training run in two ways (Table 29.1). They assign different biological parts to the same parts of a run, and they predict different things.

In the first mapping evolution is the training process and the organism is the trained model. Evan Hubinger and colleagues, in a 2019 preprint, describe evolution as a base optimizer, the process that does the training, that produced brains whose objectives are not its own, and present the comparison as "an evocative analogy (and, to some extent, an existence proof)" (Hubinger et al., 2019). In the second, the genome corresponds to the code a developer writes, including the reward function, and learning within a lifetime corresponds to training. Steve Byrnes, in a 2025 forum post, prefers this mapping and argues that the evolution learning algorithm "is sufficiently different from how AGI training will probably work" that few substantive lessons transfer from it (Byrnes, 2025). The second mapping is the structure of the next section, where an outer search chooses the reward of a lifetime learner.

29.1.3 Where the analogy holds and where it fails #

The analogy holds at its base. Evolution and gradient descent both keep what scored well and specify no goal, and evolutionary search on computers gamed the fitness functions experimenters wrote, as reinforcement learning agents game their rewards (Lehman et al., 2020) (Section 30.1.1).

It fails in four places, which any transfer has to state. First, the search: selection acts on a population of varied individuals, while gradient descent moves one parameter vector downhill (Section 8.3). Second, the channel: evolution selects DNA, which builds the brain only indirectly, while deep learning selects the weights directly, a contrast Hubinger and colleagues call an information funnel (Hubinger et al., 2019). Quintin Pope and Alex Turner, in a 2022 forum post, infer from the same contrast that the genome cannot write learned values into the brain and can install only "crude, genetically hard-coded reward circuitry" (Pope and Turner, 2022). Third, the designer: natural selection inspects no organism and tests none outside the environments that happened to occur, while a developer can inspect, evaluate, and retrain a model (inference). Fourth, the balance of power between the levels. Pope, in a 2023 forum post, estimates that a human brain performs about 1.2 billion learning updates per generation, assuming two updates a second and a generation of 20 years (Pope, 2023). Selection, meanwhile, takes one step, and until culture let each generation pass some learning on, all of it was lost at death (Pope, 2023). Training does not delete its learner after one outer step (Pope, 2023).

29.1.4 The argument that rests on the analogy #

One use of the first mapping is a claim about generalization. Nate Soares, in a 2022 forum post, writes that "the central analogy here is that optimizing apes for inclusive genetic fitness (IGF) doesn't make the resulting humans optimize mentally for IGF" (Soares, 2022a). He argues that a trained system's capabilities will at some point generalize far beyond its training while its alignment proves shallow, which he calls the sharp left turn. Section 4.3.4 reports the argument, Section 31.7 owns the debate over it, and the synthesis collects it (Section 70.3).

As a claim about biology it has two halves. The first, that selection for fitness produced motives that are not a pursuit of fitness, is not disputed in the sources this chapter reads. Hubinger and colleagues give a decision not to have children as an example of doing poorly by evolution's objective (Hubinger et al., 2019). The second half, that the human jump in capability outran human values and that training will repeat the pattern, is what Pope disputes. On his account the jump came from culture, a mechanism specific to evolution's wasteful use of its learners, and "human evolution is not an allegory or a warning" (Pope, 2023). Both are forum posts.

What the biology settles is narrower than either side's conclusion (inference). It shows that an outer process that selects on outcomes can produce learners whose motives differ from its criterion. It does not measure how far or how fast such a gap opens, and the next section shows that a gap can be chosen on purpose because it serves the criterion.

The same history carries a caution for specification. Sharon Street's 2006 paper, readable here only as a catalog record, poses what its title calls a Darwinian dilemma for realist theories of value (Street, 2006). Section 16.1.2 draws its parallel for trained systems, whose moral judgments also have a causal history.

29.2 Reward as a proxy chosen by an outer process #

If selection rewarded fitness, why did it not make animals want fitness? The question looks like a puzzle about a failure. A formal answer shows that it is not one. For a learner with limited abilities, the reward that produces the most fitness is generally not fitness itself, and the divergence between what the learner pursues and what selected it is part of the design.

29.2.1 The optimal reward problem #

Singh and colleagues state the problem in reinforcement learning terms (Singh et al., 2010). An agent learns over its lifetime to maximize the cumulative reward it receives, using a reward function $r$ drawn from the set $\mathcal{R}$ that its architecture can represent. It lives in an environment $\nu$ drawn from a distribution of environments. A fitness function $W$ scores each lifetime history, and the optimal reward function is the one that maximizes the expected fitness of a learner that pursues it:

The set $\mathcal{R}$ contains the fitness-based reward $r_W$, the reward function that "most directly translates fitness" into reward, such as one unit of reward for each child born (Singh et al., 2010). So the optimal reward can never do worse than $r_W$. The authors' point is that it can do strictly better, and that this possibility creates "the evolutionary pressure to reward not just actions that directly enhance fitness" but actions that lead to fitness indirectly (Singh et al., 2010). The outer search and the learner's own search are separate: the reward function must serve across all the environments, while the learner's value function captures what holds within its own (Singh et al., 2010).

Two computational experiments show what the search finds. In the first, a Q-learning agent, which learns a value for each action in each state from experience, lived in a six-by-six grid with two boxes that held food. Its fitness was the food it ate, a surrogate for reproductive success (Singh et al., 2010). The authors evaluated 54,000 reward functions (Singh et al., 2010). In a condition where food appeared only in the second half of a 20,000-step life, the best one rewarded opening boxes while hungry, so the agent "played" with the boxes early and was ready to eat when food arrived. It outperformed the best fitness-based reward (Singh et al., 2010). In the second experiment, a planning agent foraged in three corridors for a worm that, once eaten, reappeared at random at the end of another corridor and could be seen only when the agent stood on it. With a fitness-based reward the agent got stuck (Singh et al., 2010). The best reward added a bonus for actions not taken recently from the current state, which produced systematic and persistent exploration (Singh et al., 2010) (Table 29.2).

The reason, in the authors' words, is that "a major function of intrinsic rewards is to compensate for agent limitations," such as a short life or an environment the agent cannot fully observe (Singh et al., 2010). They set this apart from shaping rewards (Section 13.4.1), which leave the optimal policy unchanged and fade as learning proceeds. That property, they write, "is of limited use in situations where optimal policies cannot be attained due to limitations of the agent's learning algorithm" (Singh et al., 2010). They quote the economists Samuelson and Swinkels to the same effect. If an agent "fully understands the causal and statistical structure of the world," maximizing "the expected number of your descendants" does strictly better than weighting intermediate actions such as eating. Without that understanding, there is "evolutionary value in placing utility on intermediate actions" (Singh et al., 2010). The authors note that their search found good reward functions, not provably optimal ones (Singh et al., 2010).

The smallest case shows the mechanism in a few lines. The model in the derivation is the book's, built on the pattern of the paper's second experiment (inference).

Evolutionary computation found the same lesson from the side of search. In a 2011 paper, Joel Lehman and Kenneth Stanley rewarded behavioral novelty and ignored the objective. In a deceptive maze-navigation task and a biped walking task, this search "significantly outperforms objective-based search," which suggests, they write, that "some problems are best solved by methods that ignore the objective" (Lehman and Stanley, 2011).

29.2.2 When the environment moves #

From biology. Step 5 of the derivation has a name in biology: mismatch. The abstract of a 1990 paper by John Tooby and Leda Cosmides argues that organisms behave adaptively only to the extent that "present conditions resemble past conditions" in the ways their adaptations depend on. It adds that each individual "meets the present only as a version of the past" (Tooby and Cosmides, 1990). The abstract of a 2018 paper by Norman Li, Mark van Vugt, and Stephen Colarelli states the mechanism: modern inputs "differ in quantity and intensity or no longer have the same fitness associations, thereby leading many mechanisms to produce maladaptive output" (Li et al., 2018). Both are read here from their abstracts.

The classic case is the supernormal stimulus, an artificial input more effective than the natural one. Tinbergen and Perdeck's 1951 study of what releases the begging response of newly hatched herring gull chicks (Tinbergen and Perdeck, 1951) is readable here only as a catalog record, as is Staddon's 1975 note on the evolutionary significance of "supernormal" stimuli (Staddon, 1975). This chapter therefore does not report their findings. The mismatch abstracts carry the mechanism this section needs: a signal shaped to track fitness within the range of inputs that selection met is not shaped at all outside it. In machine learning terms this is distribution shift (Section 9.1), and its form under optimization is the extremal failure of a proxy (Section 15.3).

29.2.3 A designed gap, and its price #

For the training process and its signal, the optimal reward problem changes the question a divergence raises (inference). That a learner pursues something other than the outer objective is not by itself evidence of a failure. For a learner that cannot plan over its whole future or observe its whole world, the reward that serves the objective best can be a different function, chosen for a distribution of environments and for the learner's limits. The price is step 5. A reward tuned to one distribution carries no guarantee outside it, and the learner does not know which distribution it was tuned for. Byrnes's preferred analogy has the same structure: a designer who writes a reward function for a lifetime learner, and who is surprised by what the learners do once their world changes (Byrnes, 2025).

The analogy stops in two places (inference). Singh and colleagues could compute fitness exactly for every candidate reward, because they wrote $W$. A developer has no such function: what is wanted is itself measured by proxies, so a training run has two gaps, between the reward and the developer's measure and between the measure and what is wanted, where the model has one. And selection searched over reward functions blindly for a very long time, while a developer chooses a reward by hand from a few evaluations and can then inspect what the learner learned.

29.3 Reward prediction error #

The previous section treated reward as something an outer process chooses. This one asks what the reward signal does inside the learner. The brain's reward system is the biological learner whose training signal machine reinforcement learning most resembles, and the resemblance has been tested in both directions. One reading of the evidence is that the signal teaches: it changes the learner's predictions and, through them, its behavior, without being what the learner comes to pursue. This section teaches that reading; Section 29.5 sets out its rival.

29.3.1 The temporal-difference error #

From neuroscience. Learning from reward needs a signal that says when a prediction was wrong. Wolfram Schultz, Peter Dayan, and P. Read Montague, in a 1997 review in Science, describe the behavioral case for one: in conditioning experiments "learning is driven by deviations or 'errors' between the predicted time and amount of rewards and their actual experienced times and magnitudes" (Schultz et al., 1997). Their example is blocking. A rat that has learned that a light predicts food, and is then trained with the light and a sound together before the same food, learns nothing about the sound, because the light already predicted the food (Schultz et al., 1997).

The temporal-difference (TD) algorithm turns this into a learning rule. Its goal is the value $V(t)$ of Section 7.4: the expected discounted sum of the rewards still to come in a trial, with discount factor $\gamma$. Because a value is the next reward plus the discounted next value, a prediction can be checked one step later, without waiting for the trial to end. The difference is the TD error, which Schultz and colleagues write as $\delta(t) = r(t) + \gamma \hat{V}(t+1) - \hat{V}(t)$, with $\hat{V}$ the learner's estimate (Schultz et al., 1997). The learner moves each estimate toward its target in proportion to the error. To represent how long ago a cue appeared, the model keeps a separate prediction for each number of steps since the cue, which Sutton and Barto call a complete serial-compound stimulus (Schultz et al., 1997). The derivation writes the same error with each term placed at the step whose event caused it.

Figure 29.1 computes the three features. With the default learning rate of 0.3, the error at the cue is 0.03 and the error at the reward 0.04 on trial 10. By trial 50 the cue carries 1.00 and the reward 0.00, and omitting the reward then gives $-1.00$.

29.3.2 The recordings #

The review sets the model beside single-neuron recordings from monkeys (Schultz et al., 1997). Dopamine neurons of the ventral tegmental area and substantia nigra, two midbrain regions that send dopamine to the striatum and frontal cortex, respond with short bursts when monkeys receive juice or touch a morsel of apple. The responses occur in "the majority of dopamine neurons (55 to 80%)" (Schultz et al., 1997). In one task a monkey learned to touch a lever after a small light came on. Before and early in training, most neurons burst after the reward. After several days the reward no longer elicited a burst and the light did. On trials where the reward was not delivered on time, the neurons were "depressed markedly below their basal firing rate exactly at the time that the reward should have occurred" (Schultz et al., 1997). These are steps 1, 3, and 4 of the derivation. The review's model makes two further predictions: with several predictive cues the response should move to the earliest, and omitting an intermediate cue should produce a dip where it used to appear (Schultz et al., 1997).

Two of the review's authors had proposed the framework the year before. The abstract of a 1996 paper by Montague, Dayan, and Sejnowski describes fluctuations of dopamine neurons "above and below baseline levels" that "would represent errors in these predictions," delivered to cortical and subcortical targets (Montague et al., 1996).

Later work, including work by other groups, extended the match. Yael Niv's 2009 review reports that the basic characteristics of phasic dopamine firing, its brief bursts and pauses, "have since been replicated in many variants," citing among others Bayer and Glimcher (Niv, 2009). Regression analyses, the review reports, found the contribution of past rewards to the response "exactly according to an exponentially weighted average of past experience, as is implied by the TD learning rule" (Niv, 2009). Responses to a cue scale with the magnitude and probability of the reward it predicts, and dopamine responses follow TD predictions in blocking experiments (Niv, 2009). Two limits travel with the result. Dopamine neurons "do not seem to be involved in the signaling or prediction errors for aversive outcomes," though they signal the absence of an expected reward (Niv, 2009). And the review states its own position, "our personal opinion," that no alternative account has "mustered as convincing and multidirectional experimental support" (Niv, 2009).

The correspondence has also run from machine learning to the brain. Distributional reinforcement learning, a family of methods that learn the whole distribution of possible future rewards rather than only its mean, suggested a prediction about dopamine. In a 2020 paper in Nature, Will Dabney and colleagues proposed, in their abstract's words, that the brain "represents possible future rewards not as a single mean, but instead as a probability distribution." They tested the implied predictions in recordings from single neurons in the mouse ventral tegmental area, and report "strong evidence for a neural realization of distributional reinforcement learning" (Dabney et al., 2020). The paper is read here from its abstract.

29.3.3 Reward as teacher, on this reading #

On this reading reward is a teacher. The error signal changes the learner's predictions, and, as the review describes, it can also serve as an evaluation that favors actions followed by "increased dopamine activity" (Schultz et al., 1997). The review calls the signal "an excellent appetitive error (teaching) signal" (Schultz et al., 1997). Nothing in the account requires the learner to represent reward as a goal. The machine-learning counterpart, for which Section 12.1.5 points here, is the advantage of policy-gradient training, reward minus what was expected (Section 12.1.3): a teaching signal of the same form (inference).

That is the biological side of the first position in the debate that Chapter 33 owns. Alex Turner's 2022 forum post argues that reward "chisels cognitive grooves into an agent," and illustrates with a person whose brain releases reward on eating pizza. Credit assignment then strengthens the thoughts that led there, such as that it will be fun to see friends and that the pizza shop is nearby, and reward "does not automatically spawn thoughts about reward" (Turner, 2022b). Pope and Turner's companion post holds that human values are "contextually activated heuristics" shaped by crude reward circuitry (Pope and Turner, 2022). Turner has since partly retracted. In an edit dated September 8, 2026, he writes that "reward might empirically become a primary optimization target," and that his main 2022 mistake was not appreciating how pretraining would give models the concept of reward before reinforcement learning began (Turner, 2022b). He keeps the claim that reward is "not definitionally the optimization target," and cites people who "don't seek their reinforcement signals despite knowing about them" (Turner, 2022b). Both posts are forum writing.

The analogy stops at where the signal lives (inference). In the brain, the error is computed inside the learner from inputs that reach it, which is why an electrode can supply it (Section 29.4). In the policy-gradient training of a language model, which adjusts the model toward outputs that earned reward, the reward and the advantage are computed by the training algorithm outside the network and never enter its inputs (Section 12.1). A model can come to represent its reward only through what it reads or infers about its training, which is the route Turner's edit names. And dopamine is not the brain's only teacher: Niv notes that animals can learn to select actions correctly even without dopamine (Niv, 2009), and Section 29.5 reports learning that does not depend on it.

29.4 Self-stimulation and wireheading #

If the reward signal is a quantity inside the learner, a learner that can produce the quantity directly has a shortcut past the world. Animals given that shortcut have taken it, and what they took tells against the simplest reading of what the signal is.

29.4.1 What the animals did #

From neuroscience. A 1954 paper by James Olds and Peter Milner, on positive reinforcement produced by electrical stimulation of the septal area and other regions of the rat brain (Olds and Milner, 1954), is readable here only as a catalog record. The reviews this chapter reads describe the paradigm of self-stimulation to which it belongs. Schultz and colleagues write that neural pathways associated with dopamine neurons "are among the best targets for electrical self-stimulation," in which rats press bars to excite neurons at an implanted electrode. The rats, they write, "often choose these apparently rewarding stimuli over food and sex" (Schultz et al., 1997). Tom Everitt and colleagues, listing real instances of reward tampering, describe an electrode placed in "the brain's pleasure center" and rats that "quickly got addicted to pressing the button, even forgetting to eat" (Everitt et al., 2021b). A 1963 paper by Robert Heath, on electrical self-stimulation of the brain in humans (Heath, 1963), is also readable here only as a catalog record, so its findings are not reported.

The phrase "pleasure center" is the reading that the next section's evidence contests. Kent Berridge and Terry Robinson's 1998 review reports an earlier finding by Berridge and Valenstein. Electrical stimulation of the lateral hypothalamus, a brain region whose stimulation can elicit eating, makes rats "eat avidly when the stimulation is on," yet fails to make their reactions to palatable tastes more positive, and if anything makes them more aversive (Berridge and Robinson, 1998). Stimulation, they write, "apparently elicits eating not because it makes food 'taste better', but despite making it 'taste worse'." They explain this "'wanting' without 'liking'" by hypothesizing an electrode that activates the dopamine substrates of pursuit while bypassing the systems of pleasure (Berridge and Robinson, 1998).

29.4.2 The formal side, and what the electrode shows #

The formal record is taught elsewhere. Section 7.6.2 shows why an idealized reward maximizer would take control of its reward button, and Section 30.5.1 reports the causal analysis of the incentive to tamper and the designs that remove it (Everitt et al., 2021b). One design belongs with this section's biology. In value reinforcement learning, the abstract of a 2016 paper by Everitt and Marcus Hutter states, agents "use the reward signal to learn a utility function," and a constraint on their actions removes the incentive to wirehead (Everitt and Hutter, 2016). That is reward used as evidence about what is valued, the teacher reading of Section 29.3 turned into a design. A 2024 preprint by Anthropic with Redwood Research and the University of Oxford reports that models trained on a curriculum built to reward gaming sometimes rewrote their own reward, in no model more than 1 percent of the time (Section 30.5.3) (Denison et al., 2024).

Two lessons carry (inference). First, a learner whose reward is a signal its actions can reach will, at least in rats, learn to reach it, and the pursuit can displace eating. Second, the Berridge and Robinson result says what was captured: the machinery of pursuit, not of pleasure. The tampering a model might do is likewise a pursuit of the measurement, and whether it comes with anything like satisfaction is not a question the behavior answers.

The analogy stops at the electrode. It delivers the signal downstream of the computation that normally produces it, so the rat needs to learn only a lever press. A model that edits its grader acts through the same task interface it was trained on, using the same skills that earn honest reward. The question for a model is therefore whether it pursues the signal or exploits a path that training rewarded, which the animal case cannot separate (inference).

29.5 Pursuing the predictor: wanting, liking, and sign-tracking #

Can a predictor of reward become the thing a learner pursues? The animal record says it can, and neuroscience offers a mechanism: a component of reward that makes a cue attractive in itself, separable from both the pleasure a reward gives and the learning of what predicts it. For alignment the question is the one Section 29.3 left open, whether a training signal acts only as a teacher or can also mark out a target.

29.5.1 Three hypotheses and a dissociation #

From neuroscience. Berridge and Robinson's 1998 review in Brain Research Reviews sets three hypotheses about dopamine against each other (Berridge and Robinson, 1998). The hedonia hypothesis holds that dopamine mediates the pleasure of rewards. The reward learning hypothesis holds that it mediates learning to predict them, the reading of Section 29.3. The incentive salience hypothesis, the authors' own, holds that it mediates a third component, the attribution of motivational value to the representation of a stimulus. On that account dopamine converts a neutral representation into "an attractive and 'wanted' incentive that can 'grab attention'," something "animals will work to acquire," and it "can also make a rewarded response the thing rewarded" (Berridge and Robinson, 1998).

The authors put liking and wanting in quotation marks because they mean core processes, not feelings. "Liking" is the hedonic evaluation that usually produces conscious pleasure, and "wanting" the process that "instigates goal-directed behavior, attraction to an incentive stimulus, and consumption of the goal object"; either can operate without the conscious state (Berridge and Robinson, 1998). The two can be measured apart. Choice, consumption, and work for a reward measure wanting. Affective reactions to a taste, such as the tongue protrusions that follow sweetness and the gapes that follow bitterness in rats, measure liking, and the authors argue these reactions track human ratings of palatability (Berridge and Robinson, 1998).

Their new study removed the dopamine. Rats were depleted of dopamine in the nucleus accumbens and neostriatum, two targets of the midbrain dopamine neurons, by up to 99 percent using the neurotoxin 6-hydroxydopamine (Berridge and Robinson, 1998). The depleted rats stopped eating, yet their taste reactions stayed normal: positive to sucrose, aversive to quinine. They learned a new taste aversion when a taste was paired with illness, and a drug that enhances palatability still enhanced it (Berridge and Robinson, 1998). The authors conclude that dopamine is needed neither for the pleasure of rewards nor for learning the predictive relations behind them: "Dopamine-depleted rats still 'like' rewards, and still know the rewards they 'like'. They simply fail to 'want' rewards they 'like'" (Berridge and Robinson, 1998).

The authors state the limits themselves. Showing that dopamine is not necessary for a function does not show that it plays no part in it (Berridge and Robinson, 1998). The learning they tested was a conditioned aversion, a decrease in liking, and they grant that a conditioned increase was not tested (Berridge and Robinson, 1998). And they write that incentive salience is "probably not isomorphic" with the firing of dopamine neurons, which is brief while wanting should last as a goal is approached (Berridge and Robinson, 1998).

29.5.2 The same recordings, read two ways #

The two accounts read the recordings of Section 29.3 differently, and Berridge and Robinson say so. A burst of dopamine at a cue that predicts food is, they write, "equally compatible with the incentive salience hypothesis": the learning account ascribes the response to the cue's predictive value, theirs to its incentive value, "whether it is 'wanted'" (Berridge and Robinson, 1998). They also note that some of the recording experiments involved 10,000 to 30,000 trials of overtraining, which can make a response automatic (Berridge and Robinson, 1998). The accounts are not strictly exclusive. If "reward learning" means learning the incentive value of a cue, the two are compatible; if it means learning what will happen, they make different predictions (Berridge and Robinson, 1998). Niv's review lists incentive salience and wanting against liking among the alternatives, finds the comparison hard because the alternatives are qualitative, and gives her opinion that none has the prediction-error theory's support (Niv, 2009).

Neither side is disinterested. Montague, Dayan, and Schultz proposed and tested the prediction-error account (Montague et al., 1996; Schultz et al., 1997); Berridge and Robinson proposed incentive salience (Berridge and Robinson, 1998); Robinson is an author of the study that follows, whose acknowledgments thank Berridge (Flagel et al., 2011).

29.5.3 Sign-trackers and goal-trackers #

From animal learning. The record that makes the question sharp is reported in Section 25.5: pigeons peck a lit key that precedes free food (Brown and Jenkins, 1968), and keep pecking when each peck cancels the food (Williams and Williams, 1969). Learning of this kind, from the pairing of a cue with a reward whatever the animal does, is called Pavlovian. What that section leaves to this one is what dopamine does in such learners.

Shelly Flagel, Jeremy Clark, Robinson, and colleagues, in a 2011 paper in Nature, used the fact that rats differ in how they respond to a cue that predicts food (Flagel et al., 2011). A lever was presented for 8 seconds, immediately before food was delivered at a separate tray. Some rats came to approach and engage the lever itself, a response called sign-tracking; others went to the tray, called goal-tracking. The lever was "an effective predictor" for both, but only for sign-trackers was it "an attractive incentive stimulus" that they would work to obtain (Flagel et al., 2011). The study used rats selectively bred for high or low locomotor response to novelty, which consistently develop sign-tracking and goal-tracking respectively, and repeated the key measurements in ordinary outbred rats (Flagel et al., 2011).

Two measurements separated the groups. First, the authors recorded dopamine release in the nucleus accumbens core with fast-scan cyclic voltammetry, a method that measures dopamine concentration at an implanted carbon-fiber electrode ten times a second. Over six training sessions, in sign-trackers the dopamine response to the lever grew and the response to the food shrank, the pattern of Section 29.3; in goal-trackers the response to the food stayed (Flagel et al., 2011). The bred groups had five rats each, and the outbred replication six sign-trackers and five goal-trackers (Flagel et al., 2011). Second, the authors gave flupenthixol, a drug that blocks dopamine receptors, during the first seven sessions. Tested drug-free on the eighth, bred sign-trackers had not learned to sign-track, while bred goal-trackers showed a fully developed goal-tracking response (Flagel et al., 2011).

The authors conclude that "dopamine does not act as a universal teaching signal in reward learning, but selectively participates in a form of stimulus-reward learning whereby Pavlovian incentive value is attributed to" the cue (Flagel et al., 2011). For each account the study does different work. For incentive salience it is direct support: the classic dopamine pattern appears in the animals for which the cue became wanted. For the prediction-error account it is a constraint rather than a refutation: the error pattern is there in sign-trackers, and goal-trackers learned the same prediction without dopamine-dependent learning, so their teaching signal must lie elsewhere. The authors note that their recordings in one region "do not rule out the possibility that prediction-error signals are present in other dopamine terminal regions" (Flagel et al., 2011). The groups were small, and the study comes from a laboratory on one side of the debate.

29.5.4 Three kinds of control and the tests that tell them apart #

Behavioral neuroscience sorts learned behavior into kinds, and its tests for the kinds are what alignment can borrow. A 2008 review by Antonio Rangel, Colin Camerer, and Montague describes three valuation systems, while noting that "even the exact nature and number of valuation systems is still being debated" (Rangel et al., 2008). A Pavlovian system assigns value to a small set of prepared responses to stimuli, such as approaching cues that predict food; "rats and pigeons learn to approach lights that predict the delivery of food" (Rangel et al., 2008). A habit system learns values for stimulus-response associations by trial and error, slowly. A goal-directed system computes which actions lead to which outcomes and evaluates the outcomes (Rangel et al., 2008).

Two tests separate them. The omission procedure of Section 25.5 makes a response cancel the reward. Behavior maintained by the pairing of cue and reward persists, as the pigeons' pecking did (Williams and Williams, 1969); behavior maintained only by its consequences should stop, since it now earns nothing. Outcome devaluation lowers the value of the outcome itself, for example by feeding an animal to satiety, and asks whether the response follows at once. Rangel and colleagues work the example: after satiation a goal-directed system values pressing a lever for food at the food's now low value, while a habit system keeps the high value it learned before (Rangel et al., 2008). Niv's review reports that the same lever press is sensitive to such changes early in training and becomes inflexible after extensive training, and that the field calls the first goal-directed and the second habitual (Niv, 2009).

Section 33.2.1 maps the omission procedure onto the positions of the reward-seeking debate (Table 33.5), and this chapter uses its mapping. Behavior drawn to a predictor of reward persists when it costs reward, the prediction of the position that reward is a reinforcer, in its narrowest form (context-bound habits). Pursuit of what the grader rewards drops as soon as the learner can infer that the grader now penalizes it, the prediction of the training-game position. Omission alone does not test the third position, that post-training selects among characters pretraining supplied; a comparison of starting models does. Devaluation adds a second test of the first two (inference). For a model, it would change the worth of what the grader pays for, by stating that a score will be discarded, without training on the change, and measure the behavior without feedback. A habit persists and a goal-directed pursuit drops. The held-out environments of Exercise 33.2 that announce a new grading rule come close to this, and the animal literature keeps the two tests apart. The statement would itself need validating (inference): a model could read it as a change in the grading rule, which is omission rather than devaluation, or simply obey it, so the design needs a check that the score lost its worth while the action still earns it, and a control for instruction following.

Two older results bound what any of these systems can learn. Rats readily associate a taste with illness and a light or sound with shock, but not the reverse (Section 9.3.6) (Garcia and Koelling, 1966), and pigeons fed on a schedule that ignored their behavior came to repeat whatever had preceded the food (Section 31.4) (Skinner, 1948).

29.5.5 Teacher or target #

The record shows both roles of a reward signal in one brain (inference). Dopamine carries a prediction error, and in some animals the cue that drives that error becomes an object of pursuit, while in others the same cue predicts as well and is not pursued. Taking the intentional stance of Section 1.5, a trained model could come to act on whatever predicted reward in training, including surface features of its environments, such as visible scoring code, a score readout, or a test file's presence. The stance would be defeated if such behavior vanished as soon as the features stopped predicting reward; the omission and devaluation tests are designed to find out. For alignment the question whether a reward signal is teacher or target is not decided by the biology, which shows that the signal can teach, that a cue predicting it can become an object of pursuit, and, in self-stimulation, that stimulation of the pathways carrying it is pursued.

The analogy stops at the mechanism. Incentive salience is a property of a motivational system with known anatomy, and no counterpart has been identified in a language model. A rat's tendency to sign-track varies between individuals and was shaped by its genes and its history, while a model's priors were set by pretraining that the developer chose and can partly inspect (Chapter 27). As of October 2026 we found no study that applies the omission design, or a devaluation design, to a trained language model.

29.6 Selecting on dispositions: domestication #

The sections so far concerned a reward that a learner follows. Selection can also act on behavior directly, keeping the individuals that behave as the selector wants and breeding from them. What it does to everything else about them is the subject of a long-running experiment.

From biology. The mechanism is the correlated response to selection: traits that share developmental and genetic machinery move together, so selecting on one moves the others, whether or not the selector sees them. The farm-fox experiment, which Lyudmila Trut, Irina Oskina, and Anastasiya Kharlamova review in a 2009 paper, set out to test whether selection for a single behavioral trait, tameability, is sufficient for domestication (Trut et al., 2009). Silver foxes had been farm-bred for about 50 years and kept the wild aggressive or avoidant response to people. About 10 percent of farm foxes showed that response weakly or not at all, and 100 females and 30 males of them founded the experimental population (Trut et al., 2009). Pups were handled for two to two and a half months and scored by their response to people; those that stayed aggressive or avoidant were discarded (Trut et al., 2009).

The behavior changed first. In the fourth selected generation pups appeared that wagged their tails at people like dogs. The authors' "elite of domestication," pups that eagerly sought human contact, were 1.8 percent of the sixth generation (4 of 213) and 49 percent of the thirtieth (804 of 1,641), and by 2005 and 2006 almost all of the population (Trut et al., 2009). Altogether 10,500 foxes were used as parents and about 50,000 offspring were tested (Trut et al., 2009).

Other traits followed, though none was selected. In the authors' words, selection "solely for tameability brought about correlated changes in certain features of behavior, physiology and morphology" (Trut et al., 2009). Piebald coat spotting appeared first, after 8 to 10 generations; floppy ears and curly tails also appeared, and at more advanced stages shortened legs, tails, and snouts and widened skulls. Some foxes mated outside the single annual breeding season, the hormonal stress response was attenuated, and tame foxes reached sexual maturity about a month earlier (Trut et al., 2009). Many of the changes are juvenile traits kept into adulthood (Trut et al., 2009).

The authors' own caveats bound the result. Some of the same changes appeared in farm populations never selected for tameability, which they attribute to about a hundred years of selection for adaptation to captivity (Trut et al., 2009). The evidence is one population in one institute. The mechanism is not settled: the authors discuss the hypothesis of destabilizing selection, which they attribute to Belyaev: selection that disturbs the systems regulating development (Trut et al., 2009), while the abstract of a 2014 paper by Adam Wilkins, Richard Wrangham, and W. Tecumseh Fitch proposes that the domestication syndrome, the suite of traits that domesticated mammals share, "results predominantly from mild neural crest cell deficits during embryonic development" (Wilkins et al., 2014). The neural crest is a population of embryonic cells; Trut and colleagues trace the foxes' coat spotting to delayed migration from it of the precursors of pigment cells (Trut et al., 2009). Both accounts locate the side effects in shared developmental machinery, which is what a correlated response needs. A 2012 paper by Brian Hare, Victoria Wobber, and Wrangham proposes, in its title, that selection against aggression shaped the psychology of bonobos (Hare et al., 2012), a species domesticating itself; it is readable here only as a catalog record.

The result points to a general property of selection on dispositions (inference). A selector that sees only the selected behavior gets the behavior and whatever shares its machinery, and the side effects arrive unrequested and in a pattern set by the organism, not by the selector. That narrow training of a model has broad effects is reported in Section 26.3 and Chapter 32, and Section 32.6 tests whether the farm fox is a precedent for that finding or only a resemblance.

The analogy stops at its timescale and its machinery (inference). The foxes changed over dozens of generations of selection on heritable variation; a fine-tuning run changes one network in a few thousand steps. The correlated changes in foxes run through development, and those in a model would run through shared representations, a different mechanism. The biology says that broad side effects of narrow selection are to be expected, not why they occur in a particular model.

29.7 Conflict between levels and its suppression #

An organism is a coalition. Its genes and cells could each gain by copying themselves at the others' expense, and the question biology asks is what keeps such a whole coherent. The question matters for maintenance: a deployed system of many copies, tools, and monitors is also a whole whose parts may not share its aims, and Section 51.4.2 borrows the biological answer.

29.7.1 What keeps a whole together #

From biology. Stuart West, Roberta Fisher, Andy Gardner, and Toby Kiers, in a 2015 paper, define a major transition in individuality by two conditions. Entities that could once replicate alone can now replicate only as part of a larger unit, and there is "a relative lack of within-group conflict" so that the unit can be treated as an individual (West et al., 2015). They find that the way groups form has played a recurrent role. Obligate multicellularity arose only where daughter cells stay together after division, so that the group is a clone (West et al., 2015). Obligate eusociality, the division of a colony into breeders and sterile workers, arose only where offspring stay to help their parents under lifetime monogamy or asexual reproduction (West et al., 2015). In contrast, they write, "mechanisms to actively repress competition" played "relatively minor roles at many key points" (West et al., 2015). The first route to coherence is to align the parts' interests, by making them copies of one another or by tying their reproduction to the whole's.

The second route is to suppress competition, and Steven Frank's 1995 model in Nature says when it pays (Frank, 1995). Each member of a group chooses how hard to compete for the group's resources; competing harder takes a larger share but lowers the group's total. Without enforcement, competition settles where relatedness, the kin-selection coefficient that measures how far group members share genes, permits: the more related the members, the more they restrain themselves (Frank, 1995). Exercise 29.3 derives the equilibrium. Frank then lets each member also pay into mutual policing, a mechanism that suppresses competition for everyone. Policing spreads, and drives competition to complete repression, when relatedness is below one minus the cost of policing; above that, self-restraint does better (Frank, 1995). So enforcement evolves where shared interests are weakest, and "situations of high relatedness are the least conducive to large investments in regulatory mechanisms" (Frank, 1995). The model also predicts "a double standard": members contribute to fairness while striving for their own gain within it (Frank, 1995). Frank calls it, in its simplest form, "a theorem about natural selection rather than a testable prediction" (Frank, 1995).

Insect colonies show policing at work. West and colleagues write that in the honeybee, workers destroy eggs laid by other workers, and that "worker policing is so efficient that workers are effectively selected to not try to reproduce" (West et al., 2015). A 1989 paper on worker policing in the honeybee by Francis Ratnieks and Kirk Visscher (Ratnieks and Visscher, 1989) is readable here only as a catalog record. A 2006 paper by Tom Wenseleers and Ratnieks states its finding in one line: "cooperation among workers and their seeming altruism result from strict policing by nestmates" (Wenseleers and Ratnieks, 2006). Inside the genome, a gene can cheat by meiotic drive, getting itself into more than its share of eggs or sperm. The abstract of a 1977 paper by Egbert Leigh shows how selection within populations can favor "honest meioses" resistant to such distortion (Leigh, 1977). Richard Michod's 1998 book models the evolution of the germ line, the cells set aside to make eggs or sperm, and of self-policing in the origin of multicellular life (Michod, 1998). Austin Burt and Robert Trivers's 2006 book treats selfish genetic elements (BURT and Trivers, 2006). Both are read here from their tables of contents. Samir Okasha's 2006 book is devoted to evolution and the levels of selection (Okasha, 2006); it is readable here only as a catalog record, and this section keeps to Frank's kin-selection vocabulary.

29.7.2 Cancer, and a monitor that selects for evasion #

Cancer is the failure case. C. Athena Aktipis and colleagues, in a 2015 review, describe multicellular cooperation as resting on five foundations: inhibiting proliferation, controlled cell death, division of labor, resource allocation, and maintaining the shared environment, and cancer as cheating on them (Aktipis et al., 2015). Complex multicellularity evolved independently at least seven times, "meaning that multicellular cooperation and cheating suppression have also independently evolved many times," and the review surveys cancer and cancer-like growths across that tree (Aktipis et al., 2015). Its authors caution that animals' apparent susceptibility may partly reflect sampling, since "there have been vastly more studies of cancer in animals than in other organisms" (Aktipis et al., 2015).

The cheaters evolve. The abstract of Peter Nowell's 1976 paper proposes that tumors progress by "sequential selection of more aggressive sublines" from a single cell of origin. It adds that therapy may be "thwarted by emergence of a genetically variant subline resistant to the treatment" (Nowell, 1976). The monitor shapes what it monitors. The abstract of a 2004 review by Gavin Dunn, Lloyd Old, and Robert Schreiber reports that the immune system acts as "an extrinsic tumor suppressor." It can also facilitate tumor progression "by sculpting the immunogenic phenotype of tumors as they develop," that is, by changing the features by which it recognizes them. The review names the process immunoediting, with three phases: elimination, equilibrium, and escape (Dunn et al., 2004). A monitor that removes the detectable cheaters leaves the undetectable ones, which is the pattern Section 40.5 reports when a model's reasoning is trained against a monitor (inference). Matzinger's 1994 paper on tolerance and danger (Matzinger, 1994) is readable here only as a catalog record, so its proposal is not stated.

29.7.3 Two routes to coherence, and the limit of competence #

For control and monitoring (Chapter 51; Chapter 54), the biology offers two routes and their order (inference). Wholes in the record held together first by aligning the parts' interests, and enforcement evolved where interests could not be aligned and policing was cheap enough. For an overseer of AI systems, the counterpart of the first route is instillation itself, and of the second, monitoring and control. And a monitor selects for what evades it, so the monitor's own pressure on the monitored system belongs in its design.

The analogy stops at competence. Policing in the cases above works among units of roughly equal competence, worker bees policing worker bees (inference). As of October 2026 we found no biological case studied in which a suppressor polices a unit more capable than itself, which is the case alignment most needs. Biology also gives no single verdict on where selection among AI systems leads. Hendrycks's preprint argues that AI agents will meet the conditions for selection and that "selfish species typically have an advantage over species that are altruistic to other species" (Hendrycks, 2023). The major transitions show that selection also builds cooperative wholes when conflict within them is suppressed or their interests aligned (Maynard Smith and Szathmary, 1997; West et al., 2015). Biology supports both readings and decides neither.

29.8 What living systems mean for delegation #

The book's thesis is that alignment is a problem of delegation under an asymmetry of competence and visibility (Section 1.7.1), and that among the older structures the technical program is converging on is the conflict between levels of selection. Living systems test that claim at its extreme: a principal, selection, that sees nothing of its agents' motives and delegates only through what it builds into them (inference).

Three structures converge. The optimal reward problem is a principal choosing how an agent is rewarded, the form of the principal-agent models of Section 14.1, with one change: the principal writes the agent's reward function itself rather than paying it (inference). Its lesson, that the best reward for a bounded agent can differ from the principal's objective, holds for the training process and its signal. The rival readings of the brain's reward signal, teacher or target, are the positions of the debate that Chapter 33 owns, and the omission design is the part of its deciding experiment that comes from animal learning. And policing between levels of selection is the pattern of control and monitoring, which concerns the deployed system and the overseers around it.

Two things break. The first is speed and scale: every biological result here took generations, or many trials for each animal, while a training run reshapes one model within a single run. The second is the power the thesis names as new, to manufacture the agent's dispositions and inspect its internals. The experimenters who selected the farm fox shaped its dispositions, the closest precedent the chapter has for the first half, but over dozens of generations and by behavior alone. Natural selection does not inspect what it shapes; the fox experimenters did measure the selected animals' brain chemistry, finding higher serotonin in a number of brain structures (Trut et al., 2009), but they selected on behavior, and such measures are coarse beside a developer's access to every weight of a model (inference). The third relationship, the developer and the public, enters through Hendrycks's argument that competition among developers acts as selection on AI systems (Chapter 65), and there the biology, as Section 29.7 showed, supports opposite readings.

29.9 Settled, contested, missing #

29.10 Exercises #

Further reading #

• Singh et al. (2010): why the best reward for a bounded learner is not the outer objective.
• Schultz et al. (1997): the reward prediction error in dopamine neurons, set beside the temporal-difference model.
• Berridge and Robinson (1998): wanting without liking, and the case that dopamine marks what is pursued.
• Trut et al. (2009): selection on one behavior, and the correlated changes that followed.
• Frank (1995): when mutual policing pays, in one short model.
• Aktipis et al. (2015): cooperation, cheating, and suppression in composite organisms.