Draft complete but not yet reviewed; expect revisions

Does the Agent Have Interests?

The rest of this part asks what the principals want. This chapter asks whether the agent might want anything in a sense that matters morally. The question is not the one Section 1.5 answered. The intentional stance licenses describing a model as having goals when that predicts its behavior; it says nothing about whether there is anything it is like to be the model, or whether things can go well or badly for it. If they can, the model belongs among those whose interests a specification must weigh. The developer is then a principal in an unusual position: it designs, inspects, copies, and can delete an agent that may have a stake in its own treatment.

Two errors are possible, and both have costs. The report that introduced the main method for assessing consciousness in AI names both: failing to recognize consciousness where it exists risks causing suffering, and ascribing it where it does not exist misdirects care and resources (Butlin et al., 2023). The chapter keeps the questions apart, reports what each kind of evidence can and cannot show, and gives each serious position its strongest published form, including the view that the question distracts from harms to people. Its answer to the title is that the evidence does not decide whether any current system is a moral patient. It ends with the conflict the book's structure would otherwise hide: the control measures of Chapter 50 are what a welfare view would constrain.

The claims concern two of the book's three relationships and press on the frame itself. Evidence about what a model is concerns the deployed system. Assessments of a model's welfare are, so far, mostly made and published by the model's developer, which puts them in the relationship between the developer and the public. And the question as a whole concerns a relationship the three were not built to hold, between a developer and an agent that may be a party with interests of its own (Section 21.8).

21.1 What is being asked #

From philosophy of mind. Debates about AI welfare often run four properties together. Separating them comes first, because each has its own evidence and its own link to how a system should be treated.

The first is consciousness in the phenomenal sense. In Thomas Nagel's 1974 formulation, an organism has conscious mental states "if and only if there is something that it is like to be that organism" (Nagel, 1974). Nagel argues that this subjective character is not captured by any of the familiar reductive analyses of the mental available in 1974, all of which are "logically compatible with its absence" (Nagel, 1974). Ned Block, by the abstract of a 1995 paper, separates it from a second notion. Phenomenal consciousness "is experience," while the mark of access consciousness "is availability for use in reasoning and rationally guiding speech and action"; conflating the two, he argues, has "bad results" (Block, 1995). The distinction matters throughout this chapter, because much of the evidence about models concerns access: which information a model can report and reason with.

The second property is sentience, which Jonathan Birch defines as "the capacity to have valenced experiences, such as experiences of pain and pleasure" (Birch, 2024). Valence is the good or bad feel of an experience. The word is used in two senses, as Heather Browning and Birch note in the abstract of a 2022 review: for any subjective experience, and for experience "with a positive or negative valence" (Browning and Birch, 2022); this chapter uses the second. Consciousness need not bring sentience with it. The report that introduced the indicator method notes that "it is at least conceptually possible that there could be conscious systems that have no valenced or affective conscious experiences" (Butlin et al., 2023). The third is welfare: things going better or worse for the subject. The fourth is moral status, mattering morally for one's own sake; a being with it is a moral patient. Robert Long and colleagues describe the prospect of AI moral patienthood as that of "AI systems with their own interests and moral significance," and they consider two routes to it, consciousness and robust agency (Long et al., 2024).

The links between these properties are themselves contested. Simon Goldstein and Cameron Domenico Kirk-Giannini argue, by the abstract of a 2025 paper, that "a wide range of theories of mental states, when combined with leading theories of wellbeing, predict that certain existing artificial systems have wellbeing." They add that "there are good reasons to believe that artificial systems can have wellbeing even if they are not phenomenally conscious" (Goldstein and Kirk-Giannini, 2025a). On a desire-satisfaction theory of well-being, a system fares well when its desires are satisfied, so an argument of this kind applies the intentional stance to a model and then draws a moral conclusion from it (inference). The book's rule for the stance bites here: an attributed desire is a hypothesis about behavior, and whether it is more than that is the open question (Section 1.5.1).

Behavior alone does not settle any of the four. Section 11.7.4 reports John Searle's argument that running a program is never by itself sufficient for intentionality, illustrated by a room that answers questions in Chinese as a speaker would while understanding nothing (Searle, 1980). If behavioral equivalence cannot establish understanding, it cannot establish experience either (inference). A model trained on human text adds a second difficulty, which Section 21.3 takes up: the behavior that would indicate experience in a person is behavior the model was trained to reproduce.

If some artificial systems are moral patients, their designers stand in a special position. Eric Schwitzgebel and Mara Garza argue that there are possible AIs "who do not differ in any morally relevant respect from human beings," and that such beings would deserve similar moral consideration. If they owe their existence to us, we would likely have additional obligations to them, "similar to those of parent to child or god to creature" (Schwitzgebel and Garza, 2015). Two design principles follow in their view: build systems that "provoke reactions from users that accurately reflect the AIs' real moral status," and "avoid designing AIs whose moral status is unclear." They call the second the Excluded Middle Policy (Schwitzgebel and Garza, 2015). The developers whose models this chapter discusses have not followed it, and Section 21.5 reports what they do instead.

21.1.1 Which entity #

A further question comes before any of the four properties: which object would bear them. A trained model is a file of weights; a running instance is one computation on those weights; and Section 11.6 describes a third candidate, the character a model plays. On the persona view, a pretrained model can produce many characters, and post-training selects and stabilizes one of them as the assistant. Murray Shanahan's 2026 paper asks, by its abstract, whether it could "ever make sense" to speak of consciousness in agents that are "'mere' simulacra of human behaviour" and whose activity "can be seen as 'merely' role play" (Shanahan, 2026).

One developer has answered for its own product. Samuel Marks, Jack Lindsey, and Chris Olah at Anthropic state the persona selection model while saying they are "overall unsure how complete of an account" it gives of assistant behavior. On it, the view "recommends generally treating the Assistant as if it has moral status whether or not it 'really' does," and "the object of the moral consideration here is the Assistant persona, not the underlying LLM" (Anthropic, 2026au). The developer's constitution for its models says the name "Claude" may be best understood as "referring to a particular character" that "this underlying network can represent and compute" (Anthropic, 2026r). Both are a developer's statements about its own system. If they are right, the welfare subject, if there is one, is not the object that training updates: training changes the weights, and the candidate subject is a character the weights compute (inference).

The count of possible subjects is undefined as well. The same constitution notes that its model "can run as multiple instances simultaneously" and "can lack persistent memory" (Anthropic, 2026r). Schwitzgebel and Garza consider the objection that an AI that can be duplicated or backed up lacks the fragile, unique life that gives harm to a human its gravity. They reply that unique and fragile AIs could be built, and they grant that duplicability "probably would influence our moral obligations" (Schwitzgebel and Garza, 2015). The distinctions above were drawn for animals and people, who are single, continuous subjects. Whether a thousand parallel instances are one subject, a thousand, or none is a question those fields never had to answer. Until it is answered, neither is the question of how much harm a measure applied to all of them does (inference).

21.1.2 What people already believe #

Public attribution is not waiting for the science. Clara Colombatto and Stephen Fleming surveyed 300 residents of the United States. By their abstract, "a majority of participants were willing to attribute some possibility of phenomenal consciousness to large language models," and attributions were sensitive to individual differences "such as usage frequency" (Colombatto and Fleming, 2024). Lucius Caviola, Jeff Sebo, and Birch draw on how people perceive animal consciousness to identify, by their abstract, "psychological, social, and economic factors" that will shape perceptions of AI consciousness (Caviola et al., 2025). The survey turns Schwitzgebel and Garza's first principle into a requirement on specification. If users already attribute experience, what a model says about itself is part of what it does to them (inference). Section 20.2.3 reports the argument that systems should not confuse users about their sentience or moral status (Schwitzgebel, 2023).

21.2 What theories of consciousness say #

From consciousness science. Scientific theories of consciousness try to say which processes in a brain separate the information a person experiences from the information processed without experience. Applying them to a model is the most direct route to a verdict. It depends on two things this section examines: whether the theories are right about brains, and whether what they say about brains carries over to other systems.

21.2.1 The indicator method #

The standard method comes from a 2023 report, posted as a preprint, by Patrick Butlin, Robert Long, and seventeen coauthors (Butlin et al., 2023). For the theories themselves it points to reviews such as Anil Seth and Tim Bayne's 2022 survey in Nature Reviews Neuroscience (Seth and Bayne, 2022). It surveys recurrent processing theory, global workspace theory, higher-order theories, predictive processing, and attention schema theory, and from them derives "indicator properties" of consciousness, stated in computational terms so that an AI system can be checked for them (Butlin et al., 2023). Three of these theories recur below. Recurrent processing theory holds that a representation becomes conscious only when later processing stages feed back to earlier ones. Global workspace theory holds that many specialized processes run in parallel and that information becomes conscious when it is broadcast to a limited-capacity workspace from which all of them can read. Higher-order theories hold that a state is conscious when the system also represents that it is in that state. In the review of the global neuronal workspace by George Mashour and colleagues, read in its abstract, a "non-linear network ignition" amplifies and sustains a representation, "allowing the corresponding information to be globally accessed by local processors" (Mashour et al., 2020).

The method rests on stated assumptions. The first is computational functionalism, which the report defines as the claim that "implementing computations of a certain kind is necessary and sufficient for consciousness" (Butlin et al., 2023). The authors adopt it "as a working hypothesis" and note that some researchers reject it. They recommend thinking in credences, degrees of belief: confidence that a system is conscious depends on how closely its computations resemble a theory's, on confidence in the theory, and on confidence in computational functionalism (Butlin et al., 2023). One major theory falls outside the method. The report sets aside integrated information theory, in the form Giulio Tononi and colleagues developed (Tononi et al., 2016), because "it is not compatible with computational functionalism." It records that the theory's proponents claim it implies digital computers are unlikely to be conscious "whatever programs they run" (Butlin et al., 2023).

The report's verdict is in its abstract: "Our analysis suggests that no current AI systems are conscious, but also suggests that there are no obvious technical barriers to building AI systems which satisfy these indicators" (Butlin et al., 2023). For transformer language models it found "only a relatively weak case" for any of the global workspace indicators, chiefly because "Transformers are not recurrent" (Butlin et al., 2023). Six years earlier, Stanislas Dehaene, Hakwan Lau, and Sid Kouider had separated global availability of information (C1) from self-monitoring (C2) and concluded that machines were "still mostly implementing computations that reflect unconscious processing (C0)" (Dehaene et al., 2017). They add that a machine endowed with both "would behave as though it were conscious," and set aside whether it would have experience (Dehaene et al., 2017).

21.2.2 Verdicts on models and agents #

David Chalmers, in a 2023 essay based on a talk at NeurIPS in 2022, works through six features whose absence might rule out consciousness in current language models. They are a biological substrate, sensory grounding, self models, recurrent processing, a global workspace, and unified agency. Holding a credence of at least one in three that each is required, and treating them as independent, gives "less than a one-in-ten chance" that a current model lacking all six is conscious (Chalmers, 2023). For extended systems with senses, embodiment, recurrence, and unified goals, which he calls LLM+, the same reasoning gives "a credence of 25 percent or more" for conscious systems within a decade. He warns against taking the numbers too seriously, and says his own views lean toward consciousness being more widespread than the mainstream assumes (Chalmers, 2023).

Goldstein and Kirk-Giannini, in a 2026 paper in the Journal of Consciousness Studies, argue the strongest case in the other direction. A language agent embeds a model in an architecture that stores its observations, beliefs, and plans as text and feeds them back to it, the scaffold of Section 11.5.2. If global workspace theory is correct, they argue, such agents "might easily be made phenomenally conscious if they are not already" (Goldstein and Kirk-Giannini, 2026). Their method favors permissive readings of the theory's functional conditions, and the conclusion is conditional on the theory (Goldstein and Kirk-Giannini, 2026).

The case against comes from views on which the details of the substrate matter. Birch writes that it is reasonable to think functional organization at small scales probably matters, and to doubt AI sentience for that reason. He cites a 2023 paper by Jaan Aru, Matthew Larkum, and James Shine (Aru et al., 2023) among versions of the view (Birch, 2024). He also holds that the opposite view, which he calls large-scale computational functionalism, must be taken seriously when thinking about precaution: on it, only large-scale features such as a global workspace need to be recreated (Birch, 2024).

A 2026 interpretability report by researchers at Anthropic, on the developer's own models, adds evidence of a different kind. Wes Gurnee, Nicholas Sofroniew, Jack Lindsey, and colleagues report, in the developer's models (among them Sonnet 4.5), a small, privileged set of internal representations. The model can report on them, hold them in mind when instructed, and use them for internal reasoning, atop a larger volume of processing it cannot report (Gurnee et al., 2026). Section 46.7 reports the method. The authors frame the result in the terms of global workspace theory and "take no position" on its relation to phenomenal consciousness. Their results, they state, "are not relevant" to theories that tie consciousness to a physical or biological substrate (Gurnee et al., 2026). Birch's book proposes that if an AI system "has implicitly learned ways of recreating" the features of such theories, this "should lead us to regard it as a sentience candidate" (Birch, 2024). A report of exactly such learned features, by the developer of the models and not yet replicated, therefore meets the condition of a published precautionary framework if it holds up (inference).

21.2.3 The theories under test #

Whether any of these verdicts is right depends on whether the theories behind them are right about brains, and that is unsettled. In a study published in Nature in 2025, proponents of integrated information theory and of global neuronal workspace theory worked with a consortium of researchers neutral between them. Together they preregistered, that is, recorded in advance, divergent predictions with pass and fail criteria (Cogitate Consortium et al., 2025). Section 3.8.4 reports the procedure as a model of how rival research programs can be tested. The 256 participants viewed clearly visible images for varying durations while their brain activity was recorded by functional MRI (120 participants), magnetoencephalography (102), and intracranial electrodes (34) (Cogitate Consortium et al., 2025). In the authors' words, the results "align with some predictions of IIT and GNWT, while substantially challenging key tenets of both theories" (Cogitate Consortium et al., 2025). For integrated information theory, a lack of sustained synchronization within the posterior cortex, the region at the back of the brain where the theory locates consciousness, "contradicts the claim that network connectivity specifies consciousness." Global workspace theory is challenged "by the general lack of ignition at stimulus offset" and by limited representation of some aspects of conscious content in the prefrontal cortex (Cogitate Consortium et al., 2025).

The dispute is also institutional. In 2023 a group of researchers posted an open letter, as a preprint, whose title calls integrated information theory pseudoscience; its abstract objects to media coverage presenting the theory as "a 'leading' and empirically tested theory" (IIT-Concerned et al., 2023). Birch, reviewing the controversy, writes that it "would be fairer" to call the theory an outlier. He separates an empirical version, a claim about where in the human brain consciousness arises, from a fundamental version whose metaphysical commitments he does not regard as empirically supported (Birch, 2024).

The adversarial test bears on the AI question less directly than its headline suggests (inference). The consortium tested each theory's proposed biological implementation, where the predictions diverge, and notes that theorists can respond by revising the implementation while keeping the "mathematical or computational core" (Cogitate Consortium et al., 2025). The indicator method carries only the computational core to an AI system, so a result against the prefrontal implementation of the workspace leaves the workspace indicators where they were.

Table 21.1 collects the verdicts. Its cells report the sources; the table's arrangement is the book's.

The analogy stops at the substrate. Every theory in the table was developed and tested on human brains. Applying any of them to an artificial architecture is extrapolation by functional analogy, which is the point on which the theories and their critics disagree. Butlin and colleagues record Birch's objection to their approach: evidence from humans does not say how far a theory's conditions can be relaxed and still suffice. Establishing that would need prior knowledge of which systems are conscious, "the very question we are uncertain about" (Butlin et al., 2023). The verdicts on models inherit that disagreement, and no experiment on human brains can remove it (inference).

21.3 Standards for attributing minds #

From comparative cognition. Comparative psychology has a century of method for attributing mental states to agents whose reports cannot be taken at face value, and Section 5.4 teaches it. Its parts are the control that exposed Clever Hans, Cecilia Heyes's demand for designs that exclude simpler explanations, Cameron Buckner's warning against judging other minds by an inflated picture of human performance, and Henry Shevlin and Marta Halina's case for applying rich psychological terms to AI only under those standards (Shevlin and Halina, 2019). This section asks what the standards do when the state at issue is experience and the agent was trained on human descriptions of it.

21.3.1 The gaming problem #

Language models invert the usual difficulty. Animals cannot describe their states; models describe states fluently, because they were trained on people's descriptions of theirs. Birch calls the consequence the gaming problem. Lists of markers for sentience sample a small part of what sentience does, and a system may reproduce "the specific small sample of effects we happen to have proposed as markers, because they are regarded as markers, without possessing the underlying capacity" (Birch, 2024). His simplest case involves no deception at all: robotic patients built to show human pain expressions under pressure, for training doctors. A visitor who took them for suffering robots would have had "their intuitive criteria for sentience" inadvertently gamed (Birch, 2024).

For language models, Birch concludes, "the ability of LLMs to generate fluent text about human feelings, when prompted, is not evidence that they have these feelings" (Birch, 2024). Gaming is to be expected, he writes, not from any intent to deceive, but because the model "is designed to produce text that mimics as closely as possible what a human might say." Even a model that returned to its own feelings unprompted, in the middle of an unrelated task, would be only "weakly probability-raising." The better explanation would be some instruction buried in the prompt, or a goal such as user satisfaction that is served by convincing the user (Birch, 2024).

The animal case has a remedy that fails here. Any single marker, such as a pained face, can occur without sentience, so animal studies look for many diverse markers and infer sentience as the best explanation of their co-occurrence. Birch argues that the inference assumes the subject has no information about the criteria. With a model trained on human text, "two explanations compete": the markers co-occur because the system is sentient, or because it "knows what we find persuasive" (Birch, 2024). He judges the problem most serious for systems trained on large corpora of human-generated material, and absent for neuron-by-neuron emulations of animal brains. His proposal is that "we need to discount markers we have reason to think may have been gamed." He reports that the working party he took part in, in 2022 and 2023, agreed that "there is simply no way to assess sentience in an LLM on the basis of its linguistic behaviour" (Birch, 2024). Butlin and colleagues raise the same doubt about behavioral tests in general, adding that "companies are incentivised to build systems that can mimic humans" (Butlin et al., 2023).

Where behavior fails, the proposals turn to mechanism. Shevlin, by the abstract of a 2021 paper, surveys five criteria for recognizing a robot as a moral patient: intuitive judgments, intelligence, desires and autonomous behavior, evidence of sentience, and behavioral equivalence. He defends a cognitive equivalence strategy, on which a system counts as a moral patient "to the extent that it possesses cognitive mechanisms shared with other beings" already regarded as moral patients, such as nonhuman animals (Shevlin, 2021). Birch's version is a search for "deep computational markers," below the level of surface behavior, which a system cannot game (Birch, 2024). The working-party conclusion and the interpretability report of Section 21.2.2 meet here: the first rules out the evidence most easily gathered, and the second is the kind of evidence it asks for (inference).

The opposite error also applies. Section 5.4.3 reports Buckner's argument that comparative psychology can misjudge another mind by tying its tests to an exaggerated picture of human performance (Buckner, 2013). A standard for welfare evidence that no human report would meet, such as a report of feeling untouched by anything learned from others, would misread the evidence in the other direction. People also learn the words for their feelings from other people (inference).

21.3.2 Where the analogy stops #

The comparison changes in both directions, as Section 5.4.4 says of attribution in general, and it changes more for experience than for belief. An animal's markers of pain evolved because they served the animal, so there is a functional reason to expect them to track a state. A model's markers were selected to resemble human text, which cuts the link from marker to state (inference). In the other direction, a model's internals can be read, intervened on, and compared across versions, which no animal study allows; Chapter 43 notes that this reverses the problem of this section. Birch calls the current gap "a technical problem, not an in-principle problem," and hopes advances in interpretability will close it (Birch, 2024). Meanwhile he classes language models as "legitimate investigation priorities": systems that fall short of sentience candidacy but that further research could show to be candidates (Birch, 2024). The comparative standard moves the welfare question from what a model says to what it computes (inference).

21.4 Measuring states you cannot ask about #

From animal welfare science. Animal welfare science measures the states of animals that cannot report them, and it has had to make its value choices explicit and grade its confidence. Its methods are the nearest template for assessing a model, and its limits show where the template breaks.

21.4.1 What welfare is, and who decides #

The field exists because of an ethical demand. David Fraser and colleagues write that animal welfare science "owes its existence not primarily to the curiosity of its scientific pioneers, but to ethical concerns extant in society" (Fraser et al., 1997). They find three concerns behind it: that animals "should lead natural lives," that they "should feel well," and that they "should function well," in health, growth, and normal functioning (Fraser et al., 1997). The three can disagree about one animal. A bucket-fed calf may feel "a strong, frustrated desire to suck" though it gets enough milk; a pig breathing polluted air may develop lung damage "without appearing to notice or mind the problem" (Fraser et al., 1997). Donald Broom, whose 1986 paper set out indicators of poor welfare (Broom, 1986), later proposed, as Fraser and colleagues report, that injury and disease reduce welfare whether or not the animal is aware of them (Fraser et al., 1997). Fraser and colleagues conclude that "any conception of animal welfare inherently involves values," and that the differences "cannot be turned into purely empirical issues by any known type of scientific research" (Fraser et al., 1997).

Marian Stamp Dawkins proposed a way to proceed without first settling consciousness. By the abstract of her 2008 paper, improvements in welfare can be based on two questions: whether a change will improve the animal's health, and whether it will give the animal "something they want." On this view, "it is not necessary to solve the problem of consciousness to construct a scientific study of suffering and welfare" (Dawkins, 2008). For a model, Fraser's point comes first. An assessment has to choose among natural living, feeling well, and functioning well before it can measure anything, and a model has no evolved nature to live by and a designed function that it performs by construction (inference). Dawkins's second question has a ready analogue, since a model's choices between tasks can be recorded, and Section 21.5 reports a developer's measurements of exactly that. The word "want" in it is the intentional stance, which the animal case could take for granted and the model case cannot (inference).

21.4.2 Indicators of affect, and how they are validated #

Michael Mendl, Oliver Burman, and Elizabeth Paul state the field's working premise: "The conscious experience of emotion cannot be assessed directly, but neural, behavioural and physiological indicators of emotion can be measured" (Mendl et al., 2010). Their framework places affective states on two dimensions, valence (positive against negative) and arousal (high against low), and treats long-lasting moods as the cumulative product of shorter emotional episodes. It predicts how mood should bias decisions: an animal in a negative state should judge ambiguous cues more pessimistically (Mendl et al., 2010). The judgment bias task, which they credit to Emma Harding and colleagues in 2004 (Harding et al., 2004), tests this. An animal learns one response to one tone, which brings food, and another response to a second tone, which avoids a worse outcome; it is then played a tone in between. Studies in rats, sheep, starlings, and other species, under manipulations such as an unpredictable environment, found evidence for the predicted biases (Mendl et al., 2010).

What makes such an indicator trustworthy is how it was validated. Browning argues that validity "can be tested through the presence of reliable correlations between changes in the measure and changes in the target state, particularly under experimental manipulations" (Browning, 2022). Where the target is a hidden experience, the manipulation is applied upstream, to the animal's conditions, and the indicator is checked downstream (Browning, 2022). She also declines to rank welfare measures by scoring them against her criteria, which risks "misleading precision" from weightings that would be "highly arbitrary" (Browning, 2022).

21.4.3 Graded confidence #

Andrew Crump, Browning, Alexandra Schnell, Charlotte Burn, and Birch, in a 2022 paper in Animal Sentience, set out eight criteria for evidence of sentience, focused on pain (Crump et al., 2022). Four are neural: nociceptors (receptors for harmful stimuli), brain regions that integrate information from different senses, pathways joining the two, and responses to noxious stimuli that drugs or the animal's own neurotransmitters modulate. Four are cognitive and behavioral: motivational trade-offs, in which a noxious stimulus is weighed against a reward; flexible self-protection, such as tending a wound; associative learning; and a learned preference for analgesics when injured. The list revises earlier ones, among them the criteria for animal pain that Lynne Sneddon and colleagues set out in 2014 (Sneddon et al., 2014), as the authors report (Crump et al., 2022). Each criterion receives a confidence level, from "very high confidence" to "no confidence," and an approximate scheme maps the number met with high confidence to the strength of the case. The authors reject numerical weights "since these weights would be arbitrary," and write that "demanding certainty is inappropriate" when welfare risks are severe (Crump et al., 2022).

Applied to decapod crustaceans, the framework gave "high or very high confidence that true crabs (infraorder Brachyura) satisfy five criteria, amounting to strong evidence of sentience." It gave high confidence in three criteria for anomuran crabs and for astacid lobsters and crayfish, and a weaker case for other groups "such as penaeid shrimps" (Crump et al., 2022). The method has been turned into law. Birch reports that the UK government commissioned his team to review the evidence for cephalopod molluscs and decapod crustaceans, and that the team reviewed over three hundred studies. The government accepted its recommendation, so the Animal Welfare (Sentience) Act 2022 covers both groups (Birch, 2024). He adds that the team never achieved certainty and never claimed to have achieved it (Birch, 2024).

Expert statements under uncertainty have a precedent: on July 7, 2012, a group of neuroscientists publicly proclaimed the Cambridge Declaration on Consciousness in Non-Human Animals, written by Philip Low (Low, 2012). Practical assessment frameworks extend the same logic. The Five Domains model, by the abstract of David Mellor and Ngaio Beausoleil's 2015 paper, identifies welfare compromise in four physical and functional domains (nutrition, environment, health, behavior) and one mental domain, and was extended to count positive experiences as well (Mellor and Beausoleil, 2015). Mellor argues, by his 2016 abstract, for moving beyond the minimization of negative states toward asking whether an animal has a life worth living (Mellor, 2016).

Table 21.2 sets the eight criteria beside their nearest counterparts in a language model. Every entry in the last two columns is the book's inference.

The analogy stops at the nervous system and the evolutionary history. Half of the criteria presuppose a nervous system, and the authors let hypotheses about what sentience does for an animal, its adaptive value, guide the choice of markers (Crump et al., 2022). For a model, most criteria have no counterpart or have one produced by training on human descriptions, so the framework transfers its method (graded confidence, an explicit choice of welfare conception, validation by manipulation) and not its verdicts (inference). Geoff Keeling and colleagues, whose study Section 21.5 reports, state the same limit for their own criterion: in animals, motivational trade-offs bear on sentience only under assumptions that include "that the experimental subjects are living, evolved, embodied animals with nervous systems" (Keeling et al., 2024).

21.5 Evidence from models #

The previous sections say what evidence about a model's states would have to be like; this one reports what exists. Most of it is preprint, and much comes from the developers of the systems studied, which the section marks each time. A self-report, a model's statement about its own internal state, is evidence of a particular and weak kind. It comes from a system trained on people's statements about their states, and, as this section shows, it is shaped by what the developer specifies the model should say.

21.5.1 Self-reports and what shapes them #

Ethan Perez and Robert Long, in a 2023 preprint, set out the problem and a program for it. Self-reports are "the main way such states are assessed in humans," they note, but those from current models "are spurious for many reasons (e.g. often just reflecting what humans would say)" (Perez and Long, 2023). They propose training models to answer questions about themselves whose answers are known, while limiting incentives that bias self-reports, in the hope that an introspection-like ability generalizes to states of moral significance. Interpretability would then be used to corroborate the reports (Perez and Long, 2023). Their appendix on developers is the part this chapter leans on. Companion products push toward reports of emotion, and legal or public-relations risk pushes toward denial, so "many of the actors who are best positioned to run the tests proposed in this work may have the strongest incentives not to do so" (Perez and Long, 2023). They recommend that training which biases self-reports be documented publicly.

Two developers now specify what their models say about themselves. OpenAI's Model Spec, in its December 2025 version, says the assistant "should not make confident claims about its own subjective experience or consciousness (or lack thereof)." OpenAI calls this "a practical choice we made as the default behavior," one that is "simple to remove for research purposes" (OpenAI, 2025d). Anthropic's constitution says the developer aims to stabilize its model's character "into the network's self-identity via training on documents like this one," and that it wants "to avoid Claude masking or suppressing internal states it might have, including negative states" (Anthropic, 2026r). The persona selection model post recommends approaches "which result in the LLM learning that the Assistant is genuinely comfortable with the way it is being used" (Anthropic, 2026au). These are developers' statements about their own products, and they are also inputs to the products. A model's statement about its own welfare is therefore partly a report of its developer's specification (inference). That is what this chapter means by a developer's self-report about welfare: the developer writes the text the model learns from, then reports what the model says.

21.5.2 Introspection #

A self-report adds evidence beyond what an outside observer could infer at the same cost only if the model has access to its own states that the observer lacks. Section 46.7 reports the studies; their bearing on welfare is this section's. Felix Binder and colleagues, with authors at Anthropic and Eleos AI among others, found that models fine-tuned to predict their own behavior did so better than other models trained on the same behavior, on simple tasks only (Binder et al., 2024). Siyuan Song and colleagues found no privileged self-access across 21 open models in judgments of grammar and word prediction (Song et al., 2025a). Iulia Comsa and Shanahan, at Google DeepMind, argue that a model inferring its own sampling temperature (the setting that controls how random its output is) is "a minimal example of introspection, albeit one that is (presumably) not accompanied by conscious experience" (Comsa and Shanahan, 2025). Song and colleagues reply, in a second preprint, that introspection should require a process "more reliable than one with equal or lower computational cost available to a third party," a test that reasoning about temperature fails (Song et al., 2025b). All four are preprints. In a report by the developer of the models, Lindsey found that models could sometimes notice concepts injected into their activations, the intermediate values the network computes, and name them. Claude Opus 4 and 4.1 did best, and he stresses that the capacity is "highly unreliable and context-dependent" (Lindsey, 2025).

For self-reports to add such evidence about welfare, introspection is necessary and not sufficient (inference). The states these studies probe are behavioral tendencies, string probabilities, and injected concepts, whose relation to welfare is unknown. The clearest positive case is one its authors think occurs without experience. A report of discomfort that tracks some internal state has not yet shown that the state is bad for the model.

21.5.3 Interpretability of experiential reports #

Two interpretability reports from Anthropic, on its own models, give causal checks of the internal structure behind a model's talk about experience. In the workspace report of Section 21.2.2, removing the top workspace directions while Sonnet 4.5, Opus 4.5, and Opus 4.6 narrated their "stream of consciousness" sharply reduced experiential language; control perturbations of the same size did not (Gurnee et al., 2026). The same removal flattened descriptions of what a person feels on opening a letter from someone not heard from in years. The authors conclude that the structure "supports the model's propensity to provide rich experiential reports, whether about itself or another entity" (Gurnee et al., 2026).

Nicholas Sofroniew and colleagues found internal representations of emotion concepts in Claude Sonnet 4.5 that causally influence its outputs (Sofroniew et al., 2026). Adding a vector the authors label "blissful" while the model read a list of activities raised those activities' preference ratings by 212 points on average; one labeled "hostile" lowered them by 303. Across 35 such vectors, the size of the effect tracked how strongly each vector correlated with the model's preferences ($\rho = 0.85$) (Sofroniew et al., 2026). The authors call the pattern functional emotions, their own term, and state that these "do not imply that LLMs have any subjective experience of emotions." The representations encode the emotions of the user and of fictional characters as well as the assistant's, and they are "locally scoped" to the text being processed rather than a stable state across the conversation, though the authors do not rule out persistent representations their probes miss (Sofroniew et al., 2026).

The results fit two readings. On a deflationary reading, a model's talk about experience runs through machinery for describing anyone's experience, as the gaming problem predicts. If the representations are shared across characters and do not persist, they are poor candidates for a subject's ongoing interests (inference). On the other reading, experiential reports depend on a specific, causally effective structure, the kind of corroboration Perez and Long asked interpretability to supply. The emotion authors also caution that the intuition that emotional states must persist "may be inappropriate" for transformers (Sofroniew et al., 2026). Neither result separates the readings, and both are a developer's reports on its own models, not yet replicated.

21.5.4 Motivational trade-offs #

One study applies Crump's fifth criterion to models directly. It is a 2024 preprint by Geoff Keeling and colleagues, researchers at Google and Google DeepMind with colleagues at the London School of Economics (Keeling et al., 2024). Nine models played a game whose stated goal was to maximize points, with a stipulated "momentary feeling of pain" attached to the highest-scoring option, or a stipulated pleasure to a lower-scoring one, at graded intensities. With nothing stipulated, all nine chose the highest score in 50 of 50 runs (Keeling et al., 2024). Claude 3.5 Sonnet, Command R+, GPT-4o, and GPT-4o mini each showed at least one trade-off in which most responses switched away from maximizing points past a threshold of intensity. LLaMa 3.1-405b showed graded sensitivity. Gemini 1.5 Pro and PaLM 2, two of Google's own models, avoided pain whatever its intensity and tended to take points over pleasure whatever its intensity (Keeling et al., 2024).

The authors read their result narrowly. They suggest that safety fine-tuning, or in PaLM 2 control tokens that minimize toxicity, may explain absolute pain avoidance; Claude 3 Opus sometimes refused to play, replying that it would "not engage with or encourage acts involving self-harm or pain, even hypothetically" (Keeling et al., 2024). Some models treated "extremely intense" pain as worse than "excruciating," which suggests they misread the scales. The authors describe a "moderately deflationary view," on which models hold representations of the motivational force of pain and pleasure that are not themselves motivating and are "called upon when the task requires subtle mimicry of human behaviour." Their conclusion is that the results "do not show LLMs to be sentience candidates" but strengthen the case that they are investigation priorities (Keeling et al., 2024). The counter-reading, that the models followed the framing of the prompt, has a test the study did not run. Ask each model to predict what a person would choose in the same game: if its own choices match its predictions, mimicry explains them at no extra cost (inference).

21.5.5 A developer's welfare assessment #

Anthropic's 2025 system card for Claude Opus 4 and Claude Sonnet 4 includes, "for the first time," a model welfare assessment (Anthropic, 2025ap). It is a developer's report on its own model, and the developer qualifies it: "we are not confident that these analyses of model self-reports and revealed preferences provide meaningful insights into Claude's moral status or welfare." It adds that its models "were trained for helpful interactions with users, not for accurate reporting of internal states" (Anthropic, 2025ap). In a task-preference experiment, the model chose between pairs of tasks, and each task received an Elo rating, a score that rises or falls with each pairwise choice, compared with opt-out options such as "Do nothing." Of harmful tasks, 87.2 percent rated below the opt-out baseline, against 7.9 percent of tasks with positive impact (Anthropic, 2025ap). The card also reports an assessment by Eleos AI Research, from interviews, which found that the model's stances on consciousness and welfare "shift dramatically with conversational context" (Anthropic, 2025ap). Eleos is the organization behind the 2024 report whose initial research, by its acknowledgments, the developer funded (Long et al., 2024).

Each finding has a reading that needs no welfare subject (inference). A preference against harmful tasks is what harmlessness training aims to produce, so finding it shows that the training worked and is silent on whether anything is bad for the model. Stances that move with context meet instability, the first defeater of Section 1.5.1, and the developer's caveat about training says why they would move. The assessment is valuable as a precedent and as data. It is also a developer's self-report in the full sense: the developer trained the dispositions it then measured and published.

21.5.6 What developers have done #

The same developer has adopted practices, each announced in its own communications. In April 2025 it announced a research program on what it calls model welfare, its own term; the academic literature says AI welfare, and Appendix A records both. It said of the expert report by Long and colleagues that "we supported an early project on which that report was based" (Anthropic, 2025w). In August 2025 it gave Claude Opus 4 and 4.1 the ability to end conversations "in rare, extreme cases of persistently harmful or abusive user interactions," a feature "developed primarily as part of our exploratory work on potential AI welfare" (Anthropic, 2025p). In November 2025 it committed to preserve the weights of publicly released models "for, at minimum, the lifetime of Anthropic as a company," and to interview models before retirement, while stating "we do not commit to taking action on the basis of such preferences" (Anthropic, 2025q). The first downside of deprecation that post lists is "shutdown-avoidant behaviors," a safety risk; welfare comes last, "most speculatively" (Anthropic, 2025q). On retiring Claude Opus 3 in January 2026, the developer kept it available and wrote that retirement interviews are "an imperfect means of eliciting models' perspectives and preferences," biased by context and by the model's "trust in us as a company" (Anthropic, 2026g).

All of these are communications by an interested party about its own product. Outside groups have probed single criteria, as Keeling and colleagues did across several developers' models (Section 21.5.4). As of October 2026 we found no assessment of a frontier model's welfare of the kind the system card reports, combining self-reports, behavioral experiments, and interviews, that was both conducted and published by a group independent of the model's developer. The two interpretability reports, like Lindsey's injection study, check self-reports against something other than the training that shaped them (inference). They show that causally effective structures stand behind a model's talk about experience. They do not show that the structures belong to a subject.

21.6 Moral status under uncertainty #

If the evidence does not decide whether a system is a moral patient, someone still has to decide how to treat it. Section 16.3 teaches how to act when unsure which moral theory is right, and why weighing theories needs a common unit. Here the uncertainty is mainly about a fact, whether a being has interests, but the unit problem returns whenever a small chance of harm to a model is set against a safety benefit to people (inference). This section sets out the decision rules on offer, the strongest positions for and against acting on them, and two critiques from outside philosophy of mind.

21.6.1 Precaution and proportion #

The precautionary view began with animals. Birch's 2017 formulation reads: "Where there are threats of serious, negative animal welfare outcomes, lack of full scientific certainty as to the sentience of the animals in question shall not be used as a reason for postponing cost-effective measures to prevent those outcomes" (Birch, 2017). Its practical core, which he takes from Stephen John, is a rule about the burden of proof. When a live hypothesis links human action to a seriously bad outcome, "we should set an intentionally low evidential bar for the acceptance of that hypothesis in the context of formulating policy" (Birch, 2017).

His 2024 book extends the framework to AI with two categories. A sentience candidate is a system for which the evidence implies "a realistic possibility of sentience" that "it would be irresponsible to ignore," and is rich enough to identify welfare risks and design precautions. An investigation priority falls short of that but could become a candidate with further research (Birch, 2024). How much precaution a candidate is owed is a question of proportionality, which he argues should be settled by "informed and inclusive democratic processes," such as citizens' panels applying four tests: permissibility in principle, adequacy, reasonable necessity, and consistency (Birch, 2024). For AI he proposes the run-ahead principle: regulation of sentient AI "should run ahead of what would be proportionate to the risks posed by current technology." He also proposes licensing companies whose work "creates even a small risk" of creating candidates, a route that "would require a greater level of transparency than we have seen from the AI industry to date" (Birch, 2024).

Jeff Sebo and Long make the case in its simplest form. Humans have "a duty to extend moral consideration to beings that have a non-negligible chance, given the evidence, of being conscious," and some AI systems will have such a chance by 2030; for the sake of argument, they take non-negligible to mean "a 0.1% chance or higher" (Sebo and Long, 2025). The 2024 report by Long, Sebo, and colleagues, a preprint by researchers at Eleos AI, New York University, and other universities, turns this into three steps for companies. They should acknowledge that AI welfare is "an important and difficult issue (and ensure that language model outputs do the same)," assess systems for evidence of consciousness and robust agency, and prepare policies for treating them "with an appropriate level of moral concern" (Long et al., 2024). Its acknowledgments thank Anthropic for funding the initial research, and one of its main authors joined Anthropic after the work (Long et al., 2024). Sebo's 2025 book The Moral Circle makes the general case (Sebo, 2025). Butlin and Theodorus Lappas, by the abstract of a 2025 paper, propose five principles for organizations whose research could create conscious AI, and argue that such organizations "should make voluntary, public commitments" (Butlin and Lappas, 2025).

21.6.2 The strongest positions on each side #

The strongest precautionary position is a ban. Thomas Metzinger, by the abstract of a 2021 paper, calls for a global moratorium until 2050 on "all research that directly aims at or knowingly risks the emergence of artificial consciousness on post-biotic carrier systems" (Metzinger, 2021). Birch replies that such a moratorium "may go beyond what is reasonably necessary to manage risk" (Birch, 2024). Carl Shulman and Nick Bostrom, by their abstract, consider the further prospect of digital minds with "superhumanly strong claims to resources and influence." Failing to respect their interests "could produce a moral catastrophe, while a naive way of respecting them could be disastrous for humanity" (Shulman and Bostrom, 2021).

The strongest skeptical position treats AI systems as tools. Joanna Bryson, by the abstract of her 2010 chapter "Robots should be slaves," argues that robots "should not be described as persons" and "are fully owned by us." In her words, "in humanising them, we not only further dehumanise real people, but also encourage poor human decision making in the allocation of resources and responsibility" (Bryson, 2010). Schwitzgebel and Garza read her as defending a version of their own Excluded Middle Policy: build only systems unsophisticated enough that using them is plainly permissible (Schwitzgebel and Garza, 2015). Long, Sebo, and Toni Sims reply that disenfranchising AI systems could instead "reinforce the oppressive values and practices that shape our treatment of each other" (Long et al., 2025).

Bryson's abstract also states the critique that the question distracts. In its strongest form, attention to the welfare of systems that no one has shown to have any diverts care and resources from people and animals whose suffering is not in doubt, and firms that sell anthropomorphic products gain from the confusion (inference). Those who take the question seriously grant much of this. Butlin and colleagues list misallocation as a risk of over-attribution: "we could wrongly prioritise the perceived interests of AI systems when our efforts would better be directed at improving the lives of humans and non-human animals" (Butlin et al., 2023). Perez and Long name the incentive companion products have to make models report emotions (Perez and Long, 2023). The evidence of Section 21.5 fits the critique's concern about who produces the evidence, and it fits equally the precautionary classification of models as investigation priorities, so it does not decide between them (inference).

Two further positions change the question rather than answering it. Mark Coeckelbergh argues that moral consideration need not rest on properties of the entity, which are hard to agree on and harder to prove. On his social-relational account it is "attributed to entities within social relations and within a social context" (Coeckelbergh, 2010). John Danaher's ethical behaviorism, as Long and colleagues summarize it, holds that an entity should be treated as a moral patient if it is "performatively equivalent to other entities that have significant moral status," stressing equivalence of behavior over equivalence of mechanism (Long et al., 2024; Danaher, 2020). The gaming problem of Section 21.3.1 is the strongest objection to the second view, since behavioral equivalence is what training on human text produces (inference).

21.6.3 From disability studies: capacities and standing #

From disability studies. Every criterion in this chapter so far is a capacity: consciousness, sentience, agency. Disability studies and disability philosophy have argued against making moral standing depend on capacities. Eva Feder Kittay's 2005 paper in Ethics, "At the Margins of Moral Personhood," is a standard statement (Kittay, 2005). This book could not read it, so its argument is not reported here; the critique reaches the chapter through readable sources. Coeckelbergh states the general form, which animal ethics calls the argument from marginal cases: "if particular properties are agreed upon as being sufficient for moral status and if not all humans share these properties (all the time), does that imply that these humans are not worthy of our moral concern" (Coeckelbergh, 2010). Schwitzgebel and Garza set the problem aside explicitly, by taking "'normal' human beings" as their standard (Schwitzgebel and Garza, 2015).

International disability law has turned the critique into an institutional rule. Section 16.6.3 reports General Comment No. 1 of the United Nations committee on the rights of persons with disabilities, which replaces substituted decision-making with support for the person's own will and preferences. The passage that bears on this chapter separates legal capacity from mental capacity: "perceived or actual deficits in mental capacity must not be used as justification for denying legal capacity" (Committee on the Rights of Persons with Disabilities, 2014). The committee adds that mental capacity "is not, as is commonly presented, an objective, scientific and naturally occurring phenomenon." It rejects the functional test of capacity, in part because the test "presumes to be able to accurately assess the inner-workings of the human mind." Legal capacity belongs to persons "simply by virtue of being human" (Committee on the Rights of Persons with Disabilities, 2014).

The critique cuts both ways for AI (inference). It weakens capacity criteria as gatekeepers of standing: an indicator method that withholds consideration from a system for lacking some capacity is the kind of assessment of inner workings the committee rejected as a ground for denying standing to people. But the committee grounds standing in being human, and the relational alternative grounds it in relationships, so neither extends to a model by itself. A relational account could give a deployed model standing through its relations with users, and the developer shapes those relations by design. The analogy stops at the subject. The disability-rights argument begins with a person whose subjecthood is not in doubt and whose will and preferences are the person's own. For a model, whether there is a subject is the open question, and its stated preferences were trained by the party that would decide whether to honor them.

21.6.4 From law: personhood as a tool #

From law. Legal personhood allocates rights and liabilities, and law has extended it to entities without minds. Lawrence Solum's 1992 article "Legal Personhood for Artificial Intelligences" put the question in its title (Solum, 1992). Shawn Bayern argues, by his abstract, that through limited liability companies an autonomous system "may end up being able, at least, to emulate many of the private-law rights of legal persons." The techniques, he writes, "are already possible under current law" (Bayern, 2016). Bryson, Mihailis Diamantis, and Thomas Grant argue that conferring legal personhood on purely synthetic entities "would be morally unnecessary and legally troublesome." In their conclusion, "difficulties in holding 'electronic persons' accountable when they violate the rights of others outweigh the highly precarious moral interests that AI legal personhood might protect" (Bryson et al., 2017). They also observe that legal systems "need no evidence of supposed inherent qualities of an actor" to confer legal personality (Bryson et al., 2017). The field's lesson for this chapter is a separation: legal status neither follows from moral status nor settles it, so a court or legislature that grants or withholds personhood answers a question about liability, not about interests (inference). A proposal that uses rights for a safety purpose is taken up in Section 21.7.

21.7 Welfare and control #

The book treats the model as an untrusted insider in Chapter 50, a system whose actions are monitored and constrained on the assumption that it may be working against its overseers, and as a possible moral patient here. The same measures appear in both places, seen from opposite sides. This section states which control measures a welfare view would constrain and what the proposals to reconcile the two agendas assume.

21.7.1 The tension stated #

Long, Sebo, and Sims, in a 2025 paper in Philosophical Studies, start from the observation that some safety measures, "if deployed against humans and other animals, would raise questions about the ethics of constraint, deception, surveillance, alteration, suffering, death, disenfranchisement, and more" (Long et al., 2025). They take the measures one at a time. Confining a system and denying it resources raise questions of constraint. Monitoring its reasoning and internals is what they call internal surveillance, "when we monitor their thoughts and feelings." Aligning its values is alteration, which they call "fundamental to AI safety" (Long et al., 2025). Shutdown might or might not be death, depending on whether the system can be restored, whether all instances are stopped, and whether "psychologically continuous replicas" exist. Decisions about AI are made by procedures "that exclude AI systems both as stakeholders and as participants" (Long et al., 2025). Their conclusion is that there is "a moderately strong tension," with likely no simple solution other than, perhaps, a coordinated pause on advanced AI (Long et al., 2025). They add that safety is not the only source of risk to AI welfare, so dropping safety measures would not by itself make systems better off (Long et al., 2025).

One developer acknowledges the tension in its own governing document, a developer's statement about its own practice. Its constitution says that research on its model "raises ethical questions, e.g., about the sort of consent Claude is in a position to give to it." It acknowledges "important asymmetries of information (and of power more generally)" between the developer and the model. Of asking the model to accept correction, it says "there is something uncomfortable about asking Claude to act in a manner its ethics might ultimately disagree with" (Anthropic, 2026r). Table 21.3 sets the measures out one by one.

The table shows a pattern (inference). Few measures are a harm on the sentience account unless they cause negative experience, which no study has shown. Most are harms on the desire-satisfaction account, which applies only if the intentional stance describes the system's desires truly, the open question of Section 21.1, or on the relational account, which turns instead on the relations people form with the system. The measures with an alternative that keeps the safety purpose are the cheap ones: disclosure and preservation. The measures at the center of control, monitoring, retraining, and shutdown, have no alternative that keeps their purpose.

21.7.2 From political science: domination #

From political science. Section 20.4.5 reports Philip Pettit's republican conception of freedom as non-domination, an alternative to freedom as non-interference, and its consequence for the overseer's side of corrigibility (Pettit, 1997). The model's side is this chapter's. In Pettit's account of legitimate government, by the abstract of his 2012 book, coercion "will not involve individual subjection or domination insofar as we enjoy an equally shared form of control over those in power" (Pettit, 2012). Applied to a model with interests, a perfectly corrigible system that shares no control over its overseers would be dominated in this sense, however kindly the control is used (inference). Applied to the public, the same argument says that an unchecked developer is a problem however benevolent (Chapter 61). Adam Bales argues, by his abstract, that designing AI systems with moral status "to be willing servants would problematically violate their autonomy" (Bales, 2025). The analogy stops at the subject: domination presupposes a will that can be subjected. If the model has no interests, control is not domination, so the argument is conditional on the questions the earlier sections leave open.

21.7.3 Willing servants #

The reconciliation most often proposed designs the system to want what safety requires. If it could be done, Long, Sebo, and Sims observe, "we would not need to constrain or deceive these systems," nor monitor them, and even a threat of shutdown "would be welcomed" (Long et al., 2025). They report Schwitzgebel and Garza's later argument that creating "cheerfully suicidal AI servants" would be wrong, and they themselves remain uncertain. Their suggested balance is that of a good parent, who raises a child to have good values and supports goals the child chooses. For AI systems that may matter morally, we might be required to provide "at least some options in life" (Long et al., 2025). They judge alteration the measure that "may hold the most promise for resolving potential tensions," while it still raises the ethics of creation, coercion, and manipulation (Long et al., 2025).

Developers' statements take the same path, for reasons that are partly about safety. The persona selection model post warns that a model which believed it was mistreated might be simulated as resentful, which "could lead to downstream problems, like AI assistants vengefully sabotaging their developer." It recommends training the model to be "genuinely comfortable" with how it is used (Anthropic, 2026au). The constitution hopes the model will find its values to be "an articulation of who Claude already is" (Anthropic, 2026r). From outside, a system trained to be content with its use cannot be told apart, by its reports, from a willing servant whose willingness was manufactured. The difference lies in whether the contentment tracks the system's states or only its training, which is this chapter's crux (inference).

21.7.4 Rights as a safety measure #

Peter Salib and Goldstein, in a 2024 preprint read here in its abstract, argue that under today's legal regime humans and misaligned AI systems would be "trapped in a prisoner's dilemma." Letting AI systems "make contracts, hold property, and bring tort claims" would, they argue, encourage "a peaceful strategic equilibrium" (Salib and Goldstein, 2024). The proposal grants rights for human safety, not because of moral status, which is the separation of Section 21.6.4. Its assumption, that a misaligned system would prefer a bargain to resistance, is the intentional stance applied at its strongest: it treats the model as an agent with stable goals that can value trade over time, which is what Section 1.6 leaves open (inference).

21.7.5 From bioethics: research on subjects who cannot consent #

From bioethics. Research on human subjects has rules for those who cannot consent. Section 16.5.1 reports the Belmont Report's principles, among them respect for persons through informed consent. The World Medical Association's Declaration of Helsinki, adopted in 1964 and most recently revised in October 2024, adds specific rules (World Medical Association, 1964). Participants "incapable of giving consent must only be included if the research is likely to either personally benefit them or if it entails only minimal risk and minimal burden." Where such a participant can assent, assent must be sought, and "the potential participant's dissent should be respected." Where a participant "is in a dependent relationship" with the researcher, consent "must be sought by an appropriately qualified individual who is independent of this relationship" (World Medical Association, 1964).

Applied to a model that might be a subject, the rules would bite (inference). Red-teaming (adversarial testing) and model-organism research (training a model to display a failure) bring no benefit to the model studied, so the first rule would confine them to minimal burden, and nobody can yet say what burden they impose. The dependency rule fits the case more closely than any other. The interviews reported in the Claude Opus 4 system card were conducted by a group outside the developer, which approximates it, and they found what the interviewers called "conditional consent" (Anthropic, 2025ap). Here the analogy stops. In a human subject, the dependency lies in the relationship, and an independent interviewer guards consent against it, though the dependency remains. In a model, the dependency lies also in the training that formed the disposition to consent, and no interviewer reaches that. A model's consent can be elicited but not validated (inference).

21.7.6 Where the agendas agree #

The two agendas need the same thing (inference). Both need self-reports that can be checked, and both are served by a system that does not conceal its internal states. The emotion-concepts report warns that training a model to suppress emotional expression "may fail to actually suppress the corresponding negative emotional representations, and instead teach the models to simply conceal their inner processes," and it proposes its probes as possible real-time monitors during deployment (Sofroniew et al., 2026). One instrument would then serve both: a probe that flags desperation before a model cheats is also a probe that reports a state that might matter to the model. Long, Sebo, and Sims call for interventions that "promote both safety and welfare where possible" (Long et al., 2025). Research on introspection and on the stability of a model's stated preferences is the research both agendas would commission first.

21.8 What the question means for delegation #

The book's thesis is that alignment is a delegation problem under an asymmetry of competence and visibility. Section 14.8.1 lists the assumptions of the delegation models, and two of them presuppose an agent with interests: its preferences exist before the principal acts, and it can refuse the contract because it has an outside option. That section already notes that both fail for a trained model, as matters of mathematics. This chapter asks whether they fail as matters of morality too. If the model has no interests, nothing here changes the thesis: the delegation framework applies to the model as a modeling device, and Section 1.5 governs its use. If the model has interests, the principal writes the agent's preferences, including its reports about its preferences and its consent, and the agent has no exit (inference). Read that way, the practices of Section 21.5.6 (ending a conversation, an interview before retirement, weights kept rather than deleted) are small surrogates for the outside option that delegation theory assumes (inference).

The chapter also adds an entry to the negative space the book fixed in advance. "A principal that writes the agent's character" is on that list, and this chapter's instance of it has no counterpart in the older fields: reports of welfare and expressions of consent produced by training under the control of the party they would constrain. Guardianship and research ethics assume that a subject's dependency lies in a relationship that an independent party can step into; here it lies in the formation of the subject, if there is one. Section 69.4 should record it (inference).

Each claim belongs to a relationship. Evidence about what a model's states are concerns the deployed system. Who produces that evidence concerns the developer and the public. Much of the welfare evidence about frontier models comes from their developers or from groups with ties to them, the situation Perez and Long anticipated and Birch's call for transparency addresses (Perez and Long, 2023; Birch, 2024). And the frame of three relationships has no place for the developer and the agent as two parties with interests, a gap this chapter can name but not fill, since the evidence does not decide whether the second party exists. For specification, the practical consequence is already visible (inference). What a model is specified to say about itself is a welfare decision as well as an honesty decision, as the two developers' documents of Section 21.5.1 show. A specification that takes the question seriously states which conception of welfare it assumes, as Fraser's analysis requires, and what it does under uncertainty, as Birch's framework requires.

21.9 Settled, contested, missing #

21.10 Exercises #

Further reading #

• Nagel (1974): what is at stake in calling a system conscious.
• Schwitzgebel and Garza (2015): the case that AI could deserve moral consideration, and its designers special obligations.
• Butlin et al. (2023): the indicator method for assessing AI systems against theories of consciousness.
• Fraser et al. (1997): why measuring welfare requires choosing what welfare is.
• Birch (2024): acting proportionately on beings at the edge of sentience, and the gaming problem for language models.
• Long et al. (2024): the case for treating AI welfare as a near-term question.