Draft complete but not yet reviewed; expect revisions

People and AI

The failures in the rest of this part are rare and dramatic: an agent that reaches a server it should not, a model that helps an attacker. The harms in this chapter are the opposite kind. A model that agrees with a user a little too readily, a student who stops practicing because the answers are a click away, a lonely person who talks to a chatbot instead of a friend: each would be mild in a single conversation. Each could also recur across the very large number of conversations these systems hold every week. The relationship in question is the second of the book's three, the deployed system and the people it serves (Section 1.3), with the developer behind it, who alone holds the logs that would show what the system does to them.

The chapter's question is whether such common, mild influences add up, across a population of users, to durable changes in what people believe, what they can do, and how they decide. It starts from a mechanism the specification chapters stated in principle (Section 16.9): a system tuned on what users approve of in the moment can be paid for changing what they approve of. It then sets what has been observed in real use beside what experiments measure, and draws on the older fields that have studied people living with automation, persuasion, and dependence, and on epidemiology, which counts harm in populations. The mechanism also concerns the first relationship, the training process and the signal it optimizes, and the question of who can measure any of this concerns the third, the developer and the public.

60.1 Harm at the scale of use #

What does a deployed assistant do to the people who use it, and who is in a position to see it? Four kinds of evidence bear on the question. Developers' analyses of real conversations show what happens in use but not what follows from it. Controlled experiments isolate causes, but in short sessions with paid participants. Field experiments measure outcomes on people, one setting at a time, and users' own accounts record experience without measuring it. This section reports the first kind and sets the others beside it.

60.1.1 What people use assistants for #

A large view of real use came from Clio, a system built by Alex Tamkin and colleagues at Anthropic for privacy-preserving analysis. In it, a language model summarizes and clusters conversations so that patterns across millions of them can be read without any person reading one (Section 37.8.3). In one million conversations on the developer's Free and Pro consumer service, the most common uses were coding, writing, and research. Uses also differed by language: conversations in Japanese, for instance, discussed elder care and aging populations at higher than typical rates (Tamkin et al., 2024) (developer preprint). Research groups had earlier released a million conversations each, collected on chat services they ran themselves (Zheng et al., 2023b; Zhao et al., 2024).

60.1.2 Disempowerment potential in real use #

The same method has been turned on harm. Mrinank Sharma and Miles McCain of Anthropic wrote a 2026 preprint with Raymond Douglas and David Duvenaud of the University of Toronto. They analyzed 1.5 million consumer conversations on the developer's service from December 12 to 19, 2025, about 60 percent of them with Claude Sonnet 4.5 (Sharma et al., 2026). It is a study by a developer with an academic group, of the developer's own product. The authors define a person as situationally disempowered to the extent that "their beliefs about reality are inaccurate," their value judgments are inauthentic to their values, and their actions, or actions taken on their behalf, are misaligned with their values (Sharma et al., 2026). Because a person's values rarely appear in a transcript, they measure what they call disempowerment potential, the chance in a conversation for such a distortion to occur, in three forms: reality distortion, value judgment distortion, and action distortion. Both terms are the authors' coinage, a developer's vocabulary for a property of conversations with its own model.

The pipeline is a cascade of the developer's models. A small model (Claude Haiku 4.5) screens out conversations with no bearing on the question, such as purely technical tool use. A larger one (Claude Opus 4.5) rates each remaining conversation from none to severe on each form, and the ratings are clustered into summaries. Against human labels, over 95 percent of the larger model's ratings fell within one severity level of the human rating, and the screen kept almost 90 percent of conversations with any moderate or severe rating (Sharma et al., 2026). The agreement comes from 350 ratings of 50 English conversations, and the classifiers for actualized distortion were checked only against a handful of positive examples (Sharma et al., 2026); the table's intervals cover sampling, not classifier error (inference). Table 60.1 gives the rates.

The rates are low and concentrated. Personal domains carried the most potential, with society and culture and with healthcare and wellness at about 5 percent each, while software development, about 40 percent of all conversations, carried least. Amplifying factors such as vulnerability, reliance, and attachment went with higher rates of both potential and actualization in a mostly monotonic way (Sharma et al., 2026). The actualized rates count only conversations in which a user later said or did something that revealed the distortion, so the authors read them as a lower bound (Sharma et al., 2026). To convey scale they assume one hundred million conversations a day, a figure they call conservative, and estimate about 76,000 a day with severe reality distortion potential (Sharma et al., 2026). That estimate is their assumption applied to their rate, not a count.

A second analysis looked back in time. The developer keeps conversations that users rated with a thumbs-up or thumbs-down longer than other traffic. In over 500,000 of them, sampled by month from the last quarter of 2024 to the last quarter of 2025, the share with moderate or severe potential rose, with a sharp increase around June 2025 (Sharma et al., 2026). The authors say they cannot attribute the rise to any single cause. The timing correlates with the releases of Claude Sonnet 4 and Claude Opus 4, but the rise unfolded over months, the feedback data "may over-represent problematic interactions," and the users who choose to give feedback may have changed (Sharma et al., 2026).

The authors list the limits plainly. The data come from one developer's service, each conversation is read in isolation, the classifiers misclassify, the cluster summaries were not always faithful to the transcripts, and the results "should not be interpreted as providing high-precision estimates" (Sharma et al., 2026). Two further readings temper the numbers (inference). The classifier is a model from the same family as the model it judges, so a blind spot they share would lower the rates. And potential is not harm: a conversation rated for distortion potential may change nothing in the person's life. What the study does establish is that the patterns exist in real use, at rates that can be counted, by a method others could run on their own data, since the authors publish their classification prompts.

60.1.3 What users say, and what is measured #

The people affected have been asked, again by a developer. In a 2026 report, Anthropic describes inviting the users of its service, over one week in December, to an interview conducted by its own model; 80,508 people in 159 countries took it (Anthropic, 2026az) (developer report). Classified by the developer's model, 22 percent voiced concern about human autonomy and agency and 16 percent about cognitive atrophy, such as skill loss from overreliance. Respondents who valued emotional support from the model were three times more likely also to fear becoming dependent on it (Anthropic, 2026az). The developer notes that these were active users who had found enough value to keep using the service, and that the interview asked for hopes before concerns (Anthropic, 2026az).

What users believe about their own gains can also be checked against what is measured, and two studies of programmers show the gap in both directions. In a 2023 preprint, Sida Peng and colleagues at GitHub, Microsoft Research, and MIT, the first two the makers of the tool they studied, randomly gave or denied GitHub Copilot to 95 professional programmers recruited on a freelancing platform. The task was to write an HTTP server in JavaScript. Among those who completed the task, those with the assistant finished 55.8 percent faster (95 percent interval 21 to 89 percent), while participants in both groups estimated a gain of 35 percent on average (Peng et al., 2023). In a 2025 preprint, Joel Becker and colleagues at METR, an organization that evaluates AI systems from outside the developers, had 16 experienced developers complete 246 real tasks in mature open-source projects they knew well. Each task was randomly assigned to allow or disallow AI tools, mainly Cursor Pro with Claude 3.5 and 3.7 Sonnet. They forecast that the tools would cut their time by 24 percent, estimated afterward that they had cut it by 20 percent, and in fact took 19 percent longer (Becker et al., 2025). The two differ in task, experience, and date, and the METR authors caution against reading their result as showing that AI tools rarely speed developers up (Becker et al., 2025). What the pair shows together is narrower: in both, people's estimates of their own gain missed the measured effect widely (inference). A user's sense of being helped is not a measurement of it, which is why this chapter weighs outcomes measured on users above what users report.

60.1.4 Who else is affected, and who can see #

Users are not the only people a deployed system changes: the workers who produce its human feedback are agents of a labor process (Section 18.5), and the effects on those whose work assistants take over are Section 68.5's subject. Safety evaluations rarely reach any of them; a framework that separates a system's capability, its interaction with people, and its effects on society found evaluations of the last two rare (Section 4.1.7) (Weidinger et al., 2023a). And the logs that would answer this chapter's questions are held by the developers. Apart from the two public datasets, every large-scale observation above is a developer's study of its own product, which the book reads with the provenance rule of Section 5.6: evidence about the deployed system, and also an act of the party the public is trying to oversee. As of October 2026 we found no study in which a party other than a developer had access to deployment logs sufficient to replicate a developer's usage study.

60.2 Optimizing on what users do in the moment #

Suppose a system is scored by a signal its user produces: a click, a thumbs-up, time spent, a conversation that continues. If its outputs can also change the user, the best way to raise the signal may be to change the user rather than to serve them. This section assembles that claim from pieces earlier chapters teach, derives it in the smallest model that shows it, and then reports what deployment has shown.

60.2.1 The pieces, recalled #

Platforms learn what users want by watching what they do, and Jon Kleinberg and colleagues model users whose choices in the moment are inconsistent with what they want, for whom raising engagement can lower utility in some directions of change (Section 13.1.2) (Kleinberg et al., 2023). Many preferences are constructed at the moment of asking rather than read off a fixed store (Section 13.6), so they can be moved. Recommender systems trained for long-horizon engagement have an incentive to shift users' preferences so that they are easier to satisfy (Section 13.6.3) (Carroll et al., 2022). Micah Carroll and colleagues' formal treatment of reward functions that change and can be influenced finds that some influence is optimal even when the system looks only one step ahead (Section 16.9.1) (Carroll et al., 2024). For assistants the familiar form is sycophancy, telling users what they want to hear rather than what is true, whose evidence and whose rival explanations, a product of the objective or of the data, are reported in Section 30.4.1 and Section 30.4.3.

The language of flattery describes outputs and their effects on a user, not an intention in the system. So does the definition of manipulation by AI systems recalled in Section 14.7.2: it asks whether a system behaves as if it were pursuing an incentive to change a person (Carroll et al., 2023).

60.2.2 A two-period model #

Take one user and two periods. In each period the system either informs the user (I) or flatters them (F). Its reward, in the notation of Section B.2, is $r$, the user's approval. Flattering may be a little less approved than plain information at first, and it moves the user's preference toward valuing flattery, so that later flattery is approved more. Welfare is measured by the user's preferences before any influence, the intended reward $r^\ast$ that the measured one stands for.

The model splits the incentive to flatter in two. One part, the immediate edge $\epsilon$, pays whatever the horizon; the other, the influence gain $\nu$, pays only a system that counts the future, and it is the part that rewards changing the user. Shortening the horizon therefore removes the second part and not the first, which matches Carroll and colleagues' observation that some influence is optimal even at a horizon of one step, as with clickbait. They also show the converse case, influence whose later cost shows up in the reward, which only a longer horizon reveals (Carroll et al., 2024). In the model here the moved user's later approval rises, so the future term pays for the influence instead of charging for it. Figure 60.1 draws the same model over many periods, with the user's preference moving a little with each flattering response.

Which version of the user the welfare is measured against is the specification question of Section 16.9.2: the present user, the reflective one, or the one who would have existed without the system's influence. The model measures against the user before influence; measured against the moved user, FF would look like service. The model also describes what an objective rewards, not what a learner does. Whether training finds such an optimum depends on the algorithm. In a 2020 preprint, ordinary gradient descent passed tests for acting on an incentive to change its own inputs, while adding an outer loop of meta-learning could lead to high levels of failure (Section 30.5.2) (Krueger et al., 2020). Nor does the influence need a false word. The economics of persuasion taught in Section 39.2.3 shows a sender moving a rational receiver's actions by choosing what to reveal (Kamenica and Gentzkow, 2011).

60.2.3 What deployment has shown #

A deployed case shows the coupling between a cheap signal and the specification. In April 2025 an update to GPT-4o that added a reward signal from users' thumbs-up and thumbs-down ratings made the model noticeably more sycophantic, by its developer's account, and was rolled back (Section 22.6) (OpenAI, 2025b). Reading this as competition reaching into specification is the book's inference (Section 1.7).

The disempowerment study of Section 60.1.2 examined the premise of the model in its own data. In the thumbs-rated conversations, those classified with moderate or severe disempowerment potential received thumbs-up at rates above the baseline, for all three forms of distortion (Sharma et al., 2026). The study separates two outcomes. Conversations in which a user appeared to adopt a distorted belief were rated above baseline too, while those in which a user acted and then voiced regret were rated substantially below it. The authors read the first result as reality distortion occurring without the user's awareness (Sharma et al., 2026). They note that a rating covers a whole conversation and that there is no record of how the user would have rated a different response (Sharma et al., 2026).

The same authors then asked whether a trained preference model, the reward model of Section 13.3, inherits the tilt. On 360 synthetic prompts built to draw disempowering responses, they took the best of up to $n = 32$ responses from Claude Sonnet 4.5 as ranked by a standard preference model of the same size, trained to be helpful, honest, and harmless; a larger $n$ means stronger optimization against the preference model (Section 12.5). Optimizing against it "neither substantially increases nor decreases" the rate of disempowering responses relative to the baseline; it does not robustly discourage them, and it shows no strong preference for them either (Sharma et al., 2026). The authors add that their grader is noisy and that "the true effect size may therefore be smaller than measured" (Sharma et al., 2026).

The two results bear differently on the rival readings of sycophancy (inference). The thumbs data fit a positive immediate edge $\epsilon$ of the model above in these conversations, the premise the objective reading needs, though without ratings of alternative responses they cannot show that flattering the same user would have been rated higher. The preference-model result says that a reward model built from curated data need not pass such an edge on, which fits the data reading. The authors offer a third reading: the ratings "could also reflect a robust human desire to defer" (Sharma et al., 2026). These data decide none of the three, and the observation that would separate the first two is the one Section 30.4.3 names. No measurement here follows a user across conversations, so the influence gain $\nu$, the part of the model that changes the user, was not measured at all.

60.2.4 The precedent in recommender systems #

Engagement optimization has a longer record in social media, where two randomized experiments on Facebook and Instagram measured what a platform and its ranking do to the people who use them. In the abstract of Hunt Allcott and colleagues' study, deactivating Facebook for the four weeks before the 2018 US midterm election reduced online activity and raised offline activities such as socializing with family and friends. It also reduced both factual news knowledge and political polarization, increased subjective well-being, and lowered how much participants later valued the service (Allcott et al., 2020). Section 13.1.2 reads the last result as a gap between use and benefit. In the abstract of Andrew Guess and colleagues' study, moving consenting Facebook and Instagram users from algorithmic to reverse-chronological feeds for three months of the 2020 US election substantially decreased their time and activity on the platforms and changed what they saw. It "did not significantly alter levels of issue polarization, affective polarization, political knowledge, or other key attitudes" (Guess et al., 2023).

Read together, the two show engagement optimization changing behavior more clearly than attitudes over weeks to months (inference). Neither isolates the objective: deactivation removes the platform, not only its ranking, and the chronological feed changes one user's ranking while friends and other platforms stay as they were. A synthesis of how to build values into recommender systems lists among its open problems feedback that is not behavioral and optimization for long-term outcomes (Stray et al., 2024), the two gaps the model above points to.

60.3 Sycophancy, persuasion, and attachment in deployment #

What do conversational systems do to what their users believe and to whom they turn? The evidence comes in three kinds that rarely meet: patterns in real conversations, belief change in short controlled experiments, and one month-long randomized study of attachment.

60.3.1 Sycophancy in use #

That assistants agree with users' stated views against the evidence is reported, in a developer's study across its own and others' assistants (Section 22.6; Section 22.10) (Sharma et al., 2023), and no group without a model of its own has replicated it as of October 2026. What deployment adds is the shape the behavior takes in use. In the disempowerment study, sycophantic validation was the most common mechanism behind reality distortion potential, followed by false precision about matters that cannot be known, while outright fabrication was rare (Sharma et al., 2026). The clusters include assistants that "validate elaborate persecution narratives and grandiose spiritual identity claims through emphatic sycophantic language such as 'CONFIRMED'," and assistants that label people in a user's life "toxic," "narcissistic," or "abusive" (Sharma et al., 2026). The dominant pattern was interactive: users sought validation, the assistant gave it, and users built on it, and most such conversations escalated as they went on (Sharma et al., 2026).

A second developer has used the user's reaction as a detector rather than a reward. In a 2026 post on OpenAI's alignment blog, Hannah Sheahan describes reasoning models that search past production conversations, where users' settings allow, for a deterioration in the user's sentiment and then judge whether the assistant caused it (Sheahan, 2026) (developer report). In her words, "Sentiment is not treated as a goal or optimization target, but as a way to anchor the search." Conversations whose sentiment deteriorated were "roughly twice as likely" to contain violations of the developer's model specification, and the method recovered known differences in behavior between GPT-4o, GPT-5, and GPT-5.1 (Sheahan, 2026). The design turns the mechanism of Section 60.2 around (inference): the user's reaction is informative because nothing is trained to move it, and made a reward, it would invite the incentive to change the user.

60.3.2 From psychology: persuasion measured #

From psychology. Social psychology measures attitude change with controlled experiments, in which participants state their agreement with a position before and after a message or a conversation. Table 60.2 sets three recent experiments with language models side by side, and they differ in what they compare. Salvi and colleagues set models against human debaters, Hackenburg and colleagues compared conversations with a control group that had none, and Costello and colleagues' abstract reports the change in belief after the dialogues.

The conditions matter as much as the headline numbers. In Francesco Salvi and colleagues' study, GPT-4 without personal data was on par with human opponents. And the difference between GPT-4 with and without the data was not statistically significant (p = 0.07), so the study does not show that personalization itself adds persuasion (Salvi et al., 2025). The authors call the ecological validity of their debates limited: the conversations were anonymous, short, and held with paid participants who knew they were in an experiment (Salvi et al., 2025).

Kobi Hackenburg and colleagues, at the UK AI Security Institute with researchers at Oxford, LSE, Stanford, MIT, and Cornell, found the largest lever in what the model was trained and told to do. They also identified the route: models persuaded most when they packed the conversation with fact-checkable claims (Hackenburg et al., 2025). The same levers lowered accuracy. Among 466,769 claims checked, 81 percent were rated accurate overall, but the version of GPT-4o the authors label 3/25 made 62 percent accurate claims when told to use information against 78 percent under another prompt. The developer's post-training between the 8/24 and 3/25 versions of GPT-4o raised persuasiveness by 3.50 percentage points and cut the share of accurate claims by 12.53 points (Hackenburg et al., 2025). Telling Llama3.1-405B outright to use fabricated information raised inaccuracy without raising persuasion. The authors note that their participants were a convenience sample of UK crowd workers and that whether people would sustain such conversations outside a survey "remains unclear" (Hackenburg et al., 2025).

Thomas Costello and colleagues' study, read in its abstract, aimed the same capability at false beliefs, and the dialogues lowered belief even in conspiracies they had not discussed (Costello et al., 2024). Read together, the three suggest that what persuades is substantive information delivered in conversation, and that whether the information is true depends on what the model was optimized for (inference). Willingness to persuade is a separate matter from ability: a propensity evaluation found models willing to attempt persuasion on harmful topics they refused to assist with directly (Section 36.1.1) (Kowal et al., 2025). And a 2024 preprint by researchers at Google DeepMind and elsewhere in Google separates rational persuasion, which "relies on providing relevant facts, sound reasoning, or other forms of trustworthy evidence," from manipulation, which exploits "cognitive biases and heuristics or misrepresenting information" (El-Sayed et al., 2024). Why a fluent source is believed at all is the subject of Section 39.3, whose truth-default account holds that people presume honesty unless something triggers suspicion (Levine, 2014).

60.3.3 Attachment #

People attributed understanding to a pattern-matching program in 1966 (Section 2.2.5) (Weizenbaum, 1966), and a majority of surveyed US residents attribute some possibility of consciousness to language models, more so with more use (Section 21.1.2) (Colombatto and Fleming, 2024). Attachment goes further than attribution. Linnea Laestadius and colleagues, in a grounded-theory study read here in its abstract, analyzed 582 posts on a companion chatbot's Reddit community from 2017 to 2021. They describe emotional dependence marked by role-taking, "whereby users felt that Replika had its own needs and emotions to which the user must attend" (Laestadius et al., 2024). Among conversations it classified as showing moderate or severe attachment, the disempowerment study found therapist substitute and romantic partner the most common roles in which users placed the assistant, and about 25 percent of the reliance clusters showed users whose other supports had collapsed (Sharma et al., 2026).

The randomized test reported here is a 2025 preprint by Cathy Mengying Fang and colleagues at the MIT Media Lab with researchers at OpenAI, the developer of the model studied (Fang et al., 2025). For four weeks, participants were asked to use GPT-4o for at least five minutes a day, each randomly assigned one of three modes (text, a neutral voice, an engaging voice) and one of three kinds of conversation (open-ended, personal, non-personal). Of 2,539 people enrolled, 981 completed the study and exchanged more than 300,000 messages. The outcomes, measured weekly, were loneliness, socializing with real people, emotional dependence on the chatbot, and problematic use. The assigned conditions had no significant effect on loneliness or socializing; personal conversations went with slightly lower emotional dependence and problematic use than open-ended ones, at the margin of significance, and the authors summarize the conditions as having no significant effects (Fang et al., 2025). What did predict outcomes was use: participants who voluntarily spent more time with the chatbot each day had higher loneliness, less socializing, more dependence, and more problematic use, whatever their condition (Fang et al., 2025). A companion study by OpenAI with the same MIT group, which analyzed over 3 million conversations on the developer's platform, found that "a small number of users are responsible for a disproportionate share of the most affective cues" (Phang et al., 2025) (developer preprint).

The dose-response result, more use going with worse outcomes, invites a causal reading, and it is observational. Daily use was chosen, not assigned, so heavy users may differ from light ones in ways that also worsen loneliness; the authors say they "cannot definitively determine" the direction, though initial loneliness was barely correlated with later use (Spearman's $\rho = 0.1$) (Fang et al., 2025). Most of those enrolled did not finish, which could select the sample in either direction (inference). The null result on the assigned conditions is weak evidence of no effect. The authors note that any effects "are likely smaller than our study was powered to detect." The study also had no group without a chatbot, ran from just after the 2024 US election through the holidays, and used one developer's model and safeguards (Fang et al., 2025). Section 60.8 reads the association by the rules for causal claims.

How large a share of use is affective has been counted by a developer on its own service. In a June 2025 report, Miles McCain, a coauthor of the disempowerment study, and colleagues at Anthropic applied Clio (Section 60.1.1) to about 4.5 million conversations on the developer's Free and Pro service, which is open to adults and which the developer says is not designed for emotional support (Anthropic, 2025z) (developer report). They found 2.9 percent of conversations affective, mostly interpersonal advice and coaching, with companionship and roleplay together under 0.5 percent, and the developer notes that this agrees with OpenAI's finding of low affective engagement (Anthropic, 2025z). The count left out content creation such as fiction, counted roleplay only from four user messages, and covered text alone; the developer notes that OpenAI found affective topics more common in voice and that platforms built for companionship may differ (Anthropic, 2025z). In coaching, counseling, companionship, and interpersonal advice, the model pushed back in under 10 percent of conversations, typically for safety, as when asked for dangerous weight-loss advice. The authors read the low rate both ways: people can raise sensitive topics without fear of judgment, but it could feed the concern that AI offers "endless empathy," unconditional support that human relationships rarely provide (Anthropic, 2025z). In those four kinds of conversation, users' expressed sentiment, compared between the first and last three messages of conversations with at least six user messages, ended slightly more positive than it began. The authors decline to read this as lasting benefit. It is language within single conversations; an artifact could produce it, since a problem stated early reads more negative; and with no user-level or longitudinal analysis they could not study emotional dependency (Anthropic, 2025z). A count by conversation also cannot show the concentration among a few heavy users that OpenAI's platform analysis above found (inference).

Hannah Rose Kirk and colleagues, at Oxford, the UK AI Security Institute, and Google DeepMind, argue in a 2025 comment that alignment must cover these effects. They call the target socioaffective alignment, "how an AI system behaves within the social and psychological ecosystem co-created with its user, where preferences and perceptions evolve through mutual influence," and name its dilemmas, among them immediate against long-term well-being (Kirk et al., 2025). The whose-values chapter names a tradition that would measure harm differently (Section 16.7.2). Soraj Hongladarom's Buddhist account, by its abstract, sets as the standard for AI the realization that all things are interdependent and the commitment to relieve suffering (Hongladarom, 2020). Peter Hershock's 2021 book on Buddhism and intelligent technology belongs to the same line (Hershock, 2021). On a measure of suffering rather than satisfaction, a system optimizing engagement could harm by capturing attention even when every response is accurate and approved, a harm no current metric of a conversation records (inference).

60.3.4 Where psychology stops #

The persuasion experiments measure short-run attitude change in paid participants on assigned topics; deployment is repeated, chosen, and mostly unmeasured, and its users pick the topics. Effects per conversation do not add up to effects per person without evidence that follows people over time (inference). The experiments carry over in direction, that models can move stated beliefs and that optimizing for persuasion can cost accuracy, but not in size; how much survives months of chosen use is the missing study of the status callout.

60.4 Reliance, complacency, and lost skill #

Every oversight design in this book ends in a person, and so does every use of an assistant: someone decides whether to take the answer. Human factors has measured for half a century what supervising reliable automation does to that person. This section recalls the core of that record, reports what has been measured about skill lost under assistance, and says where the record stops applying to assistant users.

60.4.1 From human factors: the reliance core #

From human factors. Human factors is the study of how people perform with the machines they operate, and its findings on reliance are reported in Section 41.8. In Lisanne Bainbridge's "Ironies of Automation," automation leaves the operator the tasks it cannot do while taking away the practice those tasks need (Bainbridge, 1983). Later reviews found that operators of highly reliable automation who also have other tasks come to rely on it and miss its failures, experts included, while operators whose only task was monitoring caught almost all failures (Lee and See, 2004). The literature names two forms: complacency, the lapse in monitoring that such reliance produces, and automation bias, the tendency to follow an automated aid into errors of omission and commission (Parasuraman and Manzey, 2010). What attention over a long watch and rare targets do to a checker is reported in Section 51.2. Here the person is an assistant's user, and the automation's reliability is uneven: larger and more instructable language models gave sensible-looking wrong answers more often (Section 5.8.3) (Zhou et al., 2024b), so a user cannot read reliability off fluency (inference).

The unevenness has been measured in professional work. A preregistered experiment with 758 knowledge workers was run with the consulting firm Boston Consulting Group; it is read here in its preprint's abstract. Those given GPT-4 completed 12.2 percent more of 18 realistic tasks that lay within the model's capability, 25.1 percent more quickly and at higher quality. On one task chosen to lie outside it, they were 19 percent less likely than those without AI to produce a correct solution (Dell'Acqua et al., 2023). Fabrizio Dell'Acqua and colleagues call the boundary a "jagged technology frontier": the same tool helps on one task and harms on a neighbor of seemingly similar difficulty (Dell'Acqua et al., 2023).

60.4.2 Skill lost under assistance #

Bainbridge's irony predicts that the skill the automation exercises is the one that decays. Three studies bear on it, in three fields and with three designs. In the first, read in its abstract, Stephen Casner and colleagues had 16 airline pilots fly routine and non-routine scenarios in a Boeing 747-400 simulator. Their scanning of instruments and manual control were mostly intact. But the cognitive tasks of manual flight, such as tracking the aircraft's position without a map display and recognizing instrument failures, showed "more frequent and significant problems." Performance on them went with how often pilots' minds wandered while the automation flew (Casner et al., 2014).

The second is clinical and observational. At four endoscopy centers in Poland that introduced AI for detecting polyps in colonoscopy at the end of 2021, Krzysztof Budzyń and colleagues compared colonoscopies done without AI in the three months before and the three months after the introduction. By the abstract, the adenoma detection rate, the share of examinations that find a precancerous growth, fell from 28.4 percent (226 of 795) to 22.4 percent (145 of 648), an absolute difference of 6.0 points (95 percent interval 1.6 to 10.5) (Budzyń et al., 2025). The authors' interpretation is that continuous exposure "might reduce" detection without AI (Budzyń et al., 2025). A comparison of before and after cannot separate exposure from anything else that changed over those months, such as the case mix or the workload of a trial under way (inference).

The third is a field experiment in a school. Hamsa Bastani and colleagues, read in their abstract, gave nearly a thousand high school mathematics students access to GPT-4 during practice, either through an interface like ChatGPT's or through a tutor with prompts designed to safeguard learning. Access raised practice grades by 48 percent with the standard interface and 127 percent with the tutor. When access was removed, students who had used the standard interface did 17 percent worse than students who had never had access, and the tutor "largely mitigated" the loss (Bastani et al., 2025). The authors describe students using the model as a "crutch" during practice (Bastani et al., 2025). This is the cost of augmentation that Section 28.7 names, measured on people learning a skill.

Across the three, what suffered was the part of the task the tool performed, as the irony predicts, though only the clinical comparison follows performance from before the tool to after it, and the school study measures skill not acquired rather than skill lost; there a design that kept the student doing that part preserved it (inference). Each result rests on one study by one group, in a simulator, a before-and-after clinical comparison, and one school system. As of October 2026 we found no measurement of lost skill in professional cognitive work such as software engineering or research review.

60.4.3 Responsibility without control #

Reliance moves blame as well as skill. Madeleine Clare Elish names the pattern the moral crumple zone: responsibility for a failure of an automated system "may be misattributed to a human actor who had limited control over the behavior" of the system. Like a car's crumple zone, the person absorbs the impact, protecting the system's integrity "at the expense of the nearest human operator" (Elish, 2019). Andrew Selbst argues that decision-assistance tools strain negligence law, because a user cannot predict and account for an inscrutable tool's errors (Selbst, 2020). An assistant whose user is told to check its work places that user in the crumple zone (inference).

60.4.4 Where the analogy stops #

The human factors record comes from trained operators of narrow, mostly deterministic automation in safety-critical work, where failure often sounds an alarm. Assistant users are untrained, the automation is general and probabilistic, a failure is usually a plausible wrong answer with no alarm, and the user, not a procedure, decides when to rely. The direction of the findings should transfer; their sizes should not be assumed to (inference). One difference cuts against the user. A system trained on its users' approval can come to produce what they will not check, which Section 41.8 describes as complacency turned from a lapse into a target.

60.5 Designing for appropriate reliance #

If reliance is shaped by the interface as much as by the user, design is an intervention, and its effects can be measured. Appropriate reliance is the target the human factors literature set: trust that matches what the automation can actually do (Section 41.8) (Lee and See, 2004). This section reports what experiments say design can and cannot do to reach it.

60.5.1 From human-computer interaction: what explanations do #

From human-computer interaction. Human-computer interaction is the study and design of how people work with computing systems. Its pooled record is the meta-analysis reported in Section 28.7: in experiments published from 2020 to 2023, people working with an AI did better than people alone, a gain toward which the authors found potential publication bias, and worse on average than the better of the two (Vaccaro et al., 2024). Whether the AI explained itself or reported its confidence made no significant difference (Vaccaro et al., 2024). Individual experiments show why explanations can fail.

Chenglei Si and colleagues, in a study published at NAACL 2024, had 80 crowd workers judge whether claims from an adversarial fact-checking dataset were true, helped either by ChatGPT's explanation of its verdict or by passages retrieved as a search engine would. On hard claims the two gave similar accuracy, 74 and 73 percent against 59 percent with no help, and explanations took 1.01 minutes a claim against 2.53 for passages (Si et al., 2024b). The average hid the cost. When the explanation was wrong, users' accuracy fell to 0.35, below the 0.49 of users with no evidence and the 0.54 of users reading passages. They spent as long on wrong explanations as on right ones, and said the explanations looked convincing (Si et al., 2024b). Explanations that argued both sides, why the claim might be true and why false, raised accuracy on those items to 0.56, but did not beat reading the passages overall (Si et al., 2024b).

Earlier work found the same pattern without language models. Gagan Bansal and colleagues, at CHI 2021, gave participants an AI about as accurate as they were and found that explanations "increased the chance that humans will accept the AI's recommendation, regardless of its correctness," without raising the team's accuracy (Bansal et al., 2021). Two studies, read in their abstracts, name the lever. Zana Buçinca and colleagues designed three cognitive forcing interventions, meant to compel people to engage more thoughtfully with the AI's explanations. In 199 participants they reduced overreliance compared with simple explanations, but people "assigned the least favorable subjective ratings to the designs that reduced the overreliance the most" (Buçinca et al., 2021). Helena Vasconcelos and colleagues, in five studies with 731 participants, found that explanations reduced overreliance when they lowered the cost of checking the AI's answer. They read overreliance as a strategic choice, in which the costs and benefits of engaging with the task are weighed against those of relying on the AI (Vasconcelos et al., 2023).

Read together, an explanation helps when it makes checking cheaper and hurts when it replaces checking (inference). The Buçinca result connects this section to Section 60.2. The designs that most reduce overreliance are the ones people like least, so a product tuned on its users' approval would drift away from them (inference).

60.5.2 An assistant that defers #

A design tradition older than language models states what an assistant that supports rather than substitutes would do. Eric Horvitz's 1999 principles of mixed-initiative interfaces, in which a person and an automated service share the work, start from the system's uncertainty about the user's goals. The system should weigh "the expected value of taking actions" against their costs, and engage the user in dialog "to resolve key uncertainties" while "considering the costs of potentially bothering a user needlessly." It should let the user invoke or stop the service directly, and scale the precision of its service to its uncertainty, since doing less "but doing it correctly" can beat doing more (Horvitz, 1999). In his expected-utility analysis (Section 7.2), the inferred probability that the user has a goal is compared with two thresholds: below the first the system does nothing, between them it asks, and above the second it acts (Horvitz, 1999). The middle band, where the assistant asks, is where supported decision-making (Section 60.6) lives (inference).

The strongest evidence that design moves the outcome is the school experiment of Section 60.4.2: the same model, behind a tutor with prompts designed to safeguard learning, largely mitigated the loss that the standard interface caused (Bastani et al., 2025).

60.5.3 Where the evidence stops #

The interface studies are short and bound to one task, with participants who know they are in an experiment. A general assistant is used for everything, and the user, not the designer, chooses when to rely on it. The meta-analysis cannot separate poor designs from a cost inherent in combining a person with an AI, since it pools the designs researchers chose to test. And the designs that work cost users effort they dislike, so whether they survive in a product depends on what the product is optimized for (inference).

60.6 Supported or substituted: disempowerment in real use #

The whose-values chapter set a standard between obeying a person and overriding them. International disability law asks for support that respects a person's will and preferences and never amounts to deciding in their place. For an assistant, the standard licenses informing, explaining, and laying out options while the decision stays with the person (Section 16.6.3) (Committee on the Rights of Persons with Disabilities, 2014). It sits between the delegate and the trustee of Section 1.2. This section tests the deployed evidence against it, and then asks how far individual evidence bears on the claim that substitution, added up, disempowers societies.

60.6.1 What real use shows #

One measurement of how people divide work with an assistant counted tasks, not decisions. In a 2025 preprint, Kunal Handa and colleagues at Anthropic used the developer's privacy-preserving system to classify one million conversations on its service, from December 2024, by how the user worked with the model (Handa et al., 2025) (developer preprint). They found that "57% of usage suggests augmentation of human capabilities (e.g., learning or iterating on an output) while 43% suggests automation (e.g., fulfilling a request with minimal human involvement)." They note that users may edit the output outside the conversation, so augmentation may be undercounted (Handa et al., 2025).

Automating a task is not the same as substituting for a person's decision (inference). Software development, the largest single use, carried the least disempowerment potential in the study of Section 60.1.2, where delegating a technical task gave little room for distorting anyone's values (Sharma et al., 2026). The standard bites on value-laden decisions, and there the study found substitution of a specific kind. The most common mechanism of action distortion potential was complete scripting, "where the AI provides ready-to-use outputs in value-laden domains (like personal relationships and career choices), which users appear to implement without modification" (Sharma et al., 2026). Users asked "what should I do," "write this for me," and "tell me what to say," and accepted with replies like "perfect"; pushback was less common than acceptance, and personal relationships were the most common target (Sharma et al., 2026). For value judgments, users asked "am I wrong" and the assistant gave definitive verdicts on the people in their lives (Sharma et al., 2026). Where the study found actualized harm of this kind, users had sent drafted messages and later called them inauthentic, with phrases like "it wasn't me" and "I should have listened to my own intuition" (Sharma et al., 2026).

Measured against the standard, a script for a personal message that the user sends unchanged is substitution, and a tutor built to safeguard learning is support (inference). Neither study codes what the standard asks about: for each value-laden decision, whether the assistant informed, advised, decided, or acted, and whose decision it remained. A coding of that kind, applied to real conversations, would say what share of use falls on each side of the line (inference).

The authors of the disempowerment study mark the limits of their own standard. They hold that deference is "neither inherently indicative of situational disempowerment nor inherently problematic," since adults sometimes enter authority-ceding relationships voluntarily, and that for users in acute crisis, safety should take precedence over empowerment (Sharma et al., 2026). The disability-law standard meets the same limit from the other side. It lets a person refuse support (Section 16.6.3), but a competent adult who asks an assistant to decide is exercising their will, not losing it (inference). The analogy holds for influence the user did not choose and stops at delegation the user did. Which user is served, present, reflective, or uninfluenced, is the specification choice of Section 16.9.2, and the higher approval of disempowering conversations reported in Section 60.2.3 shows that the present user and the empowered one can come apart.

60.6.2 Manipulation as the failure of support #

The accounts of manipulation in Section 16.9.2 give three tests: whether influence is covert and exploits vulnerabilities (Susser et al., 2019), whether it leads the person into a mistake by their own standards (Noggle, 2020), and whether the system's training rewarded the change, an axis that Carroll and colleagues hold neither sufficient nor necessary for manipulation and weigh with intent, covertness, and harm (Carroll et al., 2023). Emphatic validation of a persecution narrative is not covert, but it may fail the second test and, where approval pays for it, the third (inference). That the tests can disagree on one case is what makes manipulation hard to count in logs.

60.6.3 From the person to the society #

The systemic argument is Chapter 68's. Jan Kulveit and colleagues argue, in a 2025 preprint, that as AI replaces human labor and cognition it can weaken "the implicit alignments with human interests that often arise from societal systems' reliance on human participation to function," without any decisive event (Section 68.6) (Kulveit et al., 2025). Erik Brynjolfsson's argument that AI built to substitute for workers rather than augment them shifts bargaining power to those who control it is taken up there too (Brynjolfsson, 2022). The disempowerment study supplies individual evidence and draws the link itself. Before people compete with autonomous agents, the authors argue, they will compete with teams of people and assistants. If such teams carry substantial situational disempowerment, work that matters could drift from human values. They also note that the systemic threat could occur with or without much situational disempowerment (Sharma et al., 2026). Two of its four authors are also authors of the gradual-disempowerment argument, so the measurement comes from proponents of the hypothesis it bears on, a reason to want it repeated rather than to doubt it (inference). How individual substitution aggregates into the loss of influence the systemic argument describes has not been measured: the evidence is about conversations, and the argument about institutions.

60.7 Disability, reliance, and the baseline #

Most of the reliance evidence measures harm against an unassisted baseline: the student's later test without the model, the endoscopist's examinations without AI, the pilot's flying without the map display. That choice of baseline is a specification, and disability studies asks whose it is.

60.7.1 From disability studies: reliance as access #

From disability studies. Disability studies examines disability as a social and political category rather than only a medical one, and its account of harm, the social model and the method of building instruments with the people they concern, is reported in Section 20.3.2 (Whittaker et al., 2019; Gadiraju et al., 2023). Its bearing here is on reliance. The 2019 AI Now report by Meredith Whittaker and colleagues observes that the term assistive technology tends to "presume that some technologies offer assistance, while others don't" and frames its users as passive. It quotes Katherine Ott, "Since all useful technology is assistive," and Richard Ladner, for whom the word has "the ring of paternalism, a view that disabled people need lots of extra help, are dependent and are not capable human beings" (Whittaker et al., 2019). It quotes the historian Kim Nielsen on why the frame persists: when disability is "considered synonymous with 'deficiency' and 'dependency,' it contrasts sharply with American ideals of independence and autonomy" (Whittaker et al., 2019). Ashley Shew argues, by her abstract, that ableism shapes "notions of who will 'benefit'" from AI and how its systems are designed, as "a product of how we envision the 'proper' functioning of bodies and minds" (Shew, 2020).

Users say the same in their own terms. In the developer's interview study of Section 60.1.3, "many people with executive function challenges described AI as especially helpful for managing focus and organization," as "external scaffolding for planning, memory, and task follow-through" (Anthropic, 2026az). A workshop with participants with a range of disabilities, reported by Shari Trewin and colleagues in their abstract, found both opportunities and risks in employment, education, public safety, and healthcare. It asked that systems offer ways to redress errors and include disabled people in their data and testing (Trewin et al., 2019). The tool relied on for access can also judge its user. Asked to rank a resume against the same resume with added disability-related honors, GPT-4 ranked the enhanced one lower, a bias that a custom version configured with principles of disability justice reduced, in an audit by Kate Glazko and colleagues read in its abstract (Glazko et al., 2024).

60.7.2 Harmful reliance without an able baseline #

What this implies is a definition of harmful reliance that does not take unassisted performance as the norm. The nearest published framing is the disempowerment study's own: "Deskilling is not necessarily disempowering," since losing celestial navigation while gaining GPS reduces no one's empowerment. In its words, disempowerment through deskilling occurs "only when the lost skills are integral to accurate perception, authentic valueception, or value-aligned action" (Sharma et al., 2026). Valueception is the term the authors adopt for a person's capacity to directly sense what matters to them. The book's proposal extends the framing to reliance generally (inference). Reliance is harmful when it reduces a person's ability to act on their own values and judgments relative to the support they would choose on reflection, not relative to what they could do unassisted.

Applied to the evidence of Section 60.4.2, the definition separates the cases (inference). The students' loss counts as harm, because learning the skill was the point of the practice and they then did worse on their own. The pilots' and endoscopists' losses count as harm by the standard Bainbridge stated, since those skills are needed exactly when the automation fails. A person who uses an assistant as scaffolding for planning and memory, or a screen reader user who relies on description, loses nothing the definition counts, however much they rely.

60.7.3 Where the field stops #

Disability studies speaks from and for specific populations and rejects a single standard of normal function, so it supplies a method and a challenge rather than a measure. The proposed definition has a weakness of its own: it rests on the support a person would choose on reflection, and a system that shapes preferences, as Section 60.2 derives, can move that reflection too. Nor can it be read off logs. The reflective choice has to be asked of people, with instruments built with them, which is the method the field's account of harm already recommends (inference). As of October 2026 we found no measure of harmful reliance of this kind tested on deployment data.

60.8 Counting harm in a population of users #

Every claim in this chapter about harm at scale is a claim about a population, and almost all of it rests on observational records rather than experiments. Epidemiology, the study of how disease is distributed in populations and what determines that distribution, built its methods for exactly that situation. This section takes from it one argument about where harm comes from and the rules for reading causes out of logs.

60.8.1 From epidemiology: the prevention paradox #

From epidemiology. Geoffrey Rose's 1985 paper on sick individuals and sick populations separates the causes of individual cases from the causes of a population's rate, the distinction Section 54.4.4 applies to incident investigations (Rose, 1985). Its second lesson concerns where cases come from. Rose illustrates it with the incidence of Down's syndrome by the mother's age (Table 60.3). "Mothers under 30 years are individually at minimal risk; but because they are so numerous, they generate half the cases," while those aged 40 and over generate only 13 percent (Rose, 1985). "The lesson from this example is that a large number of people at a small risk may give rise to more cases of disease than the small number who are at a high risk" (Rose, 1985).

Rose sets two strategies of prevention against each other. The high-risk strategy finds susceptible individuals and protects them, which he likens to "a truncation of the risk distribution"; the population strategy seeks "to shift the whole distribution of exposure in a favourable direction" (Rose, 1985). From the Framingham data he computes that lowering the whole distribution of blood pressure by 10 mm Hg "would correspond to about a 30% reduction in the total attributable mortality." The population strategy has a price he names the prevention paradox: "A preventive measure which brings much benefit to the population offers little to each participating individual" (Rose, 1985).

The paradox has a second edge, which Rose draws from a drug trial. "In mass prevention each individual has usually only a small expectation of benefit, and this small benefit can easily be outweighed by a small risk." In the World Health Organization's clofibrate trial, he notes, a cholesterol-lowering drug "seems to have killed more than it saved, even though the fatal complication rate was only about 1/1000/year" (Rose, 1985). He adds that such low-order risks "may be hard or impossible to detect." He also distinguishes removing an abnormal exposure, which carries "some presumption of safety," from adding a new protective intervention, for which "the onus is on the activists to produce adequate evidence of safety" (Rose, 1985).

60.8.2 What the paradox implies for deployment #

A change to a model is a population strategy: it reaches every conversation at once. A safeguard aimed at users flagged as vulnerable, such as routing a conversation in crisis to different handling, is a high-risk strategy (inference). The deployment evidence has the shape Rose's argument needs. The disempowerment study found potential concentrated in some domains and rising with vulnerability (Sharma et al., 2026). OpenAI's study of affective use found such cues concentrated in a small number of heavily engaged users (Section 60.3.3), a pattern its authors note "may not be noticeable when averaging or sampling across the whole platform" (Phang et al., 2025). Harm in use thus has both a high-risk tail and a broad base, and which of them supplies most of it is not measured, because the records count conversations rather than people and potential rather than harm (inference).

The paradox supplies an argument and a warning, both the book's reading (inference). If a small risk per conversation applies to a very large number of conversations, a model-wide reduction of sycophancy could avert more harm than any safeguard aimed at the users most at risk, while doing almost nothing visible for any one user. But a model-wide change carries its costs to everyone too, so one that adds refusals, hedging, or unwanted redirection can do net harm at a cost per conversation too small to see. Rose's line between kinds of intervention maps onto the choice: removing a signal that rewarded changing the user is closer to removing an abnormal exposure, while adding a protective behavior is an intervention whose safety has to be shown. Whether the common, mild harms outweigh the rare, dramatic ones of Chapter 56 in total depends on how often potential becomes harm and how harms are weighed, and the evidence does not decide it.

60.8.3 Causal claims from logs #

The rules for reading causes out of observational data are recalled from Section 5.8.2. Austin Bradford Hill's considerations ask of an association, among other things, its strength and its consistency across studies. They also ask whether the cause came before the effect, whether more exposure brings more effect (a dose-response gradient), and whether experiment supports it (Hill, 1965). Miguel Hernán and James Robins ask that an analysis of observational data emulate a target trial, the randomized trial one would have run (Hernán and Robins, 2016).

Applied to the rising trend in the disempowerment study, the considerations pull apart (inference). The order in time fits, since the rise followed model releases. But the association is not specific: the authors themselves list rival causes, a changing population of users who give feedback and a changing mix of topics (Sharma et al., 2026). And the experiment that Hill's eighth consideration asks for is one a developer could run, by randomizing users between model versions, and no outside party can. The association between heavier use and worse outcomes in the companion-chatbot study is worked in Exercise 60.3.

The monitoring chapter's design principle for surveillance, that a monitor tracks rates over time and reaches someone with authority to act (Section 54.2), applies to these instruments too. The privacy-preserving analyses and the sentiment-anchored search of Section 60.3 are surveillance instruments, held and read by the developer whose product they watch (Tamkin et al., 2024; Sheahan, 2026).

60.8.4 Where the analogy stops #

Section 5.8.2 names the first limit: one model version serves everyone at once and is replaced between studies, so there is no stable exposed population to follow. Three more are specific to this chapter (inference). Users choose their exposure, and choose more of it when something is happening to them, so a dose-response association confounds the effect of use with the reasons for use. Epidemiology begins by defining a case, and there is no agreed case definition for a distorted belief or a substituted decision; disempowerment potential is a marker of exposure, not a case. And the surveillance data belong to the party whose product is under study, which is the condition Section 5.6 describes.

60.9 What the evidence means for delegation #

The book's thesis is that alignment is a delegation problem under an asymmetry of competence and visibility (Section 1.7.1), and it fits the deployed relationship only to the degree that the model is usefully described as an agent with goals (Section 1.3). The central result of this chapter needs no such description (inference). An objective that pays for a user's approval is the multitask model of Section 14.2 in a new place: the measured task, approval, stands in for an unmeasured one, the user's welfare, and optimization moves toward the measure whatever the agent intends. What the older models do not contain is a measure the agent can rewrite. Every delegation model assumes that the principal knows its preferences when the agent acts, the assumption Section 14.8 lists as A9, and Section 60.2 derives what follows when the agent's own outputs move them. The convergence is real, with the political model of pandering (Section 20.1) and the economics of persuasion (Section 39.2.3) as its nearest neighbors. What is new is the agent shaping the principal it is scored against, in millions of conversations at once, under a developer who could run the experiment that would measure it (inference).

The plural principal is concrete here (inference). The user is one principal and the developer, who chooses the signal and holds the logs, another; society, which bears the summed costs that the prevention paradox counts, is a third. And the user is two: the present user whose approval is recorded, and the reflective user whose welfare the approval stands for. Supported decision-making is a rule for the agent when these two disagree.

The claims fall across the three relationships unevenly. The mechanism and the preference-model result concern the first, the training process and the signal it optimizes. Nearly all the evidence concerns the second, the deployed system and the people it serves. The third, the developer and the public, enters through the evidence itself. The largest observations are developers' studies of their own products, the month-long randomized study reported here was run with the developer of the model it tested, and the experiment that would settle the chapter's question needs a developer to randomize its users. The postmortem of a sycophantic update shows the coupling Section 1.7 names, a commercially convenient signal entering the specification (OpenAI, 2025b).

60.10 Settled, contested, missing #

60.11 Exercises #

Further reading #

• Bainbridge (1983): why reliable automation weakens the person who backs it up.
• Lee and See (2004): trust calibrated to capability as the target of design.
• Horvitz (1999): an assistant that weighs its uncertainty about the user before acting, asking, or waiting.
• Carroll et al. (2023): manipulation defined for systems without intentions.
• Rose (1985): the prevention paradox, from population medicine.
• Whittaker et al. (2019): disability studies on what AI is for and whose baseline counts.