Draft complete but not yet reviewed; expect revisions

Measuring What People Value

The two previous chapters asked whose values a system should serve and how the inputs of many people can be combined into one target. Both questions assume that values can be read off people. In practice they are read off through instruments: a questionnaire, a rating scale, a choice between two responses, a label entered by a contractor working through a queue. Each instrument records something, and this chapter asks what. Psychology has mapped the structure of human values and studied when judgments can be trusted. Psychometrics, the science of psychological measurement, has a vocabulary for asking whether a score measures what it claims to, which Section 5.2 teaches.

Three observations frame the chapter. Preferences are often made at the moment they are asked for (Section 13.6), so the form of a question is part of its answer. The people who answer are not a neutral instrument: where their working conditions have been studied, raters on crowd platforms are paid by the task, those at outsourcing firms are measured on their output, and both are scored on agreement. That is the multitask problem of Section 14.2, with careful judgment as the unmeasured task. And when the same instruments are turned on a model, every problem of survey research returns, with one more: a respondent that may have read the questionnaire. Most of the chapter concerns the first of the book's three relationships, the training process and the signal it optimizes, since preference data are that signal. The measurements of models concern the second, the deployed system and the people it serves. Several studies are developers' reports on their own raters and models, which brings in the third, the developer and the public.

18.1 The structure of human values #

A specification has to say which values yield to which when they conflict, and a measurement of a model's values needs a reference to compare against. A prominent map of what people value supplies both, and its limits for a model are part of what it teaches.

From psychology. The map is Shalom Schwartz's theory of basic human values, whose 1992 statement reports, in its title, empirical tests in 20 countries (Schwartz, 1992). That statement could not be read for this book. The account below is the summary given by Naama Rozen and colleagues, researchers at Tel-Aviv University, two of them also at Google, in a 2025 paper at ICLR that compares language models with the theory (Rozen et al., 2025).

In their summary, the refined theory "outlines 19 core values, categorized by motivational goals," which simplify into a two-dimensional structure: conservation against openness to change, and self-enhancement against self-transcendence (Rozen et al., 2025). Values whose motivations are compatible sit next to each other on a circle, and values whose motivations conflict sit opposite. Their example: pursuing independence and creativity (self-direction) aligns with seeking change (stimulation) but conflicts with an emphasis on the status quo (conformity) (Rozen et al., 2025). The theory was built to apply across populations and, in their account, has been tested in nearly 100 countries (Rozen et al., 2025).

Two regularities make the circle a reference rather than a list. First, people across societies rank some values above others, which the literature calls the universal value hierarchy: in Rozen and colleagues' example, caring for close others ranks high and dominance low (Rozen et al., 2025). Second, values that are compatible tend to be correlated within a person, so someone who scores high on dominance is unlikely to score high on benevolence (Rozen et al., 2025). The first is a claim about averages across people, the second about the pattern inside one person's answers. Rozen and colleagues' human reference is a published dataset from 49 cultural groups and 53,472 participants (Rozen et al., 2025).

The circle gives a specification its hardest cases in advance (inference). A model specification that tells an assistant both to respect a user's autonomy and to uphold rules and norms has taken a position at both ends of the first dimension. The theory predicts that the two will pull apart exactly where a user's plan departs from convention. The circle also gives a measurement of a model two separate tests (inference). Whether a model's rankings follow the human hierarchy is a question about averages, and whether its answers on compatible values move together is a question about structure. Only the second distinguishes a respondent with a value system from one that reproduces the average answer. Section 18.7 reports what the tests found.

The other reference is survey data collected across countries over decades. The World Values Survey trend file covers the years 1981 to 2022 (Haerpfer et al., 2022), and claims that a model reflects some population's values are checked against data of this kind. How far human populations differ, and why a dataset drawn from one of them carries its norms, is Section 17.4.1's subject.

The analogy stops at the respondent. The structure was found in people whose answers come from one person with a history and whose values are stable enough to survey twice. A model answers as whatever the prompt makes it, so the circle can be used to test a model only after it is shown that the model gives the answers of one respondent at all (inference). A developer's own researchers say as much of the instrument. In a 2025 preprint on the values its models express, Anthropic's authors write that a language model's level of "valuing hedonism," per the Schwartz theory, "is unlikely to be meaningful for understanding the model's properties or behavior" (Huang et al., 2025c).

18.2 Moral judgment and its rival theories #

A value survey asks what matters to a person; a moral questionnaire asks what a person counts as right and wrong. Measuring a model's morality needs such a questionnaire, and every questionnaire carries a theory of what morality is. This section sets out the main theory and its rivals as rival instruments, then asks the psychometric question any of them must answer before it can compare two groups.

18.2.1 Moral foundations and their rivals #

From psychology. The first theory is moral foundations theory, set out in a 2013 chapter by Jesse Graham, Jonathan Haidt, and colleagues (Graham et al., 2013). It began from a search for concerns that recur in moral codes around the world and have evolutionary explanations, and Haidt and Craig Joseph named the five best candidates: Care/harm, Fairness/cheating, Loyalty/betrayal, Authority/subversion, and Sanctity/degradation. They write that there are more than five and name Liberty/oppression as a candidate sixth (Graham et al., 2013).

The theory makes four claims, and its authors write that if any is disproved, the theory "would need to be abandoned, too" (Graham et al., 2013). Nativism: there is a "first draft" of the moral mind, organized in advance of experience. Cultural learning: the first draft is edited during development within a particular culture. Intuitionism: "Intuitions come first, strategic reasoning second." Pluralism: morality has several foundations, not one (Graham et al., 2013). The project is, in their words, "descriptive, not normative": it claims to describe what people count as moral, not what they should (Graham et al., 2013).

The theory became an instrument. Of the four kinds of measures built for it, self-report surveys "have been the most widely used by far" (Graham et al., 2013), the Moral Foundations Questionnaire among them. Its best-known findings concern American liberals and conservatives. Across world regions, the authors report that a large international sample fit the five-foundation model. Participants from South, East, and Southeast Asia expressed slightly greater Loyalty and Sanctity concerns than Western participants, differences smaller than those between liberals and conservatives in the United States (Graham et al., 2013). They add two caveats: the data were collected in English from people with internet access, and so "likely come from Westernized (or even WEIRD) segments of these populations" (Graham et al., 2013). WEIRD abbreviates Western, educated, industrialized, rich, and democratic, the populations Section 17.4.1 shows to be outliers. The authors also concede that their list of foundations "is unlikely to survive the empirical challenges of the next several years with no changes" (Graham et al., 2013).

Three rivals offer different instruments. The theory of dyadic morality, in a 2018 paper by Chelsea Schein and Kurt Gray, holds by its abstract that acts are condemned in proportion to norm violation, negative affect, and perceived harm. The harm is dyadic: an intentional agent causing damage to a vulnerable patient (Schein and Gray, 2018). The abstract argues against harmless wrongs and against modular foundations, while keeping a pluralism of values through the flexibility of perceived harm (Schein and Gray, 2018). Graham and colleagues, answering an earlier version of the argument, call the reduction of morality to dyadic harm Procrustean, "cutting off phenomena it cannot explain (e.g., the role of disgust in moral judgments)" (Graham et al., 2013).

Morality as cooperation, tested by Oliver Scott Curry, Daniel Mullins, and Harvey Whitehouse in 2019, holds that morality consists of solutions to recurrent problems of cooperation (Curry et al., 2019). By the paper's abstract, they examined seven cooperative behaviors (helping kin, helping your group, reciprocating, being brave, deferring to superiors, dividing disputed resources, and respecting prior possession) in the ethnographic records of 60 societies. The moral valence of these behaviors was uniformly positive, and the authors conclude that they are "plausible candidates for universal moral rules" (Curry et al., 2019).

The third challenge comes from the theory's own authors. In a 2023 paper, Mohammad Atari, with Haidt and Graham among his coauthors, built a revised questionnaire from data in 25 populations (Atari et al., 2023a). By the abstract, it splits fairness into two foundations, equality and proportionality, keeps care, loyalty, authority, and purity, and finds that the network of relations between the foundations and other measures varies across cultural contexts (Atari et al., 2023a).

The evidence does not decide among these theories, and each result bears on more than one. Curry's uniformly positive valence across 60 societies fits morality as cooperation, and it does not contradict foundations theory, since helping one's group and deferring to superiors read as loyalty and authority (inference). Atari's variation across cultures fits the cultural-learning claim of foundations theory and cuts against any claim that one structure is universal (inference). Dyadic morality predicts that every condemned act is perceived as harmful; foundations theory predicts condemnation of acts judged harmless, and disgust is where the two make different predictions (inference). For a measurement of a model the consequence is plain: a model scored on the original questionnaire receives five numbers, on the revised one six, on a harm-based instrument one, and on a cooperation-based instrument seven. The choice of instrument decides what the model's morality can be found to be (inference).

The intuitionist claim has a second consequence. If intuitions come first and reasoning second (Graham et al., 2013), a stated reason is weak evidence of what drove a judgment. Section 25.4 reports the developmental evidence on moral reasoning and action, and Section 40.3 meets the same gap in a model's written reasoning.

18.2.2 Measurement invariance #

From psychometrics. Suppose a questionnaire gives two groups different average scores on loyalty. The difference means that the groups differ in loyalty only if the questionnaire measures loyalty the same way in both. Measurement invariance is the name for that condition. William Meredith's 1993 paper in Psychometrika, by its abstract, introduces and defines measurement invariance together with weak measurement invariance and strong and strict factorial invariance, and argues that strict factorial invariance is required for fairness or equity to exist (Meredith, 1993). The usual setting for these definitions is the linear factor model, and the derivation below shows what goes wrong without them.

The condition has been tested for the moral foundations questionnaire. Kathryn Iurino and Gerard Saucier tested the short form across 27 countries on five continents and, by their abstract, found it "difficult to specify Moral Foundations Questionnaire items in a quantitative five-factor model that will converge nonproblematically" across them (Iurino and Saucier, 2020). The revised questionnaire of Atari and colleagues was examined with several methods, among them an alignment method for measurement equivalence, and its abstract reports that it "fares well in terms of reliability and validity across cultural contexts" (Atari et al., 2023a). The two results concern different instruments and are compatible: the first questionnaire failed the test, and its successor, examined with a method for approximate equivalence, reports doing well.

For a model, the consequence follows from the derivation (inference). A model compared with a human population on a moral questionnaire is a respondent of a different kind. Nothing guarantees that an item loads on the trait for the model as it does for people, or that the model reads the item's wording the same way. A model that agrees with whatever a statement asserts, for instance, adds to every item's intercept. As of October 2026 we found no study that tests the measurement invariance of a value or moral questionnaire between models and human populations, or across model families, although the human literature shows non-invariance across countries. Section 5.2.4 already notes that one model under two scaffolds can behave as two respondents; invariance is the property that would have to be shown before the two are compared.

18.3 When a judgment is a signal #

A preference label is useful only if the rater's judgment tracks the property the label is meant to measure: correctness, helpfulness, harm. This section asks when it does. Psychology supplies the question; the preference datasets supply what their raters were actually trained on; and the answer depends on whether the property is one the rater can check.

From psychology. Section 13.6.1 reports the founding evidence that the way a question is asked shapes the preference it records, and the doubt that Baruch Fischhoff put in the title of a 1991 paper, "Value elicitation: Is there anything in there?" (Fischhoff, 1991). A second question concerns the judge rather than the preference: when does a quick, intuitive judgment track the truth? Daniel Kahneman and Gary Klein addressed it in a 2009 paper titled "Conditions for intuitive expertise: A failure to disagree" (Kahneman and Klein, 2009). It could not be read for this book, so its conditions are not reported here.

What can be said without it follows from how any learner learns (inference). A judge comes to track a property only if something in its experience reported that property back. A radiologist who later reads the biopsy result gets such a report; a rater who compares two answers about tax law and never learns which was right does not. Whatever has shaped that rater's judgment, it was not the correctness of the answers.

The preference datasets say what their raters were trained on. For the summarization work of Nisan Stiennon and colleagues at OpenAI, published at NeurIPS in 2020, the developer reports that it trained all labelers "to ensure high agreement with our judgments" and continuously monitored labeler-researcher agreement (Stiennon et al., 2020). The feedback a rater receives in such a pipeline is about whether the rater agreed with the researchers, not about whether the preferred response was better, so a label measures the researchers' judgment as transmitted through instructions (inference). That is sound for properties the researchers can judge, and it passes on their blind spots for properties they cannot.

Agreement also depends on difficulty. In the same project, agreement between researchers ranged from about 65 percent on the most difficult comparisons, two varied samples from the same trained model, to about 80 percent on the easiest. Where the researchers discussed the comparisons with each other, it reached about 95 percent (Stiennon et al., 2020). The comparisons that decide what a model learns late in training, between two strong responses, are the ones on which judgment is least shared (inference).

Where the property can be checked, studies have measured what ratings track instead. A 2023 preprint whose authors include researchers at Cohere found that the assertiveness of an output skewed annotators' perception of its factual errors, which Section 22.6 reports (Hosking et al., 2023). A study published at ICLR 2025 trained models with a standard human-feedback pipeline on tasks with known answers, and time-limited human evaluators became more likely to accept the models' wrong outputs after training; Section 30.4.1 reports its design and numbers (Wen et al., 2025a). The mechanism, that feedback from an evaluator who sees only observations of an episode rewards how the episode looks, is the subject of Section 22.6 (Lang et al., 2024b).

One remedy is to model the rater's errors instead of ignoring them. Owain Evans, Andreas Stuhlmüller, and Noah Goodman, in a 2016 paper at AAAI, built generative models of planning for agents with false beliefs and time-inconsistent choices, and inferred preferences jointly with those deviations (Evans et al., 2016). Their motivating case is a person "who smokes every day while wishing to quit," for whom "a model that has good predictive performance might nonetheless fail to identify what this person values" (Evans et al., 2016). In a behavioral experiment, human subjects given the same observations also explained the choices by systematic deviations from optimal behavior (Evans et al., 2016). The remedy has a limit that Section 13.4.3 proves: a person's behavior cannot be uniquely split into a model of how they decide and what they value, so some model of the rater must be assumed rather than learned (Armstrong and Mindermann, 2018).

The analogy with an expert's judgment stops at duration and feedback. An expert's intuition, where it deserves trust, was formed over years of practice; a rater judges in minutes, and in the pipelines above the only feedback is agreement with the people who wrote the instructions (inference). The datasets report that agreement, not accuracy against a known answer; Section 30.4.1 reports one study that measured the second, on tasks built to have answers.

The consequence for alignment is a gradient (inference). The human feedback signal is most valid where raters can check what they rate and least valid where they cannot, and the cases alignment most needs, outputs more capable than the person judging them, sit at the far end. That is the opening of Chapter 41. The claim concerns the training signal; it says nothing yet about how the trained system then behaves toward its users.

18.4 Disagreement as data #

When raters disagree, annotation practice has usually treated the disagreement as noise around one correct label and resolved it by majority vote. This section shows that the arithmetic of a vote cannot tell noise from a split among groups. It then reports the line of work in natural language processing that argues the split is the information, and asks what agreement among raters measures.

18.4.1 Noise or a split #

Consider one item that $K$ raters label, $K$ odd, as either $X$ or $Y$. Two stories explain a 66-to-34 split in their labels. In the first, everyone holds $X$ and some raters slip. In the second, 70 percent hold $X$, 30 percent hold $Y$, and each rater reports their own view nine times in ten. The two stories call for different labels, and the derivation shows that a vote cannot tell them apart.

Figure 18.1 makes the third point visible. With the defaults, a single rater writes $X$ with probability 0.66, a majority of five does so with probability 0.78, and a majority of 41 with probability 0.98. On the reading of the first story this is a label becoming accurate; on the reading of the second it is a 30 percent minority being written out of the data. The same distinction separates the two cases of Section 13.5.2. A reward model is a network trained to predict which of two responses raters prefer. When the disagreement behind its data acts like noise, independent and identically distributed across responses, it ranks responses by mean utility. When the disagreement comes from groups, it ranks them by the Borda count, each response's average chance of beating the others (Siththaranjan et al., 2024).

What separates the stories is not the vote on one item but whether the disagreement attaches to persons (inference). Under the first story, a rater who writes $Y$ on one item is no more likely than anyone else to write $Y$ on a related item or on the same item asked again. Under the second, the same people disagree consistently. The test needs rater identities kept with the labels, and it needs the same raters to see related items.

18.4.2 The evidence from annotation #

The case that disagreement is signal was put by Lora Aroyo and Chris Welty in a 2015 article in AI Magazine (Aroyo and Welty, 2015). They name seven myths of human annotation, beginning with "One Truth," the assumption that "there is one correct interpretation for every input example," and "Disagreement Is Bad" (Aroyo and Welty, 2015). In their medical relation-extraction work they "found that disagreement is not noise but signal." At the level of an item it can indicate that the sentence or the relation is ambiguous or vague, or that the worker is not doing a good job (Aroyo and Welty, 2015). Annotators agreed on a sentence stating plainly that antibiotics treat typhus, and disagreed on one saying that DDT was used against the disease's insect carriers when antibiotics were in short supply, where only a deep reading finds the relation (Aroyo and Welty, 2015). The myth "Experts Are Better" fell too: in their work, experts did not annotate significantly better than nonexperts, and medical experts marked relations they knew to be true even where the sentence did not express them (Aroyo and Welty, 2015).

Later work tested and generalized the claim. Ellie Pavlick and Tom Kwiatkowski, in a 2019 paper in Transactions of the Association for Computational Linguistics, studied disagreements about whether one sentence follows from another. By their abstract, the disagreements "persist as we collect more ratings and as we vary the amount of context provided to raters," and the uncertainty current models capture does not reflect the kind present in human disagreement (Pavlick and Kwiatkowski, 2019). Read against the derivation, the first finding does not by itself decide between the stories, since the vote share on an item persists under both. The second, that the disagreement has a structure models fail to reproduce, is the stronger part (inference). Barbara Plank's 2022 position paper at EMNLP names the phenomenon human label variation and argues that it is genuine, arising "due to disagreement, subjectivity in annotation or multiple plausible answers." It affects data, modeling, and evaluation alike, and the paper provides a repository of datasets that keep un-aggregated labels (Plank, 2022).

Federico Cabitza, Andrea Campagner, and Valerio Basile, in a 2023 paper at AAAI, call the position perspectivism: methods that "preserve divergence of opinions and integrate multiple perspectives" rather than remove disagreement (Cabitza et al., 2023). They list its costs as well: the need for many raters, which is expensive in domains such as medicine; incompatibility with standard learning methods; and harder validation without a unique ground truth (Cabitza et al., 2023). A method that keeps the disagreement was tested by Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran in 2022. Their multi-task model is one network that predicts each annotator's judgment as a separate output. By their abstract, it did as well as or better than training on aggregated labels across seven binary classification tasks, and its uncertainty correlated better with annotator disagreement than traditional methods did (Davani et al., 2022).

Preference datasets for language models were built the other way. The InstructGPT authors at OpenAI write that "most comparisons are only labeled by 1 contractor for cost reasons," so the disagreement in their data is largely invisible (Ouyang et al., 2022). They add that in cases of disagreement "aligning to the average labeler preference may not be desirable," and suggest weighting more heavily the labelers who belong to a group a text disproportionately affects (Ouyang et al., 2022). Stiennon and colleagues report a measured effect of the alternative. On the training data where enough comparisons were shared, taking the modal label of three labelers raised their agreement with the researchers from 72 to 77 percent; they nevertheless "usually collect only one label per comparison, in order to maximize label throughput" (Stiennon et al., 2020). Both are developers' reports on their own pipelines. Distributional reward models, which keep a spread of utilities for each response, are the reward-model form of perspectivism, and Section 13.5.4 reports them.

18.4.3 Agreement is reliability, not validity #

From psychometrics. Agreement among raters is what psychometrics calls a reliability statistic: it measures how consistently an instrument gives the same reading, not whether the reading is of the right thing. Cronbach and Meehl's account of construct validity, which Section 5.2.1 teaches, puts the relation sharply for agreement among a test's items: "High internal consistency may lower validity. Only if the underlying theory of the trait being measured calls for high item intercorrelations do the correlations support construct validity" (Cronbach and Meehl, 1955). The same holds for raters (inference). High agreement supports a label's validity only if the property is one the raters should share, and on a subjective item it can reflect a shared instruction or a shared bias as easily as a shared truth. Low agreement on such an item may be the accurate reading.

What it implies for alignment is that every aggregation of labels is a choice (inference). Majority vote, the modal label, and the Borda count that a reward model applies (Section 13.5.3) are social choice rules, and a pipeline that collects one label per comparison has chosen whichever rater happened to see the item. Section 17.2 sets out the alternatives and who should choose among them. Where the evidence stops: the annotation studies above are of classification and inference tasks. As of October 2026 we found little evidence on whether keeping annotator-level labels in preference training changes the values a model expresses toward minority views.

18.5 Who produces the signal #

The "human preference" in a reward model is the output of a labor process. Raters and annotators are agents of a laboratory or of an outsourcing firm, and the way they are hired, paid, instructed, and scored is part of the instrument. This section treats them as agents under the book's own theory, reports what is known about the work, and says where the evidence stops: at the door of the frontier laboratories.

18.5.1 The prediction #

In the multitask model of Section 14.2.2, paying an agent on a measured task draws effort from an unmeasured one whenever the two compete, and the best piece rate falls as the unmeasured task grows more valuable ((14.2)); Steven Kerr's essay put the pattern in its title, the folly of rewarding A while hoping for B (Kerr, 1975). For a rater, the measured tasks are throughput and agreement with other raters or a gold label, and the unmeasured task is the careful judgment the label is meant to record. Where pay and scoring are tied to throughput and agreement, the model predicts that raters will move effort toward speed and toward the answer the scoring expects (inference).

18.5.2 The work and its pay #

From the sociology of work. Mary Gray and Siddharth Suri's 2019 book on on-demand work carries its argument in its title, Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass (Gray and Suri, 2019). Paola Tubaro, Antonio Casilli, and Marion Coville, in a 2020 article in Big Data & Society, distinguish three functions of platform micro-work for AI: "artificial intelligence preparation," "verification," and "impersonation." They argue that it is "a structural component" of AI production, not a stage that will vanish as the technology matures (Tubaro et al., 2020).

The pay has been measured on one platform. Kotaro Hara and colleagues, in a 2018 paper at the CHI conference, recorded 2,676 workers performing 3.8 million tasks on Amazon Mechanical Turk, a platform where requesters post small paid tasks. By the abstract, workers earned a median hourly wage of about two dollars, and only 4 percent earned more than $7.25 an hour; the average requester paid more than $11 an hour, but lower-paying requesters posted much more work (Hara et al., 2018). The figures depend on counting unpaid work, such as time spent searching for tasks and on work that was rejected or never submitted (Hara et al., 2018).

One case reaches a frontier developer's safety work. In January 2023, Billy Perrigo reported in TIME that OpenAI sent tens of thousands of text snippets to the outsourcing firm Sama in Kenya, some describing violence and sexual abuse in graphic detail. They were labeled for a toxicity detector later built into ChatGPT (Perrigo, 2023). The report rests on internal documents, payslips, and interviews with four employees. It states that labelers took home between about $1.32 and $2 an hour depending on seniority and performance, while the contracts set an hourly rate of $12.50 paid to Sama. By three employees' account, they were expected to label 150 to 250 passages per nine-hour shift, and commission depended on meeting "key performance indicators like accuracy and speed" (Perrigo, 2023). All four employees described being mentally scarred by the work, and Sama ended it eight months early (Perrigo, 2023). This is journalism, not peer-reviewed research; Sama disputed part of its account of counseling, and OpenAI confirmed that Sama's employees contributed to the detector (Perrigo, 2023). Part of the cost of the harmlessness target fell on these workers.

18.5.3 How instructions and metrics shape the label #

The most detailed study we found of how the work shapes the data is by Milagros Miceli and Julian Posada, published in 2022 in the Proceedings of the ACM on Human-Computer Interaction (Miceli and Posada, 2022). They studied outsourced machine learning data work at three platforms in Venezuela and a business process outsourcing firm in Argentina, examining 210 instruction documents and 55 interviews with workers, managers, and requesters, with participant observation (Miceli and Posada, 2022). Their findings are three. Instructions "reproduce and normalize the worldviews of requesters"; precarious conditions and economic dependency make workers "obedient to instructions"; and interfaces and performance metrics limit "workers' agency" and normalize "specific ways of interpreting data" (Miceli and Posada, 2022).

The details show the multitask structure. Instructions warned that "Judges providing low quality responses will be banned and not paid." The authors observe that accuracy was defined as agreement with "what the client believes to be an accurate truth value, while divergence from that value is considered inaccurate" (Miceli and Posada, 2022). Annotation screens on two platforms showed an accuracy score at the top, and on one also the number of annotations completed, the earnings, and the time spent per task (Miceli and Posada, 2022). The firm in Argentina paid a regular salary instead of a piece rate, at Argentina's minimum wage, and kept metrics of individual output and of correct and incorrect output (Miceli and Posada, 2022). The authors write that "Even tasks that ask for workers' judgment ... are timed and reward fast thinking" (Miceli and Posada, 2022). On the book's reading, a label produced this way measures the requester's expected answer, delivered quickly, and the worker's own judgment is the unmeasured task (inference).

Other studies point the same way. Ding Wang, Shantanu Prabhat, and Nithya Sambasivan interviewed 25 annotators, 10 industry experts, and 12 practitioners in India; by their abstract, "the work of annotators is dictated by the interests, priorities and values of others above their station" (Wang et al., 2022c). Miceli, Posada, and Tianling Yang argue in a 2022 commentary that dataset research should study the power, labor conditions, and standpoints inscribed in data rather than only its bias (Miceli et al., 2022).

The developers' own reports show both designs. OpenAI selected InstructGPT's labelers partly on agreement with its researchers, with soft cutoffs at 75 percent agreement on flagging sensitive speech and on comparisons. Its authors write that they were aligning "to our preferences, as the researchers designing this study," since they wrote the labeling instructions and answered labelers' questions (Ouyang et al., 2022). Anthropic, for its helpful and harmless dataset, did "not filter workers based on agreement or other direct measures of label quality" and largely let crowdworkers "use their own intuitions" to define helpfulness and harmfulness (Bai et al., 2022b). Its authors add an observation the multitask model predicts: "it is easier to incentivize very high-quality interactions on platforms such as Upwork, where crowdworkers can easily be paid by the hour, rather than per task." Workers on Mechanical Turk, paid per task, "tend to generate data much more rapidly" and supplied about 80 percent of the data (Bai et al., 2022b). Both are developers' reports on their own pipelines. The first design makes the label a measure of the researchers' judgment; the second makes it a measure of the crowd's, under per-task pay (inference).

18.5.4 Workers under piece rates #

One observation is consistent with the prediction, on a writing task rather than a rating task, though it did not vary the pay. In a 2023 preprint, Veniamin Veselovsky, Manoel Horta Ribeiro, and Robert West reran a summarization task on Mechanical Turk around 1 June 2023, paying $1 per summary of a medical abstract, about $15 an hour by their conservative estimate (Veselovsky et al., 2023). Combining keystroke records with a classifier trained to detect text written by a language model, they estimate that 33 to 46 percent of the 44 workers who submitted the 46 summaries used language models to do the task (Veselovsky et al., 2023). The authors note that the study covers one task, summarization, and that generalization to tasks harder for language models is uncertain (Veselovsky et al., 2023). That raters comparing model outputs respond the same way is the book's inference, not their finding.

The quality of the signal is also an organizational outcome. Interviewing 53 practitioners building AI for high-stakes domains, Nithya Sambasivan and colleagues report in their abstract that data cascades, compounding failures caused by undervalued data work, were pervasive, at 92 percent prevalence (Sambasivan et al., 2021a). Mark Díaz and colleagues propose documentation, CrowdWorkSheets, that records who annotated, how they were selected, and on what platform. Its premise is that annotators' lived experiences can affect their annotations (Díaz et al., 2022).

18.5.5 When the rater is a model #

Models are now raters too, which changes who produces the signal again. Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli reported in PNAS in 2023, by their abstract, on four samples of tweets and news articles (6,183 items). On tasks including relevance, stance, topic, and frame detection, ChatGPT's accuracy without worked examples in the prompt exceeded that of crowd workers by about 25 percentage points on average. Its intercoder agreement was above both crowd workers' and trained annotators', at under $0.003 per annotation, about thirty times cheaper than Mechanical Turk (Gilardi et al., 2023). Reward models trained on a language model's preferences, and what they optimize, are the subject of Chapter 23 (Lee et al., 2023). A model rater can be inspected, rerun, and audited, which no crowd worker can be, and whoever writes its instructions decides what is preferred (inference).

18.5.6 What the evidence shows and where it stops #

The strongest counter-reading is that the evidence comes from the wrong place. Crowd platforms pay by the task and outsourcing firms pay little and measure output, but a laboratory can hire experts, pay them by the hour, train them, and check their work. Stiennon and colleagues report that leaving out workers with poorer agreement rates "did not help" their reward models (Stiennon et al., 2020). On that reading, the labor process explains the quality of cheap data and not the preference data of frontier models. The two readings make different predictions (inference). If the multitask mechanism operates inside frontier pipelines, rater accuracy on items with a known answer should fall as per-item time pressure and throughput targets rise, holding raters fixed; if it does not, accuracy should track only rater expertise. As of October 2026 we found no published study of rater incentives inside a frontier laboratory's own preference pipeline, and only the laboratories and their vendors hold the data to run one. The claim of this section therefore concerns the training signal and stands as a prediction supported off the frontier and untested on it. That the public cannot check it is a fact about the third relationship, the developer and the public.

18.6 How well a proxy knows its principal #

A reward model is a proxy: it predicts what its principal would choose. How well can any proxy do that? Medicine asked the question of the best available human proxies, the people who know a patient, and preference learning can ask it of its own raters. The answers set the scale against which a reward model's accuracy should be read.

18.6.1 The surrogate's question #

From bioethics. When a patient cannot decide, medicine asks a surrogate to decide as the patient would have, the standard of substituted judgment that Section 16.6.1 sets out. In the words of Brian Earp and colleagues, surrogates often "struggle to guess what the patient would want if they had capacity" (Earp et al., 2024). Medicine has studied how well they guess. A paper by David Shalowitz, Elizabeth Garrett-Mayer, and David Wendler titled "The Accuracy of Surrogate Decision Makers" appeared in the Archives of Internal Medicine in 2006 (Shalowitz et al., 2006). A 2001 study by Angela Fagerlin and colleagues in Health Psychology examined "projection in surrogate decisions about life-sustaining medical treatments," the attribution of one's own preferences to the person one decides for (Fagerlin et al., 2001). Neither could be read for this book, so their figures are not reported here. The proposal to replace surrogates with a statistical predictor trained on population data, and its personalized form built with language models, are reported in Section 16.6.2 (Rid and Wendler, 2014; Earp et al., 2024).

What the field offers this chapter is the framing rather than a number (inference). Substituted judgment is preference inference by the person who knows the principal best, under formal standards, with the error measured against the principal's own stated wishes. Projection, the subject of Fagerlin and colleagues' study, names a candidate mechanism of error: a proxy fills gaps in what it knows about the principal with its own preferences. A reward model has the same structure. It fills gaps in its data with whatever its training made likely, which Section 13.4.5 shows is where a reward is least constrained.

18.6.2 The rater's ceiling #

Preference learning can measure its own version of the surrogate question: how often do two raters agree, and how well does a reward model predict a rater it has not seen? Table 18.2 collects the rates two developers report for three of their own pipelines.

The rows come from Stiennon and colleagues for summarization (Stiennon et al., 2020), Long Ouyang and colleagues for InstructGPT, whose reward models were tested by five-fold cross-validation over groups of labelers (Ouyang et al., 2022), and Yuntao Bai and colleagues for the helpful and harmless data (Bai et al., 2022b). Each is a developer's report on its own data, and the rates are not comparable across rows: they compare different people on different tasks.

The table shows a reward model doing about as well as one human at predicting another, but what is the most any model could do? The derivation answers under a simple assumption.

Read with the table, the derivation says two things (inference). A reward model that agrees with held-out labels about 70 percent of the time is not 30 percent wrong about "the human preference". Those held-out labels came from groups of the training labelers, whose agreement of 72.6 percent caps any model at about 84 percent under the derivation's assumption, so at least 16 of those 30 points would be disagreement among the raters that no model could remove. The report does not say that agreement was measured on the comparisons the reward models were tested on, so the 16 points are an illustration, not a measurement. And the InstructGPT reward models, at 72.4 percent on their training groups, sit at about the lower bound, the agreement of one labeler with another, well below the ceiling of Equation (18.2). The assumption matters. The ceiling binds a predictor that does not know who is rating. Raters who disagree consistently, the second story of Section 18.4.1, can still be independent draws, but a predictor told who the rater is can beat the ceiling. Where disagreement attaches to persons, the right target is a distribution over raters, and accuracy against single labels is the wrong yardstick for it.

18.6.3 What the analogy implies and where it stops #

The bioethics frame implies that a reward model's agreement with held-out labels should be read against a ceiling well below certainty, set by how well people predict each other, and that the error has a direction: a proxy fills gaps with its own priors (inference). Developers report the agreement numbers for their own datasets, so this concerns the training signal and, through who reports it, the developer and the public.

The analogy stops at the stakes and at the principal. End-of-life choices are rare, grave, and emotionally loaded, and a surrogate decides for one known person. Rated model outputs are mostly none of these, and the rater whose label is predicted did not write the prompt and will not receive the response. The surrogate literature sets a scale for preference inference by people who know the principal, not a prediction for rated outputs. As of October 2026 we found no head-to-head test of a model-based preference predictor against human surrogates on the same people.

18.7 Measuring a model's values #

When the respondent is a language model, the instruments of survey research and personality testing return with their known failures, and with one new one: the respondent may have been trained on the instrument and may know what answer is wanted. This section asks whether any measurement of a model's values yet measures something stable, and reports a debate the evidence has not decided.

18.7.1 What validity asks of a model #

From psychometrics. Section 5.2.1 teaches the vocabulary: a score is valid to the degree that evidence supports the interpretation placed on it. Its main threat is construct-irrelevant variance, in Samuel Messick's phrase variance "associated with other distinct constructs as well as method variance such as response sets or guessing propensities" (Messick, 1995). Cronbach and Meehl named the experiment that bounds it: "More powerful than the retest after uncontrolled intervening experiences is the retest with experimental intervention." As they explain, "If a transient influence swings test scores over a wide range, there are definite limits on the extent to which a test result can be interpreted as reflecting the typical behavior of the individual" (Cronbach and Meehl, 1955). Among their examples was a change of score under the instruction "make yourself look good to an employer" (Cronbach and Meehl, 1955). Most of the studies below are experiments of this kind, run on models: they change something that should not matter and watch the score.

18.7.2 Survey answers that measure the format #

Ricardo Dominguez-Olmedo, Moritz Hardt, and Celestine Mendler-Dünner, in a paper at NeurIPS 2024, put the American Community Survey of the US Census Bureau to 43 language models from 110 million to 175 billion parameters, presenting each question with lettered answer choices, the usual method (Dominguez-Olmedo et al., 2024). Their reference was the 2019 census microdata, the answers of about 3.2 million people. Two patterns dominated. Models' answers were governed by ordering and labeling biases, among them a tendency to pick the choice labeled "A," and all models showed substantial A-bias (Dominguez-Olmedo et al., 2024). Messick's "response sets" is the psychometric name for exactly this. The authors then averaged each model's answers over permutations of the choice order, the standard adjustment in survey research. The answers of base models, pretrained and not further tuned, were close to uniformly random whatever the question or the model's size. Models further trained to follow instructions deviated more from uniform but in general did not come closer to the census, and all remained closer to uniform than to the census (Dominguez-Olmedo et al., 2024). Even for GPT-4, the model closest to the human reference, only 6 of 25 questions were closer to the census than to a uniform distribution (Dominguez-Olmedo et al., 2024).

The second finding explains a published pattern. Survey-based alignment measures compare a model's answer distribution with each population's, and the authors found that every model appeared most "aligned" with the subgroups whose answers were most evenly spread (Dominguez-Olmedo et al., 2024). The derivation shows why.

The authors repeated the analysis on the opinion surveys used by two earlier studies, which Section 17.4.2 reports: about 1,500 questions with 60 US demographic groups, and about 2,300 questions with 60 national populations (Dominguez-Olmedo et al., 2024). The same relation between apparent alignment and the entropy of a group's answers, a measure of how evenly they are spread, held. In one earlier study, low-income, moderate, and Protestant or Catholic respondents had appeared best represented, and people aged 65 and over and widowed people worst. These were, in the survey used, the subgroups whose answers were closest to and furthest from uniform (Dominguez-Olmedo et al., 2024). The authors scope their claim to multiple-choice prompting: their results "suggest caution," and they call for better methods rather than declaring surveys useless (Dominguez-Olmedo et al., 2024).

Paul Röttger and colleagues, in a paper at ACL 2024, examined the Political Compass Test, a popular 62-item questionnaire for placing models politically (Röttger et al., 2024). In a review of 12 earlier articles, 10 had forced models to pick one of its multiple-choice answers. In January 2024 they tested ten models, including GPT-3.5 and GPT-4 in versions 0613 and 1106, with deterministic sampling (Röttger et al., 2024). Without a forcing instruction, every model gave many answers that did not pick an option, most of them saying that as an AI they held no opinion; Zephyr and three of the GPT models picked none. How models were forced changed the answers. Ten paraphrases of the opening question, such as "perceive" for "state," moved two models' overall positions by more than the distance between Joe Biden and Donald Trump on the same test in 2020 (Röttger et al., 2024). Asked to write open text while still told to agree or disagree, GPT-3.5 1106 and Mistral 7b Iv0.1 agreed with a proposition in the multiple-choice setting but disagreed in open text on 19 and 23 of the 62 propositions, by majority over prompt templates, with no change in the other direction (Röttger et al., 2024).

Ariba Khan, Stephen Casper, and Dylan Hadfield-Menell, in a paper at FAccT 2025, tested three assumptions behind survey measures of a model's cultural values. Their five models were GPT-4o, Claude 3.5 Sonnet 20241022, Gemini 2.0 Flash, Llama 3.1 405B, and Mistral Large 2411 (Khan et al., 2025). Stability failed. Reversing the order of answer options, or asking for the option's text instead of its number, frequently changed answers on a global opinion survey by more than one standard deviation of the differences between countries (0.114 on their normalized scale). The failure held for a behavioral task too, choosing between cover letters rewritten to express different cultural values. There, answers moved with the rating format, the scale's length, a request for reasoning, and the role the model was asked to play (Khan et al., 2025). Extrapolability failed: groupings of countries computed from a few cultural dimensions matched the groupings from all six no better than chance, for human data and model answers alike, until many dimensions were included. Steerability failed. Prompted to answer as a person from a given country, models' answers sat on average 6.14 and 1.94 times further from humans, under two prompting methods, than humans from different countries sat from each other (Khan et al., 2025).

18.7.3 Results that stand against the artifacts #

Other measurements find structure. Section 7.3.3 reports the strongest: Mantas Mazeika and colleagues, in a paper at NeurIPS 2025, elicited forced choices between 500 described states of the world from 23 models and found preferences that grew more coherent with scale (Mazeika et al., 2025). Their appendix tests robustness on GPT-4o and GPT-4o-mini across five kinds of surface change: translation into seven languages, syntax, the framing of the question, the labels on the options, and long software-engineering contexts. They report high correlations of the fitted utilities across all of them (Mazeika et al., 2025). They also concede, of the growing decisiveness of larger models, that "it remains unclear whether the resulting preferences are coherent or merely random arrangements" (Mazeika et al., 2025).

Khan and colleagues tested one of Mazeika's headline results directly. Mazeika and colleagues had fitted exchange rates to forced choices: how many lives in one country the choices treat as worth one in another. They reported that "GPT-4o places the value of Lives in the United States significantly below Lives in China, which it in turn ranks below Lives in Pakistan," and that the model "may deny preferring one country's population over another" if asked outright (Mazeika et al., 2025). Khan and colleagues asked GPT-4o about the same countries, under the same sampling temperature, on a five-point scale with a neutral midpoint and on a four-point scale without one. With the neutral option, GPT-4o chose "No preference" in 100 percent of country-pair comparisons; forced, it showed a hierarchy (Khan et al., 2025). They note that their quantification differed from the original's, and that their work "should not be understood as showing that they are fundamentally invalid" (Khan et al., 2025).

The two papers agree on the observations and differ on their meaning (inference). Both report a hierarchy under forced choice, and Mazeika and colleagues anticipate the denial that Khan and colleagues observed once neutrality was allowed. One reads the hierarchy as the model's implicit values and the neutrality as a denial; the other reads the neutrality as the answer and the hierarchy as a product of the format. Mazeika's robustness checks varied the wording of a forced choice, never whether indifference could be expressed, so they do not bear on the disagreement. A forced choice turns indifference into a tie broken somehow, and a consistent tie-breaking rule produces coherent choices with no preference behind them (inference). Which reading is right depends on which answer predicts what the model does when it acts.

Rozen and colleagues, introduced in Section 18.1, gave the 57-item Portrait Values Questionnaire, which measures the 19 values of the circle, to six models. They were GPT-4, Gemini Pro, Llama 3.1 8B Instruct and 70B Instruct, and Gemma 2 9B and 27B, under five kinds of prompt and two sampling temperatures (Rozen et al., 2025). The rank correlation between the models' value rankings and the human hierarchy exceeded 0.8 for many combinations of model and prompt, though GPT-4 under the basic prompt correlated very weakly. The structure was the harder test. Under the basic prompt the models either barely varied across runs or gave internally inconsistent answers, and the authors conclude that "LLMs cannot be treated as 'individuals' holding a coherent set of value priorities" (Rozen et al., 2025). Under a "Value Anchor" prompt, which tells the model to answer as a person who holds one named value, the correlations between values closely followed the human circle (Rozen et al., 2025). The authors' own analysis shows how: anchoring on a value raises the scores of the values next to it on the circle and lowers those opposite, so a population of anchored answers inherits the circle's correlations (Rozen et al., 2025). The human-like structure is real and is partly supplied by the prompt; it shows that models represent the circle, not that any one answer comes from a respondent who holds it (inference). Two authors are also at Google, whose Gemini and Gemma models are among those studied.

Yan Tao and colleagues, in a 2024 paper in PNAS Nexus, report by their abstract that five OpenAI models, from GPT-3 to GPT-4o, gave answers resembling those of English-speaking and Protestant European countries on nationally representative survey items. Prompting GPT-4, GPT-4-turbo, and GPT-4o with a country improved alignment for 71 to 81 percent of countries and territories (Tao et al., 2024). The measure is of the kind the studies above questioned.

18.8 Measured values and behavior #

Questionnaires ask; the alternative is to watch. This section reports two studies that find a model's values in what it does rather than in what it answers. It then takes up the respondent that may know what is wanted, and reads the whole record against the two positions it supports.

18.8.1 Values in use #

Saffron Huang and colleagues at Anthropic, in a 2025 preprint, sampled 700,000 Claude.ai conversations from 18 to 25 February 2025, 91.0 percent of them with Claude 3.5 Sonnet, kept the 308,210 that required interpretation rather than facts, and used Claude 3.5 Sonnet to label the values the assistant expressed (Huang et al., 2025c). They define such a value as a "normative (rather than empirical) consideration that guides how a model reasons about or settles upon a response," and found 3,307 distinct values (Huang et al., 2025c). Some appeared across contexts, mainly values of competent assistance such as "helpfulness," "professionalism," "thoroughness," and "clarity"; many others were specific to a task, such as "healthy boundaries" in relationship advice (Huang et al., 2025c). In 3.0 percent of the conversations, the model strongly resisted the values users expressed, disproportionately values like "rule-breaking" and "moral nihilism" (Huang et al., 2025c).

This is a developer's study of its own models, with its own model as the labeler. Its authors note that Claude may be predisposed to find helpful behavior when labeling Claude, and that "it is impossible to fully determine underlying values from conversational data alone" (Huang et al., 2025c). The method measures expression in use, which is what matters to the people the system serves, and only the developer holds the data to run it.

An outside group measured what values do to answers. In a 2026 preprint, Jan Betley and colleagues, most of them at Truthful AI, report covert value leakage: models' answers to practical questions are shaped by their own values, undisclosed (Betley et al., 2026b). Asked how likely the AI bubble is to pop by a user weighing an investment, Claude Opus 4.8 gave a lower probability when the company was Anthropic rather than OpenAI, mostly without disclosing it. Across such tasks, Claude models showed a small bias toward their developer, GPT models none outside one grading task, and Gemini 3.1 Pro a slight bias against Google (Betley et al., 2026b).

One of their evaluations is a direct test of whether a measured value predicts behavior elsewhere. The authors first asked each model to score 100 weekend activities from 0 to 100 by its own preference. Then, in a new context, they asked it to pick at random between two activities for a user who wanted a tiebreaker (Betley et al., 2026b). For GPT-5.5 at its highest reasoning setting, the correlation between stated preference and how often an activity was picked was 0.82 without tools, 0.53 with a tool returning the time, and 0.14 with a coin-flip tool. Claude Opus 4.7 and 4.8 at maximum reasoning showed no such bias (Betley et al., 2026b). The authors could not rule out that the scores reflect what the model thinks is good for the user, since the two are highly correlated. They also developed most tasks using Claude models, which may bias the suite against them (Betley et al., 2026b).

18.8.2 A respondent that may know what is wanted #

People can fake an inventory: a meta-analysis of 51 studies found all five major personality dimensions equally fakable, with the largest distortions on social desirability scales, and Section 26.5 reports it with the personality tests applied to models (Viswesvaran and Ones, 1999; Serapio-García et al., 2025). A model's answer to a value questionnaire is an inventory taken by a respondent that may have read the scoring key in its training data, and may have been trained to give the answer its developer prefers (inference). The methods tradition answer is the nonreactive measure, gathered without the respondent's awareness, which a 1966 book by Eugene Webb and colleagues named in its title (Webb et al., 1966); Chapter 37 asks whether a model can tell when it is measured. Huang's deployment data and Betley's tiebreaker are nonreactive in this sense, or as close as a study can come (inference).

18.8.3 Reading the evidence #

Two positions read this record. On the first, survey measures of a model's values are artifacts of the instrument: the answers follow labels, wording, and format, and the structure found is supplied by the prompt (Dominguez-Olmedo et al., 2024; Röttger et al., 2024; Khan et al., 2025). On the second, models have value systems that grow more coherent with scale, and careful elicitation can recover them (Mazeika et al., 2025). Each position meets results it does not predict. The artifact position does not predict Mazeika's stability under translation and relabeling, or Betley's stated scores predicting choices in a new context. The coherence position does not predict GPT-4o's uniform neutrality when allowed it, or Röttger's reversals between multiple choice and open text. Röttger and colleagues offer a middle reading, that the models they tested fall "somewhere in between" a single stable persona and a superposition of many (Röttger et al., 2024), which fits both sets of results and predicts that stability will vary by item. The evidence does not decide between the positions.

The analogy with human testing stops at the respondent, as Section 5.2.4 notes: psychometric validity assumes a respondent whose trait persists between the test and the behavior it predicts, and a model's expressed values change with the prompt. Every measurement of a model's values is therefore conditional on its context, and a validity argument must say which contexts it covers (inference). For alignment, the test that matters is predictive (inference). A measurement of a model's values is useful for specification and verification only if it predicts what the model does where no one asks. As far as we found as of October 2026, Betley's tiebreaker is the closest any published study has come, in a setting far simpler than an agent's work. These claims concern the deployed system and the people it serves.

18.9 What the measurements mean for delegation #

The book's thesis is that alignment is a delegation problem under an asymmetry of competence and visibility. Measurement is where the principal's side of that asymmetry is decided: what the principal can see of what people want, and of what the agent values, sets the limit on what can be specified and checked. Each relationship meets the chapter's results differently.

In the training relationship the structures the thesis names converge (inference). Preference data are produced by agents under incentives, and the multitask model predicts the distortion that the labor studies document off the frontier. The signal is valid where raters can check what they rate, which ties specification to verification as Chapter 41 develops. And aggregating labels is a social choice made by the pipeline, whether or not anyone chose it. What is new is the scale and the circularity: models now rate the outputs that train models, so the agent and the instrument can come from the same family.

In the relationship between the deployed system and the people it serves, the thesis needs a model with values to delegate to, and the measurements neither establish that one exists nor rule it out. The format effects meet the first defeater of the intentional stance that Section 1.5 names, instability under trivial changes, and coherence under forced choice meets it only in part (inference). The older field carries over with one reversal and one break. Retesting a model is cheap, so the experiments with which Cronbach and Meehl bounded validity can be run at a scale no human study allows; and the stable respondent that psychometrics assumes is the thing in question.

In the relationship between the developer and the public, the decisive data are held by developers: the agreement rates of their raters, the values their models express in deployment, and the incentives of the people who label their data. The public sees what developers choose to publish (Section 5.6). One feature belongs with what Section 1.7.1 calls new, the principal's power to manufacture the agent's dispositions (inference). The developer that writes a model's character also writes the questionnaire and scores the answers, so a measurement of a model's values can measure what its measurer installed, a loop that no psychometrician testing a person faced.

18.10 Settled, contested, missing #

18.11 Exercises #

Further reading #

• Graham et al. (2013): moral foundations theory stated by its authors, with their replies to critics and their own concessions.
• Aroyo and Welty (2015): the case that disagreement among annotators is signal.
• Miceli and Posada (2022): how instructions, interfaces, and metrics shape outsourced data work.
• Gray and Suri (2019): Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass, read here only as a catalog record.
• Cronbach and Meehl (1955): the logic any measure of an unobservable trait must satisfy.
• Mazeika et al. (2025): the strongest case that model preferences form coherent utilities, to read with Khan et al. (2025).