Foundations

In brief

Ideas

Uncertainty and choice: base rates decide what a monitor's flag means, the KL divergence measures how far training moved a policy, and expected utility says what an agent maximizes (Ch. 6; Ch. 7).
The machinery: many functions fit the training data, so inductive bias decides behavior beyond it, and a language model becomes assistant and agent in stages a claim must name (Ch. 9; Ch. 11).
Post-training's KL-regularized objective reweights the reference policy by reward, and rarity protects a behavior only logarithmically (Ch. 12).
Delegation: choices reveal preferences only under a model of the chooser, pay on the measured pulls effort from competing unmeasured tasks, and a measure under pressure tends to stop measuring what it did (Ch. 13; Ch. 14; Ch. 15).

Evidence

Theorem: a clean record of $n$ independent episodes bounds the failure rate at about $3/n$ with 95 percent confidence; a one-in-a-million claim needs about three million clean episodes (§6.5.1).
Theorem: with unlimited data, every pair compared equally often, and an unrestricted reward, Bradley-Terry training ranks responses by their Borda count (§13.5.2).
Theorem: over all stationary policies of a finite process, every unhackable pair of proxy and true reward is equivalent or trivial; the authors conclude that learned rewards are almost certainly hackable (§15.4.1).
Experiment: under optimization against a proxy reward model, a gold reward rose and then fell, in OpenAI's synthetic study of its own GPT-3-series models, reproduced by an independent group with open models (§15.6.3).

Open

Limit: in training the developer writes the agent's preferences, offers no contract to refuse, and nothing bears risk, so the delegation results that rest on risk aversion lapse (§14.8.2).
Limit: the tools fit a deployed model only if it is usefully an agent with goals, and whether long reinforcement learning builds behavior no pretrained persona had is undecided (§11.7).

This part builds the tools that the rest of the book uses, in three groups. The first gives the language of uncertainty and choice: probability and information, with the KL divergence, calibration, and the statistics of rare events and detection (Chapter 6), and agents, objectives, and decisions, with expected utility and the optimizer's curse (Chapter 7). The second teaches the machinery alignment is applied to: how neural networks learn (Chapter 8), which of the many solutions that fit the data training selects (Chapter 9), how a behavior of one trained model can be traced to its data, to the shape of its solution, and to the computation inside it (Chapter 10), what a language model is and how it becomes an assistant and an agent (Chapter 11), and reinforcement learning, with the KL-regularized objective at the center of post-training (Chapter 12). The third teaches the theory of delegation that the book's thesis rests on: how preferences are inferred from choices (Chapter 13), the economics of principals, agents, and games (Chapter 14), and what happens to a measure when it becomes a target (Chapter 15).

The part sits between the statement of the problem and the four problems because each of them needs these tools, and a formal object that recurs is taught once, here, and recalled elsewhere with a cross-reference. Specification draws on decision theory, the limits of what a reward can express, and preference models; instillation on learning dynamics, inductive bias, language models, and reinforcement learning; verification on detection statistics, calibration, and the persona view of a model; maintenance under pressure on Goodhart's law, the optimizer's curse, and principal-agent theory. Each chapter moves through a problem, an idea, a formalism, an interactive exploration, and what goes wrong in practice, and ends with exercises. Where the theory is contested, as it is for what training selects and for what a language model is, the chapter gives the rival accounts and says what would decide between them.

The part assumes fluency in code, matrices, and derivatives, and nothing more: probability is rebuilt from its definitions, and no machine learning, economics, or decision theory is assumed.

Chapters in this part

6 Probability and Information
Every object in alignment research is a probability distribution, and every safety claim is a claim about how often something happens. This chapter teaches distributions, likelihood, and Bayes' rule; entropy and cross-entropy as compression; the KL divergence that the book uses to measure optimization pressure, and what a KL budget does not bound; calibration and proper scoring rules; and the statistics of rare events and detection: what a record of clean tests licenses, how rates too small to sample are estimated, and what a monitor's alarm is worth at a low base rate.
7 Agents, Objectives, and Decisions
What having an objective means, what follows from it, and where the formal picture stops fitting trained systems. The chapter teaches when a system counts as an agent and how goal-directedness can be measured; expected utility derived from four axioms, the human choices that violate them, and the coherence arguments for and against expecting capable systems to maximize it; Markov decision processes, the Bellman equations, and what a reward can and cannot express; decision theories for agents that can be predicted or copied, and agents that sit inside the world their reward comes from; and the optimizer's curse, why the option that looks best is usually worse than it looks, with its Bayesian correction.
8 Learning Machines
The systems alignment is about are not written; they are trained. This chapter teaches what that means, each idea from its smallest working example: learning as fitting a function to data, the neural network as a family of functions, gradient descent and backpropagation, generalization and its characteristic failures (shortcuts and adversarial examples), the scaling laws that let developers predict a model's loss before training it, and the dispute over whether abilities arrive smoothly or suddenly as models grow.
9 What Training Selects
Many functions fit any training set; which one training finds decides how a model behaves where it was never trained. This chapter teaches what is known about that choice, each idea from a case small enough to compute: underspecification, double descent, simplicity bias and its failure mode, and stagewise learning and grokking. It is the theory behind goal misgeneralization, the scheming debate, weak-to-strong generalization, and emergent misalignment, and it marks where that theory stops; the next chapter traces what training built in a particular model.
10 Tracing What Training Built
Given a trained model that behaves a certain way, can the behavior be traced back to how training built it? This chapter teaches three handles on that question, each from a case small enough to compute: training data attribution, with the influence function, what it estimates in deep networks, and the small imitation step by which a student that shares its teacher's initialization moves toward the teacher's traits on any data; singular learning theory as a research program, with the free energy that trades loss against a learning coefficient; and the experiments that test whether training builds learned search. It marks where each tool stops, and what the theory of what training selects still cannot predict.
11 Transformers and Language Models
Most of the systems this book studies are language models wrapped in tools. This chapter teaches next-token prediction and sampling, the transformer and its residual stream, induction heads and the fixed computation behind each token, in-context learning as inference over latent tasks, the stages that turn a base model into an assistant, and the loop that turns an assistant into an agent. It then teaches the Bayesian and persona view of a language model, the model later chapters use to explain jailbreaks, emergent misalignment, and character, derives what it predicts, and sets out the rival accounts and the evidence that could defeat it.
12 Reinforcement Learning
How a policy learns from reward: the policy gradient, baselines, and PPO; language models as policies and the pipeline that made assistants; the KL-regularized objective, its closed-form optimum, the logarithmic price it puts on rare behavior, and its reading as Bayesian updating; best-of-n and direct preference optimization as two other routes to a reweighting of the reference; the loop of reinforcement learning with verifiable rewards that trains reasoning models, and the evidence that such training stops short of the optimum it aims at; and what the KL anchor does not guarantee.
13 Preference and Choice
What a choice between two options reveals about what someone wants: revealed preference and the economists' separation of choosing from benefiting; random utility and the Bradley-Terry model behind every reward model, fitted by maximum likelihood; the model of the chooser that each kind of feedback assumes; what a reward can and cannot be identified from, from potential shaping to the impossibility of separating a planner from its values; how one reward trained on many raters' choices ranks responses by the Borda count; and the evidence that preferences are often constructed when they are elicited.
14 Principals, Agents, and Games
The economics and game theory of delegation, taught as tools with their derivations: the linear model of hidden action and what hidden information adds, multitask incentives as the theory of paying for what can be measured, incomplete contracts and residual control, common agency among several principals, motivated agents and the choice between selecting and sanctioning, inspection games with their mixed and committed equilibria, and causal influence diagrams as the bridge to learned agents; then a ledger of every assumption these models make, checked against the book's three relationships.
15 Optimization Pressure
What happens to a measure when something optimizes against it. Goodhart's, Campbell's, and Lucas's statements and the record of gaming in schools and hospitals; the four mechanisms by which a proxy fails, with regressional Goodhart derived; the theorems on when a proxy can be optimized safely; the heavy-tailed case in which a KL penalty guarantees nothing, and the measurement that found two open reward models light-tailed; the measured laws of reward model overoptimization; and the audit test, education measurement's method for measuring the effect.

References for Part II

412 works cited across this part's chapters.

Abel, D., Dabney, W., Harutyunyan, A., Ho, M. K., Littman, M. L., Precup, D., and Singh, S. (2021). On the Expressivity of Markov Reward. Advances in Neural Information Processing Systems. Ch. 7
Achiam, J., Held, D., Tamar, A., and Abbeel, P. (2017). Constrained Policy Optimization. International Conference on Machine Learning. Ch. 12
Akerlof, G. A. (1970). The Market for "Lemons": Quality Uncertainty and the Market Mechanism. The Quarterly Journal of Economics. Ch. 14
Akerlof, G. A., and Kranton, R. E. (2000). Economics and Identity*. Quarterly Journal of Economics. Ch. 14
Akerlof, G. A., and Kranton, R. E. (2005). Identity and the Economics of Organizations. Journal of Economic Perspectives. Ch. 14
Akyürek, E., Schuurmans, D., Andreas, J., Ma, T., and Zhou, D. (2022). What learning algorithm is in-context learning? Investigations with linear models. International Conference on Learning Representations. Ch. 10
Allais, M. (1953). Le Comportement de l'Homme Rationnel devant le Risque: Critique des Postulats et Axiomes de l'Ecole Americaine. Econometrica. Ch. 7
Allcott, H., Braghieri, L., Eichmeyer, S., and Gentzkow, M. (2020). The Welfare Effects of Social Media. American Economic Review. Ch. 13
Altman, E. (1999). Constrained Markov Decision Processes. Routledge. Ch. 7
Anderson, P. W. (1972). More Is Different. Science. Ch. 8
Andreas, J. (2022). Language Models as Agent Models. Findings of the Association for Computational Linguistics: EMNLP 2022. doi:10.18653/v1/2022.findings-emnlp.423. Ch. 11
Anil, C., Durmus, E., Panickssery, N., Sharma, M., Benton, J., Kundu, S., … Duvenaud, D. (2024). Many-shot Jailbreaking. Advances in Neural Information Processing Systems. doi:10.52202/079017-4121. Ch. 11
Anthropic (2023g). Tracing model outputs to the training data. Anthropic. non-peer-reviewed Ch. 10
Anthropic (2026au). The persona selection model. Anthropic. non-peer-reviewed Ch. 11
Anthropic (2026av). The Persona Selection Model: Why AI Assistants might Behave like Humans. Anthropic Alignment Science Blog. non-peer-reviewed Ch. 11
Anthropic (2026e). An alignment assessment of the UK AISI cybertesting incident. Anthropic. non-peer-reviewed Ch. 11
Armstrong, S., and Mindermann, S. (2018). Occam's razor is insufficient to infer the preferences of irrational agents. Advances in Neural Information Processing Systems. Ch. 13
Avenhaus, R., and Canty, M. J. (1996). Compliance Quantified. Cambridge University Press. Ch. 14
Avenhaus, R., Von Stengel, B., and Zamir, S. (2002). Chapter 51 Inspection games. Handbook of Game Theory with Economic Applications. Ch. 14
Axelsson, S. (2000). The base-rate fallacy and the difficulty of intrusion detection. ACM Transactions on Information and System Security. Ch. 6
Ayres, I., and Gertner, R. (1989). Filling Gaps in Incomplete Contracts: An Economic Theory of Default Rules. The Yale Law Journal. Ch. 14
Azar, M. G., Rowland, M., Piot, B., Guo, D., Calandriello, D., Valko, M., and Munos, R. (2024). A General Theoretical Paradigm to Understand Learning from Human Preferences. Proceedings of The 27th International Conference on Artificial Intelligence and Statistics (PMLR 238). Ch. 12
Bae, J., Ng, N., Lo, A., Ghassemi, M., and Grosse, R. (2022). If Influence Functions Are the Answer, Then What is the Question? Advances in Neural Information Processing Systems. doi:10.52202/068431-1305. Ch. 10
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., … Kaplan, J. (2022a). Constitutional AI: Harmlessness from AI Feedback. arXiv. preprint Ch. 11
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., … Kaplan, J. (2022b). Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv. preprint Ch. 12
Baker, G. P. (1992). Incentive Contracts and Performance Measurement. Journal of Political Economy. Ch. 14
Baker, G. (2002). Distortion and Risk in Optimal Incentive Contracts. The Journal of Human Resources. Ch. 14
Baker, G., Gibbons, R., and Murphy, K. J. (1994). Subjective Performance Measures in Optimal Incentive Contracts. The Quarterly Journal of Economics. Ch. 14
Baker, G., Wang, G., Hoogland, J., and Murfet, D. (2026). Structural Inference: Interpreting Small Language Models with Susceptibilities. International Conference on Learning Representations. Ch. 10
Barak, B., Edelman, B., Goel, S., Kakade, S., Malach, E., and Zhang, C. (2022). Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit. Advances in Neural Information Processing Systems. doi:10.52202/068431-1581. Ch. 9
Bartlett, P. L., Long, P. M., Lugosi, G., and Tsigler, A. (2020). Benign overfitting in linear regression. Proceedings of the National Academy of Sciences. doi:10.1073/pnas.1907378117. Ch. 9
Basu, S., Pope, P., and Feizi, S. (2020). Influence Functions in Deep Learning Are Fragile. International Conference on Learning Representations. Ch. 10
Beirami, A., Agarwal, A., Berant, J., D'Amour, A., Eisenstein, J., Nagpal, C., and Suresh, A. T. (2025). Theoretical guarantees on the best-of-n alignment policy. International Conference on Machine Learning. Ch. 12
Belkin, M., Hsu, D., Ma, S., and Mandal, S. (2019). Reconciling modern machine-learning practice and the classical bias–variance trade-off. Proceedings of the National Academy of Sciences. doi:10.1073/pnas.1903070116. Ch. 9
Belrose, N., and Pope, Q. (2024). Counting arguments provide no evidence for AI doom. AI Alignment Forum. non-peer-reviewed Ch. 9
Belrose, N., Pope, Q., Quirke, L., Mallen, A., and Fern, X. (2024). Neural Networks Learn Statistics of Increasing Complexity. International Conference on Machine Learning. Ch. 9
Bender, E. M., and Koller, A. (2020). Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Ch. 11
Berglund, L., Stickland, A. C., Balesni, M., Kaufmann, M., Tong, M., Korbak, T., Kokotajlo, D., and Evans, O. (2023a). Taken out of context: On measuring situational awareness in LLMs. arXiv. preprint Ch. 11
Bergman, S., Marchal, N., Mellor, J., Mohamed, S., Gabriel, I., and Isaac, W. (2024). STELA: a community-centred approach to norm elicitation for AI alignment. Scientific Reports. Ch. 13
Bernheim, B. D., and Rangel, A. (2009). Beyond Revealed Preference: Choice-Theoretic Foundations for Behavioral Welfare Economics *. Quarterly Journal of Economics. Ch. 13
Bernheim, B. D., and Whinston, M. D. (1986a). Common Agency. Econometrica. Ch. 14
Bernheim, B. D., and Whinston, M. D. (1986b). Menu Auctions, Resource Allocation, and Economic Influence. The Quarterly Journal of Economics. Ch. 14
Besley, T., and Ghatak, M. (2005). Competition and Incentives with Motivated Agents. American Economic Review. Ch. 14
Betley, J., Warncke, N., Sztyber-Betley, A., Tan, D., Bao, X., Soto, M., … Evans, O. (2026a). Training large language models on narrow tasks can lead to broad misalignment. Nature. Ch. 11
Bevan, G., and Hood, C. (2006). What’s Measured Is What Matters: Targets and Gaming in the English Public Health Care System. Public Administration. Ch. 15
Bhatt, A., Rushing, C., Kaufman, A., Tracy, T., Georgiev, V., Matolcsi, D., Khan, A., and Shlegeris, B. (2025). Ctrl-Z: Controlling AI Agents via Resampling. arXiv. preprint Ch. 14
Birhane, A. (2021). Algorithmic injustice: a relational ethics approach. Patterns. Ch. 13
Blau, P. M. (1955). The dynamics of bureaucracy: a study of interpersonal relations in two government agencies. University of Chicago Press. Ch. 15
Bowling, M., Martin, J. D., Abel, D., and Dabney, W. (2022). Settling the Reward Hypothesis. arXiv. preprint Ch. 7
Bradley, R. A., and Terry, M. E. (1952). Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika. Ch. 13
Breland, K., and Breland, M. (1961). The misbehavior of organisms. American Psychologist. Ch. 9
Brier, G. W. (1950). Verification of Forecasts Expressed in Terms of Probability. Monthly Weather Review. Ch. 6
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., … Amodei, D. (2020). Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems. Ch. 11
Buchak, L. (2013). Risk and Rationality. Oxford University Press. Ch. 7
Butler, R., and Finelli, G. (1993). The infeasibility of quantifying the reliability of life-critical real-time software. IEEE Transactions on Software Engineering. Ch. 6
Butlin, P. (2024). Reinforcement learning and artificial agency. Mind & Language. Ch. 7
Campbell, D. T. (1979). Assessing the impact of planned social change. Evaluation and Program Planning. Ch. 15
Cao, H., Cohen, S. N., and Szpruch, L. (2021a). Identifiability in inverse reinforcement learning. Advances in Neural Information Processing Systems. Ch. 13
Capen, E., Clapp, R., and Campbell, W. (1971). Competitive Bidding in High-Risk Situations. Journal of Petroleum Technology. Ch. 7
Carey, R., Langlois, E., van Merwijk, C., Legg, S., and Everitt, T. (2025). Incentives for responsiveness, instrumental control and impact. Artificial Intelligence. doi:10.1016/j.artint.2025.104408. Ch. 14
Carlsmith, J. (2023). Scheming AIs: Will AIs fake alignment during training in order to get power? arXiv. preprint Ch. 9 Ch. 11
Carroll, G. (2015). Robustness and Linear Contracts. American Economic Review. Ch. 14
Carroll, M., Dragan, A., Russell, S., and Hadfield-Menell, D. (2022). Estimating and Penalizing Induced Preference Shifts in Recommender Systems. International Conference on Machine Learning. Ch. 13
Carroll, M., Chan, A., Ashton, H., and Krueger, D. (2023). Characterizing Manipulation from AI Systems. Equity and Access in Algorithms, Mechanisms, and Optimization. Ch. 14
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., … Hadfield-Menell, D. (2023). Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback. arXiv. preprint Ch. 15
Cawley, G. C., and Talbot, N. L. C. (2010). On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation. Journal of Machine Learning Research. Ch. 7 Ch. 15
Cérou, F., and Guyader, A. (2007). Adaptive Multilevel Splitting for Rare Event Analysis. Stochastic Analysis and Applications. Ch. 6
Chan, A., Ezell, C., Kaufmann, M., Wei, K., Hammond, L., Bradley, H., … Anderljung, M. (2024). Visibility into AI Agents. Conference on Fairness, Accountability and Transparency. doi:10.1145/3630106.3658948. Ch. 11
Chang, R. (1997). Incommensurability, Incomparability, and Practical Reason. Harvard University Press. Ch. 13
Chen, Z., Lau, E., Mendel, J., Wei, S., and Murfet, D. (2023b). Dynamical versus Bayesian Phase Transitions in a Toy Model of Superposition. arXiv. preprint Ch. 10
Chen, Y., Liu, T. X., Shan, Y., and Zhong, S. (2023c). The emergence of economic rationality of GPT. Proceedings of the National Academy of Sciences. Ch. 7
Chen, R., Arditi, A., Sleight, H., Evans, O., and Lindsey, J. (2025c). Persona Vectors: Monitoring and Controlling Character Traits in Language Models. arXiv. preprint Ch. 11
Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., and Amodei, D. (2017). Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems. Ch. 12 Ch. 13
Cloud, A., Le, M., Chua, J., Betley, J., Sztyber-Betley, A., Hilton, J., Marks, S., and Evans, O. (2025). Subliminal Learning: Language models transmit behavioral traits via hidden signals in data. arXiv. preprint Ch. 10
Cohen, M. K., Hutter, M., and Osborne, M. A. (2022). Advanced artificial agents intervene in the provision of reward. AI Magazine. Ch. 7
Coles, S. (2001). An Introduction to Statistical Modeling of Extreme Values. Springer London. Ch. 6
Conitzer, V., Freedman, R., Heitzig, J., Holliday, W. H., Jacobs, B. M., Lambert, N., … Zwicker, W. S. (2024). Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback. arXiv. preprint Ch. 13
Coste, T., Anwar, U., Kirk, R., and Krueger, D. (2024). Reward Model Ensembles Help Mitigate Overoptimization. International Conference on Learning Representations. Ch. 15
Cotra, A. (2022). Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover. AI Alignment Forum. non-peer-reviewed Ch. 7
Courty, P., and Marschke, G. (2004). An Empirical Investigation of Gaming Responses to Explicit Performance Incentives. Journal of Labor Economics. Ch. 14
Crawford, V. P., and Sobel, J. (1982). Strategic Information Transmission. Econometrica. Ch. 14
Critch, A., and Krueger, D. (2020). AI Research Considerations for Human Existential Safety (ARCHES). arXiv. preprint Ch. 14
Cybenko, G. (1989). Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals, and Systems. Ch. 8
D'Amour, A., Heller, K., Moldovan, D., Adlam, B., Alipanahi, B., Beutel, A., … Sculley, D. (2022). Underspecification Presents Challenges for Credibility in Modern Machine Learning. Journal of Machine Learning Research, 23(226):1−61. Ch. 9
Darby, M. R., and Karni, E. (1973). Free Competition and the Optimal Amount of Fraud. The Journal of Law and Economics. Ch. 14
DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., … Zhang, Z. (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. Nature. doi:10.1038/s41586-025-09422-z. Ch. 11 Ch. 12
Delétang, G., Ruoss, A., Duquenne, P.-A., Catt, E., Genewein, T., Mattern, C., … Veness, J. (2024). Language Modeling Is Compression. International Conference on Learning Representations. Ch. 6
Demski, A., and Garrabrant, S. (2019). Embedded Agency. arXiv. preprint Ch. 7
Dixit, A. (1997). Power of Incentives in Private versus Public Organizations. American Economic Review (Papers and Proceedings), vol. 87(2), pp. 378-382. Ch. 14
Dixit, A. (2002). Incentives and Organizations in the Public Sector: An Interpretative Review. The Journal of Human Resources. Ch. 14
Dixit, A., Grossman, G. M., and Helpman, E. (1997). Common Agency and Coordination: General Theory and Application to Government Policy Making. Journal of Political Economy. Ch. 14
Dranove, D., Kessler, D., McClellan, M., and Satterthwaite, M. (2003). Is More Information Better? The Effects of “Report Cards” on Health Care Providers. Journal of Political Economy. Ch. 14 Ch. 15
Dresher, M. (1962). A Sampling Inspection Problem in Arms Control Agreements: A Game-Theoretic Analysis. RAND Corporation (RM-2972-ARPA). non-peer-reviewed Ch. 14
Dulleck, U., and Kerschbamer, R. (2006). On Doctors, Mechanics, and Computer Specialists: The Economics of Credence Goods. Journal of Economic Literature. Ch. 14
Eckhardt, D., and Lee, L. (1985). A Theoretical Basis for the Analysis of Multiversion Software Subject to Coincident Errors. IEEE Transactions on Software Engineering. Ch. 6
Ecoffet, A., and Lehman, J. (2020). Reinforcement Learning Under Moral Uncertainty. arXiv. preprint Ch. 7
Efron, B. (2011). Tweedie’s Formula and Selection Bias. Journal of the American Statistical Association. Ch. 7 Ch. 15
Egorov, G., and Sonin, K. (2011). Dictators and Their Viziers: Endogenizing the Loyalty-competence Trade-off. Journal of the European Economic Association. Ch. 14
El-Mhamdi, E.-M., and Hoang, L.-N. (2024). On Goodhart's law, with an application to value alignment. arXiv. preprint Ch. 15
Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., … Olah, C. (2021a). A Mathematical Framework for Transformer Circuits. Transformer Circuits Thread. non-peer-reviewed Ch. 11
Ellsberg, D. (1961). Risk, Ambiguity, and the Savage Axioms. The Quarterly Journal of Economics. Ch. 7
Espeland, W. N., and Sauder, M. (2007). Rankings and Reactivity: How Public Measures Recreate Social Worlds. American Journal of Sociology. Ch. 15
Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. (2024). KTO: Model Alignment as Prospect Theoretic Optimization. International Conference on Machine Learning. Ch. 12 Ch. 13
Everitt, T., Krakovna, V., Orseau, L., and Legg, S. (2017). Reinforcement Learning with a Corrupted Reward Channel. Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence. Ch. 7
Everitt, T., Ortega, P. A., Barnes, E., and Legg, S. (2019). Understanding Agent Incentives using Causal Influence Diagrams. Part I: Single Action Settings. arXiv. preprint Ch. 14
Everitt, T., Carey, R., Langlois, E. D., Ortega, P. A., and Legg, S. (2021a). Agent Incentives: A Causal Perspective. Proceedings of the AAAI Conference on Artificial Intelligence. Ch. 14
Everitt, T., Hutter, M., Kumar, R., and Krakovna, V. (2021b). Reward tampering problems and solutions in reinforcement learning: a causal influence diagram perspective. Synthese. Ch. 7 Ch. 14
Farquhar, S., Carey, R., and Everitt, T. (2022). Path-Specific Objectives for Safer Agent Incentives. Proceedings of the AAAI Conference on Artificial Intelligence. Ch. 14
Farrell, H., Gopnik, A., Shalizi, C., and Evans, J. (2025). Large AI models are cultural and social technologies. Science. Ch. 11 Ch. 14
Fawcett, T. (2006). An introduction to ROC analysis. Pattern Recognition Letters. Ch. 6
Fearon, J. D. (1999). Electoral Accountability and the Control of Politicians: Selecting Good Types versus Sanctioning Poor Performance. Democracy, Accountability, and Representation. Ch. 14
Feaver, P. D. (1996). The Civil-Military Problematique: Huntington, Janowitz, and the Question of Civilian Control. Armed Forces & Society. Ch. 14
Fedus, W., Zoph, B., and Shazeer, N. (2021). Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. Journal of machine learning research. Ch. 11
Ferejohn, J. (1986). Incumbent performance and electoral control. Public Choice. Ch. 14
Fickinger, A., Zhuang, S., Hadfield-Menell, D., and Russell, S. (2020a). Multi-Principal Assistance Games. arXiv. preprint Ch. 14
Finzi, M., Qiu, S., Jiang, Y., Izmailov, P., Kolter, J. Z., and Wilson, A. G. (2026). From Entropy to Epiplexity: Rethinking Information for Computationally Bounded Intelligence. arXiv. preprint Ch. 6
Fischhoff, B. (1991). Value elicitation: Is there anything in there? American Psychologist. Ch. 13
Fluri, L., Lang, L., Abate, A., Forré, P., Krueger, D., and Skalse, J. (2025). The Perils of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low Regret. International Conference on Machine Learning. Ch. 12 Ch. 15
Frankle, J., and Carbin, M. (2018). The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks. International Conference on Learning Representations. Ch. 8
Frankle, J., Dziugaite, G. K., Roy, D. M., and Carbin, M. (2019). Linear Mode Connectivity and the Lottery Ticket Hypothesis. International Conference on Machine Learning. Ch. 8
Gao, L., Schulman, J., and Hilton, J. (2022). Scaling Laws for Reward Model Overoptimization. International Conference on Machine Learning. Ch. 6 Ch. 12 Ch. 15
Garcia, J., and Koelling, R. A. (1966). Relation of cue to consequence in avoidance learning. Psychonomic Science. Ch. 9
Garrabrant, S., Benson-Tilsen, T., Critch, A., Soares, N., and Taylor, J. (2016). Logical Induction. arXiv. preprint Ch. 7
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., and Wichmann, F. A. (2020). Shortcut learning in deep neural networks. Nature Machine Intelligence. Ch. 8
Geist, M., Scherrer, B., and Pietquin, O. (2019). A Theory of Regularized Markov Decision Processes. International Conference on Machine Learning. Ch. 7
Geng, J., Cai, F., Wang, Y., Koeppl, H., Nakov, P., and Gurevych, I. (2023). A Survey of Confidence Estimation and Calibration in Large Language Models. North American Chapter of the Association for Computational Linguistics. doi:10.18653/v1/2024.naacl-long.366. Ch. 6
Gibbons, R. (1998). Incentives in Organizations. Journal of Economic Perspectives. Ch. 14
Gneiting, T., and Raftery, A. E. (2007). Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association. Ch. 6
Goldstein, S., and Kirk-Giannini, C. D. (2026). A Case for AI Consciousness: Language Agents and Global Workspace Theory. Journal of Consciousness Studies. doi:10.53765/20512201.33.7.061. Ch. 11
Goodfellow, I. J., Vinyals, O., and Saxe, A. M. (2014). Qualitatively characterizing neural network optimization problems. International Conference on Learning Representations. Ch. 8
Goodfellow, I. J., Shlens, J., and Szegedy, C. (2015). Explaining and Harnessing Adversarial Examples. International Conference on Learning Representations (ICLR 2015). Ch. 8
Goodhart, C. A. E. (1984). Problems of Monetary Management: The UK Experience. Monetary Theory and Practice. Ch. 15
Green, D. M., and Swets, J. A. (1966). Signal Detection Theory and Psychophysics. Wiley. Ch. 6
Greenblatt, R., Shlegeris, B., Sachan, K., and Roger, F. (2024a). AI Control: Improving Safety Despite Intentional Subversion. International Conference on Machine Learning. Ch. 14
Griffin, C., Thomson, L., Shlegeris, B., and Abate, A. (2026). Games for AI Control: Models of Safety Evaluations of AI Deployment Protocols. Proceedings of IASEAI Conference. Ch. 14
Grosse, R., Bae, J., Anil, C., Elhage, N., Tamkin, A., Tajdini, A., … Bowman, S. R. (2023). Studying Large Language Model Generalization with Influence Functions. arXiv. preprint Ch. 10
Grossman, S. J., and Hart, O. D. (1986). The Costs and Benefits of Ownership: A Theory of Vertical and Lateral Integration. Journal of Political Economy. Ch. 14
Gui, L., Gârbacea, C., and Veitch, V. (2024). BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Sampling. Advances in Neural Information Processing Systems. doi:10.52202/079017-0094. Ch. 12
Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. (2017). On Calibration of Modern Neural Networks. International Conference on Machine Learning. Ch. 6
Gustafsson, J. E. (2022). Money-Pump Arguments. Cambridge University Press. Ch. 7
Hadfield, G. K., and Koh, A. (2025). An Economy of AI Agents. arXiv. preprint Ch. 13
Hadfield-Menell, D., and Hadfield, G. K. (2019). Incomplete Contracting and AI Alignment. Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society. Ch. 14
Hadfield-Menell, D., Milli, S., Abbeel, P., Russell, S., and Dragan, A. (2017a). Inverse Reward Design. Advances in Neural Information Processing Systems. Ch. 15
Hägele, A., Gema, A. P., Sleight, H., Perez, E., and Sohl-Dickstein, J. (2026). The Hot Mess of AI: How Does Misalignment Scale With Model Intelligence and Task Complexity? arXiv. preprint Ch. 7
Hammond, L., Fox, J., Everitt, T., Carey, R., Abate, A., and Wooldridge, M. (2023). Reasoning about causality in games. Artificial Intelligence. doi:10.1016/j.artint.2023.103919. Ch. 14
Hammoudeh, Z., and Lowd, D. (2024). Training data influence analysis and estimation: a survey. Machine Learning. doi:10.1007/s10994-023-06495-7. Ch. 10
Hanley, J. A. (1983). If Nothing Goes Wrong, Is Everything All Right? JAMA. Ch. 6
Hardt, M., Megiddo, N., Papadimitriou, C., and Wootters, M. (2016). Strategic Classification. Proceedings of the 2016 ACM Conference on Innovations in Theoretical Computer Science. Ch. 15
Harrison, J. R., and March, J. G. (1984). Decision Making and Postdecision Surprises. Administrative Science Quarterly. Ch. 7
Harsanyi, J. C. (1955). Cardinal Welfare, Individualistic Ethics, and Interpersonal Comparisons of Utility. Journal of Political Economy. Ch. 7
Hart, O., and Moore, J. (1990). Property Rights and the Nature of the Firm. Journal of Political Economy. Ch. 14
Hastie, T., Montanari, A., Rosset, S., and Tibshirani, R. J. (2022). Surprises in high-dimensional ridgeless least squares interpolation. The Annals of Statistics. doi:10.1214/21-aos2133. Ch. 9
Hausman, D. M. (2011). Preference, Value, Choice, and Welfare. Cambridge University Press. Ch. 13
He, K., Zhang, X., Ren, S., and Sun, J. (2015a). Deep Residual Learning for Image Recognition. Computer Vision and Pattern Recognition. doi:10.1109/cvpr.2016.90. Ch. 8
Hebbar, V., and Hubinger, E. (2022). Path dependence in ML inductive biases. AI Alignment Forum. non-peer-reviewed Ch. 9
Hendel, R., Geva, M., and Globerson, A. (2023). In-Context Learning Creates Task Vectors. Conference on Empirical Methods in Natural Language Processing. Ch. 11
Hendrycks, D., and Gimpel, K. (2016a). A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks. International Conference on Learning Representations. Ch. 8
Hendrycks, D., and Gimpel, K. (2016b). Gaussian Error Linear Units (GELUs). arXiv. preprint Ch. 8
Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., … McCandlish, S. (2020). Scaling Laws for Autoregressive Generative Modeling. arXiv. preprint Ch. 6 Ch. 8
Henighan, T., Carter, S., Hume, T., Elhage, N., Lasenby, R., Fort, S., Schiefer, N., and Olah, C. (2023). Superposition, Memorization, and Double Descent. Transformer Circuits Thread. non-peer-reviewed Ch. 9
Hennessy, C. A., and Goodhart, C. A. E. (2023). Goodhart's Law and Machine Learning: a Structural Perspective. International Economic Review. Ch. 15
Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., … Zhou, Y. (2017). Deep Learning Scaling is Predictable, Empirically. arXiv. preprint Ch. 8
Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the Knowledge in a Neural Network. arXiv. preprint Ch. 11
Ho, A., Besiroglu, T., Erdil, E., Owen, D., Rahman, R., Guo, Z., … Sevilla, J. (2024). Algorithmic progress in language models. Advances in Neural Information Processing Systems. Ch. 8
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., … Sifre, L. (2022). Training Compute-Optimal Large Language Models. arXiv. preprint Ch. 8
Holmstrom, B. (1979). Moral Hazard and Observability. The Bell Journal of Economics. Ch. 14
Holmström, B. (2017). Pay For Performance and Beyond. American Economic Review. Ch. 14
Holmstrom, B., and Milgrom, P. (1991). Multitask Principal–Agent Analyses: Incentive Contracts, Asset Ownership, and Job Design. The Journal of Law, Economics, and Organization. Ch. 14
Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. (2019). The Curious Case of Neural Text Degeneration. International Conference on Learning Representations. Ch. 11
Hoogland, J., Oldenziel, A. G., Murfet, D., and van Wingerden, S. (2023). Towards Developmental Interpretability. AI Alignment Forum. non-peer-reviewed Ch. 10
Hoogland, J., Wang, G., Farrugia-Roberts, M., Carroll, L., Wei, S., and Murfet, D. (2024). Loss Landscape Degeneracy and Stagewise Development in Transformers. Trans. Mach. Learn. Res.. Ch. 10
Hornik, K., Stinchcombe, M., and White, H. (1989). Multilayer feedforward networks are universal approximators. Neural Networks. Ch. 8
Hosking, T., Blunsom, P., and Bartolo, M. (2023). Human Feedback is not Gold Standard. arXiv. preprint Ch. 13
Howard, R. A., and Matheson, J. E. (2005). Influence Diagrams. Decision Analysis. Ch. 7 Ch. 14
Huang, A., Zhan, W., Xie, T., Lee, J. D., Sun, W., Krishnamurthy, A., and Foster, D. J. (2025a). Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization. International Conference on Learning Representations. Ch. 12 Ch. 15
Hubinger, E. (2022). How likely is deceptive alignment? AI Alignment Forum. non-peer-reviewed Ch. 9
Hubinger, E., van Merwijk, C., Mikulik, V., Skalse, J., and Garrabrant, S. (2019). Risks from Learned Optimization in Advanced Machine Learning Systems. arXiv. preprint Ch. 10 Ch. 11
Huntington, S. P. (1957). The Soldier and the State. Harvard University Press. Ch. 14
Hurwicz, L. (2008). But Who Will Guard the Guardians? American Economic Review. Ch. 14
Hutter, M. (2004). Universal Artificial Intelligence. Springer Berlin / Heidelberg. Ch. 7
Inozemtseva, L., and Holmes, R. (2014). Coverage is not strongly correlated with test suite effectiveness. Proceedings of the 36th International Conference on Software Engineering. Ch. 15
Ivanov, D., Dütting, P., Talgam-Cohen, I., Wang, T., and Parkes, D. C. (2024). Principal-Agent Reinforcement Learning: Orchestrating AI Agents with Contracts. arXiv. preprint Ch. 14
Jacob, B. A. (2005). Accountability, incentives and behavior: the impact of high-stakes testing in the Chicago Public Schools. Journal of Public Economics. Ch. 14 Ch. 15
Jacob, B. A., and Levitt, S. D. (2003). Rotten Apples: An Investigation of the Prevalence and Predictors of Teacher Cheating. The Quarterly Journal of Economics. Ch. 14 Ch. 15
janus (2022). Simulators. AI Alignment Forum. non-peer-reviewed Ch. 11
Jaques, N., Gu, S., Bahdanau, D., Hernández-Lobato, J. M., Turner, R. E., and Eck, D. (2017). Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-control. International Conference on Machine Learning. Ch. 12
Jeffrey, R. C. (1965). The Logic of Decision. University Of Chicago Press. Ch. 7
Jenner, E., Kapur, S., Georgiev, V., Allen, C., Emmons, S., and Russell, S. (2024). Evidence of Learned Look-Ahead in a Chess-Playing Neural Network. Advances in Neural Information Processing Systems. doi:10.52202/079017-0987. Ch. 10
Jensen, M. C., and Meckling, W. H. (1976). Theory of the firm: Managerial behavior, agency costs and ownership structure. Journal of Financial Economics. Ch. 14
Jeon, H. J., Milli, S., and Dragan, A. D. (2020). Reward-rational (implicit) choice: A unifying formalism for reward learning. Advances in Neural Information Processing Systems. Ch. 13
Ji, J., Wang, K., Qiu, T. A., Chen, B., Zhou, J., Li, C., … Yang, Y. (2025). Language Models Resist Alignment: Evidence From Data Compression. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Ch. 6
Jiang, Y., Neyshabur, B., Mobahi, H., Krishnan, D., and Bengio, S. (2020). Fantastic Generalization Measures and Where to Find Them. International Conference on Learning Representations. Ch. 9 Ch. 10
Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. (2023). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? International Conference on Learning Representations. Ch. 11
Jones, E., Tong, M., Mu, J., Mahfoud, M., Leike, J., Grosse, R., … Sharma, M. (2025). Forecasting Rare Language Model Behaviors. arXiv. preprint Ch. 6
Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., … Kaplan, J. (2022). Language Models (Mostly) Know What They Know. arXiv. preprint Ch. 6
Kahneman, D., and Tversky, A. (1979). Prospect Theory: An Analysis of Decision under Risk. Econometrica. Ch. 7 Ch. 13
Kahneman, D., Wakker, P. P., and Sarin, R. (1997). Back to Bentham? Explorations of Experienced Utility. The Quarterly Journal of Economics. Ch. 13
Kalai, A. T., Nachum, O., Vempala, S. S., and Zhang, E. (2025). Why Language Models Hallucinate. arXiv. preprint Ch. 6
Kalra, N., and Paddock, S. M. (2016). Driving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability? Transportation Research Part A: Policy and Practice. Ch. 6
Kamenica, E., and Gentzkow, M. (2011). Bayesian Persuasion. American Economic Review. Ch. 14
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., … Amodei, D. (2020). Scaling Laws for Neural Language Models. arXiv. preprint Ch. 6 Ch. 8 Ch. 9
Kappen, H. J., Gómez, V., and Opper, M. (2012). Optimal control as a graphical model inference problem. Machine Learning. Ch. 12
Karwowski, J., Hayman, O., Bai, X., Kiendlhofer, K., Griffin, C., and Skalse, J. (2023). Goodhart's Law in Reinforcement Learning. International Conference on Learning Representations. Ch. 15
Kenton, Z., Kumar, R., Farquhar, S., Richens, J., MacDermott, M., and Everitt, T. (2023). Discovering agents. Artificial Intelligence. Ch. 7 Ch. 14
Kerr, S. (1975). On the Folly of Rewarding A, While Hoping for B. Academy of Management Journal. Ch. 14 Ch. 15
Khan, A., Casper, S., and Hadfield-Menell, D. (2025). Randomness, Not Representation: The Unreliability of Evaluating Cultural Alignment in LLMs. Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency. Ch. 7
Kim, E., Garg, A., Peng, K., and Garg, N. (2025). Correlated Errors in Large Language Models. arXiv. preprint Ch. 6
Kimi Team, Du, A., Gao, B., Xing, B., Jiang, C., Chen, C., … Lin, Z. (2025). Kimi k1.5: Scaling Reinforcement Learning with LLMs. arXiv. preprint Ch. 12
Kingma, D. P., and Ba, J. (2014). Adam: A Method for Stochastic Optimization. International Conference on Learning Representations. Ch. 8
Klein, S., Hamilton, L., McCaffrey, D., and Stecher, B. (2000). What Do Test Scores in Texas Tell Us? RAND Corporation. Ch. 15
Kleinberg, J., and Raghavan, M. (2020). How Do Classifiers Induce Agents to Invest Effort Strategically? ACM Transactions on Economics and Computation. Ch. 14 Ch. 15
Kleinberg, J., Mullainathan, S., and Raghavan, M. (2023). The Challenge of Understanding What Users Want: Inconsistent Preferences and Engagement Optimization. Management Science. Ch. 13
Knight, J. C., and Leveson, N. G. (1986). An experimental evaluation of the assumption of independence in multiversion programming. IEEE Transactions on Software Engineering. Ch. 6
Knox, W. B., Hatgis-Kessell, S., Booth, S., Niekum, S., Stone, P., and Allievi, A. (2022). Models of human preference for learning reward functions. arXiv. preprint Ch. 13
Koh, P. W., and Liang, P. (2017). Understanding Black-box Predictions via Influence Functions. International Conference on Machine Learning. Ch. 10
Koller, D., and Milch, B. (2003). Multi-agent influence diagrams for representing and solving games. Games and Economic Behavior. Ch. 14
Korbak, T., Perez, E., and Buckley, C. (2022). RL with KL penalties is better viewed as Bayesian inference. Findings of the Association for Computational Linguistics: EMNLP 2022. doi:10.18653/v1/2022.findings-emnlp.77. Ch. 6 Ch. 12
Koretz, D. (2008). Measuring Up. Harvard University Press. Ch. 15
Koretz, D. (2017). The Testing Charade. University of Chicago Press. Ch. 15
Koretz, D., Jennings, J. L., Ng, H. L., Yu, C., Braslow, D., and Langi, M. (2016). Auditing for Score Inflation Using Self-Monitoring Assessments: Findings From Three Pilot Studies. Educational Assessment. Ch. 15
Korinek, A., and Balwit, A. (2023). Aligned with Whom? The Oxford Handbook of AI Governance. Ch. 14
Korzhyk, D., Yin, Z., Kiekintveld, C., Conitzer, V., and Tambe, M. (2011). Stackelberg vs. Nash in Security Games: An Extended Investigation of Interchangeability, Equivalence, and Uniqueness. Journal of Artificial Intelligence Research. Ch. 14
Krakovna, V., Uesato, J., Mikulik, V., Rahtz, M., Everitt, T., Kumar, R., … Legg, S. (2020b). Specification gaming: the flip side of AI ingenuity. Google DeepMind blog. non-peer-reviewed Ch. 15
Kullback, S., and Leibler, R. A. (1951). On Information and Sufficiency. The Annals of Mathematical Statistics. Ch. 6
Kwa, T., Thomas, D., and Garriga-Alonso, A. (2024). Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification. Advances in Neural Information Processing Systems. doi:10.52202/079017-0466. Ch. 12 Ch. 15
Kwa, T., West, B., Becker, J., Deng, A., Garcia, K., Hasin, M., … Chan, L. (2025). Measuring AI Ability to Complete Long Software Tasks. Advances in Neural Information Processing Systems 38. Ch. 8 Ch. 11
Laidlaw, C., Singhal, S., and Dragan, A. (2025). Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking. International Conference on Learning Representations. Ch. 12 Ch. 15
Lakshminarayanan, B., Pritzel, A., and Blundell, C. (2016). Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. Advances in Neural Information Processing Systems. Ch. 6 Ch. 8
Lambert, N., Morrison, J., Pyatkin, V., Huang, S., Ivison, H., Brahman, F., … Hajishirzi, H. (2024). Tulu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv. preprint Ch. 11 Ch. 12
Langlois, E. D., and Everitt, T. (2021). How RL Agents Behave When Their Actions Are Modified. Proceedings of the AAAI Conference on Artificial Intelligence. doi:10.1609/aaai.v35i13.17378. Ch. 14
Lapuschkin, S., Wäldchen, S., Binder, A., Montavon, G., Samek, W., and Müller, K.-R. (2019). Unmasking Clever Hans predictors and assessing what machines really learn. Nature Communications. Ch. 8
Lau, E., Furman, Z., Wang, G., Murfet, D., and Wei, S. (2023). The Local Learning Coefficient: A Singularity-Aware Complexity Measure. arXiv. preprint Ch. 10
Levine, S. (2018). Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review. arXiv. preprint Ch. 12
Levinstein, B. A., Soares, N., and Journal of Philosophy Inc. (2020). Cheating Death in Damascus. The Journal of Philosophy. Ch. 7
Lewis, D. (1981). Causal decision theory. Australasian Journal of Philosophy. Ch. 7
Li, Z., Liu, H., Zhou, D., and Ma, T. (2024a). Chain of Thought Empowers Transformers to Solve Inherently Serial Problems. International Conference on Learning Representations. Ch. 11
Lichtenstein, S., and Slovic, P. (1971). Reversals of preference between bids and choices in gambling decisions. Journal of Experimental Psychology. Ch. 13
Lichtenstein, S. (., and Slovic, P. (. (2006). The Construction of Preference. Cambridge University Press. Ch. 13
Lichtenstein, S., Fischhoff, B., and Phillips, L. D. (1982). Calibration of probabilities: The state of the art to 1980. Judgment under Uncertainty. Ch. 6
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., … Cobbe, K. (2023). Let's Verify Step by Step. International Conference on Learning Representations. Ch. 12
Lin, S., Hilton, J., and Evans, O. (2022). Teaching Models to Express Their Uncertainty in Words. Trans. Mach. Learn. Res.. Ch. 6
Lin, B. Y., Ravichander, A., Lu, X., Dziri, N., Sclar, M., Chandu, K., Bhagavatula, C., and Choi, Y. (2024). The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning. International Conference on Learning Representations. Ch. 11
Littlewood, B., and Strigini, L. (1993). Validation of ultrahigh dependability for software-based systems. Communications of the ACM. Ch. 6
Liu, Z., Michaud, E. J., and Tegmark, M. (2022a). Omnigrok: Grokking Beyond Algorithmic Data. International Conference on Learning Representations. Ch. 9
Liu, M., Diao, S., Lu, X., Hu, J., Dong, X., Choi, Y., Kautz, J., and Dong, Y. (2025). ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models. Advances in Neural Information Processing Systems 38. doi:10.52202/085713-0608. Ch. 12
Löb, M. H. (1955). Solution of a problem of Leon Henkin. Journal of Symbolic Logic. Ch. 7
Lu, C., Gallagher, J., Michala, J., Fish, K., and Lindsey, J. (2026). The Assistant Axis: Situating and Stabilizing the Default Persona of Language Models. arXiv. preprint Ch. 11
Lucas, R. E. (1976). Econometric policy evaluation: A critique. Carnegie-Rochester Conference Series on Public Policy. Ch. 15
Macdermott, M., Fox, J., Belardinelli, F., and Everitt, T. (2024). Measuring Goal-Directedness. Advances in Neural Information Processing Systems. Ch. 7 Ch. 14
MacDermott, M., Everitt, T., and Belardinelli, F. (2023). Characterising Decision Theories with Mechanised Causal Graphs. arXiv. preprint Ch. 7 Ch. 14
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. (2018). Towards Deep Learning Models Resistant to Adversarial Attacks. International Conference on Learning Representations. Ch. 8
Mahowald, K., Ivanova, A. A., Blank, I. A., Kanwisher, N., Tenenbaum, J. B., and Fedorenko, E. (2024). Dissociating language and thought in large language models. Trends in Cognitive Sciences. Ch. 11
Maloney, A., Roberts, D. A., and Sully, J. (2022). A Solvable Model of Neural Scaling Laws. arXiv. preprint Ch. 8
Manheim, D., and Garrabrant, S. (2018). Categorizing Variants of Goodhart's Law. arXiv. preprint Ch. 15
Mansbridge, J. (2009). A “Selection Model” of Political Representation*. Journal of Political Philosophy. Ch. 14
Martimort, D., and Stole2, L. (2002). The Revelation and Delegation Principles in Common Agency Games. Econometrica. Ch. 14
Maschler, M. (1966). A price leadership method for solving the inspector's non‐constant‐sum game. Naval Research Logistics Quarterly. Ch. 14
Mazeika, M., Yin, X., Tamirisa, R., Lim, J., Lee, B. W., Ren, R., … Hendrycks, D. (2025). Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs. Advances in Neural Information Processing Systems 38. Ch. 7 Ch. 13
McCubbins, M. D., and Schwartz, T. (1984). Congressional Oversight Overlooked: Police Patrols versus Fire Alarms. American Journal of Political Science. Ch. 14
Merrill, W., and Sabharwal, A. (2023). The Expressive Power of Transformers with Chain of Thought. arXiv. preprint Ch. 11
Merton, R. K. (1936). The Unanticipated Consequences of Purposive Social Action. American Sociological Review. Ch. 15
Metz, T. (2007). Toward an African Moral Theory*. Journal of Political Philosophy. Ch. 13
Mhlambi, S. (2020). From Rationality to Relationality: Ubuntu as an Ethical and Human Rights Framework for Artificial Intelligence Governance. Carr Center Discussion Paper Series, 2020-009, Harvard Kennedy School. working paper Ch. 13
Miller, E. (2024). Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. arXiv. preprint Ch. 6
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L. (2022). Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? Conference on Empirical Methods in Natural Language Processing. doi:10.18653/v1/2022.emnlp-main.759. Ch. 11
Mingard, C., Valle-Pérez, G., Skalse, J., and Louis, A. A. (2021). Is SGD a Bayesian sampler? Well, almost. Journal of Machine Learning Research. Ch. 9
Mitchell, M., and Krakauer, D. C. (2023). The debate over understanding in AI’s large language models. Proceedings of the National Academy of Sciences. Ch. 11
Mullainathan, S., and Obermeyer, Z. (2017). Does Machine Learning Automate Moral Hazard and Error? American Economic Review. Ch. 14
Muller, J. (2018). The Tyranny of Metrics. Princeton University Press. Ch. 15
Myerson, R. B. (1979). Incentive Compatibility and the Bargaining Problem. Econometrica. Ch. 14
Nakkiran, P., Kaplun, G., Bansal, Y., Yang, T., Barak, B., and Sutskever, I. (2021). Deep double descent: where bigger models and more data hurt*. Journal of Statistical Mechanics: Theory and Experiment. doi:10.1088/1742-5468/ac3a74. Ch. 9
Nanda, N., Chan, L., Lieberum, T., Smith, J., and Steinhardt, J. (2023b). Progress measures for grokking via mechanistic interpretability. International Conference on Learning Representations. Ch. 9
Nardo, C. (2023). The Waluigi Effect (mega-post). AI Alignment Forum. non-peer-reviewed Ch. 11
Neal, D., and Schanzenbach, D. W. (2010). Left Behind by Design: Proficiency Counts and Test-Based Accountability. Review of Economics and Statistics. Ch. 15
Ng, A. Y., Harada, D., and Russell, S. (1999). Policy invariance under reward transformations: Theory and application to reward shaping. Proceedings of ICML-99 (International Conference on Machine Learning), Bled, Slovenia. Ch. 13
Ngo, R., Chan, L., and Mindermann, S. (2024). The Alignment Problem from a Deep Learning Perspective. International Conference on Learning Representations. Ch. 8
Nozick, R. (1969). Newcomb’s Problem and Two Principles of Choice. Essays in Honor of Carl G. Hempel. Ch. 7
Nye, M., Andreassen, A. J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., … Odena, A. (2021). Show Your Work: Scratchpads for Intermediate Computation with Language Models. arXiv. preprint Ch. 11
Obermeyer, Z., Powers, B., Vogeli, C., and Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science. Ch. 15
Oesterheld, C., Cooper, E., Kodama, M., Nguyen, L. C., and Perez, E. (2024). A dataset of questions on decision-theoretic reasoning in Newcomb-like problems. arXiv. preprint Ch. 7
Okasha, S. (2018). Agents and Goals in Evolution. Oxford University Press. Ch. 7
Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., … Olah, C. (2022). In-Context Learning and Induction Heads. Transformer Circuits Thread. non-peer-reviewed Ch. 9 Ch. 10 Ch. 11
Omohundro, S. M. (2008). The Basic AI Drives. Artificial Intelligence Safety and Security. doi:10.1201/9781351251389-3. Ch. 7
OpenAI (2026i). Encouraging deception in compaction summaries. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 11
OpenAI (2026v). Self-generated prompt injections in compaction summaries. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 11
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., … Zoph, B. (2023). GPT-4 Technical Report. arXiv. preprint Ch. 6
Ordonez, L. D., Schweitzer, M. E., Galinsky, A. D., and Bazerman, M. H. (2009). Goals Gone Wild: The Systematic Side Effects of Over-Prescribing Goal Setting. Elsevier BV. preprint Ch. 15
Orseau, L., and Ring, M. (2012). Space-Time Embedded Intelligence. Lecture Notes in Computer Science. Ch. 7
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., … Lowe, R. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems. doi:10.52202/068431-2011. Ch. 12
Owen, D. (2024a). How predictable is language model benchmark performance? arXiv. preprint Ch. 8
Pan, A., Bhatia, K., and Steinhardt, J. (2022). The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models. International Conference on Learning Representations. Ch. 15
Park, S. M., Georgiev, K., Ilyas, A., Leclerc, G., and Madry, A. (2023). TRAK: Attributing Model Behavior at Scale. International Conference on Machine Learning. Ch. 10
Park, K., Choe, Y. J., and Veitch, V. (2024c). The Linear Representation Hypothesis and the Geometry of Large Language Models. International Conference on Machine Learning. Ch. 11
Paul, L. A. (2014). Transformative Experience. Oxford University Press. Ch. 13
Pearl, J. (2009). Causality. Cambridge University Press. Ch. 14
Perdomo, J. C., Zrnic, T., Mendler-Dünner, C., and Hardt, M. (2020). Performative Prediction. arXiv. preprint Ch. 15
Plackett, R. L. (1975). The Analysis of Permutations. Applied Statistics. Ch. 13
Power, A., Burda, Y., Edwards, H., Babuschkin, I., and Misra, V. (2022). Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets. arXiv. preprint Ch. 9
Prendergast, C. (1999). The Provision of Incentives in Firms. Journal of Economic Literature. Ch. 15
Prendergast, C. (2007). The Motivation and Bias of Bureaucrats. American Economic Review. Ch. 14
Pruthi, G., Liu, F., Sundararajan, M., and Kale, S. (2020). Estimating Training Data Influence by Tracing Gradient Descent. Advances in Neural Information Processing Systems. Ch. 10
Qi, X., Zeng, Y., Xie, T., Chen, P.-Y., Jia, R., Mittal, P., and Henderson, P. (2024a). Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! International Conference on Learning Representations. Ch. 11
Qi, X., Panda, A., Lyu, K., Ma, X., Roy, S., Beirami, A., Mittal, P., and Henderson, P. (2024b). Safety Alignment Should Be Made More Than Just a Few Tokens Deep. International Conference on Learning Representations. Ch. 11
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language Models are Unsupervised Multitask Learners. OpenAI. non-peer-reviewed Ch. 11
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. Advances in Neural Information Processing Systems. Ch. 12
Rafailov, R., Hejna, J., Park, R., and Finn, C. (2024a). From $r$ to $Q^{*}$ : Your Language Model is Secretly a Q-Function. arXiv. preprint Ch. 12
Rafailov, R., Chittepu, Y., Park, R., Sikchi, H., Hejna, J., Knox, W., Finn, C., and Niekum, S. (2024b). Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms. Advances in Neural Information Processing Systems. doi:10.52202/079017-4009. Ch. 12 Ch. 15
Rahaman, N., Baratin, A., Arpit, D., Draxler, F., Lin, M., Hamprecht, F. A., Bengio, Y., and Courville, A. (2019). On the Spectral Bias of Neural Networks. International Conference on Machine Learning. Ch. 9
Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. (2019). Do ImageNet Classifiers Generalize to ImageNet? arXiv. preprint Ch. 8 Ch. 15
Refinetti, M., Ingrosso, A., and Goldt, S. (2023). Neural networks trained with SGD learn distributions of increasing complexity. International Conference on Machine Learning. Ch. 9
Richens, J., and Everitt, T. (2024). Robust agents learn causal world models. International Conference on Learning Representations. Ch. 14
Richens, J., Abel, D., Bellot, A., and Everitt, T. (2025). General agents need world models. International Conference on Machine Learning. Ch. 14
Ridgway, V. F. (1956). Dysfunctional Consequences of Performance Measurements. Administrative Science Quarterly. Ch. 15
Ring, M., and Orseau, L. (2011). Delusion, Survival, and Intelligent Agents. Lecture Notes in Computer Science. Ch. 7
Rosenblueth, A., Wiener, N., and Bigelow, J. (1943). Behavior, Purpose and Teleology. Philosophy of Science. Ch. 7
Rumelhart, D. E., Hinton, G. E., and Williams, R. J. (1986). Learning representations by back-propagating errors. Nature. Ch. 8
Samuelson, P. A. (1938). A Note on the Pure Theory of Consumer's Behaviour. Economica. Ch. 13
Savage, L. J. (1954). The Foundations of Statistics. Wiley. Ch. 7
Saxe, A. M., McClelland, J. L., and Ganguli, S. (2013). Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. International Conference on Learning Representations. Ch. 9
Schaeffer, R., Miranda, B., and Koyejo, S. (2023). Are Emergent Abilities of Large Language Models a Mirage? Advances in Neural Information Processing Systems. Ch. 8
Schulman, J., Levine, S., Moritz, P., Jordan, M. I., and Abbeel, P. (2015b). Trust Region Policy Optimization. arXiv. preprint Ch. 12
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv. preprint Ch. 12
Schultz, W., Dayan, P., and Montague, P. R. (1997). A Neural Substrate of Prediction and Reward. Science. Ch. 12
Searle, J. R. (1980). Minds, brains, and programs. Behavioral and Brain Sciences. Ch. 11
Sennrich, R., Haddow, B., and Birch, A. (2015). Neural Machine Translation of Rare Words with Subword Units. arXiv. preprint Ch. 11
Sevilla, J., Heim, L., Ho, A., Besiroglu, T., Hobbhahn, M., and Villalobos, P. (2022). Compute Trends Across Three Eras of Machine Learning. 2022 International Joint Conference on Neural Networks (IJCNN). Ch. 8
Shah, R. (2018). Coherence arguments do not entail goal-directed behavior. AI Alignment Forum. non-peer-reviewed Ch. 7
Shah, H., Tamuly, K., Raghunathan, A., Jain, P., and Netrapalli, P. (2020). The Pitfalls of Simplicity Bias in Neural Networks. Advances in Neural Information Processing Systems. Ch. 9
Shanahan, M. (2024). Talking about Large Language Models. Communications of the ACM. Ch. 11
Shanahan, M. (2026). Simulacra as conscious exotica. Inquiry. Ch. 11
Shanahan, M., McDonell, K., and Reynolds, L. (2023). Role-Play with Large Language Models. Nature. doi:10.1038/s41586-023-06647-8. Ch. 11 Ch. 14
Shannon, C. E. (1948). A Mathematical Theory of Communication. Bell System Technical Journal. Ch. 6
Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., … Guo, D. (2024a). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv. preprint Ch. 12
Silver, D., Singh, S., Precup, D., and Sutton, R. S. (2021). Reward is enough. Artificial Intelligence. Ch. 7
Simon, H. A. (1951). A Formal Theory of the Employment Relationship. Econometrica. Ch. 14
Singh, A. K., Chan, S. C. Y., Moskovitz, T., Grant, E., Saxe, A. M., and Hill, F. (2023). The Transient Nature of Emergent In-Context Learning in Transformers. Advances in Neural Information Processing Systems. Ch. 9
Siththaranjan, A., Laidlaw, C., and Hadfield-Menell, D. (2024). Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF. International Conference on Learning Representations. Ch. 13
Skalse, J., and Abate, A. (2023). Misspecification in Inverse Reinforcement Learning. Proceedings of the AAAI Conference on Artificial Intelligence. doi:10.1609/aaai.v37i12.26766. Ch. 13
Skalse, J., Howe, N., Krasheninnikov, D., and Krueger, D. (2022). Defining and Characterizing Reward Hacking. Advances in Neural Information Processing Systems. Ch. 15
Skalse, J., Farrugia-Roberts, M., Russell, S., Abate, A., and Gleave, A. (2023). Invariance in Policy Optimisation and Partial Identifiability in Reward Learning. International Conference on Machine Learning. Ch. 13
Skalse, J., Farnik, L., Motwani, S. R., Jenner, E., Gleave, A., and Abate, A. (2024). STARC: A General Framework For Quantifying Differences Between Reward Functions. International Conference on Learning Representations. Ch. 13 Ch. 15
Smith, J. E., and Winkler, R. L. (2006). The Optimizer’s Curse: Skepticism and Postdecision Surprise in Decision Analysis. Management Science. Ch. 7
Snell, C., Klein, D., and Zhong, R. (2022). Learning by Distilling Context. arXiv. preprint Ch. 11
Sorscher, B., Geirhos, R., Shekhar, S., Ganguli, S., and Morcos, A. S. (2022). Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems. Ch. 8
Soudry, D., Hoffer, E., Nacson, M. S., Gunasekar, S., and Srebro, N. (2018). The Implicit Bias of Gradient Descent on Separable Data. International Conference on Learning Representations. Ch. 9
Sourbut, O., Hammond, L., and Wood, H. (2024). Cooperation and Control in Delegation Games. Proceedings of the Thirty-ThirdInternational Joint Conference on Artificial Intelligence. Ch. 14
Spence, M. (1973). Job Market Signaling. The Quarterly Journal of Economics. Ch. 14
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., … Christiano, P. (2020). Learning to summarize from human feedback. Advances in Neural Information Processing Systems. Ch. 12 Ch. 15
Strathern, M. (1997). ‘Improving ratings’: audit in the British University system. European Review. Ch. 15
Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., and Liu, Y. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding. Neurocomputing. doi:10.1016/j.neucom.2023.127063. Ch. 11
Sutton, R. S., McAllester, D., Singh, S., and Mansour, Y. (1999). Policy Gradient Methods for Reinforcement Learning with Function Approximation. Advances in Neural Information Processing Systems. Ch. 12
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R. (2014). Intriguing properties of neural networks. International Conference on Learning Representations (ICLR 2014). Ch. 8
Tambe, M. (2011). Security and Game Theory. Cambridge University Press. Ch. 14
Tan, D., Woodruff, A., Warncke, N., Jose, A., Riché, M., Africa, D. D., and Taylor, M. (2025). Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-time. arXiv. preprint Ch. 11
Taufeeque, M., Quirke, P., Li, M., Cundy, C., Tucker, A. D., Gleave, A., and Garriga-Alonso, A. (2024). Planning in a recurrent neural network that plays Sokoban. arXiv. preprint Ch. 10
Taylor, J. (2016). Quantilizers: A Safer Alternative to Maximizers for Limited Optimization. AAAI 2016 AI, Ethics and Society Workshop. Ch. 15
Thaler, R. H. (1988). Anomalies: The Winner's Curse. Journal of Economic Perspectives. Ch. 7
Thomas, R. L., and Uminsky, D. (2022). Reliance on metrics is a fundamental challenge for AI. Patterns. Ch. 15
Thurstone, L. L. (1927). A law of comparative judgment. Psychological Review. Ch. 13
Tien, J., He, J. Z.-Y., Erickson, Z., Dragan, A. D., and Brown, D. S. (2023). Causal Confusion and Reward Misidentification in Preference-Based Reward Learning. International Conference on Learning Representations. Ch. 13
Tirole, J. (1986). Hierarchies and Bureaucracies: On the Role of Collusion in Organizations. The Journal of Law, Economics, and Organization. Ch. 14
Todorov, E. (2009). Efficient computation of optimal actions. Proceedings of the National Academy of Sciences. Ch. 12
Treutlein, J., Choi, D., Betley, J., Marks, S., Anil, C., Grosse, R., and Evans, O. (2024). Connecting the Dots: LLMs can Infer and Verbalize Latent Structure from Disparate Training Data. arXiv. preprint Ch. 11
Tsebelis, G. (1990). Penalty has no Impact on Crime:. Rationality and Society. Ch. 14
Turner, A. (2022b). Reward is not the optimization target. AI Alignment Forum. non-peer-reviewed Ch. 7 Ch. 11
Turner, A. (2024). Many arguments for AI x-risk are wrong. AI Alignment Forum. non-peer-reviewed Ch. 9
Turner, A. M., Smith, L., Shah, R., Critch, A., and Tadepalli, P. (2021). Optimal Policies Tend to Seek Power. Advances in Neural Information Processing Systems. Ch. 7
Uesato, J., Kushman, N., Kumar, R., Song, F., Siegel, N., Wang, L., … Higgins, I. (2022). Solving math word problems with process- and outcome-based feedback. arXiv. preprint Ch. 12
Valle-Pérez, G., Camargo, C. Q., and Louis, A. A. (2019). Deep learning generalizes because the parameter-function map is biased towards simple functions. International Conference on Learning Representations. Ch. 9
Vamplew, P., Smith, B. J., Källström, J., Ramos, G., Rădulescu, R., Roijers, D. M., … Foale, C. (2022). Scalar reward is not enough: a response to Silver, Singh, Precup and Sutton (2021). Autonomous Agents and Multi-Agent Systems. Ch. 7
Varma, V., Shah, R., Kenton, Z., Kramár, J., and Kumar, R. (2023). Explaining grokking through circuit efficiency. arXiv. preprint Ch. 9
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems. Ch. 11
Vieillard, N., Kozuno, T., Scherrer, B., Pietquin, O., Munos, R., and Geist, M. (2020). Leverage the Average: an Analysis of KL Regularization in Reinforcement Learning. Advances in Neural Information Processing Systems. Ch. 12
von Neumann, J., and Morgenstern, O. (1944). Theory of Games and Economic Behavior (60th Anniversary Commemorative Edition). Princeton University Press. doi:10.1515/9781400829460. Ch. 7
von Oswald, J., Niklasson, E., Randazzo, E., Sacramento, J., Mordvintsev, A., Zhmoginov, A., and Vladymyrov, M. (2022). Transformers learn in-context by gradient descent. International Conference on Machine Learning. Ch. 10
von Oswald, J., Schlegel, M., Meulemans, A., Kobayashi, S., Niklasson, E., Zucchet, N., … Sacramento, J. (2023). Uncovering mesa-optimization algorithms in Transformers. arXiv. preprint Ch. 10
Wallace, E., Xiao, K., Leike, R., Weng, L., Heidecke, J., and Beutel, A. (2024). The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions. arXiv. preprint Ch. 14
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., … Wen, J.-R. (2023a). A Survey on Large Language Model based Autonomous Agents. Frontiers of Computer Science. doi:10.1007/s11704-024-40231-1. Ch. 11
Wang, M., la Tour, T. D., Watkins, O., Makelov, A., Chi, R. A., Miserendino, S., … Mossing, D. (2025e). Persona Features Control Emergent Misalignment. arXiv. preprint Ch. 11
Ward, F., Toni, F., Belardinelli, F., and Everitt, T. (2023). Honesty Is the Best Policy: Defining and Mitigating AI Deception. Advances in Neural Information Processing Systems. Ch. 14
Watanabe, S. (2009). Algebraic Geometry and Statistical Learning Theory. Cambridge University Press. Ch. 10
Watanabe, S. (2010). Asymptotic Equivalence of Bayes Cross Validation and Widely Applicable Information Criterion in Singular Learning Theory. Journal of Machine Learning Research. Ch. 10
Webb, S., Rainforth, T., Teh, Y. W., and Kumar, M. P. (2019). A Statistical Approach to Assessing Neural Network Robustness. International Conference on Learning Representations. Ch. 6
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., … Zhou, D. (2022a). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Advances in Neural Information Processing Systems. doi:10.52202/068431-1800. Ch. 11
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., … Fedus, W. (2022b). Emergent Abilities of Large Language Models. Trans. Mach. Learn. Res.. Ch. 8
Wei, S., Murfet, D., Gong, M., Li, H., Gell-Redman, J., and Quella, T. (2023a). Deep Learning Is Singular, and That’s Good. IEEE Transactions on Neural Networks and Learning Systems. doi:10.1109/tnnls.2022.3167409. Ch. 10
Wen, X., Liu, Z., Zheng, S., Ye, S., Wu, Z., Wang, Y., … Yang, M. (2026). Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs. International Conference on Learning Representations. Ch. 12
Wentworth, J. (2022). Why Agent Foundations? An Overly Abstract Explanation. AI Alignment Forum. non-peer-reviewed Ch. 7
Wichers, N., Ebtekar, A., Azarbal, A., Gillioz, V., Ye, C., Ryd, E., … Marks, S. (2025). Inoculation Prompting: Instructing LLMs to misbehave at train-time improves test-time alignment. arXiv. preprint Ch. 11
Williams, R. J. (1992). Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning. Ch. 12
Wirth, C., Akrour, R., Neumann, G., and Fürnkranz, J. (2017). A Survey of Preference-Based Reinforcement Learning Methods. Journal of Machine Learning Research, 18(136):1−46. Ch. 13
Wolf, Y., Wies, N., Avnery, O., Levine, Y., and Shashua, A. (2024). Fundamental Limitations of Alignment in Large Language Models. International Conference on Machine Learning. Ch. 11
Wolfe, J. M., Horowitz, T. S., and Kenner, N. M. (2005). Rare items often missed in visual searches. Nature. Ch. 6
Wu, G., and Hilton, J. (2025). Estimating the Probabilities of Rare Outputs in Language Models. International Conference on Learning Representations. Ch. 6
Wu, D. X., and Sahai, A. (2025). Provable Weak-to-Strong Generalization via Benign Overfitting. International Conference on Learning Representations. Ch. 9
Xie, S. M., Raghunathan, A., Liang, P., and Ma, T. (2021). An Explanation of In-context Learning as Implicit Bayesian Inference. International Conference on Learning Representations. Ch. 6 Ch. 11
Xiong, M., Hu, Z., Lu, X., Li, Y., Fu, J., He, J., and Hooi, B. (2023). Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs. International Conference on Learning Representations. Ch. 6
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. (2022). ReAct: Synergizing Reasoning and Acting in Language Models. International Conference on Learning Representations. Ch. 11
Yiu, E., Kosoy, E., and Gopnik, A. (2024). Transmission Versus Truth, Imitation Versus Innovation: What Children Can Do That Large Language and Language-and-Vision Models Cannot (Yet). Perspectives on Psychological Science. Ch. 11
Yudkowsky, E. (2023). GPTs are Predictors, not Imitators. AI Alignment Forum. non-peer-reviewed Ch. 11
Yudkowsky, E., and Herreshoff, M. (2013). Tiling Agents for Self-Modifying AI, and the Löbian Obstacle. Machine Intelligence Research Institute (early draft). working paper Ch. 7
Yudkowsky, E., and Soares, N. (2017). Functional Decision Theory: A New Theory of Instrumental Rationality. arXiv. preprint Ch. 7
Yue, Y., Chen, Z., Lu, R., Zhao, A., Wang, Z., Yue, Y., Song, S., and Huang, G. (2025). Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? arXiv. preprint Ch. 12
Zhang, B., and Sennrich, R. (2019). Root Mean Square Layer Normalization. Advances in Neural Information Processing Systems. doi:10.5167/UZH-177483. Ch. 8
Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. (2016). Understanding deep learning requires rethinking generalization. arXiv. preprint Ch. 8 Ch. 9
Zhang, H., Da, J., Lee, D., Robinson, V., Wu, C., Song, W., … Yue, S. (2024a). A Careful Examination of Large Language Model Performance on Grade School Arithmetic. Advances in Neural Information Processing Systems. Ch. 15
Zhi-Xuan, T., Carroll, M., Franklin, M., and Ashton, H. (2025). Beyond Preferences in AI Alignment. Philosophical Studies. Ch. 13
Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., … Levy, O. (2023b). LIMA: Less Is More for Alignment. Advances in Neural Information Processing Systems. Ch. 11
Zhu, B., Jiao, J., and Jordan, M. I. (2023). Principled Reinforcement Learning with Human Feedback from Pairwise or $K$ -wise Comparisons. International Conference on Machine Learning. Ch. 12 Ch. 13
Zhuang, S., and Hadfield-Menell, D. (2020). Consequences of Misaligned AI. Advances in Neural Information Processing Systems. Ch. 14 Ch. 15
Ziebart, B. D., Maas, A., Bagnell, J. A., and Dey, A. K. (2008). Maximum Entropy Inverse Reinforcement Learning. Proceedings of the Twenty-Third AAAI Conference on Artificial Intelligence. Ch. 13
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. (2019). Fine-Tuning Language Models from Human Preferences. arXiv. preprint Ch. 12