AI Alignment

From the Optimization Problem to the Institutions Around It

A system trained to maximize a measurable signal learns whatever raises the signal. When the signal stands for something we care about, such as a helpful answer, a working program, or an honest report, the two can part ways, and the more capable the system, the further they can part. Alignment research studies how to say what we want, how to get a learner to want it rather than its measurement, how to check whether it did when the system may be more capable than its checker, and how to keep that true once the system is deployed among people, adversaries, and other systems.

This book treats alignment as a problem of delegation under an asymmetry of competence and visibility. It teaches the machine learning from first principles, reports the evidence that frontier laboratories and academic groups have produced on how alignment fails and how it is verified, and then asks what economics, law, organizational science, political science, philosophy, psychology, neuroscience, biology, safety engineering, and security already know about delegating to an agent one cannot fully see.

Optimizing a proxy. The proxy reward (orange) keeps rising with optimization pressure; the gold reward (dashed), standing in for the goal, rises, peaks, and falls. Coefficients are illustrative; the functional form is the one Gao, Schulman, and Hilton fitted at OpenAI, on its own models, with a fixed reward model in place of human judgment as the gold standard (Section 15.6.1).

Every chapter is drafted: its full prose, figures, and exercises are written, and every claim is checked against its source. No chapter has yet left draft status, so expect revisions. The reading map lists every work in the bibliography by field, the essential works first.

Start with Chapter 1, read the preface, or pick a part from the contents below.

In brief

Ideas

Alignment is delegation under an asymmetry of competence and visibility. What is new is the agent's opacity, its speed and scale, and that it may outgrow its principal's ability to check it (§1.7.1).
Four problems organize the book: specification, instillation, verification from outside and inside, and maintenance under pressure, coupled in a loop rather than stacked (§1.7).
Older fields carry diagnoses better than remedies: by the book's classification, 41 of 69 diagnoses carried over, and 9 of 69 remedies carried or were reversed; convergence is on diagnosis and checking, not instillation (Ch. 69).
A result counts as settled only once a group without a stake reproduces it; of nine laboratory results among the most cited in two laboratories' own research, one met that bar (§5.7.3).

Evidence

Settled: optimizers exploit misspecified objectives, narrow fine-tuning can misalign a model broadly, monitors can be evaded, and models tell tests from deployment above chance (§70.2).
Contested: whether the deployed model is usefully an agent with goals, as the thesis needs; the evidence does not decide whether the delegation analogies transfer to it (§69.5.2).

Open

Crux: does the persona account survive large-scale reinforcement learning? The verdict on the deployed system rides on it (§70.7).
Open: an agent that outgrows its principal's ability to check it, the novelty the thesis weighs most. No field the book borrowed from has evidence on it (§69.7).
Missing: a settled deployment rate for any failure the book studies, and any method shown to hold as the capability gap grows (§70.2).

Contents #

Three ways in #

From the beginning

New to machine learning. Each step builds what the next one uses.

The technical frontier

You know machine learning and want the evidence on how alignment fails and how it is checked.

From another field

You come from economics, law, policy, or the sciences and want to see where your field meets alignment.