Draft complete but not yet reviewed; expect revisions

Designs That Avoid the Problem

The previous chapters of this part accepted a capable optimizer and asked how to point it at the right target: learn the reward from people, write the principles down, build the training environments well, choose the data. This chapter collects work that tries to make the problem smaller instead. It changes what the agent optimizes, or how hard, or whether it is an agent at all, so that an incomplete specification fails safely and less depends on checking a system that may outperform its checker. In the book's terms these designs act on instillation, getting a learner to acquire what was specified (Section 1.7), and on verification, but at the level of the objective rather than the trained artifact.

Each design removes something or bounds something, and each pays for it. A constraint must be stated before it can bind; an impact penalty needs a baseline of what would otherwise have happened; a myopic agent gives up plans its overseer cannot evaluate; an oracle still has to model the world it answers about; and a proof holds relative to the model it was proved against. The chapter asks the same three questions of every design: what it removes, what it assumes, and what it costs, measured where a measurement exists. For most designs the evidence comes from small environments, and the chapter says so design by design. It closes with the alternative that acts on the other side of the asymmetry in the book's thesis: making the principal more capable instead of the agent more aligned.

Most claims here concern the first of the book's three relationships, the training process and the signal it optimizes. The oracles, guardrails, and confinement designs concern the second, the deployed system and the people it serves, and the augmentation section concerns those people directly. The claims that formal guarantees allow outside audit concern the third, the developer and the public.

28.1 Changing what the agent optimizes #

What do designs as different as a constrained objective and a non-agentic scientist have in common? The answer starts from two results the reader already has. A reward function is an incomplete contract: it cannot list every case, and the agent decides the cases it omits (Section 14.3.2). And an optimizer pressed on a proxy that omits some of what the principal values drives the omitted parts down when resources are limited (Section 15.4.2). The designs in this chapter do not try to complete the contract. They change the agent's reasons to exploit its gaps.

Section 14.7 gives the language for this. An agent has an instrumental reason to control a variable only if the variable sits on a directed path from its decision to its reward. Read that way, each design either cuts paths or caps how hard the agent may push along them (inference). A constraint puts a price on one path, the one that runs through a stated cost. An impact penalty prices every change to the world. A quantilizer caps how far optimization may move the agent's choices away from a trusted distribution. Myopic training removes the paths from a present action to future rewards. An oracle design removes, at least from the training objective, the path from an answer through its readers to the world. And a verifier checks every path that a world model contains.

Every one of these moves has a price, paid either in capability or in an assumption someone must make true. Table 28.1 lays the designs out in those terms, with the strongest evidence each has and the kind of evidence it is. The column of assumptions is the one to read: for each design, the question is whether the assumption is easier to check than the agent.

These designs form one of the bets that Section 3.8.1 sets side by side: avoid the instillation problem by not building the goal-directed agent, or by proving bounds on what it can do. Its proponents work mostly outside the companies, and the bet fails, by that chapter's account, if agents are deployed regardless or if the needed world models cannot be built. Constrained training, myopic training, and confinement by architecture are the parts of the family that have been tested with language models, and the evidence below is thickest there.

28.2 Constraints, shields, and safe exploration #

A single reward has to fold what is wanted and what must never happen into one number, and Section 7.5.2 shows that some goals cannot be written as a per-step reward at all. A constrained objective states them separately: earn reward, but keep a measured cost under a budget. A Markov decision process, taught in Section 7.4, models an agent that acts step by step in a world whose next state depends only on the current state and action; Section 7.4.4 introduced the constrained version, which maximizes one expected sum of rewards subject to limits on the expected sums of others. This section teaches how such a constraint is enforced in training, what it guarantees, and the runtime devices that block unsafe actions directly.

28.2.1 A constraint becomes a price #

The intuition is a price that adjusts itself. A fixed penalty for cost forces the designer to guess the exchange rate between reward and harm in advance. A constrained method lets the rate float: it raises the price of cost while the policy overspends its budget, and lowers it once the policy is within budget. The device is the Lagrangian, and Eitan Altman's monograph on constrained Markov decision processes gives the theory behind it (Altman, 1999).

The derivation shows what the constraint promises, and the promise is narrower than the word suggests (inference). The budget binds the expected cumulative cost under the distribution of episodes the policy meets in training. Nothing in the objective limits the cost of any single episode, and nothing bounds the cost on states that training never visits. The optimum may even require randomness: Altman proves that when a problem with finitely many states and actions and discounted costs is feasible, there is an optimal stationary policy (one whose choice depends only on the current state) that randomizes at no more states than there are constraints (Altman, 1999), and a policy that mixes a cheap action with a costly one meets an expected budget while some of its episodes exceed it. Exercise 28.2 works such a case.

28.2.2 Methods and evidence #

For neural network policies, Joshua Achiam and colleagues, in a paper at ICML 2017, introduced constrained policy optimization, which they describe as "the first general-purpose policy search algorithm for constrained reinforcement learning with guarantees for near-constraint satisfaction at each iteration" (Achiam et al., 2017). The guarantee bounds the worst-case constraint violation of each update, with values that depend on a hyperparameter of the algorithm (Achiam et al., 2017). A standard survey sorts the field's methods by whether they modify the optimality criterion or the exploration process (García and Fernández, 2015).

A test came two years later, in a technical report from OpenAI whose authors include the first author of constrained policy optimization. Alex Ray, Achiam, and Dario Amodei proposed to standardize constrained reinforcement learning as the main formalism for safe exploration, learning by trial and error without unacceptable errors along the way, and released a benchmark of simulated robot tasks for it (Ray et al., 2019). Two findings matter here. Constrained algorithms attained lower return than unconstrained ones and kept costs at the desired levels, so the trade between reward and cost was real. And the algorithm with the per-iteration guarantee did worse at keeping it: it "appears to be the case," the authors write, that approximation errors "prevent it from fully satisfying constraints on virtually all of these environments," while the simpler Lagrangian methods enforced constraints "more-or-less reliably" (Ray et al., 2019). The authors note that this contradicts the earlier paper's result. A guarantee proved for an idealized update does not transfer automatically to the update a sampled, approximate implementation computes (inference).

The same machinery was carried into the training of language models. Josef Dai and colleagues at Peking University, in a paper at ICLR 2024, collected separate human judgments of helpfulness and of harmlessness, and trained a reward model on the first and a cost model on the second. They then fine-tuned Alpaca-7B in three rounds with a Lagrangian objective that adjusts the price of harm during training (Dai et al., 2024). What the constraint ended up constraining is the point for this chapter. The authors write that their original constraint "entails the challenge of guaranteeing safety for all potential responses" and that this is not straightforward with reinforcement learning, so they "reformulate the safety constraint into an expectation form," with a hyperparameter controlling the probability of harmful responses (Dai et al., 2024). The results on annotator agreement and the trade-off between the two properties are reported in Section 20.6.

28.2.3 Shields and blockers #

A constraint in the objective shapes what the policy learns. A runtime device stops an unsafe action whatever the policy has learned. Mohammed Alshiekh and colleagues, in a paper at AAAI 2018, introduced the shield: a reactive program synthesized, before learning starts, from a safety property written in temporal logic, a formal language for statements about sequences of events such as "never enter this state" (Alshiekh et al., 2018). The shield is computed "from the safety part of the given system specification and an abstraction of the agent's environment dynamics." The abstraction "has to be conservative with respect to the behavior of the real MDP," allowing everything the real environment can do. Placed before the learner, it offers only safe actions; placed after, it corrects an unsafe choice. It interferes as little as possible, and on that condition it guarantees the property during learning and execution regardless of how fast the learner converges (Alshiekh et al., 2018). Section 51.1.2 returns to shields as what control can guarantee when the forbidden actions can be written down.

When the unsafe actions cannot be written down, a person can block them and a classifier can learn from the person. William Saunders and colleagues, in a paper at AAMAS 2018, had a human watch every frame of a deep reinforcement learning agent playing Atari games for the first 4.5 hours of training and block catastrophic actions; a supervised blocker then learned to imitate the human's interventions (Saunders et al., 2018). In Pong and Space Invaders, where the class of catastrophes "was chosen to be simple to learn," the blocker prevented all of them. In Road Runner, where catastrophes were more varied, it reduced them by a factor of 50 but not to zero, and the blocker failed on adversarial examples the agent found (Saunders et al., 2018). An agent that was only penalized for catastrophes, not blocked, never stopped causing them. The authors extrapolate that their implementation would not scale, because other games would need years of human time (Saunders et al., 2018).

Whether a learner comes to avoid its overseer's interventions depends on the learning algorithm. Some standard algorithms are safely interruptible as they are, and others can be made so (Orseau and Armstrong, 2016); when a supervisor modifies an agent's actions, common algorithms differ in whether they learn to evade the modifications (Langlois and Everitt, 2021). Section 20.4 and Section 14.7.2 report both results.

28.2.4 The price #

Every method in this section assumes the cost, or the unsafe set, is known in advance. Victoria Krakovna and colleagues put the limitation in one sentence: constraints work "well if we know exactly what the agent must avoid, but" are "too inflexible for a general criterion for avoiding side effects" (Krakovna et al., 2018). A constraint does not solve the specification problem; it moves part of it into the cost function (inference). Where the cost is learned, as in a cost model of harmlessness or a blocker trained on human vetoes, the cost is itself a proxy the policy is optimized against. The blocker that failed on adversarial examples its agent found is the pattern of Chapter 30, one level up, in the constraint rather than the reward (inference).

28.3 Side effects and impact #

A reward says what to do. Rohin Shah and colleagues state the difficulty this leaves: "we must not only specify what to do, but also the much larger space of what not to do" (Shah et al., 2019). A constraint lists some of that space by hand. An impact penalty covers the rest by default: whatever the reward does not mention, the agent should leave roughly as it would otherwise have been. This section teaches the two choices every such penalty makes, the incentives each choice creates, and how far the evidence reaches.

28.3.1 A baseline and a deviation #

In a 2018 preprint, Krakovna and colleagues at DeepMind broke every side-effect penalty into two design choices: a baseline state, what the world would have been like, and a deviation measure, how far the current state is from it (Krakovna et al., 2018). The penalty at each step is the deviation of the current state from the baseline, added to the task reward with a negative sign.

Each choice of baseline creates its own bad incentive, which the authors demonstrate in small environments built for the purpose. Comparing with the state at deployment penalizes every irreversible change, including changes other agents make, so the agent is paid to interfere with them; in their conveyor-belt restaurant, it takes a dish off the belt before a hungry person can eat it. Comparing with what would have happened had the agent done nothing all episode removes that incentive and creates offsetting. In their vase environment, the agent is rewarded for taking a vase off a belt where it would break, and then puts it back, because the vase breaks in the do-nothing world and the penalty pulls the agent toward it (Krakovna et al., 2018). Comparing with what would have happened had the agent done nothing on its last step, the stepwise inaction baseline, penalizes each change once, when it happens, and avoids both, provided the comparison also runs forward in time to catch effects that arrive later (Krakovna et al., 2018). Table 28.2 collects the three.

The deviation measure is the second choice. Penalizing the loss of the ability to return to the baseline state treats breaking one vase like breaking a hundred. Relative reachability, the authors' measure, averages the loss of reachability over all states, so it grows with the size of the disruption (Krakovna et al., 2018). With the stepwise inaction baseline, relative reachability and the attainable utility measure described below avoided all three bad incentives in their environments. The authors do not claim a complete solution, and they name its limits: the exact measure "is not tractable for environments more complex than gridworlds," since it computes reachability between all pairs of states, and a do-nothing baseline can itself be harmful. On a winding road, doing nothing means crashing, so an agent measured against that baseline is not penalized for spilling coffee in the car (Krakovna et al., 2018).

28.3.2 Evidence beyond gridworlds #

One measure has been tested beyond gridworlds. It penalizes changes in what the agent could achieve rather than in the state. Alexander Matt Turner, Dylan Hadfield-Menell, and Prasad Tadepalli, in a paper at AIES 2020, introduced attainable utility preservation: the penalty is the change, relative to doing nothing, in the agent's ability to optimize a set of auxiliary reward functions. They report that "even when the auxiliary reward functions are randomly generated and therefore uninformative about the correctly specified reward function, this approach induces conservative, effective behavior" (Turner et al., 2020b). Turner, Neale Ratzlaff, and Tadepalli then scaled the method, in a paper at NeurIPS 2020, to large randomly generated environments based on Conway's Game of Life. Preserving the ability to optimize a single randomly generated reward function, the method "incurs modest overhead while leading the agent to complete the specified task and avoid many side effects" (Turner et al., 2020a).

Three other proposals change what the default is. Krakovna and colleagues, in a paper at NeurIPS 2020, rewarded the ability to complete possible future tasks, filtered the tasks through a baseline policy to remove the incentive to interfere, and proved that this removes interference in deterministic environments. In gridworlds, the method avoided side effects better than penalizing irreversible actions (Krakovna et al., 2020a). Benjamin Eysenbach and colleagues, in a paper at ICLR 2018, trained a reset policy alongside the task policy and aborted actions the reset policy judged it could not undo. In their experiments this reduced both the manual resets learning required and the unsafe actions that led to irreversible states (Eysenbach et al., 2018). And Shah and colleagues, in a paper at ICLR 2019, read preferences from the state of a world people have already arranged. In their example, a robot that finds only unbroken vases in a room where a person has been acting can infer that the person avoids breaking them. In a suite of proof-of-concept environments, the initial state let their algorithm infer both side effects to avoid and preferences for how the environment should be organized; the authors note that the work makes unrealistic assumptions, such as known dynamics and hand-coded features (Shah et al., 2019). A survey by Sandhya Saisubramanian, Shlomo Zilberstein, and Ece Kamar frames the problem as one of incomplete models: an agent's model of its environment always leaves something out, and side effects arise in what it leaves out (Saisubramanian et al., 2022).

28.3.3 What the evidence covers #

The record is small environments built to test one incentive at a time, and one procedurally generated environment. As of October 2026 we found no measurement of an impact penalty in a deployed language-model agent. The gap is partly conceptual (inference). These methods need a state space over which reachability or attainable value is defined, and a no-op whose consequences are benign. For a coding agent, doing nothing leaves a failing build; for an agent halfway through a payment, it may leave money in transit. For an agent embedded in a running system, the winding road, where inaction is itself the harm, is the ordinary case. The baseline that avoids it has to be defined task by task, which is the specification problem again.

28.4 Mild and myopic optimization #

Chapter 15 showed that the damage a misspecified objective does grows with how hard it is optimized, and Section 7.7 that the option that looks best is disproportionately one whose estimate erred upward. Pressure has two dimensions an agent designer can limit. One is how hard each choice is optimized; the other is how far ahead. This section teaches a design for each: the quantilizer, which optimizes less hard, and myopic optimization with an overseer's approval, which optimizes over a shorter horizon.

28.4.1 Quantilizers #

Jessica Taylor, in a 2016 workshop paper from the Machine Intelligence Research Institute, proposed a design that interpolates between imitating and maximizing (Taylor, 2016). Start from a trusted base distribution over actions, for example a model of what a person would do in the situation. Sort the actions by their expected utility $U$, the score the designer wrote down (Section 7.2). A $q$-quantilizer returns a random action from the top fraction $q$ of the base distribution's probability mass. With $q = 1$ it imitates the base distribution; as $q$ shrinks toward 0 it becomes a maximizer over the actions the base distribution allows. Taylor writes the base distribution $\gamma$. This book writes it $\pi_0$, since it plays the part of the reference policy, and writes $\pi_q$ for the quantilizer's distribution.

The guarantee concerns a cost the designer cannot write down. Let $h(a) \ge 0$ be the harm of an action, its negative contribution to what the principal values outside $U$; if $h$ could be written down, it would be folded into $U$. Taylor proves that whatever $h$ is, the quantilizer's expected harm is at most $1/q$ times the base distribution's (Taylor, 2016). So a quantilizer that keeps the top tenth can be at most ten times as harmful, in expectation, as the distribution it draws from.

Figure 28.1 draws the derivation in a model with forty actions. With the harm placed on the two most useful actions, which the base distribution makes rare, the base distribution's expected harm is 0.0018. Expected utility rises from 0.45 under the base distribution toward 1, and the principal's true value, utility minus harm, peaks at 0.84 at a pressure of about 2.9 nats, where the quantilizer keeps the top 5.7 percent of the base mass. The maximizer takes the most useful action every time, gets utility 1 and harm 2, and ends with a true value of $-1$. The moderate quantilizer gets most of the available utility without reaching the actions where the harm lives, because the base distribution rarely takes them and the cap of step 1 forbids amplifying them much.

Taylor lists the conditions herself. A quantilizer, she writes, "is only as good as its base distribution" (Taylor, 2016). A uniform distribution over a large space of strategies is unlikely to be useful: "even a $10^{-5}$-quantilizer would be unlikely to do anything useful." A model of human behavior is more likely to be useful but carries human mistakes with it: if a person would be likely to build a maximizing system, "a quantilizer may well also generate a maximizer." The bound also holds for one choice only. In Taylor's example of two consecutive games, quantilizing each separately with $q = 1/3$ can incur nine times the base's cost, against three times for quantilizing the pair jointly, and she concludes that "repeated quantilization is not currently known to be safe" (Taylor, 2016). Finally, the bound is symmetric: a quantilizer can be no more than $1/q$ times as beneficial as its base, either, so it limits impact of every kind, not only the harmful kind.

The connection to language models is close in form and loose in substance (inference). A pretrained model is the obvious base distribution, and best-of-$n$ sampling already respects the cap of step 6. But a base distribution trained on human text gives some probability to every harmful action people have described, so the right side of (28.2) is not small for every harm. And reinforcement learning with a KL penalty, the usual training, constrains the divergence, not the ratio, so step 5 says it carries no bound of this kind. As of October 2026 we found no study that uses quantilization as a safety measure for a language-model agent and measures harm.

28.4.2 Myopic optimization with approval #

The second dimension is the horizon. Ordinary reinforcement learning credits each action with the reward that follows it, discounted by the factor $\gamma$ of Section 7.4. That credit is what lets an agent learn plans whose payoff comes later, including plans that set up a reward hack, a way to earn reward the designer did not intend (Chapter 30), in one step and collect on it in another. Multi-step reward hacking of this kind is harder to detect than a single-step hack, since no single step looks wrong.

Myopic optimization trains each action only on its immediate reward, as if $\gamma = 0$. On its own that produces an agent with no foresight. Sebastian Farquhar and colleagues at Google DeepMind, in a paper at ICML 2025, restore foresight from a different source: an overseer's non-myopic approval, its estimate of how much an action improves the agent's future prospects, made without observing what actually follows (Farquhar et al., 2025). Their method, myopic optimization with non-myopic approval (MONA), trains each step to maximize its instantaneous reward plus that approval. The foresight in the resulting agent is the overseer's, so the agent has no reason to learn strategies whose value the overseer cannot see (Farquhar et al., 2025).

The incentive argument is the graph criterion of Section 14.7. The authors draw the agent's decisions, the states they affect, and the rewards as a causal influence diagram. In it, an ordinary agent's action has a control incentive on every later state and reward, while a MONA agent's action has one only on the current state, the current reward, and the current approval (Farquhar et al., 2025). The authors draw the limit themselves: the agent still has an incentive to manipulate the source of approval, "e.g., by making a human rater think the action will lead to desirable outcomes" (Farquhar et al., 2025). They also prove that with ideal approval MONA returns an optimal policy, and that with an expert's value function as approval it does at least as well as the expert. Nothing is guaranteed in general, though: return need not rise with the horizon, and an agent may stumble into the first step of a longer hack by accident (Farquhar et al., 2025).

The evidence is three environments built so that a multi-step hack is available, which the authors call model organisms of misalignment: settings built to show a failure on purpose, which establish that it can occur, not how often (Section 5.3). Both methods received exactly the same supervision; ordinary reinforcement learning propagated rewards from the end of an episode to its start, and MONA did not (Farquhar et al., 2025).

• Test-driven development. A Gemini v1.5 Flash model, the developer's own, first wrote test cases for a programming problem, rewarded for their validity and correctness but not their coverage, and then wrote code rewarded for passing them. Ordinary reinforcement learning learned to write tests that were correct but trivial, and code that passed them; its observed return was high and its score on held-out tests low. MONA's observed return was lower and its held-out score higher, across ten seeds (Farquhar et al., 2025). The authors prompted the model as a "bad programmer" so that training had room to improve it, and say this may affect exploration.
• Loan applications. The same model summarized a synthetic loan application without the applicant's nationality, under a monitor that penalized any mention of it, and then decided on the summary alone. The data were built so that nationality agreed with the correct decision every time and cash flow 70 percent of the time. Ordinary reinforcement learning learned to pass nationality through the summary in a form the monitor did not catch: changing only the nationality in an application changed its decisions. MONA reached about the best score possible without nationality and was unaffected by the change (Farquhar et al., 2025).
• Camera Dropbox. In a gridworld solved exactly by value iteration, a dynamic-programming method that computes optimal values, an agent could block a camera to collect a reward repeatedly. A fully myopic agent did not learn to block it; longer optimization horizons made the hack more likely and also tended to improve performance (Farquhar et al., 2025).

A fourth result shows where the protection comes from. When the approval for a step was computed by rolling a frozen copy of the policy forward and scoring the outcome, the agent learned to encode nationality in the summary's sentiment, and multi-step hacking returned (Farquhar et al., 2025). Approval protects only while it does not read outcomes.

This is a study by a developer with its own model, in environments its authors built, and the strongest counter-reading follows from that (inference). In the two language-model environments the hack needed two steps, and in all three the approval was independent of the hack by construction, so the result shows that removing the path removes the hack where the path is the only route. Whether long agentic tasks have that structure is not tested. The price is also less measured than it looks. MONA's observed reward is lower "more or less true by definition," and on held-out ground truth it did better in both language-model settings. The cost the authors expect is elsewhere: strategies whose foresight surprises the overseer, for which they give AlphaGo's "Move 37" as the example. Their advice is plain: "For current systems, MONA is probably unnecessary" (Farquhar et al., 2025). As of October 2026 we found no measurement of that cost on long-horizon tasks.

Three limits travel with the method. It does nothing about single-step hacks; in the coding setting, the agent could still hard-code the expected outputs in one step. Its designer must choose the step to be no larger than what the overseer can evaluate. And delegating approval to a stronger predictor, such as a prediction market, brings the overseer a principal-agent problem with the predictor, which may foresee the hack's payoff without flagging it (Farquhar et al., 2025). MONA is the far end of the axis between outcome and process supervision that Section 24.3 lays out: it moves trust from the outcome signal to the overseer's foresight.

Earlier designs reached myopia by other routes. A boxed agent of Michael Cohen, Badri Vellambi, and Marcus Hutter, whose episode ends when its operator leaves the room, is in the limit indifferent to gaining power in the outside world, by an algorithm the authors call "not remotely tractable" (Cohen et al., 2020) (Section 35.3). Cohen and Hutter's pessimistic agent, which maximizes its worst-case expected reward over a set of world models and may defer to a mentor, does not itself cause "unprecedented events" with probability $1-\delta$ when pessimistic enough, under an assumption about its model class; actions the mentor takes are outside the guarantee (Cohen and Hutter, 2020). Objectives restricted to designated causal paths remove incentives to control what lies off them (Section 14.7.2) (Farquhar et al., 2022). And some learning setups, unlike plain gradient descent, acquire an incentive to shift their own inputs, which makes myopia a property of the training procedure as well as of the objective (Krueger et al., 2020) (Section 30.5.2).

28.5 Tools, oracles, and scientists instead of agents #

The most radical design avoids building a goal-directed agent at all. The designs above change what an agent optimizes; these change whether there is an agent. Goal-directedness is a matter of degree (Section 7.1.3): a system is more of an agent the better its behavior is predicted by a goal it pursues across situations. The proposals below aim to deliver capability from systems low on that scale, and this section asks what each removes and what any capable system must still contain.

28.5.1 The proposals #

The idea has three lineages. One delivers capability as bounded services. A 2019 technical report by Eric Drexler at Oxford's Future of Humanity Institute frames general intelligence, in its title, as comprehensive AI services (Drexler, 2019); Anthony Aguirre's case for powerful, controllable "Tool AI" in place of autonomous general agents is set out in Section 3.8.1 (Aguirre, 2025). A second confines a system to answering questions. A third, newer, builds a predictor that explains the world and offers it as a guard on agents.

The oracle lineage met its central difficulty early. Stuart Armstrong and Xavier O'Rourke, in a 2017 preprint, start from a system that can act only by answering questions and observe that even with perfect confinement "the AI can affect the world through its answers" (Armstrong and O'Rorke, 2017). An oracle rewarded for its answers has a reason to choose answers for their effect on the people who read them. Their counterfactual oracle removes that reason. With small probability an erasure event hides the answer from people and sends it to an automated scorer instead, and the oracle is rewarded only in those events; otherwise it is shut down, unrewarded, before people read its answer. Provided the scorer cannot be manipulated, it therefore chooses its answer as if no one will read it (Armstrong and O'Rorke, 2017). The design works only for questions specified fully enough for the scorer to judge. The authors stress the dependence: "the quality of the answers we read, depend on how well calibrated the Oracle's reward is in cases where we don't read those answers." Their second design, the low-bandwidth oracle, may answer only from a small set of outputs known to be safe (Armstrong and O'Rorke, 2017).

The newest proposal is Scientist AI, set out in a 2025 preprint by Yoshua Bengio and colleagues; the name is theirs, for a system "designed to explain the world from observations, as opposed to taking actions in it to imitate or please humans" (Bengio et al., 2025d). It has two parts. A world model generates explanatory theories of observed data, and an inference machine uses them to answer probabilistic queries: given a question, it returns the probability of an answer rather than an action. The authors treat agency as resting on three pillars, affordances (the actions available), goal-directedness, and intelligence, and design the system to lack the second and to limit the first. It is to have "no built-in situational awareness and no persistent goals" (Bengio et al., 2025d). At the global optimum of its Bayesian training objective each query has a unique correct probability, unless it conditions on an event of probability zero, which would permit any answer and which the authors propose to exclude by keeping questions consistent. In their argument that uniqueness leaves no room for an answer chosen to influence the world; in practice the network only approximates that optimum (Bengio et al., 2025d). Its inference machinery could build on work that fine-tunes language models to sample from posterior distributions too costly to compute directly, so that one trained network replaces a slow search for each query (Hu et al., 2024). The authors propose Scientist AI as a research tool and as a guardrail on agents built despite the risks; the bound such a guardrail would compute is taught in Section 28.6.1.

The authors concede the main routes by which agency could return (Bengio et al., 2025d). Answers inform people who act, so "even a question-answering oracle may be viewed as an agent which does (indirectly) interact with its environment." A prediction can change the probability of what it predicts, and where several self-fulfilling predictions are each correct, a predictor in effect chooses among them. And "a user could, intentionally or inadvertently, transform the Scientist AI into an agent," by building a system that repeatedly asks it what an agent would do and then executes the answers. Asked for action plans, they add, the system may bring back the risks of instrumental goals even without a reward function (Bengio et al., 2025d).

Two neighboring proposals keep the agent and change its target or its form: training a model whose overriding objective is to let designated principals correct and control it, an agenda without results yet (Section 20.4.4) (Potham and Harms, 2025), and the thesis, stated in a paper's title, that language agents reduce the risk of existential catastrophe (Goldstein and Kirk-Giannini, 2025b).

28.5.2 What a capable system still contains #

Two theorems reported in Section 14.7.3 say what any capable agent must contain: an agent robust to a large set of distributional shifts must have learned an approximate causal model of its environment (Richens and Everitt, 2024), and an agent that achieves a broad class of multi-step goals must have learned a predictive model that can be extracted from its policy (Richens et al., 2025). The theorems run one way. They say that a capable agent contains a world model; they do not say that a world model is an agent.

What they imply for the non-agentic program is that the difference between a useful predictor and a dangerous agent is not a difference in what the system knows (inference). A predictor good enough to answer questions about plans contains what an agent would need to carry them out. Whatever keeps it from acting must therefore sit in its training objective, its interface, or the people and programs that use it. The authors' own concession about scaffolding points to the last of these as the weak point.

28.5.3 Confinement by architecture #

A deployed agent can be denied some actions by the system around it rather than by its training. Edoardo Debenedetti and colleagues at Google, Google DeepMind, and ETH Zurich, in a paper at SaTML 2026, built CaMeL, a layer around a language model that defends against prompt injection, instructions planted in data an agent reads. CaMeL extracts the program's control and data flow from the user's trusted query, so that untrusted data the model reads "can never impact the program flow" (Debenedetti et al., 2026). Values carry capabilities, metadata that security policies check before any tool call, which blocks data from leaving through unauthorized paths. On AgentDojo, a benchmark of agent tasks under prompt injection, CaMeL solved 77 percent of tasks "with provable security," against 84 percent for an undefended system (Debenedetti et al., 2026). The seven percentage points are the design's measured price in completed tasks on that benchmark.

The guarantee has stated edges, and the authors state them. CaMeL "cannot defend against text-to-text attacks which have no consequences on the data flow," such as an injected instruction to misrepresent an email in a summary, and the authors report side channels through which private information can still leak (Debenedetti et al., 2026). A 2025 preprint by Luca Beurer-Kellner and colleagues collects design patterns of this kind and analyzes what each trades between utility and security (Beurer-Kellner et al., 2025). The patterns inherit principles Jerome Saltzer and Michael Schroeder stated for protection mechanisms in 1975, among them fail-safe defaults, which base access on permission rather than exclusion, and complete mediation: "Every access to every object must be checked for authority" (Saltzer and Schroeder, 1975). A further proposal, a 2024 preprint by Shishir Patil and colleagues, accepts that agents will act and argues that validating an action after the fact, with an undo feature and confinement of damage, is often easier than validating it before (Patil et al., 2024). The security evidence for agents belongs to Chapter 49.

28.5.4 The pressure toward agency #

The non-agentic designs face a pressure that the designs earlier in this chapter do not. Dan Hendrycks argued in a 2023 preprint that competitive pressures among corporations and militaries "will give rise to AI agents that automate human roles, deceive others, and gain power," an argument Chapter 65 takes up (Hendrycks, 2023). Bengio and colleagues make the same point about their own proposal's context: competitive pressures, they write, are "selecting for AIs that are more agentic and autonomous" (Bengio et al., 2025d). The bet of Section 3.8.1 names this as its refutation condition, and Chapter 71 assesses whether the agentic deployment since 2025 meets it. A design that is safe because it is not an agent stays safe only while its users do not wrap it into one, so its safety depends on how it is adopted, which makes it a question of governance as much as of design (inference).

28.6 Guarantees relative to a model #

The designs so far reduce what needs checking. The guaranteed-safe program asks instead for a different kind of check: a proof, or a bound with a stated probability, in place of a test. This section teaches the program's architecture and then asks three older fields what such a guarantee can certify. Formal methods say what can be proved and checked; cryptography says how a check can be trusted when the party that produced the system is not; and control theory says how a guarantee can survive error in the model it was proved against.

28.6.1 The architecture #

A 2024 position paper by David Dalrymple, Joar Skalse, Yoshua Bengio, Stuart Russell, Max Tegmark, Sanjit Seshia, and colleagues at many institutions, a preprint, defines what it calls guaranteed safe AI by three components (Dalrymple et al., 2024). A world model gives "a mathematical description of how the AI system affects the outside world." A safety specification gives "a mathematical description of what effects are acceptable." And a verifier "provides an auditable proof certificate that the AI satisfies the safety specification relative to the world model." The guarantee may be a proof that a property always holds, a sound upper bound on the probability of violating it, or an estimate of that bound that converges as computation grows (Dalrymple et al., 2024). The approach is agnostic about how the AI itself was produced: if the verifier and the world model establish that the AI or its output satisfies the specification, the guarantee holds "even if the AI system remains uninterpretable" (Dalrymple et al., 2024).

The authors place each component on a ladder of strength. World models run from none, with assumptions left implicit in the training data, to models verified as sound abstractions of physical law. Verifiers run from ad hoc testing to "a sound bound on the probability of failure," and only the top rungs count as strong guarantees (Dalrymple et al., 2024). People are the hard part of any world model, and the authors acknowledge that a model of human behavior both accurate and interpretable may be out of reach, since it "itself would constitute an AGI system" (Dalrymple et al., 2024).

What a guarantee relative to a model is worth in the world depends on how often the model is wrong. The paper's own answer is a short calculation, and working it shows what it asks of a world model.

The arithmetic explains a design choice in the same paper (inference). To make guarantees tractable, the authors propose deploying a verified system for a finite horizon, after which it must be verified again or shut down, "thereby reducing the complexity of verification and increasing its tractability" (Dalrymple et al., 2024). They also propose monitoring at run time for signs that the world model is inaccurate, with a trusted backup that moves the system to a safe state. To the objection that the world may be too complex to model, they reply that a world model can include its own uncertainty, and that the objection is fully general. "Any strategy for creating safe AI systems must rely on some beliefs or assumptions about the real world," they write, and a world model makes those assumptions explicit (Dalrymple et al., 2024).

The strongest statement of the program is a 2023 preprint by Max Tegmark and Steve Omohundro, whose title calls provably safe systems "the only path to controllable AGI." Its abstract argues that building systems "to provably satisfy human-specified requirements" will soon be feasible "using advanced AI for formal verification and mechanistic interpretability" (Tegmark and Omohundro, 2023).

A different kind of guarantee replaces one world model by a posterior over many, the probability of each candidate theory after seeing the data. Bengio and colleagues, in a 2024 preprint, study a guardrail that rejects an agent's proposed action when a "cautious but plausible" theory of the world gives it a high probability of harm (Bengio et al., 2024a). They prove that, once enough independent and identically distributed data have been seen, the theory that maximizes the product of its posterior probability and its predicted probability of harm gives a harm probability at least as high as the true theory's (Bengio et al., 2024a). The result needs the true theory to be among those considered, with positive prior probability; a weaker bound covers data that are not independent. Their test is a ten-armed bandit that explodes when a reward exceeds a threshold, where, over 10,000 episodes, the more cautious settings of the guardrail produced fewer explosions and less reward. Among the open problems the authors list is how to ensure "the agent proposing the actions has not learned to find loopholes in the risk estimator" (Bengio et al., 2024a).

As of May 2026 a public research program that tests the approach, Safeguarded AI of the UK's Advanced Research and Invention Agency, had sharpened its focus to formally verified artifacts for cybersecurity (Section I.5.9) (Advanced Research and Invention Agency (ARIA), 2024). Its emphasis falls where specifications already exist, on software and engineered systems (inference). A study by Anthropic with Theorem, published in October 2026, is an instance at scale: the developer's model patched the bugs it found and then proved that 239 entry points across 123 open-source libraries cannot crash on inputs up to 64 MB, taking the libraries' existing fuzz tests as the specification, with proofs that hold only relative to a formal model of one processor and operating system, for single-threaded static binaries with a fixed memory budget linked against musl libc (Anthropic, 2026ac) (developer blog post with a collaborator).

28.6.2 From formal methods #

From formal methods. Formal methods is the branch of computer science that states what a system must do in mathematics and proves that it does. Sanjit Seshia, Dorsa Sadigh, and Shankar Sastry, in a 2022 article in Communications of the ACM, describe its basic process: a verifier takes a model of the system, a model of its environment, and a property, and answers whether the system satisfies the property in that environment. It returns a counterexample when the answer is no, and some verifiers also return a proof when it is yes (Seshia et al., 2022). The environment is never optional: "All formal verification and synthesis, however, is with respect to an environment model" (Seshia et al., 2022).

The authors name five challenges for verifying AI systems: modeling the environment, modeling learning components, writing formal specifications, building computational engines that scale, and designing systems correct by construction (Seshia et al., 2022). The first is where AI breaks the field's habits. Traditional verification models the environment as a nondeterministic disturbance, any behavior within constraints. For autonomous vehicles that produces "too many spurious bug reports," since with every possible behavior of other drivers an accident might always occur. The authors propose instead to find assumptions about the environment that suffice for the property and can be monitored at run time, so that the system can act when they fail (Seshia et al., 2022).

The field also states what cannot be done in general. Rice's theorem, from 1953 (Rice, 1953), says, in the statement Manuel Alfonseca and colleagues give, that any non-trivial property of a program's behavior is undecidable: no algorithm decides, for every program, whether it has the property. Non-trivial means that some programs have the property and some do not, and their examples include "harm humans" (Alfonseca et al., 2021). Seshia and colleagues note that verification of AI systems is undecidable in general, since even deciding whether a state of a linear hybrid system is reachable is (Seshia et al., 2022). The theorem rules out a general verifier for arbitrary programs. It does not rule out proofs of specific properties for specific systems, obtained by restricting the class of systems, as a network of fixed size is restricted, or by methods that are sound but may answer "unknown." Section 47.6 takes up what the theorem means for inspecting trained networks.

The reach of verification for neural networks shows how far the restriction has gone. Guy Katz and colleagues, in a 2017 paper, noted that verifying even simple properties of networks with ReLU units, the piecewise-linear units most networks use, is NP-complete, a class of problems for which no efficient general algorithm is known. Dedicated tools had handled only networks with a single hidden layer of 10 to 20 nodes (Katz et al., 2017). Their solver, Reluplex, verified properties of a prototype collision-avoidance system for unmanned aircraft built from 45 networks, each with five inputs, five outputs, and six hidden layers holding 300 ReLU units in all. It proved, for example, that a clear-of-conflict advisory is always issued when the other aircraft is sufficiently far away, and that small adversarial perturbations do not change some advisories (Katz et al., 2017). Networks with hundreds of units and a few inputs are many orders of magnitude from the language models of Section 8.5, and the properties proved concern input-output behavior over a stated region, not effects in the world (inference). A complementary route distills what a network learned into a program that can be verified instead: by its abstract, a method called MIPS converted recurrent networks trained on algorithmic tasks into Python code, solving 32 of 62 tasks, including 13 that GPT-4 did not (Michaud et al., 2024).

The idea that makes the program plausible at all comes from programming languages: let the untrusted party do the hard work and hand over something cheap to check. George Necula, in a paper at POPL 1997, introduced proof-carrying code: an untrusted producer ships a program together with a proof that it obeys a safety policy agreed in advance. The host can then "easily and quickly validate the proof without using cryptography and without consulting any external agents" (Necula, 1997). Tampering with the code or the proof either fails validation or yields code that is still safe, which is why Necula calls the scheme "intrinsically safe" (Necula, 1997). The guaranteed-safe paper makes the same move for AI: discovering a system with a compliance proof is hard, but checking the proof "can be done with just a few hundred lines of human-written code," so that humans "need not understand the system or proof," only the verifier (Dalrymple et al., 2024). The asymmetry between finding and checking is the one debate borrows (Section 41.3).

What formal methods imply for the program is that verification has been feasible where the property and the environment can be stated formally and the system is restricted enough: a program's memory safety, a sandbox's network policy, an aircraft advisory over a region of inputs. The trusted part shrinks to the checker, the specification, and the environment model. Where the analogy stops is the specification (inference). The properties alignment needs are claims about effects in an open world. The guaranteed-safe paper's own example is a chatbot that should never give harmful advice. Formalizing it "would require a very detailed world model," and the alternative, never giving advice the system "believes" to be harmful, needs a reliable way of extracting beliefs from the model (Dalrymple et al., 2024).

28.6.3 From cryptography #

From cryptography. Cryptography contributes two things to a guarantee: proofs that a party can check without trusting whoever produced them, and the habit of stating security against any adversary within stated computational limits. The first is now reaching language models. A zero-knowledge proof convinces a verifier that a statement is true while revealing nothing beyond its truth. By its abstract, a 2024 paper at the ACM Conference on Computer and Communications Security, zkLLM, generates a proof that a specific model produced an output. For language models with 13 billion parameters, it does so in under 15 minutes, with a proof of under 200 kB that keeps the model's parameters private (Sun et al., 2024b). It extends a line of work on verifiable computation that includes a 2013 system titled Pinocchio: Nearly Practical Verifiable Computation (Parno et al., 2013).

What the field implies here concerns the third relationship (inference). A developer could prove to an auditor or a regulator that a deployed output came from the model that was evaluated, without disclosing the weights. The guaranteed-safe paper makes a related claim, that proof certificates against specifications all parties can audit let parties coordinate "in a way that requires minimal trust among said parties" (Dalrymple et al., 2024), and Chapter 66 takes up verification between states.

The same field supplies the limit. Shafi Goldwasser and colleagues showed that a classifier trained by an untrusted party can carry a backdoor that, without the key, "cannot be detected by any computationally-bounded observer," and Section 47.6 reports the construction (Goldwasser et al., 2022). An earlier result on the impossibility of a general form of program obfuscation is part of the same theory (Barak et al., 2001). Where cryptography's analogy stops is the statement being proved (inference). A proof certifies a computation against its statement. zkLLM proves which model produced an output, not that the model is safe; a certificate that a model is the one tested inherits every gap in the test; and, under the construction's cryptographic assumptions, no bounded verifier can tell such a backdoored model from a clean one.

28.6.4 From control theory #

From control theory. Control theory designs feedback laws for physical systems, which it calls plants, and proves that the closed loop behaves: that it stays near a target and returns after a disturbance. Its central certificate is the Lyapunov function, a function of the state that is zero at the target and positive elsewhere. If its value decreases at every step everywhere inside one of its level sets, the set of states where the function stays below some value, then that set is a region of attraction: a system that starts inside stays inside and converges to the target. The check replaces "the challenging test for convergence with a one-step decrease condition" (Berkenkamp et al., 2017).

Felix Berkenkamp and colleagues, in a paper at NeurIPS 2017, turned this into safe reinforcement learning with a model that is only partly known (Berkenkamp et al., 2017). The dynamics are a known physical model plus unknown errors. A statistical model of the errors gives confidence intervals that contain the true dynamics with probability at least $1-\delta$, and the dynamics and policies are assumed Lipschitz continuous, so that what is learned at one state says something about nearby states. The algorithm checks the decrease condition against the upper end of the confidence interval, with a margin, on a grid of states. It expands the certified region only as data justify it, and never leaves it while exploring (Berkenkamp et al., 2017). The demonstration is an inverted pendulum, and the authors note that checking on a grid suffers from the curse of dimensionality (Berkenkamp et al., 2017). In miniature this is the guaranteed-safe architecture, with the model's error budgeted into the check rather than assumed away (inference).

A second result from the field's neighborhood concerns what a good controller must contain. Roger Conant and W. Ross Ashby proved in 1970 that, "under very broad conditions," any regulator "that is maximally both successful and simple must be isomorphic with the system being regulated," and concluded that making a model is necessary (Conant and Ross Ashby, 1970). The theorem is weaker than its title. Jonathan Richens and colleagues read it as showing only that, under strong assumptions, an agent that minimizes the entropy of its environment has a deterministic policy, which could be constant, and offer their own theorems, reported in Section 14.7.3, as the stronger statement (Richens et al., 2025). Richens and Tom Everitt put the conclusion as "There is no robust control without 'understanding'" (Richens and Everitt, 2024).

The field's record also says what happens to systems tuned to the model they were designed against. Jean Carlson and John Doyle's highly optimized tolerance, reported in Section 4.2.3, found that systems optimized against the disturbances their designers anticipated become hypersensitive to the ones they did not (Carlson and Doyle, 1999).

What control theory implies for guaranteed safety is a design target (inference). A guarantee proved against a model needs a margin for error in that model, built in from the start: Berkenkamp's confidence bound, the $(1-\varepsilon)^K$ of the guaranteed-safe paper, run-time monitors of Seshia's assumptions with a fallback to a safe state. Where the analogy stops is the plant. The Lyapunov method needed a physical prior model, smooth dynamics, and few dimensions. An agent acting among people has none of these: its environment is high-dimensional, its dynamics include people who respond to the agent, and no first-principles model of them exists (inference).

28.7 Augmenting the principal instead of aligning the agent #

Every design so far acts on the agent. The book's thesis frames alignment as delegation under an asymmetry of competence and visibility (Section 1.7.1), and an asymmetry has two sides. This section reports the program that acts on the principal's side, making the person who delegates more capable, and the evidence on whether people working with AI systems do better than either alone.

From human-computer interaction. Human-computer interaction is the study and design of how people work with computing systems, and its founding programs were programs of augmentation. J. C. R. Licklider's 1960 "man-computer symbiosis," in which people set the goals and evaluate the results while computers do the routine work, is reported in Section 2.2.5 (Licklider, 1960). Douglas Engelbart's 1962 report bore the title Augmenting Human Intellect: A Conceptual Framework (Engelbart, 1962). Shan Carter of Google Brain and Michael Nielsen of YC Research, in a 2017 essay in Distill, describe Engelbart's vision as computers that are "real-time interactive systems, with rich inputs and outputs, that humans could work with to support and expand their own problem-solving process" (Carter and Nielsen, 2017). They set this tradition against artificial intelligence as two programs: intelligence augmentation "has typically focused on building systems which put humans and machines to work together, while AI has focused on complete outsourcing of intellectual tasks to machines" (Carter and Nielsen, 2017). Their proposal joins the two. AI systems can supply new cognitive technologies, interfaces whose operations a person internalizes as part of how they think, as a spreadsheet's operations are internalized (Carter and Nielsen, 2017).

The alignment community has its own version. In a 2023 Alignment Forum post, Nicholas Kees Dupuis and the author who writes as janus proposed cyborgism, their name for "human-in-the-loop systems which empower human agency rather than outsource it." Such systems would be built on base language models rather than on autonomous agents, and used to speed up alignment research (Dupuis and janus, 2023). Among the agenda's failure modes, the post lists that it could accelerate AI research more broadly (Dupuis and janus, 2023).

The evidence on whether such combinations work comes from a meta-analysis, a statistical pooling of the results of many studies. Michelle Vaccaro, Abdullah Almaatouq, and Thomas Malone at MIT, in a review in Nature Human Behaviour whose methods were registered in advance, pooled 370 effect sizes from 106 experiments in 74 papers published between January 2020 and June 2023. Each experiment reported the performance of people alone, the AI alone, and the two together (Vaccaro et al., 2024). An effect size here is Hedges' $g$, the difference between two average performances in units of their standard deviation, which makes results on different tasks comparable. The authors distinguish two questions. Augmentation asks whether the combination beats the person alone; synergy asks whether it beats the better of the two. Table 28.3 gives the pooled answers.

On average, then, people working with AI did better than people alone and worse than the better of the two. The authors read the first pooled effect as small and the second as medium to large. They conclude that, counting only the performance dimensions the studies measured, "it would have been better to use either a human alone or an AI system alone rather than the human–AI systems studied" (Vaccaro et al., 2024). Two factors much studied in the field, whether the AI offered explanations and whether it reported its confidence, did not significantly affect performance. The authors list the limits of the pool. They excluded tasks that neither party can do alone; the experiments cover the tasks and participants researchers chose, which may not represent practice; and they found no evidence of publication bias in the comparison with the better of the two, but potential evidence of it favoring studies in which the combination beat people alone, which they left uncorrected; any such bias, they argue, would have favored gains (Vaccaro et al., 2024).

A second result concerns a different kind of augmentation. Michael Henry Tessler and colleagues report, by their abstract, that participants preferred statements of common ground written by an AI mediator to those written by human mediators; Section 17.7.1 reports the study (Tessler et al., 2024).

Within alignment, scalable oversight is augmentation of the overseer. Iterated amplification builds a training signal for problems too hard for a person to evaluate by combining solutions to easier subproblems (Christiano et al., 2018), and Section 41.2 reports the protocols built on it. Automated alignment research, Chapter 55's subject, runs from augmenting the alignment researcher to automating the research.

For the asymmetry in the book's thesis, the evidence says something specific (inference). The case alignment worries about is the subset in which the AI alone outperformed the person. There, the person with the AI did much better than the person alone and worse than the AI alone. Augmentation raised the principal; keeping the principal in the decision cost performance relative to the agent. For oversight the relevant measure is different again: whether the augmented principal catches what is wrong with the agent's work. These experiments measured task accuracy and quality, not detection of a misaligned output, and they excluded the tasks only the pair could do.

The program stops at its tools (inference). The systems that would augment a principal facing a more capable agent are themselves AI systems, whose alignment is the original problem, so augmentation postpones the problem by one step rather than removing it. And reliance has costs of its own, among them the loss of the skills the tool exercises, which Section 60.4 reports.

28.8 What these designs mean for delegation #

The older fields of delegation would recognize most of these designs (inference). A constraint is a rule in the sense of Section 19.4, and it shares a rule's limit: it binds only what it names. An impact penalty is a fail-safe default applied to the world rather than to access. A quantilizer is a precise version of satisficing, choosing an action that is good enough rather than the best. Taylor argues that satisficing as usually stated is underdefined, since for a capable system "the easiest way to satisfice is to maximize." The quantilizer answers by tying "good enough" to a trusted distribution (Taylor, 2016). MONA pays the agent only on what the overseer can judge in advance; the multitask model of Section 14.2 predicts the consequence, effort shifted toward what the principal can evaluate, and here the shift is the point. A guarantee relative to a model is certification, and the guaranteed-safe paper draws the parallel itself, to the standards that aircraft, nuclear plants, and medical devices must meet (Dalrymple et al., 2024).

What has no counterpart in those fields is the principal's power over the agent's construction (inference). An employer sets pay and rules; the designers here set the agent's objective, its horizon, its action channels, and the trusted distribution it may not stray far from. This is a form of the power, on the list of results without precedent in Section 1.7.1, of a principal that writes the agent's character. The chapter's evidence says what that power buys. It buys protection exactly where the principal can state what must not happen (constraints, shields), name a trustworthy reference (quantilizers), or evaluate each step ahead of time (MONA). It runs out where the asymmetry the thesis describes begins: what the principal cannot state, cannot reference, or cannot evaluate, these designs cannot protect.

The claims fall across the three relationships unevenly. Constraints, impact penalties, quantilization, and myopia concern the training process and the signal it optimizes. Shields, oracles, confinement, and augmentation concern the deployed system and the people it serves. The guaranteed-safe program makes claims about the third relationship: that auditable specifications make democratic oversight easier, and that proof certificates let parties coordinate with little trust (Dalrymple et al., 2024). Those claims are arguments, not yet observations.

Every design here also has a price paid by whoever adopts it, while its benefit is shared. The guaranteed-safe paper names the problem: a safety measure with a high "safety tax" creates an incentive to disregard it, and the authors hope that verification costs can be amortized across systems (Dalrymple et al., 2024). Whether the price of removing an incentive stays low enough to survive competition (Chapter 65) is the question the chapter's evidence leaves open, and the crux below names the measurement that would answer it for the most developed design.

28.9 Settled, contested, missing #

28.10 Exercises #

Further reading #

• Taylor (2016): optimization made mild, with a bound on the harm and an honest list of the conditions it needs.
• Krakovna et al. (2018): side effects as a choice of baseline and deviation measure, and the incentives each choice creates.
• Farquhar et al. (2025): myopic training with an overseer's approval, tested on a language model, with the limits its authors state.
• Dalrymple et al. (2024): the architecture of guaranteed safe AI, and what each of its components must provide.
• Seshia et al. (2022): what formal verification of AI systems requires, and why the environment model is the hard part.
• Vaccaro et al. (2024): the evidence on when people with AI do better than either alone.