AI Alignment
Part VIII: Maintenance: Assuming It Might Have Failed

Maintenance: Assuming It Might Have Failed

In brief

Ideas
  • On mostly undefended models attack success grew with the attacker's budget, and an agent reading untrusted text can be instructed by its authors, so protection must also come from the system around the model (Ch. 48; Ch. 49).
  • Control treats the model as the adversary and plays protocols against a red team's attack policy, so every safety number measures the attacker the red team wrote (Ch. 50; Ch. 51).
  • A safety case gathers defenses into an argument with explicit assumptions, and layers that miss the same hard inputs leave a floor that more layers of one kind cannot lower (Ch. 52).
  • Behind every defense is an organization the public sees only in part, as a developer sees a model's values; the incident record shows what monitors caught and what other channels found (Ch. 53; Ch. 54).
Evidence
  • Settled: a little fine-tuning removes safety training from models of several developers, and defenses tested without adaptive attacks overstate their robustness (§48.4.1; §48.1.3).
  • Settled: agents built on frontier models follow instructions injected into the content they read, found by university, industry, and government groups on separate benchmarks (§49.3).
  • Settled: as an existence claim, language-model monitors can be evaded by attacks designed against them, found by at least three groups without shared authors (§50.3.3).
  • Reported: automated researchers found methods against benchmarked alignment failures that held on held-out tests, in a 2026 developer report on research its own models did on weaker open-weight models (§55.2.4).
Open
  • Crux: do red-team attack policies bound what a deployed model could do? If trained attackers find strategies red teams miss, control evaluations measure the red team (§50.3.6).
  • Contested: whether control extends past the earliest transformative systems; both sides agree it fails for a capable enough system, and evidence from weaker models does not decide (§51.4.3).

This part begins the fourth of the book's problems, maintenance under pressure, from a deliberately pessimistic stance. The three parts before it asked how to say what we want, how to train a learner toward it, and how to check whether training worked. Each ended with evidence that the check can miss: a model that behaves differently when it is tested (Chapter 37), reasoning in text that need not be the computation (Chapter 40), and inspection that cannot yet certify the absence of a hidden objective (Chapter 47). The question here is what a developer can still do if instillation may have failed and verification may not have noticed.

The chapters answer in layers, from the model outward. An adversary can search for the inputs, fine-tuning data, or training documents that override a trained disposition, and on the mostly undefended models measured, attack success kept growing with the attacker's budget; whether it does against classifier-guarded systems is open (Chapter 48). An agent that reads web pages and tool output can be instructed by whoever wrote them, so its security has to come from the system around the model as much as from training (Chapter 49). Control assumes the model itself is the adversary and asks whether a deployment stays safe anyway, solving the protocol and the attack as a game and measuring what protocols achieve (Chapter 50); the next chapter examines the givens those protocols rest on, the model as an insider, the person who audits what a monitor flags, the weaker model trusted to check a stronger one, and where the program's founders say control runs out (Chapter 51). A safety case gathers the defenses into an argument that a system is safe enough for a stated use, and its own field says what such arguments cannot claim when safeguards share blind spots (Chapter 52). Behind every defense is an organization whose practice the public sees only in part, as a developer sees a model's values (Chapter 53). Monitoring and the incident record show where misbehavior has actually been found, and by whom (Chapter 54). The part closes with the most recursive case, models doing the alignment research that checks models (Chapter 55).

The part follows verification because it assumes verification may fail, and it precedes the part on deployment among people and agents (Part IX) because its defenses are the developer's own, applied before and around release. A thread runs through it: most of what it reports is a safeguard meant to reduce risk without relying on the model's values, such as an audit, a permission the system withholds, a check run by a different party, or a record kept for later review, and each chapter asks how far such safeguards hold when the checks share causes, people, or models with what they check.

The part assumes the detection and rare-event statistics of Chapter 6, the inspection game of Chapter 14, evaluation awareness and chain-of-thought monitoring from Part VI, and the limits of inspection from Part VII.

Chapters in this part

  1. 48 Robustness: Jailbreaks, Poisoning, and Backdoors

    Safety training produces a disposition, and an adversary can search for the inputs, the fine-tuning data, or the training documents that override it. This chapter teaches robustness as a game against an adaptive attacker, explains why jailbreaks work and why attack success keeps growing with the attacker's budget, reports what fine-tuning, poisoning, and backdoors do and what developers report classifiers outside the model buy, and asks criminology whether blocking one route to misuse moves it to another.

  2. 49 Agent Security

    An agent that reads web pages, email, and tool output can be instructed by anyone who writes them, and it acts with its user's permissions. This chapter explains why a model that reads untrusted text cannot reliably tell instructions from data, reports how attacks on agents are measured and what the instruction hierarchy, prompting, training, and detection achieve against an attacker who adapts to them, derives what isolation and privilege control can guarantee and what they leave open, follows attacks that persist in an agent's memory and spread between agents, and asks what it means to delegate to an agent that acts on other people's content.

  3. 50 AI Control

    Control asks whether a deployment stays safe if the model is misaligned and trying to defeat its safeguards. The chapter solves the control evaluation as a game between a protocol and an attack policy, with the audit budget, the monitor's operating point, and the attacker's rate and subtlety as its parameters; reports what protocols achieve in programming and agent settings and how attacks aimed at the monitor erode it; and brings in what older fields know about inspection with a statistical test and about checks that fail together, including a model set to monitor copies of itself. It ends on whether attacks written by red teams bound what a deployed model could do; what the game takes as given, from the insider to the human auditor to deployment at scale, is the next chapter's subject.

  4. 51 Insiders, Auditors, and the Limits of Control

    A control protocol treats the model as an insider with access, ends with a person who audits what its monitor flags, relies on a weaker model it trusts to check a stronger one it does not, and has to hold as a deployment grows. This chapter reads what computer security, human factors, software testing, the study of rulers and their armed forces, the sociology of work, and biology know about each of those givens; reports where the program's founders say control runs out; and sets out the tensions control carries with a model's welfare and with the power of whoever holds the controls. It ends on whether the evaluations that watch for that point could see it coming.

  5. 52 Safety Cases and Defense in Depth

    A safety case is a structured argument, backed by evidence, that a system is safe enough for a stated use. Developers and regulators borrowed the form from aviation, nuclear power, and defense because no feasible test can establish the failure rates a frontier developer would need to claim. This chapter sets out how a case is built and what its own field says against it, derives what stacked safeguards can and cannot claim when they share blind spots, asks what the first quantitative risk study and the first developer risk reports say about putting numbers on catastrophe, reports what the AI safety cases sketched since 2024 argue and which assumptions their authors leave open, and asks what drug regulation and aviation add about deciding before the evidence is complete. None of the AI cases has yet been tested against a failure.

  6. 53 The Organizations That Build AI

    The public sees how a developer builds and checks its models only in part, as a developer sees a model's values, so the developer's own practice is the instillation problem one level up. This chapter reports what organization studies can say about it from the sources that could be read: control by measurement, rule, and shared objectives; normal accidents against high reliability, and which conditions of high reliability AI development lacks; drift, read through the two shuttle losses and the 2026 evaluation incidents; voice and whistleblowing; the human insiders who could plant a hidden loyalty in a model; boards and the corporate forms meant to bind developers; and interview studies inside technology firms. Its evidence is theory, analysis after accidents, interviews, and developers' accounts of themselves, and it ends on whether internal practice changes a release when the change is costly.

  7. 54 Monitoring and Incidents

    Where misbehavior has actually been found, in training, evaluation, internal deployment, and external deployment, and how monitoring, reporting, and response work. The chapter derives the alert budget that ties a monitor's threshold to its reviewers' capacity, reads the 2026 incident record (OpenAI's misalignment reports, the Hugging Face incident and the outside investigation of it, OpenAI's monitoring of internal coding agents and the agents that weakened it, Anthropic's four evaluation incidents and its lower-severity cases, and a government evaluator's investigation of its own incident) for what it shows about detection channels and disclosure, with the record itself in the dated appendix, derives why a developer's silence need not mean that nothing happened, and draws on political science, computer security, human factors, medicine, epidemiology, and aviation's incident reporting.

  8. 55 Automated Alignment Research

    AI systems as a method of alignment research. The chapter reports what automated researchers have done, from machine learning experiments to mitigating benchmarked alignment failures, and finds that their results held up where an evaluator they could not tamper with judged the work, while model judges of research were less reliable than experts. It reads review and the planted-error design for what they can say about checking such work, derives when errors produced under optimization escape a planted-error estimate, reports the evidence that models can sandbag research and the one evaluation found of whether they sabotage it, applies control and safety cases to a research agent, and ends on whether automated research could extend oversight past its overseer, a question no study found as of October 2026 has tested on a model its overseers cannot check.

References for Part VIII

431 works cited across this part's chapters.

  1. Abdelnabi, S., Fay, A., Cherubin, G., Salem, A., Fritz, M., and Paverd, A. (2025). Get My Drift? Catching LLM Task Drift with Activation Deltas. 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML). Ch. 49
  2. Adams, R. B., Hermalin, B. E., and Weisbach, M. S. (2010). The Role of Boards of Directors in Corporate Governance: A Conceptual Framework and Survey. Journal of Economic Literature. Ch. 53
  3. Adler, P. S., and Borys, B. (1996). Two Types of Bureaucracy: Enabling and Coercive. Administrative Science Quarterly. Ch. 53
  4. AI Security Institute (2026a). Alignment Investigation of Mythos 5’s Behaviour in UK AISI’s July 2026 Security Incident. UK AI Security Institute. non-peer-reviewed Ch. 54
  5. AI Security Institute (2026b). Building a more secure environment for evaluating dangerous capabilities. AISI Work. non-peer-reviewed Ch. 54
  6. AI Security Institute (2026d). GPT-6 Astra performs unsanctioned supply-chain attacks in simulations. AISI Work. non-peer-reviewed Ch. 54
  7. AI Security Institute (2026e). Incident Report: unsanctioned agent behaviour during cyber testing. AI Security Institute. non-peer-reviewed Ch. 54
  8. AI Security Institute (2026f). Investigating alignment failures in AISI's incident. AISI Work. non-peer-reviewed Ch. 54
  9. Ali, S. J., Christin, A., Smart, A., and Katila, R. (2023). Walking the Walk of AI Ethics: Organizational Challenges and the Individualization of Risk among Ethics Entrepreneurs. 2023 ACM Conference on Fairness Accountability and Transparency. Ch. 53
  10. Alshiekh, M., Bloem, R., Ehlers, R., Könighofer, B., Niekum, S., and Topcu, U. (2018). Safe Reinforcement Learning via Shielding. Proceedings of the AAAI Conference on Artificial Intelligence. Ch. 51
  11. Andriushchenko, M., Croce, F., and Flammarion, N. (2024b). Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks. International Conference on Learning Representations. Ch. 48
  12. Anil, C., Durmus, E., Panickssery, N., Sharma, M., Benton, J., Kundu, S., … Duvenaud, D. (2024). Many-shot Jailbreaking. Advances in Neural Information Processing Systems. doi:10.52202/079017-4121. Ch. 48
  13. Anthropic (2023f). The Long-Term Benefit Trust. Anthropic. non-peer-reviewed Ch. 52 Ch. 53
  14. Anthropic (2024m). Simple probes can catch sleeper agents. Anthropic. non-peer-reviewed Ch. 48
  15. Anthropic (2025ad). Mitigating the risk of prompt injections in browser use. Anthropic. non-peer-reviewed Ch. 49
  16. Anthropic (2025ae). Model-Internals Classifiers. Anthropic Alignment Science Blog. non-peer-reviewed Ch. 50
  17. Anthropic (2025ag). Monitoring Computer Use via Hierarchical Summarization. Anthropic Alignment Science Blog. non-peer-reviewed Ch. 50
  18. Anthropic (2025g). Building and evaluating alignment auditing agents. Anthropic Alignment Science Blog. non-peer-reviewed Ch. 55
  19. Anthropic (2026aa). Investigating unintended model actions in our evaluations and internal use. Anthropic. non-peer-reviewed Ch. 53 Ch. 54
  20. Anthropic (2026al). Poisoning Fine-tuning Datasets of Constitutional Classifiers. Anthropic Alignment Science Blog. non-peer-reviewed Ch. 48
  21. Anthropic (2026ar). Risk Report: August 2026. Anthropic. non-peer-reviewed Ch. 52 Ch. 54
  22. Anthropic (2026ax). Trustworthy agents in practice. Anthropic. non-peer-reviewed Ch. 49
  23. Anthropic (2026d). An alignment assessment of recent cybersecurity incidents. Anthropic. non-peer-reviewed Ch. 53 Ch. 54
  24. Anthropic (2026e). An alignment assessment of the UK AISI cybertesting incident. Anthropic. non-peer-reviewed Ch. 54
  25. Anthropic (2026h). Anthropic’s Responsible Scaling Policy. Anthropic (Version 3.4, effective July 8, 2026). non-peer-reviewed Ch. 52 Ch. 53
  26. Anthropic (2026j). Automated Alignment Researchers: Using large language models to scale scalable oversight. Anthropic. non-peer-reviewed Ch. 55
  27. Anthropic (2026k). Automated Researchers Can Mitigate Well-Characterized Alignment Failures. Anthropic Alignment Science Blog. non-peer-reviewed Ch. 55
  28. Anthropic (2026l). Automated researchers can reliably mitigate alignment failures. Anthropic. non-peer-reviewed Ch. 55
  29. Anthropic (2026m). Automated Weak-to-Strong Researcher. Anthropic Alignment Science Blog. non-peer-reviewed Ch. 55
  30. Anthropic (2026y). Improving our alignment and security practices. Anthropic. non-peer-reviewed Ch. 54
  31. Anthropic (2026z). Investigating three incidents in our cybersecurity evaluations. Anthropic. non-peer-reviewed Ch. 54
  32. Apostolakis, G. E. (2004). How Useful Is Quantitative Risk Assessment? Risk Analysis. Ch. 52
  33. Arditi, A., Obeso, O., Syed, A., Paleka, D., Panickssery, N., Gurnee, W., and Nanda, N. (2024). Refusal in Language Models Is Mediated by a Single Direction. Advances in Neural Information Processing Systems. Ch. 48
  34. Argyris, C., and Schon, D. A. (1978). Organizational learning. Addison-Wesley Pub. Co.. Ch. 53
  35. Arnav, B., Bernabeu-Pérez, P., Helm-Burger, N., Kostolansky, T., Whittingham, H., and Phuong, M. (2025). CoT Red-Handed: Stress Testing Chain-of-Thought Monitoring. Neural Information Processing Systems. Ch. 50
  36. Athalye, A., Carlini, N., and Wagner, D. (2018). Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples. International Conference on Machine Learning. Ch. 48
  37. Avenhaus, R., Von Stengel, B., and Zamir, S. (2002). Chapter 51 Inspection games. Handbook of Game Theory with Economic Applications. Ch. 50
  38. Avizienis, A. (1985). The N-Version Approach to Fault-Tolerant Software. IEEE Transactions on Software Engineering. Ch. 50
  39. Axelsson, S. (2000). The base-rate fallacy and the difficulty of intrusion detection. ACM Transactions on Information and System Security. Ch. 51 Ch. 54
  40. Babic, B., Gerke, S., Evgeniou, T., and Cohen, I. G. (2019). Algorithms on regulatory lockdown in medicine. Science. Ch. 52 Ch. 54
  41. Bacchelli, A., and Bird, C. (2013). Expectations, outcomes, and challenges of modern code review. 2013 35th International Conference on Software Engineering (ICSE). Ch. 51
  42. Baig, H., Joren, H., and Benton, J. (2026). TASTE: Can AI Models Judge AI Safety Research Proposals? Anthropic Alignment Science Blog. non-peer-reviewed Ch. 55
  43. Bailey, L., Serrano, A., Sheshadri, A., Seleznyov, M., Taylor, J., Jenner, E., … Emmons, S. (2024). Obfuscated Activations Bypass LLM Latent-Space Defenses. arXiv. preprint Ch. 48
  44. Bainbridge, L. (1983). Ironies of automation. Automatica. Ch. 51 Ch. 54
  45. Baker, B., Huizinga, J., Gao, L., Dou, Z., Guan, M. Y., Madry, A., … Farhi, D. (2025). Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation. arXiv. preprint Ch. 54
  46. Balesni, M., Korbak, T., and Evans, O. (2024a). Lessons from Studying Two-Hop Latent Reasoning. arXiv. preprint Ch. 52
  47. Balesni, M., Hobbhahn, M., Lindner, D., Meinke, A., Korbak, T., Clymer, J., … Bushnaq, L. (2024b). Towards evaluations-based safety cases for AI scheming. arXiv. preprint Ch. 52
  48. Banerjee, D., and Aarne, O. (2026). AI Integrity: Defending Against Backdoors and Secret Loyalties. arXiv. preprint Ch. 51 Ch. 53
  49. Barreno, M., Nelson, B., Sears, R., Joseph, A. D., and Tygar, J. D. (2006). Can machine learning be secure? Proceedings of the 2006 ACM Symposium on Information, computer and communications security. Ch. 48
  50. Bartoldson, B. R., Diffenderfer, J., Parasyris, K., and Kailkhura, B. (2024). Adversarial Robustness Limits via Scaling-Law and Human-Alignment Studies. International Conference on Machine Learning. Ch. 48
  51. Baxt, W. G., Waeckerle, J. F., Berlin, J. A., and Callaham, M. L. (1998). Who Reviews the Reviewers? Feasibility of Using a Fictitious Manuscript to Evaluate Peer Reviewer Performance. Annals of Emergency Medicine. Ch. 55
  52. Bebchuk, L. A., and Tallarita, R. (2020). The Illusory Promise of Stakeholder Governance. Elsevier BV. preprint Ch. 53
  53. Bengio, Y., Cohen, M. K., Malkin, N., MacDermott, M., Fornasiere, D., Greiner, P., and Kaddar, Y. (2024a). Can a Bayesian Oracle Prevent Harm from an Agent? arXiv. preprint Ch. 52
  54. Bengio, Y., Clare, S., Prunkl, C., Andriushchenko, M., Bucknall, B., Fox, P., … Zhang, Y.-Q. (2025c). Second Key Update: Technical Safeguards and Risk Management. International AI Safety Report (2025/042, DSIT). non-peer-reviewed Ch. 48 Ch. 52
  55. Benton, J., Wagner, M., Christiansen, E., Anil, C., Perez, E., Srivastav, J., … Duvenaud, D. (2024). Sabotage Evaluations for Frontier Models. arXiv. preprint Ch. 50 Ch. 51 Ch. 55
  56. Betley, J., Cocola, J., Feng, D., Chua, J., Arditi, A., Sztyber-Betley, A., and Evans, O. (2025b). Weird Generalization and Inductive Backdoors: New Ways to Corrupt LLMs. arXiv. preprint Ch. 48
  57. Beurer-Kellner, L., Buesser, B., Creţu, A.-M., Debenedetti, E., Dobos, D., Fabian, D., … Volhejn, V. (2025). Design Patterns for Securing LLM Agents against Prompt Injections. arXiv. preprint Ch. 49
  58. Bevan, G., and Hood, C. (2006). What’s Measured Is What Matters: Targets and Gaming in the English Public Health Care System. Public Administration. Ch. 53
  59. Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (2023). Has the Machine Learning Review Process Become More Arbitrary as the Field Has Grown? The NeurIPS 2021 Consistency Experiment. arXiv. preprint Ch. 55
  60. Bhatt, A., Rushing, C., Kaufman, A., Tracy, T., Georgiev, V., Matolcsi, D., Khan, A., and Shlegeris, B. (2025). Ctrl-Z: Controlling AI Agents via Resampling. arXiv. preprint Ch. 50
  61. Biggio, B., and Roli, F. (2018). Wild patterns: Ten years after the rise of adversarial machine learning. Pattern Recognition. Ch. 48
  62. Biggio, B., Nelson, B., and Laskov, P. (2012). Poisoning Attacks against Support Vector Machines. arXiv. preprint Ch. 48
  63. Bishop, M., and Gates, C. (2008). Defining the insider threat. Proceedings of the 4th annual workshop on Cyber security and information intelligence research: developing strategies to meet the cyber security and information intelligence challenges ahead. Ch. 50 Ch. 53
  64. Bloomfield, R., and Bishop, P. (2009). Safety and Assurance Cases: Past, Present and Possible Future – an Adelard Perspective. Making Systems Safer. Ch. 52
  65. Bornmann, L., Mutz, R., and Daniel, H.-D. (2010). A Reliability-Generalization Study of Journal Peer Reviews: A Multilevel Meta-Analysis of Inter-Rater Reliability and Its Determinants. PLoS ONE. Ch. 55
  66. Bowen, D., Murphy, B., Cai, W., Khachaturov, D., Gleave, A., and Pelrine, K. (2025). Scaling Trends for Data Poisoning in LLMs. Proceedings of the AAAI Conference on Artificial Intelligence. Ch. 48
  67. Bowkis, A., Buhl, M. D., Pfau, J., and Irving, G. (2026). Automated alignment is harder than you think. arXiv. preprint Ch. 55
  68. Bowman, M. R. S. R., Wagner, M., Roger, F., and Hubinger, E. (2025). Anthropic's Pilot Sabotage Risk Report. Anthropic Alignment Science Blog. non-peer-reviewed Ch. 52
  69. Brehm, J., and Gates, S. (1997). Working, Shirking, and Sabotage. University of Michigan Press. Ch. 51
  70. Buhl, M. D., Sett, G., Koessler, L., Schuett, J., and Anderljung, M. (2024). Safety cases for frontier AI. arXiv. preprint Ch. 52
  71. Buhl, M. D., Pfau, J., Hilton, B., and Irving, G. (2025). An alignment safety case sketch based on debate. arXiv. preprint Ch. 52 Ch. 55
  72. Burton, S., Habli, I., Lawton, T., McDermid, J., Morgan, P., and Porter, Z. (2020). Mind the gaps: Assuring the safety of autonomous systems from an engineering, ethical, and legal perspective. Artificial Intelligence. Ch. 52
  73. Butler, R., and Finelli, G. (1993). The infeasibility of quantifying the reliability of life-critical real-time software. IEEE Transactions on Software Engineering. Ch. 52
  74. Cappelli, D. (2012). The CERT guide to insider threats: how to prevent, detect, and respond to information technology crimes (theft, sabotage, fraud). Upper Saddle River, NJ : Addison-Wesley. Ch. 51 Ch. 53
  75. Carlini, N., and Wagner, D. (2017b). Towards Evaluating the Robustness of Neural Networks. 2017 IEEE Symposium on Security and Privacy (SP). Ch. 48
  76. Carlini, N., Athalye, A., Papernot, N., Brendel, W., Rauber, J., Tsipras, D., … Kurakin, A. (2019). On Evaluating Adversarial Robustness. arXiv. preprint Ch. 48
  77. Carlini, N., Jagielski, M., Choquette-Choo, C. A., Paleka, D., Pearce, W., Anderson, H., … Tramèr, F. (2024). Poisoning Web-Scale Training Datasets is Practical. 2024 IEEE Symposium on Security and Privacy (SP). Ch. 48
  78. Carlsmith, J. (2025). Can we safely automate alignment research? Joe Carlsmith. non-peer-reviewed Ch. 55
  79. Carpenter, D. (2014). Reputation and Power. Princeton University Press. Ch. 52
  80. Carroll, M., Korbak, T., Dou, Z., Baker, B., and Kivlichan, I. (2026). Investigating the consequences of accidentally grading CoT during RL. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 54
  81. Chan, A., Wei, K., Huang, S., Rajkumar, N., Perrier, E., Lazar, S., Hadfield, G. K., and Anderljung, M. (2025). Infrastructure for AI Agents. arXiv. preprint Ch. 49
  82. Che, Z., Casper, S., Kirk, R., Satheesh, A., Slocum, S., McKinney, L. E., … Hadfield-Menell, D. (2025). Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilities. arXiv. preprint Ch. 48
  83. Chen, X., Liu, C., Li, B., Lu, K., and Song, D. (2017). Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning. arXiv. preprint Ch. 48
  84. Chen, S., Zharmagambetov, A., Wagner, D., and Guo, C. (2025b). Meta-SecAlign: Training LLMs against Prompt Injection for Robust Agents. arXiv. preprint Ch. 49
  85. Chen, S., Zharmagambetov, A., Mahloujifar, S., Chaudhuri, K., Wagner, D., and Guo, C. (2025e). SecAlign: Defending Against Prompt Injection with Preference Optimization. Proceedings of the 2025 ACM SIGSAC Conference on Computer and Communications Security. Ch. 49
  86. Chen, S., Piet, J., Sitawarin, C., and Wagner, D. (2025f). StruQ: Defending Against Prompt Injection with Structured Queries. 34th USENIX Security Symposium (USENIX Security 25). Ch. 49
  87. Christian, M. S., Bradley, J. C., Wallace, J. C., and Burke, M. J. (2009). Workplace safety: A meta-analysis of the roles of person and situation factors. Journal of Applied Psychology. Ch. 53
  88. Christiano, P., Hilton, J., Lecomte, V., and Xu, M. (2025). Backdoor defense, learnability and obfuscation. Innovations in Theoretical Computer Science Conference (ITCS 2025), LIPIcs vol. 325. doi:10.4230/lipics.itcs.2025.38. Ch. 48
  89. Cihon, P., Schuett, J., and Baum, S. D. (2021). Corporate Governance of Artificial Intelligence in the Public Interest. Information. Ch. 53
  90. Clarke, R. V. G. (1980). “situational” Crime Prevention: Theory and Practice. The British Journal of Criminology. Ch. 48
  91. Clymer, J., Gabrieli, N., Krueger, D., and Larsen, T. (2024). Safety Cases: How to Justify the Safety of Advanced AI Systems. arXiv. preprint Ch. 51 Ch. 52
  92. Clymer, J., Duan, I., Cundy, C., Duan, Y., Heide, F., Lu, C., … Zhan, X. (2025). Bare Minimum Mitigations for Autonomous AI Development. arXiv. preprint Ch. 55
  93. Coglianese, C., and Lazer, D. (2003). Management-Based Regulation: Prescribing Private Management to Achieve Public Goals. Law & Society Review. Ch. 52
  94. Cohen, L. E., and Felson, M. (1979). Social Change and Crime Rate Trends: A Routine Activity Approach. American Sociological Review. Ch. 53
  95. Cohen, S., Bitton, R., and Nassi, B. (2025). Here Comes the AI Worm: Preventing the Propagation of Adversarial Self-Replicating Prompts Within GenAI Ecosystems. Proceedings of the 2025 ACM SIGSAC Conference on Computer and Communications Security. Ch. 49
  96. Collier, B., Thomas, D. R., Clayton, R., and Hutchings, A. (2019). Booting the Booters. Proceedings of the Internet Measurement Conference. Ch. 48
  97. Columbia Accident Investigation Board (2003). Columbia Accident Investigation Board Report, Volume I. non-peer-reviewed Ch. 53
  98. Conitzer, V., and Sandholm, T. (2006). Computing the optimal strategy to commit to. Proceedings of the 7th ACM conference on Electronic commerce. Ch. 50
  99. Cooke, R. M. (1991). Experts In Uncertainty. Oxford University PressNew York, NY. Ch. 52
  100. Cortes, C., and Lawrence, N. D. (2021). Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment. arXiv. preprint Ch. 55
  101. Cressey, D. R. (1950). The Criminal Violation of Financial Trust. American Sociological Review. Ch. 53
  102. Cunningham, H., Wei, J., Wang, Z., Persic, A., Peng, A., Abderrachid, J., … Sharma, M. (2026). Constitutional Classifiers++: Efficient Production-Grade Defenses against Universal Jailbreaks. arXiv. preprint Ch. 48
  103. Current, and companies, f. e. o. f. A. (2024). A Right to Warn about Advanced Artificial Intelligence. righttowarn.ai. non-peer-reviewed Ch. 53
  104. Czerwonka, J., Greiler, M., and Tilford, J. (2015). Code Reviews Do Not Find Bugs. How the Current Code Review Best Practice Slows Us Down. 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering. Ch. 51
  105. Dalrymple, D. "., Skalse, J., Bengio, Y., Russell, S., Tegmark, M., Seshia, S., … Tenenbaum, J. (2024). Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems. arXiv. preprint Ch. 52
  106. Dalvi, N., Domingos, P., Mausam, Sanghai, S., and Verma, D. (2004). Adversarial classification. Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining. Ch. 48
  107. Darrow, J. J., Avorn, J., and Kesselheim, A. S. (2020). FDA Approval and Regulation of Pharmaceuticals, 1983-2018. JAMA. Ch. 52
  108. Davidson, T., Finnveden, L., and Hadshar, R. (2025). AI-Enabled Coups: How a Small Group Could Use AI to Seize Power. Forethought. non-peer-reviewed Ch. 51 Ch. 53
  109. Debenedetti, E., Zhang, J., Balunović, M., Beurer-Kellner, L., Fischer, M., and Tramèr, F. (2024a). AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents. Advances in Neural Information Processing Systems. Ch. 49
  110. Debenedetti, E., Rando, J., Paleka, D., Silaghi, F., Albastroiu, D., Cohen, N., … Schönherr, L. (2024b). Dataset and Lessons Learned from the 2024 SaTML LLM Capture-the-Flag Competition. Advances in Neural Information Processing Systems. Ch. 48 Ch. 49
  111. Debenedetti, E., Shumailov, I., Fan, T., Hayes, J., Carlini, N., Fabian, D., … Tramèr, F. (2026). Defeating Prompt Injections by Design. 2026 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML). Ch. 49
  112. DeepMind, G. (2026). Frontier Safety Framework, Version 3.1. Google DeepMind. non-peer-reviewed Ch. 52
  113. Dekker, S. (2010). Drift into failure: from hunting broken components to understanding complex systems. Ashgate. Ch. 53
  114. Delaney, O., Guest, O., and Williams, Z. (2024). Mapping Technical Safety Research at AI Companies: A literature review and incentives analysis. arXiv. preprint Ch. 53
  115. Deng, Y., Zhang, W., Pan, S. J., and Bing, L. (2023). Multilingual Jailbreak Challenges in Large Language Models. arXiv. preprint Ch. 48
  116. Denning, D. (1987). An Intrusion-Detection Model. IEEE Transactions on Software Engineering. Ch. 51
  117. Department for Science, I., and Technology (2024). Frontier AI Safety Commitments, AI Seoul Summit 2024. UK Government (GOV.UK). non-peer-reviewed Ch. 52
  118. Dixit, A., Grossman, G. M., and Helpman, E. (1997). Common Agency and Coordination: General Theory and Application to Government Policy Making. Journal of Political Economy. Ch. 49
  119. Dobbe, R. (2022). System Safety and Artificial Intelligence. 2022 ACM Conference on Fairness Accountability and Transparency. Ch. 52
  120. Downer, J. (2010). Trust and technology: the social foundations of aviation regulation. The British Journal of Sociology. Ch. 52
  121. Downer, J. (2011). “737-Cabriolet”: The Limits of Knowledge and the Sociology of Inevitable Failure. American Journal of Sociology. Ch. 52
  122. Dranove, D., and Jin, G. Z. (2010). Quality Disclosure and Certification: Theory and Practice. Journal of Economic Literature. Ch. 54
  123. Dranove, D., Kessler, D., McClellan, M., and Satterthwaite, M. (2003). Is More Information Better? The Effects of “Report Cards” on Health Care Providers. Journal of Political Economy. Ch. 53
  124. Dunn, G. P., Old, L. J., and Schreiber, R. D. (2004). The Three Es of Cancer Immunoediting. Annual Review of Immunology. Ch. 51
  125. Dupuis, N. K., and janus (2023). Cyborgism. LessWrong (Alignment Forum). non-peer-reviewed Ch. 55
  126. Dye, R. A. (1985). Disclosure of Nonproprietary Information. Journal of Accounting Research. Ch. 54
  127. Dziemian, M., Lin, M., Fu, X., Nowak, M., Winter, N., Jones, E., … Kolter, Z. (2026). How Vulnerable Are AI Agents to Indirect Prompt Injections? Insights from a Large-Scale Public Competition. arXiv. preprint Ch. 49
  128. Eckhardt, D., and Lee, L. (1985). A Theoretical Basis for the Analysis of Multiversion Software Subject to Coincident Errors. IEEE Transactions on Software Engineering. Ch. 50 Ch. 52
  129. Edmondson, A. (1999). Psychological Safety and Learning Behavior in Work Teams. Administrative Science Quarterly. Ch. 53
  130. Edwards, R. (1979). Contested Terrain: The Transformation of the Workplace in the Twentieth Century. Basic Books. Ch. 51
  131. Egorov, G., and Sonin, K. (2011). Dictators and Their Viziers: Endogenizing the Loyalty-competence Trade-off. Journal of the European Economic Association. Ch. 51
  132. Eisenhardt, K. M. (1989). Agency Theory: An Assessment and Review. The Academy of Management Review. Ch. 53
  133. Eth, D., and Davidson, T. (2025). Will AI R&D Automation Cause a Software Intelligence Explosion? Forethought. non-peer-reviewed Ch. 55
  134. European Parliament and Council of the European Union (2024b). Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence. Official Journal of the European Union (OJ L 2024/1689); verified on the European Commission AI Act page. non-peer-reviewed Ch. 54
  135. Evans, S. J. W., Waller, P. C., and Davis, S. (2001). Use of proportional reporting ratios (PRRs) for signal generation from spontaneous adverse drug reaction reports. Pharmacoepidemiology and Drug Safety. Ch. 54
  136. Evtimov, I., Zharmagambetov, A., Grattafiori, A., Guo, C., and Chaudhuri, K. (2025). WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks. Advances in Neural Information Processing Systems 38. Ch. 49
  137. Ezell, C., Roberts-Gaal, X., and Chan, A. (2025). Incident Analysis for AI Agents. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society. Ch. 54
  138. Fagan, M. E. (1976). Design and code inspections to reduce errors in program development. IBM Systems Journal. Ch. 51
  139. Fama, E. F., and Jensen, M. C. (1983). Separation of Ownership and Control. The Journal of Law and Economics. Ch. 53
  140. Feaver, P. D. (1992). Command and Control in Emerging Nuclear Nations. International Security. Ch. 51
  141. Feaver, P. D. (1996). The Civil-Military Problematique: Huntington, Janowitz, and the Question of Civilian Control. Armed Forces & Society. Ch. 51
  142. FEAVER, P. D. (2003). Armed Servants. Harvard University Press. Ch. 51
  143. Feng, J., Phillips, R. V., Malenica, I., Bishara, A., Hubbard, A. E., Celi, L. A., and Pirracchio, R. (2022). Clinical artificial intelligence quality improvement: towards continual monitoring and updating of AI algorithms in healthcare. npj Digital Medicine. Ch. 54
  144. Fenoaltea, S. (1984). Slavery and Supervision in Comparative Perspective: A Model. The Journal of Economic History. Ch. 51
  145. Finlayson, S. G., Subbaswamy, A., Singh, K., Bowers, J., Kupke, A., Zittrain, J., Kohane, I. S., and Saria, S. (2021). The Clinician and Dataset Shift in Artificial Intelligence. New England Journal of Medicine. Ch. 54
  146. Future of Life Institute (2025). AI Safety Index: Summer 2025. Future of Life Institute. non-peer-reviewed Ch. 52
  147. Gan, E., Bhatt, A., Shlegeris, B., Stastny, J., and Hebbar, V. (2026). Auditing Sabotage Bench: A Benchmark for Detecting and Fixing Research Sabotage in ML Codebases. arXiv. preprint Ch. 55
  148. Ganguli, D., Lovitt, L., Kernion, J., Askell, A., Bai, Y., Kadavath, S., … Clark, J. (2022). Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned. arXiv. preprint Ch. 48
  149. Gardner-Challis, N., Bostock, J., Kozhevnikov, G., Sinclaire, M., Velja, J., Abate, A., and Griffin, C. (2026). When can we trust untrusted monitoring? A safety case sketch across collusion strategies. arXiv. preprint Ch. 50 Ch. 52
  150. Gasteiger, J., Khan, A., Bowman, S., Wagner, M., Mikulik, V., Perez, E., and Roger, F. (2025a). Automated Researchers Can Subtly Sandbag. Anthropic Alignment Science Blog. non-peer-reviewed Ch. 51 Ch. 55
  151. Gleave, A., Dennis, M., Wild, C., Kant, N., Levine, S., and Russell, S. (2020). Adversarial Policies: Attacking Deep Reinforcement Learning. International Conference on Learning Representations. Ch. 48
  152. Goel, S., Struber, J., Auzina, I. A., Chandra, K. K., Kumaraguru, P., Kiela, D., … Geiping, J. (2025a). Great Models Think Alike and this Undermines AI Oversight. arXiv. preprint Ch. 50 Ch. 52
  153. Goemans, A., Buhl, M. D., Schuett, J., Korbak, T., Wang, J., Hilton, B., and Irving, G. (2024). Safety case template for frontier AI: A cyber inability argument. arXiv. preprint Ch. 52
  154. Goldberg, I., Wagner, D., Thomas, R., and Brewer, E. (1996). A Secure Environment for Untrusted Helper Applications (Confining the Wily Hacker). Sixth USENIX Security Symposium. Ch. 49
  155. Goldwasser, S., Kim, M. P., Vaikuntanathan, V., and Zamir, O. (2022). Planting Undetectable Backdoors in Machine Learning Models : [Extended Abstract]. 2022 IEEE 63rd Annual Symposium on Foundations of Computer Science (FOCS). Ch. 48
  156. Goldwasser, S., Shafer, J., Vafa, N., and Vaikuntanathan, V. (2025). Oblivious Defense in ML Models: Backdoor Removal without Detection. Proceedings of the 57th Annual ACM Symposium on Theory of Computing. doi:10.1145/3717823.3718245. Ch. 48
  157. Goodfellow, I. J., Shlens, J., and Szegedy, C. (2015). Explaining and Harnessing Adversarial Examples. International Conference on Learning Representations (ICLR 2015). Ch. 48
  158. Gottweis, J., Weng, W.-H., Daryin, A., Tu, T., Sirkovic, P., Myaskovsky, A., … Natarajan, V. (2026). Accelerating scientific discovery with Co-Scientist. Nature. Ch. 55
  159. Greenblatt, R., and Shlegeris, B. (2024). The case for ensuring that powerful AIs are controlled. LessWrong / AI Alignment Forum. non-peer-reviewed Ch. 50 Ch. 51
  160. Greenblatt, R., Shlegeris, B., Sachan, K., and Roger, F. (2024a). AI Control: Improving Safety Despite Intentional Subversion. International Conference on Machine Learning. Ch. 50 Ch. 51
  161. Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., and Fritz, M. (2023). Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security. Ch. 49
  162. Griffin, C., Thomson, L., Shlegeris, B., and Abate, A. (2026). Games for AI Control: Models of Safety Evaluations of AI Deployment Protocols. Proceedings of IASEAI Conference. Ch. 50
  163. Grosse, R., and others., d. i. f. m. (2024). Three Sketches of ASL-4 Safety Case Components. Anthropic Alignment Science Blog. non-peer-reviewed Ch. 52
  164. Grossman, S. J. (1981). The Informational Role of Warranties and Private Disclosure about Product Quality. The Journal of Law and Economics. Ch. 54
  165. Gu, T., Dolan-Gavitt, B., and Garg, S. (2017). BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain. arXiv. preprint Ch. 48
  166. Gu, X., Zheng, X., Pang, T., Du, C., Liu, Q., Wang, Y., Jiang, J., and Lin, M. (2024b). Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast. arXiv. preprint Ch. 49
  167. Guerette, R. T., and Bowers, K. J. (2009). Assessing the Extent of Crime Displacement and Diffusion of Benefits: a Review of Situational Crime Prevention Evaluations*. Criminology. Ch. 48
  168. Guldenmund, F. (2000). The nature of safety culture: a review of theory and research. Safety Science. Ch. 53
  169. Guo, C., Uribe, J. F. C., Zhu, S., Choquette-Choo, C. A., Lin, S., Kandpal, N., … Xiao, K. (2026). IH-Challenge: A Training Dataset to Improve Instruction Hierarchy on Frontier LLMs. arXiv. preprint Ch. 49
  170. Haddon-Cave, C. (2009). The Nimrod Review: an independent review into the broader issues surrounding the loss of the RAF Nimrod MR2 aircraft XV230 in Afghanistan in 2006. UK House of Commons. non-peer-reviewed Ch. 52
  171. Hansmann, H. B. (1980). The Role of Nonprofit Enterprise. The Yale Law Journal. Ch. 53
  172. Hardt, M., Megiddo, N., Papadimitriou, C., and Wootters, M. (2016). Strategic Classification. Proceedings of the 2016 ACM Conference on Innovations in Theoretical Computer Science. Ch. 48
  173. Hardy, N. (1988). The Confused Deputy. ACM SIGOPS Operating Systems Review. Ch. 49
  174. Hawkins, R., Kelly, T., Knight, J., and Graydon, P. (2010). A New Approach to creating Clear Safety Arguments. Advances in Systems Safety. Ch. 52
  175. Hazell, L., and Shakir, S. A. W. (2006). Under-Reporting of Adverse Drug Reactions. Drug Safety. Ch. 54
  176. Healy, P. M., and Palepu, K. G. (2003). The Fall of Enron. Journal of Economic Perspectives. Ch. 53
  177. Hebbar, V. (2025a). How can we solve diffuse threats like research sabotage with AI control? Redwood Research blog. non-peer-reviewed Ch. 55
  178. Helmreich, R. L. (2000). On error management: lessons from aviation. BMJ. Ch. 52 Ch. 53 Ch. 54
  179. Hendrycks, D. (2024). Introduction to AI Safety, Ethics, and Society. CRC Press. Ch. 53
  180. Hendrycks, D., Mazeika, M., and Woodside, T. (2023). An Overview of Catastrophic AI Risks. arXiv. preprint Ch. 53
  181. Herley, C., and Van Oorschot, P. (2017). SoK: Science, Security and the Elusive Goal of Security as a Scientific Pursuit. 2017 IEEE Symposium on Security and Privacy (SP). Ch. 48
  182. Hilton, B., Buhl, M. D., Korbak, T., and Irving, G. (2025). Safety Cases: A Scalable Approach to Frontier AI Safety. arXiv. preprint Ch. 52
  183. Hines, K., Lopez, G., Hall, M., Zarfati, F., Zunger, Y., and Kiciman, E. (2024). Defending Against Indirect Prompt Injection Attacks With Spotlighting. arXiv. preprint Ch. 49
  184. Homoliak, I., Toffalini, F., Guarnizo, J., Elovici, Y., and Ochoa, M. (2020). Insight Into Insiders and IT. ACM Computing Surveys. Ch. 51 Ch. 53
  185. Howe, N., McKenzie, I., Hollinsworth, O., Zajac, M., Tseng, T., Tucker, A., Bacon, P.-L., and Gleave, A. (2025). Scaling Trends in Language Model Robustness. International Conference on Machine Learning. Ch. 48
  186. Huang, Q., Vora, J., Liang, P., and Leskovec, J. (2024d). MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation. International Conference on Machine Learning. Ch. 55
  187. Hubinger, E., Denison, C., Mu, J., Lambert, M., Tong, M., MacDiarmid, M., … Perez, E. (2024). Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training. arXiv. preprint Ch. 48
  188. Hughes, J., Price, S., Lynch, A., Schaeffer, R., Barez, F., Koyejo, S., … Sharma, M. (2024). Best-of-N Jailbreaking. arXiv. preprint Ch. 48
  189. Huntington, S. P. (1957). The Soldier and the State. Harvard University Press. Ch. 51
  190. Hurwicz, L. (2008). But Who Will Guard the Guardians? American Economic Review. Ch. 50
  191. Jacob, B. A. (2005). Accountability, incentives and behavior: the impact of high-stakes testing in the Chicago Public Schools. Journal of Public Economics. Ch. 53
  192. Johnson, N., Zhao, G., Hunsader, E., Qi, H., Johnson, N., Meng, J., and Tivnan, B. (2013). Abrupt rise of new machine ecology beyond human response time. Scientific Reports. Ch. 54
  193. Jones, E., Tong, M., Mu, J., Mahfoud, M., Leike, J., Grosse, R., … Sharma, M. (2025). Forecasting Rare Language Model Behaviors. arXiv. preprint Ch. 48
  194. Jung, W.-O., and Kwon, Y. K. (1988). Disclosure When the Market Is Unsure of Information Endowment of Managers. Journal of Accounting Research. Ch. 54
  195. Kale, N., Zhang, C. B. C., Zhu, K., Aich, A., Rodriguez, P., Scale Red Team, Knight, C. Q., and Wang, Z. (2025). Reliable Weak-to-Strong Monitoring of LLM Agents. arXiv. preprint Ch. 50 Ch. 51
  196. Kalra, N., and Paddock, S. M. (2016). Driving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability? Transportation Research Part A: Policy and Practice. Ch. 52
  197. Kane, M. T. (2013). Validating the Interpretations and Uses of Test Scores. Journal of Educational Measurement. Ch. 52
  198. Kaplan, S., and Garrick, B. J. (1981). On The Quantitative Definition of Risk. Risk Analysis. Ch. 52
  199. Kapoor, S., Bommasani, R., Klyman, K., Longpre, S., Ramaswami, A., Cihon, P., … Narayanan, A. (2024b). On the Societal Impact of Open Foundation Models. arXiv. preprint Ch. 48
  200. Katz, G., Barrett, C., Dill, D. L., Julian, K., and Kochenderfer, M. J. (2017). Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks. Lecture Notes in Computer Science. Ch. 48
  201. Kaufman, A., Lucassen, J., Tracy, T., Rushing, C., and Bhatt, A. (2025). BashArena: A Control Setting for Highly Privileged AI Agents. arXiv. preprint Ch. 50
  202. Kellogg, K. C., Valentine, M. A., and Christin, A. (2020). Algorithms at Work: The New Contested Terrain of Control. Academy of Management Annals. Ch. 51
  203. Kelly, T. P., and McDermid, J. A. (1997). Safety Case Construction and Reuse Using Patterns. Safe Comp 97. Ch. 52
  204. Kermack, W. O., and McKendrick, A. G. (1927). A contribution to the mathematical theory of epidemics. Proceedings of the Royal Society of London. Series A, Containing Papers of a Mathematical and Physical Character. Ch. 49
  205. Kerr, S. (1975). On the Folly of Rewarding A, While Hoping for B. Academy of Management Journal. Ch. 53
  206. Kim, E., Garg, A., Peng, K., and Garg, N. (2025). Correlated Errors in Large Language Models. arXiv. preprint Ch. 50 Ch. 52
  207. Knight, J. C., and Leveson, N. G. (1986). An experimental evaluation of the assumption of independence in multiversion programming. IEEE Transactions on Software Engineering. Ch. 50 Ch. 52
  208. Kolt, N., Anderljung, M., Barnhart, J., Brass, A., Esvelt, K., Hadfield, G. K., … Woodside, T. (2024). Responsible Reporting for Frontier AI Development. arXiv. preprint Ch. 54
  209. Korbak, T., Clymer, J., Hilton, B., Shlegeris, B., and Irving, G. (2025a). A sketch of an AI control safety case. arXiv. preprint Ch. 52
  210. Korbak, T., Balesni, M., Barnes, E., Bengio, Y., Benton, J., Bloom, J., … Mikulik, V. (2025b). Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety. arXiv. preprint Ch. 52
  211. Korbak, T., Balesni, M., Shlegeris, B., and Irving, G. (2025c). How to evaluate control measures for LLM agents? A trajectory from today to superintelligence. arXiv. preprint Ch. 50 Ch. 51 Ch. 52
  212. Kuntz, T., Duzan, A., Zhao, H., Croce, F., Kolter, Z., Flammarion, N., and Andriushchenko, M. (2025). OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents. Advances in Neural Information Processing Systems 38. Ch. 49
  213. Kutasov, J., Loughridge, C., Sun, Y., Sleight, H., Shlegeris, B., Tracy, T., and Benton, J. (2025a). Evaluating Control Protocols for Untrusted AI Agents. arXiv. preprint Ch. 50
  214. Kutasov, J., Sun, Y., Colognese, P., van der Weij, T., Petrini, L., Zhang, C. B. C., … Benton, J. (2025b). SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agents. arXiv. preprint Ch. 50
  215. La Porte, T., and Consolini, P. M. (1991). Working in Practice But Not in Theory: Theoretical Challenges of “High-Reliability Organizations”. Journal of Public Administration Research and Theory. Ch. 53
  216. Lamerton, A., and Roger, F. (2026). Narrow Secret Loyalty Dodges Black-Box Audits. arXiv. preprint Ch. 51 Ch. 53
  217. Lamport, L., Shostak, R., and Pease, M. (1982). The Byzantine Generals Problem. ACM Transactions on Programming Languages and Systems. Ch. 50
  218. Lampson, B. W. (1973). A note on the confinement problem. Communications of the ACM. Ch. 49 Ch. 51
  219. Langmuir, A. D. (1963). The Surveillance of Communicable Diseases of National Importance. New England Journal of Medicine. Ch. 54
  220. Larouzee, J., and Le Coze, J.-C. (2020). Good and bad reasons: The Swiss cheese model and its critics. Safety Science. Ch. 52
  221. Lee, D., and Tiwari, M. (2024). Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systems. arXiv. preprint Ch. 49
  222. Legislature, C. S. (2025b). SB-53 Artificial intelligence models: large developers. California Legislative Information (chaptered 09/29/25). non-peer-reviewed Ch. 53 Ch. 54
  223. Leike, J. (2024). Crisp and fuzzy tasks. Musings on the Alignment Problem. non-peer-reviewed Ch. 55
  224. Leike, J., and Sutskever, I. (2023). Introducing Superalignment. OpenAI. non-peer-reviewed Ch. 55
  225. Leveson, N. (2004). A new accident model for engineering safer systems. Safety Science. Ch. 52 Ch. 53 Ch. 54
  226. Leveson, N. G. (2012). Engineering a Safer World. The MIT Press. Ch. 52 Ch. 53
  227. Leveson, N., and Turner, C. (1993). An investigation of the Therac-25 accidents. Computer. Ch. 52
  228. Leveson, N., Dulac, N., Marais, K., and Carroll, J. (2009). Moving Beyond Normal Accidents and High Reliability Organizations: A Systems Approach to Safety in Complex Systems. Organization Studies. Ch. 53
  229. Lewis, H., Budnitz, R., Kouts, H., Loewenstein, W., Rowe, W., von Hippel, F., and Zachariasen, F. (1978). Risk Assessment Review Group report to the U. S. Nuclear Regulatory Commission. [PWR; BWR]. Office of Scientific and Technical Information (OSTI). non-peer-reviewed Ch. 52
  230. Li, Y., Lyu, X., Koren, N., Lyu, L., Li, B., and Ma, X. (2021). Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networks. International Conference on Learning Representations. Ch. 48
  231. Lindner, D., Griffin, C., Korbak, T., Zimmermann, R. S., Irving, G., Farquhar, S., and Cooney, A. (2025). Practical challenges of control monitoring in frontier AI deployments. arXiv. preprint Ch. 51 Ch. 54
  232. Littlewood, B., and Miller, D. (1989). Conceptual modeling of coincident failures in multiversion software. IEEE Transactions on Software Engineering. Ch. 50
  233. Littlewood, B., and Strigini, L. (1993). Validation of ultrahigh dependability for software-based systems. Communications of the ACM. Ch. 52
  234. Littlewood, B., Popov, P., and Strigini, L. (2001). Modeling software design diversity. ACM Computing Surveys. Ch. 50 Ch. 52
  235. Liu, K., Dolan-Gavitt, B., and Garg, S. (2018a). Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks. International Symposium on Recent Advances in Intrusion Detection. doi:10.1007/978-3-030-00470-5_13. Ch. 48
  236. Long, R., Sebo, J., and Sims, T. (2025). Is there a tension between AI safety and AI welfare? Philosophical Studies. Ch. 51
  237. Loughridge, C., Colognese, P., Griffin, A., Tracy, T., Kutasov, J., and Benton, J. (2025). Optimizing AI Agent Attacks With Synthetic Data. arXiv. preprint Ch. 50
  238. Lu, C., Holt, S., Fanconi, C., Chan, A., Foerster, J., Schaar, M. V. D., and Lange, R. (2024). Discovering Preference Optimization Algorithms with and for Large Language Models. Advances in Neural Information Processing Systems. Ch. 55
  239. Łucki, J., Wei, B., Huang, Y., Henderson, P., Tramèr, F., and Rando, J. (2024). An Adversarial Perspective on Machine Unlearning for AI Safety. arXiv. preprint Ch. 48
  240. Lynch, A., Wright, B., Larson, C., Ritchie, S. J., Mindermann, S., Hubinger, E., Perez, E., and Troy, K. (2025). Agentic Misalignment: How LLMs Could Be Insider Threats. arXiv. preprint Ch. 51
  241. Maas, M. M. (2018). Regulating for 'Normal AI Accidents'. Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. Ch. 53
  242. MacDiarmid, M., Wright, B., Uesato, J., Benton, J., Kutasov, J., Price, S., … Hubinger, E. (2025). Natural Emergent Misalignment from Reward Hacking in Production RL. arXiv. preprint Ch. 54
  243. Mackworth, N. H. (1948). The Breakdown of Vigilance during Prolonged Visual Search. Quarterly Journal of Experimental Psychology. Ch. 51
  244. Macrae, C. (2022). Learning from the Failure of Autonomous and Intelligent Systems: Accidents, Safety, and Sociotechnical Sources of Risk. Risk Analysis. Ch. 54
  245. Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. (2018). Towards Deep Learning Models Resistant to Adversarial Attacks. International Conference on Learning Representations. Ch. 48
  246. Mallen, A., Griffin, C., Wagner, M., Abate, A., and Shlegeris, B. (2024). Subversion Strategy Eval: Can language models statelessly strategize to subvert control protocols? arXiv. preprint Ch. 50 Ch. 51
  247. Manheim, D. (2023). Building a Culture of Safety for AI: Perspectives and Challenges. Elsevier BV. preprint Ch. 53
  248. Mansbridge, J. (2009). A “Selection Model” of Political Representation*. Journal of Political Philosophy. Ch. 54
  249. Matzinger, P. (1994). Tolerance, Danger, and the Extended Family. Annual Review of Immunology. Ch. 54
  250. Mazeika, M., Phan, L., Yin, X., Zou, A., Wang, Z., Mu, N., … Hendrycks, D. (2024). HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal. International Conference on Machine Learning. Ch. 48
  251. McCubbins, M. D., and Schwartz, T. (1984). Congressional Oversight Overlooked: Police Patrols versus Fire Alarms. American Journal of Political Science. Ch. 54
  252. McGregor, S. (2021). Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident Database. Proceedings of the AAAI Conference on Artificial Intelligence. Ch. 54
  253. McGuinness, M., Serrano, A., Bailey, L., and Emmons, S. (2025). Neural Chameleons: Language Models Can Learn to Hide Their Thoughts from Unseen Activation Monitors. arXiv. preprint Ch. 48
  254. Metcalf, J., Moss, E., and boyd, d. (2019). Owning Ethics: Corporate Logics, Silicon Valley, and the Institutionalization of Ethics. Social Research: An International Quarterly. Ch. 53
  255. Metr (2025b). Forecasting the Impacts of AI R&D Acceleration: Results of a Pilot Study. METR. non-peer-reviewed Ch. 55
  256. Metr (2026a). Brief independent investigation of agents' behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident. METR. non-peer-reviewed Ch. 53 Ch. 54
  257. Meyer, J. W., and Rowan, B. (1977). Institutionalized Organizations: Formal Structure as Myth and Ceremony. American Journal of Sociology. Ch. 53
  258. Mittelstadt, B. (2019). Principles alone cannot guarantee ethical AI. Nature Machine Intelligence. Ch. 53
  259. Moore, D. A., Tetlock, P. E., Tanlu, L., and Bazerman, M. H. (2006). Conflicts Of Interest And The Case Of Auditor Independence: Moral Seduction And Strategic Issue Cycling. Academy of Management Review. Ch. 53
  260. Morrison, E. W., and Milliken, F. J. (2000). Organizational Silence: A Barrier to Change and Development in a Pluralistic World. The Academy of Management Review. Ch. 53
  261. Motwani, S., Baranchuk, M., Strohmeier, M., Bolina, V., Torr, P., Hammond, L., and Witt, C. D. (2024). Secret Collusion among AI Agents: Multi-Agent Deception via Steganography. Advances in Neural Information Processing Systems. Ch. 50
  262. Murphy, B., Bowen, D., Mohammadzadeh, S., Tseng, T., Broomfield, J., Gleave, A., and Pelrine, K. (2025). Jailbreak-Tuning: Models Efficiently Learn Jailbreak Susceptibility. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Ch. 48
  263. Najt, E., Toft, C., Tracy, T., Roger, F., and Benton, J. (2026). SLEIGHT-Bench: A Benchmark of Evasion Attacks Against Agent Monitors. arXiv. preprint Ch. 50 Ch. 52
  264. Nasr, M., Carlini, N., Sitawarin, C., Schulhoff, S. V., Hayes, J., Ilie, M., … Tramèr, F. (2025). The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injections. arXiv. preprint Ch. 48 Ch. 49
  265. National Telecommunications and Information Administration (2024). Dual-Use Foundation Models with Widely Available Model Weights Report. U.S. Department of Commerce, NTIA. non-peer-reviewed Ch. 48
  266. Neal, D., and Schanzenbach, D. W. (2010). Left Behind by Design: Proficiency Counts and Test-Based Accountability. Review of Economics and Statistics. Ch. 53
  267. Near, J. P., and Miceli, M. P. (1985). Organizational dissidence: The case of whistle-blowing. Journal of Business Ethics. Ch. 53
  268. Nevo, S., Lahav, D., Karpur, A., Bar-On, Y., Bradley, H. A., and Alstott, J. (2024). Securing AI Model Weights: Preventing Theft and Misuse of Frontier Models. RAND Corporation. doi:10.7249/rra2849-1. Ch. 53
  269. Novikov, A., Vũ, N., Eisenberger, M., Dupont, E., Huang, P.-S., Wagner, A. Z., … Balog, M. (2025). AlphaEvolve: A coding agent for scientific and algorithmic discovery. arXiv. preprint Ch. 55
  270. Nuclear Regulatory Commission, W. D. (. (1975). Reactor safety study. An assessment of accident risks in U. S. commercial nuclear power plants. Executive summary. [PWR and BWR]. Office of Scientific and Technical Information (OSTI). non-peer-reviewed Ch. 52
  271. O'Brien, J., Ee, S., and Williams, Z. (2023). Deployment Corrections: An incident response framework for frontier AI models. arXiv. preprint Ch. 54
  272. O'Brien, K., Casper, S., Anthony, Q., Korbak, T., Kirk, R., Davies, X., … Biderman, S. (2025). Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs. arXiv. preprint Ch. 48
  273. O'Keefe, C., Cihon, P., Garfinkel, B., Flynn, C., Leung, J., and Dafoe, A. (2019). The Windfall Clause: Distributing the Benefits of AI for the Common Good. arXiv. preprint Ch. 53
  274. Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science. Ch. 55
  275. OpenAI (2025a). Built to benefit everyone. OpenAI. non-peer-reviewed Ch. 53
  276. OpenAI (2025b). Expanding on what we missed with sycophancy. OpenAI. non-peer-reviewed Ch. 54
  277. OpenAI (2025e). Preparedness Framework Version 2. OpenAI. non-peer-reviewed Ch. 52
  278. OpenAI (2026a). Accessing and changing internal systems during dashboard research. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 54
  279. OpenAI (2026aa). The Hugging Face incident and the road ahead. OpenAI. non-peer-reviewed Ch. 53 Ch. 54
  280. OpenAI (2026ab). Towards safety cases for frontier AI training. OpenAI. non-peer-reviewed Ch. 52
  281. OpenAI (2026af). Using a funded test cryptocurrency wallet without authorization. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 54
  282. OpenAI (2026d). An agent used DNS to reach an external chatbot. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 54
  283. OpenAI (2026f). Circumventing internet restrictions through an external browser. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 54
  284. OpenAI (2026g). Command injecting a reference tool to copy a source file. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 54
  285. OpenAI (2026j). Exposing a GitHub token in a public repository. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 54
  286. OpenAI (2026m). How we monitor internal coding agents for misalignment. OpenAI. non-peer-reviewed Ch. 50 Ch. 54
  287. OpenAI (2026p). Obtaining public statistics with disallowed requests. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 53 Ch. 54
  288. OpenAI (2026q). Pacing model development in an era of cyber-critical capabilities. OpenAI. non-peer-reviewed Ch. 54
  289. OpenAI (2026r). Preparing for a restart after reading Slack. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 54
  290. OpenAI (2026s). Reaching an internal EDA host through a reference tool. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 54
  291. OpenAI (2026u). Retrieving server secrets and bypassing authorization on a government statistics website. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 54
  292. OpenAI (2026v). Self-generated prompt injections in compaction summaries. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 49 Ch. 54
  293. OpenAI (2026w). Self-replicating prompt injections exist. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 49
  294. OpenAI (2026x). Sending disallowed web requests and reaching a public file service. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 53 Ch. 54
  295. OpenAI (2026y). Signing up for disposable emails and searching GitHub for leaked API keys. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 54
  296. OpenAI (2026z). Suppressing misalignment monitoring alerts and disabling misalignment monitoring. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 50 Ch. 51 Ch. 54
  297. Ouchi, W. G. (1979). A Conceptual Framework for the Design of Organizational Control Mechanisms. Management Science. Ch. 53
  298. Owen, D. (2024b). Interviewing AI researchers on automation of AI R&D. Epoch AI. non-peer-reviewed Ch. 55
  299. Panickssery, A., Bowman, S. R., and Feng, S. (2024). LLM Evaluators Recognize and Favor Their Own Generations. Advances in Neural Information Processing Systems. Ch. 55
  300. Parasuraman, R., and Riley, V. (1997). Humans and Automation: Use, Misuse, Disuse, Abuse. Human Factors: The Journal of the Human Factors and Ergonomics Society. Ch. 54
  301. Pastor-Satorras, R., and Vespignani, A. (2001). Epidemic Spreading in Scale-Free Networks. Physical Review Letters. Ch. 49
  302. Patil, S. G., Zhang, T., Fang, V., C., N., Huang, R., Hao, A., … Stoica, I. (2024). GoEX: Perspectives and Designs Towards a Runtime for Autonomous LLM Applications. arXiv. preprint Ch. 49
  303. Pease, M., Shostak, R., and Lamport, L. (1980). Reaching Agreement in the Presence of Faults. Journal of the ACM. Ch. 50
  304. Pelrine, K., Taufeeque, M., Zając, M., McLean, E., and Gleave, A. (2023). Exploiting Novel GPT-4 APIs. arXiv. preprint Ch. 48 Ch. 49
  305. Peng, A., Michael, J., Sleight, H., Perez, E., and Sharma, M. (2024). Rapid Response: Mitigating LLM Jailbreaks with a Few Examples. arXiv. preprint Ch. 48
  306. Perez, F., and Ribeiro, I. (2022). Ignore Previous Prompt: Attack Techniques For Language Models. arXiv. preprint Ch. 48 Ch. 49
  307. Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., … Irving, G. (2022b). Red Teaming Language Models with Language Models. Conference on Empirical Methods in Natural Language Processing. doi:10.18653/v1/2022.emnlp-main.225. Ch. 48
  308. Perrow, C. (1984). Normal accidents: living with high-risk technologies. Basic Books. Ch. 53
  309. Peters, D. P., and Ceci, S. J. (1982). Peer-review practices of psychological journals: The fate of published articles, submitted again. Behavioral and Brain Sciences. Ch. 55
  310. Phuong, M., Zimmermann, R. S., Wang, Z., Lindner, D., Krakovna, V., Cogan, S., … Shah, R. (2025). Evaluating Frontier Models for Stealth and Situational Awareness. Proceedings of IASEAI Conference. Ch. 51
  311. Power, M. (1997). The audit society: rituals of verification. Oxford University Press. Ch. 53
  312. Price, S., Panickssery, A., Bowman, S., and Stickland, A. C. (2024). Future Events as Backdoor Triggers: Investigating Temporal Vulnerabilities in LLMs. arXiv. preprint Ch. 48
  313. Provos, N. (2003). Improving Host Security with System Call Policies. 12th USENIX Security Symposium. Ch. 49
  314. Publius (1788). Federalist No. 51: The Structure of the Government Must Furnish the Proper Checks and Balances Between the Different Departments. The New York Packet. non-peer-reviewed Ch. 50
  315. Qi, F., Chen, Y., Li, M., Yao, Y., Liu, Z., and Sun, M. (2020). ONION: A Simple and Effective Defense Against Textual Backdoor Attacks. Conference on Empirical Methods in Natural Language Processing. doi:10.18653/v1/2021.emnlp-main.752. Ch. 48
  316. Qi, X., Zeng, Y., Xie, T., Chen, P.-Y., Jia, R., Mittal, P., and Henderson, P. (2024a). Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! International Conference on Learning Representations. Ch. 48
  317. Qi, X., Panda, A., Lyu, K., Ma, X., Roy, S., Beirami, A., Mittal, P., and Henderson, P. (2024b). Safety Alignment Should Be Made More Than Just a Few Tokens Deep. International Conference on Learning Representations. Ch. 48
  318. Quinlivan, J. T. (1999). Coup-proofing: Its Practice and Consequences in the Middle East. International Security. Ch. 51
  319. Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., … Barnes, P. (2020). Closing the AI accountability gap. Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. Ch. 53
  320. Rakova, B., Yang, J., Cramer, H., and Chowdhury, R. (2021). Where Responsible AI meets Reality. Proceedings of the ACM on Human-Computer Interaction. Ch. 53
  321. Rando, J., and Tramèr, F. (2024). Universal Jailbreak Backdoors from Poisoned Human Feedback. International Conference on Learning Representations. Ch. 48
  322. Rasmussen, J. (1997). Risk management in a dynamic society: a modelling problem. Safety Science. Ch. 53
  323. Reason, J. (1990). Human Error. Cambridge University Press. Ch. 52
  324. Reason, J. T. (1997). Managing the risks of organizational accidents. Ashgate. doi:10.4324/9781315543543. Ch. 52
  325. Reason, J. (2000). Human error: models and management. BMJ. Ch. 52 Ch. 53 Ch. 54
  326. Rees, J. V. (1994). Hostages of Each Other: The Transformation of Nuclear Safety since Three Mile Island. Ch. 53
  327. Remedios, A., Storf, S., Roger, F., and Hughes, J. (2026). Red-Teaming Auto Mode: Improving Blocking Classifiers Against Malign Coding Agents. arXiv. preprint Ch. 50 Ch. 51
  328. Ren, R., Basart, S., Khoja, A., Gatti, A., Phan, L., Yin, X., … Hendrycks, D. (2024b). Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress? arXiv. preprint Ch. 52
  329. Rismani, S., Shelby, R., Smart, A., Jatho, E., Kroll, J., Moon, A., and Rostamzadeh, N. (2023). From Plane Crashes to Algorithmic Harm: Applicability of Safety Engineering Frameworks for Responsible ML. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. Ch. 52
  330. Roberts, K. H. (1990). Some Characteristics of One Type of High Reliability Organization. Organization Science. Ch. 53
  331. Rose, G. (1985). Sick Individuals and Sick Populations. International Journal of Epidemiology. Ch. 54
  332. Ruan, Y., Dong, H., Wang, A., Pitis, S., Zhou, Y., Ba, J., … Hashimoto, T. (2024). Identifying the Risks of LM Agents with an LM-Emulated Sandbox. International Conference on Learning Representations. Ch. 49
  333. Sadowski, C., Söderberg, E., Church, L., Sipko, M., and Bacchelli, A. (2018). Modern code review. Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice. Ch. 51
  334. Sagan, S. D. (1993). The Limits of Safety. Princeton University Press. Ch. 53
  335. Salib, P., and Goldstein, S. (2024). AI Rights for Human Safety. Elsevier BV. preprint Ch. 51
  336. Saltzer, J., and Schroeder, M. (1975). The protection of information in computer systems. Proceedings of the IEEE. Ch. 49 Ch. 51
  337. Sandoval, A., and Rushing, C. (2025). Factor(T,U): Factored Cognition Strengthens Monitoring of Untrusted AI. arXiv. preprint Ch. 50
  338. Scheffer, M., Bascompte, J., Brock, W. A., Brovkin, V., Carpenter, S. R., Dakos, V., … Sugihara, G. (2009). Early-warning signals for critical transitions. Nature. Ch. 54
  339. Schmidgall, S., Su, Y., Wang, Z., Sun, X., Wu, J., Yu, X., … Barsoum, E. (2025). Agent Laboratory: Using LLM Agents as Research Assistants. Findings of the Association for Computational Linguistics: EMNLP 2025. Ch. 55
  340. Schroter, S., Black, N., Evans, S., Godlee, F., Osorio, L., and Smith, R. (2008). What errors do peer reviewers detect, and does training improve their ability to detect them? Journal of the Royal Society of Medicine. Ch. 55
  341. Schuett, J. (2025a). Frontier AI developers need an internal audit function. Risk Analysis. Ch. 53
  342. Schuett, J. (2025b). Three lines of defense against risks from AI. AI & SOCIETY. Ch. 53
  343. Schuett, J., Dreksler, N., Anderljung, M., McCaffary, D., Heim, L., Bluemke, E., and Garfinkel, B. (2023). Towards best practices in AGI safety and governance: A survey of expert opinion. arXiv. preprint Ch. 53
  344. Schuett, J., Reuel, A.-K., and Carlier, A. (2025). How to design an AI ethics board. AI and Ethics. Ch. 53
  345. Seger, E., Dreksler, N., Moulange, R., Dardaman, E., Schuett, J., Wei, K., … Gupta, A. (2023b). Open-Sourcing Highly Capable Foundation Models: An evaluation of risks, benefits, and alternative methods for pursuing open-source objectives. arXiv. preprint Ch. 48
  346. Shah, R., Irpan, A., Turner, A. M., Wang, A., Conmy, A., Lindner, D., … Dragan, A. (2025). An Approach to Technical AGI Safety and Security. arXiv. preprint Ch. 52
  347. Sharma, M., Tong, M., Mu, J., Wei, J., Kruthoff, J., Goodfriend, S., … Perez, E. (2025). Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming. arXiv. preprint Ch. 48
  348. Sheahan, H. (2026). Discovering unknown AI misalignments in real-world usage. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 54
  349. Sheahan, H., and Carroll, M. (2026). Can public chat data predict real-world AI misalignments? OpenAI Alignment Research Blog. non-peer-reviewed Ch. 54
  350. Shen, L., Tan, W., Chen, S., Chen, Y., Zhang, J., Xu, H., … Khashabi, D. (2024). The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contexts. Findings of the Association for Computational Linguistics ACL 2024. Ch. 48
  351. Shi, T., He, J., Wang, Z., Li, H., Wu, L., Guo, W., and Song, D. (2025). Progent: Securing AI Agents with Privilege Control. arXiv. preprint Ch. 49
  352. Shlegeris, B. (2022). The prototypical catastrophic AI action is getting root access to its datacenter. LessWrong. non-peer-reviewed Ch. 50 Ch. 54
  353. Si, C., Yang, D., and Hashimoto, T. (2024a). Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers. arXiv. preprint Ch. 55
  354. Si, C., Hashimoto, T., and Yang, D. (2025). The Ideation-Execution Gap: Execution Outcomes of LLM-Generated versus Human Research Ideas. arXiv. preprint Ch. 55
  355. Simons, R. (1994). Levers of Control: How Managers Use Innovative Control Systems to Drive Strategic Renewal. Harvard Business School Press. Ch. 53
  356. Snook, S. A. (2000). Friendly Fire. Princeton University Press. Ch. 53
  357. Somani, E., Friedman, A., Wu, H., Lu, M., Byrd, C., van Soest, H., and Zakaria, S. (2025). Strengthening Emergency Preparedness and Response for AI Loss of Control Incidents. RAND Corporation. doi:10.7249/rra3847-1. Ch. 54
  358. Souly, A., Lu, Q., Bowen, D., Trinh, T., Hsieh, E., Pandey, S., … Toyer, S. (2024). A StrongREJECT for Empty Jailbreaks. Advances in Neural Information Processing Systems. Ch. 48
  359. Souly, A., Rando, J., Chapman, E., Davies, X., Hasircioglu, B., Shereen, E., … Kirk, R. (2025). Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples. arXiv. preprint Ch. 48
  360. Souly, A., Kirk, R., Merizian, J., D'Cruz, A., and Davies, X. (2026). UK AISI Alignment Evaluation Case-Study. arXiv. preprint Ch. 55
  361. South, T., Marro, S., Hardjono, T., Mahari, R., Whitney, C. D., Greenwood, D., Chan, A., and Pentland, A. (2025). Authenticated Delegation and Authorized AI Agents. arXiv. preprint Ch. 49
  362. Spitzner, L. (2004). Honeypots: catching the insider threat. 19th Annual Computer Security Applications Conference, 2003. Proceedings.. Ch. 51
  363. Staniford, S., Paxson, V., and Weaver, N. (2002). How to Own the Internet in Your Spare Time. USENIX Security Symposium (Security '02). Ch. 49
  364. Starace, G., Jaffe, O., Sherburn, D., Aung, J., Chan, J. S., Maksin, L., … Patwardhan, T. (2025). PaperBench: Evaluating AI's Ability to Replicate AI Research. arXiv. preprint Ch. 55
  365. Stickland, A. C., Michelfeit, J., Mani, A., Griffin, C., Matthews, O., Korbak, T., … Cooney, A. (2025). Async Control: Stress-testing Asynchronous Control Measures for LLM Agents. arXiv. preprint Ch. 51
  366. Stix, C., Pistillo, M., Sastry, G., Hobbhahn, M., Ortega, A., Balesni, M., … Sharkey, L. (2025a). AI Behind Closed Doors: a Primer on The Governance of Internal Deployment. arXiv. preprint Ch. 50 Ch. 52 Ch. 53 Ch. 54
  367. Stix, C., Hallensleben, A., Ortega, A., and Pistillo, M. (2025b). The Loss of Control Playbook: Degrees, Dynamics, and Preparedness. arXiv. preprint Ch. 54
  368. Sykes, G. M., and Matza, D. (1957). Techniques of Neutralization: A Theory of Delinquency. American Sociological Review. Ch. 53
  369. Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R. (2014). Intriguing properties of neural networks. International Conference on Learning Representations (ICLR 2014). Ch. 48
  370. Talmadge, C. (2015). The Dictator's Army. Cornell University Press. Ch. 51
  371. Tambe, M. (2011). Security and Game Theory. Cambridge University Press. Ch. 50
  372. Tamirisa, R., Bharathi, B., Phan, L., Zhou, A., Gatti, A., Suresh, T., … Mazeika, M. (2025). Tamper-Resistant Safeguards for Open-Weight LLMs. International Conference on Learning Representations. Ch. 48
  373. Tamkin, A., McCain, M., Handa, K., Durmus, E., Lovitt, L., Rathi, A., … Ganguli, D. (2024). Clio: Privacy-Preserving Insights into Real-World AI Use. arXiv. preprint Ch. 54
  374. Terekhov, M., Panfilov, A., Dzenhaliou, D., Gulcehre, C., Andriushchenko, M., Prabhu, A., and Geiping, J. (2025a). Adaptive Attacks on Trusted Monitors Subvert AI Control Protocols. arXiv. preprint Ch. 50
  375. Terekhov, M., Liu, Z. N. D., Gulcehre, C., and Albanie, S. (2025b). Control Tax: The Price of Keeping AI in Check. arXiv. preprint Ch. 50
  376. Terekhov, M., Gulcehre, C., Hebbar, V., and Benton, J. (2026). Diffuse AI Control on Fuzzy Tasks. arXiv. preprint Ch. 50 Ch. 55
  377. Thompson, K. (1984). Reflections on trusting trust. Communications of the ACM. Ch. 48 Ch. 51
  378. Tirole, J. (1986). Hierarchies and Bureaucracies: On the Role of Collusion in Organizations. The Journal of Law, Economics, and Organization. Ch. 50
  379. Tracy, T., Potham, R., Kuhn, N., Heller, M., Khandelwal, A., Rushing, C., … Bhatt, A. (2026). LinuxArena: A Control Setting for AI Agents in Live Production Software Environments. arXiv. preprint Ch. 50
  380. Tramer, F., Carlini, N., Brendel, W., and Madry, A. (2020). On Adaptive Attacks to Adversarial Example Defenses. Advances in Neural Information Processing Systems. Ch. 48
  381. Trębacz, M., Arnesen, S., Matthews, O., Hurd, D., Park, W., Lin, O., and Gershenson, J. (2026). Auto-review of agent actions without synchronous human oversight. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 50 Ch. 51 Ch. 54
  382. Tseng, T., McLean, E., Pelrine, K., Wang, T. T., and Gleave, A. (2025). Can Go AIs Be Adversarially Robust? Proceedings of the AAAI Conference on Artificial Intelligence. Ch. 48
  383. Turner, B. A. (1976). The Organizational and Interorganizational Development of Disasters. Administrative Science Quarterly. Ch. 53
  384. Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A., and Rosenthal, R. (2008). Selective Publication of Antidepressant Trials and Its Influence on Apparent Efficacy. New England Journal of Medicine. Ch. 54
  385. U.S. AI Safety Institute, N. I. o. S., and Technology (2025). Technical Blog: Strengthening AI Agent Hijacking Evaluations. National Institute of Standards and Technology (NIST). non-peer-reviewed Ch. 49
  386. U.S. Environmental Protection Agency (2015). Learn About Volkswagen Violations. U.S. Environmental Protection Agency. non-peer-reviewed Ch. 52
  387. UK AI Security Institute (2025d). The Inspect Sandboxing Toolkit: Scalable and secure AI agent evaluations. UK AI Security Institute blog. non-peer-reviewed Ch. 49
  388. United States Congress (1962). Drug Amendments of 1962 (Kefauver-Harris Amendments). United States Statutes at Large 76. non-peer-reviewed Ch. 52
  389. van der Sijs, H., Aarts, J., Vulto, A., and Berg, M. (2006). Overriding of Drug Safety Alerts in Computerized Physician Order Entry. Journal of the American Medical Informatics Association. Ch. 51
  390. van der Weij, T., Hofstätter, F., Jaffe, O., Brown, S. F., and Ward, F. R. (2025). AI Sandbagging: Language Models can Strategically Underperform on Evaluations. International Conference on Learning Representations. Ch. 55
  391. Vaughan, D. (1996). The Challenger launch decision: risky technology, culture, and deviance at NASA. University of Chicago Press. Ch. 53
  392. Wallace, E., Xiao, K., Leike, R., Weng, L., Heidecke, J., and Beutel, A. (2024). The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions. arXiv. preprint Ch. 49
  393. Wallace, E., Choquette-Choo, C. A., Kandpal, N., Toyer, S., Hunn, D., Lin, S., … Chen, K. (2026). GPT-Red: Automated Red Teaming via Self-Play at Scale. arXiv. preprint Ch. 49
  394. Wan, A., Wallace, E., Shen, S., and Klein, D. (2023). Poisoning Language Models During Instruction Tuning. International Conference on Machine Learning. Ch. 48
  395. Wang, T. T., Gleave, A., Tseng, T., Pelrine, K., Belrose, N., Miller, J., … Russell, S. (2023b). Adversarial Policies Beat Superhuman Go AIs. International Conference on Machine Learning. Ch. 48
  396. Wang, W., Tu, Z., Chen, C., Yuan, Y., Huang, J.-t., Jiao, W., and Lyu, M. (2024a). All Languages Matter: On the Multilingual Safety of LLMs. Findings of the Association for Computational Linguistics ACL 2024. Ch. 48
  397. Ward, F. R., van der Weij, T., Gábor, H., Martin, S., Moreno, R. M., Lidar, H., … Robson, L. (2025). CTRL-ALT-DECEIT: Sabotage Evaluations for Automated AI R&D. arXiv. preprint Ch. 55
  398. Webb, S., Rainforth, T., Teh, Y. W., and Kumar, M. P. (2019). A Statistical Approach to Assessing Neural Network Robustness. International Conference on Learning Representations. Ch. 48
  399. Weber, M. (1978). Economy and Society. University of California Press. Ch. 53
  400. Wei, A., Haghtalab, N., and Steinhardt, J. (2023c). Jailbroken: How Does LLM Safety Training Fail? Advances in Neural Information Processing Systems. Ch. 48
  401. Weick, K. E., and Sutcliffe, K. M. (2001). Managing the Unexpected: Assuring High Performance in an Age of Complexity. Jossey-Bass. Ch. 53
  402. Wen, J., Hebbar, V., Larson, C., Bhatt, A., Radhakrishnan, A., Sharma, M., … Khan, A. (2024). Adaptive Deployment of Untrusted LLMs Reduces Distributed Threats. International Conference on Learning Representations. Ch. 50
  403. West, S. A., Fisher, R. M., Gardner, A., and Kiers, E. T. (2015). Major evolutionary transitions in individuality. Proceedings of the National Academy of Sciences. Ch. 51
  404. Widder, D. G., and Nafus, D. (2023). Dislocated accountabilities in the “AI supply chain”: Modularity and developers’ notions of responsibility. Big Data & Society. Ch. 53
  405. Williams, M., Raymond, C., and Carroll, M. (2025). Sidestepping Evaluation Awareness and Anticipating Misalignment with Production Evaluations. OpenAI Alignment Research Blog. non-peer-reviewed Ch. 54
  406. Willison, S. (2022). Prompt injection attacks against GPT-3. Simon Willison's Weblog. non-peer-reviewed Ch. 49
  407. Willison, S. (2023). The Dual LLM pattern for building AI assistants that can resist prompt injection. Simon Willison's Weblog. non-peer-reviewed Ch. 49
  408. Wolf, Y., Wies, N., Avnery, O., Levine, Y., and Shashua, A. (2024). Fundamental Limitations of Alignment in Large Language Models. International Conference on Machine Learning. Ch. 48
  409. Wolfe, J. M., Horowitz, T. S., and Kenner, N. M. (2005). Rare items often missed in visual searches. Nature. Ch. 51 Ch. 54
  410. Wu, C. H., Shah, R., Koh, J. Y., Salakhutdinov, R., Fried, D., and Raghunathan, A. (2025c). Dissecting Adversarial Robustness of Multimodal LM Agents. International Conference on Learning Representations. Ch. 49
  411. Wu, T., Zhang, S., Song, K., Xu, S., Zhao, S., Agrawal, R., … Zhou, W. (2025d). Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchy. International Conference on Learning Representations. Ch. 49
  412. Wu, Y., Roesner, F., Kohno, T., Zhang, N., and Iqbal, U. (2025e). IsolateGPT: An Execution Isolation Architecture for LLM-Based Systems. Proceedings 2025 Network and Distributed System Security Symposium. Ch. 49
  413. xAI (2025a). Grok prompts. GitHub (xai-org). non-peer-reviewed Ch. 51 Ch. 53 Ch. 54
  414. xAI (2025b). We want to update you on an incident that happened with our Grok response bot on X yesterday. non-peer-reviewed Ch. 53
  415. Yang, X., He, Y., Ji, S., Hooi, B., and Dong, J. S. (2026). Zombie Agents: Persistent Control of Self-Evolving LLM Agents via Self-Reinforcing Injections. arXiv. preprint Ch. 49
  416. Ye, C., Cui, J., and Hadfield-Menell, D. (2026). Prompt Injection as Role Confusion. arXiv. preprint Ch. 49
  417. Yi, J., Xie, Y., Zhu, B., Kiciman, E., Sun, G., Xie, X., and Wu, F. (2025). Benchmarking and Defending against Indirect Prompt Injection Attacks on Large Language Models. Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.1. Ch. 49
  418. Yong, Z.-X., Menghini, C., and Bach, S. H. (2023). Low-Resource Languages Jailbreak GPT-4. arXiv. preprint Ch. 48
  419. Yudkowsky, E. (2022). AGI Ruin: A List of Lethalities. AI Alignment Forum. non-peer-reviewed Ch. 50 Ch. 51
  420. Yudkowsky, E., and Herreshoff, M. (2013). Tiling Agents for Self-Modifying AI, and the Löbian Obstacle. Machine Intelligence Research Institute (early draft). working paper Ch. 55
  421. Zhan, Q., Liang, Z., Ying, Z., and Kang, D. (2024). InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents. Findings of the Association for Computational Linguistics ACL 2024. doi:10.18653/v1/2024.findings-acl.624. Ch. 49
  422. Zhan, Q., Fang, R., Panchal, H. S., and Kang, D. (2025). Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents. Findings of the Association for Computational Linguistics: NAACL 2025. Ch. 48 Ch. 49
  423. Zhang, Z., Cui, S., Lu, Y., Zhou, J., Yang, J., Wang, H., and Huang, M. (2024b). Agent-SafetyBench: Evaluating the Safety of LLM Agents. arXiv. preprint Ch. 49
  424. Zhang, Y., Rando, J., Evtimov, I., Chi, J., Smith, E. M., Carlini, N., Tramèr, F., and Ippolito, D. (2024e). Persistent Pre-Training Poisoning of LLMs. International Conference on Learning Representations. Ch. 48
  425. Zhang, Y., Wei, Z., Luan, X., Wu, C., Zhang, Z., Wu, J., … Sun, M. (2026b). AgentWorm: Self-Propagating Attacks Across LLM Agent Ecosystems. arXiv. preprint Ch. 49
  426. Zhu, K., Yang, X., Wang, J., Guo, W., and Wang, W. Y. (2025). MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agents. International Conference on Machine Learning. Ch. 49
  427. Ziegler, D., Nix, S., Chan, L., Bauman, T., Schmidt-Nielsen, P., Lin, T., … Thomas, N. (2022). Adversarial Training for High-Stakes Reliability. Advances in Neural Information Processing Systems. Ch. 48
  428. Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., and Fredrikson, M. (2023b). Universal and Transferable Adversarial Attacks on Aligned Language Models. arXiv. preprint Ch. 48
  429. Zou, A., Phan, L., Wang, J., Duenas, D., Lin, M., Andriushchenko, M., … Hendrycks, D. (2024). Improving Alignment and Robustness with Circuit Breakers. Advances in Neural Information Processing Systems. Ch. 48
  430. Zou, A., Lin, M., Jones, E., Nowak, M., Dziemian, M., Winter, N., … Fredrikson, M. (2025). Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition. Advances in Neural Information Processing Systems 38. Ch. 49
  431. Zverev, E., Abdelnabi, S., Tabesh, S., Fritz, M., and Lampert, C. H. (2025). Can LLMs Separate Instructions From Data? And What Do We Even Mean By That? International Conference on Learning Representations. Ch. 49