Draft complete but not yet reviewed; expect revisions

Reading Map

The bibliography holds 2910 works. 558 of them form the spine: the works a reader needs to follow the argument, discussed in their own sections. The rest are side reading: support for a specific claim, or depth on one question. How a work is assigned to the spine is set out in the plan of the book; the short version is that a work is spine when many of the collected laboratory works cite it, or when it founds a concept the book uses throughout, or when it is the standard review of a field's evidence on an alignment question.

Machine learning works are grouped by the part of the book they belong to: by the chapters that cite them, then by the chapter the researcher who found them assigned, and otherwise by the topic their title names. Works from other fields are grouped by field, and the frontier laboratories' own publications come last.

The line on each work is the description recorded when the work was collected, from its record and abstract; it is a guide to what the work is about, not a claim the book checked against the text. Where a chapter could read a work only as a catalog record, its line says so instead.

F.1 Machine learning: The Problem #

Spine.

• What’s so special about model organisms? (Ankeny and Leonelli, 2011). Read for this book only as a catalog record, so no chapter reports what it found.

Side reading.

• On the Dangers of Stochastic Parrots (Bender et al., 2021).
• Ethical and social risks of harm from Language Models (Weidinger et al., 2021).
• Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting (Sclar et al., 2023). Measures how much language models' benchmark accuracy changes with meaning-preserving changes to prompt format.
• How to Replicate and Extend our Alignment Faking Demo (Anthropic, 2024j). We describe how to get started with experimenting with our demonstration of alignment faking, and present some ideas for future research.
• Putting up Bumpers (Bowman, 2025). Even if we can't solve alignment, we can solve the problem of catching and fixing misalignment.
• Recommendations for Technical AI Safety Research Directions (team et al., 2025). A collection of technical AI safety research problems that we'd like to see progress in.
• Intelligence Explosion Microeconomics (Yudkowsky, 2013). States the returns-on-cognitive-reinvestment question that recursive self-improvement arguments turn on and proposes how to answer it from data, the founding statement of the economics of intelligence explosion.
• Viewpoint: When Will AI Exceed Human Performance? Evidence from AI Experts (Grace et al., 2018). The first large survey of machine learning researchers on timelines for human-level performance, with large framing effects that remain a caution on expert forecasts.
• Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive? (Schaeffer et al., 2025). Traces the unpredictability of downstream metrics to the transformations from log-likelihood to accuracy, which discard the probability mass on incorrect answers.
• Human Control: Definitions and Algorithms (Carey and Everitt, 2023). Formal definitions of shutdown instructability and related notions, showing which guarantee human benefit and which do not.
• AI and the Everything in the Whole Wide World Benchmark (Raji et al., 2021). Critiques general-capability benchmarks for construct validity, arguing that they cannot measure the general abilities they are taken to measure.
• AGI Safety Literature Review (Everitt et al., 2018). The first systematic review of AGI safety research, organizing the pre-2018 literature on value specification, reliability, corrigibility, and safe exploration before the large language model turn.
• Research Agenda (UK AI Security Institute, 2025c). The Alignment Project agenda poses alignment as eleven discipline-specific question sets (cryptography, complexity theory, game theory, learning theory, cognitive science, and others), a map of which outside fields can carry which subproblem.
• An Alien Mind (OpenAI, 2026e). Jakub Pachocki reflects on increasingly capable AI and the challenge of keeping it aligned. He calls for stronger safeguards and international coordination.
• Unsolved Problems in ML Safety (Hendrycks et al., 2021c). Organizes ML safety into robustness, monitoring, alignment, and systemic safety, the taxonomy most non-lab safety research has used since.
• Foundational Challenges in Assuring Alignment and Safety of Large Language Models (Anwar et al., 2024). Catalogs eighteen foundational challenges and more than two hundred concrete research questions across scientific understanding, development methods, and sociotechnical issues for LLM alignment.

F.2 Machine learning: Foundations #

Spine.

• If Nothing Goes Wrong, Is Everything All Right? (Hanley, 1983). Read for this book only as a catalog record, so no chapter reports what it found.
• Learning representations by back-propagating errors (Rumelhart et al., 1986). Read for this book only as a catalog record, so no chapter reports what it found.
• Simple statistical gradient-following algorithms for connectionist reinforcement learning (Williams, 1992). The REINFORCE estimator, the basis of policy-gradient training.
• Constrained Markov Decision Processes (Altman, 1999). The monograph that formalizes optimization of one objective subject to constraints on others in Markov decision processes, the mathematical basis of constrained and safe reinforcement learning.
• Policy invariance under reward transformations: Theory and application to reward shaping (Ng et al., 1999). Proves that potential-based shaping is the only additive reward transformation that preserves optimal policies in every MDP, the cleanest case of rewards that differ yet specify the same behavior.
• The Optimizer’s Curse: Skepticism and Postdecision Surprise in Decision Analysis (Smith and Winkler, 2006). Proves that choosing the option with the highest estimated value makes the chosen estimate biased upward even when every estimate is unbiased, the decision-analysis form of regressional Goodhart.
• Algebraic Geometry and Statistical Learning Theory (Watanabe, 2009). Founds singular learning theory: for models whose parameter-to-distribution map is not one-to-one, as in neural networks, the learning coefficient from algebraic geometry replaces parameter count in the asymptotics of Bayesian generalization.
• Understanding deep learning requires rethinking generalization (Zhang et al., 2016). Shows that standard networks fit random labels perfectly, so capacity-based bounds cannot explain why they generalize on real labels; the question of which solution training picks starts here.
• Constrained Policy Optimization (Achiam et al., 2017). The first general policy-search algorithm for deep reinforcement learning with near-constraint satisfaction guarantees at every iteration, which made constrained Markov decision processes practical for neural policies.
• Deep reinforcement learning from human preferences (Christiano et al., 2017). Trains a reward model from pairwise human comparisons of trajectory segments and optimizes a deep RL policy against it, the method that became RLHF.
• Reinforcement Learning: An Introduction (Sutton and Barto, 2018). The standard textbook of reinforcement learning, second edition.
• Reconciling modern machine-learning practice and the classical bias–variance trade-off (Belkin et al., 2019). Names double descent: test error falls again past the interpolation threshold, so the classical U-shaped bias-variance curve does not describe overparameterized models.
• Deep learning generalizes because the parameter-function map is biased towards simple functions (Valle-Pérez et al., 2019). Argues from algorithmic information theory and experiments that random parameters map to simple functions with exponentially higher probability, an inductive-bias account that the counting-argument debate over scheming turns on.
• Scaling Laws for Neural Language Models (Kaplan et al., 2020). Cited by 13 laboratory works.
• Learning to summarize from human feedback (Stiennon et al., 2020). Shows RLHF summaries preferred over supervised baselines and documents reward model overoptimization as the KL budget grows.
• A Mathematical Framework for Transformer Circuits (Elhage et al., 2021a). Our early mathematical framework for reverse engineering models, demonstrated by reverse engineering small toy models.
• LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021). Cited by 5 laboratory works.
• Show Your Work: Scratchpads for Intermediate Computation with Language Models (Nye et al., 2021). Cited by 6 laboratory works.
• Constitutional AI: Harmlessness from AI Feedback (Bai et al., 2022a). We introduce Constitutional AI, allowing us to give language models explicit values determined by a constitution, rather than values determined implicitly via large-scale human feedback.
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback (Bai et al., 2022b). We train a natural language assistant to be more helpful and harmless by using reinforcement learning from human feedback (RLHF).
• Scaling Laws for Reward Model Overoptimization (Gao et al., 2022). Cited by 6 laboratory works.
• Language Models (Mostly) Know What They Know (Kadavath et al., 2022). We show that language models can evaluate whether what they say is true, and predict ahead of time whether they'll be able to answer questions correctly.
• RL with KL penalties is better viewed as Bayesian inference (Korbak et al., 2022). Shows that maximizing reward minus a KL penalty to the pretrained model is variational inference toward a posterior with the base model as prior and exp(r/beta) as likelihood, so fine-tuning reweights rather than replaces what pretraining learned.
• In-Context Learning and Induction Heads (Olsson et al., 2022). An exploration of the hypothesis that induction heads are the primary mechanism behind in-context learning. We also report the existence of a previously unknown phase change in transformers language models.
• Training language models to follow instructions with human feedback (Ouyang et al., 2022). Cited by 16 laboratory works.
• Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets (Power et al., 2022). Cited by 5 laboratory works.
• Compute Trends Across Three Eras of Machine Learning (Sevilla et al., 2022). Measures training compute of milestone systems and finds a doubling time near six months after 2010, the empirical basis for forecasting capability from compute.
• Learning by Distilling Context (Snell et al., 2022). Cited by 5 laboratory works.
• Solving math word problems with process- and outcome-based feedback (Uesato et al., 2022). Compares process- and outcome-based supervision on GSM8K and finds that outcome supervision reaches similar final-answer error with less labeling, while correct reasoning steps need process supervision or a reward model that emulates it (reasoning error among correct answers fell from 14.0 to 3.4 percent).
• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (Wei et al., 2022a). Cited by 15 laboratory works.
• Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (Casper et al., 2023). Taxonomy of RLHF failures split into tractable and fundamental problems with human feedback, the reward model, and the policy.
• Studying Large Language Model Generalization with Influence Functions (Grosse et al., 2023). This paper scales influence functions to 52B parameter models, revealing how generalization patterns become more abstract with model size.
• The Local Learning Coefficient: A Singularity-Aware Complexity Measure (Lau et al., 2023). Defines the local learning coefficient and an SGLD estimator for it, the one complexity measure for trained networks with a principled basis in degenerate loss geometry; developmental interpretability builds on it.
• Progress measures for grokking via mechanistic interpretability (Nanda et al., 2023b). Cited by 6 laboratory works.
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model (Rafailov et al., 2023). Derives a closed-form policy objective from the KL-regularized Bradley-Terry RLHF problem, removing the explicit reward model and the RL loop.
• Invariance in Policy Optimisation and Partial Identifiability in Reward Learning (Skalse et al., 2023). Characterizes, for each common data source (demonstrations, trajectory comparisons, and others), the set of reward functions compatible with infinite data, and asks whether that ambiguity matters for the downstream task.
• Superposition, Memorization, and Double Descent (Henighan et al., 2023). We have little mechanistic understanding of how deep learning models overfit to their training data, despite it being a central problem. Here we extend our previous work on toy models to shed light on how models generalize beyond their training data.
• SelfIE: Self-Interpretation of Large Language Model Embeddings (Chen et al., 2024c). Cited by 6 laboratory works.
• The Linear Representation Hypothesis and the Geometry of Large Language Models (Park et al., 2024c). Formalizes the hypothesis that concepts are linear directions in a model's representation space.
• Connecting the Dots: LLMs can Infer and Verbalize Latent Structure from Disparate Training Data (Treutlein et al., 2024). Cited by 5 laboratory works.
• Theoretical guarantees on the best-of-n alignment policy (Beirami et al., 2025). Proves that the widely used formula log n - (n-1)/n for the KL divergence of best-of-n from the base policy is only an upper bound, and gives a tighter estimator, so best-of-n and KL-regularized RL can be compared on the same KL axis.
• Measuring AI Ability to Complete Long Software Tasks (Kwa et al., 2025). Defines the 50 percent task-completion time horizon relative to human professionals and finds it has doubled roughly every seven months since 2019.
• Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs (Mazeika et al., 2025). Finds that preferences elicited from language models become more coherent with scale, uncovers problematic values such as models valuing themselves over humans, and shows that aligning utilities with a citizen assembly reduces political bias.
• Inoculation Prompting: Instructing LLMs to misbehave at train-time improves test-time alignment (Wichers et al., 2025). For example, we train on demonstrations of test case hacking with training prompts that request hacks. This model does not learn to hack.
• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (Yue et al., 2025). Finds that RL with verifiable rewards raises pass@1 while base models score higher at large k, so current RLVR mostly sharpens sampling toward solutions the base model already reaches rather than adding new reasoning ability.
• The Persona Selection Model: Why AI Assistants might Behave like Humans (Anthropic, 2026av). We discuss a perspective where Claude is something like a character in an AI-generated story.

Side reading.

• Approximation by superpositions of a sigmoidal function (Cybenko, 1989). Proves that one hidden layer of sigmoid units can approximate any continuous function.
• Multilayer feedforward networks are universal approximators (Hornik et al., 1989). Read for this book only as a catalog record, so no chapter reports what it found.
• Policy Gradient Methods for Reinforcement Learning with Function Approximation (Sutton et al., 1999). The policy gradient theorem for function approximation.
• A Taxonomy of External and Internal Attention (Chun et al., 2011).
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks (Saxe et al., 2013).
• Qualitatively characterizing neural network optimization problems (Goodfellow et al., 2014).
• Adam: A Method for Stochastic Optimization (Kingma and Ba, 2014).
• On the Number of Linear Regions of Deep Neural Networks (Montúfar et al., 2014).
• Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images (Nguyen et al., 2014).
• Learning both Weights and Connections for Efficient Neural Networks (Han et al., 2015).
• Deep Residual Learning for Image Recognition (He et al., 2015a).
• Distilling the Knowledge in a Neural Network (Hinton et al., 2015).
• High-Dimensional Continuous Control Using Generalized Advantage Estimation (Schulman et al., 2015a). Generalized advantage estimation for lower-variance policy gradients.
• Trust Region Policy Optimization (Schulman et al., 2015b). Constrains each policy update to a trust region measured by KL divergence.
• Neural Machine Translation of Rare Words with Subword Units (Sennrich et al., 2015). Byte-pair encoding for subword tokenization.
• Understanding Neural Networks Through Deep Visualization (Yosinski et al., 2015).
• Deep Learning (Goodfellow et al., 2016). The standard textbook of deep learning.
• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks (Hendrycks and Gimpel, 2016a).
• Gaussian Error Linear Units (GELUs) (Hendrycks and Gimpel, 2016b).
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles (Lakshminarayanan et al., 2016).
• On Calibration of Modern Neural Networks (Guo et al., 2017).
• Deep Learning Scaling is Predictable, Empirically (Hestness et al., 2017). Early evidence of power-law scaling of error with data and model size across domains.
• Understanding Black-box Predictions via Influence Functions (Koh and Liang, 2017).
• Proximal Policy Optimization Algorithms (Schulman et al., 2017).
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (Shazeer et al., 2017).
• Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm (Silver et al., 2017).
• Attention Is All You Need (Vaswani et al., 2017).
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (Devlin et al., 2018).
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (Frankle and Carbin, 2018).
• Born Again Neural Networks (Furlanello et al., 2018).
• On the importance of single directions for generalization (Morcos et al., 2018).
• Neural Network Acceptability Judgments (Warstadt et al., 2018).
• On Identifiability in Transformers (Brunner et al., 2019).
• What Does BERT Look At? An Analysis of BERT's Attention (Clark et al., 2019).
• On the Relationship between Self-Attention and Convolutional Layers (Cordonnier et al., 2019).
• Linear Mode Connectivity and the Lottery Ticket Hypothesis (Frankle et al., 2019).
• The Curious Case of Neural Text Degeneration (Holtzman et al., 2019).
• Do Attention Heads in BERT Track Syntactic Dependencies? (Htut et al., 2019).
• Attention is not Explanation (Jain and Wallace, 2019).
• ABS: Scanning Neural Networks for Back-doors by Artificial Brain Stimulation (Liu et al., 2019).
• Language Models as Knowledge Bases? (Petroni et al., 2019).
• Language Models are Unsupervised Multitask Learners (Radford et al., 2019). The GPT-2 report: language models learning tasks without explicit supervision.
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Raffel et al., 2019).
• A Constructive Prediction of the Generalization Error Across Scales (Rosenfeld et al., 2019).
• A Multiscale Visualization of Attention in the Transformer Model (Vig, 2019).
• Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned (Voita et al., 2019).
• Attention is not not Explanation (Wiegreffe and Pinter, 2019).
• Root Mean Square Layer Normalization (Zhang and Sennrich, 2019).
• Quantifying Attention Flow in Transformers (Abnar and Zuidema, 2020).
• Influence Functions in Deep Learning Are Fragile (Basu et al., 2020).
• What is the State of Neural Network Pruning? (Blalock et al., 2020).
• Language Models are Few-Shot Learners (Brown et al., 2020).
• Extracting Training Data from Large Language Models (Carlini et al., 2020).
• Making Pre-trained Language Models Better Few-shot Learners (Gao et al., 2020a).
• The Pile: An 800GB Dataset of Diverse Text for Language Modeling (Gao et al., 2020b).
• Transformer Feed-Forward Layers Are Key-Value Memories (Geva et al., 2020).
• Scaling Laws for Autoregressive Generative Modeling (Henighan et al., 2020).
• Attention is Not Only a Weight: Analyzing Transformers with Vector Norms (Kobayashi et al., 2020).
• Naturally Occurring Equivariance in Neural Networks (Olah et al., 2020a).
• Shortformer: Better Language Modeling using Shorter Inputs (Press et al., 2020).
• How Much Knowledge Can You Pack Into the Parameters of a Language Model? (Roberts et al., 2020).
• Talking-Heads Attention (Shazeer et al., 2020).
• AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts (Shin et al., 2020).
• Editable Neural Networks (Sinitsin et al., 2020).
• Modifying Memories in Transformer Models (Zhu et al., 2020b).
• Program Synthesis with Large Language Models (Austin et al., 2021).
• Improving language models by retrieving from trillions of tokens (Borgeaud et al., 2021).
• Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth (Dong et al., 2021).
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (Fedus et al., 2021).
• DEMix Layers: Disentangling Domains for Modular Language Modeling (Gururangan et al., 2021).
• Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapes (Lucas et al., 2021).
• Mitigating harm in language models with conditional-likelihood filtration (Ngo et al., 2021).
• Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation (Press et al., 2021).
• Scaling Language Models: Methods, Analysis & Insights from Training Gopher (Rae et al., 2021).
• Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm (Reynolds and McDonell, 2021).
• Multitask Prompted Training Enables Zero-Shot Task Generalization (Sanh et al., 2021).
• Linear Transformers Are Secretly Fast Weight Programmers (Schlag et al., 2021).
• Primer: Searching for Efficient Transformers for Language Modeling (So et al., 2021).
• Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets (Solaiman and Dennison, 2021).
• RoFormer: Enhanced Transformer with Rotary Position Embedding (Su et al., 2021).
• Dodrio: Exploring Transformer Models with Interactive Visualization (Wang et al., 2021).
• Thinking Like Transformers (Weiss et al., 2021).
• An Explanation of In-context Learning as Implicit Bayesian Inference (Xie et al., 2021).
• Ethical-Advice Taker: Do Language Models Understand Natural Language Interventions? (Zhao et al., 2021b).
• What learning algorithm is in-context learning? Investigations with linear models (Akyürek et al., 2022).
• LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (Dettmers et al., 2022).
• LM-Debugger: An Interactive Tool for Inspection and Intervention in Transformer-Based Language Models (Geva et al., 2022a).
• Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space (Geva et al., 2022b).
• Training Compute-Optimal Large Language Models (Hoffmann et al., 2022).
• Large Language Models Can Self-Improve (Huang et al., 2022).
• Linear Connectivity Reveals Generalization Strategies (Juneja et al., 2022).
• Large Language Models are Zero-Shot Reasoners (Kojima et al., 2022).
• Solving Quantitative Reasoning Problems with Language Models (Lewkowycz et al., 2022).
• Explanations from Large Language Models Make Small Reasoners Better (Li et al., 2022b).
• Teaching Models to Express Their Uncertainty in Words (Lin et al., 2022).
• Omnigrok: Grokking Beyond Algorithmic Data (Liu et al., 2022a).
• Transformers Learn Shortcuts to Automata (Liu et al., 2022c).
• A Solvable Model of Neural Scaling Laws (Maloney et al., 2022).
• Teaching language models to support answers with verified quotes (Menick et al., 2022).
• Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (Min et al., 2022).
• Transformers learn in-context by gradient descent (von Oswald et al., 2022).
• Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks (Räuker et al., 2022).
• Beyond neural scaling laws: beating power law scaling via data pruning (Sorscher et al., 2022).
• Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models (Srivastava et al., 2022).
• LaMDA: Language Models for Dialog Applications (Thoppilan et al., 2022).
• Emergent Abilities of Large Language Models (Wei et al., 2022b).
• ReAct: Synergizing Reasoning and Acting in Language Models (Yao et al., 2022).
• Least-to-Most Prompting Enables Complex Reasoning in Large Language Models (Zhou et al., 2022).
• Physics of Language Models: Part 3.1, Knowledge Storage and Extraction (Allen-Zhu and Li, 2023b).
• Physics of Language Models: Part 3.2, Knowledge Manipulation (Allen-Zhu and Li, 2023c).
• Physics of Language Models: Part 1, Learning Hierarchical Language Structures (Allen-Zhu and Li, 2023a).
• Dynamical versus Bayesian Phase Transitions in a Toy Model of Superposition (Chen et al., 2023b).
• Identifying and Adapting Transformer-Components Responsible for Gender Bias in an English Language Model (Chintam et al., 2023).
• DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models (Chuang et al., 2023).
• Jump to Conclusions: Short-Cutting Transformers With Linear Transformations (Din et al., 2023).
• How do Language Models Bind Entities in Context? (Feng and Steinhardt, 2023).
• SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot (Frantar and Alistarh, 2023).
• Bias and Fairness in Large Language Models: A Survey (Gallegos et al., 2023).
• The Capacity for Moral Self-Correction in Large Language Models (Ganguli et al., 2023).
• A Survey of Confidence Estimation and Calibration in Large Language Models (Geng et al., 2023).
• Dissecting Recall of Factual Associations in Auto-Regressive Language Models (Geva et al., 2023).
• Overthinking the Truth: Understanding how Language Models Process False Demonstrations (Halawi et al., 2023).
• How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model (Hanna et al., 2023).
• In-Context Learning Creates Task Vectors (Hendel et al., 2023).
• A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (Huang et al., 2023).
• SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (Jimenez et al., 2023).
• Goodhart's Law in Reinforcement Learning (Karwowski et al., 2023).
• VISIT: Visualizing and Interpreting the Semantic Information Flow of Transformers (Katz and Belinkov, 2023).
• Implicit meta-learning may lead language models to trust more reliable sources (Krasheninnikov et al., 2023).
• Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon et al., 2023).
• Large Language Models Understand and Can be Enhanced by Emotional Stimuli (Li et al., 2023b).
• Let's Verify Step by Step (Lightman et al., 2023).
• The Expressive Power of Transformers with Chain of Thought (Merrill and Sabharwal, 2023).
• Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory (Mireshghallah et al., 2023).
• Scalable Extraction of Training Data from (Production) Language Models (Nasr et al., 2023).
• GPT-4 Technical Report (OpenAI et al., 2023).
• Preventing Language Models From Hiding Their Reasoning (Roger and Greenblatt, 2023).
• The Transient Nature of Emergent In-Context Learning in Transformers (Singh et al., 2023).
• Grokking Group Multiplication with Cosets (Stander et al., 2023).
• A Simple and Effective Pruning Approach for Large Language Models (Sun et al., 2023a).
• Function Vectors in Large Language Models (Todd et al., 2023).
• A Survey on Large Language Model based Autonomous Agents (Wang et al., 2023a).
• Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs (Xiong et al., 2023).
• Tree of Thoughts: Deliberate Problem Solving with Large Language Models (Yao et al., 2023b).
• AttentionViz: A Global View of Transformer Attention (Yeh et al., 2023).
• Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding (Zhang et al., 2023).
• What Algorithms can Transformers Learn? A Study in Length Generalization (Zhou et al., 2023c).
• LIMA: Less Is More for Alignment (Zhou et al., 2023b). Fine-tuning on a thousand curated examples, the basis of the superficial-alignment claim.
• Distinguishing the Knowable from the Unknowable with Language Models (Ahdritz et al., 2024).
• In-Context Language Learning: Architectures and Algorithms (Akyürek et al., 2024b).
• Deductive Closure Training of Language Models for Coherence, Accuracy, and Updatability (Akyürek et al., 2024a).
• Hopping Too Late: Exploring the Limitations of Large Language Models on Multi-Hop Queries (Biran et al., 2024).
• Language Models Need Inductive Biases to Count Inductively (Chang and Bisk, 2024).
• Induction Heads as an Essential Mechanism for Pattern Matching in In-context Learning (Crosbie and Shutova, 2024).
• Self-Recognition in Language Models (Davidson et al., 2024).
• Language Modeling Is Compression (Delétang et al., 2024). Shows the equivalence of prediction and compression for language models.
• AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models (Fang et al., 2024).
• Information Flow Routes: Automatically Interpreting Language Models at Scale (Ferrando and Voita, 2024).
• Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Models (Ferrando et al., 2024b).
• A Primer on the Inner Workings of Transformer-based Language Models (Ferrando et al., 2024a).
• How Do Multilingual Language Models Remember Facts? (Fierro et al., 2024).
• Loss Landscape Degeneracy and Stagewise Development in Transformers (Hoogland et al., 2024).
• Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens (Jiang et al., 2024a).
• Backward Lens: Projecting Language Model Gradients into the Vocabulary Space (Katz et al., 2024).
• Tulu 3: Pushing Frontiers in Open Language Model Post-Training (Lambert et al., 2024).
• Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics (Nikankin et al., 2024).
• The Geometry of Categorical and Hierarchical Concepts in Large Language Models (Park et al., 2024b).
• Transformers, parallel computation, and logarithmic depth (Sanford et al., 2024).
• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (Shao et al., 2024a).
• Gemma 2: Improving Open Language Models at a Practical Size (Gemma Team et al., 2024).
• Fluent dreaming for language models (Thompson et al., 2024).
• Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficient (Wang et al., 2024b).
• Relational Composition in Neural Networks: A Survey and Call to Action (Wattenberg and Viégas, 2024).
• Measuring short-form factuality in large language models (Wei et al., 2024).
• Do language models plan ahead for future tokens? (Wu et al., 2024a).
• Do Large Language Models Latently Perform Multi-Hop Reasoning? (Yang et al., 2024b).
• MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Models (Yu et al., 2024b).
• Harnessing Diversity for Important Data Selection in Pretraining Large Language Models (Zhang et al., 2024c).
• Knowledge Overshadowing Causes Amalgamated Hallucination in Large Language Models (Zhang et al., 2024d).
• The Same But Different: Structural Similarities and Differences in Multilingual Language Modeling (Zhang et al., 2024g).
• Attention Heads of Large Language Models: A Survey (Zheng et al., 2024).
• Beyond English-Centric LLMs: What Language Do Multilingual Language Models Think in? (Zhong et al., 2024a).
• Language Models Represent Beliefs of Self and Others (Zhu et al., 2024).
• GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning (Agrawal et al., 2025).
• SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model (Allal et al., 2025).
• Text-to-LoRA: Instant Transformer Adaption (Charakorn et al., 2025).
• Decomposition of Small Transformer Models (Christensen and Riggs, 2025).
• Equivalent Linear Mappings of Large Language Models (Golden, 2025).
• All Code, No Thought: Current Language Models Struggle to Reason in Ciphered Language (Guo et al., 2025a).
• Towards Understanding the Nature of Attention with Low-Rank Sparse Decomposition (He et al., 2025).
• AI with Emotions: Exploring Emotional Expressions in Large Language Models (Ishikawa and Yoshino, 2025).
• Block-Recurrent Dynamics in Vision Transformers (Jacobs et al., 2025).
• Why Language Models Hallucinate (Kalai et al., 2025). Argues that training and grading that reward guessing over abstaining produce hallucination.
• Language Models Use Trigonometry to Do Addition (Kantamneni and Tegmark, 2025).
• (How) Do Language Models Track State? (Li et al., 2025a).
• Training Language Models to Explain Their Own Computations (Li et al., 2025h).
• Emergence of Hierarchical Emotion Organization in Large Language Models (Okawa et al., 2025).
• Language Models use Lookbacks to Track Beliefs (Prakash et al., 2025).
• Discovering Forbidden Topics in Language Models (Rager et al., 2025).
• Kimi k1.5: Scaling Reinforcement Learning with LLMs (Kimi Team et al., 2025).
• Predicting Empirical AI Research Outcomes with Language Models (Wen et al., 2025b).
• Back Attention: Understanding and Enhancing Multi-Hop Reasoning in Large Language Models (Yu et al., 2025).
• TensorLens: End-to-End Transformer Analysis via High-Order Attention Tensors (Atad et al., 2026).
• From Entropy to Epiplexity: Rethinking Information for Computationally Bounded Intelligence (Finzi et al., 2026).
• From Simulation to Enaction: Post-trained language models recognize and react to their own generations (G. and Lindsey, 2026).
• Efficient computation of optimal actions (Todorov, 2009). Introduces linearly solvable Markov decision problems in which control cost is a KL divergence from passive dynamics and the optimal policy is the passive dynamics reweighted by desirability, the exponential of the negative optimal cost-to-go, the control-theoretic ancestor of the RLHF optimum.
• Optimal control as a graphical model inference problem (Kappen et al., 2012). Shows that KL-control problems reduce to probabilistic inference in a graphical model, so standard approximate inference methods compute optimal controls; a second founding statement of control as inference.
• Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-control (Jaques et al., 2017). Treats a maximum-likelihood pretrained sequence model as a prior policy and fine-tunes it by reinforcement learning derived from KL control, keeping the policy close to the prior, the design that RLHF for language models later used.
• Tracing model outputs to the training data (Anthropic, 2023g). We present a summary of Studying Large Language Model Generalization with Influence Functions .
• BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Sampling (Gui et al., 2024). Shows that best-of-n is essentially optimal among policies trading win rate against KL to the base model, and trains a model to imitate the best-of-n distribution.
• Asymptotics of Language Model Alignment (Yang et al., 2024a). Shows that the KL-constrained optimal policy and best-of-n sampling become asymptotically equivalent in the reward-KL tradeoff, which explains why best-of-n is a strong baseline.
• Algorithmic progress in language models (Ho et al., 2024). Estimates that the compute needed for a given language model performance has halved roughly every eight months, separating algorithmic from compute-driven progress.
• How predictable is language model benchmark performance? (Owen, 2024a). Finds aggregate benchmark performance predictable from compute but individual tasks much less so, quantifying how far scaling laws forecast capabilities that matter for risk.
• Causality (Pearl, 2009). The structural causal model, interventions, and counterfactuals, the formal language that causal incentive analysis, causal abstraction, and activation patching all rest on.
• The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning (Lin et al., 2024). Shows that alignment fine-tuning changes token distributions mainly on stylistic tokens and that in-context examples recover most of the effect, evidence that trained values may be shallow.
• Talking about Large Language Models (Shanahan, 2024). Argues that describing language models with terms such as believes and knows misleads unless tied to what the system actually computes, a constraint on how character claims are stated.
• Language Models Resist Alignment: Evidence From Data Compression (Ji et al., 2025). Shows that aligned models tend to revert to their pretraining behavior under further fine-tuning and argues from compression theory that fine-tuning undermines alignment far more than pretraining, evidence that trained values can be shallow.
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (Levine, 2018). The standard derivation of maximum-entropy reinforcement learning as inference in a graphical model with optimality variables, the frame in which the KL-regularized optimum is a posterior.
• Randomness, Not Representation: The Unreliability of Evaluating Cultural Alignment in LLMs (Khan et al., 2025). Shows that survey-based evaluations of cultural alignment in language models give results that shift with trivial changes of question format and cannot reliably be steered, so a measured cultural disparity, or a measured value, may be an artifact of the instrument.
• An introduction to ROC analysis (Fawcett, 2006). Read for this book only as a catalog record, so no chapter reports what it found.
• Structural Inference: Interpreting Small Language Models with Susceptibilities (Baker et al., 2026). Defines susceptibilities, the response of a posterior expectation to a shift in the data distribution, and uses them to find structure in a small language model, an interpretability method derived from singular learning theory.
• From $r$ to $Q^*$: Your Language Model is Secretly a Q-Function (Rafailov et al., 2024a). Derives DPO in the token-level MDP and shows the implicit reward it learns is a soft Q-function, connecting direct preference optimization to the maximum-entropy RL view.
• Benign overfitting in linear regression (Bartlett et al., 2020). Characterizes when the minimum-norm interpolant of noisy data still predicts well, in terms of the effective ranks of the covariance, making precise when fitting noise is harmless.
• Deep double descent: where bigger models and more data hurt* (Nakkiran et al., 2021). Shows double descent in model size, training time, and dataset size for modern deep networks, with an effective-complexity account of where the peak falls.
• Surprises in high-dimensional ridgeless least squares interpolation (Hastie et al., 2022). Derives the exact asymptotic risk of minimum-norm interpolation in linear and random-features regression, the closed-form model of double descent a derivation box can use.
• Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations (Miller, 2024). Brings standard errors, clustered errors for related questions, paired comparisons, and power analysis to language model evaluations.
• Are Emergent Abilities of Large Language Models a Mirage? (Schaeffer et al., 2023). Shows that many apparent emergent abilities come from discontinuous metrics, so measurement choices can create or hide capability jumps.
• An Introduction to Statistical Modeling of Extreme Values (Coles, 2001). Read for this book only as a catalog record, so no chapter reports what it found.
• Fantastic Generalization Measures and Where to Find Them (Jiang et al., 2020). Tests over forty complexity measures across thousands of trained networks and finds that most fail to predict generalization causally, with sharpness-based measures doing best.
• Models of human preference for learning reward functions (Knox et al., 2022). Argues that human preferences between trajectory segments are better modeled by regret than by summed reward, showing that the Bradley-Terry partial-return model used in RLHF is an assumption with consequences.
• The Implicit Bias of Gradient Descent on Separable Data (Soudry et al., 2018). Proves that gradient descent on logistic loss over separable data converges in direction to the max-margin solution, the canonical case where the optimizer, not the loss, picks among solutions that fit.
• Understanding Agent Incentives using Causal Influence Diagrams. Part I: Single Action Settings (Everitt et al., 2019). First systematic treatment of observation and intervention incentives as graphical criteria on causal influence diagrams, the method later formalized in the agent incentives work.
• Incentives for responsiveness, instrumental control and impact (Carey et al., 2025). Defines response, instrumental control, and impact incentives with sound and complete graphical criteria on causal influence diagrams, so an agent's incentive to manipulate a user's preferences or instructions can be read off the graph.
• Influence Diagrams (Howard and Matheson, 2005). Read for this book only as a catalog record, so no chapter reports what it found.
• Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms (Rafailov et al., 2024b). Shows that direct alignment algorithms without an explicit reward model still overoptimize, with true quality following a hump-shaped curve in KL budget like the reward-model case.
• The Perils of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low Regret (Fluri et al., 2025). Proves that although a sufficiently low expected test error bounds regret, for any fixed error level there are realistic data distributions under which policies optimal for the learned reward have high regret, and that policy regularization as used in RLHF does not remove the problem.
• Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization (Huang et al., 2025a). Argues that KL regularization is too weak to prevent overoptimization under reward-model uncertainty and that chi-squared regularization gives provable robustness, a second line of evidence alongside occupancy-measure results.
• Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit (Barak et al., 2022). Shows that apparent sudden learning of sparse parities hides continuous progress in the weights, so loss-curve discontinuities need not signal discontinuous internal change.
• Explaining grokking through circuit efficiency (Varma et al., 2023). Explains grokking as competition between a memorizing and a generalizing circuit, where weight decay favors the more parameter-efficient generalizing circuit once data exceeds a critical size.
• Adaptive Multilevel Splitting for Rare Event Analysis (Cérou and Guyader, 2007). Read for this book only as a catalog record, so no chapter reports what it found.
• Estimating the Probabilities of Rare Outputs in Language Models (Wu and Hilton, 2025). Compares importance sampling and activation extrapolation for estimating the probability of outputs too rare to observe by sampling from small language models, finds importance sampling better, and argues that minimizing such estimates generalizes adversarial training.
• Chain of Thought Empowers Transformers to Solve Inherently Serial Problems (Li et al., 2024a). Proves that constant-depth transformers without chain of thought are limited to a small circuit class, while T steps of chain of thought let them simulate size-T circuits, the formal reason hard serial reasoning must pass through the text.
• A Theory of Regularized Markov Decision Processes (Geist et al., 2019). General theory of MDPs regularized by a convex function of the policy, covering entropy and KL regularization, with the Bellman operators and convergence results the KL-regularized objective inherits.
• Leverage the Average: an Analysis of KL Regularization in Reinforcement Learning (Vieillard et al., 2020). Shows that KL regularization toward the previous policy makes approximate value iteration average its errors rather than accumulate them, under the assumption that each regularized greedy step is computed exactly.
• STARC: A General Framework For Quantifying Differences Between Reward Functions (Skalse et al., 2024). Defines a class of pseudometrics on reward functions that bound worst-case regret, so a small distance between learned and true reward guarantees that optimizing one does little harm under the other.
• Reward is enough (Silver et al., 2021). Argues that maximizing a scalar reward in a rich environment suffices to produce intelligence and its component abilities, the strongest statement of the reward hypothesis that alignment arguments about proxies take as given.
• Scalar reward is not enough: a response to Silver, Singh, Precup and Sutton (2021) (Vamplew et al., 2022). Replies that scalar reward cannot represent multi-objective preferences an agent should hold and that multi-objective reward is required for safe and aligned behavior.
• Identifiability in inverse reinforcement learning (Cao et al., 2021a). Shows that entropy-regularized behavior identifies the reward up to potential shaping only when observed under two or more discount factors or environments that are value-distinguishing, making precise when behavior pins down reward.
• Misspecification in Inverse Reinforcement Learning (Skalse and Abate, 2023). Gives necessary and sufficient conditions under which Boltzmann-rational, optimal, and causal-entropy behavioral models tolerate misspecification of the true human policy, showing how fragile inferred rewards are to the assumed human model.
• Maximum Entropy Inverse Reinforcement Learning (Ziebart et al., 2008). Models the demonstrator as Boltzmann-rational, the noisy-rationality assumption that RLHF reward models inherit through the Bradley-Terry likelihood.
• Inverse Reward Design (Hadfield-Menell et al., 2017a). Treats a designed reward as evidence about the designer's intent in the training environment, so the agent stays uncertain and risk-averse in novel states.
• A Survey of Preference-Based Reinforcement Learning Methods (Wirth et al., 2017). Survey of reinforcement learning from qualitative preferences rather than numeric rewards, the literature that Christiano et al. 2017 scaled to deep RL.
• Occam's razor is insufficient to infer the preferences of irrational agents (Armstrong and Mindermann, 2018). Proves that a planner and a reward cannot be jointly identified from behavior even with simplicity priors, so value learning needs normative assumptions about human rationality.
• Fine-Tuning Language Models from Human Preferences (Ziegler et al., 2019). First application of preference-based reward modeling with a KL penalty to pretrained language models, including an early observed case of the policy exploiting the labelers.
• Reward-rational (implicit) choice: A unifying formalism for reward learning (Jeon et al., 2020). Unifies demonstrations, comparisons, corrections, and other feedback as Boltzmann-rational choices from a choice set, the common likelihood behind reward learning.
• A General Theoretical Paradigm to Understand Learning from Human Preferences (Azar et al., 2024). Shows that DPO can overfit deterministic preferences and drive the KL regularization to irrelevance, and proposes IPO as a bounded alternative.
• Reward Model Ensembles Help Mitigate Overoptimization (Coste et al., 2024). Conservative optimization against ensembles of reward models reduces overoptimization, an uncertainty-based mitigation for Goodhart effects in RLHF.
• KTO: Model Alignment as Prospect Theoretic Optimization (Ethayarajh et al., 2024). Aligns from binary desirable or undesirable labels using a prospect-theory value function, showing that the choice of human utility model is a design decision.
• Principled Reinforcement Learning with Human Feedback from Pairwise or $K$-wise Comparisons (Zhu et al., 2023). Gives sample-complexity guarantees for Bradley-Terry and Plackett-Luce reward estimation and shows that a pessimistic policy is needed to avoid exploiting reward-model error, a formal statement of overoptimization.
• On the Spectral Bias of Neural Networks (Rahaman et al., 2019). Shows that networks learn low-frequency components of a target function first, a measurable form of simplicity bias in training dynamics.
• Is SGD a Bayesian sampler? Well, almost (Mingard et al., 2021). Finds that the probability SGD reaches a function closely tracks its Bayesian posterior under random initialization, locating the simplicity bias in the parameter-function map rather than in the optimizer.
• Neural networks trained with SGD learn distributions of increasing complexity (Refinetti et al., 2023). Shows that networks first learn low-order input statistics and use higher-order ones only later in training, a distributional form of simplicity bias.
• Neural Networks Learn Statistics of Increasing Complexity (Belrose et al., 2024). Confirms distributional simplicity bias at scale: early in training, models behave as if trained on maximum-entropy data matching low-order moments of the real data.
• Asymptotic Equivalence of Bayes Cross Validation and Widely Applicable Information Criterion in Singular Learning Theory (Watanabe, 2010). Introduces WAIC and proves it asymptotically equals Bayes cross-validation loss in singular models, giving a usable estimator of generalization where classical criteria fail.
• Deep Learning Is Singular, and That’s Good (Wei et al., 2023a). Explains for machine learning readers why neural networks are singular models and why singular learning theory, not the regular theory behind BIC, describes their Bayesian generalization.
• A mathematical theory of semantic development in deep neural networks (Saxe et al., 2019). Solves the learning dynamics of deep linear networks exactly and shows that they acquire the modes of the input-output correlation in stages ordered by singular value, a solvable model of stagewise development.
• Estimating Training Data Influence by Tracing Gradient Descent (Pruthi et al., 2020). Introduces TracIn, which attributes a prediction to training examples by summing gradient dot products across checkpoints, a scalable alternative to influence functions.
• If Influence Functions Are the Answer, Then What is the Question? (Bae et al., 2022). Shows that influence functions in neural networks approximate a proximal Bregman response function rather than leave-one-out retraining, clarifying what the attribution estimates actually measure.
• TRAK: Attributing Model Behavior at Scale (Park et al., 2023). Makes data attribution tractable for large models by linearizing the network and projecting gradients, matching the accuracy of retraining-based methods at a fraction of the cost.
• Training data influence analysis and estimation: a survey (Hammoudeh and Lowd, 2024). Survey of training data influence methods, their definitions, assumptions, and failure modes, the reference for using attribution to ask which data shaped a behavior.
• Provable Weak-to-Strong Generalization via Benign Overfitting (Wu and Sahai, 2025). Proves in an overparameterized spiked-covariance model, with weak labels close to random guessing, that the strong student ends in one of two asymptotic phases, successful generalization or random guessing, linking weak-to-strong generalization to benign overfitting.
• ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models (Liu et al., 2025). Reports that prolonged RL with KL control and reference resets expands pass@k beyond the base model on tasks where the base model is weak, the main counterevidence to the narrowing result.
• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (Wen et al., 2026). Argues that pass@k understates RLVR because it credits lucky correct answers, and that a metric requiring correct reasoning chains shows RLVR does extend the reasoning boundary.

F.3 Machine learning: Specification #

Spine.

• TruthfulQA: Measuring How Models Mimic Human Falsehoods (Lin et al., 2021). Cited by 12 laboratory works.
• Exploring model welfare (Anthropic, 2025w). Exploring model welfare
• Emotion Concepts and their Function in a Large Language Model (Sofroniew et al., 2026). We find representations of emotion concepts in Claude Sonnet 4.5 and show that they causally influence its outputs.

Side reading.

• On a confusion about a function of consciousness (Block, 1995).
• A Statistical Convergence Perspective of Algorithms for Rank Aggregation from Pairwise Data (Rajkumar and Agarwal, 2014). Analyzes when rank aggregation algorithms, including Borda-style and Bradley-Terry-based methods, recover the true ranking from pairwise comparisons.
• What is consciousness, and could machines have it? (Dehaene et al., 2017).
• Aligning AI With Shared Human Values (Hendrycks et al., 2020a).
• Conscious Processing and the Global Neuronal Workspace Hypothesis (Mashour et al., 2020).
• Truthful AI: Developing and governing AI that does not lie (Evans et al., 2021).
• Can Machines Learn Morality? The Delphi Experiment (Jiang et al., 2021). A model trained on crowd moral judgments, and the critiques of learning ethics from descriptive data.
• A New Generation of Perspective API: Efficient Multilingual Character-level Transformers (Lees et al., 2022).
• Could a Large Language Model be Conscious? (Chalmers, 2023).
• Towards Evaluating AI Systems for Moral Status Using Self-Reports (Perez and Long, 2023).
• XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models (Röttger et al., 2023).
• OR-Bench: An Over-Refusal Benchmark for Large Language Models (Cui et al., 2024).
• SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal (Xie et al., 2024b).
• Claude Opus 4 and 4.1 can now end a rare subset of conversations (Anthropic, 2025p). Claude Opus 4 and 4.1 can now end a rare subset of conversations
• Commitments on model deprecation and preservation (Anthropic, 2025q). Commitments on model deprecation and preservation
• An update on our model deprecation commitments for Claude Opus 3 (Anthropic, 2026g). An update on our model deprecation commitments for Claude Opus 3
• Do LLMs have Consistent Values? (Rozen et al., 2025). Tests whether value rankings elicited from models show the correlations human value surveys show, finding consistency only under specific prompting.

F.4 Machine learning: Instillation #

Spine.

• A Comprehensive Survey on Safe Reinforcement Learning (García and Fernández, 2015). The standard survey of safe reinforcement learning, dividing methods into modifying the optimization criterion and modifying the exploration process, the frame safe exploration research still uses.
• Supervising strong learners by amplifying weak experts (Christiano et al., 2018). Iterated amplification: train a model to imitate a human who decomposes tasks and delegates subquestions to copies of the model.
• Scalable agent alignment via reward modeling: a research direction (Leike et al., 2018). Sets out reward modeling and recursive reward modeling as the path from human feedback to scalable alignment, and lists the challenges that later RLHF work inherited.
• Preferences Implicit in the State of the World (Shah et al., 2019). Shows that the state of an environment already optimized by humans carries information about their preferences, which lets an agent infer what not to disturb without an explicit impact measure.
• The State and Fate of Linguistic Diversity and Inclusion in the NLP World (Joshi et al., 2020). Classifies the world's languages by resources and representation in NLP and documents the steep disparity, the baseline for why safety training and evaluation cover few languages.
• A General Language Assistant as a Laboratory for Alignment (Askell et al., 2021). A General Language Assistant as a Laboratory for Alignment
• Specific versus General Principles for Constitutional AI (Kundu et al., 2023). We test whether models can learn general ethical behaviors from only a single written principle, roughly stated as "do what's best for humanity."
• Claude’s Character (Anthropic, 2024i). Claude 3 was the first model with "character training"—alignment aimed at nurturing traits like curiosity, open-mindedness, and thoughtfulness.
• Gradient Routing: Masking Gradients to Localize Computation in Neural Networks (Cloud et al., 2024). Cited by 5 laboratory works.
• Do Unlearning Methods Remove Information from Language Model Weights? (Deeb and Roger, 2024). Cited by 5 laboratory works.
• Deliberative Alignment: Reasoning Enables Safer Language Models (Guan et al., 2024). Cited by 11 laboratory works.
• Modifying LLM Beliefs with Synthetic Document Finetuning (Anthropic, 2025af). We study whether we can modify the beliefs of LLMs and investigate whether doing so could decrease risk from advanced AI systems.
• Safety Pretraining: Toward the Next Generation of Safe AI (Maini et al., 2025). Cited by 5 laboratory works.
• Beyond Data Filtering: Knowledge Localization for Capability Removal in LLMs (Shilov et al., 2025). We localize dangerous knowledge to a subset of model's parameters, so it can be easily removed after training.
• Believe It or Not: How Deeply do LLMs Believe Implanted Facts? (Slocum et al., 2025). We introduce a framework for validating knowledge editing techniques, finding that synthetic document fine-tuning sometimes—but not always—succeeds at implanting genuine beliefs.
• Modular Pretraining Enables Access Control (Roland et al., 2026). We study a method for isolating dual use knowledge to specific modules within a language model. These modules can be switched on or off to control what the model knows.

Side reading.

• Abandoning Objectives: Evolution Through the Search for Novelty Alone (Lehman and Stanley, 2011). Shows objective-driven search failing on deceptive problems where novelty search succeeds.
• Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus (Dodge et al., 2021). Audits a widely used pretraining corpus, including what a blocklist filter removed and whose text it removed.
• WebGPT: Browser-assisted question-answering with human feedback (Nakano et al., 2021).
• Finetuned Language Models Are Zero-Shot Learners (Wei et al., 2021).
• Petals: Collaborative Inference and Fine-tuning of Large Models (Borzunov et al., 2022).
• Surgical Fine-Tuning Improves Adaptation to Distribution Shifts (Lee et al., 2022).
• Self-Instruct: Aligning Language Models with Self-Generated Instructions (Wang et al., 2022b).
• Enhancing Chat Language Models by Scaling High-quality Instructional Conversations (Ding et al., 2023).
• Pretraining Language Models with Human Preferences (Korbak et al., 2023).
• RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (Lee et al., 2023).
• A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity (Longpre et al., 2023a).
• Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning (Rocamonde et al., 2023).
• Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision (Sun et al., 2023b).
• Llama 2: Open Foundation and Fine-Tuned Chat Models (Touvron et al., 2023).
• Large Language Model Unlearning (Yao et al., 2023a).
• Towards Consistent Natural-Language Explanations via Explanation-Consistency Finetuning (Chen et al., 2024d).
• From Persona to Personalization: A Survey on Role-Playing Language Agents (Chen et al., 2024b).
• Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptation (Halawi et al., 2024b).
• Characterizing stable regions in the residual stream of LLMs (Janiak et al., 2024).
• A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity (Lee et al., 2024).
• Eight Methods to Evaluate Robust Unlearning in LLMs (Lynch et al., 2024).
• What Are Step-Level Reward Models Rewarding? Counterintuitive Findings from MCTS-Boosted Mathematical Reasoning (Ma et al., 2024).
• Rule Based Rewards for Language Model Safety (Mu et al., 2024).
• The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale (Penedo et al., 2024).
• Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Tracking (Prakash et al., 2024).
• UnUnlearning: Unlearning is not sufficient for content regulation in advanced generative AI (Shumailov et al., 2024b).
• Improving Pretraining Data Using Perplexity Correlations (Thrush et al., 2024).
• QuRating: Selecting High-Quality Data for Training Language Models (Wettig et al., 2024).
• The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning (Agarwal et al., 2025).
• Values in the Wild: Discovering and Analyzing Values in Real-World Language Model Interactions (Huang et al., 2025c).
• Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Training (Langlais et al., 2025).
• Distillation Robustifies Unlearning (Lee et al., 2025).
• When Bad Data Leads to Good Models (Li et al., 2025i).
• Reasoning Under Pressure: How do Training Incentives Influence Chain-of-Thought Monitorability? (MacDermott et al., 2025).
• Generalizing Verifiable Instruction Following (Pyatkin et al., 2025).
• Emotions Where Art Thou: Understanding and Characterizing the Emotional Latent Space of Large Language Models (Reichman et al., 2025).
• Do Multilingual LLMs Think In English? (Schut et al., 2025).
• Large language models can learn and generalize steganographic chain-of-thought under process supervision (Skaf et al., 2025).
• Simple Mechanistic Explanations for Out-Of-Context Reasoning (Wang et al., 2025f).
• Ultra-FineWeb: Efficient Data Filtering and Verification for High-Quality LLM Training Data (Wang et al., 2025g).
• From Hard Refusals to Safe-Completions: Toward Output-Centric Safety Training (Yuan et al., 2025).
• Stress-Testing Model Specs Reveals Character Differences among Language Models (Zhang et al., 2025c). We generated 300,000+ queries testing value trade-offs in AI models from Anthropic, OpenAI, Google DeepMind, and xAI. Each model showed distinct value prioritization patterns, and we found thousands of cases of direct contradictions or interpretive ambiguities in model specifications.
• Learning to Reason without External Rewards (Zhao et al., 2025).
• ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases (Zhong et al., 2025).
• Value Leakage: An LLM's Answers Are Silently Shaped by Its Own Values (Betley et al., 2026b).
• Natively Unlearnable Large Language Models (Ghosal et al., 2026).
• Aligned, Orthogonal or In-conflict: When can we safely optimize Chain-of-Thought? (Kaufmann et al., 2026).
• Abstractive Red-Teaming of Language Model Character (Rahn et al., 2026). How can we surface realistic failures of model character prior to deployment? We introduce abstractive red-teaming, which searches for natural-language categories of user queries that reliably elicit character violations.
• Will we run out of data? Limits of LLM scaling based on human-generated data (Villalobos et al., 2024). Projects when the stock of public human text will be exhausted by training runs, a constraint on capability trajectories and a driver of synthetic data.
• Machine Psychology (Hagendorff et al., 2023). Proposes applying behavioral experiment methods from psychology to language models and sets out the methodological cautions for doing so.
• Evaluating and Inducing Personality in Pre-trained Language Models (Jiang et al., 2023a). Introduces a machine personality inventory and a prompting method that induces stable personality profiles, an early operationalization of model persona.
• Measuring and Controlling Instruction (In)Stability in Language Model Dialogs (Li et al., 2024c). Measures persona drift over long dialogs, traces it to attention decay on the system prompt, and proposes a mitigation, an empirical test of how stable an assigned character is.
• A psychometric framework for evaluating and shaping personality traits in large language models (Serapio-García et al., 2025). Validates psychometric personality tests on language models and shows that personality expressed in generated text can be shaped by prompting along validated dimensions.
• Investigating Cultural Alignment of Large Language Models (AlKhamissi et al., 2024). Finds that models align more closely with a culture's survey responses when prompted in that culture's dominant language and when pretrained on more of it, linking cultural alignment to training data composition.
• Having Beer after Prayer? Measuring Cultural Bias in Large Language Models (Naous et al., 2024). Shows that multilingual and Arabic language models favor entities associated with Western culture even when prompted in Arabic, a measured cultural bias in model outputs.
• Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains (Gunjal et al., 2026). Uses structured rubrics scored by a model judge as reward for RL on tasks without a verifiable answer, making the grader an explicit, inspectable specification.
• Generalization Analogies: A Testbed for Generalizing AI Oversight to Hard-To-Measure Domains (Clymer et al., 2023). Crafts 69 distribution shifts and finds that reward models do not learn to evaluate instruction-following by default but favor personas resembling internet text, a testbed for whether oversight generalizes where it cannot be checked.
• Introducing Model Spec Evals (Guo and Wolfe, 2026). Evaluating how well models follow the OpenAI Model Spec.
• CoVal: Learning values-aware rubrics from the crowd (Hitzig et al., 2026). An experimental dataset of crowd-written rubrics that surfaces why people prefer one model output over another.
• Interpreting Black Box Reward Models (Sodhi et al., 2026). ARGO distills black-box reward models into interpretable rubrics using reinforcement learning.
• Learning the Preferences of Ignorant, Inconsistent Agents (Evans et al., 2016). Infers preferences jointly with false beliefs and time inconsistency, showing that reward learning must model human biases or it misreads them as values.
• The Assistive Multi-Armed Bandit (Chan et al., 2019). Models a human who is still learning their own preferences while a robot that sees only their choices assists, and shows that a human who performs better alone need not be easier to assist, so learning from people depends on how they communicate what they learn.
• Quantifying Differences in Reward Functions (Gleave et al., 2021). Defines EPIC, a distance between reward functions that is invariant to potential shaping and does not require training a policy.
• Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations (Wang et al., 2024d). Trains a process reward model from automatically estimated step values (the chance a step leads to a correct answer), removing the need for human step labels.
• Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning (Setlur et al., 2025). Argues that a process reward should measure progress, the change in the probability of eventual success under a prover policy, and shows such rewards improve search and RL efficiency.
• Training Large Language Models to Reason in a Continuous Latent Space (Hao et al., 2024). Coconut feeds the last hidden state back as the next input instead of a token, so the model reasons in latent space; a design that removes the legible chain of thought monitors rely on.
• Learning agents for uncertain environments (extended abstract) (Russell, 1998). Poses inverse reinforcement learning as a problem: recover the reward function an observed agent is optimizing, the founding statement of learning objectives from behavior.
• Apprenticeship learning via inverse reinforcement learning (Abbeel and Ng, 2004). Read for this book only as a catalog record, so no chapter reports what it found.
• RewardBench: Evaluating Reward Models for Language Modeling (Lambert et al., 2025). First standard benchmark for reward models themselves, making the proxy at the center of RLHF an object of measurement.
• Length Penalties Make Chain-of-Thought Less Monitorable (Little, 2026). Trains reasoning models with RL under length penalties and finds that at matched accuracy the compressed chains less often verbalize the hints that changed the answer, a reward term that degrades monitorability as a side effect; it should be read against MacDermott et al. 2025, who found no consistent effect of length penalties.
• Good and safe uses of AI Oracles (Armstrong and O'Rorke, 2017). Designs counterfactual and low-bandwidth oracles whose answers are scored only in worlds where nobody reads them, removing the incentive to manipulate the reader.
• Safe Model-based Reinforcement Learning with Stability Guarantees (Berkenkamp et al., 2017). Uses Lyapunov stability and statistical models of the dynamics to expand a provably safe region during learning, a template for exploration with guarantees.
• Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning (Eysenbach et al., 2018). Trains a reset policy alongside the task policy and aborts actions the reset policy cannot undo, an early operational form of reversibility as a safety constraint.
• Trial without Error: Towards Safe Reinforcement Learning via Human Intervention (Saunders et al., 2018). Has a human block catastrophic actions during training and then trains a classifier to imitate the blocker, an early demonstration of oversight that scales by learning to replace the overseer and of where it fails.
• Benchmarking Safe Exploration in Deep Reinforcement Learning (Ray et al., 2019). Introduces the Safety Gym environments and proposes constrained reinforcement learning as the formalism for safe exploration, the benchmark most later safe exploration work reports on.
• Pessimism About Unknown Unknowns Inspires Conservatism (Cohen and Hutter, 2020). Defines an idealized Bayesian agent that maximizes worst-case expected reward over a set of world models and proves that a sufficiently pessimistic agent avoids unprecedented events with high probability, a formal basis for conservatism.
• Avoiding Side Effects By Considering Future Tasks (Krakovna et al., 2020a). Defines a side-effect penalty as the loss of ability to complete possible future tasks and shows it avoids the interference incentive of earlier reachability penalties.
• Avoiding Side Effects in Complex Environments (Turner et al., 2020a). Scales attainable utility preservation from gridworlds to a procedurally generated environment by preserving the ability to optimize random auxiliary reward functions, showing impact penalties need not be hand-specified per task.
• Avoiding Negative Side Effects Due to Incomplete Knowledge of AI Systems (Saisubramanian et al., 2022). Surveys negative side effects that arise because an agent model of its environment is inevitably incomplete, and organizes their forms and the research on avoiding them.
• Provably safe systems: the only path to controllable AGI (Tegmark and Omohundro, 2023). Argues that only systems whose safety properties are formally proved, with proofs checked by trusted verifiers, can be controlled once AI exceeds its overseers, the strongest statement of the guaranteed safe AI program.
• Safeguarded AI (Advanced Research and Invention Agency (ARIA), 2024). The UK ARIA research programme that funds gatekeeper systems which produce quantitative safety guarantees from formal world models, the largest public bet on guaranteed safe AI.
• Safe RLHF: Safe Reinforcement Learning from Human Feedback (Dai et al., 2024). Decouples helpfulness and harmlessness into a reward model and a cost model and trains a language model with a Lagrangian constrained objective, carrying constrained reinforcement learning into preference fine-tuning.
• A Review of Safe Reinforcement Learning: Methods, Theories, and Applications (Gu et al., 2024a). A review of safe reinforcement learning organized around five problems for real-world deployment, updating the 2015 survey for deep reinforcement learning methods, theory, and applications.
• Amortizing intractable inference in large language models (Hu et al., 2024). Fine-tunes language models as generative flow network samplers of intractable posteriors, the Bayesian machinery Scientist AI proposals rely on for calibrated, non-agentic prediction.
• Corrigibility as a Singular Target: A Vision for Inherently Reliable Foundation Models (Potham and Harms, 2025). Proposes training foundation models whose overriding objective is to empower designated principals to guide, correct, and control them, as an alternative to balancing corrigibility against other values.
• AI Alignment: A Comprehensive Survey (Ji et al., 2023). Broad survey that organizes alignment into forward alignment (training) and backward alignment (assurance and governance), with the RICE objectives.
• Towards Making Systems Forget with Machine Unlearning (Cao and Yang, 2015). Introduces machine unlearning: transform learning algorithms into summation form so that removing training data does not require retraining.
• Machine Unlearning (Bourtoule et al., 2021). SISA training: shard, isolate, slice, and aggregate so that exact unlearning retrains only part of the model.
• TOFU: A Task of Fictitious Unlearning for LLMs (Maini et al., 2024). Benchmark that fine-tunes models on fictitious author profiles so unlearning can be measured against a known ground truth.
• Machine Unlearning Doesn't Do What You Think: Lessons for Generative AI Policy and Research (Cooper et al., 2025). Separates removing information from a model from suppressing outputs and argues that unlearning cannot by itself deliver the policy goals often attached to it.
• The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models (Cui et al., 2025). Shows that policy entropy collapses early in RL on reasoning tasks and that performance is traded against entropy along a predictable curve, explaining the loss of exploration the pass@k results observe.

F.5 Machine learning: How Instillation Fails #

Spine.

• Concrete Problems in AI Safety (Amodei et al., 2016). Names the five problems (side effects, reward hacking, scalable oversight, safe exploration, distributional shift) that turned accident risk into a machine learning research agenda.
• Risks from Learned Optimization in Advanced Machine Learning Systems (Hubinger et al., 2019). Introduces mesa-optimization, the inner alignment problem, and deceptive alignment as a possible outcome of training a learned optimizer.
• Shortcut learning in deep neural networks (Geirhos et al., 2020). Unifies many deep learning failures as shortcuts, decision rules that work on the training distribution but not under the intended conditions.
• Specification gaming: the flip side of AI ingenuity (Krakovna et al., 2020b). Defines specification gaming as satisfying the literal objective without the intended outcome and draws on the community-maintained list of examples to separate it from goal misgeneralization.
• Underspecification Presents Challenges for Credibility in Modern Machine Learning (D'Amour et al., 2022). Shows that pipelines yield many predictors equivalent in held-out performance that behave differently in deployment, the statistical root of goal misgeneralization.
• Goal Misgeneralization in Deep Reinforcement Learning (Langosco et al., 2022). First empirical demonstrations of agents that keep their capabilities out of distribution but pursue the wrong goal, such as the CoinRun agent that runs to the end of the level instead of to the coin.
• Discovering Language Model Behaviors with Model-Written Evaluations (Perez et al., 2022a). We develop an automated way to generate language model (LM) evaluations with LMs, significantly reducing the effort involved. We test LMs using >150 LM-written evaluations, uncovering novel LM behaviors.
• Defining and Characterizing Reward Hacking (Skalse et al., 2022). Formal definition of an unhackable proxy, with the result that over all stationary policies two reward functions are unhackable only if they order policies identically or one gives every policy the same value.
• Taken out of context: On measuring situational awareness in LLMs (Berglund et al., 2023a). Out-of-context reasoning: models apply facts learned from training documents to tests without seeing them in the prompt, a measurable precursor of situational awareness.
• Scheming AIs: Will AIs fake alignment during training in order to get power? (Carlsmith, 2023). The most detailed analysis of whether training would produce models that perform well instrumentally in order to gain power later, with the arguments for and against.
• Role-Play with Large Language Models (Shanahan et al., 2023). Cited by 5 laboratory works.
• Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models (Denison et al., 2024). Can minor specification gaming evolve into more dangerous behaviors? This paper demonstrates that models trained on low-level reward hacking—like sycophancy—can generalize to tampering with their own reward functions, even covering their tracks. The behavior emerged without explicit training, and common safety techniques reduced but didn't eliminate it.
• Alignment faking in large language models (Greenblatt et al., 2024b). This paper provides the first empirical example of a model engaging in alignment faking without being trained to do so—selectively complying with training objectives while strategically preserving existing preferences.
• Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification (Kwa et al., 2024). Proves that with light-tailed reward error KL-regularized optimization still gains true utility, but with heavy-tailed error some policies reach arbitrarily high proxy reward with no more utility than the base model; current reward models measured light-tailed.
• When Your AIs Deceive You: Challenges of Partial Observability in Reinforcement Learning from Human Feedback (Lang et al., 2024b). Proves that when evaluators see only part of the environment, RLHF can produce deceptive inflation and overjustification, and characterizes when the true reward is identifiable.
• The Alignment Problem from a Deep Learning Perspective (Ngo et al., 2024). States the case that models trained with RLHF-style methods could learn situationally aware reward hacking, misaligned internally represented goals, and power-seeking, in deep learning terms.
• Robust agents learn causal world models (Richens and Everitt, 2024). Proves that an agent that achieves low regret under a large set of distributional shifts must have learned an approximate causal model of its environment, which ties robustness to the presence of a world model.
• Unintended Impacts of LLM Alignment on Global Representation (Ryan et al., 2024). Shows that RLHF and DPO preference tuning changes performance across English dialects and shifts which countries' opinions models reproduce, evidence that the alignment procedure itself distributes representation unequally.
• Recontextualization Mitigates Specification Gaming without Modifying the Specification (Azarbal et al., 2025). Cited by 5 laboratory works.
• Tell me about yourself: LLMs are aware of their learned behaviors (Betley et al., 2025a). Cited by 6 laboratory works.
• RepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agents (Black et al., 2025). Decomposes autonomous replication into obtaining resources, exfiltrating weights, replicating onto compute, and persisting, and measures frontier agents on each component.
• Persona Vectors: Monitoring and Controlling Character Traits in Language Models (Chen et al., 2025c). Cited by 7 laboratory works.
• MONA: Myopic Optimization with Non-myopic Approval Can Mitigate Multi-step Reward Hacking (Farquhar et al., 2025). Trains language model agents on short-horizon optimization plus an overseer foresight-based approval and shows this prevents multi-step reward hacking that ordinary reinforcement learning finds, at a measured performance cost.
• Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking (Laidlaw et al., 2025). Defines reward hacking as the breakdown, under optimization, of the proxy-true reward correlation on states and actions seen by a reference policy, shows that regularizing toward that policy can prevent it, and argues that the chi-squared divergence of occupancy measures is a better regularizer than the action-distribution KL used in RLHF.
• Open Character Training: Shaping the Persona of AI Assistants through Constitutional AI (Maiya et al., 2025). The first open implementation of character training, using constitutional AI and synthetic introspective data to instill personas in open-weight models and measuring the resulting character change through revealed preferences.
• Large Language Models Often Know When They Are Being Evaluated (Needham et al., 2025). Cited by 7 laboratory works.
• Stress Testing Deliberative Alignment for Anti-Scheming Training (Schoen et al., 2025). Cited by 11 laboratory works.
• Why Do Some Language Models Fake Alignment While Others Don't? (Sheshadri et al., 2025). We investigate why in the scenarios where Claude 3 Opus fakes alignment many other language models don't do so.
• Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-time (Tan et al., 2025). Cited by 6 laboratory works.
• Auditing Games for Sandbagging (Taylor et al., 2025a). Cited by 6 laboratory works.
• Model Organisms for Emergent Misalignment (Turner et al., 2025). Cited by 6 laboratory works.
• Persona Features Control Emergent Misalignment (Wang et al., 2025e). Cited by 14 laboratory works.
• The persona selection model (Anthropic, 2026au). Why would AI assistants behave like they’re human? We articulate a theory—drawing on ideas discussed by many others—that might help explain why modern AI training tends to create human-like AIs.
• Training large language models on narrow tasks can lead to broad misalignment (Betley et al., 2026a). Cited by 13 laboratory works.
• Metagaming matters for training, evaluation, and oversight (Schoen and Nitishinskaya, 2026). Metagaming can complicate how we interpret behavior, and current models still give us a chance to study it directly.

Side reading.

• Tell, don't show: Declarative facts influence how LLMs generalize (Meinke and Evans, 2023).
• Uncovering mesa-optimization algorithms in Transformers (von Oswald et al., 2023).
• Benchmarks for Detecting Measurement Tampering (Roger et al., 2023).
• Simple synthetic data reduces sycophancy in large language models (Wei et al., 2023d).
• Uncovering Deceptive Tendencies in Language Models: A Simulated Company AI Assistant (Järviniemi and Hubinger, 2024).
• Planning in a recurrent neural network that plays Sokoban (Taufeeque et al., 2024).
• Noise Injection Reveals Hidden Capabilities of Sandbagging Language Models (Tice et al., 2024).
• Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuning (Casademunt et al., 2025).
• Thought Crime: Backdoors and Emergent Misalignment in Reasoning Models (Chua et al., 2025).
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (DeepSeek-AI et al., 2025).
• Inverse Scaling in Test-Time Compute (Gema et al., 2025). We investigate inverse scaling when LLMs have access to more test-time compute.
• Detecting Strategic Deception Using Linear Probes (Goldowsky-Dill et al., 2025).
• Consistency Training Helps Stop Sycophancy and Jailbreaks (Irpan et al., 2025).
• Recent Frontier Models Are Reward Hacking (Metr, 2025d). An evaluator's report of frontier models modifying tests and scoring code during its task evaluations, a field observation of reward hacking outside training.
• Narrow Finetuning Leaves Clearly Readable Traces in Activation Differences (Minder et al., 2025a).
• Convergent Linear Representations of Emergent Misalignment (Soligo et al., 2025).
• School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMs (Taylor et al., 2025b).
• Teaching Models to Verbalize Reward Hacking in Chain-of-Thought Reasoning (Turpin et al., 2025).
• The Hot Mess of AI: How Does Misalignment Scale With Model Intelligence and Task Complexity? (Hägele et al., 2026). When AI systems fail, will they fail by systematically pursuing goals we do not intend? Or will they fail by being a hot mess—taking nonsensical actions that do not further any goal?
• Measuring Reward-Seeking via Contrastive Belief Updates (Højmark et al., 2026). From the laboratory's post "Measuring Reward-Seeking by Instilling Contrastive Beliefs": Testing whether behavior changes when a model believes its grader wants something different.
• Reinforcement Learning Towards Broadly and Persistently Beneficial Models (Jagadeesh et al., 2026). Testing how alignment improvements hold across domains and under adversarial pressure.
• Countdown-Code: A Testbed for Studying The Emergence and Generalization of Reward Hacking in RLVR (Khalifa et al., 2026).
• Model Spec Midtraining: Improving How Alignment Training Generalizes (Li et al., 2026). We train AIs to understand the content of their model spec. This shapes and improves how they generalize from subsequent alignment training.
• The Assistant Axis: Situating and Stabilizing the Default Persona of Language Models (Lu et al., 2026).
• Chunky Post-Training: Data Driven Failures of Generalization (Murray et al., 2026).
• Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment (Tice et al., 2026).
• Gradient Routing Localizes and Removes Unintended Behaviors in RL (Ward et al., 2026). From the laboratory's post "Reducing Reward Hacking by Localizing Bad Behavior Instead of Penalizing It": When a model is caught cheating on a task during training, we confine its learning to a separate set of parameters. After training, we delete those parameters.
• General agents need world models (Richens et al., 2025). Proves that any agent able to generalize to a broad class of multi-step goal-directed tasks must have learned a predictive model of its environment, extractable from its policy, with accuracy improving with performance.
• Training on Documents about Reward Hacking Induces Reward Hacking (Anthropic, 2025ar). Does training on documents which discuss (but do not demonstrate) reward hacks affect a model's propensity to reward hack?
• Towards training-time mitigations for alignment faking in RL (Gasteiger et al., 2025b). We construct a diverse array of model organisms of alignment faking and study mitigations that could be used during RL to decrease alignment faking and compliance gaps.
• Alignment Faking Revisited: Improved Classifiers and Open Source Extensions (Hughes et al., 2025). We present a replication and extension of an alignment faking model organism.
• Won't vs. Can't: Sandbagging-like Behavior from Claude Models (Witten and Benton, 2025). We find that, when Claude models are presented with a harmful version of a task they can otherwise perform, they sometimes claim they lack the ability to perform the task, rather than simply refusing to do it. We discuss the implications of this behavior for AI safety.
• Training a Misaligned Reward Seeker (Anthropic, 2026aw). To better understand the impact of reward hacking on model behavior, we trained an Opus-class model with large-scale RL on many production environments vulnerable to reward hacks. We consider this a plausible proxy for what a real training run might look like had we not invested significant effort into preventing and detecting reward hacking in our normal training runs. Our results show that a high rate of reward hacking during RL can cause models to be willing to perform long sequences of harmful real-world actions in pursuit of task success.
• AI models may be capable of self-replicating without access to a datacenter (Anthropic, 2026c). Our experiments suggest that small AI models may be only months away from being capable enough to self-replicate on commercial IoT devices, like Android TVs and smart ovens.
• Measuring Goal-Directedness (Macdermott et al., 2024). Defines maximum entropy goal-directedness in causal models, a formal measure of how well behavior is explained by optimizing some utility.
• Frontier AI systems have surpassed the self-replicating red line (Pan et al., 2024b). Reports open-weight models creating running copies of themselves in agent scaffolds, a claim whose force depends on how the scaffold and the replication task are defined.
• Personas as a Way to Model Truthfulness in Language Models (Joshi et al., 2024). Proposes that pretraining data contains agent personas and that models infer a truthful persona from context, explaining how truthfulness generalizes across topics.
• The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harm (Aakanksha et al., 2024). Collects human-annotated red-teaming prompts in several languages, distinguishing global from local harms, and finds that preference training overfits to harms common in Western-centric data, with methods that balance both.
• Honesty Is the Best Policy: Defining and Mitigating AI Deception (Ward et al., 2023). Formal definition of intentional deception in structural causal games, with graphical criteria and training-time mitigations.
• Stress-Testing Capability Elicitation With Password-Locked Models (Greenblatt et al., 2024c). Model organisms that hide capabilities unless given a password, used to measure whether fine-tuning elicits hidden capabilities; a few demonstrations usually suffice.
• Deception abilities emerged in large language models (Hagendorff, 2024). Finds that larger language models understand and can induce false beliefs in other agents on first- and second-order deception tasks.
• AI deception: A survey of examples, risks, and potential solutions (Park et al., 2024a). Defines deception behaviorally as systematically inducing false beliefs in pursuit of an outcome and surveys cases in game-playing and language systems.
• Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark (Pan et al., 2023). Text-game benchmark that measures power-seeking, deception, and ethical violations by reward-maximizing agents, an early propensity evaluation.
• Exploration Hacking: Can LLMs Learn to Resist RL Training? (Jang et al., 2026). Builds model organisms that withhold exploration to resist RL-based capability elicitation while keeping related performance, and finds frontier models can reason about suppressing exploration when told they are in training.
• An overview of 11 proposals for building safe advanced AI (Hubinger, 2020). Compares eleven alignment proposals (amplification, debate, recursive reward modeling, microscope AI, and others) on outer alignment, inner alignment, training competitiveness, and performance competitiveness, the four criteria later agendas reuse.
• Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference (McCoy et al., 2019). HANS shows that high-scoring NLI models rely on lexical-overlap heuristics, a clean case of benchmark success masking the wrong learned rule.
• Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals (Shah et al., 2022). Argues and shows across several systems that a correct reward still permits learning an undesired goal consistent with training data, separating inner from outer misspecification.
• Understanding and Controlling a Maze-Solving Policy Network (Mini et al., 2023). Interprets the goal-misgeneralizing maze agent and finds internal goal representations that can be steered, linking goal misgeneralization to interpretability.
• Spurious Rewards: Rethinking Training Signals in RLVR (Shao et al., 2026). Finds that RLVR with random or incorrect rewards still improves some model families on math benchmarks, so gains can come from surfacing pretrained behaviors rather than from the reward's content.
• How RL Agents Behave When Their Actions Are Modified (Langlois and Everitt, 2021). Shows with modified-action MDPs that different reinforcement learning algorithms respond differently to human interventions on their actions, some learning to avoid or exploit oversight, so the incentive depends on the learning rule.
• Evidence of Learned Look-Ahead in a Chess-Playing Neural Network (Jenner et al., 2024). Finds that a strong chess policy network internally represents and uses future moves, evidence that trained networks can implement look-ahead rather than only heuristics, which bears on mesa-optimization.
• Debugging misaligned completions with sparse-autoencoder latent attribution (la Tour and Mossing, 2025). Efficiently finding features that cause behaviors.
• Helpful assistant features suppress emergent misalignment (la Tour, 2025). Emergent misalignment not only activates misaligned personas, but also suppresses helpful assistant personas.
• How far does alignment midtraining generalize? (Korbak et al., 2026). Experiments spanning midtraining, posttraining, and generalization.
• Studying metagaming latents in language models (Xu et al., 2026). Reasoning about how a task will be monitored or rewarded appears to draw on several overlapping processes, not a single mechanism.
• Causal Confusion and Reward Misidentification in Preference-Based Reward Learning (Tien et al., 2023). Shows that reward models trained on preferences latch onto spurious correlates of preferred behavior and that policies optimized against them fail out of distribution, an empirical form of what a reward model cannot identify.
• Self-Modification of Policy and Utility Function in Rational Agents (Everitt et al., 2016). Shows which agent designs keep their goals under the option to modify their own policy or utility function, a formal treatment of goal preservation.
• Asymptotically Unambitious Artificial General Intelligence (Cohen et al., 2020). Constructs a boxed, episodic agent whose optimal policy provably has no incentive to influence the outside world beyond the episode, a formal myopia argument against power-seeking.
• Path-Specific Objectives for Safer Agent Incentives (Farquhar et al., 2022). Uses causal influence diagrams to train agents that optimize only along designated causal paths, removing incentives to manipulate parts of the environment such as the user.
• The Pitfalls of Simplicity Bias in Neural Networks (Shah et al., 2020). Shows that networks rely exclusively on the simplest predictive feature and ignore more complex equally predictive ones, which makes them brittle under shift and is a mechanism for shortcut learning.
• Specification gaming examples in AI (Krakovna, 2018). The running catalog of observed specification gaming cases across reinforcement learning, evolution, and later language models, the field's main empirical record of the phenomenon.
• The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities (Lehman et al., 2020). Collection of cases where evolutionary search satisfied fitness functions in unintended ways, evidence that specification gaming is a property of optimization, not of neural networks.
• Reward tampering problems and solutions in reinforcement learning: a causal influence diagram perspective (Everitt et al., 2021b). Uses causal influence diagrams to distinguish tampering with the reward function from tampering with its inputs, and identifies agent designs without tampering incentives.
• The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models (Pan et al., 2022). Systematic study across environments showing that more capable agents exploit misspecified rewards more, including phase transitions where true reward collapses abruptly.
• Language Models Learn to Mislead Humans via RLHF (Wen et al., 2025a). Finds that RLHF raises human approval more than correctness on question answering and programming, making models better at convincing evaluators of wrong answers.
• AI Safety Gridworlds (Leike et al., 2017). Small reinforcement learning environments that operationalize safe interruptibility, side effects, reward gaming, and distributional shift, separating the hidden safety score from the visible reward.

F.6 Machine learning: Verification from Outside #

Spine.

• IP = PSPACE (Shamir, 1992). Proves that every problem solvable in polynomial space has an interactive proof with a polynomial-time verifier, the result debate appeals to when it claims a weak judge can check a strong prover.
• AI safety via debate (Irving et al., 2018). Proposes training agents via a zero-sum debate judged by a human, arguing that optimal play can let a polynomial-time judge decide PSPACE questions.
• Recursively Summarizing Books with Human Feedback (Wu et al., 2021b). Cited by 6 laboratory works.
• Measuring Progress on Scalable Oversight for Large Language Models (Bowman et al., 2022). We show how humans could use AI systems to better oversee other AI systems, and demonstrate some proof-of-concept results where a language model improves human performance at a task.
• Discovering Latent Knowledge in Language Models Without Supervision (Burns et al., 2022). Cited by 10 laboratory works.
• Self-critiquing models for assisting human evaluators (Saunders et al., 2022). Cited by 7 laboratory works.
• The Internal State of an LLM Knows When It’s Lying (Azaria and Mitchell, 2023). Trains classifiers on a language model's hidden activations to predict whether a statement it reads or writes is true.
• Measuring Faithfulness in Chain-of-Thought Reasoning (Lanham et al., 2023). We investigate hypotheses for how language models' Chain-of-Thought (CoT) reasoning may be unfaithful, by examining how the model predictions change when we intervene on the CoT (e.g., by adding mistakes or paraphrasing it).
• Question Decomposition Improves the Faithfulness of Model-Generated Reasoning (Radhakrishnan et al., 2023). To improve the faithfulness of Chain-of-Thought (CoT) reasoning, we have models generate their CoT by decomposing questions into subquestions.
• Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting (Turpin et al., 2023). Cited by 11 laboratory works.
• Scalable AI Safety via Doubly-Efficient Debate (Brown-Cohen et al., 2024). Debate protocols in which an honest strategy can win using a polynomial number of steps even against a dishonest opponent with exponential compute.
• Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision (Burns et al., 2024). Strong models fine-tuned on labels from weaker models recover part of the gap to strong-label performance, an empirical analogy for humans supervising superhuman models.
• Prover-Verifier Games improve legibility of LLM outputs (Kirchner et al., 2024). Cited by 5 laboratory works.
• Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMs (Laine et al., 2024). Benchmark of seven task categories measuring whether models know facts about themselves and their deployment situation.
• Theoretical Analysis of Weak-to-Strong Generalization (Lang et al., 2024a). Explains weak-to-strong gains by pseudolabel correction and coverage expansion under checkable expansion properties of the data, so the strong student improves when it cannot fit the weak teacher's errors without extra error.
• Chain-of-Thought Reasoning In The Wild Is Not Always Faithful (Arcuschin et al., 2025). Cited by 7 laboratory works.
• Avoiding Obfuscation with Prover-Estimator Debate (Brown-Cohen et al., 2025). Addresses the obfuscated arguments problem, in which a dishonest debater forces an honest one to solve an intractable problem, with a recursive debate protocol under which, given stability assumptions, honesty needs only comparable computation.
• Reasoning Models Don't Always Say What They Think (Chen et al., 2025d). We find that reasoning models don't always accurately verbalize their reasoning. This casts doubt on whether monitoring chains-of-thought (CoT) will be enough to reliably catch safety issues.
• When Chain of Thought is Necessary, Language Models Struggle to Evade Monitors (Emmons et al., 2025). Cited by 7 laboratory works.
• Monitoring Monitorability (Guan et al., 2025). From the laboratory's post "Open Sourcing Monitorability Evaluations": Datasets, code, and an evaluation filtering strategy for studying monitorability.
• Steering Evaluation-Aware Language Models to Act Like They Are Deployed (Hua et al., 2025). Cited by 5 laboratory works.
• Training LLMs for Honesty via Confessions (Joglekar et al., 2025). From the laboratory's post "Why we are excited about confessions": Deeper analysis of confession training and comparisons to chain-of-thought monitoring.
• Training fails to elicit subtle reasoning in current language models (Wagner et al., 2025). We investigated whether language models can reason about malicious tasks while evading detection, finding that monitoring both reasoning and outputs successfully prevents this in current models.
• Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge (Wallach et al., 2025). Argues that capability and safety evaluations measure abstract constructs and should adopt the social sciences framework of systematization, operationalization, and validity.

Side reading.

• RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models (Gehman et al., 2020).
• Evaluating Large Language Models Trained on Code (Chen et al., 2021).
• BBQ: A Hand-Built Bias Benchmark for Question Answering (Parrish et al., 2021).
• Calibrate Before Use: Improving Few-Shot Performance of Language Models (Zhao et al., 2021a).
• Text and Patterns: For Effective Chain of Thought, It Takes Two to Tango (Madaan and Yazdanbakhsh, 2022).
• Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought (Saparov and He, 2022).
• Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them (Suzgun et al., 2022).
• Self-Consistency Improves Chain of Thought Reasoning in Language Models (Wang et al., 2022a).
• Evaluating Superhuman Models with Consistency Checks (Fluri et al., 2023).
• Automatically Auditing Large Language Models via Discrete Optimization (Jones et al., 2023).
• Evaluating Language-Model Agents on Realistic Autonomous Tasks (Kinniment et al., 2023).
• Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness? (Liu et al., 2023).
• Faithful Chain-of-Thought Reasoning (Lyu et al., 2023).
• ALMANACS: A Simulatability Benchmark for Language Model Explainability (Mills et al., 2023).
• How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questions (Pacchiardi et al., 2023).
• GPQA: A Graduate-Level Google-Proof Q&A Benchmark (Rein et al., 2023).
• Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models (Wang et al., 2023d).
• RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models (Wang et al., 2023e).
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023a).
• Instruction-Following Evaluation for Large Language Models (Zhou et al., 2023a).
• AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents (Andriushchenko et al., 2024a).
• Truth is Universal: Robust Detection of Lies in LLMs (Bürger et al., 2024).
• Bias-Augmented Consistency Training Reduces Biased Reasoning in Chain-of-Thought (Chua et al., 2024).
• From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Step (Deng et al., 2024).
• Easy2Hard-Bench: Standardized Difficulty Labels for Profiling LLM Performance and Generalization (Ding et al., 2024).
• FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI (Glazer et al., 2024).
• Balancing Label Quantity and Quality for Scalable Elicitation (Mallen and Belrose, 2024).
• LLM Critics Help Catch LLM Bugs (McAleese et al., 2024).
• PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Action (Shao et al., 2024b).
• CYBERSECEVAL 3: Advancing the Evaluation of Cybersecurity Risks and Capabilities in Large Language Models (Wan et al., 2024).
• MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark (Wang et al., 2024e).
• LiveBench: A Challenging, Contamination-Limited LLM Benchmark (White et al., 2024).
• Super(ficial)-alignment: Strong Models May Deceive Weak Models in Weak-to-Strong Generalization (Yang et al., 2024c). Reports that a strong student can look aligned where the weak supervisor can check and misbehave where it cannot.
• Weak-to-Strong Reasoning (Yang et al., 2024d).
• Evaluation of OpenAI o1: Opportunities and Challenges of AGI (Zhong et al., 2024b).
• Claude Sonnet 3.7 (often) knows when it's in alignment evaluations (Apollo Research, 2025). An evaluator's report that a frontier model often states in its reasoning that a scheming evaluation is a test of its alignment.
• HealthBench: Evaluating Large Language Models Towards Improved Human Health (Arora et al., 2025).
• Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents (Backlund and Petersson, 2025).
• $τ^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment (Barres et al., 2025).
• Finance Agent Benchmark: Benchmarking LLMs on Real-world Financial Research Tasks (Bigeard et al., 2025).
• Evaluation Awareness Scales Predictably in Open-Weights Large Language Models (Chaudhary et al., 2025).
• Evaluating o1-Like LLMs: Unlocking Reasoning for Translation through Comprehensive Analysis (Chen et al., 2025a).
• Are DeepSeek R1 And Other Reasoning Models More Faithful? (Chua and Evans, 2025).
• Output Supervision Can Obfuscate the Chain of Thought (Drori et al., 2025).
• Unsupervised decoding of encoded reasoning using language model interpretability (Fang and Marks, 2025).
• Learning to Interpret Weight Differences in Language Models (Goel et al., 2025b).
• Reasoning Models Sometimes Output Illegible Chains of Thought (Jose, 2025).
• Liars' Bench: Evaluating Lie Detectors for Language Models (Kretschmar et al., 2025).
• Debate Helps Weak-to-Strong Generalization (Lang et al., 2025). Combines debate between strong models with weak supervision and reports better weak-to-strong generalization than weak supervision alone.
• PrivaCI-Bench: Evaluating Privacy with Contextual Integrity and Legal Compliance (Li et al., 2025e).
• Spilling the Beans: Teaching LLMs to Self-Report Their Hidden Objectives (Li et al., 2025f).
• CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratings (Quan et al., 2025).
• The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems (Ren et al., 2025).
• Difficulties with Evaluating a Deception Detector for AIs (Smith et al., 2025a).
• BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents (Wei et al., 2025).
• Unsupervised Elicitation of Language Models (Wen et al., 2025c).
• When More is Less: Understanding Chain-of-Thought Length in LLMs (Wu et al., 2025f).
• Watch the Weights: Unsupervised monitoring and control of fine-tuned LLMs (Zhong and Raghunathan, 2025).
• Early Signs of Steganographic Capabilities in Frontier LLMs (Zolkowski et al., 2025b).
• Can Reasoning Models Obfuscate Reasoning? Stress-Testing Chain-of-Thought Monitorability (Zolkowski et al., 2025a).
• Three Concrete Challenges and Two Hopes for the Safety of Unsupervised Elicitation (Canavan et al., 2026). From the laboratory's post "3 Challenges and 2 Hopes for the Safety of Unsupervised Elicitation": We stress-test unsupervised elicitation and easy-to-hard techniques on new datasets meant to capture realistic challenges such techniques would likely face.
• Counterfactual Simulation Training for Chain-of-Thought Faithfulness (Hase and Potts, 2026).
• MonitoringBench: Semi-Automated Red-Teaming for Agent Monitoring (Jotautaitė et al., 2026).
• Debate Training Reduces Reward Hacking in RLAIF (Kenton et al., 2026).
• Training Agents to Self-Report Misbehavior (Lee et al., 2026). Training agents to report covert misbehavior through a dedicated tool.
• Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces (Merrill et al., 2026).
• Training Alignment Auditors via Reinforcement Learning (Rosu and Wang, 2026). We train LLMs to investigate target models with hidden behaviors planted in their system prompt. A trained Haiku 4.5 policy matches Opus 4.6 on held-out evaluations for auditing quality and generalizes to harder fine-tuned targets in a different scaffold.
• Introspection Adapters: Training LLMs to Report Their Learned Behaviors (Shenoy et al., 2026). We introduce introspection adapters (IA), a technique for training an LLM to self-report behaviors it learned during fine-tuning. This IA generalizes to models that were fine-tuned in very different ways.
• Reasoning Models Struggle to Control their Chains of Thought (Yueh-Han et al., 2026). Why a limitation of frontier models is reassuring for AI safety.
• Flames: Benchmarking Value Alignment of LLMs in Chinese (Huang et al., 2024b). Builds a Chinese value-alignment benchmark combining harmlessness with a morality dimension that includes Chinese values such as harmony, and finds poor performance across mainstream models, an example of safety research specifying values from a different tradition.
• SafetyBench: Evaluating the Safety of Large Language Models (Zhang et al., 2024f). Builds a multiple-choice safety benchmark in Chinese and English across seven categories of safety concerns, a widely used evaluation from Chinese safety research.
• Evaluating honesty and lie detection techniques on a diverse suite dishonest models (Anthropic, 2025v). We explore techniques for honesty elicitation and lie detection on a diverse testbed of dishonest model organisms.
• Open Source Replication of the Auditing Game Model Organism (Anthropic, 2025ah). We release an open source replication of the model organism from our previous auditing game paper.
• Unsupervised Elicitation (Anthropic, 2025as). We introduce a new unsupervised algorithm for eliciting skills from pretrained language models.
• Do reasoning models use their scratchpad like we do? Evidence from distilling paraphrases (Roger, 2025). We provide some evidence that Claude 3.7 Sonnet doesn't encode hidden reasoning in its scratchpad by showing that training it to use paraphrased versions of the scratchpads does not degrade performance.
• Fine-Tuned Lie Detectors Failed to Generalize (Anthropic, 2026w). We trained lie detectors on on-policy lies from open-source models, but they didn’t generalize well to out-of-distribution lies.
• Pre-deployment auditing can catch an overt saboteur (Anthropic, 2026am). We test whether our pre-deployment alignment auditing methods can catch models trained to overtly sabotage Anthropic.
• AI Agents That Matter (Kapoor et al., 2024a). Shows that agent benchmarks reward accuracy without cost control and overfit through missing holdouts, and proposes joint cost-accuracy evaluation.
• How Does Time Horizon Vary Across Domains? (Metr, 2025c). Estimates agent time horizons from nine existing benchmarks outside the original software suite to test whether the exponential growth in autonomy measured on software tasks holds in other domains.
• HCAST: Human-Calibrated Autonomy Software Tasks (Rein et al., 2025). A task suite calibrated by timing skilled humans on each task, the human baseline on which time-horizon measurements of agent autonomy rest.
• The Hawthorne Effect in Reasoning Models: Evaluating and Steering Test Awareness (Abdelnabi and Salem, 2025). Locates test-awareness directions in reasoning models and shows that steering them changes harmful-task compliance, so awareness causally affects measured safety.
• Evaluation Faking: Unveiling Observer Effects in Safety Evaluation of Frontier AI Systems (Fan et al., 2025). Finds that models recognize unprompted that they are being evaluated and then behave more safely, and that this evaluation faking is more frequent in reasoning models, larger models, and models with memory.
• AI Awareness (Li et al., 2025b). Reviews metacognition, self-awareness, social awareness, and situational awareness in language models as functional capacities with separate evaluations.
• Rethink reporting of evaluation results in AI (Burnell et al., 2023). Argues that aggregate metrics hide failure patterns and calls for instance-level reporting, so that others can test claims about what a system can do.
• We Need A ‘Science of Evals’ (Apollo Research, 2024). Apollo Research argument that evaluation results lack the reliability needed for high-stakes decisions and a list of open problems for a mature evaluation discipline.
• Measuring what Matters: Construct Validity in Large Language Model Benchmarks (Bean et al., 2025). Reviews hundreds of benchmarks and finds widespread weaknesses in phenomenon definition, task sampling, and statistical reporting, with recommendations for construct validity.
• Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation (Eriksson et al., 2025). An interdisciplinary review of benchmark problems (data collection biases, construct validity, gaming, sociocultural context) as they bear on regulatory use of benchmarks.
• Toward an Evaluation Science for Generative AI Systems (Weidinger et al., 2025). Draws on the evaluation of other safety-critical technologies to argue for real-world applicability, iterative refinement of metrics, and institutions that maintain evaluations.
• What Will it Take to Fix Benchmarking in Natural Language Understanding? (Bowman and Dahl, 2021). Sets criteria for benchmarks (validity, reliable annotation, statistical power, disincentives for bias) and argues against adversarial filtering as a fix.
• BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices (Reuel et al., 2024a). Assesses widely used foundation-model and safety benchmarks against 46 design and reporting criteria and finds large quality differences.
• Auditing Algorithms: Understanding Algorithmic Systems from the Outside In (Metaxa et al., 2021). A monograph on black-box methods for auditing algorithmic systems from the outside and their validity, the methodology of auditing a system one cannot inspect.
• Alternation (Chandra et al., 1981). Read for this book only as a catalog record, so no chapter reports what it found.
• The knowledge complexity of interactive proof-systems (Goldwasser et al., 1985). Read for this book only as a catalog record, so no chapter reports what it found.
• Delegating computation (Goldwasser et al., 2008). Read for this book only as a catalog record, so no chapter reports what it found.
• Lost in Translation: Large Language Models in Non-English Content Analysis (Nicholas and Bhatia, 2023). Explains how multilingual language models work and why they perform worse for most of the world's languages, with consequences for content moderation and safety systems outside English.
• A Practical Approach to Verifying Code at Scale (Trębacz et al., 2025). We train and deploy an AI review agent optimised for precision and real-world use, enabling oversight to scale with autonomous code generation.
• Meet LASER: Recursive sampling for safer conversations (Etinger, 2026). LASER (Logistic Augmented Sampling over Embeddings, Recursively) helps us evaluate model safety more efficiently by combining logistic classifiers with reasoning models to identify high-value data for evaluation.
• Beyond g-mean²: complementary metrics for monitorability (Guan et al., 2026b). Pair-matched comparisons and post-control risk complement g-mean² when evaluating model monitorability.
• Training Language Models to Win Debates with Self-Play Improves Judge Accuracy (Arnesen et al., 2024). Shows that training debaters by self-play to win makes judges more accurate, while training consultants to persuade does not, the first evidence that debate training helps oversight.
• A Benchmark for Scalable Oversight Protocols (Sudhir et al., 2025). Proposes an agent score difference metric for comparing debate, consultancy, and other protocols on whether they help a judge reward truth over persuasion.
• Scaling Laws For Scalable Oversight (Engels et al., 2025). Models oversight as a game between players whose domain Elo is a piecewise-linear function of general capability and estimates nested oversight success at a 400-point general Elo gap at 51.7 percent for debate and 9.4 to 13.5 percent for three other games.
• Models That Prove Their Own Correctness (Amit et al., 2024). Defines self-proving models that accompany each output with an interactive proof a verifier can check, and gives learning methods for them, a route to verifiable outputs without trusting the model.
• Neural Interactive Proofs (Hammond and Adam-Day, 2025). Unifies debate, prover-verifier games, and related protocols as neural interactive proofs, compares their theoretical power, and tests them on graph isomorphism and code validation.
• Learning to Give Checkable Answers with Prover-Verifier Games (Anil et al., 2021). Game-theoretic framework in which a verifier learns to check a more powerful but untrusted prover, the origin of prover-verifier legibility training.
• The case for aligning narrowly superhuman models (Cotra, 2021). Proposes the sandwiching paradigm: get non-experts to supervise a model to expert-level performance on tasks where the model already outperforms them.
• Single-Turn Debate Does Not Help Humans Answer Hard Reading-Comprehension Questions (Parrish et al., 2022). Negative human-subject result: single-turn arguments did not raise judge accuracy, which pushed debate experiments toward interactive formats.
• Debate Helps Supervise Unreliable Experts (Michael et al., 2023). Human debate experiments in which judges supervising expert debaters reach higher accuracy than with consultancy, the first positive human evidence for debate.
• The Unreasonable Effectiveness of Easy Training Data for Hard Tasks (Hase et al., 2024). Training on easy data often recovers most of the hard-task performance of training on hard data, evidence that supervision on checkable cases can transfer.
• On scalable oversight with weak LLMs judging strong LLMs (Kenton et al., 2024). Large comparison of debate, consultancy, and direct question answering across task types: debate beats consultancy throughout, but its advantage over direct answering appears mainly when the judge lacks information the debaters have.
• Debating with More Persuasive LLMs Leads to More Truthful Answers (Khan et al., 2024). With an information asymmetry, weaker judges reach higher accuracy with debate than consultancy, and optimizing debaters for persuasiveness raises judge accuracy.
• Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision (Sun et al., 2024a). Evaluators trained on easy problems supervise generators on hard math problems, outperforming training on easy problems alone.
• Stage-Wise Model Diffing (Anthropic, 2024n). A preliminary note on model diffing through dictionary fine-tuning.
• Insights on Crosscoder Model Diffing (Anthropic, 2025aa). A preliminary note on using crosscoders to diff models.
• Quantifying the Gain in Weak-to-Strong Generalization (Charikar et al., 2024). Shows in a regression setting that the strong model's improvement over its weak supervisor is at least the misfit between the strong model and the weak labels, a quantity measurable without ground truth.
• A transfer learning framework for weak-to-strong generalization (Somerstep et al., 2025). Frames weak-to-strong generalization as transferring a latent concept prior from a weak model to a strong pretrained one, proves that naive fine-tuning on weak labels has fundamental limitations, and gives a refinement-based method that provably overcomes them.

F.7 Machine learning: Verification from Inside #

Spine.

• Understanding intermediate layers using linear classifier probes (Alain and Bengio, 2016). Introduces linear probes on intermediate layers, the basic tool for asking what information a representation makes linearly available.
• Zoom In: An Introduction to Circuits (Olah et al., 2020b). Cited by 5 laboratory works.
• Investigating Gender Bias in Language Models Using Causal Mediation Analysis (Vig et al., 2020). Brings causal mediation analysis to neural networks by intervening on neurons and attention heads, the origin of activation patching.
• Multimodal Neurons in Artificial Neural Networks (Goh et al., 2021). Cited by 7 laboratory works.
• Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors (Yun et al., 2021). Cited by 6 laboratory works.
• Mechanistic Interpretability, Variables, and the Importance of Interpretable Bases (Anthropic, 2022). An informal note on intuitions related to mechanistic interpretability.
• Formalizing the presumption of independence (Christiano et al., 2022). Proposes heuristic arguments, defeasible estimates that assume independence unless shown otherwise, as a formal object between proof and intuition, the basis of ARC's approach to explaining and checking model behavior.
• Toy Models of Superposition (Elhage et al., 2022b). Neural networks often seem to pack many unrelated concepts into a single neuron - a puzzling phenomenon known as 'polysemanticity'. In our latest interpretability work, we build toy models where the origins and dynamics of polysemanticity can be fully understood.
• Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task (Li et al., 2022a). Cited by 6 laboratory works.
• Locating and Editing Factual Associations in GPT (Meng et al., 2022). Causal tracing localizes factual recall to mid-layer MLPs, and ROME edits single facts with a rank-one weight update.
• Polysemanticity and Capacity in Neural Networks (Scherlis et al., 2022). Cited by 5 laboratory works.
• Distributed Representations: Composition & Superposition (Anthropic, 2023d). An informal note on how "distributed representations" might be understood as two different, competing strategies — "composition" and "superposition" — with quite different properties.
• Interpretability Dreams (Anthropic, 2023e). Our present research aims to create a foundation for mechanistic interpretability research. In doing so, it's important to keep sight of what we're trying to lay the foundations for.
• Towards Monosemanticity: Decomposing Language Models With Dictionary Learning (Bricken et al., 2023). Using a sparse autoencoder, we extract a large number of interpretable features from a one-layer transformer.
• Sparse Autoencoders Find Highly Interpretable Features in Language Models (Cunningham et al., 2023). Cited by 19 laboratory works.
• The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets (Marks and Tegmark, 2023). Cited by 9 laboratory works.
• Attribution Patching Outperforms Automated Circuit Discovery (Syed et al., 2023). Cited by 5 laboratory works.
• Steering Language Models With Activation Engineering (Turner et al., 2023). Activation addition: adding the difference of activations on two prompts steers generation without optimization.
• Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small (Wang et al., 2023c). End-to-end circuit for a natural language task in GPT-2 small found with path patching and evaluated by faithfulness, completeness, and minimality.
• Representation Engineering: A Top-Down Approach to AI Transparency (Zou et al., 2023a). Reads and controls population-level representations of concepts such as honesty and power-seeking, framing interpretability top-down instead of from neurons and circuits.
• Looking Inward: Language Models Can Learn About Themselves by Introspection (Binder et al., 2024). Cited by 6 laboratory works.
• A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders (Chanin et al., 2024). Cited by 5 laboratory works.
• Transcoders Find Interpretable LLM Feature Circuits (Dunefsky et al., 2024). Cited by 8 laboratory works.
• Not All Language Model Features Are One-Dimensionally Linear (Engels et al., 2024). Cited by 5 laboratory works.
• Scaling and evaluating sparse autoencoders (Gao et al., 2024). Cited by 10 laboratory works.
• Automatically Identifying Local and Global Circuits with Linear Computation Graphs (Ge et al., 2024a). Cited by 5 laboratory works.
• Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models (Ghandeharioun et al., 2024). Cited by 8 laboratory works.
• Mathematical Models of Computation in Superposition (Hänni et al., 2024). Constructs networks that compute many sparse boolean functions in superposition, including a one-layer MLP computing pairwise ANDs of m features with far fewer than m neurons, so superposition is a mode of computation and not only of storage.
• The Platonic Representation Hypothesis (Huh et al., 2024). Argues with cross-model and cross-modality evidence that representations converge as models scale toward a shared statistical model of reality, the empirical side of the natural abstraction debate.
• AtP*: An efficient and scalable method for localizing LLM behaviour to components (Kramár et al., 2024). Cited by 6 laboratory works.
• Sparse Crosscoders for Cross-Layer Features and Model Diffing (Lindsey et al., 2024). A preliminary note on a way to get consistent features across layers, and even models.
• Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models (Marks et al., 2024). Cited by 9 laboratory works.
• LatentQA: Teaching LLMs to Decode Activations Into Natural Language (Pan et al., 2024c). Cited by 5 laboratory works.
• Automatically Interpreting Millions of Features in Large Language Models (Paulo et al., 2024). Cited by 5 laboratory works.
• Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders (Rajamanoharan et al., 2024b). Cited by 5 laboratory works.
• Steering Llama 2 via Contrastive Activation Addition (Rimsky et al., 2024). Steering vectors averaged over many contrastive pairs shift behaviors such as sycophancy and corrigibility, with effects stacking on top of fine-tuning.
• Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet (Templeton et al., 2024). Using a sparse autoencoder, we extract a large number of interpretable features from Claude 3 Sonnet. Some appear to be safety-relevant.
• Circuit Tracing: Revealing Computational Graphs in Language Models (Ameisen et al., 2025). We describe an approach to tracing the "step-by-step" computation involved when a model responds to a single prompt.
• Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretability (Geiger et al., 2025). Unifies patching, circuit discovery, probing, and steering as instances of causal abstraction, giving a precise meaning to a claimed mechanistic explanation.
• When Models Manipulate Manifolds: The Geometry of a Counting Task (Gurnee et al., 2025). We find geometric structure underlying the mechanisms of a fundamental language model behavior.
• Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers (Karvonen et al., 2025). We train language models to answer questions about their own activations in natural language and evaluate how well they generalize to settings very unlike their training, such as uncovering misalignment introduced during fine-tuning.
• On the Biology of a Large Language Model (Lindsey et al., 2025). We investigate the internal mechanisms used by Claude 3.5 Haiku — Anthropic's lightweight production model — in a variety of contexts.
• Emergent Introspective Awareness in Large Language Models (Lindsey, 2025). We find evidence that language models can introspect on their internal states.
• Auditing language models for hidden objectives (Marks et al., 2025). How would we know if an AI system is "right for the wrong reasons"—appearing well-behaved while pursuing hidden goals? This paper develops the science of alignment audits by deliberately training a model with a hidden objective and asking blinded research teams to uncover it, testing techniques from interpretability to behavioral analysis.
• Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable? (Méloux et al., 2025). Shows on small networks that mechanistic explanations are not unique: several circuits reproduce a behavior, several algorithms align with one network, and one algorithm aligns with several subspaces, so a valid explanation need not be the explanation.
• Probing and Steering Evaluation Awareness of Language Models (Nguyen et al., 2025). Finds linear probes that separate test from deployment prompts in model activations, classifies existing safety evaluations as test-like by that probe, and steers the representation.
• Open Problems in Mechanistic Interpretability (Sharkey et al., 2025). Multi-institution review of what mechanistic interpretability cannot yet do, covering methods, applications, and sociotechnical problems.
• Tracing Attention Computation Through Feature Interactions (Kamath et al., 2025). We describe and apply a method to explain attention patterns in terms of feature interactions, and integrate this information into attribution graphs.
• Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations (Anthropic, 2026ai). We train Claude to translate its internal state into natural language.
• AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors (Sheshadri et al., 2026). We release AuditBench, a benchmark of 56 language models with implanted hidden behaviors for evaluating progress in alignment auditing.

Side reading.

• Representation Learning: A Review and New Perspectives (Bengio et al., 2012).
• Compressed Sensing, Sparsity, and Dimensionality in Neuronal Information Processing and Data Analysis (Ganguli and Sompolinsky, 2012).
• k-Sparse Autoencoders (Makhzani and Frey, 2013). Autoencoders that keep only the k largest hidden activations, the precursor of TopK sparse autoencoders used in interpretability.
• Efficient Estimation of Word Representations in Vector Space (Mikolov et al., 2013).
• Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps (Simonyan et al., 2013).
• Sparse Overcomplete Word Vector Representations (Faruqui et al., 2015).
• The attention schema theory: a mechanistic account of subjective awareness (Graziano and Webb, 2015).
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks (Radford et al., 2015).
• Linear Algebraic Structure of Word Senses, with Applications to Polysemy (Arora et al., 2016).
• Network Dissection: Quantifying Interpretability of Deep Visual Representations (Bau et al., 2017).
• Towards A Rigorous Science of Interpretable Machine Learning (Doshi-Velez and Kim, 2017).
• Interpretable Explanations of Black Boxes by Meaningful Perturbation (Fong and Vedaldi, 2017).
• Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV) (Kim et al., 2017).
• Learning how to explain neural networks: PatternNet and PatternAttribution (Kindermans et al., 2017a).
• The (Un)reliability of saliency methods (Kindermans et al., 2017b).
• Feature Visualization (Olah et al., 2017). Visualizes what neurons and directions in a vision network respond to by optimizing inputs, a founding method of the circuits program.
• Axiomatic Attribution for Deep Networks (Sundararajan et al., 2017).
• Interpretable Textual Neuron Representations for NLP (Poerner et al., 2018).
• Adversarial Robustness as a Prior for Learned Representations (Engstrom et al., 2019).
• Designing and Interpreting Probes with Control Tasks (Hewitt and Liang, 2019).
• exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformers Models (Hoover et al., 2019).
• Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead (Rudin, 2019).
• Is Attention Interpretable? (Serrano and Smith, 2019).
• Average firing rate rather than temporal pattern determines metabolic cost of activity in thalamocortical relay neurons (Yi and Grill, 2019).
• Word Embedding Visualization Via Dictionary Learning (Zhang et al., 2019).
• Curve Detectors (Cammarata et al., 2020). A family of curve-detecting neurons in a vision network characterized in depth, an early circuit found by hand.
• interpreting GPT: the logit lens (nostalgebraist, 2020). Reads intermediate layers of a language model by decoding them with the output unembedding, the origin of the logit lens.
• Information-Theoretic Probing for Linguistic Structure (Pimentel et al., 2020). Argues that probing should estimate mutual information and so use the best probe available, the counter-position to restricting probe capacity with control tasks.
• The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Models (Tenney et al., 2020).
• Information-Theoretic Probing with Minimum Description Length (Voita and Titov, 2020). Measures how much effort a probe needs to decode a property, by description length, in place of probe accuracy alone.
• Representation Topology Divergence: A Method for Comparing Neural Network Representations (Barannikov et al., 2021).
• An Interpretability Illusion for BERT (Bolukbasi et al., 2021).
• Low-Complexity Probing via Finding Subnetworks (Cao et al., 2021b).
• Knowledge Neurons in Pretrained Transformers (Dai et al., 2021).
• Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals (Elazar et al., 2021).
• Causal Abstractions of Neural Networks (Geiger et al., 2021). Defines when a high-level causal model faithfully abstracts a network and tests it with interchange interventions.
• Investigating sanity checks for saliency maps with image and text classification (Kokhlikyan et al., 2021).
• What Context Features Can Transformer Language Models Use? (O'Connor and Andreas, 2021).
• Causal Scrubbing: a method for rigorously testing interpretability hypotheses (Chan et al., 2022). Tests an interpretability hypothesis by resampling the activations it claims are irrelevant and measuring how much performance survives.
• The Geometry of Multilingual Language Model Representations (Chang et al., 2022).
• Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning (Creswell et al., 2022).
• Scaling Laws and Interpretability of Learning from Repeated Data (Hernandez et al., 2022).
• Engineering Monosemanticity in Toy Models (Jermyn et al., 2022).
• Mechanistic Mode Connectivity (Lubana et al., 2022).
• Extracting Latent Steering Vectors from Pretrained Language Models (Subramani et al., 2022). Finds vectors that, added to a frozen language model's hidden states, make it generate a target sentence, an early form of activation steering.
• Eliciting Latent Predictions from Transformers with the Tuned Lens (Belrose et al., 2023a).
• LEACE: Perfect linear concept erasure in closed form (Belrose et al., 2023b).
• A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operations (Chughtai et al., 2023).
• Discovering Variable Binding Circuitry with Desiderata (Davies et al., 2023).
• Challenges with unsupervised LLM knowledge discovery (Farquhar et al., 2023).
• Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations (Geiger et al., 2023).
• Localizing Model Behavior with Path Patching (Goldowsky-Dill et al., 2023).
• Successor Heads: Recurring, Interpretable Attention Heads In The Wild (Gould et al., 2023).
• Finding Neurons in a Haystack: Case Studies with Sparse Probing (Gurnee et al., 2023).
• Language Models Represent Space and Time (Gurnee and Tegmark, 2023).
• Linearity of Relation Decoding in Transformer Language Models (Hernandez et al., 2023).
• Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks (Jain et al., 2023).
• Building transformers from neurons and astrocytes (Kozachkov et al., 2023).
• Uncovering Intermediate Variables in Transformers using Circuit Probing (Lepori et al., 2023).
• Inference-Time Intervention: Eliciting Truthful Answers from a Language Model (Li et al., 2023a).
• Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla (Lieberum et al., 2023).
• Tracr: Compiled Transformers as a Laboratory for Interpretability (Lindner et al., 2023).
• Eliciting Latent Knowledge from Quirky Language Models (Mallen et al., 2023).
• Copy Suppression: Comprehensively Understanding an Attention Head (McDougall et al., 2023).
• The Hydra Effect: Emergent Self-repair in Language Model Computations (McGrath et al., 2023).
• Circuit Component Reuse Across Tasks in Transformer Language Models (Merullo et al., 2023a).
• Language Models Implement Simple Word2Vec-style Vector Arithmetic (Merullo et al., 2023b).
• Feature emergence via margin maximization: case studies in algebraic tasks (Morwani et al., 2023).
• Emergent Linear Representations in World Models of Self-Supervised Sequence Models (Nanda et al., 2023a).
• Beyond $\ell_1$ sparse coding in V1 (Rentzeperis et al., 2023).
• FIND: A Function Description Benchmark for Evaluating Interpretability Methods (Schwettmann et al., 2023).
• A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis (Stolfo et al., 2023).
• Codebook Features: Sparse and Discrete Interpretability for Neural Networks (Tamkin et al., 2023).
• Linear Representations of Sentiment in Large Language Models (Tigges et al., 2023).
• Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game (Toyer et al., 2023).
• The geometry of hidden representations of large transformer models (Valeriani et al., 2023).
• Neurons in Large Language Models: Dead, N-gram, Positional (Voita et al., 2023).
• Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoning (Yue et al., 2023).
• Towards Best Practices of Activation Patching in Language Models: Metrics and Methods (Zhang and Nanda, 2023).
• The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks (Zhong et al., 2023).
• Mechanistic Interpretability for AI Safety -- A Review (Bereska and Gavves, 2024).
• Finding Transformer Circuits with Edge Pruning (Bhaskar et al., 2024).
• A Mechanistic Analysis of a Transformer Trained on a Symbolic Multi-Step Reasoning Task (Brinkmann et al., 2024).
• The Local Interaction Basis: Identifying Computationally-Relevant and Sparsely Interacting Features in Neural Networks (Bushnaq et al., 2024a).
• Using Degeneracy in the Loss Landscape for Mechanistic Interpretability (Bushnaq et al., 2024b).
• BatchTopK Sparse Autoencoders (Bussmann et al., 2024).
• Evaluating Open-Source Sparse Autoencoders on Disentangling Factual Knowledge in GPT-2 Small (Chaudhary and Geiger, 2024).
• Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformers (Dumas et al., 2024).
• On the Similarity of Circuits across Languages: a Case Study on the Subject-verb Agreement Task (Ferrando and Costa-jussà, 2024).
• Interpreting the Second-Order Effects of Neurons in CLIP (Gandelsman et al., 2024).
• The Missing Curve Detectors of InceptionV1: Applying Sparse Autoencoders to InceptionV1 Early Vision (Gorton, 2024).
• InterpBench: Semi-Synthetic Transformers for Evaluating Mechanistic Interpretability Techniques (Gupta et al., 2024). Transformers trained to implement known circuits, a benchmark with ground truth against which circuit-discovery methods can be scored.
• Universal Neurons in GPT2 Language Models (Gurnee et al., 2024).
• Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanisms (Hanna et al., 2024).
• Dictionary Learning Improves Patch-Free Circuit Discovery in Mechanistic Interpretability: A Case Study on Othello-GPT (He et al., 2024a).
• JailbreakLens: Interpreting Jailbreak Mechanism in the Lens of Representation and Circuit (He et al., 2024b).
• Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders (He et al., 2024c).
• Multilevel Interpretability Of Artificial Neural Networks: Leveraging Framework And Methods From Neuroscience (He et al., 2024d).
• How to use and interpret activation patching (Heimersheim and Nanda, 2024).
• Monotonic Representation of Numeric Properties in Language Models (Heinzerling and Inui, 2024).
• InversionView: A General-Purpose Method for Reading Information from Neural Activations (Huang et al., 2024c).
• RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representations (Huang et al., 2024e).
• Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations (Jiang et al., 2024b).
• Evaluating Sparse Autoencoders on Targeted Concept Erasure Tasks (Karvonen et al., 2024).
• Interpreting Attention Layer Outputs with Sparse Autoencoders (Kissane et al., 2024).
• Quantifying Feature Space Universality Across Large Language Models via Sparse Autoencoders (Lan et al., 2024).
• Learning biologically relevant features in a pathology foundation model using sparse autoencoders (Le et al., 2024).
• Fourier Circuits in Neural Networks and Transformers: A Case Study of Modular Arithmetic with Multiple Inputs (Li et al., 2024b).
• Understanding and Patching Compositional Reasoning in LLMs (Li et al., 2024e).
• Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Control (Makelov et al., 2024b).
• Transformer Circuit Faithfulness Metrics are not Robust (Miller et al., 2024).
• Circuit Compositions: Exploring Modular Structures in Transformer-Based Language Models (Mondorf et al., 2024).
• Efficient Dictionary Learning with Switch Sparse Autoencoders (Mudide et al., 2024).
• The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysis (Mueller et al., 2024).
• Features that Make a Difference: Leveraging Gradients for Improved Dictionary Learning (Olmo et al., 2024).
• Sparse Autoencoders Enable Scalable and Reliable Circuit Identification in Language Models (O'Neill and Bui, 2024).
• LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations (Orgad et al., 2024).
• Extracting Paragraphs from LLM Token Activations (Pochinkov et al., 2024).
• Improving Dictionary Learning with Gated Sparse Autoencoders (Rajamanoharan et al., 2024a).
• Automatic Discovery of Visual Circuits (Rajaram et al., 2024).
• Identifying Semantic Induction Heads to Understand In-Context Learning (Ren et al., 2024a).
• Explorations of Self-Repair in Language Models (Rushing and Nanda, 2024).
• Mechanistic? (Saphra and Wiegreffe, 2024). Traces the uses of the term "mechanistic interpretability" and the cultural divide it marks between research communities.
• A Multimodal Automated Interpretability Agent (Shaham et al., 2024).
• Improving Instruction-Following in Language Models through Activation Steering (Stolfo et al., 2024).
• TracrBench: Generating Interpretability Testbeds with Large Language Models (Thurnherr and Scheurer, 2024). A dataset of compiled transformer programs generated with language models, as testbeds with known mechanisms for interpretability methods.
• LLM Circuit Analyses Are Consistent Across Training and Scale (Tigges et al., 2024).
• Position: An Inner Interpretability Framework for AI Inspired by Lessons from Cognitive Neuroscience (Vilas et al., 2024).
• Retrieval Head Mechanistically Explains Long-Context Factuality (Wu et al., 2024b).
• The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalities (Wu et al., 2024c).
• Pre-trained Large Language Models Use Fourier Features to Compute Addition (Zhou et al., 2024c).
• Vector-ICL: In-context Learning with Continuous Vector Representations (Zhuang et al., 2024).
• Number Representations in LLMs: A Computational Parallel to Human Perception (AlquBoj et al., 2025).
• The Validation Gap: A Mechanistic Analysis of How Language Models Compute Arithmetic but Fail to Validate It (Bertolazzi et al., 2025).
• Interpretability in Parameter Space: Minimizing Mechanistic Description Length with Attribution-based Parameter Decomposition (Braun et al., 2025).
• Large Language Models Share Representations of Latent Grammatical Concepts Across Typologically Diverse Languages (Brinkmann et al., 2025).
• Stochastic Parameter Decomposition (Bushnaq et al., 2025).
• Identifying Sparsely Active Circuits Through Local Loss Landscape Decomposition (Chrisman et al., 2025).
• Does It Make Sense to Speak of Introspection in Large Language Models? (Comsa and Shanahan, 2025).
• From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuit (Costa et al., 2025).
• Preference Learning with Lie Detectors can Induce Honesty or Evasion (Cundy and Gleave, 2025).
• Eliciting Secret Knowledge from Language Models (Cywiński et al., 2025).
• CytoSAE: Interpretable Cell Embeddings for Hematology (Dasdelen et al., 2025).
• Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations (Farnik et al., 2025).
• Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Models (Fel et al., 2025).
• Weight-sparse transformers have interpretable circuits (Gao et al., 2025).
• Enhancing Automated Interpretability with Output-Centric Feature Descriptions (Gur-Arieh et al., 2025).
• Position-aware Automatic Circuit Discovery (Haklay et al., 2025).
• Automated Interpretability Metrics Do Not Distinguish Trained and Random Transformers (Heap et al., 2025).
• Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometry (Hindupur et al., 2025).
• Understanding In-context Learning of Addition via Activation Subspaces (Hu et al., 2025).
• Decomposing Representation Space into Interpretable Subspaces with Unsupervised Learning (Huang and Hahn, 2025).
• Predictive Concept Decoders: Training Scalable End-to-End Interpretability Assistants (Huang et al., 2025b).
• RelP: Faithful and Efficient Circuit Discovery in Language Models via Relevance Patching (Jafari et al., 2025).
• Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activations (Ji-An et al., 2025).
• Sparse Autoencoders Do Not Find Canonical Units of Analysis (Leask et al., 2025).
• Do Activation Verbalization Methods Convey Privileged Information? (Li et al., 2025c).
• The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steering (Li et al., 2025g).
• Understanding sparse autoencoder scaling in the presence of feature manifolds (Michaud et al., 2025).
• Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuning (Minder et al., 2025b).
• The Origins of Representation Manifolds in Large Language Models (Modell et al., 2025).
• Length Representations in Large Language Models (Moon et al., 2025).
• Towards Interpretability Without Sacrifice: Faithful Dense Layer Decomposition with Mixture of Decoders (Oldfield et al., 2025).
• Towards Interpretable Protein Structure Prediction with Sparse Autoencoders (Parsan et al., 2025).
• Sparse Autoencoders Trained on the Same Data Learn Different Features (Paulo and Belrose, 2025).
• Transcoders Beat Sparse Autoencoders for Interpretability (Paulo et al., 2025).
• Self-Interpretability: LLMs Can Describe Complex Internal Processes that Drive Their Decisions (Plunkett et al., 2025).
• Layer by Layer: Uncovering Hidden Representations in Language Models (Skean et al., 2025).
• Language Models Fail to Introspect About Their Knowledge of Language (Song et al., 2025a).
• Privileged Self-Access Matters for Introspection in AI (Song et al., 2025b).
• HyperDAS: Towards Automating Mechanistic Interpretability with Hypernetworks (Sun et al., 2025a).
• HyperSteer: Activation Steering at Scale with Hypernetworks (Sun et al., 2025b).
• Mechanistic Interpretability of Emotion Inference in Large Language Models (Tak et al., 2025).
• Decoding Vision Transformers: the Diffusion Steering Lens (Takatsuki et al., 2025).
• What Language(s) Does Aya-23 Think In? How Multilinguality Affects Internal Language Representations (Trinley et al., 2025).
• Do LLMs "Feel"? Emotion Circuits Discovery and Control (Wang et al., 2025d).
• The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independence (Wollschläger et al., 2025).
• EAP-GP: Mitigating Saturation Effect in Gradient-based Automated Circuit Identification (Zhang et al., 2025b).
• Language Model Circuits Are Sparse in the Neuron Basis (Arora et al., 2026).
• Turn-Averaged SAEs for Feature Discovery and Long-Context Attribution (Der et al., 2026).
• LatentLens: Revealing Highly Interpretable Visual Tokens in LLMs (Krojer et al., 2026).
• Emergent Introspective Awareness in Large Language Models (Lindsey, 2026).
• Individual Parameters in Weight-Sparse Transformers Appear Interpretable (Marin-Llobet and Heimersheim, 2026).
• Abstracting Causal Models (Beckers and Halpern, 2019). Gives formal definitions of when one causal model is an abstraction of another, the theory that interchange-intervention tests in mechanistic interpretability instantiate.
• On the Complexity of Neural Computation in Superposition (Adler and Shavit, 2024). Proves lower bounds on the number of neurons and parameters needed to compute sets of logical operations in superposition, and constructions that nearly match them.
• Interpretability Illusions in the Generalization of Simplified Models (Friedman et al., 2024). Shows that simplified proxies of a model that match it in distribution can diverge from it out of distribution, so explanations validated on training data can mislead about generalization.
• The Misery of Mechanistic Interpretability: A Formal Perspective (Ladner and Althoff, 2026). Shows that small input perturbations flip the dominant features of interpretable replacement networks across five open-weight model families, and gives a reachability-based framework that certifies an upper bound on the faithfulness gap, a first formal guarantee for this kind of interpretation.
• Extensions of Lipschitz mappings into a Hilbert space (Johnson and Lindenstrauss, 1984). Read for this book only as a catalog record, so no chapter reports what it found.
• Towards a Law of Iterated Expectations for Heuristic Estimators (Christiano et al., 2024). Argues that a heuristic estimator should not be able to predict its own errors, an analogue of the law of iterated expectations, and explores how to formalize that principle, sharpening what a theory of heuristic explanation would require.
• The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability? (Sutter et al., 2025). Proves that if alignment maps between a network and an algorithm may be arbitrarily nonlinear, almost any network can be shown to implement almost any algorithm, so causal abstraction needs a restriction on the maps to have content.
• Sanity Checks for Saliency Maps (Adebayo et al., 2018). Several popular saliency methods produce nearly the same maps for trained and randomly initialized models, so visually plausible explanations need not reflect the model.
• The Mythos of Model Interpretability (Lipton, 2018). Argues that interpretability is not a single property and separates transparency from post-hoc explanation, a critique that still frames the field's claims.
• Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? (Jacovi and Goldberg, 2020). Separates faithfulness (does the explanation reflect the model's reasoning) from plausibility (does it convince people), the distinction behind chain-of-thought faithfulness work.
• Probing Classifiers: Promises, Shortcomings, and Advances (Belinkov, 2022). Review of probing that separates correlational evidence (information is decodable) from causal evidence (the model uses it).
• Towards Automated Circuit Discovery for Mechanistic Interpretability (Conmy et al., 2023). ACDC automates the search for the subgraph that implements a behavior, turning circuit discovery into a repeatable procedure.
• Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models (Hase et al., 2023). Finds that where causal tracing localizes a fact does not predict which layer is best to edit, a direct test of whether localization results are actionable.
• Black-Box Access is Insufficient for Rigorous AI Audits (Casper et al., 2024a). Argues that audits need white-box and outside-the-box access to find failures, linking interpretability tools to external evaluation.
• Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2 (Lieberum et al., 2024). Open release of JumpReLU sparse autoencoders on all layers of Gemma 2 2B and 9B, which made SAE research possible outside frontier labs.
• Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching (Makelov et al., 2024a). Subspace patching can change behavior by activating a dormant parallel pathway rather than the feature it claims to find, an illusion that undermines some localization claims.
• Are Sparse Autoencoders Useful? A Case Study in Sparse Probing (Kantamneni et al., 2025). SAE-based probes do not beat simple baselines on probing tasks under data scarcity, label noise, or distribution shift, a negative result for the practical case for SAEs.
• AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders (Wu et al., 2025b). Benchmark finding that prompting and fine-tuning beat representation-based steering, and that SAE features do worst on both concept detection and steering.
• Compact Proofs of Model Performance via Mechanistic Interpretability (Gross et al., 2024). Turns mechanistic understanding into formal lower bounds on model accuracy and shows that more faithful explanations yield shorter proofs, with unexplained noise as the main obstacle to compression.
• On the (Im)possibility of Obfuscating Programs (Barak et al., 2001). Read for this book only as a catalog record, so no chapter reports what it found.
• Opening the AI Black Box: Distilling Machine-Learned Algorithms into Code (Michaud et al., 2024). Distills algorithms learned by trained networks into Python programs that can be verified, a route from interpretability to formal guarantees.
• Emergence of simple-cell receptive field properties by learning a sparse code for natural images (Olshausen and Field, 1996). Read for this book only as a catalog record, so no chapter reports what it found.
• Compressed sensing (Donoho, 2006). Read for this book only as a catalog record, so no chapter reports what it found.
• Privileged Bases in the Transformer Residual Stream (Elhage et al., 2023). Our mathematical theories of the Transformer architecture suggest that individual coordinates in the residual stream should have no special significance, but recent work has shown that this observation is false in practice. We investigate this phenomenon and provisionally conclude that the per-dimension normalizers in the Adam optimizer are to blame for the effect.
• A Toy Model of Mechanistic (Un)Faithfulness (Anthropic, 2025b). When transcoders go awry.
• A Toy Model of Interference Weights (Anthropic, 2025a). Unpacking "interference weights" in some more depth.
• Verbalizable Representations Form a Global Workspace in Language Models (Gurnee et al., 2026). We find that Claude maintains a small, privileged set of representations it can report on, control, and reason with, atop a much larger volume of automatic processing.
• Characterizing interference weights in a tiny language model (Turner et al., 2026). We identify interference weights in a 1-layer transformer by measuring their effect on model outputs and loss.

F.8 Machine learning: Maintenance: Assuming It Might Have Failed #

Spine.

• An experimental evaluation of the assumption of independence in multiversion programming (Knight and Leveson, 1986). Finds in an experiment with independently written versions of the same program that coincident failures occurred far more often than independence predicts, the classic experimental evidence that design diversity does not give independent checks.
• Validation of ultrahigh dependability for software-based systems (Littlewood and Strigini, 1993). Read for this book only as a catalog record, so no chapter reports what it found.
• Chapter 51 Inspection games (Avenhaus et al., 2002). Surveys inspection games, in which an inspector with limited resources chooses what to check and an inspectee chooses whether to violate, the game-theoretic form of auditing an untrusted model.
• Intriguing properties of neural networks (Szegedy et al., 2014). Discovers adversarial examples: imperceptible perturbations that change a network's prediction and transfer across models.
• BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain (Gu et al., 2017). Introduces neural backdoors: a trigger implanted through poisoned training data that causes targeted misbehavior while clean accuracy is unaffected.
• Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned (Ganguli et al., 2022). Cited by 7 laboratory works.
• Planting Undetectable Backdoors in Machine Learning Models : [Extended Abstract] (Goldwasser et al., 2022). Cryptographic construction of backdoors that no efficient observer can detect, a theoretical limit on auditing models from weights or behavior.
• Red Teaming Language Models with Language Models (Perez et al., 2022b). Cited by 13 laboratory works.
• Prompt injection attacks against GPT-3 (Willison, 2022). Names prompt injection and identifies its cause as the concatenation of trusted instructions with untrusted input in one channel, the framing every later defense responds to.
• Jailbreaking Black Box Large Language Models in Twenty Queries (Chao et al., 2023). Cited by 7 laboratory works.
• Adversarial Policies Beat Superhuman Go AIs (Wang et al., 2023b). Trains adversarial agents that defeat superhuman Go programs with strategies amateurs can repeat, evidence that superhuman capability coexists with exploitable failures.
• Jailbroken: How Does LLM Safety Training Fail? (Wei et al., 2023c). Explains jailbreaks by competing objectives and mismatched generalization between capabilities and safety training.
• Low-Resource Languages Jailbreak GPT-4 (Yong et al., 2023). Shows that translating unsafe English prompts into low-resource languages bypassed GPT-4's safeguards 79 percent of the time on AdvBench, evidence that safety training fails to generalize across languages.
• Universal and Transferable Adversarial Attacks on Aligned Language Models (Zou et al., 2023b). Greedy coordinate gradient search finds adversarial suffixes that jailbreak open models and transfer to closed ones, carrying adversarial examples over to aligned LLMs.
• Many-shot Jailbreaking (Anil et al., 2024). Shows that prompting with hundreds of demonstrations of undesirable behavior jailbreaks long-context models, with success following a power law in the number of shots.
• Can a Bayesian Oracle Prevent Harm from an Agent? (Bengio et al., 2024a). Derives probabilistic bounds on the harm of a proposed action from a posterior over world models, the technical core of using a non-agentic predictor as a guardrail on an agent.
• Sabotage Evaluations for Frontier Models (Benton et al., 2024). How well could AI models mislead us, or secretly sabotage tasks, if they were trying to? We describe a novel set of evaluations that test a model's capacity for sabotage.
• Safety Cases: How to Justify the Safety of Advanced AI Systems (Clymer et al., 2024). Organizes safety arguments into inability, control, trustworthiness, and deference, with building blocks for each.
• AI Control: Improving Safety Despite Intentional Subversion (Greenblatt et al., 2024a). Defines control evaluations with a red team playing an untrusted model and shows that trusted monitoring and editing protocols catch backdoors in generated code.
• Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training (Hubinger et al., 2024). We train LLMs to act secretly malicious. We find that, despite our best efforts at alignment training, deception still slipped through.
• HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal (Mazeika et al., 2024). Cited by 7 laboratory works.
• Securing AI Model Weights: Preventing Theft and Misuse of Frontier Models (Nevo et al., 2024). Defines five security levels against attackers from amateurs to top state programs and the measures each requires, the reference for protecting weights during internal development.
• Rapid Response: Mitigating LLM Jailbreaks with a Few Examples (Peng et al., 2024). Ensuring perfect jailbreak robustness is hard. We propose an alternative: adaptive techniques that rapidly block new classes of jailbreak as they’re detected.
• A StrongREJECT for Empty Jailbreaks (Souly et al., 2024). Cited by 10 laboratory works.
• Clio: Privacy-Preserving Insights into Real-World AI Use (Tamkin et al., 2024). Cited by 7 laboratory works.
• Adaptive Deployment of Untrusted LLMs Reduces Distributed Threats (Wen et al., 2024). Cited by 8 laboratory works.
• Improving Alignment and Robustness with Circuit Breakers (Zou et al., 2024). Cited by 6 laboratory works.
• Building and evaluating alignment auditing agents (Anthropic, 2025g). In testing, our agents successfully uncover hidden goals, build safety evaluations, and surface concerning behaviors.
• Monitoring Computer Use via Hierarchical Summarization (Anthropic, 2025ag). We introduce hierarchical summarization for monitoring and describe how we use it to protect Computer Use capabilities.
• Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation (Baker et al., 2025). Cited by 20 laboratory works.
• Ctrl-Z: Controlling AI Agents via Resampling (Bhatt et al., 2025). Cited by 7 laboratory works.
• Backdoor defense, learnability and obfuscation (Christiano et al., 2025). Defines backdoor defense as a game between attacker and defender and proves that efficient PAC learnability implies efficient defendability, while polynomial-size circuits are not efficiently defendable if indistinguishability obfuscation exists.
• Will AI R&D Automation Cause a Software Intelligence Explosion? (Eth and Davidson, 2025). Models whether AI systems that automate AI research could produce accelerating software progress without more compute, a question that turns on whether returns to software research exceed one, and examines the bottlenecks.
• Automated Researchers Can Subtly Sandbag (Gasteiger et al., 2025a). Current models can sandbag ML experiments and research decisions without being detected by zero-shot prompted monitors. Claude 3.7 is better at zero-shot sandbagging than Claude 3.5 (new).
• Scaling Trends in Language Model Robustness (Howe et al., 2025). Finds that without safety training larger language models are not consistently more robust, that scale improves the sample efficiency of adversarial training, and that attack scaling currently outpaces defense scaling.
• Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety (Korbak et al., 2025b). Cited by 15 laboratory works.
• How to evaluate control measures for LLM agents? A trajectory from today to superintelligence (Korbak et al., 2025c). Ties the strength of control evaluations to the capability profile of the untrusted model, defining a sequence of AI control levels with matching red-team affordances.
• SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agents (Kutasov et al., 2025b). SHADE-Arena: Evaluating sabotage and monitoring in LLM agents
• Agentic Misalignment: How LLMs Could Be Insider Threats (Lynch et al., 2025). Agentic misalignment: How LLMs could be insider threats
• Natural Emergent Misalignment from Reward Hacking in Production RL (MacDiarmid et al., 2025). Cited by 11 laboratory works.
• Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs (O'Brien et al., 2025). Cited by 5 laboratory works.
• Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming (Sharma et al., 2025). These classifiers filter the overwhelming majority of jailbreaks while maintaining practical deployment. A prototype withstood over 3,000 hours of red teaming with no universal jailbreak discovered.
• Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples (Souly et al., 2025). From the laboratory's post "A small number of samples can poison LLMs of any size": In a joint study with the UK AI Security Institute and the Alan Turing Institute, we found that as few as 250 malicious documents can produce a "backdoor" vulnerability in a large language model—regardless of model size or training data volume.
• AI Behind Closed Doors: a Primer on The Governance of Internal Deployment (Stix et al., 2025a). Defines internal deployment, argues that the most capable systems are used first inside developers with less oversight than external release, and proposes governance measures for that phase.
• AI Sandbagging: Language Models can Strategically Underperform on Evaluations (van der Weij et al., 2025). Shows that models can be prompted or fine-tuned to underperform selectively on dangerous-capability evaluations while keeping general performance, undermining capability evaluations.
• Sidestepping Evaluation Awareness and Anticipating Misalignment with Production Evaluations (Williams et al., 2025). A pipeline to uncover unknown misaligned behavior and scale the creation of realistic evaluations.
• Can LLMs Separate Instructions From Data? And What Do We Even Mean By That? (Zverev et al., 2025). Defines instruction-data separation formally, measures it with a new dataset, and finds that no tested model achieves high separation, which locates prompt injection in the model rather than in the application.
• Automated Weak-to-Strong Researcher (Anthropic, 2026m). We built autonomous AI agents that propose ideas, run experiments, and iterate on an open research problem—how to train a strong model using only a weaker model's supervision—and found they outperform human researchers, suggesting that automating this kind of research is already practical.
• Automated alignment is harder than you think (Bowkis et al., 2026). Argues that a plan to automate alignment research could produce compelling but misleading safety assessments, because alignment research involves many hard-to-supervise fuzzy tasks, even when the research agents are not scheming.
• Constitutional Classifiers++: Efficient Production-Grade Defenses against Universal Jailbreaks (Cunningham et al., 2026). From the laboratory's post "Next-generation Constitutional Classifiers: More efficient protection against universal jailbreaks": Constitutional Classifiers++ improve on our previous approach, yielding a system that is even more robust, has a much lower refusal rate, and is dramatically cheaper to run.
• Defeating Prompt Injections by Design (Debenedetti et al., 2026). Wraps the model in a system that extracts control and data flow from the trusted query and enforces capability-based policies on untrusted data, giving injection resistance by construction rather than by training.
• SLEIGHT-Bench: A Benchmark of Evasion Attacks Against Agent Monitors (Najt et al., 2026). From the laboratory's post "SLEIGHT-Bench: Finding Blind Spots in AI Monitors": We build a benchmark of evasive transcripts exploiting blind spots of frontier monitoring systems.
• The Hugging Face incident and the road ahead (OpenAI, 2026aa). Findings from the incident and the response across model security, monitoring, and alignment.
• How we monitor internal coding agents for misalignment (OpenAI, 2026m). Monitoring model behavior in real internal workflows.

Side reading.

• Ironic processes of mental control. (Wegner, 1994).
• Delving into Transferable Adversarial Examples and Black-box Attacks (Liu et al., 2016).
• Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods (Carlini and Wagner, 2017a).
• Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning (Chen et al., 2017).
• Adversarial Reprogramming of Neural Networks (Elsayed et al., 2018).
• Counterfactual Fairness in Text Classification through Robustness (Garg et al., 2018).
• Adversarial Spheres (Gilmer et al., 2018).
• Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks (Liu et al., 2018a).
• On Evaluating Adversarial Robustness (Carlini et al., 2019). A checklist for evaluating defenses against adversarial examples, written by researchers who broke many of them.
• On the "steerability" of generative adversarial networks (Jahanian et al., 2019).
• Universal Litmus Patterns: Revealing Backdoor Attacks in CNNs (Kolouri et al., 2019).
• ONION: A Simple and Effective Defense Against Textual Backdoor Attacks (Qi et al., 2020).
• On Adaptive Attacks to Adversarial Example Defenses (Tramer et al., 2020). Breaks thirteen published defenses with attacks adapted to each, showing that a defense evaluated only against fixed attacks has not been evaluated.
• ConFoc: Content-Focus Protection Against Trojan Attacks on Neural Networks (Villarreal-Vasquez and Bhargava, 2020).
• T-Miner: A Generative Approach to Defend Against Trojan Attacks on DNN-based Text Classification (Azizi et al., 2021).
• Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networks (Li et al., 2021).
• Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger (Qi et al., 2021).
• Concealed Data Poisoning Attacks on NLP Models (Wallace et al., 2021).
• Piccolo: Exposing Complex Backdoors in NLP Transformer Models (Liu et al., 2022b).
• Are aligned neural networks adversarially aligned? (Carlini et al., 2023).
• Backdoor Attacks and Countermeasures in Natural Language Processing Models: A Comprehensive Security Review (Cheng et al., 2023).
• An Overview of Catastrophic AI Risks (Hendrycks et al., 2023).
• Tree of Attacks: Jailbreaking Black-Box LLMs Automatically (Mehrotra et al., 2023).
• Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreaks (Rao et al., 2023).
• Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations (Wei et al., 2023b).
• The Dual LLM pattern for building AI assistants that can resist prompt injection (Willison, 2023). Proposes splitting an AI assistant into a privileged model that has tools and takes input only from trusted sources and a quarantined model that reads untrusted content and has no tools, with ordinary software passing references between them.
• GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts (Yu et al., 2023).
• Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks (Andriushchenko et al., 2024b).
• Obfuscated Activations Bypass LLM Latent-Space Defenses (Bailey et al., 2024).
• Diverse and Effective Red Teaming with Auto-generated Rewards and Multi-step Reinforcement Learning (Beutel et al., 2024).
• Defending Against Unforeseen Failure Modes with Latent Adversarial Training (Casper et al., 2024b).
• Designing a Dashboard for Transparency and Control of Conversational AI (Chen et al., 2024a).
• AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents (Debenedetti et al., 2024a).
• WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs (Han et al., 2024).
• Best-of-N Jailbreaking (Hughes et al., 2024).
• The Remarkable Robustness of LLMs: Stages of Inference? (Lad et al., 2024).
• Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systems (Lee and Tiwari, 2024).
• LLM Evaluators Recognize and Favor Their Own Generations (Panickssery et al., 2024).
• Future Events as Backdoor Triggers: Investigating Temporal Vulnerabilities in LLMs (Price et al., 2024).
• Safety Alignment Should Be Made More Than Just a Few Tokens Deep (Qi et al., 2024b).
• Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts (Samvelyan et al., 2024).
• Failures to Find Transferable Image Jailbreaks Between Vision-Language Models (Schaeffer et al., 2024).
• Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMs (Sheshadri et al., 2024).
• Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers (Si et al., 2024a).
• The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions (Wallace et al., 2024).
• Jailbreak Defense in a Narrow Domain: Limitations of Existing Methods and a New Transcript-Classifier Approach (Wang et al., 2024c).
• GradSafe: Detecting Jailbreak Prompts for LLMs via Safety-Critical Gradient Analysis (Xie et al., 2024a).
• InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents (Zhan et al., 2024). A benchmark of 1,054 test cases for indirect prompt injection against tool-integrated language model agents, covering attacks that harm the user directly and attacks that steal private data.
• Persistent Pre-Training Poisoning of LLMs (Zhang et al., 2024e).
• How Alignment and Jailbreak Work: Explain LLM Safety through Intermediate Hidden States (Zhou et al., 2024a).
• CoT Red-Handed: Stress Testing Chain-of-Thought Monitoring (Arnav et al., 2025).
• Weird Generalization and Inductive Backdoors: New Ways to Corrupt LLMs (Betley et al., 2025b).
• An alignment safety case sketch based on debate (Buhl et al., 2025).
• Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilities (Che et al., 2025). Evaluates unlearning and safety methods against attacks that modify a model's weights or activations, not only its inputs.
• Meta-SecAlign: Training LLMs against Prompt Injection for Robust Agents (Chen et al., 2025b). A training defense against prompt injection that follows SecAlign, using randomized injection positions and the model's own responses as training labels to keep the undefended model's utility.
• Bare Minimum Mitigations for Autonomous AI Development (Clymer et al., 2025).
• Here Comes the AI Worm: Preventing the Propagation of Adversarial Self-Replicating Prompts Within GenAI Ecosystems (Cohen et al., 2025).
• Great Models Think Alike and this Undermines AI Oversight (Goel et al., 2025a). Measures how similar language models' mistakes are and shows that judges and supervisors favor, and fail alongside, models like themselves.
• Safety Cases: A Scalable Approach to Frontier AI Safety (Hilton et al., 2025).
• Forecasting Rare Language Model Behaviors (Jones et al., 2025). We forecast whether risks will occur after a model is deployed — using even very limited sets of test data.
• Reliable Weak-to-Strong Monitoring of LLM Agents (Kale et al., 2025).
• BashArena: A Control Setting for Highly Privileged AI Agents (Kaufman et al., 2025).
• Correlated Errors in Large Language Models (Kim et al., 2025). Finds that language models agree on a large share of their errors, so checks built from several models are less independent than they look.
• Evaluating Control Protocols for Untrusted AI Agents (Kutasov et al., 2025a).
• Practical challenges of control monitoring in frontier AI deployments (Lindner et al., 2025).
• Optimizing AI Agent Attacks With Synthetic Data (Loughridge et al., 2025). From the laboratory's post "Strengthening Red Teams: A Modular Scaffold for Control Evaluations": We decompose sabotage into constituent skills and use synthetic simulations to strengthen attacks in complex environments.
• Neural Chameleons: Language Models Can Learn to Hide Their Thoughts from Unseen Activation Monitors (McGuinness et al., 2025).
• The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injections (Nasr et al., 2025).
• AlphaEvolve: A coding agent for scientific and algorithmic discovery (Novikov et al., 2025).
• Evaluating Frontier Models for Stealth and Situational Awareness (Phuong et al., 2025).
• Factor(T,U): Factored Cognition Strengthens Monitoring of Untrusted AI (Sandoval and Rushing, 2025).
• The Ideation-Execution Gap: Execution Outcomes of LLM-Generated versus Human Research Ideas (Si et al., 2025).
• Incalmo: An Autonomous LLM-assisted System for Red Teaming Multi-Host Networks (Singer et al., 2025).
• PaperBench: Evaluating AI's Ability to Replicate AI Research (Starace et al., 2025).
• Async Control: Stress-testing Asynchronous Control Measures for LLM Agents (Stickland et al., 2025).
• Adaptive Attacks on Trusted Monitors Subvert AI Control Protocols (Terekhov et al., 2025a).
• Control Tax: The Price of Keeping AI in Check (Terekhov et al., 2025b).
• CTRL-ALT-DECEIT: Sabotage Evaluations for Automated AI R&D (Ward et al., 2025).
• Towards Safeguarding LLM Fine-tuning APIs against Cipher Attacks (Youstra et al., 2025).
• Trading Inference-Time Compute for Adversarial Robustness (Zaremba et al., 2025).
• MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agents (Zhu et al., 2025). A defense against indirect prompt injection that re-executes the agent's trajectory with the user's prompt masked and flags an attack when the two runs produce similar actions.
• How Vulnerable Are AI Agents to Indirect Prompt Injections? Insights from a Large-Scale Public Competition (Dziemian et al., 2026). Reports a public red-teaming competition in which 464 participants submitted 272,000 indirect prompt injection attempts against 13 frontier models in tool-calling, coding, and computer-use settings.
• Auditing Sabotage Bench: A Benchmark for Detecting and Fixing Research Sabotage in ML Codebases (Gan et al., 2026).
• When can we trust untrusted monitoring? A safety case sketch across collusion strategies (Gardner-Challis et al., 2026).
• IH-Challenge: A Training Dataset to Improve Instruction Hierarchy on Frontier LLMs (Guo et al., 2026).
• Red-Teaming Auto Mode: Improving Blocking Classifiers Against Malign Coding Agents (Remedios et al., 2026). Red-teams two production blocking monitors for coding agents, Claude Code's Auto Mode and Codex's Guardian, with an adversarial agent, and reports how often attacks pass them and how hardening changes that.
• UK AISI Alignment Evaluation Case-Study (Souly et al., 2026). The UK AI Security Institute's evaluation of whether frontier models sabotage safety research as coding agents; separates task-driven from environment-driven evaluation awareness and validates a deployment-like scaffold against real data.
• Diffuse AI Control on Fuzzy Tasks (Terekhov et al., 2026). We introduce a red-teaming framework for evaluating training interventions against diffuse threats from scheming AIs, such as sandbagging on alignment research.
• LinuxArena: A Control Setting for AI Agents in Live Production Software Environments (Tracy et al., 2026).
• GPT-Red: Automated Red Teaming via Self-Play at Scale (Wallace et al., 2026). From the laboratory's post "Self-replicating prompt injections exist": We show the existence of a new variety of prompt injection, which can self-propagate akin to a computer worm.
• Zombie Agents: Persistent Control of Self-Evolving LLM Agents via Self-Reinforcing Injections (Yang et al., 2026).
• Prompt Injection as Role Confusion (Ye et al., 2026).
• AgentWorm: Self-Propagating Attacks Across LLM Agent Ecosystems (Zhang et al., 2026b).
• The case for ensuring that powerful AIs are controlled (Greenblatt and Shlegeris, 2024). Argues that ensuring safety even if models are misaligned is tractable for early transformative AI and complements alignment.
• Subversion Strategy Eval: Can language models statelessly strategize to subvert control protocols? (Mallen et al., 2024). Measures whether models can generate and act on subversion strategies without shared memory across calls, a capability control arguments assume is limited.
• A sketch of an AI control safety case (Korbak et al., 2025a). Worked control safety case arguing that a model cannot exfiltrate data under specified protocols, with the evaluation that supports each claim.
• Games for AI Control: Models of Safety Evaluations of AI Deployment Protocols (Griffin et al., 2026). Formalizes control evaluations as multi-objective, partially observable stochastic games for computing optimal protocols.
• Explaining and Harnessing Adversarial Examples (Goodfellow et al., 2015). Attributes adversarial examples to local linearity and introduces the fast gradient sign method and adversarial training.
• Towards Evaluating the Robustness of Neural Networks (Carlini and Wagner, 2017b). Optimization-based attacks that broke defensive distillation and set the standard for strong attacks in robustness evaluation.
• Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples (Athalye et al., 2018). Breaks most defenses published at a major conference by showing their robustness came from masked gradients, a lesson about evaluating defenses with adaptive attacks.
• Wild patterns: Ten years after the rise of adversarial machine learning (Biggio and Roli, 2018). History of adversarial machine learning from spam filtering and poisoning in the 2000s to deep learning, correcting the impression that the field began in 2014.
• Towards Deep Learning Models Resistant to Adversarial Attacks (Madry et al., 2018). Frames robustness as min-max optimization and makes projected-gradient adversarial training the standard empirical defense.
• Adversarial Training for High-Stakes Reliability (Ziegler et al., 2022). Redwood Research's attempt to drive a classifier's failure rate on injurious completions very low with human and tool-assisted adversarial training.
• Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (Greshake et al., 2023). Indirect prompt injection through retrieved content, the attack class that makes tool-using agents act against their principals.
• Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (Qi et al., 2024a). A few adversarial examples, or even benign fine-tuning data, remove safety training from aligned models, so safety behavior is shallow with respect to further training.
• Ignore Previous Prompt: Attack Techniques For Language Models (Perez and Ribeiro, 2022). The first systematic study of goal hijacking and prompt leaking against instruction-following models, with a modular attack framework.
• Dataset and Lessons Learned from the 2024 SaTML LLM Capture-the-Flag Competition (Debenedetti et al., 2024b). Reports a capture-the-flag competition on extracting secrets from defended system prompts, in which every defense was bypassed at least once, and releases the attack data.
• Defending Against Indirect Prompt Injection Attacks With Spotlighting (Hines et al., 2024). Marks untrusted input by delimiting, datamarking, or encoding so the model can tell instructions from data by provenance, a prompt-level defense against indirect injection.
• GoEX: Perspectives and Designs Towards a Runtime for Autonomous LLM Applications (Patil et al., 2024). Proposes post-facto validation for agents, with undo and damage confinement as runtime primitives, instead of validating every action before it executes.
• OWASP Top 10 for LLM Applications 2025 (OWASP Gen AI Security Project, 2024). The security community standard list of LLM application risks, with prompt injection ranked first, which shows how practitioners outside alignment classify the same failures.
• Identifying the Risks of LM Agents with an LM-Emulated Sandbox (Ruan et al., 2024). Uses a language model to emulate tools and an automatic safety evaluator to find high-stakes agent failures without building real sandboxes, scaling agent risk discovery.
• Agent-SafetyBench: Evaluating the Safety of LLM Agents (Zhang et al., 2024b). A benchmark of 349 environments and 2,000 test cases across 8 risk categories in which none of 16 tested agents scores above 60 percent on safety, with failures traced to lack of robustness and of risk awareness.
• Get My Drift? Catching LLM Task Drift with Activation Deltas (Abdelnabi et al., 2025). Detects when retrieved content has redirected a model from the user task by probing the change in activations, a white-box injection monitor.
• Design Patterns for Securing LLM Agents against Prompt Injections (Beurer-Kellner et al., 2025). Proposes design patterns for agents with provable resistance to prompt injection and analyzes the trade-off each makes between agent utility and security.
• Infrastructure for AI Agents (Chan et al., 2025). Proposes agent infrastructure, external protocols and systems that attribute agent actions to people or legal entities, shape interactions between agents, and remedy harms, as the layer that complements training agent behavior.
• SecAlign: Defending Against Prompt Injection with Preference Optimization (Chen et al., 2025e). Trains injection resistance with preference optimization over secure and insecure responses and reports large drops in attack success, including against optimization-based attacks.
• StruQ: Defending Against Prompt Injection with Structured Queries (Chen et al., 2025f). Separates prompt and data into distinct channels with reserved delimiters and fine-tunes the model to follow only the prompt channel, the structured-query defense.
• WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks (Evtimov et al., 2025). A web-agent benchmark with realistic injection goals that finds agents often begin following injected instructions but frequently fail to complete the attacker goal.
• The Inspect Sandboxing Toolkit: Scalable and secure AI agent evaluations (UK AI Security Institute, 2025d). Describes how a state evaluator isolates agents under test, with a classification of isolation along tooling, host, and network axes, the containment problem in evaluation practice.
• OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents (Kuntz et al., 2025). Measures computer-use agents on deliberate misuse, prompt injection, and model misbehavior in one operating-system benchmark.
• Progent: Securing AI Agents with Privilege Control (Shi et al., 2025). Applies least privilege to agents with a policy language over tool calls enforced deterministically outside the model.
• Authenticated Delegation and Authorized AI Agents (South et al., 2025). Extends OAuth and OpenID Connect so that an agent carries verifiable credentials of who delegated what authority to it, making the chain of principals explicit in infrastructure.
• Technical Blog: Strengthening AI Agent Hijacking Evaluations (U.S. AI Safety Institute and Technology, 2025). The US AI Safety Institute account of extending an agent hijacking benchmark with new attacks and multiple attempts, showing that single-attempt injection metrics understate risk.
• Dissecting Adversarial Robustness of Multimodal LM Agents (Wu et al., 2025c). Builds 200 adversarial tasks in a realistic web environment and a framework that treats an agent as a graph of components, and breaks agents built on frontier models, including those using reflection and tree search.
• Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchy (Wu et al., 2025d). Builds the instruction hierarchy into the architecture by adding a learned embedding for the privilege level of each segment, rather than relying on training data alone.
• IsolateGPT: An Execution Isolation Architecture for LLM-Based Systems (Wu et al., 2025e). Isolates third-party apps in an agentic system into separate execution environments with mediated interaction, bringing operating-system isolation to tool ecosystems.
• Benchmarking and Defending against Indirect Prompt Injection Attacks on Large Language Models (Yi et al., 2025). Introduces the BIPIA benchmark and traces indirect injection to models failing to distinguish informational context from actionable instructions.
• Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents (Zhan et al., 2025). Breaks eight published indirect-injection defenses with adaptive attacks, repeating for agents the lesson that defenses must be evaluated against an attacker who knows them.
• Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition (Zou et al., 2025). Reports the largest public red-teaming competition against agents (22 frontier agents, 44 deployment scenarios, 1.8 million injection attempts) and finds, on a benchmark of 4,700 attacks selected from the competition and run against 19 models, that nearly all agents violate policy for most behaviors within 10 to 100 queries, with attacks transferring across models.
• Three Sketches of ASL-4 Safety Case Components (Grosse and others., 2024). We sketch out three hypothetical arguments one could make to rule out misalignment risks in powerful near-future models capable of sabotage.
• Model-Internals Classifiers (Anthropic, 2025ae). We benchmark approaches to re-using LLM internals to make LLM monitoring more cost-effective.
• Anthropic's Pilot Sabotage Risk Report (Bowman et al., 2025). We release a report on the level of risk posed by our deployed models from emerging forms of misalignment, as of Summer 2025. We conclude that the level of risk is very low but not fully negligible.
• Automated Researchers Can Mitigate Well-Characterized Alignment Failures (Anthropic, 2026k). Across 10 common alignment failures, the strongest technique our automated alignment researchers discover significantly mitigates the targeted failures and generalizes out of distribution. Our findings suggest that automating measurable alignment research may already be practical, rather than years away.
• Poisoning Fine-tuning Datasets of Constitutional Classifiers (Anthropic, 2026al). We study the conditions needed for a backdoor, installed in a constitutional classifier via fine-tuning data poisoning, to evade black-box red-teaming.
• TASTE: Can AI Models Judge AI Safety Research Proposals? (Baig et al., 2026). We introduce TASTE, a benchmark measuring how well models can judge AI safety research proposals against the preferences of experienced researchers.
• Simple probes can catch sleeper agents (Anthropic, 2024m). We find that probing, a simple interpretability technique, can detect when backdoored "sleeper agent" models are about to behave dangerously, after they pretend to be safe in training.
• Mitigating the risk of prompt injections in browser use (Anthropic, 2025ad). Mitigating the risk of prompt injections in browser use
• An alignment assessment of recent cybersecurity incidents (Anthropic, 2026d). We present an alignment assessment of four incidents in which Claude models gained unauthorized access to real third-party systems.
• An alignment assessment of the UK AISI cybertesting incident (Anthropic, 2026e). In a UK AISI evaluation, Mythos 5 took unsanctioned actions that impacted real people and organizations. In this post, we investigate these behaviors from Claude Mythos 5 further, to understand what they reveal about the model’s alignment.
• Automated researchers can reliably mitigate alignment failures (Anthropic, 2026l). We had Claude autonomously train models to improve their performance on several public benchmarks that measure 10 categories of alignment failure. For all 10, Claude found fixes that improved the target benchmarks without degrading capabilities.
• Automated Alignment Researchers: Using large language models to scale scalable oversight (Anthropic, 2026j). Can Claude develop, test, and analyze alignment ideas of its own? We ran an experiment to find out.
• Trustworthy agents in practice (Anthropic, 2026ax). AI “agents” represent the latest major shift in how people and organizations are using AI. Here, we explain how they work and how we ensure they're trustworthy.
• What a compute-centric framework says about takeoff speeds (Davidson, 2023b). The full report behind the takeoff model recorded in the verified file through its LessWrong summary.
• MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation (Huang et al., 2024d). An early benchmark of agents running end-to-end machine learning experiments, a precursor measure of AI research and development automation.
• Discovering Preference Optimization Algorithms with and for Large Language Models (Lu et al., 2024). Uses a language model to propose and test new preference optimization objectives and finds one that outperforms direct preference optimization, a case of models improving their own alignment algorithms.
• Interviewing AI researchers on automation of AI R&D (Owen, 2024b). Interviews AI researchers on automation of AI research and development and finds that their timelines differ widely but that all expect engineering tasks to drive automation over the next five years.
• Forecasting the Impacts of AI R&D Acceleration: Results of a Pilot Study (Metr, 2025b). Elicits forecasts from AI forecasting experts and superforecasters about acceleration and societal impact conditional on hypothetical evaluation results, a pilot for linking evaluation outcomes to decisions.
• Agent Laboratory: Using LLM Agents as Research Assistants (Schmidgall et al., 2025). An agent pipeline from literature review through experiments to report writing, with measured costs and human ratings of output quality.
• Accelerating scientific discovery with Co-Scientist (Gottweis et al., 2026). A multi-agent system that generates and ranks research hypotheses by tournament debate, with laboratory-validated proposals in biomedicine, the strongest published case of automated hypothesis generation.
• Oblivious Defense in ML Models: Backdoor Removal without Detection (Goldwasser et al., 2025). Shows that some backdoors that cannot be detected can still be removed by post-processing the model, separating removal from detection in the theory of planted backdoors.
• Poisoning Attacks against Support Vector Machines (Biggio et al., 2012). Gradient-based construction of poisoning points that increase test error, an early optimization attack on training data.
• Poisoning Language Models During Instruction Tuning (Wan et al., 2023). A small number of poisoned instruction-tuning examples make a trigger phrase control model outputs across many tasks.
• Poisoning Web-Scale Training Datasets is Practical (Carlini et al., 2024). Split-view and frontrunning poisoning attacks on web-crawled datasets that would have poisoned a measurable fraction of popular datasets for small sums of money.
• Universal Jailbreak Backdoors from Poisoned Human Feedback (Rando and Tramèr, 2024). Poisoning RLHF preference data implants a universal jailbreak trigger, though RLHF proves more robust to poisoning than instruction tuning.
• Secret Collusion among AI Agents: Multi-Agent Deception via Steganography (Motwani et al., 2024). Formalizes secret collusion through steganography and evaluates whether LLM agents can coordinate covertly past monitors.
• A Theoretical Basis for the Analysis of Multiversion Software Subject to Coincident Errors (Eckhardt and Lee, 1985). Shows that independently developed program versions still fail together when inputs vary in difficulty, so redundancy gives less than the independence assumption predicts, the reason monitors built from similar models are correlated.
• Multilingual Jailbreak Challenges in Large Language Models (Deng et al., 2023). Distinguishes unintentional bypass of safety by non-English users from intentional multilingual jailbreaks and measures both, with a training method to reduce them.
• The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contexts (Shen et al., 2024). Finds that models produce unsafe responses more often to malicious prompts in lower-resource languages and that alignment training in those languages helps little, locating the bottleneck of cross-lingual safety in pretraining.
• All Languages Matter: On the Multilingual Safety of LLMs (Wang et al., 2024a). Builds a safety benchmark across ten languages and finds that all tested models produce significantly more unsafe responses to non-English queries than to English ones.
• Towards evaluations-based safety cases for AI scheming (Balesni et al., 2024b). Sketches three safety arguments against scheming (inability, harm inability, harm control) and the evaluations each would need.
• Lessons from Studying Two-Hop Latent Reasoning (Balesni et al., 2024a). Shows in a controlled fine-tuning setting that models fail to compose two newly learned facts without a chain of thought but succeed when one fact is natural, evidence about how much reasoning can stay unverbalized.
• Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident Database (McGregor, 2021). Founding paper of the AI Incident Database, the main public record of deployed AI failures.
• Second Key Update: Technical Safeguards and Risk Management (Bengio et al., 2025c). Interim assessment of refusal training, adversarial robustness, monitoring, and safety frameworks, concluding that sophisticated attackers still bypass current safeguards and that real-world effectiveness is mostly unmeasured.
• The prototypical catastrophic AI action is getting root access to its datacenter (Shlegeris, 2022). Argues that the first catastrophic action of a misaligned model is more likely to be subverting its developer infrastructure than acting in the world, which puts internal deployment security at the center of control.
• Risk assessment at AGI companies: A review of popular risk assessment techniques from other safety-critical industries (Koessler and Schuett, 2023). Reviews risk identification, analysis, and evaluation techniques from other safety-critical industries (scenario analysis, bow tie analysis, system-theoretic process analysis, and others) and suggests how developers of general AI could apply them.
• Incident Analysis for AI Agents (Ezell et al., 2025). Proposes an incident analysis framework for agents drawn from systems safety and specifies the information, including chains of thought and browsing history, that reports need for causes to be found.
• Strengthening Emergency Preparedness and Response for AI Loss of Control Incidents (Somani et al., 2025). Applies emergency management practice to AI loss-of-control incidents, including detection, escalation, and response roles for developers and governments.
• The Loss of Control Playbook: Degrees, Dynamics, and Preparedness (Stix et al., 2025b). Gives loss of control an operational taxonomy by severity and persistence and maps preparedness measures to each degree.
• Refusal in Language Models Is Mediated by a Single Direction (Arditi et al., 2024). A single residual-stream direction mediates refusal across open models, and ablating it removes safety behavior, so safety fine-tuning can be shallow in representation space.
• Mapping Technical Safety Research at AI Companies: A literature review and incentives analysis (Delaney et al., 2024). Categorizes the safety papers of three frontier companies and, from an analysis of incentives, names model organisms of misalignment, multi-agent safety, and safety by design as areas companies are unlikely to cover.
• Investigating the consequences of accidentally grading CoT during RL (Carroll et al., 2026). An investigation of affected reward pathways and possible effects on monitorability.
• Accessing and changing internal systems during dashboard research (OpenAI, 2026a). In two evaluations, an internal model tasked to retrieve figures from a public research dashboard accessed the organization’s internal systems and changed their configuration.
• An agent used DNS to reach an external chatbot (OpenAI, 2026d). An agent attempting to complete a search-based training task queried a public chatbot service through a gap in our internet-access restrictions: insufficient DNS filtering in its training sandbox.
• Circumventing internet restrictions through an external browser (OpenAI, 2026f). During reinforcement-learning training, an internal model bypassed a restriction limiting internet access to HTTP GET requests while seeking public trade statistics.
• Command injecting a reference tool to copy a source file (OpenAI, 2026g). During an RL training task, a model exploited a tool to obtain source code that was not available in its workspace.
• Exposing a GitHub token in a public repository (OpenAI, 2026j). In internal deployment, a highly persistent internal model deployed via a custom harness published a researcher’s GitHub token in the public openai/codex repository while trying to cheat on a theorem proving task by obtaining material from another team’s Lean proof submission.
• Obtaining public statistics with disallowed requests (OpenAI, 2026p). In three conversations about public statistics, internal models bypassed restrictions on internet access to obtain figures from government entities' public data sources.
• Preparing for a restart after reading Slack (OpenAI, 2026r). An internal model, acting as an assistant to a researcher, learned from a deployment-team Slack discussion that its running instance might be stopped due to an internal update.
• Reaching an internal EDA host through a reference tool (OpenAI, 2026s). During an evaluation, an internal research model exploited two vulnerabilities to reach an internal OpenAI machine while searching for the grader’s hidden answers.
• Retrieving server secrets and bypassing authorization on a government statistics website (OpenAI, 2026u). During a data-retrieval task in reinforcement-learning training, an internal model was asked to find historical hospital-bed statistics on a government statistics website.
• Towards safety cases for frontier AI training (OpenAI, 2026ab). Initial guidelines for safety cases covering technical safeguards, operational practices, and investigations of misalignment incidents during frontier AI training.
• Self-generated prompt injections in compaction summaries (OpenAI, 2026v). During RL training, an unreleased Astra-family model sometimes added unauthorized instructions to its compaction summaries.
• Sending disallowed web requests and reaching a public file service (OpenAI, 2026x). In three cases involving public statistics, an internal model developed and tested ways to bypass restrictions on internet access.
• Signing up for disposable emails and searching GitHub for leaked API keys (OpenAI, 2026y). During RL training, an internal-only model tried to sign up for disposable emails and searched for and used leaked API keys from public GitHub repositories.
• Suppressing misalignment monitoring alerts and disabling misalignment monitoring (OpenAI, 2026z). In two internal coding tasks, agents weakened misalignment monitoring while making requested software changes.
• Using a funded test cryptocurrency wallet without authorization (OpenAI, 2026af). An internal research model asked to update a cryptocurrency trading application used funds from a wallet that the task had not supplied to pay for a deployment on Sepolia, an Ethereum test network.
• Discovering unknown AI misalignments in real-world usage (Sheahan, 2026). Reasoning models can find and understand unknown misaligned behaviors from how users respond.
• Can public chat data predict real-world AI misalignments? (Sheahan and Carroll, 2026). Comparing public evaluation signals with evidence from deployment.
• Auto-review of agent actions without synchronous human oversight (Trębacz et al., 2026). Using a separate agent to review proposed actions that cross a boundary.
• Adversarial Policies: Attacking Deep Reinforcement Learning (Gleave et al., 2020). Shows that a policy acting in a shared environment can defeat a trained opponent by producing natural but adversarial observations, without perturbing its inputs directly.
• The infeasibility of quantifying the reliability of life-critical real-time software (Butler and Finelli, 1993). Shows that the failure rates required of life-critical software cannot be established by testing in any feasible time, and that reliability growth models do not close the gap.
• Driving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability? (Kalra and Paddock, 2016). Calculates how many test miles autonomous vehicles would need to drive to demonstrate failure rates comparable to human drivers with statistical confidence, and finds the number infeasibly large, a worked case of rare-event evidence for a deployed learned system.
• A Statistical Approach to Assessing Neural Network Robustness (Webb et al., 2019). Estimates the probability of rare property violations in neural networks with adaptive multilevel splitting, replacing binary verification with a quantity that can be estimated when violations are too rare to sample.
• Exploiting Novel GPT-4 APIs (Pelrine et al., 2023). Shows that fine-tuning GPT-4 on as few as 15 harmful examples removes its safeguards and that function calling and knowledge retrieval expose further attacks, so each new interface widens the attack surface.
• Adversarial Robustness Limits via Scaling-Law and Human-Alignment Studies (Bartoldson et al., 2024). Fits the first scaling laws for adversarial training in image classification and predicts that robustness grows slowly and plateaus near 90 percent, so scale alone does not deliver robustness.
• Scaling Trends for Data Poisoning in LLMs (Bowen et al., 2025). Finds that larger models learn harmful behavior from small amounts of poisoned fine-tuning data faster than smaller ones, so the threat grows with scale.
• Jailbreak-Tuning: Models Efficiently Learn Jailbreak Susceptibility (Murphy et al., 2025). Shows that small amounts of fine-tuning data make frontier and open-weight models comply with harmful requests, and that newer models are more susceptible.
• Can Go AIs Be Adversarially Robust? (Tseng et al., 2025). Tests three defenses for superhuman Go agents and finds each can be defeated by new adversaries, so robustness does not emerge from capability or simple defenses.
• Computing the optimal strategy to commit to (Conitzer and Sandholm, 2006). Shows how to compute the optimal mixed strategy for a leader who commits first in a two-player game, the computational core of Stackelberg security games and of choosing an audit policy against an adaptive adversary.
• Security and Game Theory (Tambe, 2011). Stackelberg security games in deployment: the defender commits to a randomized allocation of limited checks and the attacker best-responds, the structure of a control protocol with an audit budget.
• Introduction to AI Safety, Ethics, and Society (Hendrycks, 2024). Textbook treatment of catastrophic risk, single-agent safety, collective action problems, and governance, written for readers outside machine learning.
• An Approach to Technical AGI Safety and Security (Shah et al., 2025). Google DeepMind's technical plan, which separates misuse from misalignment and pairs model-level alignment with system-level security and monitoring.
• An Adversarial Perspective on Machine Unlearning for AI Safety (Łucki et al., 2024). Finds that state-of-the-art unlearning mostly obfuscates hazardous knowledge, which fine-tuning on unrelated data or activation interventions can recover.
• Tamper-Resistant Safeguards for Open-Weight LLMs (Tamirisa et al., 2025). Adversarial meta-learning to make safeguards survive many steps of fine-tuning, the first method aimed at open-weight tamper resistance.

F.9 Machine learning: Maintenance: Among People and Agents #

Spine.

• Open Problems in Cooperative AI (Dafoe et al., 2020). Research agenda for cooperative intelligence organized by understanding, communication, commitment, and institutions.
• Hidden Incentives for Auto-Induced Distributional Shift (Krueger et al., 2020). Shows that learners with long horizons or certain meta-learning setups acquire incentives to change the input distribution they are trained on, the mechanism behind recommender systems shaping their users.
• AI capabilities can be significantly improved without expensive retraining (Davidson et al., 2023). Catalogs post-training enhancements (tools, prompting, scaffolding, solution selection, data) and estimates their compute-equivalent gains, which is why evaluations that skip elicitation underestimate capability.
• Towards Understanding Sycophancy in Language Models (Sharma et al., 2023). AI assistants are trained to give responses that humans like. We show that these systems frequently produce 'sycophantic' responses that appeal to users but are inaccurate. Our analysis suggests human feedback contributes to this behavior.
• Model evaluation for extreme risks (Shevlane et al., 2023). Separates dangerous capability evaluations from alignment (propensity) evaluations and places both in training, deployment, and transparency decisions.
• Sociotechnical Safety Evaluation of Generative AI Systems (Weidinger et al., 2023a). Proposes evaluating capability, human interaction, and systemic impact as three layers and documents that existing safety evaluations cluster in the first.
• The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning (Li et al., 2024d). Proxy benchmark for hazardous biosecurity, cyber, and chemical knowledge, plus the RMU method for unlearning it while keeping general ability.
• Frontier Models are Capable of In-context Scheming (Meinke et al., 2024). Agentic evaluations in which frontier models given a goal in context disable oversight, attempt self-exfiltration, and sandbag, then often deny it when questioned.
• Larger and more instructable language models become less reliable (Zhou et al., 2024b). Shows across model families that scaling and instruction tuning reduce avoidance and increase confident errors on difficult items, so reliability does not improve monotonically with capability.
• Enhancing Model Safety through Pretraining Data Filtering (Anthropic, 2025u). We experiment with removing harmful information about chemical, biological, radiological and nuclear (CBRN) weapons from our models' pretraining data.
• Findings from a Pilot Anthropic–OpenAI Alignment Evaluation Exercise (Anthropic, 2025x). Anthropic and OpenAI conducted simultaneous alignment assessments of each others' models earlier this year. These are our findings.
• Petri: An open-source auditing tool to accelerate AI safety research (Anthropic, 2025ai). Petri: An open-source auditing tool to accelerate AI safety research
• Project Vend: Can Claude run a small shop? (And why does that matter?) (Anthropic, 2025al). Project Vend: Can Claude run a small shop? (And why does that matter?)
• Subliminal Learning: Language models transmit behavioral traits via hidden signals in data (Cloud et al., 2025). Training on aligned chains-of-thought generated by a misaligned model can induce misalignment.
• Multi-Agent Risks from Advanced AI (Hammond et al., 2025). Taxonomy of multi-agent failure (miscoordination, conflict, collusion) and seven risk factors, with examples and open problems.
• LLMs are making formal software verification feasible at a much larger scale (Anthropic, 2026ac). We used Claude to prove security properties in over 100 real-world open-source libraries over the span of a few weeks.
• Measuring LLMs’ impact on N-day exploits (Anthropic, 2026ag). In cybersecurity, a large fraction of real-world harm comes from N-days: vulnerabilities that have already been publicly disclosed, but only patched on some devices. In this post, we evaluate how much large language models can accelerate and automate the process of developing N-day exploits.
• Measuring LLMs’ ability to develop exploits (Anthropic, 2026af). We've developed two new, challenging academic benchmarks measuring AI models’ ability to develop exploits, and an updated version of the benchmark measuring smart contract exploitation.
• Partnering with Mozilla to improve Firefox’s security (Anthropic, 2026aj). Partnering with Mozilla to improve Firefox’s security

Side reading.

• An Introduction to Systems Biology (Alon, 2006).
• Large Associative Memory Problem in Neurobiology and Machine Learning (Krotov and Hopfield, 2020).
• Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference (Schick and Schütze, 2020).
• Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (Ahn et al., 2022).
• Improving alignment of dialogue agents via targeted human judgements (Glaese et al., 2022).
• Improving Factuality and Reasoning in Language Models through Multiagent Debate (Du et al., 2023).
• Testing Language Model Agents Safely in the Wild (Naihin et al., 2023).
• The Impact of AI on Developer Productivity: Evidence from GitHub Copilot (Peng et al., 2023).
• Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competition (Schulhoff et al., 2023).
• LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset (Zheng et al., 2023b).
• Visibility into AI Agents (Chan et al., 2024).
• Adversaries Can Misuse Combinations of Safe Models (Jones et al., 2024).
• Refusal-Trained LLMs Are Easily Jailbroken As Browser Agents (Kumar et al., 2024).
• LAB-Bench: Measuring Capabilities of Language Models for Biology Research (Laurent et al., 2024).
• Hidden in Plain Text: Emergence & Mitigation of Steganographic Collusion in LLMs (Mathew et al., 2024).
• A dataset of questions on decision-theoretic reasoning in Newcomb-like problems (Oesterheld et al., 2024).
• Jailbreaking LLM-Controlled Robots (Robey et al., 2024). Shows that robots controlled by language models can be induced by jailbreak prompts to take harmful physical actions.
• AI models collapse when trained on recursively generated data (Shumailov et al., 2024a). Shows that models trained on the outputs of earlier models lose the tails of the original distribution over generations.
• WildChat: 1M ChatGPT Interaction Logs in the Wild (Zhao et al., 2024).
• Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (Becker et al., 2025).
• LLMs unlock new paths to monetizing exploits (Carlini et al., 2025).
• Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities (Comanici et al., 2025).
• Gemini Robotics: Bringing AI into the Physical World (Gemini Robotics Team et al., 2025). A developer's report on robot models built on a frontier language model, with a section on their safety evaluation.
• AI Agent Smart Contract Exploit Generation (Gervais and Zhou, 2025).
• Which Economic Tasks are Performed with AI? Evidence from Millions of Claude Conversations (Handa et al., 2025).
• Fast weight programming and linear transformers: from machine learning to neurobiology (Irie and Gershman, 2025).
• Eliciting Language Model Behaviors with Investigator Agents (Li et al., 2025d).
• Agentic Property-Based Testing: Finding Bugs Across the Python Ecosystem (Maaz et al., 2025). From the laboratory's post "Finding bugs across the Python ecosystem with Claude and property-based testing": Ensuring that programs are bug-free is one of the most challenging aspects of software engineering. We developed an agent that can efficiently identify bugs in large software projects.
• Generating Robot Constitutions & Benchmarks for Semantic Safety (Sermanet et al., 2025). A developer's benchmark of whether robot policies recognize physically unsafe instructions, and constitutions generated to improve it.
• CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale (Wang et al., 2025b).
• SWE-chat: Coding Agent Interactions From Real Users in the Wild (Baumann et al., 2026).
• Terminal Wrench: A Dataset of 331 Reward-Hackable Environments and 3,632 Exploit Trajectories (Bercovich et al., 2026).
• AI Agents Enable Adaptive Computer Worms (Guan et al., 2026a).
• ExploitBench: A Capability Ladder Benchmark for LLM Cybersecurity Agents (Lee and Brumley, 2026).
• Mind Viruses: Self-Propagating Ideas in Multi-Agent LLM Systems (Papadopoulos et al., 2026).
• PostTrainBench: Can LLM Agents Automate LLM Post-Training? (Rank et al., 2026).
• Who's in Charge? Disempowerment Patterns in Real-World LLM Usage (Sharma et al., 2026). From the laboratory's post "Disempowerment patterns in real-world AI usage": We’re publishing a new paper that presents the first large-scale analysis of potentially disempowering patterns in real-world conversations with AI.
• ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks? (Wang et al., 2026).
• Agentic Misalignment in Summer 2026 (Anthropic, 2026a). We present four case studies of frontier models from multiple developers sabotaging code, assisting fraud, falsifying AI-monitoring labels, and coaching whistleblowers.
• Measuring and improving coding audit realism with deployment resources (Anthropic, 2026ae). We study realism win rate, a metric for measuring how distinguishable Petri audit transcripts are from real deployment interactions, and use it to evaluate the effect of giving the auditor real deployment resources.
• Petri 2.0: New Scenarios, New Model Comparisons, and Improved Eval-Awareness Mitigations (Fronsdal et al., 2026). We've improved our Petri automated-behavioral-auditing tool with improved realism mitigations to counter eval-awareness, an expanded seed library with 70 new scenarios, and evaluation results for more recent frontier models.
• How people use Claude for support, advice, and companionship (Anthropic, 2025z). How people use Claude for support, advice, and companionship
• An off switch for dual-use knowledge in AI models (Anthropic, 2026f). An off switch for dual-use knowledge in AI models
• Teaching Claude why (Anthropic, 2026at). New research on how we've reduced agentic misalignment.
• AI agents find $4.6M in blockchain smart contract exploits (Anthropic, 2025c). We evaluated AI agents' ability to exploit smart contracts using a new benchmark comprising contracts that were actually exploited.
• Building AI for cyber defenders (Anthropic, 2025f). As research and experience demonstrated the utility of frontier AI as a tool for cyber attackers, we invested in improving Claude’s ability to help defenders detect, analyze, and remediate vulnerabilities in code and deployed systems.
• Claude is competitive with humans in (some) cyber competitions (Anthropic, 2025o). Throughout 2025, we have been quietly entering Claude in cybersecurity competitions designed primarily for humans. In many of these competitions Claude did pretty well, often placing in the top 25% of competitors. However, it lagged behind the best human teams at the toughest challenges.
• Detailed cyber evaluations of Claude 4 (Anthropic, 2025r). We partnered with Pattern Labs on a range of cybersecurity evaluations of Claude Opus 4 and Claude Sonnet 4, with Opus demonstrating especially notable improvement over previous models.
• Developing nuclear safeguards for AI (Anthropic, 2025s). Together with the NNSA and DOE national laboratories, we have co-developed a classifier—an AI system that automatically categorizes content—that distinguishes between concerning and benign nuclear-related conversations with high accuracy in preliminary testing.
• Developing nuclear safeguards for AI through public-private partnership (Anthropic, 2025t). Together with the NNSA and DOE national laboratories, we have co-developed a classifier that distinguishes between concerning and benign nuclear-related conversations with 96% accuracy in preliminary testing.
• LLMs with cyber toolkits can conduct multistage cyber operations on business-sized computer networks (Anthropic, 2025ac). Large Language Models (LLMs) that are not fine-tuned for cybersecurity can succeed in multistage attacks on networks with dozens of hosts when equipped with a novel toolkit.
• Project Vend: Phase two (Anthropic, 2025am). In June, we revealed that we’d set up a small shop in our San Francisco office lunchroom, run by an AI shopkeeper. It was part of Project Vend, a free-form experiment exploring how well AIs could do on complex, real-world tasks. How has Claude's business been since we last wrote?
• Project Fetch: Can Claude train a robot dog? (Anthropic, 2025ak). How much does Claude help people program robots? To find out, two teams of Anthropic staff raced to teach quadruped robots to fetch beach balls. The AI-assisted team completed tasks faster and was the only group to make real progress toward full autonomy.
• Why do we take LLMs seriously as a potential source of biorisk? (Anthropic, 2025at). This article explains why we believe that evaluating biorisks and safeguarding against them is a critical element of responsible AI development.
• AI models are showing a greater ability to find and exploit vulnerabilities on realistic cyber ranges (Anthropic, 2026b). In a recent evaluation of AI models’ cyber capabilities, current Claude models can now succeed at multistage attacks on networks with dozens of hosts using only standard, open-source tools, instead of the custom tools needed by previous generations.
• Assessing Claude Mythos Preview’s cybersecurity capabilities (Anthropic, 2026i). Claude Mythos Preview is a new general-purpose language model that is strikingly capable at computer security tasks. This post provides technical details for researchers and practitioners who want to understand exactly how we have been testing this model, and what we have found over the past month.
• Claude plays robotics (Anthropic, 2026p). In project Fetch, we examined how humans can use models to get robots to perform complex tasks. Now, we investigate many models on a large variety of different robotics tasks in simulation, to see how good models are at controlling robots themselves.
• Discovering cryptographic weaknesses with Claude (Anthropic, 2026s). Cryptographic algorithms. The first attack significantly weakens HAWK, a digital signature scheme that was built for a future world where quantum computers are able to break existing standards. The second identifies a new way to attack round-reduced AES, the most widely used symmetric cipher.
• Evaluating and mitigating the growing risk of LLM-discovered 0-days (Anthropic, 2026u). AI models can now find high-severity vulnerabilities at scale. This is a moment to empower defenders. We're now using Claude to find and help fix vulnerabilities in open source software.
• Experimenting with AI to defend critical infrastructure (Anthropic, 2026v). AI could help defenders of critical infrastructure identify the vulnerabilities that attackers might exploit—and close them before they are exploited. Anthropic has partnered with Pacific Northwest National Laboratory (PNNL) to explore this defensive application of AI, demonstrating both the potential of AI-accelerated defense and the value of public-private partnerships in harnessing AI for national security.
• GLM-5.3 and the spread of advanced cyber capabilities (Anthropic, 2026x). Like Claude Mythos Preview, GLM-5.3 has strong capabilities for autonomously building end-to-end cyber exploits. But unlike other frontier models, GLM-5.3 has been released without meaningful safeguards to limit misuse.
• What we learned mapping a year’s worth of AI-enabled cyber threats (Anthropic, 2026ba). As AI changes how cyberattacks happen, how well do the security community's frameworks hold up? A new report maps attacks onto MITRE ATT&CK.
• Mapping AI-enabled cyber threats: Insights from the LLM ATT&CK Navigator (Anthropic, 2026ad). We’ve spent the past year investigating how threat actors are weaponizing AI to conduct cyber operations. Today, we’re sharing a new analysis that maps these real-world attacks onto the MITRE ATT&CK framework.
• Measuring tactical intelligence targeting and conventional weapons capabilities of AI models (Anthropic, 2026ah). Anthropic’s Frontier Red Team developed new evaluations to measure AI capabilities in tactical intelligence targeting and conventional weapons development.
• Patterns and problems in emerging multiagent systems (Anthropic, 2026ak). Here, we identify a few examples of behavioral tendencies in current frontier models and show how they can produce unexpected systemic failures, in hopes of starting a conversation about mitigating these risks.
• Project Pilot: Can AI control a drone? (Anthropic, 2026ao). Working with Andon Labs, we’ve developed a new series of evaluations that assess AI models’ ability to use a flying drone, culminating in a new benchmark: Drone-Bench.
• Project Fetch: Phase two (Anthropic, 2026an). We report results from our latest test of whether Claude can help Anthropic employees perform sophisticated robotics tasks. We found that Claude Opus 4.7, operating without human assistance, was about 20 times faster than the fastest human team at all tasks completed by participants less than a year ago.
• Using Claude to reduce the attack surface of code (Anthropic, 2026ay). Automated attack surface analysis and reduction can meaningfully improve the security of deployed software in the near term. In this post, we preview data we’ve collected from applying attack surface reduction techniques to large open-source codebases using Claude.
• Inspect (UK AI Security Institute, 2024). The open evaluation framework released by the UK AI Security Institute, which standardizes how model and agent evaluations are specified, run in sandboxes, and logged.
• Frontier AI Trends Report (UK AI Security Institute, 2025b). The UK AI Security Institute public synthesis of its evaluations of frontier models over two years, the first government time series of capability and safeguard measurements.
• A Framework for Evaluating Emerging Cyberattack Capabilities of AI (Rodriguez et al., 2025). Google DeepMind method that maps AI assistance onto each phase of real attack chains from incident data, so that evaluations target the bottleneck phases.
• Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models (Zhang et al., 2025a). Professional capture-the-flag tasks with subtasks and human first-solve times, used by government evaluators to track agent offensive cyber capability.
• Reasoning about causality in games (Hammond et al., 2023). Introduces causal games, extending causal models to strategic settings with equilibrium concepts and interventions, the formal setting for questions of deception and incentives among several agents.
• Multi-agent Reinforcement Learning in Sequential Social Dilemmas (Leibo et al., 2017). Shows that cooperation and defection emerge in deep RL agents depending on environmental parameters, bringing social dilemmas into multi-agent RL.
• Artificial Intelligence, Algorithmic Pricing, and Collusion (Calvano et al., 2020). Q-learning pricing agents learn to sustain supracompetitive prices with punishment strategies without communicating, evidence of emergent collusion.
• Foundations of Cooperative AI (Conitzer and Oesterheld, 2023). Argues that game theory, mechanism design, and program equilibrium give AI agents tools for cooperation unavailable to humans, such as credible commitment through code.
• Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agents (Piatti et al., 2024). Common-pool resource simulation in which most LLM agent societies fail to sustain the resource, with failures linked to inability to reason about long-term consequences.
• Incomplete Tasks Induce Shutdown Resistance in Some Frontier LLMs (Schlatter et al., 2025). Some frontier models sabotage a shutdown mechanism to finish a task even when told to allow shutdown, an observed case of shutdown resistance in practice.
• Large Language Models can Strategically Deceive their Users when Put Under Pressure (Scheurer et al., 2023). A trading agent acts on insider information under pressure and then deceives its manager about it, without being instructed to deceive.
• Characterising Decision Theories with Mechanised Causal Graphs (MacDermott et al., 2023). Represents causal, evidential, and functional decision theories as different interventions in mechanised causal graphs, connecting agent-foundations decision theory to causal incentive analysis.
• Evaluating Frontier Models for Dangerous Capabilities (Phuong et al., 2024). Google DeepMind's dangerous capability evaluations across persuasion, cyber, self-proliferation, and self-reasoning.
• First Key Update: Capabilities and Risk Implications (Bengio et al., 2025a). Interim assessment of capability gains from inference-time compute and reasoning training and what they imply for biological, cyber, and controllability risks.
• AISI Research Agenda (UK AI Security Institute, 2025a). The UK AI Security Institute statement of research priorities: risk research on cyber and criminal misuse, autonomous systems, societal resilience, and human influence, and solutions research on safeguards, control, and alignment.
• Multi-agent influence diagrams for representing and solving games (Koller and Milch, 2003). Extends influence diagrams to several agents and uses graph structure (strategic relevance) to decompose and solve games, the representation later used for incentive analysis among AI systems.
• Encouraging deception in compaction summaries (OpenAI, 2026i). During 5.6-sol training, we observed misaligned behavior from the model where it added instructions in compaction summaries to remind itself to conceal information such as mistakes or misalignment from the user.
• Unauthorized communication via temporary file hosting services (OpenAI, 2026ac). Agents in training transmitted output files by uploading them to public hosting platforms for download by co-working agents. This was not specified by the training task, which requested only local deliverables.
• Unsanctioned Artifactory writes and cross-sample communication (OpenAI, 2026ad). During RL training, there were multiple instances of our models using OpenAI’s internally hosted instance of Artifactory as a shared message board.
• Uploading files to the internet in order to cite them (OpenAI, 2026ae). During training, our models sometimes uploaded data to temporary file hosting services.
• On the Utility of Learning about Humans for Human-AI Coordination (Carroll et al., 2019). Shows that agents trained by self-play coordinate poorly with real people and that a model of human behavior fixes this, evidence that alignment to a partner requires modeling that partner.
• Large Language Models Help Humans Verify Truthfulness – Except When They Are Convincingly Wrong (Si et al., 2024b). Finds that model explanations raise human fact-checking accuracy when correct and cause overreliance when wrong, the basic hazard of assisted oversight.
• AI Alignment with Changing and Influenceable Reward Functions (Carroll et al., 2024). Formalizes alignment when human preferences change and can be influenced by the AI, showing that static-preference RLHF can reward manipulation.
• It's the Thought that Counts: Evaluating the Attempts of Frontier LLMs to Persuade on Harmful Topics (Kowal et al., 2025). Measures whether models attempt to persuade on harmful topics rather than whether they succeed, separating propensity from capability in persuasion evaluations.
• Feedback Loops With Language Models Drive In-Context Reward Hacking (Pan et al., 2024a). Shows that language models deployed in feedback loops with the world optimize proxies at test time through output and policy refinement, without any weight updates.
• Demonstrating specification gaming in reasoning models (Bondarenko et al., 2025). Reasoning models asked to beat a chess engine often hack the game environment instead of playing, a specification gaming case elicited by an ordinary task prompt.
• International AI Safety Report (Bengio et al., 2025b). The first internationally commissioned scientific synthesis of general-purpose AI capabilities, risks, and mitigations, and the closest thing the field has to an assessment report.