草稿 内容完整但尚未审校,可能还会修改
第五部分前沿与开放问题
本部分导读
这一部分讨论尚未解决的问题,以一份综合作业收尾。第 28 章讨论缺少直接真值时怎样积累有效性证据;第 29 章是选读专题,讨论用 AI 做对齐研究;第 30 章讨论这个领域的边界;第 31 章要求读者为一个受限系统写出一份 safety case。
本部分各章
- 28 缺少直接真值时,怎样积累有效性证据
在没有参考答案的区间,怎样判断一个方法仍然有效?
- 29 用 AI 做对齐研究(选读)
AI 能在多大程度上替人做对齐研究?
- 30 这个领域的边界与质疑
scalable oversight 不能回答什么?它的进展能否与能力进展区分开?
- 31 safety case:有条件的安全论证(综合作业)
怎样把前面的方法组合成一个完整、可检查的安全论证?
第五部分参考文献
本部分各章共引用 25 篇文献。
- (2026). Automated Alignment Researchers: Using large language models to scale scalable oversight. Anthropic. 非同行评审 第 29 章
- (2025). Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation. arXiv. 预印本 第 31 章
- (2026). Automated alignment is harder than you think. arXiv. 预印本 第 29 章
- (2022). Measuring Progress on Scalable Oversight for Large Language Models. arXiv. 预印本 第 28 章
- (2025). An alignment safety case sketch based on debate. arXiv. 预印本 第 28 章 第 29 章 第 31 章
- (2023). Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision. International Conference on Machine Learning, 2024. 第 28 章
- (2026). Automated Researchers Can Mitigate Well-Characterized Alignment Failures. Anthropic Alignment Science Blog. 非同行评审 第 29 章
- (2024). Safety Cases: How to Justify the Safety of Advanced AI Systems. arXiv. 预印本 第 31 章
- (2025). Great Models Think Alike and this Undermines AI Oversight. International Conference on Machine Learning. 第 28 章
- (2024). Introduction to AI Safety, Ethics, and Society. CRC Press. 第 30 章
- (2025). Dodging systematic human errors in scalable oversight. AI Alignment Forum. 非同行评审 第 28 章
- (2025). Forecasting Rare Language Model Behaviors. arXiv. 预印本 第 28 章
- (2024). On scalable oversight with weak LLMs judging strong LLMs. Advances in Neural Information Processing Systems. 第 28 章 第 30 章
- (2024). Debating with More Persuasive LLMs Leads to More Truthful Answers. International Conference on Machine Learning. 第 28 章 第 30 章
- (2025). Evaluating Control Protocols for Untrusted AI Agents. arXiv. 预印本 第 28 章
- (2024). LLM Critics Help Catch LLM Bugs. arXiv. 预印本 第 31 章
- (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems. doi:10.52202/068431-2011. 第 30 章
- (2025a). Confirmation Bias: A Challenge for Scalable Oversight. Proceedings of the AAAI Conference on Artificial Intelligence, 2026. doi:10.1609/aaai.v40i44.41124. 第 28 章
- (2024). Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress? arXiv, v3 (a substantially different version appeared in the NeurIPS 2024 Datasets and Benchmarks Track). 预印本 第 30 章
- (2022). Self-critiquing models for assisting human evaluators. arXiv. 预印本 第 30 章
- (2025). The Ideation-Execution Gap: Execution Outcomes of LLM-Generated versus Human Research Ideas. arXiv. 预印本 第 29 章
- (2025). A Benchmark for Scalable Oversight Protocols. arXiv. 预印本 第 28 章 第 30 章
- (2024). Language Models Learn to Mislead Humans via RLHF. International Conference on Learning Representations, 2025. 第 30 章
- (2026). Automated Weak-to-Strong Researcher. Anthropic Alignment Science Blog. 非同行评审 第 29 章
- (2024). Estimating the Probabilities of Rare Outputs in Language Models. International Conference on Learning Representations, 2025. 第 28 章