综合
本书最后把各条线索汇集在一起。一次比较实际上混合了多种成分:稳定的偏好、评价选项时的结构化噪声、提问本身引起的改变,以及不完备或有意随机的回答。因此,偏好优化既是估计,也是干预;好的系统与好的研究应有的样子也随之改变。
三章依次说明一次比较测量什么;为构建与评估偏好优化系统给出具体建议,包括更简单的方法何时应当胜出;最后讨论未解决问题与最能改变我们认识的那个实验,并展望若干研究方向,在这些方向上,这些方法所针对的问题正日益重要。
本部分取材于全书,但可以单独阅读。
本部分各章
- 45 一次比较测量什么
该领域如今的难点所在,其他学科中哪些结果经受住了检验,以及一次比较测量的是什么:稳定偏好、结构化评价噪声、查询引起的改变与背后并无偏好的回答。一次偏好优化会话既是估计,也是干预。
- 46 构建与评估偏好优化系统
具体建议:先判断究竟是否应当采用偏好优化,再依次讨论观测模型、代理模型、查询设计、非平稳性、停止、评测与伦理,并说明哪些情形下应当默认采用更简单的方法。
- 47 未解决问题与判定实验
十八个未解决问题,每个都配有一个能够解决它的实验;一个尚无人做过的实验(随机化查询顺序,一周后重测),其描述具体到可以照此实施;以及全书的结论。
第十部分参考文献
本部分各章共引用 127 篇文献。
- (2026). Benchmarking self-driving labs. Digital Discovery. 第 46 章 第 47 章
- (2025). Ranges of Randomization. Review of Economics and Statistics. 第 45 章
- (2023). Identifying Nontransitive Preferences. University of Zurich. 工作论文 第 45 章
- (2022). Evaluating Deliberative Competence: A Simple Method with an Application to Financial Choice. American Economic Review. 第 45 章
- (2025). No evidence for decision fatigue using large-scale field data from healthcare. Communications Psychology. 第 45 章
- (2022). Solutions to preference manipulation in recommender systems require knowledge of meta-preferences. FAccTRec Workshop (RecSys 2022). 研讨会论文 第 45 章
- (2023). qEUBO: A Decision-Theoretic Acquisition Function for Preferential Bayesian Optimization. International Conference on Artificial Intelligence and Statistics. 第 45 章
- (2025). A systematic review and meta-analyses of the temporal stability and convergent validity of risk preference measures. Nature Human Behaviour. doi:10.1038/s41562-024-02085-2. 第 45 章
- (2023). The functional form of value normalization in human reinforcement learning. eLife. 第 45 章
- (2018). Reference-point centering and range-adaptation enhance human reinforcement learning at the cost of irrational preferences. Nature Communications. 第 45 章
- (2024). The online metacognitive control of decisions. Communications Psychology. doi:10.1038/s44271-024-00071-y. 第 46 章
- (2009). Beyond Revealed Preference: Choice-Theoretic Foundations for Behavioral Welfare Economics *. Quarterly Journal of Economics. 第 45 章
- (2017). Noisy preferences in risky choice: A cautionary note. Psychological Review. 第 45 章
- (2022). A meta-analysis on the effect of visual attention on choice. Journal of Experimental Psychology: General. 第 46 章
- (2019). Asking Easy Questions: A User-Friendly Approach to Active Reward Learning. CoRL 2019. 第 46 章
- (2024). Modelling individual aesthetic judgements over time. Philosophical Transactions of the Royal Society B: Biological Sciences. 第 45 章
- (2024). Meta-analysis of Empirical Estimates of Loss Aversion. Journal of Economic Literature. 第 45 章 第 46 章
- (2022). Estimating and Penalizing Induced Preference Shifts in Recommender Systems. ICML 2022. 第 45 章 第 46 章
- (2023). Characterizing Manipulation from AI Systems. Equity and Access in Algorithms, Mechanisms, and Optimization. 第 45 章 第 46 章
- (2024). AI Alignment with Changing and Influenceable Reward Functions. International Conference on Machine Learning. 第 45 章 第 47 章
- (2019). Revealed preferences under uncertainty: Incomplete preferences and preferences for randomization. Journal of Economic Theory. 第 45 章
- (2024). What’s so Hard about Hard Choices? Erasmus Journal for Philosophy and Economics. doi:10.23941/ejpe.v17i1.872. 第 45 章
- (2026). Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences. International Conference on Artificial Intelligence and Statistics. 第 45 章 第 46 章
- (2023). Extracting medicinal chemistry intuition via preference machine learning. Nature Communications. doi:10.1038/s41467-023-42242-1. 第 45 章
- (2022). Choice, deferral, and consistency. Quantitative Economics. 第 45 章
- (2025). How to Capture Human Preference: Commissioning of a Robotic Use-Case via Preferential Bayesian Optimisation. arXiv. 预印本 第 45 章
- (2022). Preference Dynamics Under Personalized Recommendations. EC 2022. 第 45 章 第 46 章 第 47 章
- (2025). Preferential Bayesian optimization improves the efficiency of printing objects with subjective qualities. Digital Discovery. 第 47 章
- (2026). User preference in the personalized control of an ankle prosthesis: a case study. Journal of NeuroEngineering and Rehabilitation. doi:10.1186/s12984-026-01931-w. 第 46 章
- (2018). Human-in-the-Loop Optimization of Hip Assistance with a Soft Exosuit during Walking. Science Robotics. 第 47 章
- (2024). Quality Diversity through Human Feedback: Towards Open-Ended Diversity-Driven Optimization. ICML 2024. 第 46 章
- (2026). We Still Don't Understand High-Dimensional Bayesian Optimization. AISTATS 2026 (best student paper). 第 45 章 第 46 章
- (2010). Reconsidering the Effect of Market Experience on the "Endowment Effect". Econometrica. 第 45 章
- (2021). Choice changes preferences, not merely reflects them: A meta-analysis of the artifact-free free-choice paradigm. Journal of Personality and Social Psychology. 第 45 章 第 47 章
- (2025). Consecutive Preferential Bayesian Optimization. arXiv. 预印本 第 46 章
- (2022b). Regulation (EU) 2022/2065 on a Single Market for Digital Services (Digital Services Act). Official Journal of the European Union (EUR-Lex). 非同行评审 第 46 章
- (2011). On the multi-utility representation of preference relations. Journal of Mathematical Economics. 第 45 章
- (2021). Human-in-the-loop optimization of retinal prostheses encoders. Sorbonne Université. 学位论文 第 47 章
- (2026). Agents, Alignment, and the Many Faces of Autonomy. Minds and Machines. 第 45 章
- (2020). Discrete Choice and Rational Inattention: A General Equivalence Result. International Economic Review. 第 45 章
- (2021). Preference stability in discrete choice experiments. Some evidence using eye-tracking. Journal of Behavioral and Experimental Economics. 第 45 章
- (2014). The Limits of Attraction. Journal of Marketing Research. 第 45 章
- (2019). Goal congruency dominates reward value in accounting for behavioral and neural correlates of value-based decision-making. Nature Communications. 第 45 章
- (2024). Axioms for AI Alignment from Human Feedback. Advances in Neural Information Processing Systems 37. 第 45 章
- (2023). The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Types. AAAI. 第 45 章 第 46 章
- (2025). Optimal adaptive Bayesian design in choice experiments: performance for the population versus individuals. Marketing Letters. 第 46 章
- (2025). How human–AI feedback loops alter human perceptual, emotional and social judgements. Nature Human Behaviour. doi:10.1038/s41562-024-02077-2. 第 45 章 第 46 章
- (2026). Multiobjective Optimisation for Others: How Anchoring Effects Change Based on Who Guides the Interaction. Journal of Multi-Criteria Decision Analysis. 第 46 章
- (2025). Performative Prediction: Past and Future. Statistical Science. 第 45 章
- (2005). The Impact of Utility Balance and Endogeneity in Conjoint Analysis. Marketing Science. 第 46 章
- (2019). Active ranking from pairwise comparisons and when parametric assumptions do not help. The Annals of Statistics. 第 45 章
- (2019). Graph Resistance and Learning from Pairwise Comparisons. ICML. 第 45 章 第 46 章
- (2024). Vanilla Bayesian Optimization Performs Great in High Dimensions. International Conference on Machine Learning. 第 46 章 第 47 章
- (2022). The role of user preference in the customized control of robotic exoskeletons. Science Robotics. 第 46 章
- (2011). Statistical ranking and combinatorial Hodge theory. Mathematical Programming. 第 45 章
- (2017). Contemporary Guidance for Stated Preference Studies. Journal of the Association of Environmental and Resource Economists. 第 46 章
- (2026). Constructive Alignment: Governing Preference Dynamics in Human-AI Interaction. AAAI-26 Workshop on Machine Ethics. 研讨会论文 第 45 章 第 46 章
- (2025). Bayesian Optimization from Human Feedback: Near-Optimal Regret Bounds. International Conference on Machine Learning. 第 45 章 第 47 章
- (2024). Stochastic Monotonicity and Random Utility Models: The Good and The Ugly. arXiv preprint 2409.00704. 预印本 第 46 章
- (2024). On the Pros and Cons of Active Learning for Moral Preference Elicitation. AIES. 第 46 章
- (2026). Moral Change or Noise? On Problems of Aligning AI With Temporally Unstable Human Feedback. AAAI. 第 45 章 第 46 章 第 47 章
- (2026). PRISM-X: Experiments on Personalised Fine-Tuning with Human and Simulated Users. arXiv. 预印本 第 46 章 第 47 章
- (2020). Sequential Gallery for Interactive Visual Design Optimization. ACM Transactions on Graphics 39(4) (SIGGRAPH 2020). 第 46 章
- (2026). A Finite Time Analysis of Thompson Sampling for Bayesian Optimization with Preferential Feedback. International Conference on Artificial Intelligence and Statistics. 第 45 章 第 47 章
- (2024a). Enhancing Preference-based Linear Bandits via Human Response Time. Advances in Neural Information Processing Systems. 第 45 章
- (2026c). Curiosity is Knowledge: Self-Consistent Learning and No-Regret Optimization with Active Inference. arXiv preprint 2602.06029. 预印本 第 45 章
- (2026f). Preference-Guided Prompt Optimization for Text-to-Image Generation. CHI 2026. 第 47 章
- (2026). Efficient Human-in-the-Loop Optimization via Priors Learned from User Models. CHI 2026. 第 46 章
- (2026b). GimmBO: Interactive Generative Image Model Merging via Bayesian Optimization. ACM Transactions on Graphics. doi:10.1145/3811293. 第 46 章 第 47 章
- (2020). Testing the Random Utility Hypothesis Directly. The Economic Journal. doi:10.1093/ej/uez039. 第 45 章
- (2010). NAUTILUS method: An interactive technique in multiobjective optimization based on the nadir point. European Journal of Operational Research. 第 46 章
- (2021). Whence the Expected Free Energy? Neural Computation. 第 45 章
- (2022). When Choices Are Mistakes. American Economic Review. 第 45 章 第 46 章
- (2026). Revealed Incomplete Preferences. Working paper (author's website). 工作论文 第 45 章 第 46 章 第 47 章
- (2015). On making the right choice: A meta-analysis and large-scale replication attempt of the unconscious thought advantage. Judgment and Decision Making. 第 45 章
- (2025). Cooperative Design Optimization through Natural Language Interaction. UIST 2025. 第 45 章 第 46 章
- (2017). The evolution of paired preference tests from forced choice to the use of ‘No Preference’ options, from preference frequencies to d′ values, from placebo pairs to signal detection. Trends in Food Science & Technology. 第 46 章 第 47 章
- (2026b). Random Is Hard to Beat: Active Selection in online DPO with Modern LLMs. ICLR 2026 Workshop: I Can't Believe It's Not Better (ICBINB). 研讨会论文 第 45 章 第 46 章 第 47 章
- (2022). Indifference, indecisiveness, experimentation, and stochastic choice. Theoretical Economics. 第 45 章 第 46 章 第 47 章
- (2024). Decisions under Risk Are Decisions under Complexity. American Economic Review. 第 45 章
- (2022). The Human in the Infinite Loop: A Case Study on Revealing and Explaining Human-AI Interaction Loop Failures. Mensch und Computer 2022. 第 45 章 第 46 章
- (2026). Learning Feasibility-Aware Latent Spaces for Preference-Based Exploration of Procedural Automotive Wheel Designs. arXiv. 预印本 第 46 章
- (2024). Bandits with Preference Feedback: A Stackelberg Game Perspective. Advances in Neural Information Processing Systems. doi:10.52202/079017-0383. 第 45 章
- (2026). Efficient Personalization of Generative User Interfaces. arXiv. 预印本 第 45 章 第 46 章
- (2020). Performative Prediction. ICML. 第 45 章
- (2023). Nudging for changing selves. Synthese. 第 45 章 第 46 章
- (2021). How adaptation, training, and customization contribute to benefits from exoskeleton assistance. Science Robotics. 第 46 章 第 47 章
- (2019). Efficient coding of subjective value. Nature Neuroscience. 第 45 章
- (2023). The intrinsic variance of beauty judgment. Attention, Perception, & Psychophysics. 第 45 章
- (2022). Efficient coding of numbers explains decision bias and noise. Nature Human Behaviour. 第 45 章
- (2026). Personalized Image Generation via Human-in-the-loop Bayesian Optimization. International Conference on Machine Learning. 第 47 章
- (2026). Observation Moderates the Moral Licensing Effect: A Meta-Analytic Test of Interpersonal and Intrapsychic Mechanisms. Personality and Social Psychology Bulletin. 第 45 章
- (2019). Ellipsoidal Methods for Adaptive Choice-Based Conjoint Analysis. Operations Research. 第 46 章
- (2025). Preference Learning with Response Time: Robust Losses and Guarantees. NeurIPS. 第 46 章
- (2026). User preference-based human-in-the-loop tuning of exoskeleton assistance during walking. npj Biomedical Innovations. doi:10.1038/s44385-026-00085-7. 第 45 章 第 46 章 第 47 章
- (2025). Evaluating Deep Human-in-the-Loop Optimization for Retinal Implants Using Sighted Participants. 2025 47th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). doi:10.1109/embc58623.2025.11253762. 第 45 章 第 46 章
- (2016). Estimation from Pairwise Comparisons: Sharp Minimax Bounds with Topology Dependence. Journal of Machine Learning Research. 第 45 章 第 46 章
- (2026). Adaptive KappaSharp: Condition-Number Shaping for Preferential Bayesian Optimization. arXiv. 预印本 第 45 章 第 46 章 第 47 章
- (2025b). Early versus late noise differentially enhances or degrades context-dependent choice. Nature Communications. doi:10.1038/s41467-025-59140-3. 第 45 章 第 46 章
- (2022). High-value decisions are fast and accurate, inconsistent with diminishing value sensitivity. Proceedings of the National Academy of Sciences. 第 45 章 第 46 章
- (2021). Bayesian reaction optimization as a tool for chemical synthesis. Nature. 第 47 章
- (2024). Response Time Improves Gaussian Process Models for Perception and Preferences. Uncertainty in Artificial Intelligence. 第 45 章 第 46 章 第 47 章
- (2021). Preferential Batch Bayesian Optimization. IEEE MLSP 2021. 第 46 章
- (2024). Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF. ICLR 2024. 第 45 章
- (2024). Behavioral Biases Are Temporally Stable. Working paper (author's website). 工作论文 第 45 章
- (2022). The Network HHD: Quantifying Cyclic Competition in Trait-Performance Models of Tournaments. SIAM Review. 第 45 章
- (2018b). Stagewise Safe Bayesian Optimization with Gaussian Processes. International Conference on Machine Learning. 第 47 章
- (2026). Bayesian Preference Elicitation: Human-In-The-Loop Optimization of An Active Prosthesis. arXiv. 预印本 第 45 章 第 46 章
- (2023). Towards Practical Preferential Bayesian Optimization with Skew Gaussian Processes. International Conference on Machine Learning. 第 46 章 第 47 章
- (2024). A Multilab Replication of the Induced-Compliance Paradigm of Cognitive Dissonance. Advances in Methods and Practices in Psychological Science. 第 45 章
- (2020). Stability and change of basic personal values in early adolescence: A 2‐year longitudinal study. Journal of Personality. 第 45 章 第 47 章
- (2018). Stronger shared taste for natural aesthetic domains than for artifacts of human culture. Cognition. 第 45 章 第 46 章 第 47 章
- (2020). On the equivalence of optimal recommendation sets and myopically optimal query sets. Artificial Intelligence. 第 45 章
- (2021). A Multisite Preregistered Paradigmatic Test of the Ego-Depletion Effect. Psychological Science. 第 45 章
- (2022). Personalizing over-the-counter hearing aids using pairwise comparisons. Smart Health. doi:10.1016/j.smhl.2021.100231. 第 47 章
- (2024). The Effects of Generative AI on Design Fixation and Divergent Thinking. Proceedings of the CHI Conference on Human Factors in Computing Systems. 第 46 章
- (2025). On Targeted Manipulation and Deception when Optimizing LLMs for User Feedback. ICLR 2025. 第 45 章 第 46 章
- (2024). Stopping Bayesian Optimization with Probabilistic Regret Bounds. NeurIPS 2024. 第 46 章 第 47 章
- (2026). Knowledge Gradient for Preference Learning. arXiv. 预印本 第 45 章 第 47 章
- (2026). Cost-aware Stopping for Bayesian Optimization. International Conference on Machine Learning. 第 46 章 第 47 章
- (2024b). Principled Preferential Bayesian Optimization. International Conference on Machine Learning. 第 45 章
- (2017). The effect of mood on judgments of subjective well-being: Nine tests of the judgment model. Journal of Personality and Social Psychology. 第 45 章
- (2025). Loss aversion is not robust: A re-meta-analysis. Journal of Economic Psychology. 第 45 章
- (2025a). PABBO: Preferential Amortized Black-Box Optimization. ICLR 2025. 第 45 章
- (2026b). Vibrotactile Preference Learning: Uncertainty-Aware Preference Learning for Personalized Vibration Feedback. UMAP 2026 (per Semantic Scholar). 第 45 章
- (2024). Value construction through sequential sampling explains serial dependencies in decision making. eLife. doi:10.7554/eLife.96997. 第 45 章 第 46 章 第 47 章