Machine Learning from Human Preferences and Active Learning从人类偏好和主动学习中进行机器学习

The following content is generated by LLMs and may contain inaccuracies.

Context

Learning from human preferences has emerged as a critical bottleneck in deploying AI systems that genuinely serve human values. While traditional supervised learning assumes labeled ground truth, preference learning acknowledges that many real-world objectives—from LLM safety to personalized control systems—lack objective labels and must be inferred from comparative judgments. This matters acutely now as RLHF has become the dominant paradigm for LLM alignment, yet the sample efficiency of preference collection remains poor. The tension: preference data is expensive to collect, but passive collection scales poorly with system complexity.

Key Insights

Active learning can dramatically reduce labeling costs. Recent work by Zhao & Jun (2026) provides the first instance-dependent complexity bounds for active preference learning in LLM alignment, demonstrating that query selection tailored to preference structure (rather than generic experimental design criteria like D-optimality) improves sample efficiency. This challenges the common practice of applying classical active learning objectives without adapting them to the comparative nature of preferences.

Preferences are inherently contextual and heterogeneous. Wang et al. (2025) show that personalized building climate control requires contextual preferential Bayesian optimization to account for both individual differences and environmental factors like outdoor temperature. Similarly, Ozaki et al. (2023) address multi-objective scenarios where decision-maker preferences over Pareto-optimal solutions must be learned interactively. Both works highlight that a single utility function rarely captures real-world complexity.

Preference aggregation introduces normative challenges. The Stanford CS329H curriculum explicitly covers “preference heterogeneity and aggregation” and asks “whose preferences?” These aren’t just technical questions—aggregating preferences involves value judgments about whose feedback counts and how disagreements are resolved, linking machine learning design to social choice theory.

Open Questions

• How do we balance exploration-exploitation tradeoffs when the preference model itself is uncertain? Current active learning methods optimize for either objective uncertainty or preference uncertainty, but not both simultaneously in a principled way.

• Can we develop theoretically grounded methods for preference aggregation that go beyond majority voting while remaining computationally tractable? The connection to classical impossibility results in social choice (Arrow’s theorem, etc.) suggests fundamental limits that ML practitioners rarely engage with.