Open Problems and the Decisive Experiment

Preferential Bayesian optimization (PBO) finds the design a person likes best by asking them to compare options and fitting a model of their utility to the answers (Chapter 19). The chapters before this one reported what is known about it: the methods, the theory, the studies with people, and what other disciplines say a preference is. The two previous chapters drew two conclusions from that record. A single comparison mixes a stable preference, structured noise, change caused by the questions, and answers with no preference behind them (Chapter 45), and a system built on comparisons should be instrumented and judged against strong simple baselines (Chapter 46).

This chapter lists what is not known. It collects eighteen open problems, and for each one names an experiment and the result that would decide it. The first group concerns preference itself, the second theory and dimension, and the third the choice of methods in practice. One experiment, randomizing the order of queries and retesting a week later, would change our understanding more than any other, and the chapter describes it concretely enough to run. The chapter, and the book, end with a conclusion.

The experiments are proposals. Where we add detail to them (sample sizes, numbers of retest pairs, how an arm is balanced), the detail is our own inference, offered as a starting point for a design rather than as a finding.

47.1 Problems about preference itself #

The first problems decide how a comparison should be understood: as a noisy readout of a fixed utility, or as something the session partly makes. Table 47.1 lists them.

Choice history. Choosing an option raises its value for the chooser (Enisman et al., 2021; Zylberberg et al., 2024), and in a PBO session the current best is the option chosen most often. The test is a model comparison on held-out answers: if a likelihood with a term for recent choices predicts better than one without, and options the person rejected become more likely to win again as time passes, the effect is present in design comparisons, where it has never been measured (Section 45.4).

Noise, drift, and incompleteness. Repeated pairs at different lags separate noise, which does not depend on the lag, from drift, which does. One caution from Figure 46.1: induced change that fades also makes agreement fall with the lag, so a falling curve alone does not identify drift (inference); comparing query orders does (Section 47.4). Placebo pairs, the same design shown twice, estimate how often people express a preference where none can exist (O'Mahony and Wichchukit, 2017). A "can't compare" answer that concentrates on complex pairs would show that the fourth component of Section 45.4 is real in design tasks; in experiments with lotteries and money, 40% to 50% of participants used such an answer when allowed (Nielsen and Rigotti, 2026).

Incomplete answers as a default. Forced choice turns indifference, indecisiveness, and experimentation into the same coin flip (Ok and Tserenjigmid, 2022). If allowing the extra answers reduces transitivity violations (choosing $a$ over $b$ and $b$ over $c$ but $c$ over $a$) and makes the final design hold up better at a delayed retest, the cost of the extra buttons is repaid (inference).

The hierarchical view. One position in the literature holds that preferences over ultimate goals are stable while preferences over intermediate outcomes are elicited (Section 45.3.2). The evidence for stability comes from self-reported values (Vecchione et al., 2020) and the evidence for construction from choices, so the two levels have never been tested on the same task. Asking the same people, across sessions, to judge both an abstract goal and concrete configurations would test the view where PBO operates.

Population priors by domain. Shared taste is high for faces and landscapes and low for architecture and artworks (Vessel et al., 2018). If a strong population prior helps in the first domains and a weak or group-specific prior in the second, the strength of the prior is a parameter to set per domain (Section 46.3).

47.2 Problems about theory and dimension #

The next three problems decide what the theory guarantees and where the dimension limit of PBO really comes from (Table 47.2). Two quantities from regret theory appear in them (Section 45.1.2): $\gamma_T$, the maximum information gain of the kernel after $T$ queries, and $\kappa$, an upper bound on the inverse slope of the link function. A third is $B$, a bound on the norm of the utility function in the kernel's own function space, which limits how rough the utility may be.

Lower bounds. All regret results for kernelized preference feedback are upper bounds. The batched algorithm MR-LPF reaches $\tilde O(\sqrt{\gamma_T T})$, matching the scalar rate, but needs a warm-up phase whose constant does not depend on $T$ (Kayal et al., 2025), and the question is whether that constant hides a dependence on $\kappa$ or on $\exp(B)$, which could be very large. No lower bound exists under a logistic or probit link (Section 29.7).

The pipeline in practice. Theory analyzes elimination, optimism, and Thompson sampling on frequentist kernel estimators; practice runs a Laplace posterior with EUBO (Section 45.1.2). A regret analysis of that pipeline, or a counterexample showing that it can fail to converge, would close the gap between what is proved and what is used.

The dimension ceiling. A common claim is that PBO fails beyond 10 to 20 dimensions. The evidence points instead to a property of the default configuration: PairwiseGP's lengthscale prior does not scale with dimension, so the default kernel treats points in high dimension as nearly unrelated (Section 46.3 gives the arithmetic; see also Section 27.5 and Section 30.3.4). For scalar feedback, dimension-scaled priors repaired this (Hvarfner et al., 2024), but with pairwise feedback and human budgets the repair is untested. The proposed experiment separates two explanations: if the dimension-scaled prior closes most of the gap to a reduced representation, the ceiling was the defaults; if only the reduced representation works, the limit is how little each answer carries. Recording the gradients of the marginal likelihood shows whether the lengthscales are being learned at all. Simulation comes first, so that the experiment with people, in a 20- to 50-dimensional latent space of a generative model, is not confounded by the default prior.

Figure 30.2 runs the scalar version of this experiment on a toy problem: a six-dimensional function hidden among up to 50 inputs, with fixed-scale and dimension-scaled priors and two ways of searching the acquisition function. Its lengthscale panel shows whether the lengthscales are learned, and in those runs the answer is a matter of degree: under the fixed-scale prior the inputs that matter get lengthscales only about three times shorter than the rest, while the dimension-scaled prior switches most of the irrelevant inputs off. The figure's regret curves show why the result must be read together with the acquisition search.

47.3 Problems about practice #

The remaining problems decide which methods to use (Table 47.3). Most of them could be settled by an experiment with people that compares a handful of variants on the same task.

Manual against algorithmic tuning. In exoskeleton tuning, people adjusting four hip-timing parameters with a thumbstick reached a 16.6% reduction in metabolic cost in about 11 minutes (Schäfer et al., 2026), of the same order as algorithmic tuning reported in other studies with different devices and controls. No study has compared the two on the same participants against a preregistered endpoint, an outcome fixed before data collection (Section 33.3, Section 24.5). Because people adapt to a device over about 109 minutes of assisted walking (Poggensee and Collins, 2021), the order of the two conditions has to be balanced across participants.

Acquisition functions with people. The candidates in the table are the decision-theoretic qEUBO (Section 19.4), two older heuristics, dueling Thompson sampling, which takes the best option under one random sample of the posterior and pairs it with the option whose duel against it is most uncertain, and the maximally uncertain challenge, which pits the option with the largest posterior mean against the option whose comparison with it is most uncertain in the model's knowledge (Section 28.1), and random queries as a control. We found no study that randomized people to different acquisition functions with the same interface and budget (Section 28.9). If the differences between acquisition functions turn out smaller than the differences between people and the noise of a retest, the work on acquisition functions matters less, in practice, than the observation model (inference).

Response times, simulated users, and data. Response times improved models on real human data (Shvartsman et al., 2024), but they have not been tested in closed-loop optimization. Language-model simulations of users reproduce population-level rankings almost perfectly but agree with individuals at a Kendall $\tau$ (a rank correlation from $-1$ to $1$) of only 0.11 to 0.22 (Kirk et al., 2026), so whether they can stand in for people in a PBO loop is open (Section 35.2.4). A public data set of individual comparisons, with the timing and order information listed in the table, is the prerequisite for most offline comparisons in this list (Section 31.5).

The default pipeline. EUBO's tendency to collapse toward the estimated maximum, and the rank-deficient Hessian that its isolated pairs cause, are documented in 2026 preprints (Wu and Gardner, 2026; Shao et al., 2026), and the error of the Laplace approximation was measured at very low noise (Takeno et al., 2023). The alternatives in the table are expectation propagation and exact sampling of the skew Gaussian process posterior (Chapter 17), and the hallucination believer, which plugs a single posterior sample in as if it were data (Section 19.3). Whether these failures cost anything with real people, at real noise levels, is the question (Section 27.6, Section 27.4).

Language models and stopping. In online direct preference optimization, a 2026 workshop paper found active selection only negligibly better than random (Oh et al., 2026b); whether an explicit posterior over rewards changes that is open (Section 35.3.5). Stopping rules from scalar Bayesian optimization (Wilson, 2024; Xie et al., 2026) have not been carried over to a pairwise likelihood, and Section 46.6 explains why a person's cost per query is not constant.

47.4 The decisive experiment #

Of all the problems above, one would change our understanding most: whether the order of queries changes the final preference. It decides between two readings of every PBO session, noisy discovery, in which repeated comparisons recover a preference that was there before, and preference scaffolding or construction, in which the questions help make the preference they measure (Section 45.3). It also supplies the controls that judging the legitimacy of a system-induced change requires (Section 45.5), and its retests separate noise from drift. As of September 2026, no study has run it.

47.4.1 The design #

In outline, the design is simple. Everyone gets the same candidate space and the same budget of comparisons. Participants are randomly assigned to a sequence of queries chosen by the acquisition function, a random sequence, or a sequence in balanced order. Before the session, each person's attitude toward having their taste changed is recorded. At the end of the session and again a week later, without the system's framing, each person chooses between the final design and options they rejected early. Algorithm 47.1 fills in the details we would choose; every number in it is our suggestion (inference).

47.4.2 What decides it #

The decision criterion has two halves. If the final designs of the acquisition arm lie closer to their own early posterior mean, and their delayed retest agrees less often than in the random arm, then change induced by the queries is a substantive component of the answers. If the distribution of final designs and the delayed agreement do not differ between the arms, noisy discovery is enough to explain the data.

Our analysis of the model in Figure 45.2 suggests a refinement (inference). In that model, the second half of the criterion, comparing the delayed agreement of the two arms directly, can fail even when induced change is present and fades. At the figure's default settings, the acquisition arm's final designs are still preferred more often a week later than the random arm's (78% against 73%), because the acquisition arm reaches a higher agreement at the end of the session (88% against 76%) and part of that lead survives. The signature that holds up is the drop in agreement from the end of the session to the delayed retest, compared across arms: 10 percentage points in the acquisition arm against 3 in the random arm, while with induced change set to zero both drops vanish. A reader can reproduce this by setting the figure's sliders and reading its retest panel.

The model also shows the limit of the drop. When the induced change lasts (the Lasting share slider at 1), neither arm drops, and only the first half of the criterion, final designs close to the early favorite and a different distribution of final designs across arms, still detects it (Exercise 45.1). We would therefore preregister three contrasts (inference):

• Primary: the mean drop from the end-of-session retest to the delayed retest, acquisition arm against random arm (a difference in differences).
• Secondary: the distance between the final design and the early favorite, and the distribution of final designs, compared across arms; and the direct comparison of delayed agreement between arms.
• Diagnostic: agreement on repeated pairs by lag and the false-preference rate on placebo pairs, pooled over arms, which separate noise and drift as in Figure 46.1.

47.4.3 How many people #

The experiment is cheap in equipment and expensive in participants, because the primary outcome is a small difference between two differences of proportions. Figure 47.1 gives a planning approximation: a person's measured drop varies because of the binomial noise of a few retest choices at each time and because people differ, and a standard two-sample test then needs the number of participants per arm that the figure marks.

At the defaults, a difference of 7 percentage points in the drop, 8 retest pairs per person, agreement around 80%, and a spread of 0.1 between people, about 161 participants per arm reach 80% power, 483 in three arms. Doubling the retest pairs to 16 cuts that to 97 per arm; a difference of 10 points with 16 pairs needs 48. The number of retest pairs is the cheapest lever, since each pair costs seconds while each participant costs a session and a return visit a week later (inference). Numbers of this size are within reach of an online study with a visual task, and out of reach of a laboratory study with a device, which is one reason to run the experiment first where it is cheap.

47.4.4 Practical notes #

Keep the task low-stakes. The experiment deliberately includes an arm that may shape preferences more than others. A design task without consequences beyond the session, informed consent that explains that the system may influence choices, and a debriefing after the delayed retest keep it within ordinary research ethics (inference).

Extensions. The same design answers other problems in this chapter with one more arm each: an arm with manual self-tuning answers whether the algorithm is needed at all, arms with different acquisition functions answer how much they differ with people, and an arm with the extra answers "about the same" and "can't compare" answers whether incomplete answers should be the default.

Why it has not been run. It needs no new method, only a commitment to measuring the system's effect on the people it optimizes for. The two retests, the random arm, and the attitude question are the same controls that Algorithm 46.2 recommends for every session, so a study that already follows those recommendations has most of the decisive experiment built in.

47.5 Settled, contested, missing #

47.6 Outlook: where the problem is growing #

The open problems above are about what is not yet known. This section is about where the research is worth doing, and it is the book's own view rather than a summary of evidence: every paragraph in it could carry the mark (inference), so it is marked once here.

It helps to separate two questions. Bayesian optimization and its preferential variant are answers to one problem: finding what is good when each evaluation is expensive, and, in the preferential case, when the only instrument is a person's judgment. The first question is whether the algorithm family has much further to go. The second is whether the problem it answers is becoming more or less important.

The algorithm family. Its future is modest. The core is mature: decision-theoretic acquisition functions with a multi-option form (Section 19.4), a default pipeline in widely used software (Section 31.1.1), and regret theory that has caught up with scalar feedback at the level of upper bounds (Section 29.4). The evidence with people is small: the median user study in the evidence map of Figure 32.3 has 12 participants, and simpler methods often do as well (Section 45.1.5). Another acquisition function is unlikely to change that picture. Better measurement could.

The problem. It is growing. Systems increasingly generate, adapt, and personalize, and each of those needs to learn what one particular person wants from a handful of judgments. The directions below are where that problem meets an opening, ordered roughly from the measurement questions at the center of this book outward. Each names what exists, what is missing, and the combination of skills it rewards.

47.6.1 Measurement science for preferences #

The niche closest to this book's argument. The observation model, probit or logistic noise of constant size, has never been compared with an alternative on human data; noise that depends on how hard a comparison is, response times, "can't tell" answers, and drift over a session are all open (Section 47.1, Section 27.2). The decisive experiment of Section 47.4, randomized query order with a retest a week later, would separate a preference that was found from one that was shaped, and nobody has run it. And no public data set of individual pairwise judgments exists (Section 31.5), so almost every offline comparison of methods runs on simulated people. The work is cheap, mostly participant time with existing interfaces, and has high leverage, because every method in this book sits on top of the observation model. It rewards someone who can design a careful experiment with people and also write the model: human-computer interaction or psychophysics together with probabilistic machine learning.

47.6.2 Personalizing generative models #

A foundation model is a strong prior over what people in general like and a weak one about what a specific person likes. A handful of well-chosen comparisons on top of such a prior is exactly the preferential setup, with a far better prior than a Gaussian process on raw parameters. The first systems exist: GimmBO searches the merging weights of 20 to 30 diffusion adapters (Liu et al., 2026b), MultiBO shows several images per round (Rajagopalan et al., 2026), and APPO hides the text prompt and asks only for binary preferences between images while a language model rewrites the prompt (Li et al., 2026f) (Section 32.1.6). Missing are a surrogate that uses the generative model itself rather than a Gaussian process over its latent space (Section 30.5), tests with people in the 20 to 50 dimensions such spaces have (Section 30.3.4), and an evaluation that separates what the person specifically likes from what the population likes (Section 32.5). It rewards generative modeling together with Bayesian optimization and user studies.

47.6.3 Choosing which comparisons to collect for language models #

Reward models and direct preference optimization learn from pairwise comparisons at scale, and they face the questions this book studies: which pair to ask about next, and when to stop (Section 35.3.3, Section 35.3.4). The setting differs in ways that matter: millions of comparisons instead of dozens, many raters instead of one person, and a population preference as the target (Section 35.6.2). Whether choosing pairs actively beats random pairs at that scale is itself unsettled (Section 35.3.5), and disagreement between raters may be signal rather than noise, the problem social choice studies (Section 40.7). This is where most of the funding and attention are. It rewards large-scale machine-learning engineering together with the theory of active learning, and it gains from the measurement work above, since raters are people too.

47.6.4 Devices on the body #

Exoskeletons, prostheses, hearing aids, and neurostimulation share the conditions under which preferential optimization makes most sense: comparisons are natural, objective measures are slow or expensive, and people differ more than any population default allows. The strongest real-world evidence in this book comes from here. Bayesian optimization tuned two timing parameters of a soft exosuit in 21.4 ± 1.0 minutes (Ding et al., 2018); wearers tuning four parameters themselves with a thumbstick remote control took 10.9 minutes (Schäfer et al., 2026); preset selection for over-the-counter hearing aids has been posed as a dueling bandit (Vyas et al., 2022); a variant of safe optimization that takes preference feedback was applied to spinal cord stimulation (Sui et al., 2018b); and a thesis tuned the encoders of retinal prostheses with people in the loop (Fauvel, 2021) (Chapter 33). Missing are a head-to-head comparison of manual and algorithmic tuning on the same participants (Section 33.3), exploration that stays safe for the body, and studies that follow a person for weeks while their body and preference adapt. It rewards biomechanics or clinical research together with safe and non-stationary Bayesian optimization.

47.6.5 Simulated people and model judges #

Because studies with people are expensive, nearly every comparison of methods uses simulated people (Section 31.7), and language models now act as judges and stand-in users. One measurement shows the risk: simulated users reproduced population-level rankings almost perfectly but agreed with individuals at a Kendall $\tau$ of only 0.11 to 0.22 (Kirk et al., 2026) (Section 35.2.4). A simulator validated against real comparison data, with order effects and drift, would make offline benchmarks informative; an unvalidated one makes them circular. It rewards user modeling together with evaluation methodology, and it depends on the public data set the first direction would produce.

47.6.6 Science and engineering with experts in the loop #

Self-driving laboratories now report how much faster an optimizer reaches a target than a reference strategy (Adesiji et al., 2026). Expert judgment helped where the goal had no sensor, as in printing objects with subjective qualities (Deneault et al., 2025), and lost to the optimizer where yield could be measured (Shields et al., 2021) (Section 34.2). Missing are models that combine measured objectives with an expert's pairwise preferences, queries that treat the expert's time as a cost, and studies with more than one expert in the loop. It rewards domain science together with multi-objective and cost-aware Bayesian optimization.

47.6.7 Many people, one setting #

A thermostat in a shared office, a recommender, or a design meant for a population is tuned by the preferences of many people at once (Section 34.1.1). Aggregation, fairness, and strategic answers are questions social choice has studied for decades (Section 40.7), and a prior learned from earlier people in the same domain could cut the number of questions for each new one (Section 47.1). It rewards mechanism design or social choice together with preference learning.

47.6.8 Theory for the pipeline people deploy #

All regret results for kernelized preference feedback are upper bounds; there is no lower bound under a logistic or probit link, and the algorithms that theory analyzes, elimination and optimism, are not the decision-theoretic acquisition with a Laplace approximation that software runs (Section 47.2). Preferences that drift during a session have almost no theory. It rewards learning theory, with enough contact with practice to analyze what is actually deployed.

47.6.9 Signals beyond the click #

A comparison records more than which option won: how long the person took, and how much they hesitated. A drift-diffusion model links choice and time (Section 39.3), and response times improved models on real human data (Shvartsman et al., 2024), but they have not been used to choose the next query. Eye movements and physiological signals are further candidates. The open question is which of these signals are robust across people and interfaces, and whether they improve the choice of queries and not only the fit. It rewards cognitive modeling together with Bayesian optimization.

47.6.10 The intervention as a design problem #

If a session can change a preference, a system should be able to detect when it does, and a study should state the conditions under which that change is acceptable (Section 45.5.2, Section 41.2). Measuring preference change within a session, stopping rules that account for it, and interfaces that show a person how their answers moved are technical problems, not only ethical ones. It rewards human-computer interaction together with statistics and moral philosophy.

What the directions share is the book's argument. The bottleneck has moved from algorithms to measurement, and most of the valuable work sits where a careful experiment with people meets a well-specified model. People who can do both are scarce, which is the opportunity.

47.7 Conclusion #

Between 2017 and 2026 the largest change in PBO was not a new methodological paradigm but a shift in where the attention went. Acquisition functions gained a decision-theoretic foundation, regret theory caught up with scalar feedback at the level of upper bounds, and software fixed a default pipeline. At the same time, every layer of that default was found to rest on an untested assumption: the lengthscale prior does not scale with dimension, the Laplace approximation is ill-conditioned when the comparison graph is disconnected, and the probit link with constant noise has never been compared with an alternative on human data. The evidence gives an order of work: first get the observation model and the representation shown to the person right, then choose the acquisition function.

About preference itself, the evidence since 2017 supports neither pure discovery nor pure construction. A single pairwise judgment contains a stable component, structured evaluation noise, change caused by the query, and incomplete or deliberately randomized answers, and which of them dominates depends on the domain, the similarity of the options, and the length of the session. The practical consequence is that the output of a PBO session is both an estimate and an intervention. A study that does not randomize the order of queries or retest after a delay cannot tell whether what converged was the preference or the system's effect on the person, and because the controls needed to tell them apart are the controls needed to judge whether that effect was legitimate, the methodological question and the normative one can be answered in the same experiment (inference).

Three positions in the literature come out of this with qualified support. The hierarchical view of preference is supported where it says that stability lies at the broad, abstract, self-reported level, and not supported where it predicts that iterated querying converges on an underlying preference (Section 45.3.2). Preference scaffolding is partly supported as a description of what PBO does, but as a normative position it needs conditions of legitimacy, and endorsement after the fact cannot supply them on its own (Section 45.3.3). And the common claim that PBO stops working beyond 10 to 20 dimensions describes the default configuration, not a limit of preference feedback (Section 30.3.4).

Four pieces of work are the most likely to change these conclusions: randomized query order with a delayed retest; manual self-tuning against algorithmic tuning on the same participants; dimension-scaled priors tested with human comparisons on 20- to 50-dimensional representations; and a public data set of individual pairwise judgments. The first two can be done with existing devices and interfaces, and their main cost is participants' time; the third needs simulation first, to rule out the default prior as a confound; the fourth is the precondition for most of the offline comparisons in this chapter. On the theoretical side, the most important gap is a lower bound under a logistic or probit link.

Until these results exist, the evidence supports a modest practice: treat PBO as one of several baselines, instrument every session so that it measures the person as well as the design, and, in every study, measure what the system does to the people it learns from.

47.8 Exercises #

Further reading #

• Carroll et al. (2024) and Dean and Morgenstern (2022) explain why the order of queries is not a detail when the system can move the preference it measures.
• Keswani et al. (2026) is the closest existing measurement of instability in repeated pairwise answers across sessions, and a model for the retests of the decisive experiment.
• Schäfer et al. (2026) is the strongest current evidence that a simple baseline can match algorithmic tuning, and the starting point for the manual-tuning experiment.
• Kayal et al. (2025) and Lazzaro et al. (2026) state the best current upper bounds and their conditions, which a lower bound would have to match.