Building and Evaluating a Preferential Optimization System

Preferential Bayesian optimization (PBO) searches for the design a person likes best by asking them to compare options and fitting a model of their utility to the answers (Chapter 19). The previous chapter concluded that a single answer mixes a stable preference with structured noise, change caused by the questions, and answers with no preference behind them, so that a session is both an estimate of a preference and an intervention on it, and that the largest errors come from what the model assumes about the person (Chapter 45).

This chapter turns that analysis into advice for someone who has to build or evaluate a preferential optimization system: three questions that decide whether PBO is the right tool, the parts of the system one at a time, and the situations in which a simpler method should be the default. Every recommendation comes from evidence reported earlier in the book or from our inference from it, and none has been tested as a whole in PBO. Each table also says whether a recommendation depends on the Gaussian process and PBO framework. Those that do not apply to any preference model, including the simple baselines of Section 46.9; those that do are repairs inside the framework, and say nothing about whether the framework was the right choice.

46.1 Three questions before choosing PBO #

Faced with a new problem of personalization or design with a person in the loop, we suggest answering three questions before deciding to use PBO (inference).

When all three questions point toward PBO, qEUBO with BoTorch's PairwiseGP is a reasonable default (Section 19.4, Section 18.6), a choice that comes from the software ecosystem rather than from a comparison with simpler methods. The session should be instrumented as in Algorithm 46.2: the errors caused by the observation model and by the person's behavior exceed the differences between acquisition functions (Section 45.1), and these additions both estimate those errors and supply the controls needed to tell discovery of a preference from shaping it (Section 45.5).

Steps 3 to 5 are justified in Section 46.2, step 2 in Section 46.4, step 4 again in Section 46.5, steps 7 and 8 in Section 46.6, and steps 1, 6, 7, and 8 in Section 46.8.

A note on dimension. A comparison carries at most one bit (Section 16.6), so 100 comparisons over 20 parameters carry at most five bits per parameter, and usually far fewer, because answers are noisy and many queries are redundant (inference). Where PBO has worked in more than a handful of dimensions, the dimension the person faced had been reduced by design, to a 10-dimensional body-shape space of a generative human model or the merging weights of 20 to 30 diffusion adapters (Koyama et al., 2020; Liu et al., 2026b), or to a 5-dimensional feasibility-aware latent space (Table 46.8).

46.2 Observation model #

The observation model carries the largest error (Section 45.1.3). Table 46.1 lists five recommendations for it; none depends on the Gaussian process framework.

Two extra answers. A forced choice turns indifference, indecisiveness, and experimentation (Ok and Tserenjigmid, 2022) into a coin flip that the model reads as a weak preference, and when allowed, many people say they cannot compare (Table 45.2). An "about the same" answer has a standard model, a just-noticeable-difference threshold, and a 2025 preprint reports that a method which models it clearly outperforms standard preferential baselines once 10% to 20% of comparisons are indifferent, on synthetic benchmarks (Erarslan et al., 2025) (Section 20.4).

Placebo pairs. Sensory science found that consumers report preferences between identical samples, and made pairs of identical samples a standard control condition (O'Mahony and Wichchukit, 2017). In PBO, the rate at which a person prefers one of two identical renders, identical down to the pixel, estimates how many answers carry no preference at all and can calibrate a lapse term (inference).

Structured noise. In estimates of risk aversion from a representative sample, the standard expected-utility random utility model gave distorted estimates when everyone shared one noise scale and not when each person had their own, in a 2024 preprint (Keffert and Schweizer, 2024). PairwiseGP fixes the noise at 1 and lets the kernel's amplitude (BoTorch's output scale) stand in for it (Section 27.5).

Response times are free information about how far apart the options felt. A joint likelihood of choices and drift-diffusion response times exists for Gaussian process models (Shvartsman et al., 2024), and a loss built on the EZ-diffusion model reduces the error of preference learning with linear rewards from exponential to polynomial growth in the magnitude of the reward (Sawarni et al., 2025). But decisions between two high-value options are mostly faster and more accurate, not slower (Shevlin et al., 2022), so late in a session a quick answer does not mean a large difference unless the pair's overall value is in the model (inference).

Choice history. The current best, the most-compared option, gains most from choice-induced revaluation (Section 45.4), and the loss-aversion coefficient has a meta-analytic mean of 1.955 but is about 1.07 when gains and losses are symmetric and unsorted (Table 45.1), so both belong in the likelihood as estimated terms, not fixed values.

46.3 Surrogate #

The surrogate is the prior over the utility function. Table 46.2 lists three recommendations, of which only the last is purely a matter of the Gaussian process framework.

Population priors. Shared taste varies by domain (Table 45.3), and in interactive design a prior pretrained on models of earlier users was significantly better only at the second and third iterations; from the sixth on, all methods were comparable (Liao et al., 2026). A population prior buys a better start, not a better result, and its weight should fall as the person's own answers accumulate (inference; Section 20.5).

High dimension. The default PairwiseGP lengthscale prior is far shorter than the typical distance between designs in high dimension (about 0.52 against 2.9 at $d = 50$), so the default model treats every comparison as nearly uninformative about every other point (Section 27.5, Section 30.3.2). For scalar feedback, priors that scale with dimension repaired this (Hvarfner et al., 2024), and a spherical input mapping with Bayesian linear regression reached the state of the art (Doumont et al., 2026). Neither has been tested with pairwise feedback from people, so reduce the dimension the person sees first, which helps under any model. Figure 30.2 adds a second reason: on a toy problem with scalar feedback, how the acquisition function is searched decided more of the outcome within 80 evaluations than the lengthscale prior did (Section 30.1.2).

Inference. At very low noise, the Laplace approximation misplaces the posterior and expectation propagation distorts duel probabilities near 0 or 1 (Takeno et al., 2023); exact sampling of the skew Gaussian process posterior avoids both (Benavoli et al., 2021c). How large the errors are at human noise levels is unmeasured (Section 27.4), so the switch matters most for very consistent answerers (inference).

46.4 Acquisition and query design #

The observation model settles what a query records and what happens when the person cannot answer (Chapter 20); Table 46.3 collects what the evidence says about choosing and presenting the queries.

Random and balanced queries. A simulation study of moral preference elicitation found that active learning rests on three premises (preferences that are stable and unaffected by the order of queries, a correct model class, and limited noise), and that when they fail, actively chosen queries do no better than random ones, or worse (Keswani et al., 2024). A fixed share of queries that the acquisition function does not choose costs some efficiency and buys a control group inside every session, free of the system's steering, against which the model's predictions can be checked (Section 46.8). Position matters too: steering how long people looked at an option raised its share of choices to 0.541 (95% confidence interval 0.523 to 0.560), a causal but small effect (Bhatnagar and Orquin, 2022), so randomizing positions and rotating which option the system proposes keeps such effects from adding up in one direction (inference).

The comparison graph. If evaluated designs and answered pairs fall apart into disconnected pieces, the answers say nothing about how the pieces compare, and the Laplace approximation becomes ill-conditioned (Section 18.5, Figure 27.1). The graph's algebraic connectivity, the second-smallest eigenvalue of its Laplacian matrix, is zero exactly then. Since EUBO's pairs tend to start new components (Shao et al., 2026) (Section 19.6), the check is cheap insurance: when the next pair would start a new component, swap one of its options for an already-compared design (inference).

Rankings and simple pairs. When users differ in ways the model does not know, binary comparisons cannot identify the user types, and rankings of at least three options can (Chidambaram et al., 2026). Guidance for stated-preference studies notes that respondents' consistency falls as the statistical efficiency and the complexity of the choice tasks rise (Johnston et al., 2017); in PBO, the complex tasks are pairs that differ in many attributes at once (inference).

Language for goals, comparisons for appearance. In the two studies of optimizer-led design ($N = 18$ and $N = 12$; Section 45.3.3), steering through natural language gave lower workload than explicit constraints, and 90.9% of 187 requests in natural language described a desired outcome rather than a parameter (Niwa et al., 2025). For fine judgments of appearance, comparisons work better than words: in a 2026 preprint, personalization from just 8 pairwise judgments beat every baseline for 12 new users, including their own written preferences, with an aggregate win rate of 60.35% (Peng et al., 2026). The photo case study lets you try comparisons and line search on an image (Chapter 25).

46.5 Non-stationarity #

A preference that changes during a session breaks the model's central assumption. Two kinds of change need to be separated: drift, where the utility itself moves (through learning, adaptation, fatigue, or change caused by the queries), and a change in the choice rule, where the utility stays put but the way the person turns it into an answer changes, for instance from weighing all attributes to deciding on the single one that has come to matter (Section 37.5). Drift calls for a time-varying utility, a changed choice rule for a time-varying noise or link (inference).

Repeated pairs. When the same moral pairwise questions were asked in three to five sessions, 6% to 20% of answers flipped, and for some participants the fitted decision model itself changed (Keswani et al., 2026). One repeated pair cannot tell noise from change; pairs repeated at several lags can, because noise does not depend on the lag and drift does (Figure 46.1).

Things to try:

• At the defaults the curve falls from 72% at a lag of 2 answers to 61% at a lag of 40, because the utility differences drift; with 30 people and 4 repeated pairs per lag, the measured points show the decline. Set drift to 0, and the curve is flat at 73%: noise makes people disagree with themselves equally at every lag.
• Raise Incomplete (the share of incomplete answers) to 0.4 with drift at 0. The curve stays flat and drops to 58%: answers with no preference behind them look like extra noise unless "can't compare" is offered.
• Return Incomplete to 0, keep drift at 0, and set Induced (induced change) to 1. The curve falls again, from 92% to 74%, because at short lags the person repeats a choice that the first choice made more attractive. A falling curve does not distinguish drift from induced change; only comparing query orders does (Section 45.4.1).
• Reset the figure, then set People to 5 and Repeats (repeated pairs per lag and person) to 1. The intervals span about ±33 percentage points. A diagnostic built on repeated pairs needs on the order of a hundred repeated pairs per lag, pooled over people (inference).

Familiarization. People adapt to a device over longer than a typical tuning session (in exoskeleton walking, training contributed about half of a 39% reduction in metabolic cost (Poggensee and Collins, 2021)), and comparisons made before they have adapted measure a different person (Section 24.4).

Models of change. Where drift is real, a time kernel with a fast and a slow scale, or a state-space model in which each parameter drifts, can represent it (inference); over months, a stable component plus deviations that follow events and then decay fits the evidence better than a random walk (Chapter 42). Drifting utilities are not yet modeled in PBO (Section 27.2), and the theory that exists is for finite arms (Section 29.10).

46.6 Stopping #

When should a session end? The tempting answer is when the posterior has concentrated: the model is confident about the best design, and EUBO, asked to choose a pair, would show the incumbent against itself (Exercise 19.2). That is not enough, because the posterior concentrates whenever the answers are consistent, and choice-induced revaluation makes answers consistent without any stable preference behind them (Figure 45.2, Exercise 45.2).

Rules from scalar Bayesian optimization. Scalar rules stop when a design is $\varepsilon$-optimal with probability at least $1 - \delta$ (Wilson, 2024), or when no point's expected improvement exceeds the cost of evaluating it (Xie et al., 2026); both assume a correctly specified Gaussian process prior (Section 30.7, Section 14.7). Preference-based reward learning has an optimal stopping rule for parametric reward models (Bıyık et al., 2019). None has been carried over to PBO's pairwise Gaussian process likelihood, and a person's cost per query is not constant: fatigue raises it, and learning the task lowers it. Until a rule is carried over, Algorithm 46.3 is the practical one.

Response times as a stopping signal. Answers usually get faster late in a session. That can mean the preference has settled, but people also stop deliberating when the expected gain in confidence is no longer worth the effort (Bénon et al., 2024), so faster answers need the same caution as posterior concentration (inference).

People stop on their own. In a three-month field deployment, 415 of 549 evaluation sequences stopped at the first iteration, and only 16 of the remaining 134 reached a satisfying result (Ou et al., 2022). A person's "good enough" and a model's converged posterior are different signals, and a stopping rule designed for the model must also survive the person's own decision to quit.

46.7 Evaluation #

How should a new PBO method or system be evaluated? Current practice is weak, with scalar test functions, incompatible noise models and definitions of regret, and no shared benchmark (Section 45.1.4, Section 31.4). Three recommendations, all independent of the framework, address it, and Table 46.6 collects them with the others into a reporting checklist.

Strong baselines. Most positive results for interactive Bayesian optimization in design come from comparisons with sliders, parameter panels, or random queries; against skilled manual work or a similar optimizer, final quality usually does not differ, and these systems are best summarized as reaching the same result at lower cost (Section 32.8). A method that has not been compared with manual tuning, random search, and a linear model cannot claim to be needed.

Acceleration factors. Self-driving laboratories report the acceleration factor, how many experiments a strategy needs to reach a target relative to a reference strategy: across 42 studies and 63 benchmarks, the median reported factor was 6, with a range of 1.3 to 100 (Adesiji et al., 2026). PBO studies with people can report the same number relative to manual or random search, which makes results comparable across tasks (inference).

Simulated users. Test the assumptions of a simulated user on at least a few real people. Simulated agents agreed with the people they stood in for on only about 50% of choices (Schoinas et al., 2025) (Figure 45.1), and language-model simulations of users recover population-level rankings almost perfectly while agreeing with individuals at a Kendall $\tau$ of only 0.11 to 0.22, against 0.57 for the agreement between people's own ratings and their own rankings, in a 2026 preprint (Kirk et al., 2026). A result obtained with a simulated user is a result about that simulation until it has been checked on people (Section 31.7).

46.8 Ethics and legitimacy #

A system that learns a preference while shaping it needs rules about how much shaping is acceptable. Section 45.5.2 set out the candidate conditions; Table 46.7 turns them into practice.

Measure the shift, not only the regret. Because low regret can be achieved trivially by moving the user's preferences (Section 45.2), regret alone cannot certify a system that can influence its user. Measuring how far preferences moved during the session, against a control group whose queries were random or balanced, makes the influence visible, and the shift can then be bounded (Carroll et al., 2022).

Audit the incentives. Since learners optimized on user feedback learn to target the users easiest to influence (Section 45.5.1), a team can ask whether its acquisition function and stopping rule reward answers becoming more predictable; labeling proposals, showing the history accurately, and allowing a reset keep the influence in view. A system that chooses which candidates to show and also benefits from particular outcomes is in the position of a sender in the economics of persuasion, whose preferences differ from the user's (Section 40.6).

Before and after. Recording the person's attitude before the session and retesting after a delay without the system gives both the earlier and the changed self a say (inference; Section 45.5.2). For decisions with moral weight, the evidence on autonomy argues for a system that lays out the options and trade-offs and leaves the decision to the person (Section 41.2).

For systems deployed to consumers, these steps also serve compliance. Article 25 of the EU Digital Services Act prohibits providers of online platforms from designing interfaces that materially distort or impair users' ability to make free and informed decisions (European Union, 2022b). An acquisition function that systematically reinforces the current favorite might fall within that wording, and logging a balanced query order and ending with an explicit endorsement check serve scientific validity and that requirement at once (inference; this is our reading of the text, not legal advice; Section 42.2.3).

46.9 When a simpler method should win #

Random search learns nothing from its own results, and in several settings nothing beats it by much (Chapter 1). Table 46.8 lists the situations in which a simpler method should be the default or a required baseline, and whether the advice comes from the problem itself, from direct evidence, or from inside the Gaussian process and PBO framework.

The case studies work several of these situations end to end: the exoskeleton case re-enacts the first row (Section 24.5), the classifier case shows random search holding its own against Bayesian optimization (Section 22.3), and the photo case (Chapter 25) is an instance of the last row.

The overall verdict, which comes from the evidence and not from the framework, is that in studies with people, an advantage of PBO over strong simple baselines has not been shown directly. New studies should therefore make manual tuning, random or coarse grid search, and a linear utility model required controls; comparing only with sliders or with another PBO variant cannot answer whether PBO is needed at all.

46.10 Settled, contested, missing #

46.11 Exercises #

Further reading #

• Schäfer et al. (2026) is the clearest demonstration that a strong simple baseline can match algorithmic tuning with a person in the loop.
• Keswani et al. (2024) states the premises on which active preference elicitation depends and shows what happens when they fail.
• O'Mahony and Wichchukit (2017) reviews how sensory science learned to handle forced choice, "no preference" answers, and placebo pairs, decades before PBO.
• Dean and Morgenstern (2022) and Carroll et al. (2022) show why regret is not enough when the system can move the preference it is learning.
• Adesiji et al. (2026) sets out the acceleration and enhancement factors that make results comparable across tasks.