Interactive Design and Human-Computer Interaction

Part IV built preferential Bayesian optimization (PBO) around a simulated person: a fixed utility, constant noise, and unlimited patience, and Section 19.7 warned that real people are none of these things. This chapter collects what human-computer interaction (HCI), computer graphics, and interactive design learned between 2017 and September 2026 by putting real people in the loop: designers tuning photographs, artists simplifying 3D meshes, crowd workers moving sliders, drivers rating dashboards.

The chapter reads "preferential" broadly: besides pairwise duels, it includes sliders, galleries, and rankings, as the authors of one main system do when they use the term PBO "in a broader sense" (Koyama et al., 2020), and it includes optimization driven by ratings or by measured performance, because most of what is known about agency and population priors comes from those studies. Each study is labeled with its feedback type, preference (choices, rankings, comparisons), rating (a score on a scale), or performance (a measurement of what the person did, such as task time), because a result about one type does not automatically transfer to another. $N$ is the number of participants.

Three findings carry the chapter, and each bears on the book's argument that the hard part of PBO is now measurement, including what asking does to the person (Section 45.1). When the optimizer leads, outcomes improve but the person's sense of agency drops, and most of it returns when the person can steer. Feedback in real sessions is unstable in ways the standard noise model does not describe. And against strong comparisons, the usual result is the same outcome at lower cost, not a better outcome, from samples that in pure preference studies are mostly 6 to 60 people.

32.1 Research lines #

Two papers mark the start: Brochu et al. (2007) used gallery-style active preference learning to design materials, and González et al. (2017) gave the formal definition based on duels in 2017. Since then, HCI systems have mostly changed the form of the query and rarely the acquisition function, the rule that picks the next question (Section 12.1), which is usually expected improvement (Section 12.3) or the upper confidence bound (Section 12.4) rewritten to fit the query (inference, from the system descriptions below). After 2024 the emphasis moved toward priors borrowed from other users and toward language and generative models. The subsections follow the research groups; Table 32.1 collects the user studies.

32.1.1 Koyama and collaborators: the query interface as the variable #

Yuki Koyama and colleagues (University of Tokyo, AIST, and OMRON SINIC X) treat the query interface as the main thing to study: from 2017 to 2025 each paper added one channel for feedback, from a single slider to a gallery, several sliders, implicit slider logs, references, constraints, and natural language (inference, from the designs below).

Sequential line search turns each query into a single-slider microtask along a line through the design space (Koyama et al., 2017) (Section 20.2; Chapter 25 runs it on a photograph). Its crowdsourced pipeline cost about $5.25 and 68 minutes per result (15 iterations of 7 microtasks at $0.05 each), and against the automatic enhancement of Photoshop and Lightroom the optimized versions of three photos received 32, 26, and 29 crowd votes, and each alternative 0 to 3 (Koyama and Igarashi, 2018). The same book chapter argues that absolute ratings generally work poorly because a rater must know the design space, whereas a pairwise comparison "is easy to answer even for non-experts".

Sequential Gallery replaces the slider with a zoomable grid (5 by 5 for photos) from which the person picks the best item (Koyama et al., 2020) (Section 20.3). Its plane search beat line search in simulation, and in a preliminary study with $N = 6$, participants were satisfied after 5.36 iterations on average and rated the grid as a source of inspiration at 6.50 on a 7-point scale. The authors list as limitations that the method suits only visual designs, handles neither discrete parameters nor prior knowledge, performs poorly above about 20 dimensions, and assumes that the user's perceptual function "does not change over time".

Systems from the same period search the latent space of a generative model, the compact input vector that a trained network decodes into an image, a melody, or a font. Chiu et al. (2020) built slider subspaces in it without Bayesian optimization; Chong et al. (2021) let people blend image candidates with several sliders and guide the search by painting; and Zhou, Koyama, Goto, and Igarashi had people pick the best of several generated melodies, first in a pilot study (Zhou et al., 2020), then with a slider that sets the exploration weight of the upper confidence bound (Zhou et al., 2021).

BO as Assistant removes the explicit query loop: it extracts preference pairs from ordinary slider edits and offers suggestions asynchronously, so the designer can "take the full initiative"; it had no formal user study, and its authors state that usability, usefulness, and agency were not evaluated (Koyama and Goto, 2022). Photographer-in-the-loop lighting is PBO with a gallery of four photographs, which in the authors' demonstrations found pleasing lighting in about 10 iterations (Yamamoto et al., 2022). FontCraft explores font styles from text, image, and font references, with a history whose choices can be withdrawn (Tatsukawa et al., 2025). And constrained PBO asks banner-ad designers only for pairwise preferences while a model of the click-through rate (the fraction of viewers who click an ad) acts as a constraint; its acquisition function, EUBOC, is the expected utility of the best option (EUBO, Section 19.4) with the constraint added (Iwai et al., 2025). The line then turned to language: designers intervening in a system-led optimization in natural language (Niwa et al., 2025), a language model turning vague preferences into a combinatorial problem (Kuroki et al., 2026), and, in a 2026 preprint, a feasibility-aware latent space of car-wheel designs (Owaki et al., 2026).

32.1.2 Cambridge, Aalto, and NYCU: multi-objective optimization with designers #

The second line, around Per Ola Kristensson, John Dudley, Antti Oulasvirta, Liwei Chan, George Mo, and Yi-Chi Liao (Cambridge, Aalto, and National Yang Ming Chiao Tung University), mostly uses multi-objective Bayesian optimization with performance objectives, not preference feedback. Multi-objective optimization (Section 14.5) measures several objectives at once, such as speed and accuracy, and returns the Pareto front, the designs that cannot be improved on one objective without getting worse on another.

Dudley et al. (2019) optimized a map search interface for task time, with 200 crowd workers in its first experiment. Chan et al. (2022) compared optimizer-led with designer-led design of a virtual reality touch technique (4 parameters) in a between-subjects experiment, in which each participant saw only one condition, with 40 novices; its results anchor Section 32.2. Shen et al. (2022) let 12 users pick their final gesture-keyboard design from their own Pareto front; Liao et al. (2023) had 8 designers work with the optimizer and with their own strategy; Mo et al. (2024) proposed cooperative optimization, in which designers mark designs and regions to guide the optimizer; and Liao et al. (2024b) pooled sessions with a "global Gaussian process" (the surrogate of Chapter 8) and started new users from a "warm-start Gaussian process". Later papers color-graded 360-degree images with two-level PBO (Yuan et al., 2025), built the cost of each prototype into the acquisition function (Langerak et al., 2026), and separated expert opinion, empirical studies, and simulators as feedback sources in MUSE (Fernandes Junior et al., 2026), accepted at ACM Transactions on Interactive Intelligent Systems (TiiS). A review by Dudley, Oulasvirta, Chan, and Kristensson appeared online in ACM Computing Surveys in August 2026; we could not obtain its text (Dudley et al., 2026).

32.1.3 ETH Zurich and Saarland: priors learned from other people #

Since 2024 a third line, around Yi-Chi Liao, Zhipeng Li, Christoph Gebhardt, Christian Holz, and Anna Maria Feit (ETH Zurich and Saarland University), has studied population priors: data from earlier users, so that a new user needs to be asked as little as possible. Liao et al. (2024a), a collaboration of Aalto University and Meta, built a population model from 14 participants and used it as the prior of a transfer acquisition function for 11 new ones, and continual human-in-the-loop optimization represents the whole population with a Bayesian neural network, one that keeps a distribution over its weights (Liao et al., 2025); both use performance feedback. Meta-PO meta-learns over the Gaussian process models of earlier users with the Sequential Gallery interface (Li et al., 2025a), and Song et al. (2025) inferred objective weights from users' manual edits of mixed-reality layouts. Three CHI 2026 papers followed: HOMI trained a neural acquisition function on simulated users before any real user arrived (Liao et al., 2026); APPO hides an image generator's text prompt and asks only for binary preferences while a language model rewrites the prompt (Li et al., 2026f); and AutoOptimization turns spoken preferences into the objectives of a layout optimizer (Li et al., 2026a).

32.1.4 Ulm: automotive interfaces rated by drivers and passengers #

The group of Pascal Jansen, Mark Colley, and Enrico Rukzio at Ulm University optimizes automotive interfaces on questionnaire ratings (trust, perceived safety, mental demand, predictability), with some of the largest samples in the field: the visualizations of an automated vehicle in OptiCarVis ($N = 117$, online, between-subjects) (Jansen et al., 2025), external vehicle displays ($N = 37$) (Colley et al., 2025), an air-taxi simulation with and without a motion seat ($N = 40$) (Meinhardt et al., 2025), a proactive in-car assistant in ProVoice ($N = 19$, within-subjects, meaning every participant experienced every condition) (Susak et al., 2026), blur in virtual-reality driving in BlurDriving (IMWUT, September 2026) (Li et al., 2026b), and a three-day study accepted at AutomotiveUI 2026 ($N = 74$) that compared continued optimization with keeping the day-one design (Colley et al., 2026).

32.1.5 LMU Munich: expertise and field deployment #

Ou, Buschek, Mayer, and Butz built, with an industry partner, a PBO system for simplifying 3D meshes (9 parameters), in which artists' ratings were converted into pairwise relations; it ran for 3 months with two full-time technical artists and then in a lab study with $N = 20$ (Ou et al., 2022). Ou et al. (2023) compared 60 users of different expertise on three tasks with an interface that asked for a ranking of 4 candidates, offered a "don't know" area, and ended when the user pressed a satisfaction button. These two studies supply much of Section 32.3 and Section 32.6.

32.1.6 Guiding generative models #

Four more systems put preference queries inside a generative model, often a diffusion model, the kind of generator behind most current text-to-image systems. BOgen builds 3D shapes part by part with PBO and a variational autoencoder (a network that decodes a short latent vector into an output), and was compared with a baseline by 30 designers (Lee et al., 2026). ROMBO personalizes music generation, evaluated in simulation and with 16 volunteers who scored each piece from 0 to 10 (Marcos et al., 2025). GimmBO runs PBO over the merging weights of 20 to 30 adapters, small add-on networks that each push a diffusion model toward a style (Liu et al., 2026b). MultiBO (ICML 2026) has the user pick, from 4 generated images, the one closest to the image they have in mind, and steers edits in the model's attention space for up to 50 rounds (Rajagopalan et al., 2026). It reports an image similarity of 0.9364 against 0.8365 for DiffusionDPO and won 70.82% of comparisons rated by 30 users, and its authors observe that the preference signal weakens as the images approach the target.

32.1.7 Other domains: touch, text entry, reading, architecture, engineering #

Catkin and Patoglu (2023) optimized the perceived realism of rendered springs and friction from pairwise comparisons, validated against a model tuned by experts, and in Zhang et al. (2026b) 13 participants made 40 rounds of comparisons with a 5-level confidence rating, for a held-out accuracy of 92.3% (range 85% to 100%). AdaptiFont optimized a generative font space for reading speed (performance, $N = 11$ in the main study) (Kadner et al., 2021); Tanaka et al. (2026) had 40 design students design a pavilion with 6 parameters, by optimization driven by ratings from 0 to 100 or by sliders; and Nandy and Goucher-Lambert (2025) adapted a parametric mug to the attribute "comfortable" ($N = 31$ from scratch, $N = 25$ from the previous group's data) and improved the perceived match between intent and design over a non-interactive model. McCourt and Dewancker (2019), a workshop paper, had to extend PBO to handle ties when coloring artwork, and Peng et al. (2026), a preprint, personalized generative user interfaces from pairwise judgments. For text entry the only study we found is the gesture keyboard of Shen et al. In visualization design, the applications of Bayesian optimization we found tune particular displays (OptiCarVis, a case in MUSE), and none of them is a PBO study.

32.1.8 The studies at a glance #

Table 32.1 lists the main user studies by year. The questionnaires named in it are explained in Section 32.2. Hypervolume measures how much of the objective space a Pareto front covers; larger is better.

32.2 Agency versus performance #

The most consistent result of the period concerns who leads the search. When the optimizer leads, outcome measures are better and mental demand is lower, but the person's agency (the sense of being in control of the process), ownership (the sense that the result is one's own), and expressiveness (the sense of being able to put one's own ideas into the result) fall clearly. When designers get ways to steer, most of the agency comes back at a small cost in outcome quality. Almost all of this evidence comes from multi-objective optimization with performance objectives.

Two questionnaires recur. The Creativity Support Index scores support for creative work on a 0 to 100 scale, from factors such as exploration, expressiveness, and enjoyment; the NASA Task Load Index (NASA-TLX) measures workload from six subscales, among them mental demand and effort. $p$ is the probability of a difference at least this large if the conditions truly did not differ, and $t(38)$ is a $t$ statistic with 38 degrees of freedom whose sign says which group scored lower.

The only study large enough to isolate the effect. In the between-subjects experiment of Chan et al. (2022), with 40 novice designers and performance objectives, the optimizer-led group reported lower agency ($t(38) = -5.523$, $p < 0.001$) and ownership ($t(38) = -3.892$, $p < 0.001$), while satisfaction and confidence did not differ. The Creativity Support Index was 75.3 for designer-led and 65.4 for optimizer-led design ($p = .011$), a difference that came mainly from expressiveness (44.9 against 23.0, $p = .001$). The NASA-TLX total did not differ (57.6 against 49.6, $p = .758$), but the optimizer-led group reported lower mental demand (14.9 against 8.4, $p = .011$) and effort (24.9 against 13.7, $p = .040$). Optimizer-led sessions were longer, 78.0 minutes against 51.8, their designs had better spatial error ($t(38) = 2.237$, $p < 0.05$), and the group explored a larger share of the design space. In the words of the abstract, designers guided by an optimizer "reported lower mental effort but also felt less creative and less in charge of the progress", and the authors conclude that such optimization can support novice designers "in cases where agency is not critical".

Three levels of control in one study. Mo et al. (2024) compared designer-led, cooperative, and optimizer-led design within subjects ($N = 18$). For "I felt that I had control over searching different areas of the design space", the median ratings were 6.0, 5.5, and 1.5 on a 7-point scale. Twelve participants preferred the cooperative condition, 4 the designer-led one, and 2 the optimizer-led one, with medians for willingness to use the method again of 6.0, 5.0, and 3.0 in the same order. The cooperative condition's relative hypervolume did not differ significantly from the optimizer-led one, and it needed fewer formal evaluations but found fewer Pareto-optimal designs.

Smaller studies point the same way. Among the 8 designers of Liao et al. (2023), optimizer-found designs showed "very similar performance" to the designers' own at significantly lower workload, while "designers may become detached from the design process when typical aspects of their role are subsumed" by the optimizer. In the melody study of Zhou et al. (2021) (preference feedback, 12 novice composers), letting users set the balance between exploration and exploitation themselves raised the Creativity Support Index total ($p < 0.01$) and satisfaction ($p = 0.041$), and 11 of 12 preferred balancing by hand. In study 1 of Niwa et al. (2025) ($N = 18$), the optimizer-led condition reached a higher relative hypervolume than designer-led design ($p < 0.001$) and than cooperation in natural language ($p = 0.001$) but had significantly lower agency than both (corrected $p = 0.002$); in study 2 ($N = 12$), language and explicit constraints performed the same ($p = 0.204$), language gave lower task load ($p = 0.034$), and constraints gave more agency ($p = 0.032$). Some participants were unsure "how much their instructions were actually being followed".

Weaker evidence. FontCraft's participants (preference feedback, $N = 10$) reported more agency with multimodal references than with the baseline, and 7 of the 10 mentioned the history view (Tatsukawa et al., 2025). In Colley et al. (2025), "I felt in control of the design process" averaged 5.81 and "the final design is mine" 5.30 on 7-point scales. Tanaka et al. (2026) found "process disclosure" (parameter plots, the range of evaluations, the direction of optimization) essential for agency and trust, even when participants' understanding stayed superficial, and Song et al. (2025) note that an unconstrained optimizer can move elements the user had placed by hand.

You can feel the trade-off in Figure 32.1: run the same budget three times, choosing every design yourself, letting the optimizer choose, and letting it suggest while you decide, and rate after each run how much you controlled where the search went.

Things to try:

• Do the You run first, without revealing the landscape. You will probably find the broad hill first; notice whether you spend the remaining tries polishing it.
• Switch to Optimizer and press Run to the end. Compare its best score with yours, then compare your two control ratings. Which gap is larger?
• Press New landscape and repeat. On the first landscape the optimizer's exploration finds the narrow, taller peak; on most of the next ones it does not reach that peak within its 10 tries, and your own search can do as well.

The map shows the whole design space at a glance. The design spaces in the studies were larger (4 parameters in Chan et al., 6 in the pavilion study, 9 in the mesh simplification of Ou et al.), and a person moving 9 sliders can no longer see which regions they have not tried, which is where an optimizer's systematic exploration should count for more (inference); Figure 19.4 shows from the other side how much less forty comparisons accomplish as parameters are added.

What it means for preferential optimization. Agency falls in the order designer-led, cooperative, user-guided, optimizer-led, and the differences in agency (sense-of-control medians of 6.0 against 1.5) are much larger than the differences in outcome between cooperative and optimizer-led design. For design tasks, this favors adding channels through which the person can steer over making the optimizer more autonomous (inference). Plain PBO, in which the person only answers the optimizer's questions, is the optimizer-led condition by construction (inference). The effect was isolated with performance objectives; among preference-feedback systems only Zhou et al. and FontCraft measured anything close to agency, with 12 and 10 participants (inference). A team building a preference tool should therefore measure agency in its own study, with an item such as Mo et al.'s, rather than assume the effect transfers (inference).

32.3 Unstable feedback and non-convergence #

A PBO model assumes that each answer is a noisy reading of one fixed utility, with independent, identically distributed (i.i.d.) noise (Chapter 19). The longest observation of real use shows how far practice can be from that. In the deployment of Ou et al. (2022) with two professional 3D artists, 415 of 549 evaluation sequences stopped at the first iteration, without any optimization being requested. The remaining 134 sequences ran 4.1 iterations on average (standard deviation 4.2, range 1 to 23), and only 16 of them (11.9%) produced a satisfactory result. In the lab study, 97 of 200 sequences (48.5%) ended satisfied. Figure 32.2 draws every sequence.

Things to try:

• With Count against set to All sequences, compare the two percentages: 2.9% in the field, 48.5% in the lab.
• Switch to Sequences that asked for optimization. The 415 gray squares drop out of the count and the field figure rises to 11.9%. Both numbers are correct; Exercise 32.1 asks which one describes the tool in practice.

The authors write that participants, if not rating at random, "at least behave highly unstable and inconsistent in the rating process". Asked about specific inconsistent ratings, the experts pointed to anchoring on grids they had seen before, the availability and representativeness heuristics, loss aversion, and diminishing returns, the judgment biases of Section 37.2. The authors concluded that the assumptions of a stable and complete preference and of i.i.d. noise do not hold: "human judgment is a fragile function to optimize for". Reading this as direct evidence that preferences are constructed during the session is an interpretation: the paper documents instability and non-convergence, for which construction is one explanation (inference).

Scattered but consistent signs elsewhere. A Sequential Gallery participant selected designs "based on criteria that I didn't have at the beginning" (Koyama et al., 2020); AdaptiFont finds the best font at the time of use rather than a stable optimum, since the font that maximizes reading speed may depend on the text, fatigue, and display (Kadner et al., 2021); and Dudley et al. (2019) had to cope with noise from differences between users and tasks and from learning effects.

Real people against simulated ones. In personalizing a retinal implant with 17 sighted participants, only about 50% of choices agreed with the simulated agent's (Schoinas et al., 2025) (Section 33.5). In the preprint of Peng et al. (2026), 20 participants with varying interface-design experience judged the same 600 pairs of generated interfaces and agreed with each other at only 0.25 on a chance-corrected agreement statistic where 0 means chance and 1 perfect agreement (Krippendorff's $\alpha$; Cohen's $\kappa$ is the same to two decimals). The vibrotactile study notes that comparisons between nearby candidates are "dominated by response noise" and that its model assumes stationarity, with no account of adaptation, fatigue, or drift (Zhang et al., 2026b). Meta-PO, Song et al., and LAPPI list static preferences as a limitation without measuring change (Li et al., 2025a; Song et al., 2025; Kuroki et al., 2026).

The models that could absorb such effects are discussed in Section 27.2 and Section 29.10, and the experiment that would separate noise from drift in Section 47.4.

32.4 Forms of feedback #

What should the system ask? Chapter 20 treats the query forms as methods; this section collects what was learned with people. Before 2026 each study varied one aspect of the query. Koyama and Igarashi (2018) argue that a query over a continuous space carries much more information than a comparison but is hard for crowd workers, so the single slider is their compromise. Chong et al. (2021) measured 17.2 seconds per iteration with one slider and 53.4 with four; four converged faster per iteration and were preferred, but $N = 3$. In virtual-reality color grading, users preferred comparing two options to four (Yuan et al., 2025). A "don't know" area let people express incomplete preferences (Ou et al., 2023); in the field study, rating 4 models instead of picking 1 did not remove inconsistency (Ou et al., 2022) (Section 20.4 covers ties and abstention). And Mikkola et al. (2020) argue that slider-like projective queries greatly reduce user workload.

The first comparisons on the same task, in 2026. GimmBO ($N = 12$, within-subjects, 20 iterations) found that ranking beat sliders in similarity to the target, in success rate, and in recovering the right sparse set of adapters; sliders switched on superfluous adapters and Sequential Gallery users got stuck in local minima, but ranking took longer per step (50.5 against 34.7 seconds) (Liu et al., 2026b). APPO, with binary preferences only, converged faster and with lower load but was less expressive than baselines that allowed text feedback or manual edits (Li et al., 2026f). MultiBO argues that choosing one of 4 candidates per round balances information against load better than a pair, from a design analysis rather than a controlled experiment (Rajagopalan et al., 2026). In an online study with 12 new users, Peng et al. (2026) (a preprint) found that personalization from 8 pairwise judgments beat every baseline, including shared design guidelines and the users' own written preferences, with an aggregate win rate of 60.35% against 50.9% for a judge conditioned on a persona. And Mo et al. (2024) criticize galleries because all options are still supplied by the optimizer, leaving the user nothing to express beyond liking or disliking them.

These results point the same way: lower the burden of each query or raise the information it carries. No study yet compares pairs, galleries, sliders, rankings, and ratings on one task with real users, so advice on format rests mainly on simulation, very small samples, and the authors' arguments (inference). Until it exists, the cheapest safeguard is a short pilot of two candidate formats on the real task, timing each answer and repeating a few queries (inference). Section 20.6 takes up why the interface is part of the model.

32.5 Individual differences and population priors #

A crowdsourcing framework assumes one preference shared by the crowd, though crowds from different backgrounds "can have clearly different preferences" (Koyama and Igarashi, 2018). Population priors borrow other people's data to help each new person. They work in experiments, but their gains are concentrated in the first iterations.

The evidence for population priors. The global Gaussian process of Liao et al. (2024b) (performance) produced a design that cut completion time by 5.5% and spatial error by 48%, and the warm-start Gaussian process raised hypervolume by 38% for experienced and 18% for novice users, though a global design oriented toward speed was not better than the baseline in spatial error. Meta-Bayesian optimization of wrist interaction improved absolute pointing by 22.92% over standard Bayesian optimization and 21.35% over manual calibration, and relative pointing by 25.43% and 13.60% (Liao et al., 2024a). Meta-PO (preference feedback) cut the iterations to satisfaction from 9.54 without transfer (standard deviation 2.19) to 5.86 when transferring between users with the same theme (standard deviation 1.20, $t(11) = 5.11$, $p < 0.001$) and to 7.41 across themes (standard deviation 1.28, $p < 0.01$); across themes needed more iterations than within a theme ($p < 0.01$), so the more the goals differ, the less transfer helps (Li et al., 2025a). HOMI was significantly better only at iterations 2 and 3 ($p = .040$ and $p = .002$); from iteration 6 all methods converged to the same level, and NASA-TLX did not differ (Liao et al., 2026).

The evidence for individual differences. Every study that measured individual differences found them large: hearing-aid gain adjustments ($N = 20$) (Søgaard Jensen et al., 2019) and preferred compression ratios (Baltzell et al., 2018) (Section 33.4), the font that maximized reading speed (Kadner et al., 2021), the trade-off between typing speed and accuracy (Shen et al., 2022), design preferences for air-taxi interfaces (Meinhardt et al., 2025), and preferred blur in driving (Li et al., 2026b). Even expert preference and measured outcome can disagree: in a pre-study, professional ad designers' preferences showed no positive correlation with actual click-through rates, which is why constrained PBO hands the click-through rate to the machine (Iwai et al., 2025).

A population prior shifts the unit of analysis from the person to the population, which sits uneasily with these individual differences. The studies show that population priors speed up early convergence; none shows that they do no harm when an individual departs from the population (inference). A system using such a prior should therefore let the person's own answers outweigh it quickly, and report results separately for the people farthest from the population (inference). Section 20.5 and Section 27.3 describe the models.

32.6 Novices and experts #

Who is being optimized for matters as much as how. In the study of Ou et al. (2023) ($N = 60$), "novices can achieve an expert level of quality performance, but participants with higher expertise led to more optimization iteration with more explicit preference while keeping satisfaction low. In contrast, novices were more easily satisfied and terminated faster." The authors' explanation is that "experts seek more diverse outcomes while the machine reaches optimal results". In the field study, too, lab participants were more easily satisfied than the expert artists, who applied quality criteria the lab participants ignored (Ou et al., 2022).

Experts repeatedly asked for direct control: to manipulate parameters outside the gallery (Koyama et al., 2020), or to edit glyphs, switch off style propagation, and see a fuller history (Tatsukawa et al., 2025). They were positive when the optimizer took over what they could not judge themselves: the 11 ad designers of constrained PBO, with 5.82 years of experience on average, endorsed letting the system manage click-through rate (Iwai et al., 2025). Materials-science experiments also found experts and non-experts behaving differently (Section 34.2).

Most samples, however, are novices or students: 40 novices in Chan et al., 18 participants aged 20 to 36 in Mo et al., 10 non-experts in FontCraft, participants all with a computing or machine-learning background in GimmBO, and participants aged 24 to 28 in APPO (Chan et al., 2022; Mo et al., 2024; Liu et al., 2026b; Li et al., 2026f). What is known about professional designers using PBO tools comes mainly from two technical artists and two interviews (inference).

32.7 Explanation and trust #

Explanations improved task performance in experiments. In a between-subjects experiment with $N = 213$ tuning 6 parameters of a simulated egg-boiling task, any explanation (a bar chart, rules, or text) improved task success, reduced the trials needed, and improved understanding and confidence, without adding workload (Chakraborty et al., 2025b). MOLONE's authors report that comparative explanations over inputs and outcomes gave significantly faster convergence in a user study whose size we could not obtain (Chakraborty et al., 2025a). ShapleyBO, a preprint, explains each proposal with Shapley values, which divide the proposal's expected value among the input parameters; in personalizing an exosuit, teams that saw the explanations had lower cumulative regret, the total shortfall from the best design over the session (Chapter 13) (Rodemann et al., 2024). CoExBO explains each candidate to foster trust and comes with a no-harm guarantee: even under adversarial input from the user, it converges like ordinary Bayesian optimization (Adachi et al., 2024). Participants of Niwa et al. (2025) found that explanations made unexpected suggestions more acceptable, while some found them too generic or too long.

Understanding the optimizer can also change the feedback. Colella et al. (2020) found that users who understood the optimizer gave strategically biased answers (21 people, a one-dimensional task, scalar feedback), unlike the truthful simulated users of most papers. Sandholtz et al. (2023) (Bayesian Analysis, 2023) found that many participants explored in ways standard acquisition functions do not capture, and a preprint by Weichert et al. (2025) documents an industrial case in which adding data and expert knowledge made Bayesian optimization worse.

The "trust" in the Ulm studies is trust in the vehicle, an objective being optimized, not trust in the optimizer (Jansen et al., 2025), and trust in the optimizer itself has almost never been measured directly in HCI studies of PBO (inference); a study that wants to claim it should ask about the optimizer by name. Making the search visible and editable (explicit constraints, process disclosure, a withdrawable history) appears to sustain agency better than text explanations alone (inference).

32.8 Against strong baselines #

A method can look good by beating a weak competitor. Table 32.2 sorts the main comparisons by how strong the comparison condition was. The grading is this chapter's judgment (inference): random queries, a single slider, and no visualization count as weak; designers' own design, manual parameter tuning, and settings from the literature as medium; similar optimizers, Bayesian optimization that ignores cost, and expert designs as strong.

Figure 32.3 places the studies on two axes, sample size and the strength of the comparison, colored by feedback type.

The pattern is clear (inference). The stronger positive results come from performance objectives: task time, typing speed, pointing error. Studies of pure preference feedback more often report satisfaction or iteration counts, or a win over a weak comparison such as a single slider (ROMBO's win over random queries was obtained with ratings). Against skilled manual work or a similar optimizer, final quality usually does not differ. What interactive Bayesian optimization delivered in design from 2017 to 2026 is best summarized as the same result at lower cost, in fewer iterations, evaluations, or minutes of effort, rather than a better result (inference). Section 46.7 turns this into advice on how to evaluate a new system.

32.9 Critiques and alternatives from design research #

Between 2017 and 2025 the critiques came mainly from the authors of optimization papers and from neighboring HCI research, and they are of two kinds. The first concerns agency, with the evidence of Section 32.2: Chan et al. propose "to push the optimizer to the background, making its suggestions recommendations and not dictations" (Chan et al., 2022), and Mo et al. (2024) add that an optimizer typically cannot "leverage the designer's expertise in quickly identifying that a given 'bad' design is not worth" evaluating.

The second concerns the utility model itself. Ou et al. (2022) conclude that "we need to generally rethink basic assumptions and approaches in the design of HITL systems". Koyama and Igarashi (2018) ask "Whose preference?": crowd frameworks assume "a 'general' (or universal) preference shared among crowds", while in some domains "only experts can adequately assess the quality of designs". Koyama and Goto (2022) leave open whether BO-generated suggestions are creative, and Dudley et al. (2019) could not tell whether their optimizer truly optimized or only excluded poorly performing regions. In 2026, Langerak et al. (2026) wrote that a predefined parameter space limits early exploration while the design space is still evolving, and Owaki et al. (2026) argue that the representation shown to the user is part of interaction design; in their study ($N = 40$) a 5-dimensional feasibility-aware latent space beat the raw 9-dimensional parameters.

Fixation. Design fixation is the tendency to stay close to examples one has seen, or to one's first idea. An AI image generator used during ideation led to more fixation, fewer ideas, and lower variety and originality ($N = 60$) (Wadinambiarachchi et al., 2024), and students can fixate more on their own first idea than on given examples (Leahy et al., 2020). A PBO gallery is a stream of system-supplied examples with the current best as the first idea, so the same fixation can be expected when PBO is used for ideation (inference). Gmeiner et al. (2023) watched 14 trained designers struggle to understand and steer generative design tools.

Preferences formed in the interaction. Research on co-creation with generative AI in 2025 and 2026 argued that preferences form during the interaction, that premature convergence and fixation are the main failures, and that some friction may help. Switchable divergent and convergent modes (HAICo, a preprint) scored higher than ChatGPT on every factor of the Creativity Support Index ($N = 24$, $p < 0.002$) (Wen et al., 2026); IdeaBlocks led designers to explore 2.13 times as many images, with 12.5% higher visual diversity (Choi et al., 2026); a workshop paper argues for keeping reflective friction (Avelino et al., 2026); a preprint found that elicitation surfaces preferences users had not yet formed (Kim et al., 2026); and Saracay et al. (2026) (COLM 2026) argue that agents should help users construct preferences, with a simulated-user benchmark whose user model a study with 25 people supports. Chapter 45 returns to what a comparison measures if preferences are partly built while answering.

The alternatives move toward mixed initiative, in which either side can take the lead: the systems of Section 32.2, a withdrawable history (Tatsukawa et al., 2025), constraints handled by the machine (Iwai et al., 2025), and the designer as a curator of constraints rather than a maker of prototypes (Jansen, 2025), a workshop paper. For ideation, Koch et al. (2019) used cooperative contextual bandits, a sequential recommender that adapts to feedback, to suggest inspirational material for mood boards instead of converging on an optimum; 14 of 16 professional designers preferred the tool.

Counter-evidence: optimization can support exploration. Optimizer-led designers explored more of the design space (Chan et al., 2022), cooperative designers moved farther between evaluations than designer-led ones (Mo et al., 2024), participants of Niwa et al. described suggestions that broke their fixation as "Oh, I see!" moments (Niwa et al., 2025), and 85% of the architecture students endorsed the diversity the optimizer brought ($p < 0.001$) (Tanaka et al., 2026).

Where the two critiques stand. The agency critique already has tested remedies (cooperative control, explicit constraints, an exploration slider) that restored most of the agency in experiments. The utility-model critique has only proposed remedies inside PBO, such as discarding history, decaying old data, and withdrawing choices, and a PBO tool has yet to be compared with an ideation tool built for divergence on the same creative task (inference). Critics and optimization authors largely agree on the diagnosis (fixation, unclear goals, preferences that form during the interaction) and disagree on the remedy.

The evidence suggests a working rule for design tools (inference): put the optimizer in the role of an advisor; give the machine the measurable sub-goals, such as click-through rate, feasibility, or task time; leave judgments of appearance to comparisons or rankings; make the state of the search visible and editable; and evaluate against manual tuning or skilled designers, not only against a single slider or random queries. The rule comes from the empirical results of this chapter and does not depend on the Gaussian process framework.

32.10 Common claims, checked #

Table 32.3 checks claims that circulate in secondary accounts against the primary sources.

Two citation details are easy to get wrong. The 2020 SIGGRAPH paper is called Sequential Gallery; "sequential plane search" is the method inside it (Koyama et al., 2020). And the melody work is two papers, a 2020 pilot (Zhou et al., 2020) and the IUI 2021 study with 12 participants (Zhou et al., 2021).

32.11 Settled, contested, missing #

32.12 Exercises #

Further reading #

• Chan et al. (2022) is the clearest controlled study of agency against performance; read its discussion of mixed initiative alongside the results.
• Mo et al. (2024) show what a cooperative middle ground looks like and measure it against both extremes in one within-subjects study.
• Ou et al. (2022) is the only long field deployment of PBO with professionals, and the most detailed record of how real feedback departs from the model.
• Koyama et al. (2020) and the book chapter Koyama and Igarashi (2018) give the interface line's reasoning about query forms and its frank list of limitations.
• Niwa et al. (2025) measure outcome and agency together when the designer steers in natural language, and are the bridge to the language-model systems of Chapter 35.