Bayesian Optimization
Part V: Case Studies
中文

Case Studies

The first four parts built the methods on test functions chosen to make each idea visible. This part puts them to work on real problems, one per chapter, and keeps the details that test functions leave out: what an evaluation actually costs, where the noise comes from, which choices the practitioner has to make before the first query, and what the published studies found when the method met the problem.

Two chapters use measured data, so the optimizer in the figures runs against real outcomes rather than a formula: the validation error of a classifier on a real data set, and the yields of a fully enumerated set of chemical reactions. One chapter re-enacts human-in-the-loop exoskeleton tuning with a simulated walker calibrated to published studies, and says plainly which parts are simulated. In the last chapter the problem is real because you are part of it: you enhance a photograph by comparing versions, and the method learns from your eye.

The part assumes Part III and Part IV; each chapter can be read on its own.

Chapters in this part

  1. 22 Tuning a Classifier

    Bayesian optimization on measured outcomes: the validation-error landscape of a support vector machine over two hyperparameters, which you search yourself before grid, random, and Bayesian search replay it, then recorded searches over seven hyperparameters of a gradient-boosted tree model, with costs, noise, and held-out scores.

  2. 23 Optimizing a Chemical Reaction

    Bayesian optimization replayed on a published, fully measured reaction data set: 1,728 combinations of ligand, base, solvent, concentration, and temperature, each with a measured yield. You play the chemists' optimization game on the same data, then watch the optimizer with one-hot or descriptor encodings and different batch sizes, against random selection, 50 recorded chemists, and the paper's own runs.

  3. 24 Tuning an Exoskeleton with a Person in the Loop

    A re-enactment of human-in-the-loop exoskeleton tuning on a simulated walker calibrated to published studies: what a two-minute metabolic estimate is worth, a session in which you tune the device by feel against Bayesian optimization, an evolution strategy, and preferential optimization on the same walking time, and what a person who is still adapting does to all of them.

  4. 25 Enhancing a Photo by Comparison

    Preferential Bayesian optimization on a real photograph with six adjustments: you enhance it by picking one of two versions or by sliding along a line, the model learns your taste, and the chapter sets what a session reveals (consistency, drift, dependence on the order of questions) against the published studies.

References for Part V

60 works cited across this part's chapters.

  1. Abram, S. J., Poggensee, K. L., Sánchez, N., Simha, S. N., Finley, J. M., Collins, S. H., and Donelan, J. M. (2022). General variability leads to specific adaptation toward optimal movement policies. Current Biology. Ch. 24
  2. Adesiji, A. D., Wang, J., Kuo, C.-S., and Brown, K. A. (2026). Benchmarking self-driving labs. Digital Discovery. Ch. 23
  3. Alpaydin, E., and Kaynak, C. (1998). Optical Recognition of Handwritten Digits. UCI Machine Learning Repository, data set, CC BY 4.0. doi:10.24432/C50P49. non-peer-reviewed Ch. 22
  4. Bergstra, J., and Bengio, Y. (2012). Random Search for Hyper-Parameter Optimization. Journal of Machine Learning Research. Ch. 22
  5. Cawley, G. C., and Talbot, N. L. C. (2010). On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation. Journal of Machine Learning Research. Ch. 22
  6. Chang, S., Kim, C.-Y., and Cho, Y. S. (2017). Sequential effects in preference decision: Prior preference assimilates current preference. PLOS ONE. Ch. 25
  7. Chong, T. L. H., Shen, I.-C., Sato, I., and Igarashi, T. (2021). Interactive Optimization of Generative Image Modelling using Sequential Subspace Search and Content-based Guidance. Computer Graphics Forum. doi:10.1111/cgf.14188. Ch. 25
  8. Clites, T. R., Shepherd, M. K., Ingraham, K. A., Wontorcik, L., and Rouse, E. J. (2021). Understanding patient preference in prosthetic ankle stiffness. Journal of NeuroEngineering and Rehabilitation. Ch. 24
  9. Cortes, C., and Vapnik, V. (1995). Support-Vector Networks. Machine Learning. Ch. 22
  10. Deneault, J. R., Kim, W., Kim, J., Gu, Y., Chang, J., Maruyama, B., Myung, J. I., and Pitt, M. A. (2025). Preferential Bayesian optimization improves the efficiency of printing objects with subjective qualities. Digital Discovery. Ch. 23
  11. Díaz, M. A., Nuzzo, S., Flynn, L., Beckerle, P., Verstraten, T., and De Pauw, K. (2026). User preference in the personalized control of an ankle prosthesis: a case study. Journal of NeuroEngineering and Rehabilitation. doi:10.1186/s12984-026-01931-w. Ch. 24
  12. Ding, Y., Kim, M., Kuindersma, S., and Walsh, C. J. (2018). Human-in-the-Loop Optimization of Hip Assistance with a Soft Exosuit during Walking. Science Robotics. Ch. 24
  13. Feurer, M., and Hutter, F. (2019). Hyperparameter Optimization. Automated Machine Learning: Methods, Systems, Challenges. Ch. 22
  14. Friedman, J. H. (2001). Greedy Function Approximation: A Gradient Boosting Machine. The Annals of Statistics. Ch. 22
  15. Ginsbourger, D., Le Riche, R., and Carraro, L. (2010). Kriging Is Well-Suited to Parallelize Optimization. Computational Intelligence in Expensive Optimization Problems. Ch. 23
  16. Gupta, R., Hartford, J., and Liu, B. (2025). LLMs for Bayesian Optimization in Scientific Domains: Are We There Yet? Findings of the Association for Computational Linguistics: EMNLP 2025. Ch. 23
  17. Hansen, N., and Ostermeier, A. (2001). Completely Derandomized Self-Adaptation in Evolution Strategies. Evolutionary Computation. Ch. 24
  18. Hopkins, M., Reeber, E., Forman, G., and Suermondt, J. (1999). Spambase. UCI Machine Learning Repository, data set, CC BY 4.0. doi:10.24432/C53G6X. non-peer-reviewed Ch. 22
  19. Hsu, C.-W., Chang, C.-C., and Lin, C.-J. (2003). A Practical Guide to Support Vector Classification. Department of Computer Science, National Taiwan University. non-peer-reviewed Ch. 22
  20. Hutter, F., Hoos, H., and Leyton-Brown, K. (2014). An Efficient Approach for Assessing Hyperparameter Importance. International Conference on Machine Learning. Ch. 22
  21. Ingraham, K. A., Remy, C. D., and Rouse, E. J. (2022). The role of user preference in the customized control of robotic exoskeletons. Science Robotics. Ch. 24
  22. Iwai, K., Kumagae, Y., Koyama, Y., Hamasaki, M., and Goto, M. (2025). Constrained Preferential Bayesian Optimization and Its Application in Banner Ad Design. Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence. Ch. 25
  23. Kalinin, S. V., Liu, Y., Biswas, A., Duscher, G., Pratiush, U., Roccapriore, K., Ziatdinov, M., and Vasudevan, R. (2024). Human-in-the-loop: The future of Machine Learning in Automated Electron Microscopy. Microscopy Today. doi:10.1093/mictod/qaad096. Ch. 23
  24. Keerthi, S. S., and Lin, C.-J. (2003). Asymptotic Behaviors of Support Vector Machines with Gaussian Kernel. Neural Computation. Ch. 22
  25. Kim, G., and Sergi, F. (2026). Validation of Dynamic Bayesian Optimization for Human-in-the-Loop Optimization of Exoskeleton Control at User-Driven Walking Speed. bioRxiv. preprint Ch. 24
  26. Klein, A., Falkner, S., Bartels, S., Hennig, P., and Hutter, F. (2017). Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets. Artificial Intelligence and Statistics. Ch. 22
  27. Koyama, Y., and Igarashi, T. (2018). Computational Design with Crowds. Computational Interaction. Ch. 25
  28. Koyama, Y., Sato, I., Sakamoto, D., and Igarashi, T. (2017). Sequential line search for efficient visual design optimization by crowds. ACM Transactions on Graphics. Ch. 25
  29. Koyama, Y., Sato, I., and Goto, M. (2020). Sequential Gallery for Interactive Visual Design Optimization. ACM Transactions on Graphics 39(4) (SIGGRAPH 2020). Ch. 25
  30. Kutulakos, Z., and Slade, P. (2024). Simulating human-in-the-loop optimization of exoskeleton assistance to compare optimization algorithm performance. bioRxiv. preprint Ch. 24
  31. Lee, U. H., Shetty, V. S., Franks, P. W., Tan, J., Evangelopoulos, G., Ha, S., and Rouse, E. J. (2023). User preference optimization for control of ankle exoskeletons using sample efficient active learning. Science Robotics. Ch. 24
  32. Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., and Talwalkar, A. (2018). Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization. Journal of Machine Learning Research. Ch. 22
  33. Liu, C., Ling, S., and Jacobson, A. (2026b). GimmBO: Interactive Generative Image Model Merging via Bayesian Optimization. ACM Transactions on Graphics. doi:10.1145/3811293. Ch. 25
  34. Liu, X.-Y., Li, G., Wang, W., and Hou, Z.-G. (2026d). Personalized Lower-limb Exoskeleton Assistance via Preference-based Bayesian Optimization. arXiv. preprint Ch. 24
  35. Maberry, A., and Martin, A. E. (2026). Just Noticeable Difference of Impedance Parameters While Walking in an Ankle Exoskeleton. IEEE Transactions on Neural Systems and Rehabilitation Engineering. Ch. 24
  36. Mikkola, P., Todorović, M., Järvi, J., Rinke, P., and Kaski, S. (2020). Projective Preferential Bayesian Optimization. International Conference on Machine Learning. Ch. 25
  37. Ou, C., Buschek, D., Mayer, S., and Butz, A. (2022). The Human in the Infinite Loop: A Case Study on Revealing and Explaining Human-AI Interaction Loop Failures. Mensch und Computer 2022. Ch. 25
  38. Ou, C., Mayer, S., and Butz, A. (2023). The Impact of Expertise in the Loop for Exploring Machine Rationality. IUI 2023. Ch. 25
  39. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., … Duchesnay, É. (2011). Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research. Ch. 22
  40. Poggensee, K. L., and Collins, S. H. (2021). How adaptation, training, and customization contribute to benefits from exoskeleton assistance. Science Robotics. Ch. 24
  41. Pombo, M., Brielmann, A. A., and Pelli, D. G. (2023). The intrinsic variance of beauty judgment. Attention, Perception, & Psychophysics. Ch. 25
  42. Pratiush, U., Roccapriore, K. M., Liu, Y., Duscher, G., Ziatdinov, M., and Kalinin, S. V. (2025). Building Workflows for Interactive Human in the Loop Automated Experiment (hAE) in STEM-EELS. Digital Discovery. doi:10.1039/d5dd00033e. Ch. 23
  43. Probst, P., Boulesteix, A.-L., and Bischl, B. (2019). Tunability: Importance of Hyperparameters of Machine Learning Algorithms. Journal of Machine Learning Research. Ch. 22
  44. Ranković, B., Griffiths, R.-R., and Schwaller, P. (2026). Large language models as uncertainty-calibrated optimizers for experimental discovery. Nature Machine Intelligence. doi:10.1038/s42256-026-01283-z. Ch. 23
  45. Schäfer, N., Zhao, G., Li, B., Kupnik, M., Seyfarth, A., Beckerle, P., and Grimmer, M. (2026). User preference-based human-in-the-loop tuning of exoskeleton assistance during walking. npj Biomedical Innovations. doi:10.1038/s44385-026-00085-7. Ch. 24
  46. Schoinas, E., Rastogi, A., Carter, A., Granley, J., and Beyeler, M. (2025). Evaluating Deep Human-in-the-Loop Optimization for Retinal Implants Using Sighted Participants. 2025 47th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). doi:10.1109/embc58623.2025.11253762. Ch. 25
  47. Selinger, J. C., and Donelan, J. M. (2014). Estimating instantaneous energetic cost during non-steady-state gait. Journal of Applied Physiology. Ch. 24
  48. Shields, B. J. (2021). EDBO: Experimental Design via Bayesian Optimization. GitHub repository, MIT License; direct arylation data in experiments/data/direct_arylation, commit 9b41eac. software Ch. 23
  49. Shields, B. J., and Li, J. (2020). EvML: Expert versus Machine Learning. GitHub repository, MIT License; reaction optimization game records and analysis, commit 9fb4655. software Ch. 23
  50. Shields, B. J., Stevens, J., Li, J., Parasram, M., Damani, F., Alvarado, J. I. M., … Doyle, A. G. (2021). Bayesian reaction optimization as a tool for chemical synthesis. Nature. Ch. 23
  51. Siivola, E., Vehtari, A., Vanhatalo, J., González, J., and Andersen, M. R. (2018). Correcting Boundary Over-Exploration Deficiencies in Bayesian Optimization with Virtual Derivative Sign Observations. 2018 IEEE 28th International Workshop on Machine Learning for Signal Processing (MLSP). Ch. 22
  52. Slade, P., Kochenderfer, M. J., Delp, S. L., and Collins, S. H. (2022). Personalizing exoskeleton assistance while walking in the real world. Nature. Ch. 24
  53. Snoek, J., Larochelle, H., and Adams, R. P. (2012). Practical Bayesian Optimization of Machine Learning Algorithms. Advances in Neural Information Processing Systems 25 (NeurIPS 2012). Ch. 22
  54. Tucker, M., Cheng, M., Novoseller, E., Cheng, R., Yue, Y., Burdick, J. W., and Ames, A. D. (2020a). Human Preference-Based Learning for High-dimensional Optimization of Exoskeleton Walking Gaits. IROS 2020. Ch. 24
  55. Tucker, M., Novoseller, E., Kann, C., Sui, Y., Yue, Y., Burdick, J. W., and Ames, A. D. (2020b). Preference-Based Learning for Exoskeleton Gait Optimization. 2020 IEEE International Conference on Robotics and Automation (ICRA). Ch. 24
  56. Turner, R., Eriksson, D., McCourt, M., Kiili, J., Laaksonen, E., Xu, Z., and Guyon, I. (2021). Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020. NeurIPS 2020 Competition and Demonstration Track. Ch. 22
  57. Xie, Q., Astudillo, R., Frazier, P. I., Scully, Z., and Terenin, A. (2024). Cost-aware Bayesian Optimization via the Pandora's Box Gittins Index. NeurIPS 2024. Ch. 22
  58. Xie, Q., Cai, L., Terenin, A., Frazier, P. I., and Scully, Z. (2026). Cost-aware Stopping for Bayesian Optimization. International Conference on Machine Learning. Ch. 22
  59. Zhang, J., Fiers, P., Witte, K. A., Jackson, R. W., Poggensee, K. L., Atkeson, C. G., and Collins, S. H. (2017). Human-in-the-loop optimization of exoskeleton assistance during walking. Science. Ch. 24
  60. Zhang, R., Zhu, X., Pourebadi Khotbehsara, M., Dao, W., Bıyık, E., and Culbertson, H. (2026b). Vibrotactile Preference Learning: Uncertainty-Aware Preference Learning for Personalized Vibration Feedback. UMAP 2026 (per Semantic Scholar). Ch. 25