LLM-Guided Bayesian Optimization for Faster ConvergenceLLM引导的贝叶斯优化加速收敛

The following content is generated by LLMs and may contain inaccuracies.

Context

This sits at the intersection of automated machine learning (AutoML), Bayesian optimization (BO), and large language model (LLM) reasoning. Traditional BO excels at sample-efficient optimization but struggles with cold-start problems and lacks domain priors that human experts bring. The 2023 work you reference (Papagiannopoulou et al.) proposes using LLMs to encode structured domain knowledge—replacing or augmenting the acquisition function that guides where to sample next. This matters now because LLMs have proven effective at extracting patterns from text-based technical knowledge that would otherwise require costly human-in-the-loop guidance.

Key Insights

• LLMs as surrogate priors: The core innovation is using LLMs to propose promising regions of the search space by leveraging scientific literature, API documentation, or past optimization logs. Papagiannopoulou’s LLAMBO demonstrates that GPT-4 can recommend hyperparameters competitive with Gaussian process surrogates, especially when search spaces are semantically structured (e.g., learning rates, architectural choices). However, LLMs hallucinate numerical relationships—they excel at categorical/ordinal decisions but need guardrails when suggesting continuous values.

• Self-referential optimization loops: Your observation about “LLMs writing for LLMs” connects to recent prompt optimization frameworks like DSPy and Textgrad, which differentiate through LLM calls to optimize prompts. Extending this to BO means the LLM doesn’t just suggest parameters—it iteratively refines its own suggestion strategy based on observed outcomes. The risk: LLMs lack calibrated uncertainty estimates, so integrating them into BO’s exploration-exploitation tradeoff remains fragile without explicit uncertainty quantification (e.g., ensembles or conformal prediction wrappers).

• Where heuristics beat surrogates: LLMs shine in high-dimensional discrete spaces where Gaussian processes fail (e.g., code generation hyperparameters, graph neural network architectures). But standard BO already converges fast in low-dimensional continuous problems—LLM overhead may not justify gains there. The sweet spot is mixed discrete-continuous spaces with interpretable parameters.

Open Questions

1. Can LLMs learn to query themselves strategically—deciding when to inject domain knowledge versus deferring to BO’s probabilistic model—without degrading sample efficiency?
2. How do we prevent LLMs from amplifying biases in optimization literature (e.g., over-indexing on popular heuristics like Adam over niche but effective alternatives)?