Paper · 2020 · Open access

Efficient bayesian hyperparameter optimization

Aaron Klein

FreiDok plus (Universitätsbibliothek Freiburg) · Published 1 January 2020 · DOI 10.6094/unifr/166637

Open full textCite this paper

In brief

“Efficient bayesian hyperparameter optimization” is a 2020 paper by Aaron Klein, published in FreiDok plus (Universitätsbibliothek Freiburg). OpenAlex counts 2 citations as of 19 November 2026. It is open access: the full text is free to read at FreiDok plus (Universitätsbibliothek Freiburg).

Citations
2
Most cited in
2021 (1)
Field-weighted impact
0.14
Authors
1

Abstract

Automated machine learning emerged as a new research field inside of machine learning that tries to progressively automate different steps of common machine learning pipelines which are traditionally executed by humans. One of its core tasks is the automated search for the right hyperparameters of a given machine learning algorithm which in practice is often essential to achieve good performance. Compare to other optimization problems, hyperparameter optimization is usually particularly expensive, since in each iteration, it requires to train and validate the underlying algorithm. One of the most successful approaches for hyperparameter optimization is Bayesian optimization. At its core, Bayesian optimization fits a probabilistic model of the objective function, which together with an additional acquisition function is used to guide the search towards the global optimum. In this thesis we present several extensions to standard Bayesian optimization to improve its performance for hyperparameter optimization problems. First, we introduce a new probabilistic model based on Bayesian neural networks, that allows to model the performance of hyperparameter configurations across different tasks and thereby scales much better with the number of data points and dimensions than Gaussian processes which are traditionally used inside Bayesian optimization. In hyperparameter optimization, often approximations, so-called fidelities, of the objective function are available which are much cheaper to evaluate. We present two new Bayesian optimization methods that can leverage such fidelities, such as learning curves or dataset subsets, to improve the overall search process in terms of wall-clock time by orders of magnitude. Furthermore, based on our proposed Bayesian neural network model, we present a new neural network architecture which models the learning curve of iterative machine learning methods, such as neural networks. Finally, due to the high computational cost of hyperparameter optimization, thorough benchmarking and evaluation of new developed methods is often prohibitively expensive. We show that one can approximate continuous and discrete benchmarks by surrogate benchmarks that capture the characteristics of the original benchmark but take only milliseconds to evaluate. This allows us to performa rigorous analysis and comparison of various different hyperparameter optimization methods from the literature.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Klein, A. (2020). Efficient bayesian hyperparameter optimization. FreiDok plus (Universitätsbibliothek Freiburg). https://doi.org/10.6094/unifr/166637

MLA 9

Klein, Aaron. “Efficient bayesian hyperparameter optimization.” FreiDok plus (Universitätsbibliothek Freiburg), 2020. https://doi.org/10.6094/unifr/166637.

Chicago

Klein, Aaron. 2020. “Efficient bayesian hyperparameter optimization.” FreiDok plus (Universitätsbibliothek Freiburg). https://doi.org/10.6094/unifr/166637.

Harvard

Klein, A. (2020) ‘Efficient bayesian hyperparameter optimization’, FreiDok plus (Universitätsbibliothek Freiburg). doi:10.6094/unifr/166637.

IEEE

A. Klein, “Efficient bayesian hyperparameter optimization,” FreiDok plus (Universitätsbibliothek Freiburg), 2020, doi: 10.6094/unifr/166637.

BibTeX

@article{klein2020efficient,
  title     = {{Efficient bayesian hyperparameter optimization}},
  author    = {Klein, Aaron},
  journal   = {FreiDok plus (Universitätsbibliothek Freiburg)},
  year      = {2020},
  doi       = {10.6094/unifr/166637},
  url       = {https://doi.org/10.6094/unifr/166637}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Social Sciences › Decision Sciences › Management Science and Operations Research › Advanced Bandit Algorithms Research

  • hyperparameter optimization
  • Bayesian optimization
  • Bayesian neural networks
  • acquisition functions
  • fidelity
  • learning curve
  • automated machine learning
  • Gaussian processes

Work with this paper

Most-cited papers in Decision Sciences

All highly cited Decision Sciences papers

Questions about this paper

How do I cite this paper in APA style?

Klein, A. (2020). Efficient bayesian hyperparameter optimization. FreiDok plus (Universitätsbibliothek Freiburg). https://doi.org/10.6094/unifr/166637

How many times has this paper been cited?

2 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at FreiDok plus (Universitätsbibliothek Freiburg): sign in and open the full text from this page.

Who wrote this paper?

Aaron Klein.

What is the DOI of this paper?

10.6094/unifr/166637, permanently at https://doi.org/10.6094/unifr/166637. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.