Preprint · 2021 · Open access

Automatic prior selection for meta Bayesian optimization with a case study on tuning deep neural network optimizers.

Zi Wang, George E. Dahl, Kevin Swersky, Chansoo Lee, Zelda Mariet, Zachary Nado, Justin Gilmer, Jasper Snoek, Zoubin Ghahramani

arXiv (Cornell University) · Published 16 September 2021

Open full textCite this paper

In brief

“Automatic prior selection for meta Bayesian optimization with a case study on tuning deep neural network optimizers.” is a 2021 preprint by Zi Wang and 8 co-authors, published in arXiv (Cornell University). OpenAlex counts 3 citations as of 19 November 2026. It is open access: the full text is free to read at arXiv (Cornell University).

Citations
3
Most cited in
2022 (1)
References
34
Authors
9

Abstract

The performance of deep neural networks can be highly sensitive to the choice of a variety of meta-parameters, such as optimizer parameters and model hyperparameters. Tuning these well, however, often requires extensive and costly experimentation. Bayesian optimization (BO) is a principled approach to solve such expensive hyperparameter tuning problems efficiently. Key to the performance of BO is specifying and refining a distribution over functions, which is used to reason about the optima of the underlying function being optimized. In this work, we consider the scenario where we have data from similar functions that allows us to specify a tighter distribution a priori. Specifically, we focus on the common but potentially costly task of tuning optimizer parameters for training neural networks. Building on the meta BO method from Wang et al. (2018), we develop practical improvements that (a) boost its performance by leveraging tuning results on multiple tasks without requiring observations for the same meta-parameter points across all tasks, and (b) retain its regret bound for a special case of our method. As a result, we provide a coherent BO solution for iterative optimization of continuous optimizer parameters. To verify our approach in realistic model training setups, we collected a large multi-task hyperparameter tuning dataset by training tens of thousands of configurations of near-state-of-the-art models on popular image and text datasets, as well as a protein sequence dataset. Our results show that on average, our method is able to locate good hyperparameters at least 3 times more efficiently than the best competing methods.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Wang, Z., Dahl, G. E., Swersky, K., Lee, C., Mariet, Z., Nado, Z., Gilmer, J., Snoek, J., & Ghahramani, Z. (2021). Automatic prior selection for meta Bayesian optimization with a case study on tuning deep neural network optimizers. arXiv (Cornell University). https://arxiv.org/pdf/2109.08215.pdf

MLA 9

Wang, Zi, et al. “Automatic prior selection for meta Bayesian optimization with a case study on tuning deep neural network optimizers.” arXiv (Cornell University), 2021. https://arxiv.org/pdf/2109.08215.pdf.

Chicago

Wang, Zi, George E. Dahl, Kevin Swersky, Chansoo Lee, Zelda Mariet, Zachary Nado, Justin Gilmer, Jasper Snoek, and Zoubin Ghahramani. 2021. “Automatic prior selection for meta Bayesian optimization with a case study on tuning deep neural network optimizers.” arXiv (Cornell University). https://arxiv.org/pdf/2109.08215.pdf.

Harvard

Wang, Z. et al. (2021) ‘Automatic prior selection for meta Bayesian optimization with a case study on tuning deep neural network optimizers.’, arXiv (Cornell University). Available at: https://arxiv.org/pdf/2109.08215.pdf.

IEEE

Z. Wang et al., “Automatic prior selection for meta Bayesian optimization with a case study on tuning deep neural network optimizers.,” arXiv (Cornell University), 2021.

BibTeX

@misc{wang2021automatic,
  title     = {{Automatic prior selection for meta Bayesian optimization with a case study on tuning deep neural network optimizers.}},
  author    = {Wang, Zi and Dahl, George E. and Swersky, Kevin and Lee, Chansoo and Mariet, Zelda and Nado, Zachary and Gilmer, Justin and Snoek, Jasper and Ghahramani, Zoubin},
  howpublished = {arXiv (Cornell University)},
  year      = {2021},
  url       = {https://arxiv.org/pdf/2109.08215.pdf}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Citations by year

YearCitationsShare of the best year
202211
202411
202511

Topic and keywords

Social Sciences › Decision Sciences › Management Science and Operations Research › Advanced Bandit Algorithms Research

  • Bayesian optimization
  • hyperparameter optimization
  • multi-task learning
  • regret bounds

Work with this paper

Most-cited papers in Decision Sciences

All highly cited Decision Sciences papers

Questions about this paper

How do I cite this paper in APA style?

Wang, Z., Dahl, G. E., Swersky, K., Lee, C., Mariet, Z., Nado, Z., Gilmer, J., Snoek, J., & Ghahramani, Z. (2021). Automatic prior selection for meta Bayesian optimization with a case study on tuning deep neural network optimizers. arXiv (Cornell University). https://arxiv.org/pdf/2109.08215.pdf

How many times has this paper been cited?

3 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at arXiv (Cornell University): sign in and open the full text from this page.

Who wrote this paper?

Zi Wang, George E. Dahl, Kevin Swersky, Chansoo Lee, Zelda Mariet, Zachary Nado, Justin Gilmer, Jasper Snoek, Zoubin Ghahramani.

Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.