Preprint · 2021 · Open access
Automatic prior selection for meta Bayesian optimization with a case study on tuning deep neural network optimizers.
arXiv (Cornell University) · Published 16 September 2021
In brief
“Automatic prior selection for meta Bayesian optimization with a case study on tuning deep neural network optimizers.” is a 2021 preprint by Zi Wang and 8 co-authors, published in arXiv (Cornell University). OpenAlex counts 3 citations as of 19 November 2026. It is open access: the full text is free to read at arXiv (Cornell University).
- Citations
- 3
- Most cited in
- 2022 (1)
- References
- 34
- Authors
- 9
Abstract
The performance of deep neural networks can be highly sensitive to the choice of a variety of meta-parameters, such as optimizer parameters and model hyperparameters. Tuning these well, however, often requires extensive and costly experimentation. Bayesian optimization (BO) is a principled approach to solve such expensive hyperparameter tuning problems efficiently. Key to the performance of BO is specifying and refining a distribution over functions, which is used to reason about the optima of the underlying function being optimized. In this work, we consider the scenario where we have data from similar functions that allows us to specify a tighter distribution a priori. Specifically, we focus on the common but potentially costly task of tuning optimizer parameters for training neural networks. Building on the meta BO method from Wang et al. (2018), we develop practical improvements that (a) boost its performance by leveraging tuning results on multiple tasks without requiring observations for the same meta-parameter points across all tasks, and (b) retain its regret bound for a special case of our method. As a result, we provide a coherent BO solution for iterative optimization of continuous optimizer parameters. To verify our approach in realistic model training setups, we collected a large multi-task hyperparameter tuning dataset by training tens of thousands of configurations of near-state-of-the-art models on popular image and text datasets, as well as a protein sequence dataset. Our results show that on average, our method is able to locate good hyperparameters at least 3 times more efficiently than the best competing methods.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Wang, Z., Dahl, G. E., Swersky, K., Lee, C., Mariet, Z., Nado, Z., Gilmer, J., Snoek, J., & Ghahramani, Z. (2021). Automatic prior selection for meta Bayesian optimization with a case study on tuning deep neural network optimizers. arXiv (Cornell University). https://arxiv.org/pdf/2109.08215.pdf
MLA 9
Wang, Zi, et al. “Automatic prior selection for meta Bayesian optimization with a case study on tuning deep neural network optimizers.” arXiv (Cornell University), 2021. https://arxiv.org/pdf/2109.08215.pdf.
Chicago
Wang, Zi, George E. Dahl, Kevin Swersky, Chansoo Lee, Zelda Mariet, Zachary Nado, Justin Gilmer, Jasper Snoek, and Zoubin Ghahramani. 2021. “Automatic prior selection for meta Bayesian optimization with a case study on tuning deep neural network optimizers.” arXiv (Cornell University). https://arxiv.org/pdf/2109.08215.pdf.
Harvard
Wang, Z. et al. (2021) ‘Automatic prior selection for meta Bayesian optimization with a case study on tuning deep neural network optimizers.’, arXiv (Cornell University). Available at: https://arxiv.org/pdf/2109.08215.pdf.
IEEE
Z. Wang et al., “Automatic prior selection for meta Bayesian optimization with a case study on tuning deep neural network optimizers.,” arXiv (Cornell University), 2021.
BibTeX
@misc{wang2021automatic,
title = {{Automatic prior selection for meta Bayesian optimization with a case study on tuning deep neural network optimizers.}},
author = {Wang, Zi and Dahl, George E. and Swersky, Kevin and Lee, Chansoo and Mariet, Zelda and Nado, Zachary and Gilmer, Justin and Snoek, Jasper and Ghahramani, Zoubin},
howpublished = {arXiv (Cornell University)},
year = {2021},
url = {https://arxiv.org/pdf/2109.08215.pdf}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Citations by year
| Year | Citations | Share of the best year |
|---|---|---|
| 2022 | 1 | |
| 2024 | 1 | |
| 2025 | 1 |
Topic and keywords
Social Sciences › Decision Sciences › Management Science and Operations Research › Advanced Bandit Algorithms Research
- Bayesian optimization
- hyperparameter optimization
- multi-task learning
- regret bounds
Work with this paper
Most-cited papers in Decision Sciences
- The PRISMA 2020 statement: an updated guideline for reporting systematic reviewsMatthew James Page, Joanne E. McKenzie, Patrick M. Bossuyt and 23 more · BMJ · 2021 · 144,919 citations
- Evaluating Structural Equation Models with Unobservable Variables and Measurement ErrorClaes Fornell, David F. Larcker · Journal of Marketing Research · 1981 · 118,336 citations
- Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternativesLi‐tze Hu, Peter M. Bentler · Structural Equation Modeling A Multidisciplinary Journal · 1999 · 117,461 citations
- Common method biases in behavioral research: A critical review of the literature and recommended remedies.Philip M. Podsakoff, SCOTT B. MACKENZIE, Jeong Yeon Lee and 1 more · Journal of Applied Psychology · 2003 · 89,248 citations
- Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statementDavid Moher, Alessandro Liberati, Jennifer Tetzlaff and 2 more · BMJ · 2009 · 85,524 citations
Questions about this paper
How do I cite this paper in APA style?
Wang, Z., Dahl, G. E., Swersky, K., Lee, C., Mariet, Z., Nado, Z., Gilmer, J., Snoek, J., & Ghahramani, Z. (2021). Automatic prior selection for meta Bayesian optimization with a case study on tuning deep neural network optimizers. arXiv (Cornell University). https://arxiv.org/pdf/2109.08215.pdf
How many times has this paper been cited?
3 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at arXiv (Cornell University): sign in and open the full text from this page.
Who wrote this paper?
Zi Wang, George E. Dahl, Kevin Swersky, Chansoo Lee, Zelda Mariet, Zachary Nado, Justin Gilmer, Jasper Snoek, Zoubin Ghahramani.
Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.