Dissertation · 2018 · Open access
A Comparative Study of Black-box Optimization Algorithms for Tuning of Hyper-parameters in Deep Neural Networks
DiVA (Luleå University of Technology) · Published 1 January 2018
In brief
“A Comparative Study of Black-box Optimization Algorithms for Tuning of Hyper-parameters in Deep Neural Networks” is a 2018 dissertation by Olof Skogby Steinholtz, published in DiVA (Luleå University of Technology). OpenAlex counts 31 citations as of 19 November 2026. It is open access: the full text is free to read at DiVA (Luleå University of Technology).
- Citations
- 31
- Most cited in
- 2024 (7)
- Authors
- 1
Abstract
Deep neural networks (DNNs) have successfully been applied across various data intensive applications ranging from computer vision, language modeling, bioinformatics and search engines. Hyper-parameters of a DNN are defined as parameters that remain fixed during model training and heavily influence the DNN performance. Hence, regardless of application, the design-phase of constructing a DNN model becomes critical. Framing the selection and tuning of hyper-parameters as an expensive black-box optimization (BBO) problem, obstacles encountered in manual by-hand tuning could be addressed by taking instead an automated algorithmic approach. In this work, the following BBO algorithms: Nelder-Mead Algorithm (NM), ParticleSwarm Optmization (PSO), Bayesian Optimization with Gaussian Processes (BO-GP) and Tree-structured Parzen Estimator (TPE), are evaluated side-by-side for two hyper-parameter optimization problem instances. These instances are: Problem 1, incorporating a convolutionalneural network and Problem 2, incorporating a recurrent neural network. A simple Random Search (RS) algorithm acting as a baseline for performance comparison is also included in the experiments. Results in this work show that the TPE algorithm achieves the overall highest performance with respect to mean solution quality, speed ofimprovement and with a comparatively low trial-to-trial variability for both Problem 1 and Problem 2. The NM, PSO and BO-GP algorithms are shown capable of outperforming the RS baseline for Problem 1, but fails to do so in Problem 2.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Steinholtz, O. S. (2018). A Comparative Study of Black-box Optimization Algorithms for Tuning of Hyper-parameters in Deep Neural Networks. DiVA (Luleå University of Technology). http://urn.kb.se/resolve?urn=urn:nbn:se:ltu:diva-69865
MLA 9
Steinholtz, Olof Skogby. “A Comparative Study of Black-box Optimization Algorithms for Tuning of Hyper-parameters in Deep Neural Networks.” DiVA (Luleå University of Technology), 2018. http://urn.kb.se/resolve?urn=urn:nbn:se:ltu:diva-69865.
Chicago
Steinholtz, Olof Skogby. 2018. “A Comparative Study of Black-box Optimization Algorithms for Tuning of Hyper-parameters in Deep Neural Networks.” DiVA (Luleå University of Technology). http://urn.kb.se/resolve?urn=urn:nbn:se:ltu:diva-69865.
Harvard
Steinholtz, O. S. (2018) ‘A Comparative Study of Black-box Optimization Algorithms for Tuning of Hyper-parameters in Deep Neural Networks’, DiVA (Luleå University of Technology). Available at: http://urn.kb.se/resolve?urn=urn:nbn:se:ltu:diva-69865.
IEEE
O. S. Steinholtz, “A Comparative Study of Black-box Optimization Algorithms for Tuning of Hyper-parameters in Deep Neural Networks,” DiVA (Luleå University of Technology), 2018.
BibTeX
@phdthesis{steinholtz2018comparative,
title = {{A Comparative Study of Black-box Optimization Algorithms for Tuning of Hyper-parameters in Deep Neural Networks}},
author = {Steinholtz, Olof Skogby},
year = {2018},
url = {http://urn.kb.se/resolve?urn=urn:nbn:se:ltu:diva-69865}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Citations by year
| Year | Citations | Share of the best year |
|---|---|---|
| 2019 | 1 | |
| 2020 | 3 | |
| 2021 | 3 | |
| 2022 | 4 | |
| 2023 | 6 | |
| 2024 | 7 | |
| 2025 | 6 | |
| 2026 | 1 |
Topic and keywords
Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- hyperparameter optimization
- black-box optimization
- deep neural network
- Nelder-Mead algorithm
- particle swarm optimization
- Bayesian optimization
- Gaussian processes
- tree-structured Parzen estimator
Work with this paper
Most-cited papers in Computer Science
- ImageNet classification with deep convolutional neural networksAlex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton · Communications of the ACM · 2017 · 114,726 citations
- Long Short-Term MemorySepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 109,097 citations
- A Mathematical Theory of CommunicationClaude E. Shannon · Bell System Technical Journal · 1948 · 91,736 citations
- Deep learningYann LeCun, Yoshua Bengio, Geoffrey E. Hinton · Nature · 2015 · 90,226 citations
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksShaoqing Ren, Kaiming He, Ross Girshick and 1 more · IEEE Transactions on Pattern Analysis and Machine Intelligence · 2016 · 74,143 citations
Questions about this paper
How do I cite this paper in APA style?
Steinholtz, O. S. (2018). A Comparative Study of Black-box Optimization Algorithms for Tuning of Hyper-parameters in Deep Neural Networks. DiVA (Luleå University of Technology). http://urn.kb.se/resolve?urn=urn:nbn:se:ltu:diva-69865
How many times has this paper been cited?
31 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at DiVA (Luleå University of Technology): sign in and open the full text from this page.
Who wrote this paper?
Olof Skogby Steinholtz.
Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.