Dissertation · 2018 · Open access

A Comparative Study of Black-box Optimization Algorithms for Tuning of Hyper-parameters in Deep Neural Networks

Olof Skogby Steinholtz

DiVA (Luleå University of Technology) · Published 1 January 2018

Open full textCite this paper

In brief

“A Comparative Study of Black-box Optimization Algorithms for Tuning of Hyper-parameters in Deep Neural Networks” is a 2018 dissertation by Olof Skogby Steinholtz, published in DiVA (Luleå University of Technology). OpenAlex counts 31 citations as of 19 November 2026. It is open access: the full text is free to read at DiVA (Luleå University of Technology).

Citations
31
Most cited in
2024 (7)
Authors
1

Abstract

Deep neural networks (DNNs) have successfully been applied across various data intensive applications ranging from computer vision, language modeling, bioinformatics and search engines. Hyper-parameters of a DNN are defined as parameters that remain fixed during model training and heavily influence the DNN performance. Hence, regardless of application, the design-phase of constructing a DNN model becomes critical. Framing the selection and tuning of hyper-parameters as an expensive black-box optimization (BBO) problem, obstacles encountered in manual by-hand tuning could be addressed by taking instead an automated algorithmic approach. In this work, the following BBO algorithms: Nelder-Mead Algorithm (NM), ParticleSwarm Optmization (PSO), Bayesian Optimization with Gaussian Processes (BO-GP) and Tree-structured Parzen Estimator (TPE), are evaluated side-by-side for two hyper-parameter optimization problem instances. These instances are: Problem 1, incorporating a convolutionalneural network and Problem 2, incorporating a recurrent neural network. A simple Random Search (RS) algorithm acting as a baseline for performance comparison is also included in the experiments. Results in this work show that the TPE algorithm achieves the overall highest performance with respect to mean solution quality, speed ofimprovement and with a comparatively low trial-to-trial variability for both Problem 1 and Problem 2. The NM, PSO and BO-GP algorithms are shown capable of outperforming the RS baseline for Problem 1, but fails to do so in Problem 2.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Steinholtz, O. S. (2018). A Comparative Study of Black-box Optimization Algorithms for Tuning of Hyper-parameters in Deep Neural Networks. DiVA (Luleå University of Technology). http://urn.kb.se/resolve?urn=urn:nbn:se:ltu:diva-69865

MLA 9

Steinholtz, Olof Skogby. “A Comparative Study of Black-box Optimization Algorithms for Tuning of Hyper-parameters in Deep Neural Networks.” DiVA (Luleå University of Technology), 2018. http://urn.kb.se/resolve?urn=urn:nbn:se:ltu:diva-69865.

Chicago

Steinholtz, Olof Skogby. 2018. “A Comparative Study of Black-box Optimization Algorithms for Tuning of Hyper-parameters in Deep Neural Networks.” DiVA (Luleå University of Technology). http://urn.kb.se/resolve?urn=urn:nbn:se:ltu:diva-69865.

Harvard

Steinholtz, O. S. (2018) ‘A Comparative Study of Black-box Optimization Algorithms for Tuning of Hyper-parameters in Deep Neural Networks’, DiVA (Luleå University of Technology). Available at: http://urn.kb.se/resolve?urn=urn:nbn:se:ltu:diva-69865.

IEEE

O. S. Steinholtz, “A Comparative Study of Black-box Optimization Algorithms for Tuning of Hyper-parameters in Deep Neural Networks,” DiVA (Luleå University of Technology), 2018.

BibTeX

@phdthesis{steinholtz2018comparative,
  title     = {{A Comparative Study of Black-box Optimization Algorithms for Tuning of Hyper-parameters in Deep Neural Networks}},
  author    = {Steinholtz, Olof Skogby},
  year      = {2018},
  url       = {http://urn.kb.se/resolve?urn=urn:nbn:se:ltu:diva-69865}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Citations by year

YearCitationsShare of the best year
201911
202033
202133
202244
202366
202477
202566
202611

Topic and keywords

Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification

  • hyperparameter optimization
  • black-box optimization
  • deep neural network
  • Nelder-Mead algorithm
  • particle swarm optimization
  • Bayesian optimization
  • Gaussian processes
  • tree-structured Parzen estimator

Work with this paper

Most-cited papers in Computer Science

All highly cited Computer Science papers

Questions about this paper

How do I cite this paper in APA style?

Steinholtz, O. S. (2018). A Comparative Study of Black-box Optimization Algorithms for Tuning of Hyper-parameters in Deep Neural Networks. DiVA (Luleå University of Technology). http://urn.kb.se/resolve?urn=urn:nbn:se:ltu:diva-69865

How many times has this paper been cited?

31 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at DiVA (Luleå University of Technology): sign in and open the full text from this page.

Who wrote this paper?

Olof Skogby Steinholtz.

Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.