Journal article · 2026 · Open access
A Survey of Optimization Methods in Machine Learning: From Gradient Descent to Convex Optimization
Applied and Computational Engineering, 231(1), 62–68 · Published 6 May 2026 · DOI 10.54254/2755-2721/2026.ba33212
In brief
“A Survey of Optimization Methods in Machine Learning: From Gradient Descent to Convex Optimization” is a 2026 journal article by Shizheng Song, published in Applied and Computational Engineering. OpenAlex counts 0 citations as of 16 November 2026. It is open access under a CC BY licence: the full text is free to read at Applied and Computational Engineering.
- Citations
- 0
- References
- 3
- Authors
- 1
Abstract
Optimization plays a fundamental role in machine learning, as most learning tasks can be formulated as the minimization of a loss function. From classical gradient descent to modern convex optimization theory, optimization algorithms have continuously evolved to meet the demands of large-scale data and high-dimensional models. This paper reviews the development of optimization methods in machine learning, focusing on gradient descent and its variants, stochastic optimization, and convex optimization theory. Through literature analysis, this study examines the theoretical foundations, convergence properties, and practical applications of these methods. Specifically, the research addresses three key questions: how gradient-based methods have evolved, what advantages convex optimization provides, and what challenges arise in non-convex optimization. The paper concludes that convex optimization offers strong theoretical guarantees, while gradient-based algorithms dominate practical large-scale machine learning tasks, especially in deep learning.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Song, S. (2026). A Survey of Optimization Methods in Machine Learning: From Gradient Descent to Convex Optimization. Applied and Computational Engineering, 231(1), 62–68. https://doi.org/10.54254/2755-2721/2026.ba33212
MLA 9
Song, Shizheng. “A Survey of Optimization Methods in Machine Learning: From Gradient Descent to Convex Optimization.” Applied and Computational Engineering, vol. 231, no. 1, 2026, pp. 62–68. https://doi.org/10.54254/2755-2721/2026.ba33212.
Chicago
Song, Shizheng. 2026. “A Survey of Optimization Methods in Machine Learning: From Gradient Descent to Convex Optimization.” Applied and Computational Engineering 231 (1): 62–68. https://doi.org/10.54254/2755-2721/2026.ba33212.
Harvard
Song, S. (2026) ‘A Survey of Optimization Methods in Machine Learning: From Gradient Descent to Convex Optimization’, Applied and Computational Engineering, 231(1), pp. 62–68. doi:10.54254/2755-2721/2026.ba33212.
IEEE
S. Song, “A Survey of Optimization Methods in Machine Learning: From Gradient Descent to Convex Optimization,” Applied and Computational Engineering, vol. 231, no. 1, pp. 62–68, 2026, doi: 10.54254/2755-2721/2026.ba33212.
BibTeX
@article{song2026survey,
title = {{A Survey of Optimization Methods in Machine Learning: From Gradient Descent to Convex Optimization}},
author = {Song, Shizheng},
journal = {Applied and Computational Engineering},
year = {2026},
volume = {231},
number = {1},
pages = {62--68},
doi = {10.54254/2755-2721/2026.ba33212},
url = {https://doi.org/10.54254/2755-2721/2026.ba33212}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- optimization methods
- gradient descent
- convex optimization
- stochastic optimization
- non-convex optimization
- convergence properties
- loss function optimization
- large-scale machine learning
Work with this paper
Most-cited papers in Computer Science
- ImageNet classification with deep convolutional neural networksAlex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton · Communications of the ACM · 2017 · 114,726 citations
- Long Short-Term MemorySepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 109,097 citations
- A Mathematical Theory of CommunicationClaude E. Shannon · Bell System Technical Journal · 1948 · 91,736 citations
- Deep learningYann LeCun, Yoshua Bengio, Geoffrey E. Hinton · Nature · 2015 · 90,226 citations
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksShaoqing Ren, Kaiming He, Ross Girshick and 1 more · IEEE Transactions on Pattern Analysis and Machine Intelligence · 2016 · 74,143 citations
Questions about this paper
How do I cite this paper in APA style?
Song, S. (2026). A Survey of Optimization Methods in Machine Learning: From Gradient Descent to Convex Optimization. Applied and Computational Engineering, 231(1), 62–68. https://doi.org/10.54254/2755-2721/2026.ba33212
How many times has this paper been cited?
0 times, according to OpenAlex on 16 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access: read the free full text at Applied and Computational Engineering.
Who wrote this paper?
Shizheng Song.
What is the DOI of this paper?
10.54254/2755-2721/2026.ba33212. The permanent link is https://doi.org/10.54254/2755-2721/2026.ba33212.
Bibliographic data from OpenAlex (CC0), last checked 16 November 2026. ResearchGuru does not host this paper; the links go to the publisher or the repository that does.