Journal article · 2026 · Open access
Towards Quantifying the Hessian Structure of Neural Networks
IEEE Transactions on Information Theory, 1 · Published 1 January 2026 · DOI 10.1109/tit.2026.3750559
In brief
“Towards Quantifying the Hessian Structure of Neural Networks” is a 2026 journal article by Zhaorui Dong and 3 co-authors, published in IEEE Transactions on Information Theory. OpenAlex counts 18 citations as of 17 November 2026, which puts it in the top 4% of works published in 2026 by citations. It is open access under a CC BY licence: the full text is free to read at IEEE Transactions on Information Theory.
- Citations
- 18
- Most cited in
- 2026 (14)
- Field-weighted impact
- 19.53
- References
- 72
- Authors
- 4
Abstract
Empirical studies reported that the Hessian matrix of neural networks (NNs) exhibits a near-block-diagonal structure, yet its theoretical foundation remains unclear. In this work, we reveal that the reported Hessian structure comes from a mixture of two forces: a “static force” rooted in the architecture design, and a “dynamic force” arisen from training. We then provide a rigorous theoretical analysis of “static force” at random initialization. We study linear models and 1-hidden-layer networks for classification tasks withCclasses. By leveraging random matrix theory, we compare the limit distributions of the diagonal and off-diagonal Hessian blocks and find that the block-diagonal structure arises asCbecomes large, where one block corresponds to one neuron. Our findings reveal thatCis one primary driver of the near-block-diagonal structure. These results may shed new light on the Hessian structure of large language models (LLMs), which typically operate with a largeCexceeding 104.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Dong, Z., Zhang, Y., Yao, J., & Sun, R. (2026). Towards Quantifying the Hessian Structure of Neural Networks. IEEE Transactions on Information Theory, 1. https://doi.org/10.1109/tit.2026.3750559
MLA 9
Dong, Zhaorui, et al. “Towards Quantifying the Hessian Structure of Neural Networks.” IEEE Transactions on Information Theory, 2026, p. 1. https://doi.org/10.1109/tit.2026.3750559.
Chicago
Dong, Zhaorui, Yushun Zhang, Jianfeng Yao, and Ruoyu Sun. 2026. “Towards Quantifying the Hessian Structure of Neural Networks.” IEEE Transactions on Information Theory: 1. https://doi.org/10.1109/tit.2026.3750559.
Harvard
Dong, Z. et al. (2026) ‘Towards Quantifying the Hessian Structure of Neural Networks’, IEEE Transactions on Information Theory, p. 1. doi:10.1109/tit.2026.3750559.
IEEE
Z. Dong, Y. Zhang, J. Yao, and R. Sun, “Towards Quantifying the Hessian Structure of Neural Networks,” IEEE Transactions on Information Theory, p. 1, 2026, doi: 10.1109/tit.2026.3750559.
BibTeX
@article{dong2026towards,
title = {{Towards Quantifying the Hessian Structure of Neural Networks}},
author = {Dong, Zhaorui and Zhang, Yushun and Yao, Jianfeng and Sun, Ruoyu},
journal = {IEEE Transactions on Information Theory},
year = {2026},
pages = {1},
doi = {10.1109/tit.2026.3750559},
url = {https://doi.org/10.1109/tit.2026.3750559}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Hessian matrix
- artificial neural networks
- block diagonal structure
- random initialization
- random matrix theory
- number of classes
- linear models
- training dynamics
Work with this paper
Most-cited papers in Computer Science
- ImageNet classification with deep convolutional neural networksAlex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton · Communications of the ACM · 2017 · 114,726 citations
- Long Short-Term MemorySepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 109,097 citations
- A Mathematical Theory of CommunicationClaude E. Shannon · Bell System Technical Journal · 1948 · 91,736 citations
- Deep learningYann LeCun, Yoshua Bengio, Geoffrey E. Hinton · Nature · 2015 · 90,226 citations
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksShaoqing Ren, Kaiming He, Ross Girshick and 1 more · IEEE Transactions on Pattern Analysis and Machine Intelligence · 2016 · 74,143 citations
Questions about this paper
How do I cite this paper in APA style?
Dong, Z., Zhang, Y., Yao, J., & Sun, R. (2026). Towards Quantifying the Hessian Structure of Neural Networks. IEEE Transactions on Information Theory, 1. https://doi.org/10.1109/tit.2026.3750559
How many times has this paper been cited?
18 times, according to OpenAlex on 17 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at IEEE Transactions on Information Theory: sign in and open the full text from this page.
Who wrote this paper?
Zhaorui Dong, Yushun Zhang, Jianfeng Yao, Ruoyu Sun.
What is the DOI of this paper?
10.1109/tit.2026.3750559, permanently at https://doi.org/10.1109/tit.2026.3750559. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 17 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.