Journal article · 2026 · Open access

Towards Quantifying the Hessian Structure of Neural Networks

Zhaorui Dong, Yushun Zhang, Jianfeng Yao, Ruoyu Sun

IEEE Transactions on Information Theory, 1 · Published 1 January 2026 · DOI 10.1109/tit.2026.3750559

Open full textCite this paper

In brief

“Towards Quantifying the Hessian Structure of Neural Networks” is a 2026 journal article by Zhaorui Dong and 3 co-authors, published in IEEE Transactions on Information Theory. OpenAlex counts 18 citations as of 17 November 2026, which puts it in the top 4% of works published in 2026 by citations. It is open access under a CC BY licence: the full text is free to read at IEEE Transactions on Information Theory.

Citations
18
Most cited in
2026 (14)
Field-weighted impact
19.53
References
72
Authors
4

Abstract

Empirical studies reported that the Hessian matrix of neural networks (NNs) exhibits a near-block-diagonal structure, yet its theoretical foundation remains unclear. In this work, we reveal that the reported Hessian structure comes from a mixture of two forces: a “static force” rooted in the architecture design, and a “dynamic force” arisen from training. We then provide a rigorous theoretical analysis of “static force” at random initialization. We study linear models and 1-hidden-layer networks for classification tasks withCclasses. By leveraging random matrix theory, we compare the limit distributions of the diagonal and off-diagonal Hessian blocks and find that the block-diagonal structure arises asCbecomes large, where one block corresponds to one neuron. Our findings reveal thatCis one primary driver of the near-block-diagonal structure. These results may shed new light on the Hessian structure of large language models (LLMs), which typically operate with a largeCexceeding 104.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Dong, Z., Zhang, Y., Yao, J., & Sun, R. (2026). Towards Quantifying the Hessian Structure of Neural Networks. IEEE Transactions on Information Theory, 1. https://doi.org/10.1109/tit.2026.3750559

MLA 9

Dong, Zhaorui, et al. “Towards Quantifying the Hessian Structure of Neural Networks.” IEEE Transactions on Information Theory, 2026, p. 1. https://doi.org/10.1109/tit.2026.3750559.

Chicago

Dong, Zhaorui, Yushun Zhang, Jianfeng Yao, and Ruoyu Sun. 2026. “Towards Quantifying the Hessian Structure of Neural Networks.” IEEE Transactions on Information Theory: 1. https://doi.org/10.1109/tit.2026.3750559.

Harvard

Dong, Z. et al. (2026) ‘Towards Quantifying the Hessian Structure of Neural Networks’, IEEE Transactions on Information Theory, p. 1. doi:10.1109/tit.2026.3750559.

IEEE

Z. Dong, Y. Zhang, J. Yao, and R. Sun, “Towards Quantifying the Hessian Structure of Neural Networks,” IEEE Transactions on Information Theory, p. 1, 2026, doi: 10.1109/tit.2026.3750559.

BibTeX

@article{dong2026towards,
  title     = {{Towards Quantifying the Hessian Structure of Neural Networks}},
  author    = {Dong, Zhaorui and Zhang, Yushun and Yao, Jianfeng and Sun, Ruoyu},
  journal   = {IEEE Transactions on Information Theory},
  year      = {2026},
  pages     = {1},
  doi       = {10.1109/tit.2026.3750559},
  url       = {https://doi.org/10.1109/tit.2026.3750559}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques

  • Hessian matrix
  • artificial neural networks
  • block diagonal structure
  • random initialization
  • random matrix theory
  • number of classes
  • linear models
  • training dynamics

Work with this paper

Most-cited papers in Computer Science

All highly cited Computer Science papers

Questions about this paper

How do I cite this paper in APA style?

Dong, Z., Zhang, Y., Yao, J., & Sun, R. (2026). Towards Quantifying the Hessian Structure of Neural Networks. IEEE Transactions on Information Theory, 1. https://doi.org/10.1109/tit.2026.3750559

How many times has this paper been cited?

18 times, according to OpenAlex on 17 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at IEEE Transactions on Information Theory: sign in and open the full text from this page.

Who wrote this paper?

Zhaorui Dong, Yushun Zhang, Jianfeng Yao, Ruoyu Sun.

What is the DOI of this paper?

10.1109/tit.2026.3750559, permanently at https://doi.org/10.1109/tit.2026.3750559. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 17 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.