Journal article · 2021 · Open access

Entropy and Relative Entropy From Information-Theoretic Principles

Gilad Gour, Marco Tomamichel

IEEE Transactions on Information Theory, 67(10), 6313–6327 · Published 14 May 2021 · DOI 10.1109/tit.2021.3078337

Free PDFView at the publisherCite this paper

In brief

“Entropy and Relative Entropy From Information-Theoretic Principles” is a 2021 journal article by Gilad Gour and Marco Tomamichel, published in IEEE Transactions on Information Theory. OpenAlex counts 40 citations as of 15 November 2026. It is open access: the full text is free to read at arXiv (Cornell University).

Citations
40
Most cited in
2025 (9)
Field-weighted impact
1.88
References
28
Authors
2

Abstract

We introduce an axiomatic approach to entropies and relative entropies that relies only on minimal information-theoretic axioms, namely monotonicity under mixing and data-processing as well as additivity for product distributions. We find that these axioms induce sufficient structure to establish continuity in the interior of the probability simplex and meaningful upper and lower bounds, e.g., we find that every relative entropy satisfying these axioms must lie between the Rényi divergences of order 0 and ∞. We further show simple conditions for positive definiteness of such relative entropies and a characterisation in terms of a variant of relative trumping. Our main result is a one-to-one correspondence between entropies and relative entropies.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Gour, G., & Tomamichel, M. (2021). Entropy and Relative Entropy From Information-Theoretic Principles. IEEE Transactions on Information Theory, 67(10), 6313–6327. https://doi.org/10.1109/tit.2021.3078337

MLA 9

Gour, Gilad, and Marco Tomamichel. “Entropy and Relative Entropy From Information-Theoretic Principles.” IEEE Transactions on Information Theory, vol. 67, no. 10, 2021, pp. 6313–6327. https://doi.org/10.1109/tit.2021.3078337.

Chicago

Gour, Gilad and Marco Tomamichel. 2021. “Entropy and Relative Entropy From Information-Theoretic Principles.” IEEE Transactions on Information Theory 67 (10): 6313–6327. https://doi.org/10.1109/tit.2021.3078337.

Harvard

Gour, G. and Tomamichel, M. (2021) ‘Entropy and Relative Entropy From Information-Theoretic Principles’, IEEE Transactions on Information Theory, 67(10), pp. 6313–6327. doi:10.1109/tit.2021.3078337.

IEEE

G. Gour and M. Tomamichel, “Entropy and Relative Entropy From Information-Theoretic Principles,” IEEE Transactions on Information Theory, vol. 67, no. 10, pp. 6313–6327, 2021, doi: 10.1109/tit.2021.3078337.

BibTeX

@article{gour2021entropy,
  title     = {{Entropy and Relative Entropy From Information-Theoretic Principles}},
  author    = {Gour, Gilad and Tomamichel, Marco},
  journal   = {IEEE Transactions on Information Theory},
  year      = {2021},
  volume    = {67},
  number    = {10},
  pages     = {6313--6327},
  doi       = {10.1109/tit.2021.3078337},
  url       = {https://doi.org/10.1109/tit.2021.3078337}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Citations by year

YearCitationsShare of the best year
202011
202133
202277
202344
202488
202599
202688

Its field-weighted citation impact is 1.88: it has been cited 1.88 times as often as the average work of the same field, type and year (1.00 is the world average).

Topic and keywords

Physical Sciences › Physics and Astronomy › Statistical and Nonlinear Physics › Statistical Mechanics and Entropy

  • entropy (information theory)
  • Kullback-Leibler divergence
  • Rényi divergence
  • data processing inequality
  • positive definiteness
  • probability simplex

Work with this paper

Most-cited papers in Physics and Astronomy

All highly cited Physics and Astronomy papers

Questions about this paper

How do I cite this paper in APA style?

Gour, G., & Tomamichel, M. (2021). Entropy and Relative Entropy From Information-Theoretic Principles. IEEE Transactions on Information Theory, 67(10), 6313–6327. https://doi.org/10.1109/tit.2021.3078337

How many times has this paper been cited?

40 times, according to OpenAlex on 15 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access: read the free full text at arXiv (Cornell University).

Who wrote this paper?

Gilad Gour, Marco Tomamichel.

What is the DOI of this paper?

10.1109/tit.2021.3078337. The permanent link is https://doi.org/10.1109/tit.2021.3078337.

Bibliographic data from OpenAlex (CC0), last checked 15 November 2026. ResearchGuru does not host this paper; the links go to the publisher or the repository that does.