Journal article · 2025 · Open access

Probabilistic Measures Afford Fair Comparisons of AIWP and NWP Model Output

Tilmann Gneiting, Tobias Biegert, Kristof Kraus, Eva‐Maria Walz, Alexander I. Jordan, Sebastian Lerch

Artificial Intelligence for the Earth Systems, 5(1) · Published 29 December 2025 · DOI 10.1175/aies-d-25-0054.1

Free PDFView at the publisherCite this paper

In brief

“Probabilistic Measures Afford Fair Comparisons of AIWP and NWP Model Output” is a 2025 journal article by Tilmann Gneiting and 5 co-authors, published in Artificial Intelligence for the Earth Systems. OpenAlex counts 7 citations as of 15 November 2026. It is open access: the full text is free to read at arXiv (Cornell University).

Citations
7
Most cited in
2026 (6)
Field-weighted impact
2.15
References
38
Authors
6

Abstract

We introduce a new measure for fair and meaningful comparisons of single-valued output from artificial-intelligence-based weather prediction (AIWP) and numerical weather prediction (NWP) models, called potential continuous ranked probability score (PCRPS). In a nutshell, we subject the deterministic backbone of physics-based and data-driven models post hoc to the same statistical postprocessing technique, namely, isotonic distributional regression (IDR). Then, we find PCRPS as the mean continuous ranked probability score (CRPS) of the postprocessed probabilistic forecasts. The nonnegative PCRPS measure quantifies potential predictive performance and is invariant under strictly increasing transformations of the model output. PCRPS attains its most desirable value of zero if, and only if, the weather outcome is a fixed, nondecreasing function of the model output. The PCRPS measure is recorded in the unit of the outcome, has an upper bound of one-half times the mean absolute difference between outcomes, and serves as a proxy for the mean CRPS of real-time, operational probabilistic products. When applied to WeatherBench 2 data, our approach demonstrates that the data-driven GraphCast model outperforms the leading physics-based European Centre for Medium-Range Weather Forecasts (ECMWF) high-resolution (HRES) model. Furthermore, the PCRPS measure for the HRES and GraphCast models aligns exceptionally well with the mean CRPS of the operational ECMWF ensemble and the GenCast ensemble, respectively. Across application domains, our approach affords comparisons of single-valued forecasts in settings where the prespecification of a loss function—which is the usual, and principally superior, procedure in forecast contests, administrative, and benchmark settings—places competitors on unequal footings. Significance Statement Until recently, weather forecasts have relied on physics-based numerical weather prediction (NWP) models. Since late 2022, the advent of artificial-intelligence-based weather prediction (AIWP) models that learn exclusively from data, without the laws of physics being invoked, challenges extant practice. While AIWP models might eventually supersede NWP models, meaningful comparisons of forecast quality remain difficult due to differences in training schemes, initial conditions, and postprocessing. Our paper introduces a new measure, called potential continuous ranked probability score (PCRPS), that accounts for these differences and, therefore, affords fair comparisons. Using benchmark data, we confirm the superiority of AIWP models for gridpoint-based global forecasts of surface weather variables at lead times up to 10 days.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Gneiting, T., Biegert, T., Kraus, K., Walz, E., Jordan, A. I., & Lerch, S. (2025). Probabilistic Measures Afford Fair Comparisons of AIWP and NWP Model Output. Artificial Intelligence for the Earth Systems, 5(1). https://doi.org/10.1175/aies-d-25-0054.1

MLA 9

Gneiting, Tilmann, et al. “Probabilistic Measures Afford Fair Comparisons of AIWP and NWP Model Output.” Artificial Intelligence for the Earth Systems, vol. 5, no. 1, 2025. https://doi.org/10.1175/aies-d-25-0054.1.

Chicago

Gneiting, Tilmann, Tobias Biegert, Kristof Kraus, Eva‐Maria Walz, Alexander I. Jordan, and Sebastian Lerch. 2025. “Probabilistic Measures Afford Fair Comparisons of AIWP and NWP Model Output.” Artificial Intelligence for the Earth Systems 5 (1). https://doi.org/10.1175/aies-d-25-0054.1.

Harvard

Gneiting, T. et al. (2025) ‘Probabilistic Measures Afford Fair Comparisons of AIWP and NWP Model Output’, Artificial Intelligence for the Earth Systems, 5(1). doi:10.1175/aies-d-25-0054.1.

IEEE

T. Gneiting, T. Biegert, K. Kraus, E. Walz, A. I. Jordan, and S. Lerch, “Probabilistic Measures Afford Fair Comparisons of AIWP and NWP Model Output,” Artificial Intelligence for the Earth Systems, vol. 5, no. 1, 2025, doi: 10.1175/aies-d-25-0054.1.

BibTeX

@article{gneiting2025probabilistic,
  title     = {{Probabilistic Measures Afford Fair Comparisons of AIWP and NWP Model Output}},
  author    = {Gneiting, Tilmann and Biegert, Tobias and Kraus, Kristof and Walz, Eva‐Maria and Jordan, Alexander I. and Lerch, Sebastian},
  journal   = {Artificial Intelligence for the Earth Systems},
  year      = {2025},
  volume    = {5},
  number    = {1},
  doi       = {10.1175/aies-d-25-0054.1},
  url       = {https://doi.org/10.1175/aies-d-25-0054.1}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Physical Sciences › Earth and Planetary Sciences › Atmospheric Science › Meteorological Phenomena and Simulations

  • numerical weather prediction
  • continuous ranked probability score
  • probabilistic forecast evaluation
  • GraphCast
  • forecast benchmarking

Work with this paper

Most-cited papers in Earth and Planetary Sciences

All highly cited Earth and Planetary Sciences papers

Questions about this paper

How do I cite this paper in APA style?

Gneiting, T., Biegert, T., Kraus, K., Walz, E., Jordan, A. I., & Lerch, S. (2025). Probabilistic Measures Afford Fair Comparisons of AIWP and NWP Model Output. Artificial Intelligence for the Earth Systems, 5(1). https://doi.org/10.1175/aies-d-25-0054.1

How many times has this paper been cited?

7 times, according to OpenAlex on 15 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access: read the free full text at arXiv (Cornell University).

Who wrote this paper?

Tilmann Gneiting, Tobias Biegert, Kristof Kraus, Eva‐Maria Walz, Alexander I. Jordan, Sebastian Lerch.

What is the DOI of this paper?

10.1175/aies-d-25-0054.1. The permanent link is https://doi.org/10.1175/aies-d-25-0054.1.

Bibliographic data from OpenAlex (CC0), last checked 15 November 2026. ResearchGuru does not host this paper; the links go to the publisher or the repository that does.