Journal article · 2025 · Open access
Probabilistic Measures Afford Fair Comparisons of AIWP and NWP Model Output
Artificial Intelligence for the Earth Systems, 5(1) · Published 29 December 2025 · DOI 10.1175/aies-d-25-0054.1
In brief
“Probabilistic Measures Afford Fair Comparisons of AIWP and NWP Model Output” is a 2025 journal article by Tilmann Gneiting and 5 co-authors, published in Artificial Intelligence for the Earth Systems. OpenAlex counts 7 citations as of 15 November 2026. It is open access: the full text is free to read at arXiv (Cornell University).
- Citations
- 7
- Most cited in
- 2026 (6)
- Field-weighted impact
- 2.15
- References
- 38
- Authors
- 6
Abstract
We introduce a new measure for fair and meaningful comparisons of single-valued output from artificial-intelligence-based weather prediction (AIWP) and numerical weather prediction (NWP) models, called potential continuous ranked probability score (PCRPS). In a nutshell, we subject the deterministic backbone of physics-based and data-driven models post hoc to the same statistical postprocessing technique, namely, isotonic distributional regression (IDR). Then, we find PCRPS as the mean continuous ranked probability score (CRPS) of the postprocessed probabilistic forecasts. The nonnegative PCRPS measure quantifies potential predictive performance and is invariant under strictly increasing transformations of the model output. PCRPS attains its most desirable value of zero if, and only if, the weather outcome is a fixed, nondecreasing function of the model output. The PCRPS measure is recorded in the unit of the outcome, has an upper bound of one-half times the mean absolute difference between outcomes, and serves as a proxy for the mean CRPS of real-time, operational probabilistic products. When applied to WeatherBench 2 data, our approach demonstrates that the data-driven GraphCast model outperforms the leading physics-based European Centre for Medium-Range Weather Forecasts (ECMWF) high-resolution (HRES) model. Furthermore, the PCRPS measure for the HRES and GraphCast models aligns exceptionally well with the mean CRPS of the operational ECMWF ensemble and the GenCast ensemble, respectively. Across application domains, our approach affords comparisons of single-valued forecasts in settings where the prespecification of a loss function—which is the usual, and principally superior, procedure in forecast contests, administrative, and benchmark settings—places competitors on unequal footings. Significance Statement Until recently, weather forecasts have relied on physics-based numerical weather prediction (NWP) models. Since late 2022, the advent of artificial-intelligence-based weather prediction (AIWP) models that learn exclusively from data, without the laws of physics being invoked, challenges extant practice. While AIWP models might eventually supersede NWP models, meaningful comparisons of forecast quality remain difficult due to differences in training schemes, initial conditions, and postprocessing. Our paper introduces a new measure, called potential continuous ranked probability score (PCRPS), that accounts for these differences and, therefore, affords fair comparisons. Using benchmark data, we confirm the superiority of AIWP models for gridpoint-based global forecasts of surface weather variables at lead times up to 10 days.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Gneiting, T., Biegert, T., Kraus, K., Walz, E., Jordan, A. I., & Lerch, S. (2025). Probabilistic Measures Afford Fair Comparisons of AIWP and NWP Model Output. Artificial Intelligence for the Earth Systems, 5(1). https://doi.org/10.1175/aies-d-25-0054.1
MLA 9
Gneiting, Tilmann, et al. “Probabilistic Measures Afford Fair Comparisons of AIWP and NWP Model Output.” Artificial Intelligence for the Earth Systems, vol. 5, no. 1, 2025. https://doi.org/10.1175/aies-d-25-0054.1.
Chicago
Gneiting, Tilmann, Tobias Biegert, Kristof Kraus, Eva‐Maria Walz, Alexander I. Jordan, and Sebastian Lerch. 2025. “Probabilistic Measures Afford Fair Comparisons of AIWP and NWP Model Output.” Artificial Intelligence for the Earth Systems 5 (1). https://doi.org/10.1175/aies-d-25-0054.1.
Harvard
Gneiting, T. et al. (2025) ‘Probabilistic Measures Afford Fair Comparisons of AIWP and NWP Model Output’, Artificial Intelligence for the Earth Systems, 5(1). doi:10.1175/aies-d-25-0054.1.
IEEE
T. Gneiting, T. Biegert, K. Kraus, E. Walz, A. I. Jordan, and S. Lerch, “Probabilistic Measures Afford Fair Comparisons of AIWP and NWP Model Output,” Artificial Intelligence for the Earth Systems, vol. 5, no. 1, 2025, doi: 10.1175/aies-d-25-0054.1.
BibTeX
@article{gneiting2025probabilistic,
title = {{Probabilistic Measures Afford Fair Comparisons of AIWP and NWP Model Output}},
author = {Gneiting, Tilmann and Biegert, Tobias and Kraus, Kristof and Walz, Eva‐Maria and Jordan, Alexander I. and Lerch, Sebastian},
journal = {Artificial Intelligence for the Earth Systems},
year = {2025},
volume = {5},
number = {1},
doi = {10.1175/aies-d-25-0054.1},
url = {https://doi.org/10.1175/aies-d-25-0054.1}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Physical Sciences › Earth and Planetary Sciences › Atmospheric Science › Meteorological Phenomena and Simulations
- numerical weather prediction
- continuous ranked probability score
- probabilistic forecast evaluation
- GraphCast
- forecast benchmarking
Work with this paper
Most-cited papers in Earth and Planetary Sciences
- The ERA5 global reanalysisHans Hersbach, Bill Bell, Paul Berrisford and 40 more · Quarterly Journal of the Royal Meteorological Society · 2020 · 35,424 citations
- The NCEP/NCAR 40-Year Reanalysis ProjectEugenia Kalnay, Masao Kanamitsu, Robert E. Kistler and 19 more · Bulletin of the American Meteorological Society · 1996 · 28,486 citations
- Chemical and isotopic systematics of oceanic basalts: implications for mantle composition and processesShen‐Su Sun, W. F. McDonough · Geological Society London Special Publications · 1989 · 26,603 citations
- The ERA‐Interim reanalysis: configuration and performance of the data assimilation systemDick P. Dee, Sakari M. Uppala, A. J. Simmons and 33 more · Quarterly Journal of the Royal Meteorological Society · 2011 · 25,245 citations
- PAST: PALEONTOLOGICAL STATISTICAL SOFTWARE PACKAGE FOR EDUCATION AND DATA ANALYSISØyvind Hammer, David A. T. Harper, Paul D. Ryan · 2001 · 17,871 citations
Questions about this paper
How do I cite this paper in APA style?
Gneiting, T., Biegert, T., Kraus, K., Walz, E., Jordan, A. I., & Lerch, S. (2025). Probabilistic Measures Afford Fair Comparisons of AIWP and NWP Model Output. Artificial Intelligence for the Earth Systems, 5(1). https://doi.org/10.1175/aies-d-25-0054.1
How many times has this paper been cited?
7 times, according to OpenAlex on 15 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access: read the free full text at arXiv (Cornell University).
Who wrote this paper?
Tilmann Gneiting, Tobias Biegert, Kristof Kraus, Eva‐Maria Walz, Alexander I. Jordan, Sebastian Lerch.
What is the DOI of this paper?
10.1175/aies-d-25-0054.1. The permanent link is https://doi.org/10.1175/aies-d-25-0054.1.
Bibliographic data from OpenAlex (CC0), last checked 15 November 2026. ResearchGuru does not host this paper; the links go to the publisher or the repository that does.