Preprint · 2026 · Open access
Can We Trust Evaluations of AI Weather Forecasts for Extremes? How Evaluation Protocols Shape Model Rankings
arXiv (Cornell University) · Published 22 October 2026 · DOI 10.48550/arxiv.2610.31237
In brief
“Can We Trust Evaluations of AI Weather Forecasts for Extremes? How Evaluation Protocols Shape Model Rankings” is a 2026 preprint by Tobias Biegert and 3 co-authors, published in arXiv (Cornell University). OpenAlex counts 0 citations as of 15 November 2026. It is open access: the full text is free to read at arXiv (Cornell University).
- Citations
- 0
- References
- 26
- Authors
- 4
Abstract
AI weather models perform competitively with traditional physics-based models at a fraction of the computational cost, and they are thus already used operationally. However, studies reach conflicting conclusions about their ability to forecast extreme weather events, and comparisons across studies are complicated by differing evaluation protocols. We discuss the challenges that arise when comparing AI and physics-based weather forecasts for extreme events. Within a common experimental design, we find that model rankings are highly sensitive to the choice of evaluation metrics: the considered physics-based model tends to perform best when evaluation is restricted to observed extreme events, whereas AI models generally perform better when evaluated using weighted scoring functions. A contingency table analysis shows that AI models underpredict the frequency of extremes, so that their scores are generally dominated by misses, while the physics-based model forecasts extremes more frequently and its scores are dominated by false alarms. Different evaluation metrics penalise these errors differently. Further score decompositions confirm that the score differences arise primarily from miscalibration rather than discrimination ability. This sheds light on the contrasting results in the literature, and highlights the importance of the evaluation metrics used to evaluate AI models.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Biegert, T., Allen, S., Zhang, Z., & Lerch, S. (2026). Can We Trust Evaluations of AI Weather Forecasts for Extremes? How Evaluation Protocols Shape Model Rankings. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2610.31237
MLA 9
Biegert, Tobias, et al. “Can We Trust Evaluations of AI Weather Forecasts for Extremes? How Evaluation Protocols Shape Model Rankings.” arXiv (Cornell University), 2026. https://doi.org/10.48550/arxiv.2610.31237.
Chicago
Biegert, Tobias, Sam Allen, Zhongwei Zhang, and Sebastian Lerch. 2026. “Can We Trust Evaluations of AI Weather Forecasts for Extremes? How Evaluation Protocols Shape Model Rankings.” arXiv (Cornell University). https://doi.org/10.48550/arxiv.2610.31237.
Harvard
Biegert, T. et al. (2026) ‘Can We Trust Evaluations of AI Weather Forecasts for Extremes? How Evaluation Protocols Shape Model Rankings’, arXiv (Cornell University). doi:10.48550/arxiv.2610.31237.
IEEE
T. Biegert, S. Allen, Z. Zhang, and S. Lerch, “Can We Trust Evaluations of AI Weather Forecasts for Extremes? How Evaluation Protocols Shape Model Rankings,” arXiv (Cornell University), 2026, doi: 10.48550/arxiv.2610.31237.
BibTeX
@misc{biegert2026trust,
title = {{Can We Trust Evaluations of AI Weather Forecasts for Extremes? How Evaluation Protocols Shape Model Rankings}},
author = {Biegert, Tobias and Allen, Sam and Zhang, Zhongwei and Lerch, Sebastian},
howpublished = {arXiv (Cornell University)},
year = {2026},
doi = {10.48550/arxiv.2610.31237},
url = {https://doi.org/10.48550/arxiv.2610.31237}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Physical Sciences › Earth and Planetary Sciences › Atmospheric Science › Meteorological Phenomena and Simulations
- extreme weather events
- forecast evaluation metrics
- model ranking
- contingency table analysis
- calibration error
- discrimination (statistics)
- operational forecasting
Work with this paper
Most-cited papers in Earth and Planetary Sciences
- The ERA5 global reanalysisHans Hersbach, Bill Bell, Paul Berrisford and 40 more · Quarterly Journal of the Royal Meteorological Society · 2020 · 35,424 citations
- The NCEP/NCAR 40-Year Reanalysis ProjectEugenia Kalnay, Masao Kanamitsu, Robert E. Kistler and 19 more · Bulletin of the American Meteorological Society · 1996 · 28,486 citations
- Chemical and isotopic systematics of oceanic basalts: implications for mantle composition and processesShen‐Su Sun, W. F. McDonough · Geological Society London Special Publications · 1989 · 26,603 citations
- The ERA‐Interim reanalysis: configuration and performance of the data assimilation systemDick P. Dee, Sakari M. Uppala, A. J. Simmons and 33 more · Quarterly Journal of the Royal Meteorological Society · 2011 · 25,245 citations
- PAST: PALEONTOLOGICAL STATISTICAL SOFTWARE PACKAGE FOR EDUCATION AND DATA ANALYSISØyvind Hammer, David A. T. Harper, Paul D. Ryan · 2001 · 17,871 citations
Questions about this paper
How do I cite this paper in APA style?
Biegert, T., Allen, S., Zhang, Z., & Lerch, S. (2026). Can We Trust Evaluations of AI Weather Forecasts for Extremes? How Evaluation Protocols Shape Model Rankings. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2610.31237
How many times has this paper been cited?
0 times, according to OpenAlex on 15 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access: read the free full text at arXiv (Cornell University).
Who wrote this paper?
Tobias Biegert, Sam Allen, Zhongwei Zhang, Sebastian Lerch.
What is the DOI of this paper?
10.48550/arxiv.2610.31237. The permanent link is https://doi.org/10.48550/arxiv.2610.31237.
Bibliographic data from OpenAlex (CC0), last checked 15 November 2026. ResearchGuru does not host this paper; the links go to the publisher or the repository that does.