Journal article · 2025 · Open access
Distributional bias compromises leave-one-out cross-validation
Science Advances, 11(48), eadx6976 · Published 28 November 2025 · DOI 10.1126/sciadv.adx6976
In brief
“Distributional bias compromises leave-one-out cross-validation” is a 2025 journal article by George I. Austin and 2 co-authors, published in Science Advances. OpenAlex counts 55 citations as of 17 November 2026, which puts it in the top 7% of works published in 2025 by citations. It is open access under a CC BY licence: the full text is free to read at Science Advances.
- Citations
- 55
- Most cited in
- 2026 (48)
- Field-weighted impact
- 11.48
- References
- 46
- Authors
- 3
Abstract
Cross-validation is a common method for evaluating machine learning models. “Leave-one-out cross-validation,” in which each data instance is used to test a model trained on all other instances, is often used in data-scarce regimes. As common metrics such as the R 2 score cannot be calculated for a single prediction, predictions are commonly aggregated across folds for performance evaluation. Here, we prove that this creates “distributional bias”: a negative correlation between the average label of each training fold and the label of its corresponding test instance. As machine learning models tend to regress to the mean of their training data, this bias tends to negatively affect performance evaluation and hyperparameter optimization. We demonstrate that distributional bias exists across diverse tasks, models, and evaluation approaches, and can bias against stronger regularization. To address it, we developed a generalizable rebalanced cross-validation that is robust to distributional bias in both classification and regression, and demonstrates improved performance in simulations, machine learning benchmarks, and several published analyses.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Austin, G. I., Pe’er, I., & Korem, T. (2025). Distributional bias compromises leave-one-out cross-validation. Science Advances, 11(48), eadx6976. https://doi.org/10.1126/sciadv.adx6976
MLA 9
Austin, George I., et al. “Distributional bias compromises leave-one-out cross-validation.” Science Advances, vol. 11, no. 48, 2025, p. eadx6976. https://doi.org/10.1126/sciadv.adx6976.
Chicago
Austin, George I., Itsik Pe’er, and Tal Korem. 2025. “Distributional bias compromises leave-one-out cross-validation.” Science Advances 11 (48): eadx6976. https://doi.org/10.1126/sciadv.adx6976.
Harvard
Austin, G. I., Pe’er, I. and Korem, T. (2025) ‘Distributional bias compromises leave-one-out cross-validation’, Science Advances, 11(48), p. eadx6976. doi:10.1126/sciadv.adx6976.
IEEE
G. I. Austin, I. Pe’er, and T. Korem, “Distributional bias compromises leave-one-out cross-validation,” Science Advances, vol. 11, no. 48, p. eadx6976, 2025, doi: 10.1126/sciadv.adx6976.
BibTeX
@article{austin2025distributional,
title = {{Distributional bias compromises leave-one-out cross-validation}},
author = {Austin, George I. and Pe’er, Itsik and Korem, Tal},
journal = {Science Advances},
year = {2025},
volume = {11},
number = {48},
pages = {eadx6976},
doi = {10.1126/sciadv.adx6976},
url = {https://doi.org/10.1126/sciadv.adx6976}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Citations by year
| Year | Citations | Share of the best year |
|---|---|---|
| 2024 | 2 | |
| 2025 | 5 | |
| 2026 | 48 |
Its field-weighted citation impact is 11.48: it has been cited 11.48 times as often as the average work of the same field, type and year (1.00 is the world average).
Topic and keywords
Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- leave-one-out cross-validation
- distributional bias
- performance evaluation
- hyperparameter optimization
- regression to the mean
- data scarcity
- regularization
- classification
Work with this paper
Most-cited papers in Computer Science
- ImageNet classification with deep convolutional neural networksAlex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton · Communications of the ACM · 2017 · 114,726 citations
- Long Short-Term MemorySepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 109,097 citations
- A Mathematical Theory of CommunicationClaude E. Shannon · Bell System Technical Journal · 1948 · 91,736 citations
- Deep learningYann LeCun, Yoshua Bengio, Geoffrey E. Hinton · Nature · 2015 · 90,226 citations
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksShaoqing Ren, Kaiming He, Ross Girshick and 1 more · IEEE Transactions on Pattern Analysis and Machine Intelligence · 2016 · 74,143 citations
Questions about this paper
How do I cite this paper in APA style?
Austin, G. I., Pe’er, I., & Korem, T. (2025). Distributional bias compromises leave-one-out cross-validation. Science Advances, 11(48), eadx6976. https://doi.org/10.1126/sciadv.adx6976
How many times has this paper been cited?
55 times, according to OpenAlex on 17 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at Science Advances: sign in and open the PDF from this page.
Who wrote this paper?
George I. Austin, Itsik Pe’er, Tal Korem.
What is the DOI of this paper?
10.1126/sciadv.adx6976, permanently at https://doi.org/10.1126/sciadv.adx6976. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 17 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.