Journal article · 2025 · Open access

Distributional bias compromises leave-one-out cross-validation

George I. Austin, Itsik Pe’er, Tal Korem

Science Advances, 11(48), eadx6976 · Published 28 November 2025 · DOI 10.1126/sciadv.adx6976

Open PDFView at the publisherCite this paper

In brief

“Distributional bias compromises leave-one-out cross-validation” is a 2025 journal article by George I. Austin and 2 co-authors, published in Science Advances. OpenAlex counts 55 citations as of 17 November 2026, which puts it in the top 7% of works published in 2025 by citations. It is open access under a CC BY licence: the full text is free to read at Science Advances.

Citations
55
Most cited in
2026 (48)
Field-weighted impact
11.48
References
46
Authors
3

Abstract

Cross-validation is a common method for evaluating machine learning models. “Leave-one-out cross-validation,” in which each data instance is used to test a model trained on all other instances, is often used in data-scarce regimes. As common metrics such as the R 2 score cannot be calculated for a single prediction, predictions are commonly aggregated across folds for performance evaluation. Here, we prove that this creates “distributional bias”: a negative correlation between the average label of each training fold and the label of its corresponding test instance. As machine learning models tend to regress to the mean of their training data, this bias tends to negatively affect performance evaluation and hyperparameter optimization. We demonstrate that distributional bias exists across diverse tasks, models, and evaluation approaches, and can bias against stronger regularization. To address it, we developed a generalizable rebalanced cross-validation that is robust to distributional bias in both classification and regression, and demonstrates improved performance in simulations, machine learning benchmarks, and several published analyses.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Austin, G. I., Pe’er, I., & Korem, T. (2025). Distributional bias compromises leave-one-out cross-validation. Science Advances, 11(48), eadx6976. https://doi.org/10.1126/sciadv.adx6976

MLA 9

Austin, George I., et al. “Distributional bias compromises leave-one-out cross-validation.” Science Advances, vol. 11, no. 48, 2025, p. eadx6976. https://doi.org/10.1126/sciadv.adx6976.

Chicago

Austin, George I., Itsik Pe’er, and Tal Korem. 2025. “Distributional bias compromises leave-one-out cross-validation.” Science Advances 11 (48): eadx6976. https://doi.org/10.1126/sciadv.adx6976.

Harvard

Austin, G. I., Pe’er, I. and Korem, T. (2025) ‘Distributional bias compromises leave-one-out cross-validation’, Science Advances, 11(48), p. eadx6976. doi:10.1126/sciadv.adx6976.

IEEE

G. I. Austin, I. Pe’er, and T. Korem, “Distributional bias compromises leave-one-out cross-validation,” Science Advances, vol. 11, no. 48, p. eadx6976, 2025, doi: 10.1126/sciadv.adx6976.

BibTeX

@article{austin2025distributional,
  title     = {{Distributional bias compromises leave-one-out cross-validation}},
  author    = {Austin, George I. and Pe’er, Itsik and Korem, Tal},
  journal   = {Science Advances},
  year      = {2025},
  volume    = {11},
  number    = {48},
  pages     = {eadx6976},
  doi       = {10.1126/sciadv.adx6976},
  url       = {https://doi.org/10.1126/sciadv.adx6976}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Citations by year

YearCitationsShare of the best year
202422
202555
20264848

Its field-weighted citation impact is 11.48: it has been cited 11.48 times as often as the average work of the same field, type and year (1.00 is the world average).

Topic and keywords

Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification

  • leave-one-out cross-validation
  • distributional bias
  • performance evaluation
  • hyperparameter optimization
  • regression to the mean
  • data scarcity
  • regularization
  • classification

Work with this paper

Most-cited papers in Computer Science

All highly cited Computer Science papers

Questions about this paper

How do I cite this paper in APA style?

Austin, G. I., Pe’er, I., & Korem, T. (2025). Distributional bias compromises leave-one-out cross-validation. Science Advances, 11(48), eadx6976. https://doi.org/10.1126/sciadv.adx6976

How many times has this paper been cited?

55 times, according to OpenAlex on 17 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at Science Advances: sign in and open the PDF from this page.

Who wrote this paper?

George I. Austin, Itsik Pe’er, Tal Korem.

What is the DOI of this paper?

10.1126/sciadv.adx6976, permanently at https://doi.org/10.1126/sciadv.adx6976. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 17 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.