Journal article · 2023 · Open access
Leave-one-out cross-validation, penalization, and differential bias of some prediction model performance measures—a simulation study
Diagnostic and Prognostic Research, 7(1), 9 · Published 2 May 2023 · DOI 10.1186/s41512-023-00146-0
In brief
“Leave-one-out cross-validation, penalization, and differential bias of some prediction model performance measures—a simulation study” is a 2023 journal article by Angelika Geroldinger and 3 co-authors, published in Diagnostic and Prognostic Research. OpenAlex counts 112 citations as of 22 November 2026, which puts it in the top 3% of works published in 2023 by citations. It is open access under a CC BY licence: the full text is free to read at Diagnostic and Prognostic Research.
- Citations
- 112
- Most cited in
- 2026 (52)
- Field-weighted impact
- 12.63
- References
- 30
- Authors
- 4
Abstract
BACKGROUND: The performance of models for binary outcomes can be described by measures such as the concordance statistic (c-statistic, area under the curve), the discrimination slope, or the Brier score. At internal validation, data resampling techniques, e.g., cross-validation, are frequently employed to correct for optimism in these model performance criteria. Especially with small samples or rare events, leave-one-out cross-validation is a popular choice. METHODS: Using simulations and a real data example, we compared the effect of different resampling techniques on the estimation of c-statistics, discrimination slopes, and Brier scores for three estimators of logistic regression models, including the maximum likelihood and two maximum penalized likelihood estimators. RESULTS: Our simulation study confirms earlier studies reporting that leave-one-out cross-validated c-statistics can be strongly biased towards zero. In addition, our study reveals that this bias is even more pronounced for model estimators shrinking estimated probabilities towards the observed event fraction, such as ridge regression. Leave-one-out cross-validation also provided pessimistic estimates of the discrimination slope but nearly unbiased estimates of the Brier score. CONCLUSIONS: We recommend to use leave-pair-out cross-validation, fivefold cross-validation with repetitions, the enhanced or the .632+ bootstrap to estimate c-statistics, and leave-pair-out or fivefold cross-validation to estimate discrimination slopes.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Geroldinger, A., Lusa, L., Nold, M., & Heinze, G. (2023). Leave-one-out cross-validation, penalization, and differential bias of some prediction model performance measures—a simulation study. Diagnostic and Prognostic Research, 7(1), 9. https://doi.org/10.1186/s41512-023-00146-0
MLA 9
Geroldinger, Angelika, et al. “Leave-one-out cross-validation, penalization, and differential bias of some prediction model performance measures—a simulation study.” Diagnostic and Prognostic Research, vol. 7, no. 1, 2023, p. 9. https://doi.org/10.1186/s41512-023-00146-0.
Chicago
Geroldinger, Angelika, Lara Lusa, Mariana Nold, and Georg Heinze. 2023. “Leave-one-out cross-validation, penalization, and differential bias of some prediction model performance measures—a simulation study.” Diagnostic and Prognostic Research 7 (1): 9. https://doi.org/10.1186/s41512-023-00146-0.
Harvard
Geroldinger, A. et al. (2023) ‘Leave-one-out cross-validation, penalization, and differential bias of some prediction model performance measures—a simulation study’, Diagnostic and Prognostic Research, 7(1), p. 9. doi:10.1186/s41512-023-00146-0.
IEEE
A. Geroldinger, L. Lusa, M. Nold, and G. Heinze, “Leave-one-out cross-validation, penalization, and differential bias of some prediction model performance measures—a simulation study,” Diagnostic and Prognostic Research, vol. 7, no. 1, p. 9, 2023, doi: 10.1186/s41512-023-00146-0.
BibTeX
@article{geroldinger2023leave,
title = {{Leave-one-out cross-validation, penalization, and differential bias of some prediction model performance measures—a simulation study}},
author = {Geroldinger, Angelika and Lusa, Lara and Nold, Mariana and Heinze, Georg},
journal = {Diagnostic and Prognostic Research},
year = {2023},
volume = {7},
number = {1},
pages = {9},
doi = {10.1186/s41512-023-00146-0},
url = {https://doi.org/10.1186/s41512-023-00146-0}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Citations by year
| Year | Citations | Share of the best year |
|---|---|---|
| 2023 | 6 | |
| 2024 | 15 | |
| 2025 | 39 | |
| 2026 | 52 |
Its field-weighted citation impact is 12.63: it has been cited 12.63 times as often as the average work of the same field, type and year (1.00 is the world average).
Topic and keywords
Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- leave-one-out cross-validation
- model performance measures
- binary outcomes
- Brier score
- optimism bias
- penalized likelihood
- ridge regression
- logistic regression
Work with this paper
Most-cited papers in Mathematics
- Random ForestsLeo Breiman · Machine Learning · 2001 · 142,830 citations
- Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple TestingYoav Benjamini, Yosef Hochberg · Journal of the Royal Statistical Society Series B (Statistical Methodology) · 1995 · 113,923 citations
- Fitting Linear Mixed-Effects Models Using lme4Douglas M. Bates, Martin Maechler, Benjamin M. Bolker and 1 more · Journal of Statistical Software · 2015 · 100,328 citations
- The moderator-mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations.Reuben M. Baron, David A. Kenny · Journal of Personality and Social Psychology · 1986 · 95,866 citations
- Using multivariate statisticsBarbara Gerson Tabachnick, Linda S. Fidell · 1983 · 79,867 citations
Questions about this paper
How do I cite this paper in APA style?
Geroldinger, A., Lusa, L., Nold, M., & Heinze, G. (2023). Leave-one-out cross-validation, penalization, and differential bias of some prediction model performance measures—a simulation study. Diagnostic and Prognostic Research, 7(1), 9. https://doi.org/10.1186/s41512-023-00146-0
How many times has this paper been cited?
112 times, according to OpenAlex on 22 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at Diagnostic and Prognostic Research: sign in and open the PDF from this page.
Who wrote this paper?
Angelika Geroldinger, Lara Lusa, Mariana Nold, Georg Heinze.
What is the DOI of this paper?
10.1186/s41512-023-00146-0, permanently at https://doi.org/10.1186/s41512-023-00146-0. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 22 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.