Journal article · 2021 · Open access
Internal-external cross-validation helped to evaluate the generalizability of prediction models in large clustered datasets
Journal of Clinical Epidemiology, 137, 83–91 · Published 6 April 2021 · DOI 10.1016/j.jclinepi.2021.03.025
In brief
“Internal-external cross-validation helped to evaluate the generalizability of prediction models in large clustered datasets” is a 2021 journal article by Toshihiko Takada and 7 co-authors, published in Journal of Clinical Epidemiology. OpenAlex counts 94 citations as of 16 November 2026, which puts it in the top 5% of works published in 2021 by citations. It is open access under a CC BY licence: the full text is free to read at Journal of Clinical Epidemiology.
- Citations
- 94
- Most cited in
- 2026 (43)
- Field-weighted impact
- 5.96
- References
- 31
- Authors
- 8
Abstract
OBJECTIVE: To illustrate how to evaluate the need of complex strategies for developing generalizable prediction models in large clustered datasets. STUDY DESIGN AND SETTING: We developed eight Cox regression models to estimate the risk of heart failure using a large population-level dataset. These models differed in the number of predictors, the functional form of the predictor effects (non-linear effects and interaction) and the estimation method (maximum likelihood and penalization). Internal-external cross-validation was used to evaluate the models' generalizability across the included general practices. RESULTS: Among 871,687 individuals from 225 general practices, 43,987 (5.5%) developed heart failure during a median follow-up time of 5.8 years. For discrimination, the simplest prediction model yielded a good concordance statistic, which was not much improved by adopting complex strategies. Between-practice heterogeneity in discrimination was similar in all models. For calibration, the simplest model performed satisfactorily. Although accounting for non-linear effects and interaction slightly improved the calibration slope, it also led to more heterogeneity in the observed/expected ratio. Similar results were found in a second case study involving patients with stroke. CONCLUSION: In large clustered datasets, prediction model studies may adopt internal-external cross-validation to evaluate the generalizability of competing models, and to identify promising modelling strategies.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Takada, T., Nijman, S. W. J., Denaxas, S., Snell, K. I. E., Uijl, A., Nguyen, T., Asselbergs, F. W., & Debray, T. P. A. (2021). Internal-external cross-validation helped to evaluate the generalizability of prediction models in large clustered datasets. Journal of Clinical Epidemiology, 137, 83–91. https://doi.org/10.1016/j.jclinepi.2021.03.025
MLA 9
Takada, Toshihiko, et al. “Internal-external cross-validation helped to evaluate the generalizability of prediction models in large clustered datasets.” Journal of Clinical Epidemiology, vol. 137, 2021, pp. 83–91. https://doi.org/10.1016/j.jclinepi.2021.03.025.
Chicago
Takada, Toshihiko, Steven W J Nijman, Spiros Denaxas, Kym I E Snell, Alicia Uijl, Tri‐Long Nguyen, Folkert Wouter Asselbergs, and Thomas P. A. Debray. 2021. “Internal-external cross-validation helped to evaluate the generalizability of prediction models in large clustered datasets.” Journal of Clinical Epidemiology 137: 83–91. https://doi.org/10.1016/j.jclinepi.2021.03.025.
Harvard
Takada, T. et al. (2021) ‘Internal-external cross-validation helped to evaluate the generalizability of prediction models in large clustered datasets’, Journal of Clinical Epidemiology, 137, pp. 83–91. doi:10.1016/j.jclinepi.2021.03.025.
IEEE
T. Takada et al., “Internal-external cross-validation helped to evaluate the generalizability of prediction models in large clustered datasets,” Journal of Clinical Epidemiology, vol. 137, pp. 83–91, 2021, doi: 10.1016/j.jclinepi.2021.03.025.
BibTeX
@article{takada2021internal,
title = {{Internal-external cross-validation helped to evaluate the generalizability of prediction models in large clustered datasets}},
author = {Takada, Toshihiko and Nijman, Steven W J and Denaxas, Spiros and Snell, Kym I E and Uijl, Alicia and Nguyen, Tri‐Long and Asselbergs, Folkert Wouter and Debray, Thomas P. A.},
journal = {Journal of Clinical Epidemiology},
year = {2021},
volume = {137},
pages = {83--91},
doi = {10.1016/j.jclinepi.2021.03.025},
url = {https://doi.org/10.1016/j.jclinepi.2021.03.025}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Citations by year
| Year | Citations | Share of the best year |
|---|---|---|
| 2021 | 3 | |
| 2022 | 12 | |
| 2023 | 8 | |
| 2024 | 14 | |
| 2025 | 14 | |
| 2026 | 43 |
Its field-weighted citation impact is 5.96: it has been cited 5.96 times as often as the average work of the same field, type and year (1.00 is the world average).
Topic and keywords
Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- clustered data
- heart failure prediction
- Cox proportional hazards model
- interaction terms
- penalized regression
- model discrimination
- calibration
- stroke risk prediction
Work with this paper
Most-cited papers in Mathematics
- Random ForestsLeo Breiman · Machine Learning · 2001 · 142,830 citations
- Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple TestingYoav Benjamini, Yosef Hochberg · Journal of the Royal Statistical Society Series B (Statistical Methodology) · 1995 · 113,923 citations
- Fitting Linear Mixed-Effects Models Using lme4Douglas M. Bates, Martin Maechler, Benjamin M. Bolker and 1 more · Journal of Statistical Software · 2015 · 100,328 citations
- The moderator-mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations.Reuben M. Baron, David A. Kenny · Journal of Personality and Social Psychology · 1986 · 95,866 citations
- Using multivariate statisticsBarbara Gerson Tabachnick, Linda S. Fidell · 1983 · 79,867 citations
Questions about this paper
How do I cite this paper in APA style?
Takada, T., Nijman, S. W. J., Denaxas, S., Snell, K. I. E., Uijl, A., Nguyen, T., Asselbergs, F. W., & Debray, T. P. A. (2021). Internal-external cross-validation helped to evaluate the generalizability of prediction models in large clustered datasets. Journal of Clinical Epidemiology, 137, 83–91. https://doi.org/10.1016/j.jclinepi.2021.03.025
How many times has this paper been cited?
94 times, according to OpenAlex on 16 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access: read the free full text at Journal of Clinical Epidemiology.
Who wrote this paper?
Toshihiko Takada, Steven W J Nijman, Spiros Denaxas, Kym I E Snell, Alicia Uijl, Tri‐Long Nguyen, Folkert Wouter Asselbergs, Thomas P. A. Debray.
What is the DOI of this paper?
10.1016/j.jclinepi.2021.03.025. The permanent link is https://doi.org/10.1016/j.jclinepi.2021.03.025.
Bibliographic data from OpenAlex (CC0), last checked 16 November 2026. ResearchGuru does not host this paper; the links go to the publisher or the repository that does.