Review article · 2021 · Open access
Developing more generalizable prediction models from pooled studies and large clustered data sets
Statistics in Medicine, 40(15), 3533–3559 · Published 5 May 2021 · DOI 10.1002/sim.8981
In brief
“Developing more generalizable prediction models from pooled studies and large clustered data sets” is a 2021 review article by Valentijn M. T. de Jong and 4 co-authors, published in Statistics in Medicine. OpenAlex counts 62 citations as of 16 November 2026. It is open access under a CC BY licence: the full text is free to read at Statistics in Medicine.
- Citations
- 62
- Most cited in
- 2026 (27)
- Field-weighted impact
- 0.93
- References
- 93
- Authors
- 5
Abstract
Prediction models often yield inaccurate predictions for new individuals. Large data sets from pooled studies or electronic healthcare records may alleviate this with an increased sample size and variability in sample characteristics. However, existing strategies for prediction model development generally do not account for heterogeneity in predictor-outcome associations between different settings and populations. This limits the generalizability of developed models (even from large, combined, clustered data sets) and necessitates local revisions. We aim to develop methodology for producing prediction models that require less tailoring to different settings and populations. We adopt internal-external cross-validation to assess and reduce heterogeneity in models' predictive performance during the development. We propose a predictor selection algorithm that optimizes the (weighted) average performance while minimizing its variability across the hold-out clusters (or studies). Predictors are added iteratively until the estimated generalizability is optimized. We illustrate this by developing a model for predicting the risk of atrial fibrillation and updating an existing one for diagnosing deep vein thrombosis, using individual participant data from 20 cohorts (N = 10 873) and 11 diagnostic studies (N = 10 014), respectively. Meta-analysis of calibration and discrimination performance in each hold-out cluster shows that trade-offs between average and heterogeneity of performance occurred. Our methodology enables the assessment of heterogeneity of prediction model performance during model development in multiple or clustered data sets, thereby informing researchers on predictor selection to improve the generalizability to different settings and populations, and reduce the need for model tailoring. Our methodology has been implemented in the R package metamisc.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
de Jong, V. M. T., Moons, K. G., Eijkemans, M. J., Riley, R. D., & Debray, T. P. A. (2021). Developing more generalizable prediction models from pooled studies and large clustered data sets. Statistics in Medicine, 40(15), 3533–3559. https://doi.org/10.1002/sim.8981
MLA 9
de Jong, Valentijn M. T., et al. “Developing more generalizable prediction models from pooled studies and large clustered data sets.” Statistics in Medicine, vol. 40, no. 15, 2021, pp. 3533–3559. https://doi.org/10.1002/sim.8981.
Chicago
de Jong, Valentijn M. T., Karel G.M. Moons, Marinus J.C. Eijkemans, Richard D Riley, and Thomas P. A. Debray. 2021. “Developing more generalizable prediction models from pooled studies and large clustered data sets.” Statistics in Medicine 40 (15): 3533–3559. https://doi.org/10.1002/sim.8981.
Harvard
de Jong, V. M. T. et al. (2021) ‘Developing more generalizable prediction models from pooled studies and large clustered data sets’, Statistics in Medicine, 40(15), pp. 3533–3559. doi:10.1002/sim.8981.
IEEE
V. M. T. de Jong, K. G. Moons, M. J. Eijkemans, R. D. Riley, and T. P. A. Debray, “Developing more generalizable prediction models from pooled studies and large clustered data sets,” Statistics in Medicine, vol. 40, no. 15, pp. 3533–3559, 2021, doi: 10.1002/sim.8981.
BibTeX
@article{dejong2021developing,
title = {{Developing more generalizable prediction models from pooled studies and large clustered data sets}},
author = {de Jong, Valentijn M. T. and Moons, Karel G.M. and Eijkemans, Marinus J.C. and Riley, Richard D and Debray, Thomas P. A.},
journal = {Statistics in Medicine},
year = {2021},
volume = {40},
number = {15},
pages = {3533--3559},
doi = {10.1002/sim.8981},
url = {https://doi.org/10.1002/sim.8981}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Citations by year
| Year | Citations | Share of the best year |
|---|---|---|
| 2012 | 2 | |
| 2021 | 1 | |
| 2022 | 4 | |
| 2023 | 13 | |
| 2024 | 6 | |
| 2025 | 9 | |
| 2026 | 27 |
Its field-weighted citation impact is 0.93: it has been cited 0.93 times as often as the average work of the same field, type and year (1.00 is the world average).
Topic and keywords
Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- clustered data
- individual patient data
- atrial fibrillation risk prediction
- discrimination and calibration
Work with this paper
Most-cited papers in Mathematics
- Random ForestsLeo Breiman · Machine Learning · 2001 · 142,830 citations
- Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple TestingYoav Benjamini, Yosef Hochberg · Journal of the Royal Statistical Society Series B (Statistical Methodology) · 1995 · 113,923 citations
- Fitting Linear Mixed-Effects Models Using lme4Douglas M. Bates, Martin Maechler, Benjamin M. Bolker and 1 more · Journal of Statistical Software · 2015 · 100,328 citations
- The moderator-mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations.Reuben M. Baron, David A. Kenny · Journal of Personality and Social Psychology · 1986 · 95,866 citations
- Using multivariate statisticsBarbara Gerson Tabachnick, Linda S. Fidell · 1983 · 79,867 citations
Questions about this paper
How do I cite this paper in APA style?
de Jong, V. M. T., Moons, K. G., Eijkemans, M. J., Riley, R. D., & Debray, T. P. A. (2021). Developing more generalizable prediction models from pooled studies and large clustered data sets. Statistics in Medicine, 40(15), 3533–3559. https://doi.org/10.1002/sim.8981
How many times has this paper been cited?
62 times, according to OpenAlex on 16 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access: read the free full text at Statistics in Medicine.
Who wrote this paper?
Valentijn M. T. de Jong, Karel G.M. Moons, Marinus J.C. Eijkemans, Richard D Riley, Thomas P. A. Debray.
What is the DOI of this paper?
10.1002/sim.8981. The permanent link is https://doi.org/10.1002/sim.8981.
Bibliographic data from OpenAlex (CC0), last checked 16 November 2026. ResearchGuru does not host this paper; the links go to the publisher or the repository that does.