Journal article · 2022 · Open access
Generalizability of Cardiovascular Disease Clinical Prediction Models: 158 Independent External Validations of 104 Unique Models
Circulation Cardiovascular Quality and Outcomes, 15(4), e008487 · Published 31 March 2022 · DOI 10.1161/circoutcomes.121.008487
In brief
“Generalizability of Cardiovascular Disease Clinical Prediction Models: 158 Independent External Validations of 104 Unique Models” is a 2022 journal article by Gaurav Gulati and 11 co-authors, published in Circulation Cardiovascular Quality and Outcomes. OpenAlex counts 67 citations as of 15 November 2026, which puts it in the top 1% of works published in 2022 by citations. It is open access under a CC BY-NC-ND licence: the full text is free to read at Circulation Cardiovascular Quality and Outcomes.
- Citations
- 67
- Most cited in
- 2026 (21)
- Field-weighted impact
- 7.17
- References
- 108
- Authors
- 12
Abstract
Background: While clinical prediction models (CPMs) are used increasingly commonly to guide patient care, the performance and clinical utility of these CPMs in new patient cohorts is poorly understood. Methods: We performed 158 external validations of 104 unique CPMs across 3 domains of cardiovascular disease (primary prevention, acute coronary syndrome, and heart failure). Validations were performed in publicly available clinical trial cohorts and model performance was assessed using measures of discrimination, calibration, and net benefit. To explore potential reasons for poor model performance, CPM-clinical trial cohort pairs were stratified based on relatedness, a domain-specific set of characteristics to qualitatively grade the similarity of derivation and validation patient populations. We also examined the model-based C-statistic to assess whether changes in discrimination were because of differences in case-mix between the derivation and validation samples. The impact of model updating on model performance was also assessed. Results: Discrimination decreased significantly between model derivation (0.76 [interquartile range 0.73–0.78]) and validation (0.64 [interquartile range 0.60–0.67], P <0.001), but approximately half of this decrease was because of narrower case-mix in the validation samples. CPMs had better discrimination when tested in related compared with distantly related trial cohorts. Calibration slope was also significantly higher in related trial cohorts (0.77 [interquartile range, 0.59–0.90]) than distantly related cohorts (0.59 [interquartile range 0.43–0.73], P =0.001). When considering the full range of possible decision thresholds between half and twice the outcome incidence, 91% of models had a risk of harm (net benefit below default strategy) at some threshold; this risk could be reduced substantially via updating model intercept, calibration slope, or complete re-estimation. Conclusions: There are significant decreases in model performance when applying cardiovascular disease CPMs to new patient populations, resulting in substantial risk of harm. Model updating can mitigate these risks. Care should be taken when using CPMs to guide clinical decision-making.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Gulati, G., Upshaw, J., Wessler, B. S., Brazil, R. J., Nelson, J., Klaveren, D. V., Lundquist, C. M., Park, J. G., McGinnes, H., Steyerberg, E. W., VAN CALSTER, B., & Kent, D. M. (2022). Generalizability of Cardiovascular Disease Clinical Prediction Models: 158 Independent External Validations of 104 Unique Models. Circulation Cardiovascular Quality and Outcomes, 15(4), e008487. https://doi.org/10.1161/circoutcomes.121.008487
MLA 9
Gulati, Gaurav, et al. “Generalizability of Cardiovascular Disease Clinical Prediction Models: 158 Independent External Validations of 104 Unique Models.” Circulation Cardiovascular Quality and Outcomes, vol. 15, no. 4, 2022, p. e008487. https://doi.org/10.1161/circoutcomes.121.008487.
Chicago
Gulati, Gaurav, Jenica Upshaw, Benjamin S. Wessler, Riley J. Brazil, Jason Nelson, David V. Klaveren, Christine M. Lundquist, et al. 2022. “Generalizability of Cardiovascular Disease Clinical Prediction Models: 158 Independent External Validations of 104 Unique Models.” Circulation Cardiovascular Quality and Outcomes 15 (4): e008487. https://doi.org/10.1161/circoutcomes.121.008487.
Harvard
Gulati, G. et al. (2022) ‘Generalizability of Cardiovascular Disease Clinical Prediction Models: 158 Independent External Validations of 104 Unique Models’, Circulation Cardiovascular Quality and Outcomes, 15(4), p. e008487. doi:10.1161/circoutcomes.121.008487.
IEEE
G. Gulati et al., “Generalizability of Cardiovascular Disease Clinical Prediction Models: 158 Independent External Validations of 104 Unique Models,” Circulation Cardiovascular Quality and Outcomes, vol. 15, no. 4, p. e008487, 2022, doi: 10.1161/circoutcomes.121.008487.
BibTeX
@article{gulati2022generalizability,
title = {{Generalizability of Cardiovascular Disease Clinical Prediction Models: 158 Independent External Validations of 104 Unique Models}},
author = {Gulati, Gaurav and Upshaw, Jenica and Wessler, Benjamin S. and Brazil, Riley J. and Nelson, Jason and Klaveren, David V. and Lundquist, Christine M. and Park, Jinny G. and McGinnes, Hannah and Steyerberg, Ewout Willem and VAN CALSTER, Ben and Kent, David M.},
journal = {Circulation Cardiovascular Quality and Outcomes},
year = {2022},
volume = {15},
number = {4},
pages = {e008487},
doi = {10.1161/circoutcomes.121.008487},
url = {https://doi.org/10.1161/circoutcomes.121.008487}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Citations by year
| Year | Citations | Share of the best year |
|---|---|---|
| 2022 | 10 | |
| 2023 | 12 | |
| 2024 | 12 | |
| 2025 | 12 | |
| 2026 | 21 |
Its field-weighted citation impact is 7.17: it has been cited 7.17 times as often as the average work of the same field, type and year (1.00 is the world average).
Topic and keywords
Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- clinical prediction model
- cardiovascular disease
- external validation
- generalizability
- model updating
- discrimination (statistics)
- calibration
- net benefit
Work with this paper
Most-cited papers in Mathematics
- Random ForestsLeo Breiman · Machine Learning · 2001 · 142,830 citations
- Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple TestingYoav Benjamini, Yosef Hochberg · Journal of the Royal Statistical Society Series B (Statistical Methodology) · 1995 · 113,923 citations
- Fitting Linear Mixed-Effects Models Using lme4Douglas M. Bates, Martin Maechler, Benjamin M. Bolker and 1 more · Journal of Statistical Software · 2015 · 100,328 citations
- The moderator-mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations.Reuben M. Baron, David A. Kenny · Journal of Personality and Social Psychology · 1986 · 95,866 citations
- Using multivariate statisticsBarbara Gerson Tabachnick, Linda S. Fidell · 1983 · 79,867 citations
Questions about this paper
How do I cite this paper in APA style?
Gulati, G., Upshaw, J., Wessler, B. S., Brazil, R. J., Nelson, J., Klaveren, D. V., Lundquist, C. M., Park, J. G., McGinnes, H., Steyerberg, E. W., VAN CALSTER, B., & Kent, D. M. (2022). Generalizability of Cardiovascular Disease Clinical Prediction Models: 158 Independent External Validations of 104 Unique Models. Circulation Cardiovascular Quality and Outcomes, 15(4), e008487. https://doi.org/10.1161/circoutcomes.121.008487
How many times has this paper been cited?
67 times, according to OpenAlex on 15 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access: read the free full text at Circulation Cardiovascular Quality and Outcomes.
Who wrote this paper?
Gaurav Gulati, Jenica Upshaw, Benjamin S. Wessler, Riley J. Brazil, Jason Nelson, David V. Klaveren, Christine M. Lundquist, Jinny G. Park, Hannah McGinnes, Ewout Willem Steyerberg, Ben VAN CALSTER, David M. Kent.
What is the DOI of this paper?
10.1161/circoutcomes.121.008487. The permanent link is https://doi.org/10.1161/circoutcomes.121.008487.
Bibliographic data from OpenAlex (CC0), last checked 15 November 2026. ResearchGuru does not host this paper; the links go to the publisher or the repository that does.