Review article · 2021 · Open access
External Validations of Cardiovascular Clinical Prediction Models: A Large-Scale Review of the Literature
Circulation Cardiovascular Quality and Outcomes, 14(8), e007858 · Published 1 August 2021 · DOI 10.1161/circoutcomes.121.007858
In brief
“External Validations of Cardiovascular Clinical Prediction Models: A Large-Scale Review of the Literature” is a 2021 review article by Benjamin S. Wessler and 11 co-authors, published in Circulation Cardiovascular Quality and Outcomes. OpenAlex counts 93 citations as of 15 November 2026, which puts it in the top 2% of works published in 2021 by citations. It is open access under a CC BY-NC-ND licence: the full text is free to read at Circulation Cardiovascular Quality and Outcomes.
- Citations
- 93
- Most cited in
- 2026 (29)
- Field-weighted impact
- 1.85
- References
- 47
- Authors
- 12
Abstract
Background: There are many clinical prediction models (CPMs) available to inform treatment decisions for patients with cardiovascular disease. However, the extent to which they have been externally tested, and how well they generally perform has not been broadly evaluated. Methods: A SCOPUS citation search was run on March 22, 2017 to identify external validations of cardiovascular CPMs in the Tufts Predictive Analytics and Comparative Effectiveness CPM Registry. We assessed the extent of external validation, performance heterogeneity across databases, and explored factors associated with model performance, including a global assessment of the clinical relatedness between the derivation and validation data. Results: We identified 2030 external validations of 1382 CPMs. Eight hundred seven (58%) of the CPMs in the Registry have never been externally validated. On average, there were 1.5 validations per CPM (range, 0–94). The median external validation area under the receiver operating characteristic curve was 0.73 (25th–75th percentile [interquartile range (IQR)], 0.66–0.79), representing a median percent decrease in discrimination of −11.1% (IQR, −32.4% to +2.7%) compared with performance on derivation data. 81% (n=1333) of validations reporting area under the receiver operating characteristic curve showed discrimination below that reported in the derivation dataset. 53% (n=983) of the validations report some measure of CPM calibration. For CPMs evaluated more than once, there was typically a large range of performance. Of 1702 validations classified by relatedness, the percent change in discrimination was −3.7% (IQR, −13.2 to 3.1) for closely related validations (n=123), −9.0 (IQR, −27.6 to 3.9) for related validations (n=862), and −17.2% (IQR, −42.3 to 0) for distantly related validations (n=717; P <0.001). Conclusions: Many published cardiovascular CPMs have never been externally validated, and for those that have, apparent performance during development is often overly optimistic. A single external validation appears insufficient to broadly understand the performance heterogeneity across different settings.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Wessler, B. S., Nelson, J., Park, J. G., McGinnes, H. L., Gulati, G., Brazil, R., VAN CALSTER, B., Klaveren, D. V., Venema, E., Steyerberg, E. W., Paulus, J. K., & Kent, D. M. (2021). External Validations of Cardiovascular Clinical Prediction Models: A Large-Scale Review of the Literature. Circulation Cardiovascular Quality and Outcomes, 14(8), e007858. https://doi.org/10.1161/circoutcomes.121.007858
MLA 9
Wessler, Benjamin S., et al. “External Validations of Cardiovascular Clinical Prediction Models: A Large-Scale Review of the Literature.” Circulation Cardiovascular Quality and Outcomes, vol. 14, no. 8, 2021, p. e007858. https://doi.org/10.1161/circoutcomes.121.007858.
Chicago
Wessler, Benjamin S., Jason Nelson, Jinny G. Park, Hannah L. McGinnes, Gaurav Gulati, Riley Brazil, Ben VAN CALSTER, et al. 2021. “External Validations of Cardiovascular Clinical Prediction Models: A Large-Scale Review of the Literature.” Circulation Cardiovascular Quality and Outcomes 14 (8): e007858. https://doi.org/10.1161/circoutcomes.121.007858.
Harvard
Wessler, B. S. et al. (2021) ‘External Validations of Cardiovascular Clinical Prediction Models: A Large-Scale Review of the Literature’, Circulation Cardiovascular Quality and Outcomes, 14(8), p. e007858. doi:10.1161/circoutcomes.121.007858.
IEEE
B. S. Wessler et al., “External Validations of Cardiovascular Clinical Prediction Models: A Large-Scale Review of the Literature,” Circulation Cardiovascular Quality and Outcomes, vol. 14, no. 8, p. e007858, 2021, doi: 10.1161/circoutcomes.121.007858.
BibTeX
@article{wessler2021external,
title = {{External Validations of Cardiovascular Clinical Prediction Models: A Large-Scale Review of the Literature}},
author = {Wessler, Benjamin S. and Nelson, Jason and Park, Jinny G. and McGinnes, Hannah L. and Gulati, Gaurav and Brazil, Riley and VAN CALSTER, Ben and Klaveren, David V. and Venema, Esmée and Steyerberg, Ewout Willem and Paulus, Jessica K. and Kent, David M.},
journal = {Circulation Cardiovascular Quality and Outcomes},
year = {2021},
volume = {14},
number = {8},
pages = {e007858},
doi = {10.1161/circoutcomes.121.007858},
url = {https://doi.org/10.1161/circoutcomes.121.007858}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Citations by year
| Year | Citations | Share of the best year |
|---|---|---|
| 2021 | 6 | |
| 2022 | 15 | |
| 2023 | 14 | |
| 2024 | 13 | |
| 2025 | 16 | |
| 2026 | 29 |
Its field-weighted citation impact is 1.85: it has been cited 1.85 times as often as the average work of the same field, type and year (1.00 is the world average).
Topic and keywords
Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- external validation
- model discrimination
- model calibration
- performance heterogeneity
- optimism bias
- cardiovascular disease
Work with this paper
Most-cited papers in Mathematics
- Random ForestsLeo Breiman · Machine Learning · 2001 · 142,830 citations
- Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple TestingYoav Benjamini, Yosef Hochberg · Journal of the Royal Statistical Society Series B (Statistical Methodology) · 1995 · 113,923 citations
- Fitting Linear Mixed-Effects Models Using lme4Douglas M. Bates, Martin Maechler, Benjamin M. Bolker and 1 more · Journal of Statistical Software · 2015 · 100,328 citations
- The moderator-mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations.Reuben M. Baron, David A. Kenny · Journal of Personality and Social Psychology · 1986 · 95,866 citations
- Using multivariate statisticsBarbara Gerson Tabachnick, Linda S. Fidell · 1983 · 79,867 citations
Questions about this paper
How do I cite this paper in APA style?
Wessler, B. S., Nelson, J., Park, J. G., McGinnes, H. L., Gulati, G., Brazil, R., VAN CALSTER, B., Klaveren, D. V., Venema, E., Steyerberg, E. W., Paulus, J. K., & Kent, D. M. (2021). External Validations of Cardiovascular Clinical Prediction Models: A Large-Scale Review of the Literature. Circulation Cardiovascular Quality and Outcomes, 14(8), e007858. https://doi.org/10.1161/circoutcomes.121.007858
How many times has this paper been cited?
93 times, according to OpenAlex on 15 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access: read the free full text at Circulation Cardiovascular Quality and Outcomes.
Who wrote this paper?
Benjamin S. Wessler, Jason Nelson, Jinny G. Park, Hannah L. McGinnes, Gaurav Gulati, Riley Brazil, Ben VAN CALSTER, David V. Klaveren, Esmée Venema, Ewout Willem Steyerberg, Jessica K. Paulus, David M. Kent.
What is the DOI of this paper?
10.1161/circoutcomes.121.007858. The permanent link is https://doi.org/10.1161/circoutcomes.121.007858.
Bibliographic data from OpenAlex (CC0), last checked 15 November 2026. ResearchGuru does not host this paper; the links go to the publisher or the repository that does.