Review article · 2021 · Open access

External Validations of Cardiovascular Clinical Prediction Models: A Large-Scale Review of the Literature

Benjamin S. Wessler, Jason Nelson, Jinny G. Park, Hannah L. McGinnes, Gaurav Gulati, Riley Brazil, Ben VAN CALSTER, David V. Klaveren, Esmée Venema, Ewout Willem Steyerberg, Jessica K. Paulus, David M. Kent

Circulation Cardiovascular Quality and Outcomes, 14(8), e007858 · Published 1 August 2021 · DOI 10.1161/circoutcomes.121.007858

Free PDFView at the publisherCite this paper

In brief

“External Validations of Cardiovascular Clinical Prediction Models: A Large-Scale Review of the Literature” is a 2021 review article by Benjamin S. Wessler and 11 co-authors, published in Circulation Cardiovascular Quality and Outcomes. OpenAlex counts 93 citations as of 15 November 2026, which puts it in the top 2% of works published in 2021 by citations. It is open access under a CC BY-NC-ND licence: the full text is free to read at Circulation Cardiovascular Quality and Outcomes.

Citations
93
Most cited in
2026 (29)
Field-weighted impact
1.85
References
47
Authors
12

Abstract

Background: There are many clinical prediction models (CPMs) available to inform treatment decisions for patients with cardiovascular disease. However, the extent to which they have been externally tested, and how well they generally perform has not been broadly evaluated. Methods: A SCOPUS citation search was run on March 22, 2017 to identify external validations of cardiovascular CPMs in the Tufts Predictive Analytics and Comparative Effectiveness CPM Registry. We assessed the extent of external validation, performance heterogeneity across databases, and explored factors associated with model performance, including a global assessment of the clinical relatedness between the derivation and validation data. Results: We identified 2030 external validations of 1382 CPMs. Eight hundred seven (58%) of the CPMs in the Registry have never been externally validated. On average, there were 1.5 validations per CPM (range, 0–94). The median external validation area under the receiver operating characteristic curve was 0.73 (25th–75th percentile [interquartile range (IQR)], 0.66–0.79), representing a median percent decrease in discrimination of −11.1% (IQR, −32.4% to +2.7%) compared with performance on derivation data. 81% (n=1333) of validations reporting area under the receiver operating characteristic curve showed discrimination below that reported in the derivation dataset. 53% (n=983) of the validations report some measure of CPM calibration. For CPMs evaluated more than once, there was typically a large range of performance. Of 1702 validations classified by relatedness, the percent change in discrimination was −3.7% (IQR, −13.2 to 3.1) for closely related validations (n=123), −9.0 (IQR, −27.6 to 3.9) for related validations (n=862), and −17.2% (IQR, −42.3 to 0) for distantly related validations (n=717; P <0.001). Conclusions: Many published cardiovascular CPMs have never been externally validated, and for those that have, apparent performance during development is often overly optimistic. A single external validation appears insufficient to broadly understand the performance heterogeneity across different settings.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Wessler, B. S., Nelson, J., Park, J. G., McGinnes, H. L., Gulati, G., Brazil, R., VAN CALSTER, B., Klaveren, D. V., Venema, E., Steyerberg, E. W., Paulus, J. K., & Kent, D. M. (2021). External Validations of Cardiovascular Clinical Prediction Models: A Large-Scale Review of the Literature. Circulation Cardiovascular Quality and Outcomes, 14(8), e007858. https://doi.org/10.1161/circoutcomes.121.007858

MLA 9

Wessler, Benjamin S., et al. “External Validations of Cardiovascular Clinical Prediction Models: A Large-Scale Review of the Literature.” Circulation Cardiovascular Quality and Outcomes, vol. 14, no. 8, 2021, p. e007858. https://doi.org/10.1161/circoutcomes.121.007858.

Chicago

Wessler, Benjamin S., Jason Nelson, Jinny G. Park, Hannah L. McGinnes, Gaurav Gulati, Riley Brazil, Ben VAN CALSTER, et al. 2021. “External Validations of Cardiovascular Clinical Prediction Models: A Large-Scale Review of the Literature.” Circulation Cardiovascular Quality and Outcomes 14 (8): e007858. https://doi.org/10.1161/circoutcomes.121.007858.

Harvard

Wessler, B. S. et al. (2021) ‘External Validations of Cardiovascular Clinical Prediction Models: A Large-Scale Review of the Literature’, Circulation Cardiovascular Quality and Outcomes, 14(8), p. e007858. doi:10.1161/circoutcomes.121.007858.

IEEE

B. S. Wessler et al., “External Validations of Cardiovascular Clinical Prediction Models: A Large-Scale Review of the Literature,” Circulation Cardiovascular Quality and Outcomes, vol. 14, no. 8, p. e007858, 2021, doi: 10.1161/circoutcomes.121.007858.

BibTeX

@article{wessler2021external,
  title     = {{External Validations of Cardiovascular Clinical Prediction Models: A Large-Scale Review of the Literature}},
  author    = {Wessler, Benjamin S. and Nelson, Jason and Park, Jinny G. and McGinnes, Hannah L. and Gulati, Gaurav and Brazil, Riley and VAN CALSTER, Ben and Klaveren, David V. and Venema, Esmée and Steyerberg, Ewout Willem and Paulus, Jessica K. and Kent, David M.},
  journal   = {Circulation Cardiovascular Quality and Outcomes},
  year      = {2021},
  volume    = {14},
  number    = {8},
  pages     = {e007858},
  doi       = {10.1161/circoutcomes.121.007858},
  url       = {https://doi.org/10.1161/circoutcomes.121.007858}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Citations by year

YearCitationsShare of the best year
202166
20221515
20231414
20241313
20251616
20262929

Its field-weighted citation impact is 1.85: it has been cited 1.85 times as often as the average work of the same field, type and year (1.00 is the world average).

Topic and keywords

Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology

  • external validation
  • model discrimination
  • model calibration
  • performance heterogeneity
  • optimism bias
  • cardiovascular disease

Work with this paper

Most-cited papers in Mathematics

All highly cited Mathematics papers

Questions about this paper

How do I cite this paper in APA style?

Wessler, B. S., Nelson, J., Park, J. G., McGinnes, H. L., Gulati, G., Brazil, R., VAN CALSTER, B., Klaveren, D. V., Venema, E., Steyerberg, E. W., Paulus, J. K., & Kent, D. M. (2021). External Validations of Cardiovascular Clinical Prediction Models: A Large-Scale Review of the Literature. Circulation Cardiovascular Quality and Outcomes, 14(8), e007858. https://doi.org/10.1161/circoutcomes.121.007858

How many times has this paper been cited?

93 times, according to OpenAlex on 15 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access: read the free full text at Circulation Cardiovascular Quality and Outcomes.

Who wrote this paper?

Benjamin S. Wessler, Jason Nelson, Jinny G. Park, Hannah L. McGinnes, Gaurav Gulati, Riley Brazil, Ben VAN CALSTER, David V. Klaveren, Esmée Venema, Ewout Willem Steyerberg, Jessica K. Paulus, David M. Kent.

What is the DOI of this paper?

10.1161/circoutcomes.121.007858. The permanent link is https://doi.org/10.1161/circoutcomes.121.007858.

Bibliographic data from OpenAlex (CC0), last checked 15 November 2026. ResearchGuru does not host this paper; the links go to the publisher or the repository that does.