Journal article · 2022 · Open access

Generalizability of Cardiovascular Disease Clinical Prediction Models: 158 Independent External Validations of 104 Unique Models

Gaurav Gulati, Jenica Upshaw, Benjamin S. Wessler, Riley J. Brazil, Jason Nelson, David V. Klaveren, Christine M. Lundquist, Jinny G. Park, Hannah McGinnes, Ewout Willem Steyerberg, Ben VAN CALSTER, David M. Kent

Circulation Cardiovascular Quality and Outcomes, 15(4), e008487 · Published 31 March 2022 · DOI 10.1161/circoutcomes.121.008487

Free PDFView at the publisherCite this paper

In brief

“Generalizability of Cardiovascular Disease Clinical Prediction Models: 158 Independent External Validations of 104 Unique Models” is a 2022 journal article by Gaurav Gulati and 11 co-authors, published in Circulation Cardiovascular Quality and Outcomes. OpenAlex counts 67 citations as of 15 November 2026, which puts it in the top 1% of works published in 2022 by citations. It is open access under a CC BY-NC-ND licence: the full text is free to read at Circulation Cardiovascular Quality and Outcomes.

Citations
67
Most cited in
2026 (21)
Field-weighted impact
7.17
References
108
Authors
12

Abstract

Background: While clinical prediction models (CPMs) are used increasingly commonly to guide patient care, the performance and clinical utility of these CPMs in new patient cohorts is poorly understood. Methods: We performed 158 external validations of 104 unique CPMs across 3 domains of cardiovascular disease (primary prevention, acute coronary syndrome, and heart failure). Validations were performed in publicly available clinical trial cohorts and model performance was assessed using measures of discrimination, calibration, and net benefit. To explore potential reasons for poor model performance, CPM-clinical trial cohort pairs were stratified based on relatedness, a domain-specific set of characteristics to qualitatively grade the similarity of derivation and validation patient populations. We also examined the model-based C-statistic to assess whether changes in discrimination were because of differences in case-mix between the derivation and validation samples. The impact of model updating on model performance was also assessed. Results: Discrimination decreased significantly between model derivation (0.76 [interquartile range 0.73–0.78]) and validation (0.64 [interquartile range 0.60–0.67], P <0.001), but approximately half of this decrease was because of narrower case-mix in the validation samples. CPMs had better discrimination when tested in related compared with distantly related trial cohorts. Calibration slope was also significantly higher in related trial cohorts (0.77 [interquartile range, 0.59–0.90]) than distantly related cohorts (0.59 [interquartile range 0.43–0.73], P =0.001). When considering the full range of possible decision thresholds between half and twice the outcome incidence, 91% of models had a risk of harm (net benefit below default strategy) at some threshold; this risk could be reduced substantially via updating model intercept, calibration slope, or complete re-estimation. Conclusions: There are significant decreases in model performance when applying cardiovascular disease CPMs to new patient populations, resulting in substantial risk of harm. Model updating can mitigate these risks. Care should be taken when using CPMs to guide clinical decision-making.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Gulati, G., Upshaw, J., Wessler, B. S., Brazil, R. J., Nelson, J., Klaveren, D. V., Lundquist, C. M., Park, J. G., McGinnes, H., Steyerberg, E. W., VAN CALSTER, B., & Kent, D. M. (2022). Generalizability of Cardiovascular Disease Clinical Prediction Models: 158 Independent External Validations of 104 Unique Models. Circulation Cardiovascular Quality and Outcomes, 15(4), e008487. https://doi.org/10.1161/circoutcomes.121.008487

MLA 9

Gulati, Gaurav, et al. “Generalizability of Cardiovascular Disease Clinical Prediction Models: 158 Independent External Validations of 104 Unique Models.” Circulation Cardiovascular Quality and Outcomes, vol. 15, no. 4, 2022, p. e008487. https://doi.org/10.1161/circoutcomes.121.008487.

Chicago

Gulati, Gaurav, Jenica Upshaw, Benjamin S. Wessler, Riley J. Brazil, Jason Nelson, David V. Klaveren, Christine M. Lundquist, et al. 2022. “Generalizability of Cardiovascular Disease Clinical Prediction Models: 158 Independent External Validations of 104 Unique Models.” Circulation Cardiovascular Quality and Outcomes 15 (4): e008487. https://doi.org/10.1161/circoutcomes.121.008487.

Harvard

Gulati, G. et al. (2022) ‘Generalizability of Cardiovascular Disease Clinical Prediction Models: 158 Independent External Validations of 104 Unique Models’, Circulation Cardiovascular Quality and Outcomes, 15(4), p. e008487. doi:10.1161/circoutcomes.121.008487.

IEEE

G. Gulati et al., “Generalizability of Cardiovascular Disease Clinical Prediction Models: 158 Independent External Validations of 104 Unique Models,” Circulation Cardiovascular Quality and Outcomes, vol. 15, no. 4, p. e008487, 2022, doi: 10.1161/circoutcomes.121.008487.

BibTeX

@article{gulati2022generalizability,
  title     = {{Generalizability of Cardiovascular Disease Clinical Prediction Models: 158 Independent External Validations of 104 Unique Models}},
  author    = {Gulati, Gaurav and Upshaw, Jenica and Wessler, Benjamin S. and Brazil, Riley J. and Nelson, Jason and Klaveren, David V. and Lundquist, Christine M. and Park, Jinny G. and McGinnes, Hannah and Steyerberg, Ewout Willem and VAN CALSTER, Ben and Kent, David M.},
  journal   = {Circulation Cardiovascular Quality and Outcomes},
  year      = {2022},
  volume    = {15},
  number    = {4},
  pages     = {e008487},
  doi       = {10.1161/circoutcomes.121.008487},
  url       = {https://doi.org/10.1161/circoutcomes.121.008487}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Citations by year

YearCitationsShare of the best year
20221010
20231212
20241212
20251212
20262121

Its field-weighted citation impact is 7.17: it has been cited 7.17 times as often as the average work of the same field, type and year (1.00 is the world average).

Topic and keywords

Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology

  • clinical prediction model
  • cardiovascular disease
  • external validation
  • generalizability
  • model updating
  • discrimination (statistics)
  • calibration
  • net benefit

Work with this paper

Most-cited papers in Mathematics

All highly cited Mathematics papers

Questions about this paper

How do I cite this paper in APA style?

Gulati, G., Upshaw, J., Wessler, B. S., Brazil, R. J., Nelson, J., Klaveren, D. V., Lundquist, C. M., Park, J. G., McGinnes, H., Steyerberg, E. W., VAN CALSTER, B., & Kent, D. M. (2022). Generalizability of Cardiovascular Disease Clinical Prediction Models: 158 Independent External Validations of 104 Unique Models. Circulation Cardiovascular Quality and Outcomes, 15(4), e008487. https://doi.org/10.1161/circoutcomes.121.008487

How many times has this paper been cited?

67 times, according to OpenAlex on 15 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access: read the free full text at Circulation Cardiovascular Quality and Outcomes.

Who wrote this paper?

Gaurav Gulati, Jenica Upshaw, Benjamin S. Wessler, Riley J. Brazil, Jason Nelson, David V. Klaveren, Christine M. Lundquist, Jinny G. Park, Hannah McGinnes, Ewout Willem Steyerberg, Ben VAN CALSTER, David M. Kent.

What is the DOI of this paper?

10.1161/circoutcomes.121.008487. The permanent link is https://doi.org/10.1161/circoutcomes.121.008487.

Bibliographic data from OpenAlex (CC0), last checked 15 November 2026. ResearchGuru does not host this paper; the links go to the publisher or the repository that does.