Journal article · 2021 · Open access

Large-scale validation of the prediction model risk of bias assessment Tool (PROBAST) using a short form: high risk of bias models show poorer discrimination

Esmée Venema, Benjamin S. Wessler, Jessica K. Paulus, Rehab Salah, Gowri Raman, Lester Y. Leung, Benjamin C. Koethe, Jason Nelson, Jinny G. Park, David V. Klaveren, Ewout Willem Steyerberg, David M. Kent

Journal of Clinical Epidemiology, 138, 32–39 · Published 24 June 2021 · DOI 10.1016/j.jclinepi.2021.06.017

Free PDFView at the publisherCite this paper

In brief

“Large-scale validation of the prediction model risk of bias assessment Tool (PROBAST) using a short form: high risk of bias models show poorer discrimination” is a 2021 journal article by Esmée Venema and 11 co-authors, published in Journal of Clinical Epidemiology. OpenAlex counts 66 citations as of 15 November 2026. It is open access under a CC BY-NC-ND licence: the full text is free to read at Journal of Clinical Epidemiology.

Citations
66
Most cited in
2022 (16)
Field-weighted impact
4.08
References
42
Authors
12

Abstract

OBJECTIVE: To assess whether the Prediction model Risk Of Bias ASsessment Tool (PROBAST) and a shorter version of this tool can identify clinical prediction models (CPMs) that perform poorly at external validation. STUDY DESIGN AND SETTING: We evaluated risk of bias (ROB) on 102 CPMs from the Tufts CPM Registry, comparing PROBAST to a short form consisting of six PROBAST items anticipated to best identify high ROB. We then applied the short form to all CPMs in the Registry with at least 1 validation (n=556) and assessed the change in discrimination (dAUC) in external validation cohorts (n=1,147). RESULTS: PROBAST classified 98/102 CPMS as high ROB. The short form identified 96 of these 98 as high ROB (98% sensitivity), with perfect specificity. In the full CPM registry, 527 of 556 CPMs (95%) were classified as high ROB, 20 (3.6%) low ROB, and 9 (1.6%) unclear ROB. Only one model with unclear ROB was reclassified to high ROB after full PROBAST assessment of all low and unclear ROB models. Median change in discrimination was significantly smaller in low ROB models (dAUC -0.9%, IQR -6.2-4.2%) compared to high ROB models (dAUC -11.7%, IQR -33.3-2.6%; P<0.001). CONCLUSION: High ROB is pervasive among published CPMs. It is associated with poor discriminative performance at validation, supporting the application of PROBAST or a shorter version in CPM reviews.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Venema, E., Wessler, B. S., Paulus, J. K., Salah, R., Raman, G., Leung, L. Y., Koethe, B. C., Nelson, J., Park, J. G., Klaveren, D. V., Steyerberg, E. W., & Kent, D. M. (2021). Large-scale validation of the prediction model risk of bias assessment Tool (PROBAST) using a short form: high risk of bias models show poorer discrimination. Journal of Clinical Epidemiology, 138, 32–39. https://doi.org/10.1016/j.jclinepi.2021.06.017

MLA 9

Venema, Esmée, et al. “Large-scale validation of the prediction model risk of bias assessment Tool (PROBAST) using a short form: high risk of bias models show poorer discrimination.” Journal of Clinical Epidemiology, vol. 138, 2021, pp. 32–39. https://doi.org/10.1016/j.jclinepi.2021.06.017.

Chicago

Venema, Esmée, Benjamin S. Wessler, Jessica K. Paulus, Rehab Salah, Gowri Raman, Lester Y. Leung, Benjamin C. Koethe, et al. 2021. “Large-scale validation of the prediction model risk of bias assessment Tool (PROBAST) using a short form: high risk of bias models show poorer discrimination.” Journal of Clinical Epidemiology 138: 32–39. https://doi.org/10.1016/j.jclinepi.2021.06.017.

Harvard

Venema, E. et al. (2021) ‘Large-scale validation of the prediction model risk of bias assessment Tool (PROBAST) using a short form: high risk of bias models show poorer discrimination’, Journal of Clinical Epidemiology, 138, pp. 32–39. doi:10.1016/j.jclinepi.2021.06.017.

IEEE

E. Venema et al., “Large-scale validation of the prediction model risk of bias assessment Tool (PROBAST) using a short form: high risk of bias models show poorer discrimination,” Journal of Clinical Epidemiology, vol. 138, pp. 32–39, 2021, doi: 10.1016/j.jclinepi.2021.06.017.

BibTeX

@article{venema2021large,
  title     = {{Large-scale validation of the prediction model risk of bias assessment Tool (PROBAST) using a short form: high risk of bias models show poorer discrimination}},
  author    = {Venema, Esmée and Wessler, Benjamin S. and Paulus, Jessica K. and Salah, Rehab and Raman, Gowri and Leung, Lester Y. and Koethe, Benjamin C. and Nelson, Jason and Park, Jinny G. and Klaveren, David V. and Steyerberg, Ewout Willem and Kent, David M.},
  journal   = {Journal of Clinical Epidemiology},
  year      = {2021},
  volume    = {138},
  pages     = {32--39},
  doi       = {10.1016/j.jclinepi.2021.06.017},
  url       = {https://doi.org/10.1016/j.jclinepi.2021.06.017}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Citations by year

YearCitationsShare of the best year
202111
20221616
20231515
20241313
20251212
202699

Its field-weighted citation impact is 4.08: it has been cited 4.08 times as often as the average work of the same field, type and year (1.00 is the world average).

Topic and keywords

Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews

  • PROBAST
  • risk of bias
  • clinical prediction model
  • external validation
  • discrimination (statistics)

Work with this paper

Most-cited papers in Decision Sciences

All highly cited Decision Sciences papers

Questions about this paper

How do I cite this paper in APA style?

Venema, E., Wessler, B. S., Paulus, J. K., Salah, R., Raman, G., Leung, L. Y., Koethe, B. C., Nelson, J., Park, J. G., Klaveren, D. V., Steyerberg, E. W., & Kent, D. M. (2021). Large-scale validation of the prediction model risk of bias assessment Tool (PROBAST) using a short form: high risk of bias models show poorer discrimination. Journal of Clinical Epidemiology, 138, 32–39. https://doi.org/10.1016/j.jclinepi.2021.06.017

How many times has this paper been cited?

66 times, according to OpenAlex on 15 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access: read the free full text at Journal of Clinical Epidemiology.

Who wrote this paper?

Esmée Venema, Benjamin S. Wessler, Jessica K. Paulus, Rehab Salah, Gowri Raman, Lester Y. Leung, Benjamin C. Koethe, Jason Nelson, Jinny G. Park, David V. Klaveren, Ewout Willem Steyerberg, David M. Kent.

What is the DOI of this paper?

10.1016/j.jclinepi.2021.06.017. The permanent link is https://doi.org/10.1016/j.jclinepi.2021.06.017.

Bibliographic data from OpenAlex (CC0), last checked 15 November 2026. ResearchGuru does not host this paper; the links go to the publisher or the repository that does.