Journal article · 2025 · Open access

Assessing the quality of prediction models in health care using the Prediction model Risk Of Bias ASsessment Tool (PROBAST): an evaluation of its use and practical application

Tabea Kaul, Johanna AAG Damen, Laure Wynants, Ben VAN CALSTER, Maarten van Smeden, Lotty Hooft, Karel G.M. Moons

Journal of Clinical Epidemiology, 181, 111732 · Published 25 February 2025 · DOI 10.1016/j.jclinepi.2025.111732

Free PDFView at the publisherCite this paper

In brief

“Assessing the quality of prediction models in health care using the Prediction model Risk Of Bias ASsessment Tool (PROBAST): an evaluation of its use and practical application” is a 2025 journal article by Tabea Kaul and 6 co-authors, published in Journal of Clinical Epidemiology. OpenAlex counts 20 citations as of 15 November 2026, which puts it in the top 3% of works published in 2025 by citations. It is open access under a CC BY licence: the full text is free to read at Journal of Clinical Epidemiology.

Citations
20
Most cited in
2026 (15)
Field-weighted impact
4.16
References
17
Authors
7

Abstract

BACKGROUND AND OBJECTIVES: Since 2019, the Prediction model Risk Of Bias ASsessment Tool (PROBAST; www.probast.org) has supported methodological quality assessments of prediction model studies. Most prediction model studies are rated with a "High" risk of bias (ROB) and researchers report low interrater reliability (IRR) using PROBAST. We aimed to (1) assess the IRR of PROBAST ratings between assessors of the same study and understand reasons for discrepancies, (2) determine which items contribute most to domain-level ROB ratings, and (3) explore the impact of consensus meetings. STUDY DESIGN AND SETTING: We used PROBAST assessments from a systematic review of diagnostic and prognostic COVID-19 prediction models as a case study. Assessors included international experts in prediction model studies or their reviews. We assessed IRR using prevalence-adjusted bias-adjusted kappa (PABAK) before consensus meetings, examined bias ratings per domain-level ROB judgments, and evaluated the impact of consensus meetings by identifying rating changes after discussion. RESULTS: We analyzed 2167 PROBAST assessments from 27 assessor pairs covering 760 prediction models: 384 developments, 242 validations, and 134 mixed assessments (including both). The IRR using PABAK was higher for overall ROB judgments (development: 0.82 [0.76; 0.89]; validation: 0.78 [0.68; 0.88]) compared to domain- and item-level judgments. Some PROBAST items frequently contributed to domain-level ROB judgments, eg, 3.5 Outcome blinding and 4.1 Sample size. Consensus discussions mainly led to item-level and never to overall ROB rating changes. CONCLUSION: Within this case study, PROBAST assessments received high IRR at the overall ROB level, with some variation at item- and domain-level. To reduce variability, PROBAST assessors should standardize item- and domain-level judgments and hold well-structured consensus meetings between assessors of the same study. PLAIN LANGUAGE SUMMARY: The Prediction model Risk Of Bias ASsessment Tool (PROBAST; www.probast.org) provides a set of items to assess the quality of medical studies on so-called prediction tools that calculate an individual's probability of having or developing a certain disease or health outcome. Previous research found low interrater reliability (IRR; ie, how consistently two assessors rate aspects of the same study) when using PROBAST. To understand why this is the case, we conducted a large study involving more than 30 experts from around the world, all of whom applied PROBAST to the same set of prediction tool studies. Based on more than 2150 PROBAST assessments, we identified which PROBAST items led to the most disagreements between raters, explored reasons for these disagreements, and examined whether the use of so-called consensus meetings (ie, different assessors of the same study discuss their ratings and decide on a finalized rating)…

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Kaul, T., Damen, J. A., Wynants, L., VAN CALSTER, B., van Smeden, M., Hooft, L., & Moons, K. G. (2025). Assessing the quality of prediction models in health care using the Prediction model Risk Of Bias ASsessment Tool (PROBAST): an evaluation of its use and practical application. Journal of Clinical Epidemiology, 181, 111732. https://doi.org/10.1016/j.jclinepi.2025.111732

MLA 9

Kaul, Tabea, et al. “Assessing the quality of prediction models in health care using the Prediction model Risk Of Bias ASsessment Tool (PROBAST): an evaluation of its use and practical application.” Journal of Clinical Epidemiology, vol. 181, 2025, p. 111732. https://doi.org/10.1016/j.jclinepi.2025.111732.

Chicago

Kaul, Tabea, Johanna AAG Damen, Laure Wynants, Ben VAN CALSTER, Maarten van Smeden, Lotty Hooft, and Karel G.M. Moons. 2025. “Assessing the quality of prediction models in health care using the Prediction model Risk Of Bias ASsessment Tool (PROBAST): an evaluation of its use and practical application.” Journal of Clinical Epidemiology 181: 111732. https://doi.org/10.1016/j.jclinepi.2025.111732.

Harvard

Kaul, T. et al. (2025) ‘Assessing the quality of prediction models in health care using the Prediction model Risk Of Bias ASsessment Tool (PROBAST): an evaluation of its use and practical application’, Journal of Clinical Epidemiology, 181, p. 111732. doi:10.1016/j.jclinepi.2025.111732.

IEEE

T. Kaul et al., “Assessing the quality of prediction models in health care using the Prediction model Risk Of Bias ASsessment Tool (PROBAST): an evaluation of its use and practical application,” Journal of Clinical Epidemiology, vol. 181, p. 111732, 2025, doi: 10.1016/j.jclinepi.2025.111732.

BibTeX

@article{kaul2025assessing,
  title     = {{Assessing the quality of prediction models in health care using the Prediction model Risk Of Bias ASsessment Tool (PROBAST): an evaluation of its use and practical application}},
  author    = {Kaul, Tabea and Damen, Johanna AAG and Wynants, Laure and VAN CALSTER, Ben and van Smeden, Maarten and Hooft, Lotty and Moons, Karel G.M.},
  journal   = {Journal of Clinical Epidemiology},
  year      = {2025},
  volume    = {181},
  pages     = {111732},
  doi       = {10.1016/j.jclinepi.2025.111732},
  url       = {https://doi.org/10.1016/j.jclinepi.2025.111732}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews

  • PROBAST
  • risk of bias assessment
  • prediction model
  • inter-rater reliability
  • consensus conference
  • sample size
  • methodological quality assessment
  • prediction model validation

Work with this paper

Most-cited papers in Decision Sciences

All highly cited Decision Sciences papers

Questions about this paper

How do I cite this paper in APA style?

Kaul, T., Damen, J. A., Wynants, L., VAN CALSTER, B., van Smeden, M., Hooft, L., & Moons, K. G. (2025). Assessing the quality of prediction models in health care using the Prediction model Risk Of Bias ASsessment Tool (PROBAST): an evaluation of its use and practical application. Journal of Clinical Epidemiology, 181, 111732. https://doi.org/10.1016/j.jclinepi.2025.111732

How many times has this paper been cited?

20 times, according to OpenAlex on 15 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access: read the free full text at Journal of Clinical Epidemiology.

Who wrote this paper?

Tabea Kaul, Johanna AAG Damen, Laure Wynants, Ben VAN CALSTER, Maarten van Smeden, Lotty Hooft, Karel G.M. Moons.

What is the DOI of this paper?

10.1016/j.jclinepi.2025.111732. The permanent link is https://doi.org/10.1016/j.jclinepi.2025.111732.

Bibliographic data from OpenAlex (CC0), last checked 15 November 2026. ResearchGuru does not host this paper; the links go to the publisher or the repository that does.