Review article · 2023 · Open access

Systematic metareview of prediction studies demonstrates stable trends in bias and low PROBAST inter-rater agreement

Liselotte F.S. Langenhuijsen, Roemer J. Janse, Esmée Venema, David M. Kent, Merel van Diepen, Friedo Wilhelm Dekker, Ewout Willem Steyerberg, Ype de Jong

Journal of Clinical Epidemiology, 159, 159–173 · Published 2 May 2023 · DOI 10.1016/j.jclinepi.2023.04.012

Free PDFView at the publisherCite this paper

In brief

“Systematic metareview of prediction studies demonstrates stable trends in bias and low PROBAST inter-rater agreement” is a 2023 review article by Liselotte F.S. Langenhuijsen and 7 co-authors, published in Journal of Clinical Epidemiology. OpenAlex counts 20 citations as of 15 November 2026. It is open access under a CC BY licence: the full text is free to read at Journal of Clinical Epidemiology.

Citations
20
Most cited in
2026 (8)
Field-weighted impact
0.95
References
53
Authors
8

Abstract

OBJECTIVES: To (1) explore trends of risk of bias (ROB) in prediction research over time following key methodological publications, using the Prediction model Risk Of Bias ASsessment Tool (PROBAST) and (2) assess the inter-rater agreement of the PROBAST. STUDY DESIGN AND SETTING: PubMed and Web of Science were searched for reviews with extractable PROBAST scores on domain and signaling question (SQ) level. ROB trends were visually correlated with yearly citations of key publications. Inter-rater agreement was assessed using Cohen's Kappa. RESULTS: One hundred and thirty nine systematic reviews were included, of which 85 reviews (containing 2,477 single studies) on domain level and 54 reviews (containing 2,458 single studies) on SQ level. High ROB was prevalent, especially in the Analysis domain, and overall trends of ROB remained relatively stable over time. The inter-rater agreement was low, both on domain (Kappa 0.04-0.26) and SQ level (Kappa -0.14 to 0.49). CONCLUSION: Prediction model studies are at high ROB and time trends in ROB as assessed with the PROBAST remain relatively stable. These results might be explained by key publications having no influence on ROB or recency of key publications. Moreover, the trend may suffer from the low inter-rater agreement and ceiling effect of the PROBAST. The inter-rater agreement could potentially be improved by altering the PROBAST or providing training on how to apply the PROBAST.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Langenhuijsen, L. F., Janse, R. J., Venema, E., Kent, D. M., van Diepen, M., Dekker, F. W., Steyerberg, E. W., & de Jong, Y. (2023). Systematic metareview of prediction studies demonstrates stable trends in bias and low PROBAST inter-rater agreement. Journal of Clinical Epidemiology, 159, 159–173. https://doi.org/10.1016/j.jclinepi.2023.04.012

MLA 9

Langenhuijsen, Liselotte F.S., et al. “Systematic metareview of prediction studies demonstrates stable trends in bias and low PROBAST inter-rater agreement.” Journal of Clinical Epidemiology, vol. 159, 2023, pp. 159–173. https://doi.org/10.1016/j.jclinepi.2023.04.012.

Chicago

Langenhuijsen, Liselotte F.S., Roemer J. Janse, Esmée Venema, David M. Kent, Merel van Diepen, Friedo Wilhelm Dekker, Ewout Willem Steyerberg, and Ype de Jong. 2023. “Systematic metareview of prediction studies demonstrates stable trends in bias and low PROBAST inter-rater agreement.” Journal of Clinical Epidemiology 159: 159–173. https://doi.org/10.1016/j.jclinepi.2023.04.012.

Harvard

Langenhuijsen, L. F. et al. (2023) ‘Systematic metareview of prediction studies demonstrates stable trends in bias and low PROBAST inter-rater agreement’, Journal of Clinical Epidemiology, 159, pp. 159–173. doi:10.1016/j.jclinepi.2023.04.012.

IEEE

L. F. Langenhuijsen et al., “Systematic metareview of prediction studies demonstrates stable trends in bias and low PROBAST inter-rater agreement,” Journal of Clinical Epidemiology, vol. 159, pp. 159–173, 2023, doi: 10.1016/j.jclinepi.2023.04.012.

BibTeX

@article{langenhuijsen2023systematic,
  title     = {{Systematic metareview of prediction studies demonstrates stable trends in bias and low PROBAST inter-rater agreement}},
  author    = {Langenhuijsen, Liselotte F.S. and Janse, Roemer J. and Venema, Esmée and Kent, David M. and van Diepen, Merel and Dekker, Friedo Wilhelm and Steyerberg, Ewout Willem and de Jong, Ype},
  journal   = {Journal of Clinical Epidemiology},
  year      = {2023},
  volume    = {159},
  pages     = {159--173},
  doi       = {10.1016/j.jclinepi.2023.04.012},
  url       = {https://doi.org/10.1016/j.jclinepi.2023.04.012}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Citations by year

YearCitationsShare of the best year
202311
202455
202566
202688

Its field-weighted citation impact is 0.95: it has been cited 0.95 times as often as the average work of the same field, type and year (1.00 is the world average).

Topic and keywords

Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews

  • PROBAST
  • risk of bias
  • prediction model
  • interobserver agreement
  • bias assessment tools
  • methodological quality

Work with this paper

Most-cited papers in Decision Sciences

All highly cited Decision Sciences papers

Questions about this paper

How do I cite this paper in APA style?

Langenhuijsen, L. F., Janse, R. J., Venema, E., Kent, D. M., van Diepen, M., Dekker, F. W., Steyerberg, E. W., & de Jong, Y. (2023). Systematic metareview of prediction studies demonstrates stable trends in bias and low PROBAST inter-rater agreement. Journal of Clinical Epidemiology, 159, 159–173. https://doi.org/10.1016/j.jclinepi.2023.04.012

How many times has this paper been cited?

20 times, according to OpenAlex on 15 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access: read the free full text at Journal of Clinical Epidemiology.

Who wrote this paper?

Liselotte F.S. Langenhuijsen, Roemer J. Janse, Esmée Venema, David M. Kent, Merel van Diepen, Friedo Wilhelm Dekker, Ewout Willem Steyerberg, Ype de Jong.

What is the DOI of this paper?

10.1016/j.jclinepi.2023.04.012. The permanent link is https://doi.org/10.1016/j.jclinepi.2023.04.012.

Bibliographic data from OpenAlex (CC0), last checked 15 November 2026. ResearchGuru does not host this paper; the links go to the publisher or the repository that does.