Journal article · 2023 · Open access
Inter-Rater Agreement in Assessing Risk of Bias in Melanoma Prediction Studies Using the Prediction Model Risk of Bias Assessment Tool (PROBAST): Results from a Controlled Experiment on the Effect of Specific Rater Training
Journal of Clinical Medicine, 12(5), 1976 · Published 2 March 2023 · DOI 10.3390/jcm12051976
In brief
“Inter-Rater Agreement in Assessing Risk of Bias in Melanoma Prediction Studies Using the Prediction Model Risk of Bias Assessment Tool (PROBAST): Results from a Controlled Experiment on the Effect of Specific Rater Training” is a 2023 journal article by Isabelle Kaiser and 7 co-authors, published in Journal of Clinical Medicine. OpenAlex counts 19 citations as of 15 November 2026, which puts it in the top 7% of works published in 2023 by citations. It is open access under a CC BY licence: the full text is free to read at Journal of Clinical Medicine.
- Citations
- 19
- Most cited in
- 2026 (8)
- Field-weighted impact
- 1.86
- References
- 81
- Authors
- 8
Abstract
Assessing the risk of bias (ROB) of studies is an important part of the conduct of systematic reviews and meta-analyses in clinical medicine. Among the many existing ROB tools, the Prediction Model Risk of Bias Assessment Tool (PROBAST) is a rather new instrument specifically designed to assess the ROB of prediction studies. In our study we analyzed the inter-rater reliability (IRR) of PROBAST and the effect of specialized training on the IRR. Six raters independently assessed the risk of bias (ROB) of all melanoma risk prediction studies published until 2021 (n = 42) using the PROBAST instrument. The raters evaluated the ROB of the first 20 studies without any guidance other than the published PROBAST literature. The remaining 22 studies were assessed after receiving customized training and guidance. Gwet’s AC1 was used as the primary measure to quantify the pairwise and multi-rater IRR. Depending on the PROBAST domain, results before training showed a slight to moderate IRR (multi-rater AC1 ranging from 0.071 to 0.535). After training, the multi-rater AC1 ranged from 0.294 to 0.780 with a significant improvement for the overall ROB rating and two of the four domains. The largest net gain was achieved in the overall ROB rating (difference in multi-rater AC1: 0.405, 95%-CI 0.149–0.630). In conclusion, without targeted guidance, the IRR of PROBAST is low, questioning its use as an appropriate ROB instrument for prediction studies. Intensive training and guidance manuals with context-specific decision rules are needed to correctly apply and interpret the PROBAST instrument and to ensure consistency of ROB ratings.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Kaiser, I., Pfahlberg, A. B., Mathes, S., Uter, W. J. C., Diehl, K., Steeb, T., Heppt, M. V., & Gefeller, O. (2023). Inter-Rater Agreement in Assessing Risk of Bias in Melanoma Prediction Studies Using the Prediction Model Risk of Bias Assessment Tool (PROBAST): Results from a Controlled Experiment on the Effect of Specific Rater Training. Journal of Clinical Medicine, 12(5), 1976. https://doi.org/10.3390/jcm12051976
MLA 9
Kaiser, Isabelle, et al. “Inter-Rater Agreement in Assessing Risk of Bias in Melanoma Prediction Studies Using the Prediction Model Risk of Bias Assessment Tool (PROBAST): Results from a Controlled Experiment on the Effect of Specific Rater Training.” Journal of Clinical Medicine, vol. 12, no. 5, 2023, p. 1976. https://doi.org/10.3390/jcm12051976.
Chicago
Kaiser, Isabelle, Annette B. Pfahlberg, Sonja Mathes, Wolfgang J. C. Uter, Katharina Diehl, Theresa Steeb, Markus Vincent Heppt, and Olaf Gefeller. 2023. “Inter-Rater Agreement in Assessing Risk of Bias in Melanoma Prediction Studies Using the Prediction Model Risk of Bias Assessment Tool (PROBAST): Results from a Controlled Experiment on the Effect of Specific Rater Training.” Journal of Clinical Medicine 12 (5): 1976. https://doi.org/10.3390/jcm12051976.
Harvard
Kaiser, I. et al. (2023) ‘Inter-Rater Agreement in Assessing Risk of Bias in Melanoma Prediction Studies Using the Prediction Model Risk of Bias Assessment Tool (PROBAST): Results from a Controlled Experiment on the Effect of Specific Rater Training’, Journal of Clinical Medicine, 12(5), p. 1976. doi:10.3390/jcm12051976.
IEEE
I. Kaiser et al., “Inter-Rater Agreement in Assessing Risk of Bias in Melanoma Prediction Studies Using the Prediction Model Risk of Bias Assessment Tool (PROBAST): Results from a Controlled Experiment on the Effect of Specific Rater Training,” Journal of Clinical Medicine, vol. 12, no. 5, p. 1976, 2023, doi: 10.3390/jcm12051976.
BibTeX
@article{kaiser2023inter,
title = {{Inter-Rater Agreement in Assessing Risk of Bias in Melanoma Prediction Studies Using the Prediction Model Risk of Bias Assessment Tool (PROBAST): Results from a Controlled Experiment on the Effect of Specific Rater Training}},
author = {Kaiser, Isabelle and Pfahlberg, Annette B. and Mathes, Sonja and Uter, Wolfgang J. C. and Diehl, Katharina and Steeb, Theresa and Heppt, Markus Vincent and Gefeller, Olaf},
journal = {Journal of Clinical Medicine},
year = {2023},
volume = {12},
number = {5},
pages = {1976},
doi = {10.3390/jcm12051976},
url = {https://doi.org/10.3390/jcm12051976}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Citations by year
| Year | Citations | Share of the best year |
|---|---|---|
| 2023 | 2 | |
| 2024 | 4 | |
| 2025 | 5 | |
| 2026 | 8 |
Its field-weighted citation impact is 1.86: it has been cited 1.86 times as often as the average work of the same field, type and year (1.00 is the world average).
Topic and keywords
Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- risk of bias
- PROBAST
- inter-rater reliability
- melanoma risk prediction
- rater training
- systematic review methodology
- Gwet's AC1
Work with this paper
Most-cited papers in Decision Sciences
- The PRISMA 2020 statement: an updated guideline for reporting systematic reviewsMatthew James Page, Joanne E. McKenzie, Patrick M. Bossuyt and 23 more · BMJ · 2021 · 144,919 citations
- Evaluating Structural Equation Models with Unobservable Variables and Measurement ErrorClaes Fornell, David F. Larcker · Journal of Marketing Research · 1981 · 118,336 citations
- Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternativesLi‐tze Hu, Peter M. Bentler · Structural Equation Modeling A Multidisciplinary Journal · 1999 · 117,461 citations
- Common method biases in behavioral research: A critical review of the literature and recommended remedies.Philip M. Podsakoff, SCOTT B. MACKENZIE, Jeong Yeon Lee and 1 more · Journal of Applied Psychology · 2003 · 89,248 citations
- Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statementDavid Moher, Alessandro Liberati, Jennifer Tetzlaff and 2 more · BMJ · 2009 · 85,524 citations
Questions about this paper
How do I cite this paper in APA style?
Kaiser, I., Pfahlberg, A. B., Mathes, S., Uter, W. J. C., Diehl, K., Steeb, T., Heppt, M. V., & Gefeller, O. (2023). Inter-Rater Agreement in Assessing Risk of Bias in Melanoma Prediction Studies Using the Prediction Model Risk of Bias Assessment Tool (PROBAST): Results from a Controlled Experiment on the Effect of Specific Rater Training. Journal of Clinical Medicine, 12(5), 1976. https://doi.org/10.3390/jcm12051976
How many times has this paper been cited?
19 times, according to OpenAlex on 15 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access: read the free full text at Journal of Clinical Medicine.
Who wrote this paper?
Isabelle Kaiser, Annette B. Pfahlberg, Sonja Mathes, Wolfgang J. C. Uter, Katharina Diehl, Theresa Steeb, Markus Vincent Heppt, Olaf Gefeller.
What is the DOI of this paper?
10.3390/jcm12051976. The permanent link is https://doi.org/10.3390/jcm12051976.
Bibliographic data from OpenAlex (CC0), last checked 15 November 2026. ResearchGuru does not host this paper; the links go to the publisher or the repository that does.