Journal article · 2023 · Open access

Inter-Rater Agreement in Assessing Risk of Bias in Melanoma Prediction Studies Using the Prediction Model Risk of Bias Assessment Tool (PROBAST): Results from a Controlled Experiment on the Effect of Specific Rater Training

Isabelle Kaiser, Annette B. Pfahlberg, Sonja Mathes, Wolfgang J. C. Uter, Katharina Diehl, Theresa Steeb, Markus Vincent Heppt, Olaf Gefeller

Journal of Clinical Medicine, 12(5), 1976 · Published 2 March 2023 · DOI 10.3390/jcm12051976

Free PDFView at the publisherCite this paper

In brief

“Inter-Rater Agreement in Assessing Risk of Bias in Melanoma Prediction Studies Using the Prediction Model Risk of Bias Assessment Tool (PROBAST): Results from a Controlled Experiment on the Effect of Specific Rater Training” is a 2023 journal article by Isabelle Kaiser and 7 co-authors, published in Journal of Clinical Medicine. OpenAlex counts 19 citations as of 15 November 2026, which puts it in the top 7% of works published in 2023 by citations. It is open access under a CC BY licence: the full text is free to read at Journal of Clinical Medicine.

Citations
19
Most cited in
2026 (8)
Field-weighted impact
1.86
References
81
Authors
8

Abstract

Assessing the risk of bias (ROB) of studies is an important part of the conduct of systematic reviews and meta-analyses in clinical medicine. Among the many existing ROB tools, the Prediction Model Risk of Bias Assessment Tool (PROBAST) is a rather new instrument specifically designed to assess the ROB of prediction studies. In our study we analyzed the inter-rater reliability (IRR) of PROBAST and the effect of specialized training on the IRR. Six raters independently assessed the risk of bias (ROB) of all melanoma risk prediction studies published until 2021 (n = 42) using the PROBAST instrument. The raters evaluated the ROB of the first 20 studies without any guidance other than the published PROBAST literature. The remaining 22 studies were assessed after receiving customized training and guidance. Gwet’s AC1 was used as the primary measure to quantify the pairwise and multi-rater IRR. Depending on the PROBAST domain, results before training showed a slight to moderate IRR (multi-rater AC1 ranging from 0.071 to 0.535). After training, the multi-rater AC1 ranged from 0.294 to 0.780 with a significant improvement for the overall ROB rating and two of the four domains. The largest net gain was achieved in the overall ROB rating (difference in multi-rater AC1: 0.405, 95%-CI 0.149–0.630). In conclusion, without targeted guidance, the IRR of PROBAST is low, questioning its use as an appropriate ROB instrument for prediction studies. Intensive training and guidance manuals with context-specific decision rules are needed to correctly apply and interpret the PROBAST instrument and to ensure consistency of ROB ratings.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Kaiser, I., Pfahlberg, A. B., Mathes, S., Uter, W. J. C., Diehl, K., Steeb, T., Heppt, M. V., & Gefeller, O. (2023). Inter-Rater Agreement in Assessing Risk of Bias in Melanoma Prediction Studies Using the Prediction Model Risk of Bias Assessment Tool (PROBAST): Results from a Controlled Experiment on the Effect of Specific Rater Training. Journal of Clinical Medicine, 12(5), 1976. https://doi.org/10.3390/jcm12051976

MLA 9

Kaiser, Isabelle, et al. “Inter-Rater Agreement in Assessing Risk of Bias in Melanoma Prediction Studies Using the Prediction Model Risk of Bias Assessment Tool (PROBAST): Results from a Controlled Experiment on the Effect of Specific Rater Training.” Journal of Clinical Medicine, vol. 12, no. 5, 2023, p. 1976. https://doi.org/10.3390/jcm12051976.

Chicago

Kaiser, Isabelle, Annette B. Pfahlberg, Sonja Mathes, Wolfgang J. C. Uter, Katharina Diehl, Theresa Steeb, Markus Vincent Heppt, and Olaf Gefeller. 2023. “Inter-Rater Agreement in Assessing Risk of Bias in Melanoma Prediction Studies Using the Prediction Model Risk of Bias Assessment Tool (PROBAST): Results from a Controlled Experiment on the Effect of Specific Rater Training.” Journal of Clinical Medicine 12 (5): 1976. https://doi.org/10.3390/jcm12051976.

Harvard

Kaiser, I. et al. (2023) ‘Inter-Rater Agreement in Assessing Risk of Bias in Melanoma Prediction Studies Using the Prediction Model Risk of Bias Assessment Tool (PROBAST): Results from a Controlled Experiment on the Effect of Specific Rater Training’, Journal of Clinical Medicine, 12(5), p. 1976. doi:10.3390/jcm12051976.

IEEE

I. Kaiser et al., “Inter-Rater Agreement in Assessing Risk of Bias in Melanoma Prediction Studies Using the Prediction Model Risk of Bias Assessment Tool (PROBAST): Results from a Controlled Experiment on the Effect of Specific Rater Training,” Journal of Clinical Medicine, vol. 12, no. 5, p. 1976, 2023, doi: 10.3390/jcm12051976.

BibTeX

@article{kaiser2023inter,
  title     = {{Inter-Rater Agreement in Assessing Risk of Bias in Melanoma Prediction Studies Using the Prediction Model Risk of Bias Assessment Tool (PROBAST): Results from a Controlled Experiment on the Effect of Specific Rater Training}},
  author    = {Kaiser, Isabelle and Pfahlberg, Annette B. and Mathes, Sonja and Uter, Wolfgang J. C. and Diehl, Katharina and Steeb, Theresa and Heppt, Markus Vincent and Gefeller, Olaf},
  journal   = {Journal of Clinical Medicine},
  year      = {2023},
  volume    = {12},
  number    = {5},
  pages     = {1976},
  doi       = {10.3390/jcm12051976},
  url       = {https://doi.org/10.3390/jcm12051976}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Citations by year

YearCitationsShare of the best year
202322
202444
202555
202688

Its field-weighted citation impact is 1.86: it has been cited 1.86 times as often as the average work of the same field, type and year (1.00 is the world average).

Topic and keywords

Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews

  • risk of bias
  • PROBAST
  • inter-rater reliability
  • melanoma risk prediction
  • rater training
  • systematic review methodology
  • Gwet's AC1

Work with this paper

Most-cited papers in Decision Sciences

All highly cited Decision Sciences papers

Questions about this paper

How do I cite this paper in APA style?

Kaiser, I., Pfahlberg, A. B., Mathes, S., Uter, W. J. C., Diehl, K., Steeb, T., Heppt, M. V., & Gefeller, O. (2023). Inter-Rater Agreement in Assessing Risk of Bias in Melanoma Prediction Studies Using the Prediction Model Risk of Bias Assessment Tool (PROBAST): Results from a Controlled Experiment on the Effect of Specific Rater Training. Journal of Clinical Medicine, 12(5), 1976. https://doi.org/10.3390/jcm12051976

How many times has this paper been cited?

19 times, according to OpenAlex on 15 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access: read the free full text at Journal of Clinical Medicine.

Who wrote this paper?

Isabelle Kaiser, Annette B. Pfahlberg, Sonja Mathes, Wolfgang J. C. Uter, Katharina Diehl, Theresa Steeb, Markus Vincent Heppt, Olaf Gefeller.

What is the DOI of this paper?

10.3390/jcm12051976. The permanent link is https://doi.org/10.3390/jcm12051976.

Bibliographic data from OpenAlex (CC0), last checked 15 November 2026. ResearchGuru does not host this paper; the links go to the publisher or the repository that does.