Journal article · 2024 · Open access
Statistical inference for agreement between multiple raters on a binary scale
British Journal of Mathematical and Statistical Psychology, 77(2), 245–260 · Published 17 January 2024 · DOI 10.1111/bmsp.12333
In brief
“Statistical inference for agreement between multiple raters on a binary scale” is a 2024 journal article by Sophie Vanbelle, published in British Journal of Mathematical and Statistical Psychology. OpenAlex counts 7 citations as of 15 November 2026, which puts it in the top 7% of works published in 2024 by citations. It is open access under a CC BY-NC licence: the full text is free to read at British Journal of Mathematical and Statistical Psychology.
- Citations
- 7
- Most cited in
- 2024 (3)
- Field-weighted impact
- 0.75
- References
- 42
- Authors
- 1
Abstract
Agreement studies often involve more than two raters or repeated measurements. In the presence of two raters, the proportion of agreement and of positive agreement are simple and popular agreement measures for binary scales. These measures were generalized to agreement studies involving more than two raters with statistical inference procedures proposed on an empirical basis. We present two alternatives. The first is a Wald confidence interval using standard errors obtained by the delta method. The second involves Bayesian statistical inference not requiring any specific Bayesian software. These new procedures show better statistical behaviour than the confidence intervals initially proposed. In addition, we provide analytical formulas to determine the minimum number of persons needed for a given number of raters when planning an agreement study. All methods are implemented in the R package simpleagree and the Shiny app simpleagree.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Vanbelle, S. (2024). Statistical inference for agreement between multiple raters on a binary scale. British Journal of Mathematical and Statistical Psychology, 77(2), 245–260. https://doi.org/10.1111/bmsp.12333
MLA 9
Vanbelle, Sophie. “Statistical inference for agreement between multiple raters on a binary scale.” British Journal of Mathematical and Statistical Psychology, vol. 77, no. 2, 2024, pp. 245–260. https://doi.org/10.1111/bmsp.12333.
Chicago
Vanbelle, Sophie. 2024. “Statistical inference for agreement between multiple raters on a binary scale.” British Journal of Mathematical and Statistical Psychology 77 (2): 245–260. https://doi.org/10.1111/bmsp.12333.
Harvard
Vanbelle, S. (2024) ‘Statistical inference for agreement between multiple raters on a binary scale’, British Journal of Mathematical and Statistical Psychology, 77(2), pp. 245–260. doi:10.1111/bmsp.12333.
IEEE
S. Vanbelle, “Statistical inference for agreement between multiple raters on a binary scale,” British Journal of Mathematical and Statistical Psychology, vol. 77, no. 2, pp. 245–260, 2024, doi: 10.1111/bmsp.12333.
BibTeX
@article{vanbelle2024statistical,
title = {{Statistical inference for agreement between multiple raters on a binary scale}},
author = {Vanbelle, Sophie},
journal = {British Journal of Mathematical and Statistical Psychology},
year = {2024},
volume = {77},
number = {2},
pages = {245--260},
doi = {10.1111/bmsp.12333},
url = {https://doi.org/10.1111/bmsp.12333}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Citations by year
| Year | Citations | Share of the best year |
|---|---|---|
| 2024 | 3 | |
| 2025 | 2 | |
| 2026 | 2 |
Its field-weighted citation impact is 0.75: it has been cited 0.75 times as often as the average work of the same field, type and year (1.00 is the world average).
Topic and keywords
Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- interobserver agreement
- multiple raters
- Wald confidence intervals
- delta method
- Bayesian inference
- sample size determination
Work with this paper
Most-cited papers in Decision Sciences
- The PRISMA 2020 statement: an updated guideline for reporting systematic reviewsMatthew James Page, Joanne E. McKenzie, Patrick M. Bossuyt and 23 more · BMJ · 2021 · 144,919 citations
- Evaluating Structural Equation Models with Unobservable Variables and Measurement ErrorClaes Fornell, David F. Larcker · Journal of Marketing Research · 1981 · 118,336 citations
- Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternativesLi‐tze Hu, Peter M. Bentler · Structural Equation Modeling A Multidisciplinary Journal · 1999 · 117,461 citations
- Common method biases in behavioral research: A critical review of the literature and recommended remedies.Philip M. Podsakoff, SCOTT B. MACKENZIE, Jeong Yeon Lee and 1 more · Journal of Applied Psychology · 2003 · 89,248 citations
- Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statementDavid Moher, Alessandro Liberati, Jennifer Tetzlaff and 2 more · BMJ · 2009 · 85,524 citations
Questions about this paper
How do I cite this paper in APA style?
Vanbelle, S. (2024). Statistical inference for agreement between multiple raters on a binary scale. British Journal of Mathematical and Statistical Psychology, 77(2), 245–260. https://doi.org/10.1111/bmsp.12333
How many times has this paper been cited?
7 times, according to OpenAlex on 15 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access: read the free full text at British Journal of Mathematical and Statistical Psychology.
Who wrote this paper?
Sophie Vanbelle.
What is the DOI of this paper?
10.1111/bmsp.12333. The permanent link is https://doi.org/10.1111/bmsp.12333.
Bibliographic data from OpenAlex (CC0), last checked 15 November 2026. ResearchGuru does not host this paper; the links go to the publisher or the repository that does.