Journal article · 2024 · Open access

Statistical inference for agreement between multiple raters on a binary scale

Sophie Vanbelle

British Journal of Mathematical and Statistical Psychology, 77(2), 245–260 · Published 17 January 2024 · DOI 10.1111/bmsp.12333

Free PDFView at the publisherCite this paper

In brief

“Statistical inference for agreement between multiple raters on a binary scale” is a 2024 journal article by Sophie Vanbelle, published in British Journal of Mathematical and Statistical Psychology. OpenAlex counts 7 citations as of 15 November 2026, which puts it in the top 7% of works published in 2024 by citations. It is open access under a CC BY-NC licence: the full text is free to read at British Journal of Mathematical and Statistical Psychology.

Citations
7
Most cited in
2024 (3)
Field-weighted impact
0.75
References
42
Authors
1

Abstract

Agreement studies often involve more than two raters or repeated measurements. In the presence of two raters, the proportion of agreement and of positive agreement are simple and popular agreement measures for binary scales. These measures were generalized to agreement studies involving more than two raters with statistical inference procedures proposed on an empirical basis. We present two alternatives. The first is a Wald confidence interval using standard errors obtained by the delta method. The second involves Bayesian statistical inference not requiring any specific Bayesian software. These new procedures show better statistical behaviour than the confidence intervals initially proposed. In addition, we provide analytical formulas to determine the minimum number of persons needed for a given number of raters when planning an agreement study. All methods are implemented in the R package simpleagree and the Shiny app simpleagree.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Vanbelle, S. (2024). Statistical inference for agreement between multiple raters on a binary scale. British Journal of Mathematical and Statistical Psychology, 77(2), 245–260. https://doi.org/10.1111/bmsp.12333

MLA 9

Vanbelle, Sophie. “Statistical inference for agreement between multiple raters on a binary scale.” British Journal of Mathematical and Statistical Psychology, vol. 77, no. 2, 2024, pp. 245–260. https://doi.org/10.1111/bmsp.12333.

Chicago

Vanbelle, Sophie. 2024. “Statistical inference for agreement between multiple raters on a binary scale.” British Journal of Mathematical and Statistical Psychology 77 (2): 245–260. https://doi.org/10.1111/bmsp.12333.

Harvard

Vanbelle, S. (2024) ‘Statistical inference for agreement between multiple raters on a binary scale’, British Journal of Mathematical and Statistical Psychology, 77(2), pp. 245–260. doi:10.1111/bmsp.12333.

IEEE

S. Vanbelle, “Statistical inference for agreement between multiple raters on a binary scale,” British Journal of Mathematical and Statistical Psychology, vol. 77, no. 2, pp. 245–260, 2024, doi: 10.1111/bmsp.12333.

BibTeX

@article{vanbelle2024statistical,
  title     = {{Statistical inference for agreement between multiple raters on a binary scale}},
  author    = {Vanbelle, Sophie},
  journal   = {British Journal of Mathematical and Statistical Psychology},
  year      = {2024},
  volume    = {77},
  number    = {2},
  pages     = {245--260},
  doi       = {10.1111/bmsp.12333},
  url       = {https://doi.org/10.1111/bmsp.12333}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Citations by year

YearCitationsShare of the best year
202433
202522
202622

Its field-weighted citation impact is 0.75: it has been cited 0.75 times as often as the average work of the same field, type and year (1.00 is the world average).

Topic and keywords

Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement

  • interobserver agreement
  • multiple raters
  • Wald confidence intervals
  • delta method
  • Bayesian inference
  • sample size determination

Work with this paper

Most-cited papers in Decision Sciences

All highly cited Decision Sciences papers

Questions about this paper

How do I cite this paper in APA style?

Vanbelle, S. (2024). Statistical inference for agreement between multiple raters on a binary scale. British Journal of Mathematical and Statistical Psychology, 77(2), 245–260. https://doi.org/10.1111/bmsp.12333

How many times has this paper been cited?

7 times, according to OpenAlex on 15 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access: read the free full text at British Journal of Mathematical and Statistical Psychology.

Who wrote this paper?

Sophie Vanbelle.

What is the DOI of this paper?

10.1111/bmsp.12333. The permanent link is https://doi.org/10.1111/bmsp.12333.

Bibliographic data from OpenAlex (CC0), last checked 15 November 2026. ResearchGuru does not host this paper; the links go to the publisher or the repository that does.