Journal article · 2015 · Open access
Measures of agreement between many raters for ordinal classifications
Statistics in Medicine, 34(23), 3116–3132 · Published 21 June 2015 · DOI 10.1002/sim.6546
In brief
“Measures of agreement between many raters for ordinal classifications” is a 2015 journal article by Kerrie P. Nelson and Don Edwards, published in Statistics in Medicine. OpenAlex counts 84 citations as of 18 November 2026. It is open access: the full text is free to read at Europe PMC (PubMed Central).
- Citations
- 84
- Most cited in
- 2023 (14)
- Field-weighted impact
- 2.00
- References
- 54
- Authors
- 2
Abstract
Screening and diagnostic procedures often require a physician's subjective interpretation of a patient's test result using an ordered categorical scale to define the patient's disease severity. Because of wide variability observed between physicians' ratings, many large-scale studies have been conducted to quantify agreement between multiple experts' ordinal classifications in common diagnostic procedures such as mammography. However, very few statistical approaches are available to assess agreement in these large-scale settings. Many existing summary measures of agreement rely on extensions of Cohen's kappa. These are prone to prevalence and marginal distribution issues, become increasingly complex for more than three experts, or are not easily implemented. Here we propose a model-based approach to assess agreement in large-scale studies based upon a framework of ordinal generalized linear mixed models. A summary measure of agreement is proposed for multiple experts assessing the same sample of patients' test results according to an ordered categorical scale. This measure avoids some of the key flaws associated with Cohen's kappa and its extensions. Simulation studies are conducted to demonstrate the validity of the approach with comparison with commonly used agreement measures. The proposed methods are easily implemented using the software package R and are applied to two large-scale cancer agreement studies.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Nelson, K. P., & Edwards, D. (2015). Measures of agreement between many raters for ordinal classifications. Statistics in Medicine, 34(23), 3116–3132. https://doi.org/10.1002/sim.6546
MLA 9
Nelson, Kerrie P., and Don Edwards. “Measures of agreement between many raters for ordinal classifications.” Statistics in Medicine, vol. 34, no. 23, 2015, pp. 3116–3132. https://doi.org/10.1002/sim.6546.
Chicago
Nelson, Kerrie P. and Don Edwards. 2015. “Measures of agreement between many raters for ordinal classifications.” Statistics in Medicine 34 (23): 3116–3132. https://doi.org/10.1002/sim.6546.
Harvard
Nelson, K. P. and Edwards, D. (2015) ‘Measures of agreement between many raters for ordinal classifications’, Statistics in Medicine, 34(23), pp. 3116–3132. doi:10.1002/sim.6546.
IEEE
K. P. Nelson and D. Edwards, “Measures of agreement between many raters for ordinal classifications,” Statistics in Medicine, vol. 34, no. 23, pp. 3116–3132, 2015, doi: 10.1002/sim.6546.
BibTeX
@article{nelson2015measures,
title = {{Measures of agreement between many raters for ordinal classifications}},
author = {Nelson, Kerrie P. and Edwards, Don},
journal = {Statistics in Medicine},
year = {2015},
volume = {34},
number = {23},
pages = {3116--3132},
doi = {10.1002/sim.6546},
url = {https://doi.org/10.1002/sim.6546}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Citations by year
| Year | Citations | Share of the best year |
|---|---|---|
| 2017 | 4 | |
| 2018 | 9 | |
| 2019 | 13 | |
| 2020 | 10 | |
| 2021 | 7 | |
| 2022 | 5 | |
| 2023 | 14 | |
| 2024 | 4 | |
| 2025 | 10 | |
| 2026 | 4 |
Its field-weighted citation impact is 2.00: it has been cited 2.00 times as often as the average work of the same field, type and year (1.00 is the world average).
Topic and keywords
Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- interobserver agreement
- ordinal classification
- multiple raters
- Cohen's kappa
- mammography interpretation
- cancer diagnosis
- disease severity assessment
Work with this paper
Most-cited papers in Decision Sciences
- The PRISMA 2020 statement: an updated guideline for reporting systematic reviewsMatthew James Page, Joanne E. McKenzie, Patrick M. Bossuyt and 23 more · BMJ · 2021 · 144,919 citations
- Evaluating Structural Equation Models with Unobservable Variables and Measurement ErrorClaes Fornell, David F. Larcker · Journal of Marketing Research · 1981 · 118,336 citations
- Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternativesLi‐tze Hu, Peter M. Bentler · Structural Equation Modeling A Multidisciplinary Journal · 1999 · 117,461 citations
- Common method biases in behavioral research: A critical review of the literature and recommended remedies.Philip M. Podsakoff, SCOTT B. MACKENZIE, Jeong Yeon Lee and 1 more · Journal of Applied Psychology · 2003 · 89,248 citations
- Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statementDavid Moher, Alessandro Liberati, Jennifer Tetzlaff and 2 more · BMJ · 2009 · 85,524 citations
Questions about this paper
How do I cite this paper in APA style?
Nelson, K. P., & Edwards, D. (2015). Measures of agreement between many raters for ordinal classifications. Statistics in Medicine, 34(23), 3116–3132. https://doi.org/10.1002/sim.6546
How many times has this paper been cited?
84 times, according to OpenAlex on 18 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at Europe PMC (PubMed Central): sign in and open the full text from this page.
Who wrote this paper?
Kerrie P. Nelson, Don Edwards.
What is the DOI of this paper?
10.1002/sim.6546, permanently at https://doi.org/10.1002/sim.6546. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 18 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.