Journal article · 2021 · Open access
A Comparison of Reliability Coefficients for Ordinal Rating Scales
Journal of Classification, 38(3), 519–543 · Published 22 April 2021 · DOI 10.1007/s00357-021-09386-5
In brief
“A Comparison of Reliability Coefficients for Ordinal Rating Scales” is a 2021 journal article by Alexandra de Raadt and 3 co-authors, published in Journal of Classification. OpenAlex counts 173 citations as of 18 November 2026. It is open access under a CC BY licence: the full text is free to read at Journal of Classification.
- Citations
- 173
- Most cited in
- 2025 (47)
- Field-weighted impact
- 6.05
- References
- 72
- Authors
- 4
Abstract
Kappa coefficients are commonly used for quantifying reliability on a categorical scale, whereas correlation coefficients are commonly applied to assess reliability on an interval scale. Both types of coefficients can be used to assess the reliability of ordinal rating scales. In this study, we compare seven reliability coefficients for ordinal rating scales: the kappa coefficients included are Cohen’s kappa, linearly weighted kappa, and quadratically weighted kappa; the correlation coefficients included are intraclass correlation ICC(3,1), Pearson’s correlation, Spearman’s rho, and Kendall’s tau-b. The primary goal is to provide a thorough understanding of these coefficients such that the applied researcher can make a sensible choice for ordinal rating scales. A second aim is to find out whether the choice of the coefficient matters. We studied to what extent we reach the same conclusions about inter-rater reliability with different coefficients, and to what extent the coefficients measure agreement in a similar way, using analytic methods, and simulated and empirical data. Using analytical methods, it is shown that differences between quadratic kappa and the Pearson and intraclass correlations increase if agreement becomes larger. Differences between the three coefficients are generally small if differences between rater means and variances are small. Furthermore, using simulated and empirical data, it is shown that differences between all reliability coefficients tend to increase if agreement between the raters increases. Moreover, for the data in this study, the same conclusion about inter-rater reliability was reached in virtually all cases with the four correlation coefficients. In addition, using quadratically weighted kappa, we reached a similar conclusion as with any correlation coefficient a great number of times. Hence, for the data in this study, it does not really matter which of these five coefficients is used. Moreover, the four correlation coefficients and quadratically weighted kappa tend to measure agreement in a similar way: their values are very highly correlated for the data in this study.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
de Raadt, A., Warrens, M. J., Bosker, R., & Kiers, H. A. L. (2021). A Comparison of Reliability Coefficients for Ordinal Rating Scales. Journal of Classification, 38(3), 519–543. https://doi.org/10.1007/s00357-021-09386-5
MLA 9
de Raadt, Alexandra, et al. “A Comparison of Reliability Coefficients for Ordinal Rating Scales.” Journal of Classification, vol. 38, no. 3, 2021, pp. 519–543. https://doi.org/10.1007/s00357-021-09386-5.
Chicago
de Raadt, Alexandra, Matthijs J. Warrens, Roel Bosker, and Henk A. L. Kiers. 2021. “A Comparison of Reliability Coefficients for Ordinal Rating Scales.” Journal of Classification 38 (3): 519–543. https://doi.org/10.1007/s00357-021-09386-5.
Harvard
de Raadt, A. et al. (2021) ‘A Comparison of Reliability Coefficients for Ordinal Rating Scales’, Journal of Classification, 38(3), pp. 519–543. doi:10.1007/s00357-021-09386-5.
IEEE
A. de Raadt, M. J. Warrens, R. Bosker, and H. A. L. Kiers, “A Comparison of Reliability Coefficients for Ordinal Rating Scales,” Journal of Classification, vol. 38, no. 3, pp. 519–543, 2021, doi: 10.1007/s00357-021-09386-5.
BibTeX
@article{deraadt2021comparison,
title = {{A Comparison of Reliability Coefficients for Ordinal Rating Scales}},
author = {de Raadt, Alexandra and Warrens, Matthijs J. and Bosker, Roel and Kiers, Henk A. L.},
journal = {Journal of Classification},
year = {2021},
volume = {38},
number = {3},
pages = {519--543},
doi = {10.1007/s00357-021-09386-5},
url = {https://doi.org/10.1007/s00357-021-09386-5}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Citations by year
| Year | Citations | Share of the best year |
|---|---|---|
| 2021 | 1 | |
| 2022 | 11 | |
| 2023 | 35 | |
| 2024 | 32 | |
| 2025 | 47 | |
| 2026 | 46 |
Its field-weighted citation impact is 6.05: it has been cited 6.05 times as often as the average work of the same field, type and year (1.00 is the world average).
Topic and keywords
Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- inter-rater reliability
- ordinal rating scales
- kappa statistic
- weighted kappa
- intraclass correlation coefficient
- Pearson correlation
- Spearman correlation
- Kendall's tau-b
Work with this paper
Most-cited papers in Decision Sciences
- The PRISMA 2020 statement: an updated guideline for reporting systematic reviewsMatthew James Page, Joanne E. McKenzie, Patrick M. Bossuyt and 23 more · BMJ · 2021 · 144,919 citations
- Evaluating Structural Equation Models with Unobservable Variables and Measurement ErrorClaes Fornell, David F. Larcker · Journal of Marketing Research · 1981 · 118,336 citations
- Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternativesLi‐tze Hu, Peter M. Bentler · Structural Equation Modeling A Multidisciplinary Journal · 1999 · 117,461 citations
- Common method biases in behavioral research: A critical review of the literature and recommended remedies.Philip M. Podsakoff, SCOTT B. MACKENZIE, Jeong Yeon Lee and 1 more · Journal of Applied Psychology · 2003 · 89,248 citations
- Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statementDavid Moher, Alessandro Liberati, Jennifer Tetzlaff and 2 more · BMJ · 2009 · 85,524 citations
Questions about this paper
How do I cite this paper in APA style?
de Raadt, A., Warrens, M. J., Bosker, R., & Kiers, H. A. L. (2021). A Comparison of Reliability Coefficients for Ordinal Rating Scales. Journal of Classification, 38(3), 519–543. https://doi.org/10.1007/s00357-021-09386-5
How many times has this paper been cited?
173 times, according to OpenAlex on 18 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at Journal of Classification: sign in and open the PDF from this page.
Who wrote this paper?
Alexandra de Raadt, Matthijs J. Warrens, Roel Bosker, Henk A. L. Kiers.
What is the DOI of this paper?
10.1007/s00357-021-09386-5, permanently at https://doi.org/10.1007/s00357-021-09386-5. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 18 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.