Journal article · 2021 · Open access

A Comparison of Reliability Coefficients for Ordinal Rating Scales

Alexandra de Raadt, Matthijs J. Warrens, Roel Bosker, Henk A. L. Kiers

Journal of Classification, 38(3), 519–543 · Published 22 April 2021 · DOI 10.1007/s00357-021-09386-5

Open PDFView at the publisherCite this paper

In brief

“A Comparison of Reliability Coefficients for Ordinal Rating Scales” is a 2021 journal article by Alexandra de Raadt and 3 co-authors, published in Journal of Classification. OpenAlex counts 173 citations as of 18 November 2026. It is open access under a CC BY licence: the full text is free to read at Journal of Classification.

Citations
173
Most cited in
2025 (47)
Field-weighted impact
6.05
References
72
Authors
4

Abstract

Kappa coefficients are commonly used for quantifying reliability on a categorical scale, whereas correlation coefficients are commonly applied to assess reliability on an interval scale. Both types of coefficients can be used to assess the reliability of ordinal rating scales. In this study, we compare seven reliability coefficients for ordinal rating scales: the kappa coefficients included are Cohen’s kappa, linearly weighted kappa, and quadratically weighted kappa; the correlation coefficients included are intraclass correlation ICC(3,1), Pearson’s correlation, Spearman’s rho, and Kendall’s tau-b. The primary goal is to provide a thorough understanding of these coefficients such that the applied researcher can make a sensible choice for ordinal rating scales. A second aim is to find out whether the choice of the coefficient matters. We studied to what extent we reach the same conclusions about inter-rater reliability with different coefficients, and to what extent the coefficients measure agreement in a similar way, using analytic methods, and simulated and empirical data. Using analytical methods, it is shown that differences between quadratic kappa and the Pearson and intraclass correlations increase if agreement becomes larger. Differences between the three coefficients are generally small if differences between rater means and variances are small. Furthermore, using simulated and empirical data, it is shown that differences between all reliability coefficients tend to increase if agreement between the raters increases. Moreover, for the data in this study, the same conclusion about inter-rater reliability was reached in virtually all cases with the four correlation coefficients. In addition, using quadratically weighted kappa, we reached a similar conclusion as with any correlation coefficient a great number of times. Hence, for the data in this study, it does not really matter which of these five coefficients is used. Moreover, the four correlation coefficients and quadratically weighted kappa tend to measure agreement in a similar way: their values are very highly correlated for the data in this study.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

de Raadt, A., Warrens, M. J., Bosker, R., & Kiers, H. A. L. (2021). A Comparison of Reliability Coefficients for Ordinal Rating Scales. Journal of Classification, 38(3), 519–543. https://doi.org/10.1007/s00357-021-09386-5

MLA 9

de Raadt, Alexandra, et al. “A Comparison of Reliability Coefficients for Ordinal Rating Scales.” Journal of Classification, vol. 38, no. 3, 2021, pp. 519–543. https://doi.org/10.1007/s00357-021-09386-5.

Chicago

de Raadt, Alexandra, Matthijs J. Warrens, Roel Bosker, and Henk A. L. Kiers. 2021. “A Comparison of Reliability Coefficients for Ordinal Rating Scales.” Journal of Classification 38 (3): 519–543. https://doi.org/10.1007/s00357-021-09386-5.

Harvard

de Raadt, A. et al. (2021) ‘A Comparison of Reliability Coefficients for Ordinal Rating Scales’, Journal of Classification, 38(3), pp. 519–543. doi:10.1007/s00357-021-09386-5.

IEEE

A. de Raadt, M. J. Warrens, R. Bosker, and H. A. L. Kiers, “A Comparison of Reliability Coefficients for Ordinal Rating Scales,” Journal of Classification, vol. 38, no. 3, pp. 519–543, 2021, doi: 10.1007/s00357-021-09386-5.

BibTeX

@article{deraadt2021comparison,
  title     = {{A Comparison of Reliability Coefficients for Ordinal Rating Scales}},
  author    = {de Raadt, Alexandra and Warrens, Matthijs J. and Bosker, Roel and Kiers, Henk A. L.},
  journal   = {Journal of Classification},
  year      = {2021},
  volume    = {38},
  number    = {3},
  pages     = {519--543},
  doi       = {10.1007/s00357-021-09386-5},
  url       = {https://doi.org/10.1007/s00357-021-09386-5}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Citations by year

YearCitationsShare of the best year
202111
20221111
20233535
20243232
20254747
20264646

Its field-weighted citation impact is 6.05: it has been cited 6.05 times as often as the average work of the same field, type and year (1.00 is the world average).

Topic and keywords

Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement

  • inter-rater reliability
  • ordinal rating scales
  • kappa statistic
  • weighted kappa
  • intraclass correlation coefficient
  • Pearson correlation
  • Spearman correlation
  • Kendall's tau-b

Work with this paper

Most-cited papers in Decision Sciences

All highly cited Decision Sciences papers

Questions about this paper

How do I cite this paper in APA style?

de Raadt, A., Warrens, M. J., Bosker, R., & Kiers, H. A. L. (2021). A Comparison of Reliability Coefficients for Ordinal Rating Scales. Journal of Classification, 38(3), 519–543. https://doi.org/10.1007/s00357-021-09386-5

How many times has this paper been cited?

173 times, according to OpenAlex on 18 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at Journal of Classification: sign in and open the PDF from this page.

Who wrote this paper?

Alexandra de Raadt, Matthijs J. Warrens, Roel Bosker, Henk A. L. Kiers.

What is the DOI of this paper?

10.1007/s00357-021-09386-5, permanently at https://doi.org/10.1007/s00357-021-09386-5. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 18 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.