Paper · 1998 · Open access
Assessing five common measures of interobserver reliability, proposing new refined measures
Library and Archives Canada (Government of Canada) · Published 1 September 1998
In brief
“Assessing five common measures of interobserver reliability, proposing new refined measures” is a 1998 paper by Shawn M. Chmil, published in Library and Archives Canada (Government of Canada). OpenAlex counts 0 citations as of 18 November 2026. It is open access: the full text is free to read at Mspace (University of Manitoba).
- Citations
- 0
- References
- 31
- Authors
- 1
Abstract
It is frequently desired to determine the extent of agreement between two raters when the data are measured on an ordinal scale. Five common measures of interobserver reliability are the overall proportion of agreement, Cohen's kappa, weighted kappa, the disagreement rate and the concordance between raters. A number of studies have assessed interobserver reliability including ones which have reservations about the measures of reliability and others which recognize several paradoxes. It is known that chance-corrected measures of agreement are prone to exhibit paradoxical and counter-intuitive results. Also, if measures are to be adjusted for chance agreement, then the guessing mechanism needs to be specified properly and precisely, as the current assumption that all observations are guessed is simply impractical. The inadequacies of these measures are discussed and, in light of their deficiencies, new measures are proposed. The assumption that some but not all observations are guessed is used to develop three new measures of interobserver reliability, namely, partial-chance proportion, partial-chance kappa and the expected-chance proportion. Simulations are used to compare the finite sample performance of these measures. In the simulations, the concordance between raters produced the best results, closely followed by partial-chance proportion, expected-chance proportion and partial-chance kappa, in terms of bias, efficiency and the empirical distributions of critical ratios. Recommended measures of interobserver reliability are the concordance between raters, partial-chance proportion, expected-chance proportion and partial-chance kappa. Although the concordance between raters is highly advised, its usage should be cautioned as it is based on assumptions that are impractical in clinical practice.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Chmil, S. M. (1998). Assessing five common measures of interobserver reliability, proposing new refined measures. Library and Archives Canada (Government of Canada). http://hdl.handle.net/1993/1368
MLA 9
Chmil, Shawn M. “Assessing five common measures of interobserver reliability, proposing new refined measures.” Library and Archives Canada (Government of Canada), 1998. http://hdl.handle.net/1993/1368.
Chicago
Chmil, Shawn M. 1998. “Assessing five common measures of interobserver reliability, proposing new refined measures.” Library and Archives Canada (Government of Canada). http://hdl.handle.net/1993/1368.
Harvard
Chmil, S. M. (1998) ‘Assessing five common measures of interobserver reliability, proposing new refined measures’, Library and Archives Canada (Government of Canada). Available at: http://hdl.handle.net/1993/1368.
IEEE
S. M. Chmil, “Assessing five common measures of interobserver reliability, proposing new refined measures,” Library and Archives Canada (Government of Canada), 1998.
BibTeX
@article{chmil1998assessing,
title = {{Assessing five common measures of interobserver reliability, proposing new refined measures}},
author = {Chmil, Shawn M.},
journal = {Library and Archives Canada (Government of Canada)},
year = {1998},
url = {http://hdl.handle.net/1993/1368}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- inter-rater reliability
- interobserver agreement
- ordinal data
- Cohen's kappa
- weighted kappa
- finite sample properties
Work with this paper
Most-cited papers in Decision Sciences
- The PRISMA 2020 statement: an updated guideline for reporting systematic reviewsMatthew James Page, Joanne E. McKenzie, Patrick M. Bossuyt and 23 more · BMJ · 2021 · 144,919 citations
- Evaluating Structural Equation Models with Unobservable Variables and Measurement ErrorClaes Fornell, David F. Larcker · Journal of Marketing Research · 1981 · 118,336 citations
- Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternativesLi‐tze Hu, Peter M. Bentler · Structural Equation Modeling A Multidisciplinary Journal · 1999 · 117,461 citations
- Common method biases in behavioral research: A critical review of the literature and recommended remedies.Philip M. Podsakoff, SCOTT B. MACKENZIE, Jeong Yeon Lee and 1 more · Journal of Applied Psychology · 2003 · 89,248 citations
- Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statementDavid Moher, Alessandro Liberati, Jennifer Tetzlaff and 2 more · BMJ · 2009 · 85,524 citations
Questions about this paper
How do I cite this paper in APA style?
Chmil, S. M. (1998). Assessing five common measures of interobserver reliability, proposing new refined measures. Library and Archives Canada (Government of Canada). http://hdl.handle.net/1993/1368
How many times has this paper been cited?
0 times, according to OpenAlex on 18 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at Mspace (University of Manitoba): sign in and open the full text from this page.
Who wrote this paper?
Shawn M. Chmil.
Bibliographic data from OpenAlex (CC0), last checked 18 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.