Paper · 1998 · Open access

Assessing five common measures of interobserver reliability, proposing new refined measures

Shawn M. Chmil

Library and Archives Canada (Government of Canada) · Published 1 September 1998

Open full textCite this paper

In brief

“Assessing five common measures of interobserver reliability, proposing new refined measures” is a 1998 paper by Shawn M. Chmil, published in Library and Archives Canada (Government of Canada). OpenAlex counts 0 citations as of 18 November 2026. It is open access: the full text is free to read at Mspace (University of Manitoba).

Citations
0
References
31
Authors
1

Abstract

It is frequently desired to determine the extent of agreement between two raters when the data are measured on an ordinal scale. Five common measures of interobserver reliability are the overall proportion of agreement, Cohen's kappa, weighted kappa, the disagreement rate and the concordance between raters. A number of studies have assessed interobserver reliability including ones which have reservations about the measures of reliability and others which recognize several paradoxes. It is known that chance-corrected measures of agreement are prone to exhibit paradoxical and counter-intuitive results. Also, if measures are to be adjusted for chance agreement, then the guessing mechanism needs to be specified properly and precisely, as the current assumption that all observations are guessed is simply impractical. The inadequacies of these measures are discussed and, in light of their deficiencies, new measures are proposed. The assumption that some but not all observations are guessed is used to develop three new measures of interobserver reliability, namely, partial-chance proportion, partial-chance kappa and the expected-chance proportion. Simulations are used to compare the finite sample performance of these measures. In the simulations, the concordance between raters produced the best results, closely followed by partial-chance proportion, expected-chance proportion and partial-chance kappa, in terms of bias, efficiency and the empirical distributions of critical ratios. Recommended measures of interobserver reliability are the concordance between raters, partial-chance proportion, expected-chance proportion and partial-chance kappa. Although the concordance between raters is highly advised, its usage should be cautioned as it is based on assumptions that are impractical in clinical practice.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Chmil, S. M. (1998). Assessing five common measures of interobserver reliability, proposing new refined measures. Library and Archives Canada (Government of Canada). http://hdl.handle.net/1993/1368

MLA 9

Chmil, Shawn M. “Assessing five common measures of interobserver reliability, proposing new refined measures.” Library and Archives Canada (Government of Canada), 1998. http://hdl.handle.net/1993/1368.

Chicago

Chmil, Shawn M. 1998. “Assessing five common measures of interobserver reliability, proposing new refined measures.” Library and Archives Canada (Government of Canada). http://hdl.handle.net/1993/1368.

Harvard

Chmil, S. M. (1998) ‘Assessing five common measures of interobserver reliability, proposing new refined measures’, Library and Archives Canada (Government of Canada). Available at: http://hdl.handle.net/1993/1368.

IEEE

S. M. Chmil, “Assessing five common measures of interobserver reliability, proposing new refined measures,” Library and Archives Canada (Government of Canada), 1998.

BibTeX

@article{chmil1998assessing,
  title     = {{Assessing five common measures of interobserver reliability, proposing new refined measures}},
  author    = {Chmil, Shawn M.},
  journal   = {Library and Archives Canada (Government of Canada)},
  year      = {1998},
  url       = {http://hdl.handle.net/1993/1368}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement

  • inter-rater reliability
  • interobserver agreement
  • ordinal data
  • Cohen's kappa
  • weighted kappa
  • finite sample properties

Work with this paper

Most-cited papers in Decision Sciences

All highly cited Decision Sciences papers

Questions about this paper

How do I cite this paper in APA style?

Chmil, S. M. (1998). Assessing five common measures of interobserver reliability, proposing new refined measures. Library and Archives Canada (Government of Canada). http://hdl.handle.net/1993/1368

How many times has this paper been cited?

0 times, according to OpenAlex on 18 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at Mspace (University of Manitoba): sign in and open the full text from this page.

Who wrote this paper?

Shawn M. Chmil.

Bibliographic data from OpenAlex (CC0), last checked 18 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.