Dissertation · 2004 · Open access

A Comparative Study of Interrater Reliability Coefficients Obtained from Different Statistical Procedures Using Monte Carlo Simulation Techniques

Ebrima Nying

ScholarWorks - WMU (Western Michigan University) · Published 1 January 2004

Free PDFView at the publisherCite this paper

In brief

“A Comparative Study of Interrater Reliability Coefficients Obtained from Different Statistical Procedures Using Monte Carlo Simulation Techniques” is a 2004 dissertation by Ebrima Nying, published in ScholarWorks - WMU (Western Michigan University). OpenAlex counts 0 citations as of 16 November 2026. It is open access: the full text is free to read at ScholarWorks - WMU (Western Michigan University).

Citations
0
References
20
Authors
1

Abstract

Reliability estimation is a key research component within the global area of educational assessment. The literature reports numerous studies using different statistical techniques for estimating reliability of educational measures. However, few have focused on the estimation of interrater reliability of performance assessment (Abedi, Baker, & Herl, 1995). Specifically, this study compared three different methods for estimating interrater reliability to determine if there are differences among these estimates as a function of: sample size, measurement scale, number of raters and the theoretical population reliability (rho). The three methods of estimation were the Intraclass Correlation (ICC(2, k )) (Shrout & Fleiss, 1979), Kendall's Coefficient of Concordance (Kendall, 1938) and Kappa for Multiple Raters (Fleiss, 1971).\nThis study employed a 4-between (three measurement scales, four sample sizes, four rater groups, and two population rho's), 1-within (three estimation methods) mixed design. ANOVA results indicated two significant 4-way interactions involving measure*Mscale* rater*rho and measure*Mscale* size*rho. Simple effect analyses of the 4-way interactions focused on each level of rho and parallel 3-way ANOVAs (measure*Mscale*size or rater) were conducted. Post hoc analyses indicated that both 3-way interactions were significant. To further understand these 3-way interactions, they were followed by 2-way designs holding Measure constant.\nFinally, Tukey HSD tests were then examined. This post hoc strategy consistently showed mean differences in the reliability estimates among the differentmeasurement scales as a function of both number of raters and sample size regardless of rho.\nRecommendations for future research and limitations of this study are provided. Although population rho had a predictable effect, the present findings should be replicated and extended with different values of population rho and since this study only considered normally distributed data, the influence of population distribution should be examined, especially with response scales having fewer levels.\nPractitioners and researchers who are faced with estimating interrater reliability should be comforted by the striking stability of these methods. When the data are dichotomous, Kappa for Multiple Raters is recommended, and when the data are ordinal, Kendall's Coefficient of Concordance is recommended.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Nying, E. (2004). A Comparative Study of Interrater Reliability Coefficients Obtained from Different Statistical Procedures Using Monte Carlo Simulation Techniques. ScholarWorks - WMU (Western Michigan University). https://scholarworks.wmich.edu/dissertations/1267

MLA 9

Nying, Ebrima. “A Comparative Study of Interrater Reliability Coefficients Obtained from Different Statistical Procedures Using Monte Carlo Simulation Techniques.” ScholarWorks - WMU (Western Michigan University), 2004. https://scholarworks.wmich.edu/dissertations/1267.

Chicago

Nying, Ebrima. 2004. “A Comparative Study of Interrater Reliability Coefficients Obtained from Different Statistical Procedures Using Monte Carlo Simulation Techniques.” ScholarWorks - WMU (Western Michigan University). https://scholarworks.wmich.edu/dissertations/1267.

Harvard

Nying, E. (2004) ‘A Comparative Study of Interrater Reliability Coefficients Obtained from Different Statistical Procedures Using Monte Carlo Simulation Techniques’, ScholarWorks - WMU (Western Michigan University). Available at: https://scholarworks.wmich.edu/dissertations/1267.

IEEE

E. Nying, “A Comparative Study of Interrater Reliability Coefficients Obtained from Different Statistical Procedures Using Monte Carlo Simulation Techniques,” ScholarWorks - WMU (Western Michigan University), 2004.

BibTeX

@phdthesis{nying2004comparative,
  title     = {{A Comparative Study of Interrater Reliability Coefficients Obtained from Different Statistical Procedures Using Monte Carlo Simulation Techniques}},
  author    = {Nying, Ebrima},
  year      = {2004},
  url       = {https://scholarworks.wmich.edu/dissertations/1267}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement

  • inter-rater reliability
  • performance evaluation
  • intraclass correlation coefficient
  • Kendall's coefficient of concordance
  • Fleiss' kappa
  • measurement scales
  • sample size
  • number of raters

Work with this paper

Most-cited papers in Decision Sciences

All highly cited Decision Sciences papers

Questions about this paper

How do I cite this paper in APA style?

Nying, E. (2004). A Comparative Study of Interrater Reliability Coefficients Obtained from Different Statistical Procedures Using Monte Carlo Simulation Techniques. ScholarWorks - WMU (Western Michigan University). https://scholarworks.wmich.edu/dissertations/1267

How many times has this paper been cited?

0 times, according to OpenAlex on 16 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access: read the free full text at ScholarWorks - WMU (Western Michigan University).

Who wrote this paper?

Ebrima Nying.

Bibliographic data from OpenAlex (CC0), last checked 16 November 2026. ResearchGuru does not host this paper; the links go to the publisher or the repository that does.