Dissertation · 2004 · Open access
A Comparative Study of Interrater Reliability Coefficients Obtained from Different Statistical Procedures Using Monte Carlo Simulation Techniques
ScholarWorks - WMU (Western Michigan University) · Published 1 January 2004
In brief
“A Comparative Study of Interrater Reliability Coefficients Obtained from Different Statistical Procedures Using Monte Carlo Simulation Techniques” is a 2004 dissertation by Ebrima Nying, published in ScholarWorks - WMU (Western Michigan University). OpenAlex counts 0 citations as of 16 November 2026. It is open access: the full text is free to read at ScholarWorks - WMU (Western Michigan University).
- Citations
- 0
- References
- 20
- Authors
- 1
Abstract
Reliability estimation is a key research component within the global area of educational assessment. The literature reports numerous studies using different statistical techniques for estimating reliability of educational measures. However, few have focused on the estimation of interrater reliability of performance assessment (Abedi, Baker, & Herl, 1995). Specifically, this study compared three different methods for estimating interrater reliability to determine if there are differences among these estimates as a function of: sample size, measurement scale, number of raters and the theoretical population reliability (rho). The three methods of estimation were the Intraclass Correlation (ICC(2, k )) (Shrout & Fleiss, 1979), Kendall's Coefficient of Concordance (Kendall, 1938) and Kappa for Multiple Raters (Fleiss, 1971).\nThis study employed a 4-between (three measurement scales, four sample sizes, four rater groups, and two population rho's), 1-within (three estimation methods) mixed design. ANOVA results indicated two significant 4-way interactions involving measure*Mscale* rater*rho and measure*Mscale* size*rho. Simple effect analyses of the 4-way interactions focused on each level of rho and parallel 3-way ANOVAs (measure*Mscale*size or rater) were conducted. Post hoc analyses indicated that both 3-way interactions were significant. To further understand these 3-way interactions, they were followed by 2-way designs holding Measure constant.\nFinally, Tukey HSD tests were then examined. This post hoc strategy consistently showed mean differences in the reliability estimates among the differentmeasurement scales as a function of both number of raters and sample size regardless of rho.\nRecommendations for future research and limitations of this study are provided. Although population rho had a predictable effect, the present findings should be replicated and extended with different values of population rho and since this study only considered normally distributed data, the influence of population distribution should be examined, especially with response scales having fewer levels.\nPractitioners and researchers who are faced with estimating interrater reliability should be comforted by the striking stability of these methods. When the data are dichotomous, Kappa for Multiple Raters is recommended, and when the data are ordinal, Kendall's Coefficient of Concordance is recommended.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Nying, E. (2004). A Comparative Study of Interrater Reliability Coefficients Obtained from Different Statistical Procedures Using Monte Carlo Simulation Techniques. ScholarWorks - WMU (Western Michigan University). https://scholarworks.wmich.edu/dissertations/1267
MLA 9
Nying, Ebrima. “A Comparative Study of Interrater Reliability Coefficients Obtained from Different Statistical Procedures Using Monte Carlo Simulation Techniques.” ScholarWorks - WMU (Western Michigan University), 2004. https://scholarworks.wmich.edu/dissertations/1267.
Chicago
Nying, Ebrima. 2004. “A Comparative Study of Interrater Reliability Coefficients Obtained from Different Statistical Procedures Using Monte Carlo Simulation Techniques.” ScholarWorks - WMU (Western Michigan University). https://scholarworks.wmich.edu/dissertations/1267.
Harvard
Nying, E. (2004) ‘A Comparative Study of Interrater Reliability Coefficients Obtained from Different Statistical Procedures Using Monte Carlo Simulation Techniques’, ScholarWorks - WMU (Western Michigan University). Available at: https://scholarworks.wmich.edu/dissertations/1267.
IEEE
E. Nying, “A Comparative Study of Interrater Reliability Coefficients Obtained from Different Statistical Procedures Using Monte Carlo Simulation Techniques,” ScholarWorks - WMU (Western Michigan University), 2004.
BibTeX
@phdthesis{nying2004comparative,
title = {{A Comparative Study of Interrater Reliability Coefficients Obtained from Different Statistical Procedures Using Monte Carlo Simulation Techniques}},
author = {Nying, Ebrima},
year = {2004},
url = {https://scholarworks.wmich.edu/dissertations/1267}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- inter-rater reliability
- performance evaluation
- intraclass correlation coefficient
- Kendall's coefficient of concordance
- Fleiss' kappa
- measurement scales
- sample size
- number of raters
Work with this paper
Most-cited papers in Decision Sciences
- The PRISMA 2020 statement: an updated guideline for reporting systematic reviewsMatthew James Page, Joanne E. McKenzie, Patrick M. Bossuyt and 23 more · BMJ · 2021 · 144,919 citations
- Evaluating Structural Equation Models with Unobservable Variables and Measurement ErrorClaes Fornell, David F. Larcker · Journal of Marketing Research · 1981 · 118,336 citations
- Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternativesLi‐tze Hu, Peter M. Bentler · Structural Equation Modeling A Multidisciplinary Journal · 1999 · 117,461 citations
- Common method biases in behavioral research: A critical review of the literature and recommended remedies.Philip M. Podsakoff, SCOTT B. MACKENZIE, Jeong Yeon Lee and 1 more · Journal of Applied Psychology · 2003 · 89,248 citations
- Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statementDavid Moher, Alessandro Liberati, Jennifer Tetzlaff and 2 more · BMJ · 2009 · 85,524 citations
Questions about this paper
How do I cite this paper in APA style?
Nying, E. (2004). A Comparative Study of Interrater Reliability Coefficients Obtained from Different Statistical Procedures Using Monte Carlo Simulation Techniques. ScholarWorks - WMU (Western Michigan University). https://scholarworks.wmich.edu/dissertations/1267
How many times has this paper been cited?
0 times, according to OpenAlex on 16 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access: read the free full text at ScholarWorks - WMU (Western Michigan University).
Who wrote this paper?
Ebrima Nying.
Bibliographic data from OpenAlex (CC0), last checked 16 November 2026. ResearchGuru does not host this paper; the links go to the publisher or the repository that does.