Review article · 2026 · Open access
The Reliability of Human Evaluation of Large Language Models in Health Care Settings: Scoping Review
Journal of Medical Internet Research, 28, e98184 · Published 19 August 2026 · DOI 10.2196/98184
In brief
“The Reliability of Human Evaluation of Large Language Models in Health Care Settings: Scoping Review” is a 2026 review article by Euijun Yang and 2 co-authors, published in Journal of Medical Internet Research. OpenAlex counts 1 citations as of 19 November 2026. It is open access under a CC BY licence: the full text is free to read at Journal of Medical Internet Research.
- Citations
- 1
- Most cited in
- 2026 (1)
- Field-weighted impact
- 0.36
- References
- 170
- Authors
- 3
Abstract
BACKGROUND: Integration of large language models (LLMs) into health care has accelerated rapidly, yet reliability concerns pose potential risks to patient safety. Although human evaluation has been widely used as an important approach for assessing LLM reliability, a systematic understanding of how such evaluations have been operationalized across studies remains limited. OBJECTIVE: This study aimed to characterize the current landscape of human evaluation frameworks for LLM reliability in health care and to identify similarities and differences between the clinical and public health domains. METHODS: In line with the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews) guidelines, PubMed, Web of Science, the Cochrane Library, CINAHL, and Google Scholar were searched for studies published from January 2016 to July 2025. Eligible studies were English-language original research conducted in health care settings that assessed the reliability of LLM-generated responses through human evaluation. Key exclusion criteria were studies without human evaluation and studies focused primarily on LLM model selection, performance optimization, or technical development. Extracted data were analyzed across 3 dimensions: what was evaluated, who evaluated, and how evaluation was conducted. Reported methodological limitations were also categorized and compared between the clinical and public health domains. RESULTS: Of the 4347 records identified, 71 studies were included in the final analysis (clinical, n=26; public health, n=45). Six reliability indicators were used: accuracy, relevance, completeness, clarity, safety, and consistency. The clinical domain more frequently assessed guideline concordance, internal consistency, and structural coherence, whereas the public health domain more frequently assessed understandability, harm potential, and repeat response consistency. Single-specialty clinicians were the most common evaluators in both domains, although mixed evaluator panels were observed only in the public health domain. Evaluator panels generally consisted of 5 or fewer members. Five-point Likert scales and researcher-defined rubrics were commonly used evaluation approaches in both domains. Key methodological limitations included evaluator subjectivity, nonstandardized indicators, and limited evaluation scope and settings. CONCLUSIONS: To our knowledge, this is the first review to systematically examine how human evaluations of LLM reliability have been conducted across health care. The focus of reliability evaluation differed across domains, with clinical evaluations giving relatively greater attention to clinical validity and logical rigor, whereas public health evaluations gave relatively greater attention to understandability, practical use, and safe use. These differences suggest that the reliability of health care LLMs is difficult to evaluate adequately using a single universal standard. In addition, the methodological limitations identified in this review indicate that current human evaluation approaches are insufficiently standardized. Therefore, future evaluations of health care…
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Yang, E., Ko, S., & Woo, H. (2026). The Reliability of Human Evaluation of Large Language Models in Health Care Settings: Scoping Review. Journal of Medical Internet Research, 28, e98184. https://doi.org/10.2196/98184
MLA 9
Yang, Euijun, et al. “The Reliability of Human Evaluation of Large Language Models in Health Care Settings: Scoping Review.” Journal of Medical Internet Research, vol. 28, 2026, p. e98184. https://doi.org/10.2196/98184.
Chicago
Yang, Euijun, Siyeon Ko, and Hyekyung Woo. 2026. “The Reliability of Human Evaluation of Large Language Models in Health Care Settings: Scoping Review.” Journal of Medical Internet Research 28: e98184. https://doi.org/10.2196/98184.
Harvard
Yang, E., Ko, S. and Woo, H. (2026) ‘The Reliability of Human Evaluation of Large Language Models in Health Care Settings: Scoping Review’, Journal of Medical Internet Research, 28, p. e98184. doi:10.2196/98184.
IEEE
E. Yang, S. Ko, and H. Woo, “The Reliability of Human Evaluation of Large Language Models in Health Care Settings: Scoping Review,” Journal of Medical Internet Research, vol. 28, p. e98184, 2026, doi: 10.2196/98184.
BibTeX
@article{yang2026reliability,
title = {{The Reliability of Human Evaluation of Large Language Models in Health Care Settings: Scoping Review}},
author = {Yang, Euijun and Ko, Siyeon and Woo, Hyekyung},
journal = {Journal of Medical Internet Research},
year = {2026},
volume = {28},
pages = {e98184},
doi = {10.2196/98184},
url = {https://doi.org/10.2196/98184}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- large language models
- human evaluation
- LLM reliability
- health care
- clinical settings
- evaluation framework
- evaluator subjectivity
- methodological limitations
Work with this paper
Most-cited papers in Medicine
- Yaşlı Kadınlarda Üreme SağlığıDemet Kokanalı, Yaprak Engin Üstün · Jinekoloji-Obstetrik ve Neonatoloji Tıp Dergisi · 2019 · 169,630 citations
- Global Cancer Statistics 2020: GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 CountriesHyuna Sung, Adalberto M. Filho, Mathieu Laversanne and 5 more · CA A Cancer Journal for Clinicians · 2021 · 119,885 citations
- “Mini-mental state”Marshal F. Folstein, Susan E. Folstein, Paul R. McHugh · Journal of Psychiatric Research · 1975 · 90,424 citations
- Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countriesFreddie Bray, Jacques Ferlay, Isabelle Soerjomataram and 3 more · CA A Cancer Journal for Clinicians · 2018 · 87,396 citations
- Hallmarks of Cancer: The Next GenerationDouglas Hanahan, Robert A. Weinberg · Cell · 2011 · 69,758 citations
Questions about this paper
How do I cite this paper in APA style?
Yang, E., Ko, S., & Woo, H. (2026). The Reliability of Human Evaluation of Large Language Models in Health Care Settings: Scoping Review. Journal of Medical Internet Research, 28, e98184. https://doi.org/10.2196/98184
How many times has this paper been cited?
1 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at Journal of Medical Internet Research: sign in and open the full text from this page.
Who wrote this paper?
Euijun Yang, Siyeon Ko, Hyekyung Woo.
What is the DOI of this paper?
10.2196/98184, permanently at https://doi.org/10.2196/98184. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.