Review article · 2026 · Open access

The Reliability of Human Evaluation of Large Language Models in Health Care Settings: Scoping Review

Euijun Yang, Siyeon Ko, Hyekyung Woo

Journal of Medical Internet Research, 28, e98184 · Published 19 August 2026 · DOI 10.2196/98184

Open full textCite this paper

In brief

“The Reliability of Human Evaluation of Large Language Models in Health Care Settings: Scoping Review” is a 2026 review article by Euijun Yang and 2 co-authors, published in Journal of Medical Internet Research. OpenAlex counts 1 citations as of 19 November 2026. It is open access under a CC BY licence: the full text is free to read at Journal of Medical Internet Research.

Citations
1
Most cited in
2026 (1)
Field-weighted impact
0.36
References
170
Authors
3

Abstract

BACKGROUND: Integration of large language models (LLMs) into health care has accelerated rapidly, yet reliability concerns pose potential risks to patient safety. Although human evaluation has been widely used as an important approach for assessing LLM reliability, a systematic understanding of how such evaluations have been operationalized across studies remains limited. OBJECTIVE: This study aimed to characterize the current landscape of human evaluation frameworks for LLM reliability in health care and to identify similarities and differences between the clinical and public health domains. METHODS: In line with the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews) guidelines, PubMed, Web of Science, the Cochrane Library, CINAHL, and Google Scholar were searched for studies published from January 2016 to July 2025. Eligible studies were English-language original research conducted in health care settings that assessed the reliability of LLM-generated responses through human evaluation. Key exclusion criteria were studies without human evaluation and studies focused primarily on LLM model selection, performance optimization, or technical development. Extracted data were analyzed across 3 dimensions: what was evaluated, who evaluated, and how evaluation was conducted. Reported methodological limitations were also categorized and compared between the clinical and public health domains. RESULTS: Of the 4347 records identified, 71 studies were included in the final analysis (clinical, n=26; public health, n=45). Six reliability indicators were used: accuracy, relevance, completeness, clarity, safety, and consistency. The clinical domain more frequently assessed guideline concordance, internal consistency, and structural coherence, whereas the public health domain more frequently assessed understandability, harm potential, and repeat response consistency. Single-specialty clinicians were the most common evaluators in both domains, although mixed evaluator panels were observed only in the public health domain. Evaluator panels generally consisted of 5 or fewer members. Five-point Likert scales and researcher-defined rubrics were commonly used evaluation approaches in both domains. Key methodological limitations included evaluator subjectivity, nonstandardized indicators, and limited evaluation scope and settings. CONCLUSIONS: To our knowledge, this is the first review to systematically examine how human evaluations of LLM reliability have been conducted across health care. The focus of reliability evaluation differed across domains, with clinical evaluations giving relatively greater attention to clinical validity and logical rigor, whereas public health evaluations gave relatively greater attention to understandability, practical use, and safe use. These differences suggest that the reliability of health care LLMs is difficult to evaluate adequately using a single universal standard. In addition, the methodological limitations identified in this review indicate that current human evaluation approaches are insufficiently standardized. Therefore, future evaluations of health care…

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Yang, E., Ko, S., & Woo, H. (2026). The Reliability of Human Evaluation of Large Language Models in Health Care Settings: Scoping Review. Journal of Medical Internet Research, 28, e98184. https://doi.org/10.2196/98184

MLA 9

Yang, Euijun, et al. “The Reliability of Human Evaluation of Large Language Models in Health Care Settings: Scoping Review.” Journal of Medical Internet Research, vol. 28, 2026, p. e98184. https://doi.org/10.2196/98184.

Chicago

Yang, Euijun, Siyeon Ko, and Hyekyung Woo. 2026. “The Reliability of Human Evaluation of Large Language Models in Health Care Settings: Scoping Review.” Journal of Medical Internet Research 28: e98184. https://doi.org/10.2196/98184.

Harvard

Yang, E., Ko, S. and Woo, H. (2026) ‘The Reliability of Human Evaluation of Large Language Models in Health Care Settings: Scoping Review’, Journal of Medical Internet Research, 28, p. e98184. doi:10.2196/98184.

IEEE

E. Yang, S. Ko, and H. Woo, “The Reliability of Human Evaluation of Large Language Models in Health Care Settings: Scoping Review,” Journal of Medical Internet Research, vol. 28, p. e98184, 2026, doi: 10.2196/98184.

BibTeX

@article{yang2026reliability,
  title     = {{The Reliability of Human Evaluation of Large Language Models in Health Care Settings: Scoping Review}},
  author    = {Yang, Euijun and Ko, Siyeon and Woo, Hyekyung},
  journal   = {Journal of Medical Internet Research},
  year      = {2026},
  volume    = {28},
  pages     = {e98184},
  doi       = {10.2196/98184},
  url       = {https://doi.org/10.2196/98184}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education

  • large language models
  • human evaluation
  • LLM reliability
  • health care
  • clinical settings
  • evaluation framework
  • evaluator subjectivity
  • methodological limitations

Work with this paper

Most-cited papers in Medicine

All highly cited Medicine papers

Questions about this paper

How do I cite this paper in APA style?

Yang, E., Ko, S., & Woo, H. (2026). The Reliability of Human Evaluation of Large Language Models in Health Care Settings: Scoping Review. Journal of Medical Internet Research, 28, e98184. https://doi.org/10.2196/98184

How many times has this paper been cited?

1 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at Journal of Medical Internet Research: sign in and open the full text from this page.

Who wrote this paper?

Euijun Yang, Siyeon Ko, Hyekyung Woo.

What is the DOI of this paper?

10.2196/98184, permanently at https://doi.org/10.2196/98184. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.