Preprint · 2026 · Open access

LLM-as-a-Judge in Healthcare: A Scoping Analysis of Applications, Methods, and Human Alignment

Lingyao Li, Deyi Li, Chen Chen, Renkai Ma, Runlong Yu, Mingquan Lin, Rui Yin, Lizhou Fan, Cathy Shyr, Siyuan Ma, Mei Liu, Steven J. Bethard

arXiv (Cornell University) · Published 24 May 2026 · DOI 10.48550/arxiv.2605.25273

Open PDFView at the publisherCite this paper

In brief

“LLM-as-a-Judge in Healthcare: A Scoping Analysis of Applications, Methods, and Human Alignment” is a 2026 preprint by Lingyao Li and 11 co-authors, published in arXiv (Cornell University). OpenAlex counts 12 citations as of 19 November 2026. It is open access under a CC BY licence: the full text is free to read at arXiv (Cornell University).

Citations
12
Most cited in
2026 (12)
References
156
Authors
12

Abstract

Large language models (LLMs) are increasingly deployed across healthcare applications, including clinical documentation, diagnostic reasoning, medicine recommendation, and medical education. Their outputs are largely unstructured clinical text, which is difficult to reliably evaluate at scale. LLM-as-a-Judge, in which an LLM evaluates another system's output against task-specific criteria, offers a scalable alternative and is increasingly used in clinical evaluation, yet its validity in healthcare remains underexamined. Existing reviews focus on general-purpose LLM evaluation or on risk framework, rather than systematically characterizing how LLM-as-a-Judge is applied in healthcare and how well their judgments align with human experts. We therefore conduct a PRISMA-guided comprehensive review of LLM-as-a-Judge applications in healthcare, searching five databases for studies published between January 2023 and February 2026. After screening 541 records, 134 studies meet the eligibility and are coded by health scenario, judge configuration, technical approach, and validation design. LLM-as-a-Judge is concentrated in clinical decision support, clinical natural language processing (NLP), medical knowledge and question answering (QA), and medical communication. OpenAI models are the most frequently used judges, and prompt engineering appears in nearly all studies, with ensemble, multi-agent, and retrieval-augmented designs as common extensions. Among studies reporting human validation, LLM judges often show moderate to strong alignment with expert judgments, although reliability varies substantially across tasks. Overall, this review positions LLM-as-a-Judge as a promising framework for scalable healthcare AI evaluation, while emphasizing that its clinical value depends on model design and rigorous validation.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Li, L., Li, D., Chen, C., Ma, R., Yu, R., Lin, M., Yin, R., Fan, L., Shyr, C., Ma, S., Liu, M., & Bethard, S. J. (2026). LLM-as-a-Judge in Healthcare: A Scoping Analysis of Applications, Methods, and Human Alignment. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2605.25273

MLA 9

Li, Lingyao, et al. “LLM-as-a-Judge in Healthcare: A Scoping Analysis of Applications, Methods, and Human Alignment.” arXiv (Cornell University), 2026. https://doi.org/10.48550/arxiv.2605.25273.

Chicago

Li, Lingyao, Deyi Li, Chen Chen, Renkai Ma, Runlong Yu, Mingquan Lin, Rui Yin, et al. 2026. “LLM-as-a-Judge in Healthcare: A Scoping Analysis of Applications, Methods, and Human Alignment.” arXiv (Cornell University). https://doi.org/10.48550/arxiv.2605.25273.

Harvard

Li, L. et al. (2026) ‘LLM-as-a-Judge in Healthcare: A Scoping Analysis of Applications, Methods, and Human Alignment’, arXiv (Cornell University). doi:10.48550/arxiv.2605.25273.

IEEE

L. Li et al., “LLM-as-a-Judge in Healthcare: A Scoping Analysis of Applications, Methods, and Human Alignment,” arXiv (Cornell University), 2026, doi: 10.48550/arxiv.2605.25273.

BibTeX

@misc{li2026judge,
  title     = {{LLM-as-a-Judge in Healthcare: A Scoping Analysis of Applications, Methods, and Human Alignment}},
  author    = {Li, Lingyao and Li, Deyi and Chen, Chen and Ma, Renkai and Yu, Runlong and Lin, Mingquan and Yin, Rui and Fan, Lizhou and Shyr, Cathy and Ma, Siyuan and Liu, Mei and Bethard, Steven J.},
  howpublished = {arXiv (Cornell University)},
  year      = {2026},
  doi       = {10.48550/arxiv.2605.25273},
  url       = {https://doi.org/10.48550/arxiv.2605.25273}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education

  • LLM-as-a-judge
  • large language models
  • human alignment
  • clinical decision support
  • clinical natural language processing
  • medical question answering
  • prompt engineering
  • multi-agent systems

Work with this paper

Most-cited papers in Medicine

All highly cited Medicine papers

Questions about this paper

How do I cite this paper in APA style?

Li, L., Li, D., Chen, C., Ma, R., Yu, R., Lin, M., Yin, R., Fan, L., Shyr, C., Ma, S., Liu, M., & Bethard, S. J. (2026). LLM-as-a-Judge in Healthcare: A Scoping Analysis of Applications, Methods, and Human Alignment. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2605.25273

How many times has this paper been cited?

12 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at arXiv (Cornell University): sign in and open the PDF from this page.

Who wrote this paper?

Lingyao Li, Deyi Li, Chen Chen, Renkai Ma, Runlong Yu, Mingquan Lin, Rui Yin, Lizhou Fan, Cathy Shyr, Siyuan Ma, Mei Liu, Steven J. Bethard.

What is the DOI of this paper?

10.48550/arxiv.2605.25273, permanently at https://doi.org/10.48550/arxiv.2605.25273. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.