Preprint · 2026 · Open access

A Scoping Review of LLM-as-a-Judge in Healthcare and the MedJUDGE Framework

Chenyu Li, Zohaib Akhtar, Mingu Kwak, Yuelyu Ji, Hang Zhang, Tracey Obi, Yufan Ren, Xizhi Wu, Sonish Sivarajkumar, Harold P. Lehmann, Shyam Visweswaran, Michael J. Becich and 4 more

arXiv (Cornell University) · Published 3 April 2026 · DOI 10.48550/arxiv.2604.25933

Open PDFView at the publisherCite this paper

In brief

“A Scoping Review of LLM-as-a-Judge in Healthcare and the MedJUDGE Framework” is a 2026 preprint by Chenyu Li and 15 co-authors, published in arXiv (Cornell University). OpenAlex counts 6 citations as of 19 November 2026. It is open access: the full text is free to read at arXiv (Cornell University).

Citations
6
Most cited in
2026 (6)
References
72
Authors
16

Abstract

As large language models (LLMs) increasingly generate and process clinical text, scalable evaluation has become critical. LLM-as-a-Judge (LaaJ), which uses LLMs to evaluate model outputs, offers a scalable alternative to costly expert review, but its healthcare adoption raises safety and bias concerns. We conducted a PRISMA-ScR scoping review of six databases (January 2020-January 2026), screening 11,727 studies and including 49. The landscape was dominated by evaluation and benchmarking applications (n=37, 75.5%), pointwise scoring (n=42, 85.7%), and GPT-family judges (n=36, 73.5%). Despite growing adoption, validation rigor was limited: among 36 studies with human involvement, the median number of expert validators was 3, while 13 (26.5%) used none. Risk of bias testing was absent in 36 studies (73.5%), only 1 (2.0%) examined demographic fairness, and none assessed temporal stability or patient context. Deployment remained limited, with 1 study (2.0%) reaching production and four (8.2%) prototype stage. Importantly, these gaps may interact: when judges and evaluated systems share training data or architectures, they may inherit similar blind spots, and agreement metrics may fail to distinguish true validity from shared errors. Minimal human oversight, limited bias assessment, and model monoculture together represent a governance gap where current validation may miss clinically significant errors. To address this, we propose MedJUDGE (Medical Judge Utility, De-biasing, Governance and Evaluation), a risk-stratified three-pillar framework organized around validity, safety, and accountability across clinical risk tiers, providing deployment-oriented evaluation guidance for healthcare LaaJ systems.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Li, C., Akhtar, Z., Kwak, M., Ji, Y., Zhang, H., Obi, T., Ren, Y., Wu, X., Sivarajkumar, S., Lehmann, H. P., Visweswaran, S., Becich, M. J., Mowery, D. L., Liu, R., Sun, H., & Wang, Y. (2026). A Scoping Review of LLM-as-a-Judge in Healthcare and the MedJUDGE Framework. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2604.25933

MLA 9

Li, Chenyu, et al. “A Scoping Review of LLM-as-a-Judge in Healthcare and the MedJUDGE Framework.” arXiv (Cornell University), 2026. https://doi.org/10.48550/arxiv.2604.25933.

Chicago

Li, Chenyu, Zohaib Akhtar, Mingu Kwak, Yuelyu Ji, Hang Zhang, Tracey Obi, Yufan Ren, et al. 2026. “A Scoping Review of LLM-as-a-Judge in Healthcare and the MedJUDGE Framework.” arXiv (Cornell University). https://doi.org/10.48550/arxiv.2604.25933.

Harvard

Li, C. et al. (2026) ‘A Scoping Review of LLM-as-a-Judge in Healthcare and the MedJUDGE Framework’, arXiv (Cornell University). doi:10.48550/arxiv.2604.25933.

IEEE

C. Li et al., “A Scoping Review of LLM-as-a-Judge in Healthcare and the MedJUDGE Framework,” arXiv (Cornell University), 2026, doi: 10.48550/arxiv.2604.25933.

BibTeX

@misc{li2026scoping,
  title     = {{A Scoping Review of LLM-as-a-Judge in Healthcare and the MedJUDGE Framework}},
  author    = {Li, Chenyu and Akhtar, Zohaib and Kwak, Mingu and Ji, Yuelyu and Zhang, Hang and Obi, Tracey and Ren, Yufan and Wu, Xizhi and Sivarajkumar, Sonish and Lehmann, Harold P. and Visweswaran, Shyam and Becich, Michael J. and Mowery, Danielle L. and Liu, Renxuan and Sun, Haoyang and Wang, Yanshan},
  howpublished = {arXiv (Cornell University)},
  year      = {2026},
  doi       = {10.48550/arxiv.2604.25933},
  url       = {https://doi.org/10.48550/arxiv.2604.25933}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education

  • LLM-as-a-judge
  • large language models
  • LLM evaluation
  • benchmarking
  • bias assessment
  • safety
  • accountability
  • clinical risk stratification

Work with this paper

Most-cited papers in Medicine

All highly cited Medicine papers

Questions about this paper

How do I cite this paper in APA style?

Li, C., Akhtar, Z., Kwak, M., Ji, Y., Zhang, H., Obi, T., Ren, Y., Wu, X., Sivarajkumar, S., Lehmann, H. P., Visweswaran, S., Becich, M. J., Mowery, D. L., Liu, R., Sun, H., & Wang, Y. (2026). A Scoping Review of LLM-as-a-Judge in Healthcare and the MedJUDGE Framework. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2604.25933

How many times has this paper been cited?

6 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at arXiv (Cornell University): sign in and open the PDF from this page.

Who wrote this paper?

Chenyu Li, Zohaib Akhtar, Mingu Kwak, Yuelyu Ji, Hang Zhang, Tracey Obi, Yufan Ren, Xizhi Wu, Sonish Sivarajkumar, Harold P. Lehmann, Shyam Visweswaran, Michael J. Becich and 4 more.

What is the DOI of this paper?

10.48550/arxiv.2604.25933, permanently at https://doi.org/10.48550/arxiv.2604.25933. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.