Preprint · 2026 · Open access
Validating LLM judges for automated oversight of patient communication
medRxiv · Published 17 September 2026 · DOI 10.64898/2026.09.16.26363176
In brief
“Validating LLM judges for automated oversight of patient communication” is a 2026 preprint by Zidu Xu and 19 co-authors, published in medRxiv. OpenAlex counts 0 citations as of 19 November 2026. It is open access under a CC BY-NC licence: the full text is free to read at medRxiv.
- Citations
- 0
- References
- 54
- Authors
- 20
Abstract
LLMs are increasingly used to mediate patient communication, yet scalable evaluation of their safety, accuracy, and communication quality remains an open problem. LLM judges have emerged as automated evaluators, but whether they can holistically replicate human expert judgment is unvalidated. Informed consent for clinical trials presents a demanding case for such validation because it requires conveying complex information to lay audiences under ethical and safety constraints. We developed a stakeholder-informed seven-criterion evaluation rubric spanning safety, reliability, and communication quality. Clinician reference ratings showed strong interrater reliability across all criteria. We validated the rubric on the Informed CONsent Benchmark (ICON-Bench) and benchmarked 19 LLM judges across multiple implementation strategies. LLM judges achieved strong clinician agreement for safety screening and factual verification (Spearman ρ > 0.80) but weaker agreement for communication quality ( ρ < 0.60). Safety-specialized guard models underperformed general-purpose models. Patient advocates rated communication quality lower than both clinicians and LLM judges. These findings support LLM judges for scalable patient communication oversight while demonstrating the need for recalibration to patient-centered evaluation standards.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Xu, Z., Zeng, J., Zhou, S., Zhang, Z., Heintz, T., Tonneau, M., Yaghoubi, A., Ye, B., Goddla, V., Lehmann, L. S., Chen, Y.-H., Sharon, E., Kozono, D., Revette, A. C., Maués, J., Brown, T., Catalano, P. J., Mak, R. H., Dligach, D., & Bitterman, D. S. (2026). Validating LLM judges for automated oversight of patient communication. medRxiv. https://doi.org/10.64898/2026.09.16.26363176
MLA 9
Xu, Zidu, et al. “Validating LLM judges for automated oversight of patient communication.” medRxiv, 2026. https://doi.org/10.64898/2026.09.16.26363176.
Chicago
Xu, Zidu, Johnathan Zeng, Shuang Zhou, Zhihong Zhang, Thibault Heintz, Marion Tonneau, Arsalan Yaghoubi, et al. 2026. “Validating LLM judges for automated oversight of patient communication.” medRxiv. https://doi.org/10.64898/2026.09.16.26363176.
Harvard
Xu, Z. et al. (2026) ‘Validating LLM judges for automated oversight of patient communication’, medRxiv. doi:10.64898/2026.09.16.26363176.
IEEE
Z. Xu et al., “Validating LLM judges for automated oversight of patient communication,” medRxiv, 2026, doi: 10.64898/2026.09.16.26363176.
BibTeX
@misc{xu2026validating,
title = {{Validating LLM judges for automated oversight of patient communication}},
author = {Xu, Zidu and Zeng, Johnathan and Zhou, Shuang and Zhang, Zhihong and Heintz, Thibault and Tonneau, Marion and Yaghoubi, Arsalan and Ye, Bingyang and Goddla, Vikram and Lehmann, Lisa Soleymani and Chen, Yu-Hui and Sharon, Elad and Kozono, David and Revette, Anna C. and Maués, Julia and Brown, Thelma and Catalano, Paul J. and Mak, Raymond H. and Dligach, Dmitriy and Bitterman, Danielle S.},
howpublished = {medRxiv},
year = {2026},
doi = {10.64898/2026.09.16.26363176},
url = {https://doi.org/10.64898/2026.09.16.26363176}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- LLM-as-a-judge
- large language models
- informed consent
- patient communication
- automated assessment
- assessment rubrics
- inter-rater reliability
- communication quality
Work with this paper
Most-cited papers in Medicine
- Yaşlı Kadınlarda Üreme SağlığıDemet Kokanalı, Yaprak Engin Üstün · Jinekoloji-Obstetrik ve Neonatoloji Tıp Dergisi · 2019 · 169,630 citations
- Global Cancer Statistics 2020: GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 CountriesHyuna Sung, Adalberto M. Filho, Mathieu Laversanne and 5 more · CA A Cancer Journal for Clinicians · 2021 · 119,885 citations
- “Mini-mental state”Marshal F. Folstein, Susan E. Folstein, Paul R. McHugh · Journal of Psychiatric Research · 1975 · 90,424 citations
- Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countriesFreddie Bray, Jacques Ferlay, Isabelle Soerjomataram and 3 more · CA A Cancer Journal for Clinicians · 2018 · 87,396 citations
- Hallmarks of Cancer: The Next GenerationDouglas Hanahan, Robert A. Weinberg · Cell · 2011 · 69,758 citations
Questions about this paper
How do I cite this paper in APA style?
Xu, Z., Zeng, J., Zhou, S., Zhang, Z., Heintz, T., Tonneau, M., Yaghoubi, A., Ye, B., Goddla, V., Lehmann, L. S., Chen, Y.-H., Sharon, E., Kozono, D., Revette, A. C., Maués, J., Brown, T., Catalano, P. J., Mak, R. H., Dligach, D., & Bitterman, D. S. (2026). Validating LLM judges for automated oversight of patient communication. medRxiv. https://doi.org/10.64898/2026.09.16.26363176
How many times has this paper been cited?
0 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at medRxiv: sign in and open the PDF from this page.
Who wrote this paper?
Zidu Xu, Johnathan Zeng, Shuang Zhou, Zhihong Zhang, Thibault Heintz, Marion Tonneau, Arsalan Yaghoubi, Bingyang Ye, Vikram Goddla, Lisa Soleymani Lehmann, Yu-Hui Chen, Elad Sharon and 8 more.
What is the DOI of this paper?
10.64898/2026.09.16.26363176, permanently at https://doi.org/10.64898/2026.09.16.26363176. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.