Preprint · 2026 · Open access

Validating LLM judges for automated oversight of patient communication

Zidu Xu, Johnathan Zeng, Shuang Zhou, Zhihong Zhang, Thibault Heintz, Marion Tonneau, Arsalan Yaghoubi, Bingyang Ye, Vikram Goddla, Lisa Soleymani Lehmann, Yu-Hui Chen, Elad Sharon and 8 more

medRxiv · Published 17 September 2026 · DOI 10.64898/2026.09.16.26363176

Open PDFView at the publisherCite this paper

In brief

“Validating LLM judges for automated oversight of patient communication” is a 2026 preprint by Zidu Xu and 19 co-authors, published in medRxiv. OpenAlex counts 0 citations as of 19 November 2026. It is open access under a CC BY-NC licence: the full text is free to read at medRxiv.

Citations
0
References
54
Authors
20

Abstract

LLMs are increasingly used to mediate patient communication, yet scalable evaluation of their safety, accuracy, and communication quality remains an open problem. LLM judges have emerged as automated evaluators, but whether they can holistically replicate human expert judgment is unvalidated. Informed consent for clinical trials presents a demanding case for such validation because it requires conveying complex information to lay audiences under ethical and safety constraints. We developed a stakeholder-informed seven-criterion evaluation rubric spanning safety, reliability, and communication quality. Clinician reference ratings showed strong interrater reliability across all criteria. We validated the rubric on the Informed CONsent Benchmark (ICON-Bench) and benchmarked 19 LLM judges across multiple implementation strategies. LLM judges achieved strong clinician agreement for safety screening and factual verification (Spearman ρ > 0.80) but weaker agreement for communication quality ( ρ < 0.60). Safety-specialized guard models underperformed general-purpose models. Patient advocates rated communication quality lower than both clinicians and LLM judges. These findings support LLM judges for scalable patient communication oversight while demonstrating the need for recalibration to patient-centered evaluation standards.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Xu, Z., Zeng, J., Zhou, S., Zhang, Z., Heintz, T., Tonneau, M., Yaghoubi, A., Ye, B., Goddla, V., Lehmann, L. S., Chen, Y.-H., Sharon, E., Kozono, D., Revette, A. C., Maués, J., Brown, T., Catalano, P. J., Mak, R. H., Dligach, D., & Bitterman, D. S. (2026). Validating LLM judges for automated oversight of patient communication. medRxiv. https://doi.org/10.64898/2026.09.16.26363176

MLA 9

Xu, Zidu, et al. “Validating LLM judges for automated oversight of patient communication.” medRxiv, 2026. https://doi.org/10.64898/2026.09.16.26363176.

Chicago

Xu, Zidu, Johnathan Zeng, Shuang Zhou, Zhihong Zhang, Thibault Heintz, Marion Tonneau, Arsalan Yaghoubi, et al. 2026. “Validating LLM judges for automated oversight of patient communication.” medRxiv. https://doi.org/10.64898/2026.09.16.26363176.

Harvard

Xu, Z. et al. (2026) ‘Validating LLM judges for automated oversight of patient communication’, medRxiv. doi:10.64898/2026.09.16.26363176.

IEEE

Z. Xu et al., “Validating LLM judges for automated oversight of patient communication,” medRxiv, 2026, doi: 10.64898/2026.09.16.26363176.

BibTeX

@misc{xu2026validating,
  title     = {{Validating LLM judges for automated oversight of patient communication}},
  author    = {Xu, Zidu and Zeng, Johnathan and Zhou, Shuang and Zhang, Zhihong and Heintz, Thibault and Tonneau, Marion and Yaghoubi, Arsalan and Ye, Bingyang and Goddla, Vikram and Lehmann, Lisa Soleymani and Chen, Yu-Hui and Sharon, Elad and Kozono, David and Revette, Anna C. and Maués, Julia and Brown, Thelma and Catalano, Paul J. and Mak, Raymond H. and Dligach, Dmitriy and Bitterman, Danielle S.},
  howpublished = {medRxiv},
  year      = {2026},
  doi       = {10.64898/2026.09.16.26363176},
  url       = {https://doi.org/10.64898/2026.09.16.26363176}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education

  • LLM-as-a-judge
  • large language models
  • informed consent
  • patient communication
  • automated assessment
  • assessment rubrics
  • inter-rater reliability
  • communication quality

Work with this paper

Most-cited papers in Medicine

All highly cited Medicine papers

Questions about this paper

How do I cite this paper in APA style?

Xu, Z., Zeng, J., Zhou, S., Zhang, Z., Heintz, T., Tonneau, M., Yaghoubi, A., Ye, B., Goddla, V., Lehmann, L. S., Chen, Y.-H., Sharon, E., Kozono, D., Revette, A. C., Maués, J., Brown, T., Catalano, P. J., Mak, R. H., Dligach, D., & Bitterman, D. S. (2026). Validating LLM judges for automated oversight of patient communication. medRxiv. https://doi.org/10.64898/2026.09.16.26363176

How many times has this paper been cited?

0 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at medRxiv: sign in and open the PDF from this page.

Who wrote this paper?

Zidu Xu, Johnathan Zeng, Shuang Zhou, Zhihong Zhang, Thibault Heintz, Marion Tonneau, Arsalan Yaghoubi, Bingyang Ye, Vikram Goddla, Lisa Soleymani Lehmann, Yu-Hui Chen, Elad Sharon and 8 more.

What is the DOI of this paper?

10.64898/2026.09.16.26363176, permanently at https://doi.org/10.64898/2026.09.16.26363176. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.