Journal article · 2026 · Open access
Cross-paradigm audit of large language model judges for clinical text evaluation
npj Digital Medicine, 9(1) · Published 4 November 2026 · DOI 10.1038/s41746-026-03443-2
In brief
“Cross-paradigm audit of large language model judges for clinical text evaluation” is a 2026 journal article by Tabea M. G. Pakull and 11 co-authors, published in npj Digital Medicine. OpenAlex counts 0 citations as of 17 November 2026. It is open access under a CC BY licence: the full text is free to read at npj Digital Medicine.
- Citations
- 0
- References
- 31
- Authors
- 12
Abstract
Large language models are increasingly being used to evaluate clinical texts, a concept known as LLM-as-a-judge. However, most of the existing evidence focuses on English data and uses pooled agreement metrics that obscure clinically relevant failure modes. This study audited six open-weight models against expert judges on four German clinical datasets covering report and nursing documentation summarization, discharge summary generation, and patient letter generation. The models were evaluated across three paradigms and four prompt configurations. Alignment with experts varied by criterion and corpus, and no configuration proved reliable across paradigms. All judges exhibited position and identity bias. Substituting LLM-as-a-judge for the expert panel shifted 57.2% of pointwise categories, swapped 41.7% of pairwise winners, missed 10.2% of clinical errors, and inverted clinical priorities by overrating the criteria succinct and accurate. Therefore, LLM-as-a-judge is not a safe substitute for experts in German clinical text evaluations, and deployment thresholds should follow decision-impact analysis for LLM-as-a-judge in clinical contexts.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Pakull, T. M. G., Warmer, S., Arzideh, K., Hilser, T., Hosters, B., Damm, H., Nensa, F., Schüler, M., Borys, K., Horn, P. A., Hosch, R., & Friedrich, C. M. (2026). Cross-paradigm audit of large language model judges for clinical text evaluation. npj Digital Medicine, 9(1). https://doi.org/10.1038/s41746-026-03443-2
MLA 9
Pakull, Tabea M. G., et al. “Cross-paradigm audit of large language model judges for clinical text evaluation.” npj Digital Medicine, vol. 9, no. 1, 2026. https://doi.org/10.1038/s41746-026-03443-2.
Chicago
Pakull, Tabea M. G., Sina Warmer, Kamyar Arzideh, Thomas Hilser, Bernadette Hosters, Hendrik Damm, Felix Nensa, et al. 2026. “Cross-paradigm audit of large language model judges for clinical text evaluation.” npj Digital Medicine 9 (1). https://doi.org/10.1038/s41746-026-03443-2.
Harvard
Pakull, T. M. G. et al. (2026) ‘Cross-paradigm audit of large language model judges for clinical text evaluation’, npj Digital Medicine, 9(1). doi:10.1038/s41746-026-03443-2.
IEEE
T. M. G. Pakull et al., “Cross-paradigm audit of large language model judges for clinical text evaluation,” npj Digital Medicine, vol. 9, no. 1, 2026, doi: 10.1038/s41746-026-03443-2.
BibTeX
@article{pakull2026cross,
title = {{Cross-paradigm audit of large language model judges for clinical text evaluation}},
author = {Pakull, Tabea M. G. and Warmer, Sina and Arzideh, Kamyar and Hilser, Thomas and Hosters, Bernadette and Damm, Hendrik and Nensa, Felix and Schüler, Martin and Borys, Katarzyna and Horn, Peter Alexander and Hosch, René and Friedrich, Christoph M.},
journal = {npj Digital Medicine},
year = {2026},
volume = {9},
number = {1},
doi = {10.1038/s41746-026-03443-2},
url = {https://doi.org/10.1038/s41746-026-03443-2}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- LLM-as-a-judge
- large language models
- clinical judgment
- expert judges
- open-weight models
Work with this paper
Most-cited papers in Medicine
- Yaşlı Kadınlarda Üreme SağlığıDemet Kokanalı, Yaprak Engin Üstün · Jinekoloji-Obstetrik ve Neonatoloji Tıp Dergisi · 2019 · 169,630 citations
- Global Cancer Statistics 2020: GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 CountriesHyuna Sung, Adalberto M. Filho, Mathieu Laversanne and 5 more · CA A Cancer Journal for Clinicians · 2021 · 119,885 citations
- “Mini-mental state”Marshal F. Folstein, Susan E. Folstein, Paul R. McHugh · Journal of Psychiatric Research · 1975 · 90,424 citations
- Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countriesFreddie Bray, Jacques Ferlay, Isabelle Soerjomataram and 3 more · CA A Cancer Journal for Clinicians · 2018 · 87,396 citations
- Hallmarks of Cancer: The Next GenerationDouglas Hanahan, Robert A. Weinberg · Cell · 2011 · 69,758 citations
Questions about this paper
How do I cite this paper in APA style?
Pakull, T. M. G., Warmer, S., Arzideh, K., Hilser, T., Hosters, B., Damm, H., Nensa, F., Schüler, M., Borys, K., Horn, P. A., Hosch, R., & Friedrich, C. M. (2026). Cross-paradigm audit of large language model judges for clinical text evaluation. npj Digital Medicine, 9(1). https://doi.org/10.1038/s41746-026-03443-2
How many times has this paper been cited?
0 times, according to OpenAlex on 17 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at npj Digital Medicine: sign in and open the PDF from this page.
Who wrote this paper?
Tabea M. G. Pakull, Sina Warmer, Kamyar Arzideh, Thomas Hilser, Bernadette Hosters, Hendrik Damm, Felix Nensa, Martin Schüler, Katarzyna Borys, Peter Alexander Horn, René Hosch, Christoph M. Friedrich.
What is the DOI of this paper?
10.1038/s41746-026-03443-2, permanently at https://doi.org/10.1038/s41746-026-03443-2. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 17 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.