Journal article · 2026 · Open access

Cross-paradigm audit of large language model judges for clinical text evaluation

Tabea M. G. Pakull, Sina Warmer, Kamyar Arzideh, Thomas Hilser, Bernadette Hosters, Hendrik Damm, Felix Nensa, Martin Schüler, Katarzyna Borys, Peter Alexander Horn, René Hosch, Christoph M. Friedrich

npj Digital Medicine, 9(1) · Published 4 November 2026 · DOI 10.1038/s41746-026-03443-2

Open PDFView at the publisherCite this paper

In brief

“Cross-paradigm audit of large language model judges for clinical text evaluation” is a 2026 journal article by Tabea M. G. Pakull and 11 co-authors, published in npj Digital Medicine. OpenAlex counts 0 citations as of 17 November 2026. It is open access under a CC BY licence: the full text is free to read at npj Digital Medicine.

Citations
0
References
31
Authors
12

Abstract

Large language models are increasingly being used to evaluate clinical texts, a concept known as LLM-as-a-judge. However, most of the existing evidence focuses on English data and uses pooled agreement metrics that obscure clinically relevant failure modes. This study audited six open-weight models against expert judges on four German clinical datasets covering report and nursing documentation summarization, discharge summary generation, and patient letter generation. The models were evaluated across three paradigms and four prompt configurations. Alignment with experts varied by criterion and corpus, and no configuration proved reliable across paradigms. All judges exhibited position and identity bias. Substituting LLM-as-a-judge for the expert panel shifted 57.2% of pointwise categories, swapped 41.7% of pairwise winners, missed 10.2% of clinical errors, and inverted clinical priorities by overrating the criteria succinct and accurate. Therefore, LLM-as-a-judge is not a safe substitute for experts in German clinical text evaluations, and deployment thresholds should follow decision-impact analysis for LLM-as-a-judge in clinical contexts.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Pakull, T. M. G., Warmer, S., Arzideh, K., Hilser, T., Hosters, B., Damm, H., Nensa, F., Schüler, M., Borys, K., Horn, P. A., Hosch, R., & Friedrich, C. M. (2026). Cross-paradigm audit of large language model judges for clinical text evaluation. npj Digital Medicine, 9(1). https://doi.org/10.1038/s41746-026-03443-2

MLA 9

Pakull, Tabea M. G., et al. “Cross-paradigm audit of large language model judges for clinical text evaluation.” npj Digital Medicine, vol. 9, no. 1, 2026. https://doi.org/10.1038/s41746-026-03443-2.

Chicago

Pakull, Tabea M. G., Sina Warmer, Kamyar Arzideh, Thomas Hilser, Bernadette Hosters, Hendrik Damm, Felix Nensa, et al. 2026. “Cross-paradigm audit of large language model judges for clinical text evaluation.” npj Digital Medicine 9 (1). https://doi.org/10.1038/s41746-026-03443-2.

Harvard

Pakull, T. M. G. et al. (2026) ‘Cross-paradigm audit of large language model judges for clinical text evaluation’, npj Digital Medicine, 9(1). doi:10.1038/s41746-026-03443-2.

IEEE

T. M. G. Pakull et al., “Cross-paradigm audit of large language model judges for clinical text evaluation,” npj Digital Medicine, vol. 9, no. 1, 2026, doi: 10.1038/s41746-026-03443-2.

BibTeX

@article{pakull2026cross,
  title     = {{Cross-paradigm audit of large language model judges for clinical text evaluation}},
  author    = {Pakull, Tabea M. G. and Warmer, Sina and Arzideh, Kamyar and Hilser, Thomas and Hosters, Bernadette and Damm, Hendrik and Nensa, Felix and Schüler, Martin and Borys, Katarzyna and Horn, Peter Alexander and Hosch, René and Friedrich, Christoph M.},
  journal   = {npj Digital Medicine},
  year      = {2026},
  volume    = {9},
  number    = {1},
  doi       = {10.1038/s41746-026-03443-2},
  url       = {https://doi.org/10.1038/s41746-026-03443-2}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education

  • LLM-as-a-judge
  • large language models
  • clinical judgment
  • expert judges
  • open-weight models

Work with this paper

Most-cited papers in Medicine

All highly cited Medicine papers

Questions about this paper

How do I cite this paper in APA style?

Pakull, T. M. G., Warmer, S., Arzideh, K., Hilser, T., Hosters, B., Damm, H., Nensa, F., Schüler, M., Borys, K., Horn, P. A., Hosch, R., & Friedrich, C. M. (2026). Cross-paradigm audit of large language model judges for clinical text evaluation. npj Digital Medicine, 9(1). https://doi.org/10.1038/s41746-026-03443-2

How many times has this paper been cited?

0 times, according to OpenAlex on 17 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at npj Digital Medicine: sign in and open the PDF from this page.

Who wrote this paper?

Tabea M. G. Pakull, Sina Warmer, Kamyar Arzideh, Thomas Hilser, Bernadette Hosters, Hendrik Damm, Felix Nensa, Martin Schüler, Katarzyna Borys, Peter Alexander Horn, René Hosch, Christoph M. Friedrich.

What is the DOI of this paper?

10.1038/s41746-026-03443-2, permanently at https://doi.org/10.1038/s41746-026-03443-2. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 17 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.