Journal article · 2026 · Open access

Addressing benchmarking gaps in large language models for health and medicine with dynamic red-teaming

Jiazhen Pan, Bailiang Jian, Paul Hager, Yundi Zhang, Che Liu, Friedrike Jungmann, Hongwei Li, Julian Canisius, Chenyu You, Junde Wu, Jiayuan Zhu, Fenglin Liu and 10 more

Nature Health, 1(11), 1254–1270 · Published 15 July 2026 · DOI 10.1038/s44360-026-00152-8

Open PDFView at the publisherCite this paper

In brief

“Addressing benchmarking gaps in large language models for health and medicine with dynamic red-teaming” is a 2026 journal article by Jiazhen Pan and 21 co-authors, published in Nature Health. OpenAlex counts 29 citations as of 19 November 2026, which puts it in the top 3% of works published in 2026 by citations. It is open access under a CC BY licence: the full text is free to read at Nature Health.

Citations
29
Most cited in
2026 (23)
Field-weighted impact
25.23
References
58
Authors
22

Abstract

Large language models (LLMs) are increasingly used to answer health-related questions and support healthcare workflows, yet evidence for their safety still relies heavily on static benchmarks that can rapidly become obsolete or be optimized against. Here we introduce a Dynamic, Automatic and Systematic (DAS) red-teaming audit framework that continuously stress-tests LLMs for health across four safety-critical axes: robustness, privacy, bias and hallucination. Validated against board-certified clinicians, a suite of adversarial agents autonomously mutates health-related test cases to uncover vulnerabilities in real time. Applying DAS to 15 state-of-the-art LLMs revealed a profound gap between high static benchmark performance and low dynamic reliability-the 'benchmarking gap'. Despite median MedQA accuracy exceeding 80%, 94% of previously correct answers failed under dynamic robustness testing. This brittleness generalized to the realistic, open-ended HealthBench dataset, where top-tier models exhibited failure rates exceeding 70%, suggesting that high scores on established static benchmarks may reflect superficial memorization. We observed similarly high failure rates across other domains: privacy leaks were elicited in 86% of scenarios, cognitive bias priming altered recommendations in 81% of fairness tests and hallucination rates exceeded 74% in widely used models. By converting LLM safety evaluation for health from a static checklist into a living adversarial audit, DAS provides a scalable framework for surfacing latent risks before such systems are deployed in consumer-facing health assistants and broader clinical workflows.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Pan, J., Jian, B., Hager, P., Zhang, Y., Liu, C., Jungmann, F., Li, H., Canisius, J., You, C., Wu, J., Zhu, J., Liu, F., Liu, Y., Bubeck, N., Knolle, M. A., Chen, C., Wachinger, C., Gong, Z., Ouyang, C., . . . Rueckert, D. (2026). Addressing benchmarking gaps in large language models for health and medicine with dynamic red-teaming. Nature Health, 1(11), 1254–1270. https://doi.org/10.1038/s44360-026-00152-8

MLA 9

Pan, Jiazhen, et al. “Addressing benchmarking gaps in large language models for health and medicine with dynamic red-teaming.” Nature Health, vol. 1, no. 11, 2026, pp. 1254–1270. https://doi.org/10.1038/s44360-026-00152-8.

Chicago

Pan, Jiazhen, Bailiang Jian, Paul Hager, Yundi Zhang, Che Liu, Friedrike Jungmann, Hongwei Li, et al. 2026. “Addressing benchmarking gaps in large language models for health and medicine with dynamic red-teaming.” Nature Health 1 (11): 1254–1270. https://doi.org/10.1038/s44360-026-00152-8.

Harvard

Pan, J. et al. (2026) ‘Addressing benchmarking gaps in large language models for health and medicine with dynamic red-teaming’, Nature Health, 1(11), pp. 1254–1270. doi:10.1038/s44360-026-00152-8.

IEEE

J. Pan et al., “Addressing benchmarking gaps in large language models for health and medicine with dynamic red-teaming,” Nature Health, vol. 1, no. 11, pp. 1254–1270, 2026, doi: 10.1038/s44360-026-00152-8.

BibTeX

@article{pan2026addressing,
  title     = {{Addressing benchmarking gaps in large language models for health and medicine with dynamic red-teaming}},
  author    = {Pan, Jiazhen and Jian, Bailiang and Hager, Paul and Zhang, Yundi and Liu, Che and Jungmann, Friedrike and Li, Hongwei and Canisius, Julian and You, Chenyu and Wu, Junde and Zhu, Jiayuan and Liu, Fenglin and Liu, Yuyuan and Bubeck, Niklas and Knolle, Moritz A. and Chen, Chen and Wachinger, Christian and Gong, Zhenyu and Ouyang, Cheng and Kaissis, Georgios A. and Wiestler, Benedikt and Rueckert, Daniel},
  journal   = {Nature Health},
  year      = {2026},
  volume    = {1},
  number    = {11},
  pages     = {1254--1270},
  doi       = {10.1038/s44360-026-00152-8},
  url       = {https://doi.org/10.1038/s44360-026-00152-8}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education

  • large language models
  • red teaming
  • LLM safety evaluation
  • adversarial testing
  • robustness
  • privacy leakage
  • algorithmic bias
  • hallucinations

Work with this paper

Most-cited papers in Medicine

All highly cited Medicine papers

Questions about this paper

How do I cite this paper in APA style?

Pan, J., Jian, B., Hager, P., Zhang, Y., Liu, C., Jungmann, F., Li, H., Canisius, J., You, C., Wu, J., Zhu, J., Liu, F., Liu, Y., Bubeck, N., Knolle, M. A., Chen, C., Wachinger, C., Gong, Z., Ouyang, C., . . . Rueckert, D. (2026). Addressing benchmarking gaps in large language models for health and medicine with dynamic red-teaming. Nature Health, 1(11), 1254–1270. https://doi.org/10.1038/s44360-026-00152-8

How many times has this paper been cited?

29 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at Nature Health: sign in and open the PDF from this page.

Who wrote this paper?

Jiazhen Pan, Bailiang Jian, Paul Hager, Yundi Zhang, Che Liu, Friedrike Jungmann, Hongwei Li, Julian Canisius, Chenyu You, Junde Wu, Jiayuan Zhu, Fenglin Liu and 10 more.

What is the DOI of this paper?

10.1038/s44360-026-00152-8, permanently at https://doi.org/10.1038/s44360-026-00152-8. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.