Journal article · 2025 · Open access

A novel evaluation benchmark for medical LLMs illuminating safety and effectiveness in clinical domains

Shirui Wang, Zhihui Tang, Huaxia Yang, Qiuhong Gong, Tiantian Gu, Hongyang Ma, Yongxin Wang, Wubin Sun, Zeliang Lian, Kehang Mao, Yinan Jiang, Zhicheng Huang and 26 more

npj Digital Medicine, 9(1), 91 · Published 26 December 2025 · DOI 10.1038/s41746-025-02277-8

Open full textCite this paper

In brief

“A novel evaluation benchmark for medical LLMs illuminating safety and effectiveness in clinical domains” is a 2025 journal article by Shirui Wang and 37 co-authors, published in npj Digital Medicine. OpenAlex counts 78 citations as of 17 November 2026, which puts it in the top 3% of works published in 2025 by citations. It is open access under a CC BY licence: the full text is free to read at npj Digital Medicine.

Citations
78
Most cited in
2026 (73)
Field-weighted impact
11.49
References
37
Authors
38

Abstract

Large language models (LLMs) hold promise in clinical decision support but face major challenges in safety evaluation and effectiveness validation. We developed the Clinical Safety-Effectiveness Dual-Track Benchmark (CSEDB), a multidimensional framework built on clinical expert consensus, encompassing 30 metrics covering critical areas like critical illness recognition, guideline adherence, and medication safety, with weighted consequence measures. Thirty-two specialist physicians developed and revised 2069 open-ended Q&A items aligned with these criteria, spanning 26 clinical departments to simulate real-world scenarios. Benchmark testing of six LLMs revealed moderate overall performance (average total score 57.2%, safety 54.7%, effectiveness 62.3%), with a significant 13.3% performance drop in high-risk scenarios (p < 0.0001). Domain-specific medical LLMs showed consistent performance advantages over general-purpose models, with relatively higher top scores in safety (0.912) and effectiveness (0.861). The findings of this study not only provide a standardized metric for evaluating the clinical application of medical LLMs, facilitating comparative analyses, risk exposure identification, and improvement directions across different scenarios, but also hold the potential to promote safer and more effective deployment of large language models in healthcare environments.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Wang, S., Tang, Z., Yang, H., Gong, Q., Gu, T., Ma, H., Wang, Y., Sun, W., Lian, Z., Mao, K., Jiang, Y., Huang, Z., Ma, L., Shen, W. D., Ji, Y., Tan, Y., Wang, C., Gao, Y., Ye, Q., . . . Wu, J. (2025). A novel evaluation benchmark for medical LLMs illuminating safety and effectiveness in clinical domains. npj Digital Medicine, 9(1), 91. https://doi.org/10.1038/s41746-025-02277-8

MLA 9

Wang, Shirui, et al. “A novel evaluation benchmark for medical LLMs illuminating safety and effectiveness in clinical domains.” npj Digital Medicine, vol. 9, no. 1, 2025, p. 91. https://doi.org/10.1038/s41746-025-02277-8.

Chicago

Wang, Shirui, Zhihui Tang, Huaxia Yang, Qiuhong Gong, Tiantian Gu, Hongyang Ma, Yongxin Wang, et al. 2025. “A novel evaluation benchmark for medical LLMs illuminating safety and effectiveness in clinical domains.” npj Digital Medicine 9 (1): 91. https://doi.org/10.1038/s41746-025-02277-8.

Harvard

Wang, S. et al. (2025) ‘A novel evaluation benchmark for medical LLMs illuminating safety and effectiveness in clinical domains’, npj Digital Medicine, 9(1), p. 91. doi:10.1038/s41746-025-02277-8.

IEEE

S. Wang et al., “A novel evaluation benchmark for medical LLMs illuminating safety and effectiveness in clinical domains,” npj Digital Medicine, vol. 9, no. 1, p. 91, 2025, doi: 10.1038/s41746-025-02277-8.

BibTeX

@article{wang2025novel,
  title     = {{A novel evaluation benchmark for medical LLMs illuminating safety and effectiveness in clinical domains}},
  author    = {Wang, Shirui and Tang, Zhihui and Yang, Huaxia and Gong, Qiuhong and Gu, Tiantian and Ma, Hongyang and Wang, Yongxin and Sun, Wubin and Lian, Zeliang and Mao, Kehang and Jiang, Yinan and Huang, Zhicheng and Ma, Lingyun and Shen, W D and Ji, Yajie and Tan, Yunhui and Wang, Chunbo and Gao, Yunlu and Ye, Qianling and Lin, Rui and Chen, MingYu and Niu, Lijuan and Wang, Zhihao and Yu, Peng and Lang, Mengran and Liu, Yue and Zhang, Huimin and Shen, Haitao and Chen, Long and Zhao, Q M and others},
  journal   = {npj Digital Medicine},
  year      = {2025},
  volume    = {9},
  number    = {1},
  pages     = {91},
  doi       = {10.1038/s41746-025-02277-8},
  url       = {https://doi.org/10.1038/s41746-025-02277-8}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education

  • medical large language models
  • benchmark evaluation
  • clinical safety
  • clinical effectiveness
  • clinical decision support
  • guideline adherence
  • drug safety
  • expert consensus

Work with this paper

Most-cited papers in Medicine

All highly cited Medicine papers

Questions about this paper

How do I cite this paper in APA style?

Wang, S., Tang, Z., Yang, H., Gong, Q., Gu, T., Ma, H., Wang, Y., Sun, W., Lian, Z., Mao, K., Jiang, Y., Huang, Z., Ma, L., Shen, W. D., Ji, Y., Tan, Y., Wang, C., Gao, Y., Ye, Q., . . . Wu, J. (2025). A novel evaluation benchmark for medical LLMs illuminating safety and effectiveness in clinical domains. npj Digital Medicine, 9(1), 91. https://doi.org/10.1038/s41746-025-02277-8

How many times has this paper been cited?

78 times, according to OpenAlex on 17 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at npj Digital Medicine: sign in and open the full text from this page.

Who wrote this paper?

Shirui Wang, Zhihui Tang, Huaxia Yang, Qiuhong Gong, Tiantian Gu, Hongyang Ma, Yongxin Wang, Wubin Sun, Zeliang Lian, Kehang Mao, Yinan Jiang, Zhicheng Huang and 26 more.

What is the DOI of this paper?

10.1038/s41746-025-02277-8, permanently at https://doi.org/10.1038/s41746-025-02277-8. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 17 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.