Journal article · 2025 · Open access

Automating expert-level medical reasoning evaluation of large language models

Shuang Zhou, Wenya Xie, Jiaxi Li, Zaifu Zhan, Meijia Song, Han Yang, Cheyenna Espinoza, Lindsay L. Welton, Xinnie Mai, Yanwei Jin, Zidu Xu, Yuen-Hei Chung and 7 more

npj Digital Medicine, 9(1), 34 · Published 6 December 2025 · DOI 10.1038/s41746-025-02208-7

Open PDFView at the publisherCite this paper

In brief

“Automating expert-level medical reasoning evaluation of large language models” is a 2025 journal article by Shuang Zhou and 18 co-authors, published in npj Digital Medicine. OpenAlex counts 84 citations as of 17 November 2026, which puts it in the top 4% of works published in 2025 by citations. It is open access under a CC BY licence: the full text is free to read at npj Digital Medicine.

Citations
84
Most cited in
2026 (80)
Field-weighted impact
12.37
References
69
Authors
19

Abstract

As large language models (LLMs) become increasingly integrated into clinical decision-making, ensuring trustworthy reasoning is paramount. However, current evaluation strategies of LLMs' medical reasoning capability either suffer from unsatisfactory assessment or poor scalability, and a rigorous benchmark remains absent. To address this, we present MedThink-Bench, a benchmark designed for rigorous and scalable assessment of LLMs' medical reasoning. MedThink-Bench comprises 500 high-complexity questions spanning ten medical domains, accompanied by expert-authored, step-by-step rationales that elucidate intermediate reasoning processes. Further, we introduce LLM-w-Rationale, an evaluation framework that combines fine-grained rationale assessment with an LLM-as-a-Judge paradigm, enabling expert-level fidelity in evaluating reasoning quality while preserving scalability. Results show that LLM-w-Rationale correlates strongly with expert evaluation (Pearson coefficient up to 0.87) while requiring only 1.4% of the evaluation time. Overall, MedThink-Bench establishes a rigorous and scalable standard for evaluating medical reasoning in LLMs, advancing their safe and responsible deployment in clinical practice.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Zhou, S., Xie, W., Li, J., Zhan, Z., Song, M., Yang, H., Espinoza, C., Welton, L. L., Mai, X., Jin, Y., Xu, Z., Chung, Y.-H., Xing, Y., Tsai, M.-H., Schaffer, E., Shi, Y., Liu, N., Liu, Z., & Zhang, R. (2025). Automating expert-level medical reasoning evaluation of large language models. npj Digital Medicine, 9(1), 34. https://doi.org/10.1038/s41746-025-02208-7

MLA 9

Zhou, Shuang, et al. “Automating expert-level medical reasoning evaluation of large language models.” npj Digital Medicine, vol. 9, no. 1, 2025, p. 34. https://doi.org/10.1038/s41746-025-02208-7.

Chicago

Zhou, Shuang, Wenya Xie, Jiaxi Li, Zaifu Zhan, Meijia Song, Han Yang, Cheyenna Espinoza, et al. 2025. “Automating expert-level medical reasoning evaluation of large language models.” npj Digital Medicine 9 (1): 34. https://doi.org/10.1038/s41746-025-02208-7.

Harvard

Zhou, S. et al. (2025) ‘Automating expert-level medical reasoning evaluation of large language models’, npj Digital Medicine, 9(1), p. 34. doi:10.1038/s41746-025-02208-7.

IEEE

S. Zhou et al., “Automating expert-level medical reasoning evaluation of large language models,” npj Digital Medicine, vol. 9, no. 1, p. 34, 2025, doi: 10.1038/s41746-025-02208-7.

BibTeX

@article{zhou2025automating,
  title     = {{Automating expert-level medical reasoning evaluation of large language models}},
  author    = {Zhou, Shuang and Xie, Wenya and Li, Jiaxi and Zhan, Zaifu and Song, Meijia and Yang, Han and Espinoza, Cheyenna and Welton, Lindsay L. and Mai, Xinnie and Jin, Yanwei and Xu, Zidu and Chung, Yuen-Hei and Xing, Yiyun and Tsai, Meng-Han and Schaffer, Emma and Shi, Yucheng and Liu, Ninghao and Liu, Zirui and Zhang, Rui},
  journal   = {npj Digital Medicine},
  year      = {2025},
  volume    = {9},
  number    = {1},
  pages     = {34},
  doi       = {10.1038/s41746-025-02208-7},
  url       = {https://doi.org/10.1038/s41746-025-02208-7}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education

  • large language models
  • medical reasoning
  • LLM evaluation
  • benchmark
  • LLM-as-a-judge
  • expert evaluation
  • clinical decision support

Work with this paper

Most-cited papers in Medicine

All highly cited Medicine papers

Questions about this paper

How do I cite this paper in APA style?

Zhou, S., Xie, W., Li, J., Zhan, Z., Song, M., Yang, H., Espinoza, C., Welton, L. L., Mai, X., Jin, Y., Xu, Z., Chung, Y.-H., Xing, Y., Tsai, M.-H., Schaffer, E., Shi, Y., Liu, N., Liu, Z., & Zhang, R. (2025). Automating expert-level medical reasoning evaluation of large language models. npj Digital Medicine, 9(1), 34. https://doi.org/10.1038/s41746-025-02208-7

How many times has this paper been cited?

84 times, according to OpenAlex on 17 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at npj Digital Medicine: sign in and open the PDF from this page.

Who wrote this paper?

Shuang Zhou, Wenya Xie, Jiaxi Li, Zaifu Zhan, Meijia Song, Han Yang, Cheyenna Espinoza, Lindsay L. Welton, Xinnie Mai, Yanwei Jin, Zidu Xu, Yuen-Hei Chung and 7 more.

What is the DOI of this paper?

10.1038/s41746-025-02208-7, permanently at https://doi.org/10.1038/s41746-025-02208-7. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 17 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.