Journal article · 2025 · Open access
Automating expert-level medical reasoning evaluation of large language models
npj Digital Medicine, 9(1), 34 · Published 6 December 2025 · DOI 10.1038/s41746-025-02208-7
In brief
“Automating expert-level medical reasoning evaluation of large language models” is a 2025 journal article by Shuang Zhou and 18 co-authors, published in npj Digital Medicine. OpenAlex counts 84 citations as of 17 November 2026, which puts it in the top 4% of works published in 2025 by citations. It is open access under a CC BY licence: the full text is free to read at npj Digital Medicine.
- Citations
- 84
- Most cited in
- 2026 (80)
- Field-weighted impact
- 12.37
- References
- 69
- Authors
- 19
Abstract
As large language models (LLMs) become increasingly integrated into clinical decision-making, ensuring trustworthy reasoning is paramount. However, current evaluation strategies of LLMs' medical reasoning capability either suffer from unsatisfactory assessment or poor scalability, and a rigorous benchmark remains absent. To address this, we present MedThink-Bench, a benchmark designed for rigorous and scalable assessment of LLMs' medical reasoning. MedThink-Bench comprises 500 high-complexity questions spanning ten medical domains, accompanied by expert-authored, step-by-step rationales that elucidate intermediate reasoning processes. Further, we introduce LLM-w-Rationale, an evaluation framework that combines fine-grained rationale assessment with an LLM-as-a-Judge paradigm, enabling expert-level fidelity in evaluating reasoning quality while preserving scalability. Results show that LLM-w-Rationale correlates strongly with expert evaluation (Pearson coefficient up to 0.87) while requiring only 1.4% of the evaluation time. Overall, MedThink-Bench establishes a rigorous and scalable standard for evaluating medical reasoning in LLMs, advancing their safe and responsible deployment in clinical practice.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Zhou, S., Xie, W., Li, J., Zhan, Z., Song, M., Yang, H., Espinoza, C., Welton, L. L., Mai, X., Jin, Y., Xu, Z., Chung, Y.-H., Xing, Y., Tsai, M.-H., Schaffer, E., Shi, Y., Liu, N., Liu, Z., & Zhang, R. (2025). Automating expert-level medical reasoning evaluation of large language models. npj Digital Medicine, 9(1), 34. https://doi.org/10.1038/s41746-025-02208-7
MLA 9
Zhou, Shuang, et al. “Automating expert-level medical reasoning evaluation of large language models.” npj Digital Medicine, vol. 9, no. 1, 2025, p. 34. https://doi.org/10.1038/s41746-025-02208-7.
Chicago
Zhou, Shuang, Wenya Xie, Jiaxi Li, Zaifu Zhan, Meijia Song, Han Yang, Cheyenna Espinoza, et al. 2025. “Automating expert-level medical reasoning evaluation of large language models.” npj Digital Medicine 9 (1): 34. https://doi.org/10.1038/s41746-025-02208-7.
Harvard
Zhou, S. et al. (2025) ‘Automating expert-level medical reasoning evaluation of large language models’, npj Digital Medicine, 9(1), p. 34. doi:10.1038/s41746-025-02208-7.
IEEE
S. Zhou et al., “Automating expert-level medical reasoning evaluation of large language models,” npj Digital Medicine, vol. 9, no. 1, p. 34, 2025, doi: 10.1038/s41746-025-02208-7.
BibTeX
@article{zhou2025automating,
title = {{Automating expert-level medical reasoning evaluation of large language models}},
author = {Zhou, Shuang and Xie, Wenya and Li, Jiaxi and Zhan, Zaifu and Song, Meijia and Yang, Han and Espinoza, Cheyenna and Welton, Lindsay L. and Mai, Xinnie and Jin, Yanwei and Xu, Zidu and Chung, Yuen-Hei and Xing, Yiyun and Tsai, Meng-Han and Schaffer, Emma and Shi, Yucheng and Liu, Ninghao and Liu, Zirui and Zhang, Rui},
journal = {npj Digital Medicine},
year = {2025},
volume = {9},
number = {1},
pages = {34},
doi = {10.1038/s41746-025-02208-7},
url = {https://doi.org/10.1038/s41746-025-02208-7}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- large language models
- medical reasoning
- LLM evaluation
- benchmark
- LLM-as-a-judge
- expert evaluation
- clinical decision support
Work with this paper
Most-cited papers in Medicine
- Yaşlı Kadınlarda Üreme SağlığıDemet Kokanalı, Yaprak Engin Üstün · Jinekoloji-Obstetrik ve Neonatoloji Tıp Dergisi · 2019 · 169,630 citations
- Global Cancer Statistics 2020: GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 CountriesHyuna Sung, Adalberto M. Filho, Mathieu Laversanne and 5 more · CA A Cancer Journal for Clinicians · 2021 · 119,885 citations
- “Mini-mental state”Marshal F. Folstein, Susan E. Folstein, Paul R. McHugh · Journal of Psychiatric Research · 1975 · 90,424 citations
- Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countriesFreddie Bray, Jacques Ferlay, Isabelle Soerjomataram and 3 more · CA A Cancer Journal for Clinicians · 2018 · 87,396 citations
- Hallmarks of Cancer: The Next GenerationDouglas Hanahan, Robert A. Weinberg · Cell · 2011 · 69,758 citations
Questions about this paper
How do I cite this paper in APA style?
Zhou, S., Xie, W., Li, J., Zhan, Z., Song, M., Yang, H., Espinoza, C., Welton, L. L., Mai, X., Jin, Y., Xu, Z., Chung, Y.-H., Xing, Y., Tsai, M.-H., Schaffer, E., Shi, Y., Liu, N., Liu, Z., & Zhang, R. (2025). Automating expert-level medical reasoning evaluation of large language models. npj Digital Medicine, 9(1), 34. https://doi.org/10.1038/s41746-025-02208-7
How many times has this paper been cited?
84 times, according to OpenAlex on 17 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at npj Digital Medicine: sign in and open the PDF from this page.
Who wrote this paper?
Shuang Zhou, Wenya Xie, Jiaxi Li, Zaifu Zhan, Meijia Song, Han Yang, Cheyenna Espinoza, Lindsay L. Welton, Xinnie Mai, Yanwei Jin, Zidu Xu, Yuen-Hei Chung and 7 more.
What is the DOI of this paper?
10.1038/s41746-025-02208-7, permanently at https://doi.org/10.1038/s41746-025-02208-7. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 17 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.