Journal article · 2024 · Open access
Evaluating large language models as agents in the clinic
npj Digital Medicine, 7(1), 84 · Published 3 April 2024 · DOI 10.1038/s41746-024-01083-y
In brief
“Evaluating large language models as agents in the clinic” is a 2024 journal article by Nikita Mehandru and 5 co-authors, published in npj Digital Medicine. OpenAlex counts 253 citations as of 22 November 2026, which puts it in the top 1% of works published in 2024 by citations. It is open access under a CC BY licence: the full text is free to read at npj Digital Medicine.
- Citations
- 253
- Most cited in
- 2026 (117)
- Field-weighted impact
- 16.22
- References
- 30
- Authors
- 6
Abstract
Recent developments in large language models (LLMs) have unlocked opportunities for healthcare, from information synthesis to clinical decision support. These LLMs are not just capable of modeling language, but can also act as intelligent “agents” that interact with stakeholders in open-ended conversations and even influence clinical decision-making. Rather than relying on benchmarks that measure a model’s ability to process clinical data or answer standardized test questions, LLM agents can be modeled in high-fidelity simulations of clinical settings and should be assessed for their impact on clinical workflows. These evaluation frameworks, which we refer to as “Artificial Intelligence Structured Clinical Examinations” (“AI-SCE”), can draw from comparable technologies where machines operate with varying degrees of self-governance, such as self-driving cars, in dynamic environments with multiple stakeholders. Developing these robust, real-world clinical evaluations will be crucial towards deploying LLM agents in medical settings.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Mehandru, N., Miao, B. Y., Almaraz, E. R., Sushil, M., Butte, A. J., & Alaa, A. M. (2024). Evaluating large language models as agents in the clinic. npj Digital Medicine, 7(1), 84. https://doi.org/10.1038/s41746-024-01083-y
MLA 9
Mehandru, Nikita, et al. “Evaluating large language models as agents in the clinic.” npj Digital Medicine, vol. 7, no. 1, 2024, p. 84. https://doi.org/10.1038/s41746-024-01083-y.
Chicago
Mehandru, Nikita, Brenda Y. Miao, Eduardo Rodriguez Almaraz, Madhumita Sushil, Atul Janardhan Butte, and Ahmed M. Alaa. 2024. “Evaluating large language models as agents in the clinic.” npj Digital Medicine 7 (1): 84. https://doi.org/10.1038/s41746-024-01083-y.
Harvard
Mehandru, N. et al. (2024) ‘Evaluating large language models as agents in the clinic’, npj Digital Medicine, 7(1), p. 84. doi:10.1038/s41746-024-01083-y.
IEEE
N. Mehandru, B. Y. Miao, E. R. Almaraz, M. Sushil, A. J. Butte, and A. M. Alaa, “Evaluating large language models as agents in the clinic,” npj Digital Medicine, vol. 7, no. 1, p. 84, 2024, doi: 10.1038/s41746-024-01083-y.
BibTeX
@article{mehandru2024evaluating,
title = {{Evaluating large language models as agents in the clinic}},
author = {Mehandru, Nikita and Miao, Brenda Y. and Almaraz, Eduardo Rodriguez and Sushil, Madhumita and Butte, Atul Janardhan and Alaa, Ahmed M.},
journal = {npj Digital Medicine},
year = {2024},
volume = {7},
number = {1},
pages = {84},
doi = {10.1038/s41746-024-01083-y},
url = {https://doi.org/10.1038/s41746-024-01083-y}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Citations by year
| Year | Citations | Share of the best year |
|---|---|---|
| 2024 | 28 | |
| 2025 | 108 | |
| 2026 | 117 |
Its field-weighted citation impact is 16.22: it has been cited 16.22 times as often as the average work of the same field, type and year (1.00 is the world average).
Topic and keywords
Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- large language models
- LLM agents
- clinical decision support
- AI evaluation framework
- clinical simulation
- clinical workflow
- clinical implementation of AI
- benchmarking
Work with this paper
Most-cited papers in Medicine
- Yaşlı Kadınlarda Üreme SağlığıDemet Kokanalı, Yaprak Engin Üstün · Jinekoloji-Obstetrik ve Neonatoloji Tıp Dergisi · 2019 · 169,630 citations
- Global Cancer Statistics 2020: GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 CountriesHyuna Sung, Adalberto M. Filho, Mathieu Laversanne and 5 more · CA A Cancer Journal for Clinicians · 2021 · 119,885 citations
- “Mini-mental state”Marshal F. Folstein, Susan E. Folstein, Paul R. McHugh · Journal of Psychiatric Research · 1975 · 90,424 citations
- Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countriesFreddie Bray, Jacques Ferlay, Isabelle Soerjomataram and 3 more · CA A Cancer Journal for Clinicians · 2018 · 87,396 citations
- Hallmarks of Cancer: The Next GenerationDouglas Hanahan, Robert A. Weinberg · Cell · 2011 · 69,758 citations
Questions about this paper
How do I cite this paper in APA style?
Mehandru, N., Miao, B. Y., Almaraz, E. R., Sushil, M., Butte, A. J., & Alaa, A. M. (2024). Evaluating large language models as agents in the clinic. npj Digital Medicine, 7(1), 84. https://doi.org/10.1038/s41746-024-01083-y
How many times has this paper been cited?
253 times, according to OpenAlex on 22 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at npj Digital Medicine: sign in and open the PDF from this page.
Who wrote this paper?
Nikita Mehandru, Brenda Y. Miao, Eduardo Rodriguez Almaraz, Madhumita Sushil, Atul Janardhan Butte, Ahmed M. Alaa.
What is the DOI of this paper?
10.1038/s41746-024-01083-y, permanently at https://doi.org/10.1038/s41746-024-01083-y. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 22 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.