Review article · 2024 · Open access
Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review
JAMA, 333(4), 319 · Published 15 October 2024 · DOI 10.1001/jama.2024.21700
In brief
“Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review” is a 2024 review article by Suhana Bedi and 18 co-authors, published in JAMA. OpenAlex counts 1,042 citations as of 19 November 2026, which puts it in the top 1% of works published in 2024 by citations. It is open access under a CC BY-NC-ND licence: the full text is free to read at medRxiv.
- Citations
- 1,042
- Most cited in
- 2026 (676)
- Field-weighted impact
- 15.48
- References
- 71
- Authors
- 19
Abstract
Importance Large language models (LLMs) can assist in various health care activities, but current evaluation approaches may not adequately identify the most useful application areas. Objective To summarize existing evaluations of LLMs in health care in terms of 5 components: (1) evaluation data type, (2) health care task, (3) natural language processing (NLP) and natural language understanding (NLU) tasks, (4) dimension of evaluation, and (5) medical specialty. Data Sources A systematic search of PubMed and Web of Science was performed for studies published between January 1, 2022, and February 19, 2024. Study Selection Studies evaluating 1 or more LLMs in health care. Data Extraction and Synthesis Three independent reviewers categorized studies via keyword searches based on the data used, the health care tasks, the NLP and NLU tasks, the dimensions of evaluation, and the medical specialty. Results Of 519 studies reviewed, published between January 1, 2022, and February 19, 2024, only 5% used real patient care data for LLM evaluation. The most common health care tasks were assessing medical knowledge such as answering medical licensing examination questions (44.5%) and making diagnoses (19.5%). Administrative tasks such as assigning billing codes (0.2%) and writing prescriptions (0.2%) were less studied. For NLP and NLU tasks, most studies focused on question answering (84.2%), while tasks such as summarization (8.9%) and conversational dialogue (3.3%) were infrequent. Almost all studies (95.4%) used accuracy as the primary dimension of evaluation; fairness, bias, and toxicity (15.8%), deployment considerations (4.6%), and calibration and uncertainty (1.2%) were infrequently measured. Finally, in terms of medical specialty area, most studies were in generic health care applications (25.6%), internal medicine (16.4%), surgery (11.4%), and ophthalmology (6.9%), with nuclear medicine (0.6%), physical medicine (0.4%), and medical genetics (0.2%) being the least represented. Conclusions and Relevance Existing evaluations of LLMs mostly focus on accuracy of question answering for medical examinations, without consideration of real patient care data. Dimensions such as fairness, bias, and toxicity and deployment considerations received limited attention. Future evaluations should adopt standardized applications and metrics, use clinical data, and broaden focus to include a wider range of tasks and specialties.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Bedi, S., Liu, Y., Orr-Ewing, L., Dash, D., Koyejo, S., Callahan, A., Fries, J., Wornow, M., Swaminathan, A., Lehmann, L. S., Hong, H. J., Kashyap, M., Chaurasia, A., Shah, N. R., Singh, K., Tazbaz, T., Milstein, A. S., Pfeffer, M. A., & Shah, N. H. (2024). Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review. JAMA, 333(4), 319. https://doi.org/10.1001/jama.2024.21700
MLA 9
Bedi, Suhana, et al. “Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review.” JAMA, vol. 333, no. 4, 2024, p. 319. https://doi.org/10.1001/jama.2024.21700.
Chicago
Bedi, Suhana, Yutong Liu, Lucy Orr-Ewing, Dev Dash, Sanmi Koyejo, Alison Callahan, Jason Fries, et al. 2024. “Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review.” JAMA 333 (4): 319. https://doi.org/10.1001/jama.2024.21700.
Harvard
Bedi, S. et al. (2024) ‘Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review’, JAMA, 333(4), p. 319. doi:10.1001/jama.2024.21700.
IEEE
S. Bedi et al., “Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review,” JAMA, vol. 333, no. 4, p. 319, 2024, doi: 10.1001/jama.2024.21700.
BibTeX
@article{bedi2024testing,
title = {{Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review}},
author = {Bedi, Suhana and Liu, Yutong and Orr-Ewing, Lucy and Dash, Dev and Koyejo, Sanmi and Callahan, Alison and Fries, Jason and Wornow, Michael and Swaminathan, Akshay and Lehmann, Lisa Soleymani and Hong, Hyo Jung and Kashyap, Mehr and Chaurasia, Akash and Shah, Nirav R. and Singh, Karandeep and Tazbaz, Troy and Milstein, Arnold S. and Pfeffer, Michael A. and Shah, Nigam Haresh},
journal = {JAMA},
year = {2024},
volume = {333},
number = {4},
pages = {319},
doi = {10.1001/jama.2024.21700},
url = {https://doi.org/10.1001/jama.2024.21700}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Citations by year
| Year | Citations | Share of the best year |
|---|---|---|
| 2023 | 1 | |
| 2024 | 21 | |
| 2025 | 344 | |
| 2026 | 676 |
Its field-weighted citation impact is 15.48: it has been cited 15.48 times as often as the average work of the same field, type and year (1.00 is the world average).
Topic and keywords
Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- large language models
- health care applications
- LLM evaluation
- natural language processing
- natural language understanding
- question answering
- medical licensing examinations
- medical question answering
Work with this paper
Most-cited papers in Medicine
- Yaşlı Kadınlarda Üreme SağlığıDemet Kokanalı, Yaprak Engin Üstün · Jinekoloji-Obstetrik ve Neonatoloji Tıp Dergisi · 2019 · 169,630 citations
- Global Cancer Statistics 2020: GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 CountriesHyuna Sung, Adalberto M. Filho, Mathieu Laversanne and 5 more · CA A Cancer Journal for Clinicians · 2021 · 119,885 citations
- “Mini-mental state”Marshal F. Folstein, Susan E. Folstein, Paul R. McHugh · Journal of Psychiatric Research · 1975 · 90,424 citations
- Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countriesFreddie Bray, Jacques Ferlay, Isabelle Soerjomataram and 3 more · CA A Cancer Journal for Clinicians · 2018 · 87,396 citations
- Hallmarks of Cancer: The Next GenerationDouglas Hanahan, Robert A. Weinberg · Cell · 2011 · 69,758 citations
Questions about this paper
How do I cite this paper in APA style?
Bedi, S., Liu, Y., Orr-Ewing, L., Dash, D., Koyejo, S., Callahan, A., Fries, J., Wornow, M., Swaminathan, A., Lehmann, L. S., Hong, H. J., Kashyap, M., Chaurasia, A., Shah, N. R., Singh, K., Tazbaz, T., Milstein, A. S., Pfeffer, M. A., & Shah, N. H. (2024). Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review. JAMA, 333(4), 319. https://doi.org/10.1001/jama.2024.21700
How many times has this paper been cited?
1,042 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at medRxiv: sign in and open the PDF from this page.
Who wrote this paper?
Suhana Bedi, Yutong Liu, Lucy Orr-Ewing, Dev Dash, Sanmi Koyejo, Alison Callahan, Jason Fries, Michael Wornow, Akshay Swaminathan, Lisa Soleymani Lehmann, Hyo Jung Hong, Mehr Kashyap and 7 more.
What is the DOI of this paper?
10.1001/jama.2024.21700, permanently at https://doi.org/10.1001/jama.2024.21700. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.