Review article · 2024 · Open access

Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review

Suhana Bedi, Yutong Liu, Lucy Orr-Ewing, Dev Dash, Sanmi Koyejo, Alison Callahan, Jason Fries, Michael Wornow, Akshay Swaminathan, Lisa Soleymani Lehmann, Hyo Jung Hong, Mehr Kashyap and 7 more

JAMA, 333(4), 319 · Published 15 October 2024 · DOI 10.1001/jama.2024.21700

Open PDFView at the publisherCite this paper

In brief

“Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review” is a 2024 review article by Suhana Bedi and 18 co-authors, published in JAMA. OpenAlex counts 1,042 citations as of 19 November 2026, which puts it in the top 1% of works published in 2024 by citations. It is open access under a CC BY-NC-ND licence: the full text is free to read at medRxiv.

Citations
1,042
Most cited in
2026 (676)
Field-weighted impact
15.48
References
71
Authors
19

Abstract

Importance Large language models (LLMs) can assist in various health care activities, but current evaluation approaches may not adequately identify the most useful application areas. Objective To summarize existing evaluations of LLMs in health care in terms of 5 components: (1) evaluation data type, (2) health care task, (3) natural language processing (NLP) and natural language understanding (NLU) tasks, (4) dimension of evaluation, and (5) medical specialty. Data Sources A systematic search of PubMed and Web of Science was performed for studies published between January 1, 2022, and February 19, 2024. Study Selection Studies evaluating 1 or more LLMs in health care. Data Extraction and Synthesis Three independent reviewers categorized studies via keyword searches based on the data used, the health care tasks, the NLP and NLU tasks, the dimensions of evaluation, and the medical specialty. Results Of 519 studies reviewed, published between January 1, 2022, and February 19, 2024, only 5% used real patient care data for LLM evaluation. The most common health care tasks were assessing medical knowledge such as answering medical licensing examination questions (44.5%) and making diagnoses (19.5%). Administrative tasks such as assigning billing codes (0.2%) and writing prescriptions (0.2%) were less studied. For NLP and NLU tasks, most studies focused on question answering (84.2%), while tasks such as summarization (8.9%) and conversational dialogue (3.3%) were infrequent. Almost all studies (95.4%) used accuracy as the primary dimension of evaluation; fairness, bias, and toxicity (15.8%), deployment considerations (4.6%), and calibration and uncertainty (1.2%) were infrequently measured. Finally, in terms of medical specialty area, most studies were in generic health care applications (25.6%), internal medicine (16.4%), surgery (11.4%), and ophthalmology (6.9%), with nuclear medicine (0.6%), physical medicine (0.4%), and medical genetics (0.2%) being the least represented. Conclusions and Relevance Existing evaluations of LLMs mostly focus on accuracy of question answering for medical examinations, without consideration of real patient care data. Dimensions such as fairness, bias, and toxicity and deployment considerations received limited attention. Future evaluations should adopt standardized applications and metrics, use clinical data, and broaden focus to include a wider range of tasks and specialties.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Bedi, S., Liu, Y., Orr-Ewing, L., Dash, D., Koyejo, S., Callahan, A., Fries, J., Wornow, M., Swaminathan, A., Lehmann, L. S., Hong, H. J., Kashyap, M., Chaurasia, A., Shah, N. R., Singh, K., Tazbaz, T., Milstein, A. S., Pfeffer, M. A., & Shah, N. H. (2024). Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review. JAMA, 333(4), 319. https://doi.org/10.1001/jama.2024.21700

MLA 9

Bedi, Suhana, et al. “Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review.” JAMA, vol. 333, no. 4, 2024, p. 319. https://doi.org/10.1001/jama.2024.21700.

Chicago

Bedi, Suhana, Yutong Liu, Lucy Orr-Ewing, Dev Dash, Sanmi Koyejo, Alison Callahan, Jason Fries, et al. 2024. “Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review.” JAMA 333 (4): 319. https://doi.org/10.1001/jama.2024.21700.

Harvard

Bedi, S. et al. (2024) ‘Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review’, JAMA, 333(4), p. 319. doi:10.1001/jama.2024.21700.

IEEE

S. Bedi et al., “Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review,” JAMA, vol. 333, no. 4, p. 319, 2024, doi: 10.1001/jama.2024.21700.

BibTeX

@article{bedi2024testing,
  title     = {{Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review}},
  author    = {Bedi, Suhana and Liu, Yutong and Orr-Ewing, Lucy and Dash, Dev and Koyejo, Sanmi and Callahan, Alison and Fries, Jason and Wornow, Michael and Swaminathan, Akshay and Lehmann, Lisa Soleymani and Hong, Hyo Jung and Kashyap, Mehr and Chaurasia, Akash and Shah, Nirav R. and Singh, Karandeep and Tazbaz, Troy and Milstein, Arnold S. and Pfeffer, Michael A. and Shah, Nigam Haresh},
  journal   = {JAMA},
  year      = {2024},
  volume    = {333},
  number    = {4},
  pages     = {319},
  doi       = {10.1001/jama.2024.21700},
  url       = {https://doi.org/10.1001/jama.2024.21700}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Citations by year

YearCitationsShare of the best year
202311
20242121
2025344344
2026676676

Its field-weighted citation impact is 15.48: it has been cited 15.48 times as often as the average work of the same field, type and year (1.00 is the world average).

Topic and keywords

Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education

  • large language models
  • health care applications
  • LLM evaluation
  • natural language processing
  • natural language understanding
  • question answering
  • medical licensing examinations
  • medical question answering

Work with this paper

Most-cited papers in Medicine

All highly cited Medicine papers

Questions about this paper

How do I cite this paper in APA style?

Bedi, S., Liu, Y., Orr-Ewing, L., Dash, D., Koyejo, S., Callahan, A., Fries, J., Wornow, M., Swaminathan, A., Lehmann, L. S., Hong, H. J., Kashyap, M., Chaurasia, A., Shah, N. R., Singh, K., Tazbaz, T., Milstein, A. S., Pfeffer, M. A., & Shah, N. H. (2024). Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review. JAMA, 333(4), 319. https://doi.org/10.1001/jama.2024.21700

How many times has this paper been cited?

1,042 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at medRxiv: sign in and open the PDF from this page.

Who wrote this paper?

Suhana Bedi, Yutong Liu, Lucy Orr-Ewing, Dev Dash, Sanmi Koyejo, Alison Callahan, Jason Fries, Michael Wornow, Akshay Swaminathan, Lisa Soleymani Lehmann, Hyo Jung Hong, Mehr Kashyap and 7 more.

What is the DOI of this paper?

10.1001/jama.2024.21700, permanently at https://doi.org/10.1001/jama.2024.21700. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.