Review article · 2025 · Open access

Testing and evaluation of generative large language models in electronic health record applications: a systematic review

Xinsong Du, Zhengyang Zhou, Yifei Wang, Ya-Wen Chuang, Yiming Li, Richard C Yang, Wenyu Zhang, Xinyi Wang, Xinyu Chen, Hao Guan, John Lian, Pengyu Hong and 2 more

Journal of the American Medical Informatics Association, 33(3), 743–753 · Published 23 December 2025 · DOI 10.1093/jamia/ocaf233

Open PDFView at the publisherCite this paper

In brief

“Testing and evaluation of generative large language models in electronic health record applications: a systematic review” is a 2025 review article by Xinsong Du and 13 co-authors, published in Journal of the American Medical Informatics Association. OpenAlex counts 71 citations as of 17 November 2026, which puts it in the top 7% of works published in 2025 by citations. It is open access under a CC BY licence: the full text is free to read at PubMed Central.

Citations
71
Most cited in
2026 (54)
Field-weighted impact
2.79
References
198
Authors
14

Abstract

BACKGROUND: The use of generative large language models (LLMs) with electronic health record (EHR) data is rapidly expanding to support clinical and research tasks. This systematic review characterizes the clinical fields and use cases that have been studied and evaluated to date. METHODS: We followed the Preferred Reporting Items for Systematic Review and Meta-Analyses guidelines to conduct a systematic review of articles from PubMed and Web of Science published between January 1, 2023, and November 9, 2024. Studies were included if they used generative LLMs to analyze real-world EHR data and reported quantitative performance evaluations. Through data extraction, we identified clinical specialties and tasks for each included article, and summarized evaluation methods. RESULTS: Of the 18 735 articles retrieved, 196 met our criteria. Most studies focused on radiology (26.0%), oncology (10.7%), and emergency medicine (6.6%). Regarding clinical tasks, clinical decision support made up the largest proportion of studies (62.2%), while summarizations and patient communications made up the smallest, at 5.6% and 5.1%, respectively. In addition, GPT-4 and GPT-3.5 were the most commonly used generative LLMs, appearing in 60.2% and 57.7% of studies, respectively. Across these studies, we identified 22 unique non-NLP metrics and 35 unique NLP metrics. While NLP metrics offer greater scalability, none demonstrated a strong correlation with gold-standard human evaluations. CONCLUSION: Our findings highlight the need to evaluate generative LLMs on EHR data across a broader range of clinical specialties and tasks, as well as the urgent need for standardized, scalable, and clinically meaningful evaluation frameworks.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Du, X., Zhou, Z., Wang, Y., Chuang, Y.-W., Li, Y., Yang, R. C., Zhang, W., Wang, X., Chen, X., Guan, H., Lian, J., Hong, P., Bates, D. W., & Zhou, L. (2025). Testing and evaluation of generative large language models in electronic health record applications: a systematic review. Journal of the American Medical Informatics Association, 33(3), 743–753. https://doi.org/10.1093/jamia/ocaf233

MLA 9

Du, Xinsong, et al. “Testing and evaluation of generative large language models in electronic health record applications: a systematic review.” Journal of the American Medical Informatics Association, vol. 33, no. 3, 2025, pp. 743–753. https://doi.org/10.1093/jamia/ocaf233.

Chicago

Du, Xinsong, Zhengyang Zhou, Yifei Wang, Ya-Wen Chuang, Yiming Li, Richard C Yang, Wenyu Zhang, et al. 2025. “Testing and evaluation of generative large language models in electronic health record applications: a systematic review.” Journal of the American Medical Informatics Association 33 (3): 743–753. https://doi.org/10.1093/jamia/ocaf233.

Harvard

Du, X. et al. (2025) ‘Testing and evaluation of generative large language models in electronic health record applications: a systematic review’, Journal of the American Medical Informatics Association, 33(3), pp. 743–753. doi:10.1093/jamia/ocaf233.

IEEE

X. Du et al., “Testing and evaluation of generative large language models in electronic health record applications: a systematic review,” Journal of the American Medical Informatics Association, vol. 33, no. 3, pp. 743–753, 2025, doi: 10.1093/jamia/ocaf233.

BibTeX

@article{du2025testing,
  title     = {{Testing and evaluation of generative large language models in electronic health record applications: a systematic review}},
  author    = {Du, Xinsong and Zhou, Zhengyang and Wang, Yifei and Chuang, Ya-Wen and Li, Yiming and Yang, Richard C and Zhang, Wenyu and Wang, Xinyi and Chen, Xinyu and Guan, Hao and Lian, John and Hong, Pengyu and Bates, David Westfall and Zhou, Li},
  journal   = {Journal of the American Medical Informatics Association},
  year      = {2025},
  volume    = {33},
  number    = {3},
  pages     = {743--753},
  doi       = {10.1093/jamia/ocaf233},
  url       = {https://doi.org/10.1093/jamia/ocaf233}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Citations by year

YearCitationsShare of the best year
202422
20251515
20265454

Its field-weighted citation impact is 2.79: it has been cited 2.79 times as often as the average work of the same field, type and year (1.00 is the world average).

Topic and keywords

Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education

  • generative large language models
  • electronic health records
  • clinical decision support
  • LLM evaluation metrics
  • human evaluation
  • radiology
  • emergency medicine
  • GPT-4

Work with this paper

Most-cited papers in Medicine

All highly cited Medicine papers

Questions about this paper

How do I cite this paper in APA style?

Du, X., Zhou, Z., Wang, Y., Chuang, Y.-W., Li, Y., Yang, R. C., Zhang, W., Wang, X., Chen, X., Guan, H., Lian, J., Hong, P., Bates, D. W., & Zhou, L. (2025). Testing and evaluation of generative large language models in electronic health record applications: a systematic review. Journal of the American Medical Informatics Association, 33(3), 743–753. https://doi.org/10.1093/jamia/ocaf233

How many times has this paper been cited?

71 times, according to OpenAlex on 17 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at PubMed Central: sign in and open the PDF from this page.

Who wrote this paper?

Xinsong Du, Zhengyang Zhou, Yifei Wang, Ya-Wen Chuang, Yiming Li, Richard C Yang, Wenyu Zhang, Xinyi Wang, Xinyu Chen, Hao Guan, John Lian, Pengyu Hong and 2 more.

What is the DOI of this paper?

10.1093/jamia/ocaf233, permanently at https://doi.org/10.1093/jamia/ocaf233. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 17 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.