Review article · 2025 · Open access
Testing and evaluation of generative large language models in electronic health record applications: a systematic review
Journal of the American Medical Informatics Association, 33(3), 743–753 · Published 23 December 2025 · DOI 10.1093/jamia/ocaf233
In brief
“Testing and evaluation of generative large language models in electronic health record applications: a systematic review” is a 2025 review article by Xinsong Du and 13 co-authors, published in Journal of the American Medical Informatics Association. OpenAlex counts 71 citations as of 17 November 2026, which puts it in the top 7% of works published in 2025 by citations. It is open access under a CC BY licence: the full text is free to read at PubMed Central.
- Citations
- 71
- Most cited in
- 2026 (54)
- Field-weighted impact
- 2.79
- References
- 198
- Authors
- 14
Abstract
BACKGROUND: The use of generative large language models (LLMs) with electronic health record (EHR) data is rapidly expanding to support clinical and research tasks. This systematic review characterizes the clinical fields and use cases that have been studied and evaluated to date. METHODS: We followed the Preferred Reporting Items for Systematic Review and Meta-Analyses guidelines to conduct a systematic review of articles from PubMed and Web of Science published between January 1, 2023, and November 9, 2024. Studies were included if they used generative LLMs to analyze real-world EHR data and reported quantitative performance evaluations. Through data extraction, we identified clinical specialties and tasks for each included article, and summarized evaluation methods. RESULTS: Of the 18 735 articles retrieved, 196 met our criteria. Most studies focused on radiology (26.0%), oncology (10.7%), and emergency medicine (6.6%). Regarding clinical tasks, clinical decision support made up the largest proportion of studies (62.2%), while summarizations and patient communications made up the smallest, at 5.6% and 5.1%, respectively. In addition, GPT-4 and GPT-3.5 were the most commonly used generative LLMs, appearing in 60.2% and 57.7% of studies, respectively. Across these studies, we identified 22 unique non-NLP metrics and 35 unique NLP metrics. While NLP metrics offer greater scalability, none demonstrated a strong correlation with gold-standard human evaluations. CONCLUSION: Our findings highlight the need to evaluate generative LLMs on EHR data across a broader range of clinical specialties and tasks, as well as the urgent need for standardized, scalable, and clinically meaningful evaluation frameworks.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Du, X., Zhou, Z., Wang, Y., Chuang, Y.-W., Li, Y., Yang, R. C., Zhang, W., Wang, X., Chen, X., Guan, H., Lian, J., Hong, P., Bates, D. W., & Zhou, L. (2025). Testing and evaluation of generative large language models in electronic health record applications: a systematic review. Journal of the American Medical Informatics Association, 33(3), 743–753. https://doi.org/10.1093/jamia/ocaf233
MLA 9
Du, Xinsong, et al. “Testing and evaluation of generative large language models in electronic health record applications: a systematic review.” Journal of the American Medical Informatics Association, vol. 33, no. 3, 2025, pp. 743–753. https://doi.org/10.1093/jamia/ocaf233.
Chicago
Du, Xinsong, Zhengyang Zhou, Yifei Wang, Ya-Wen Chuang, Yiming Li, Richard C Yang, Wenyu Zhang, et al. 2025. “Testing and evaluation of generative large language models in electronic health record applications: a systematic review.” Journal of the American Medical Informatics Association 33 (3): 743–753. https://doi.org/10.1093/jamia/ocaf233.
Harvard
Du, X. et al. (2025) ‘Testing and evaluation of generative large language models in electronic health record applications: a systematic review’, Journal of the American Medical Informatics Association, 33(3), pp. 743–753. doi:10.1093/jamia/ocaf233.
IEEE
X. Du et al., “Testing and evaluation of generative large language models in electronic health record applications: a systematic review,” Journal of the American Medical Informatics Association, vol. 33, no. 3, pp. 743–753, 2025, doi: 10.1093/jamia/ocaf233.
BibTeX
@article{du2025testing,
title = {{Testing and evaluation of generative large language models in electronic health record applications: a systematic review}},
author = {Du, Xinsong and Zhou, Zhengyang and Wang, Yifei and Chuang, Ya-Wen and Li, Yiming and Yang, Richard C and Zhang, Wenyu and Wang, Xinyi and Chen, Xinyu and Guan, Hao and Lian, John and Hong, Pengyu and Bates, David Westfall and Zhou, Li},
journal = {Journal of the American Medical Informatics Association},
year = {2025},
volume = {33},
number = {3},
pages = {743--753},
doi = {10.1093/jamia/ocaf233},
url = {https://doi.org/10.1093/jamia/ocaf233}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Citations by year
| Year | Citations | Share of the best year |
|---|---|---|
| 2024 | 2 | |
| 2025 | 15 | |
| 2026 | 54 |
Its field-weighted citation impact is 2.79: it has been cited 2.79 times as often as the average work of the same field, type and year (1.00 is the world average).
Topic and keywords
Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- generative large language models
- electronic health records
- clinical decision support
- LLM evaluation metrics
- human evaluation
- radiology
- emergency medicine
- GPT-4
Work with this paper
Most-cited papers in Medicine
- Yaşlı Kadınlarda Üreme SağlığıDemet Kokanalı, Yaprak Engin Üstün · Jinekoloji-Obstetrik ve Neonatoloji Tıp Dergisi · 2019 · 169,630 citations
- Global Cancer Statistics 2020: GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 CountriesHyuna Sung, Adalberto M. Filho, Mathieu Laversanne and 5 more · CA A Cancer Journal for Clinicians · 2021 · 119,885 citations
- “Mini-mental state”Marshal F. Folstein, Susan E. Folstein, Paul R. McHugh · Journal of Psychiatric Research · 1975 · 90,424 citations
- Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countriesFreddie Bray, Jacques Ferlay, Isabelle Soerjomataram and 3 more · CA A Cancer Journal for Clinicians · 2018 · 87,396 citations
- Hallmarks of Cancer: The Next GenerationDouglas Hanahan, Robert A. Weinberg · Cell · 2011 · 69,758 citations
Questions about this paper
How do I cite this paper in APA style?
Du, X., Zhou, Z., Wang, Y., Chuang, Y.-W., Li, Y., Yang, R. C., Zhang, W., Wang, X., Chen, X., Guan, H., Lian, J., Hong, P., Bates, D. W., & Zhou, L. (2025). Testing and evaluation of generative large language models in electronic health record applications: a systematic review. Journal of the American Medical Informatics Association, 33(3), 743–753. https://doi.org/10.1093/jamia/ocaf233
How many times has this paper been cited?
71 times, according to OpenAlex on 17 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at PubMed Central: sign in and open the PDF from this page.
Who wrote this paper?
Xinsong Du, Zhengyang Zhou, Yifei Wang, Ya-Wen Chuang, Yiming Li, Richard C Yang, Wenyu Zhang, Xinyi Wang, Xinyu Chen, Hao Guan, John Lian, Pengyu Hong and 2 more.
What is the DOI of this paper?
10.1093/jamia/ocaf233, permanently at https://doi.org/10.1093/jamia/ocaf233. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 17 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.