Journal article · 2024 · Open access
A framework for human evaluation of large language models in healthcare derived from literature review
npj Digital Medicine, 7(1), 258 · Published 27 September 2024 · DOI 10.1038/s41746-024-01258-7
In brief
“A framework for human evaluation of large language models in healthcare derived from literature review” is a 2024 journal article by Thomas Yu Chow Tam and 14 co-authors, published in npj Digital Medicine. OpenAlex counts 615 citations as of 19 November 2026, which puts it in the top 3% of works published in 2024 by citations. It is open access under a CC BY licence: the full text is free to read at npj Digital Medicine.
- Citations
- 615
- Most cited in
- 2026 (424)
- Field-weighted impact
- 40.45
- References
- 107
- Authors
- 15
Abstract
With generative artificial intelligence (GenAI), particularly large language models (LLMs), continuing to make inroads in healthcare, assessing LLMs with human evaluations is essential to assuring safety and effectiveness. This study reviews existing literature on human evaluation methodologies for LLMs in healthcare across various medical specialties and addresses factors such as evaluation dimensions, sample types and sizes, selection, and recruitment of evaluators, frameworks and metrics, evaluation process, and statistical analysis type. Our literature review of 142 studies shows gaps in reliability, generalizability, and applicability of current human evaluation practices. To overcome such significant obstacles to healthcare LLM developments and deployments, we propose QUEST, a comprehensive and practical framework for human evaluation of LLMs covering three phases of workflow: Planning, Implementation and Adjudication, and Scoring and Review. QUEST is designed with five proposed evaluation principles: Quality of Information, Understanding and Reasoning, Expression Style and Persona, Safety and Harm, and Trust and Confidence.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Tam, T. Y. C., Sivarajkumar, S., Kapoor, S., Stolyar, A. V., Polanska, K., McCarthy, K. R., Osterhoudt, H., Wu, X., Visweswaran, S., Fu, S., Mathur, P., Cacciamani, G. E., Sun, C., Peng, Y., & Wang, Y. (2024). A framework for human evaluation of large language models in healthcare derived from literature review. npj Digital Medicine, 7(1), 258. https://doi.org/10.1038/s41746-024-01258-7
MLA 9
Tam, Thomas Yu Chow, et al. “A framework for human evaluation of large language models in healthcare derived from literature review.” npj Digital Medicine, vol. 7, no. 1, 2024, p. 258. https://doi.org/10.1038/s41746-024-01258-7.
Chicago
Tam, Thomas Yu Chow, Sonish Sivarajkumar, Sumit Kapoor, Alisa V Stolyar, Katelyn Polanska, Karleigh R McCarthy, Hunter Osterhoudt, et al. 2024. “A framework for human evaluation of large language models in healthcare derived from literature review.” npj Digital Medicine 7 (1): 258. https://doi.org/10.1038/s41746-024-01258-7.
Harvard
Tam, T. Y. C. et al. (2024) ‘A framework for human evaluation of large language models in healthcare derived from literature review’, npj Digital Medicine, 7(1), p. 258. doi:10.1038/s41746-024-01258-7.
IEEE
T. Y. C. Tam et al., “A framework for human evaluation of large language models in healthcare derived from literature review,” npj Digital Medicine, vol. 7, no. 1, p. 258, 2024, doi: 10.1038/s41746-024-01258-7.
BibTeX
@article{tam2024framework,
title = {{A framework for human evaluation of large language models in healthcare derived from literature review}},
author = {Tam, Thomas Yu Chow and Sivarajkumar, Sonish and Kapoor, Sumit and Stolyar, Alisa V and Polanska, Katelyn and McCarthy, Karleigh R and Osterhoudt, Hunter and Wu, Xizhi and Visweswaran, Shyam and Fu, Sunyang and Mathur, Piyush and Cacciamani, Giovanni Enrico and Sun, Cong and Peng, Yifan and Wang, Yanshan},
journal = {npj Digital Medicine},
year = {2024},
volume = {7},
number = {1},
pages = {258},
doi = {10.1038/s41746-024-01258-7},
url = {https://doi.org/10.1038/s41746-024-01258-7}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Citations by year
| Year | Citations | Share of the best year |
|---|---|---|
| 2021 | 1 | |
| 2024 | 6 | |
| 2025 | 184 | |
| 2026 | 424 |
Its field-weighted citation impact is 40.45: it has been cited 40.45 times as often as the average work of the same field, type and year (1.00 is the world average).
Topic and keywords
Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- large language models
- human evaluation
- health care
- generative AI
- evaluation framework
- evaluation metrics
- assessment reliability
- trust and confidence
Work with this paper
Most-cited papers in Medicine
- Yaşlı Kadınlarda Üreme SağlığıDemet Kokanalı, Yaprak Engin Üstün · Jinekoloji-Obstetrik ve Neonatoloji Tıp Dergisi · 2019 · 169,630 citations
- Global Cancer Statistics 2020: GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 CountriesHyuna Sung, Adalberto M. Filho, Mathieu Laversanne and 5 more · CA A Cancer Journal for Clinicians · 2021 · 119,885 citations
- “Mini-mental state”Marshal F. Folstein, Susan E. Folstein, Paul R. McHugh · Journal of Psychiatric Research · 1975 · 90,424 citations
- Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countriesFreddie Bray, Jacques Ferlay, Isabelle Soerjomataram and 3 more · CA A Cancer Journal for Clinicians · 2018 · 87,396 citations
- Hallmarks of Cancer: The Next GenerationDouglas Hanahan, Robert A. Weinberg · Cell · 2011 · 69,758 citations
Questions about this paper
How do I cite this paper in APA style?
Tam, T. Y. C., Sivarajkumar, S., Kapoor, S., Stolyar, A. V., Polanska, K., McCarthy, K. R., Osterhoudt, H., Wu, X., Visweswaran, S., Fu, S., Mathur, P., Cacciamani, G. E., Sun, C., Peng, Y., & Wang, Y. (2024). A framework for human evaluation of large language models in healthcare derived from literature review. npj Digital Medicine, 7(1), 258. https://doi.org/10.1038/s41746-024-01258-7
How many times has this paper been cited?
615 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at npj Digital Medicine: sign in and open the PDF from this page.
Who wrote this paper?
Thomas Yu Chow Tam, Sonish Sivarajkumar, Sumit Kapoor, Alisa V Stolyar, Katelyn Polanska, Karleigh R McCarthy, Hunter Osterhoudt, Xizhi Wu, Shyam Visweswaran, Sunyang Fu, Piyush Mathur, Giovanni Enrico Cacciamani and 3 more.
What is the DOI of this paper?
10.1038/s41746-024-01258-7, permanently at https://doi.org/10.1038/s41746-024-01258-7. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.