Journal article · 2025 · Open access
Clinical Large Language Model Evaluation by Expert Review (CLEVER): Framework Development and Validation
JMIR AI, 4, e72153 · Published 2 November 2025 · DOI 10.2196/72153
In brief
“Clinical Large Language Model Evaluation by Expert Review (CLEVER): Framework Development and Validation” is a 2025 journal article by Veysel Kocaman and 3 co-authors, published in JMIR AI. OpenAlex counts 55 citations as of 19 November 2026, which puts it in the top 1% of works published in 2025 by citations. It is open access under a CC BY licence: the full text is free to read at JMIR AI.
- Citations
- 55
- Most cited in
- 2026 (55)
- Field-weighted impact
- 8.06
- References
- 45
- Authors
- 4
Abstract
Background: The proliferation of both general purpose and health care-specific large language models (LLMs) has intensified the challenge of effectively evaluating and comparing them. Data contamination plagues the validity of public benchmarks, self-preference distorts LLM-as-a-judge approaches, and there is a gap between the tasks used to test models and those used in clinical practice. Objective: In response, we propose CLEVER (Clinical Large Language Model Evaluation-Expert Review), a methodology for blind, randomized, preference-based evaluation by practicing medical doctors on specific tasks. Methods: We demonstrate the methodology by comparing GPT-4o (OpenAI) against 2 health care-specific LLMs, with 8 billion and 70 billion parameters, over 3 tasks: clinical text summarization, clinical information extraction, and question answering on biomedical research. Results: Medical doctors prefer the medical model-small LLM trained by John Snow Labs over GPT-4o 45% to 92% more often on the dimensions of factuality, clinical relevance, and conciseness. Conclusions: The models show comparable performance on open-ended medical question answering, suggesting that health care-specific LLMs can outperform much larger general purpose LLMs in tasks that require understanding of clinical context. We test the validity of CLEVER evaluations by conducting interannotator agreement, interclass correlation, and washout period analysis.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Kocaman, V., Kaya, M. A., Feier, A. M., & Talby, D. (2025). Clinical Large Language Model Evaluation by Expert Review (CLEVER): Framework Development and Validation. JMIR AI, 4, e72153. https://doi.org/10.2196/72153
MLA 9
Kocaman, Veysel, et al. “Clinical Large Language Model Evaluation by Expert Review (CLEVER): Framework Development and Validation.” JMIR AI, vol. 4, 2025, p. e72153. https://doi.org/10.2196/72153.
Chicago
Kocaman, Veysel, Mustafa Aytuğ Kaya, Andrei Marian Feier, and David Talby. 2025. “Clinical Large Language Model Evaluation by Expert Review (CLEVER): Framework Development and Validation.” JMIR AI 4: e72153. https://doi.org/10.2196/72153.
Harvard
Kocaman, V. et al. (2025) ‘Clinical Large Language Model Evaluation by Expert Review (CLEVER): Framework Development and Validation’, JMIR AI, 4, p. e72153. doi:10.2196/72153.
IEEE
V. Kocaman, M. A. Kaya, A. M. Feier, and D. Talby, “Clinical Large Language Model Evaluation by Expert Review (CLEVER): Framework Development and Validation,” JMIR AI, vol. 4, p. e72153, 2025, doi: 10.2196/72153.
BibTeX
@article{kocaman2025clinical,
title = {{Clinical Large Language Model Evaluation by Expert Review (CLEVER): Framework Development and Validation}},
author = {Kocaman, Veysel and Kaya, Mustafa Aytuğ and Feier, Andrei Marian and Talby, David},
journal = {JMIR AI},
year = {2025},
volume = {4},
pages = {e72153},
doi = {10.2196/72153},
url = {https://doi.org/10.2196/72153}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- large language models
- expert review
- GPT-4o
- clinical text summarization
- clinical information extraction
- biomedical question answering
- inter-annotator agreement
- assessment validity
Work with this paper
Most-cited papers in Medicine
- Yaşlı Kadınlarda Üreme SağlığıDemet Kokanalı, Yaprak Engin Üstün · Jinekoloji-Obstetrik ve Neonatoloji Tıp Dergisi · 2019 · 169,630 citations
- Global Cancer Statistics 2020: GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 CountriesHyuna Sung, Adalberto M. Filho, Mathieu Laversanne and 5 more · CA A Cancer Journal for Clinicians · 2021 · 119,885 citations
- “Mini-mental state”Marshal F. Folstein, Susan E. Folstein, Paul R. McHugh · Journal of Psychiatric Research · 1975 · 90,424 citations
- Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countriesFreddie Bray, Jacques Ferlay, Isabelle Soerjomataram and 3 more · CA A Cancer Journal for Clinicians · 2018 · 87,396 citations
- Hallmarks of Cancer: The Next GenerationDouglas Hanahan, Robert A. Weinberg · Cell · 2011 · 69,758 citations
Questions about this paper
How do I cite this paper in APA style?
Kocaman, V., Kaya, M. A., Feier, A. M., & Talby, D. (2025). Clinical Large Language Model Evaluation by Expert Review (CLEVER): Framework Development and Validation. JMIR AI, 4, e72153. https://doi.org/10.2196/72153
How many times has this paper been cited?
55 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at JMIR AI: sign in and open the full text from this page.
Who wrote this paper?
Veysel Kocaman, Mustafa Aytuğ Kaya, Andrei Marian Feier, David Talby.
What is the DOI of this paper?
10.2196/72153, permanently at https://doi.org/10.2196/72153. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.