Journal article · 2025 · Open access

Clinical Large Language Model Evaluation by Expert Review (CLEVER): Framework Development and Validation

Veysel Kocaman, Mustafa Aytuğ Kaya, Andrei Marian Feier, David Talby

JMIR AI, 4, e72153 · Published 2 November 2025 · DOI 10.2196/72153

Open full textCite this paper

In brief

“Clinical Large Language Model Evaluation by Expert Review (CLEVER): Framework Development and Validation” is a 2025 journal article by Veysel Kocaman and 3 co-authors, published in JMIR AI. OpenAlex counts 55 citations as of 19 November 2026, which puts it in the top 1% of works published in 2025 by citations. It is open access under a CC BY licence: the full text is free to read at JMIR AI.

Citations
55
Most cited in
2026 (55)
Field-weighted impact
8.06
References
45
Authors
4

Abstract

Background: The proliferation of both general purpose and health care-specific large language models (LLMs) has intensified the challenge of effectively evaluating and comparing them. Data contamination plagues the validity of public benchmarks, self-preference distorts LLM-as-a-judge approaches, and there is a gap between the tasks used to test models and those used in clinical practice. Objective: In response, we propose CLEVER (Clinical Large Language Model Evaluation-Expert Review), a methodology for blind, randomized, preference-based evaluation by practicing medical doctors on specific tasks. Methods: We demonstrate the methodology by comparing GPT-4o (OpenAI) against 2 health care-specific LLMs, with 8 billion and 70 billion parameters, over 3 tasks: clinical text summarization, clinical information extraction, and question answering on biomedical research. Results: Medical doctors prefer the medical model-small LLM trained by John Snow Labs over GPT-4o 45% to 92% more often on the dimensions of factuality, clinical relevance, and conciseness. Conclusions: The models show comparable performance on open-ended medical question answering, suggesting that health care-specific LLMs can outperform much larger general purpose LLMs in tasks that require understanding of clinical context. We test the validity of CLEVER evaluations by conducting interannotator agreement, interclass correlation, and washout period analysis.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Kocaman, V., Kaya, M. A., Feier, A. M., & Talby, D. (2025). Clinical Large Language Model Evaluation by Expert Review (CLEVER): Framework Development and Validation. JMIR AI, 4, e72153. https://doi.org/10.2196/72153

MLA 9

Kocaman, Veysel, et al. “Clinical Large Language Model Evaluation by Expert Review (CLEVER): Framework Development and Validation.” JMIR AI, vol. 4, 2025, p. e72153. https://doi.org/10.2196/72153.

Chicago

Kocaman, Veysel, Mustafa Aytuğ Kaya, Andrei Marian Feier, and David Talby. 2025. “Clinical Large Language Model Evaluation by Expert Review (CLEVER): Framework Development and Validation.” JMIR AI 4: e72153. https://doi.org/10.2196/72153.

Harvard

Kocaman, V. et al. (2025) ‘Clinical Large Language Model Evaluation by Expert Review (CLEVER): Framework Development and Validation’, JMIR AI, 4, p. e72153. doi:10.2196/72153.

IEEE

V. Kocaman, M. A. Kaya, A. M. Feier, and D. Talby, “Clinical Large Language Model Evaluation by Expert Review (CLEVER): Framework Development and Validation,” JMIR AI, vol. 4, p. e72153, 2025, doi: 10.2196/72153.

BibTeX

@article{kocaman2025clinical,
  title     = {{Clinical Large Language Model Evaluation by Expert Review (CLEVER): Framework Development and Validation}},
  author    = {Kocaman, Veysel and Kaya, Mustafa Aytuğ and Feier, Andrei Marian and Talby, David},
  journal   = {JMIR AI},
  year      = {2025},
  volume    = {4},
  pages     = {e72153},
  doi       = {10.2196/72153},
  url       = {https://doi.org/10.2196/72153}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education

  • large language models
  • expert review
  • GPT-4o
  • clinical text summarization
  • clinical information extraction
  • biomedical question answering
  • inter-annotator agreement
  • assessment validity

Work with this paper

Most-cited papers in Medicine

All highly cited Medicine papers

Questions about this paper

How do I cite this paper in APA style?

Kocaman, V., Kaya, M. A., Feier, A. M., & Talby, D. (2025). Clinical Large Language Model Evaluation by Expert Review (CLEVER): Framework Development and Validation. JMIR AI, 4, e72153. https://doi.org/10.2196/72153

How many times has this paper been cited?

55 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at JMIR AI: sign in and open the full text from this page.

Who wrote this paper?

Veysel Kocaman, Mustafa Aytuğ Kaya, Andrei Marian Feier, David Talby.

What is the DOI of this paper?

10.2196/72153, permanently at https://doi.org/10.2196/72153. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.