Review article · 2025 · Open access

Knowledge-Practice Performance Gap in Clinical Large Language Models: Systematic Review of 39 Benchmarks

Eun Jeong Gong, Chang Seok Bang, Jae Jun Lee, Gwang Ho Baik

Journal of Medical Internet Research, 27, e84120 · Published 1 December 2025 · DOI 10.2196/84120

Open PDFView at the publisherCite this paper

In brief

“Knowledge-Practice Performance Gap in Clinical Large Language Models: Systematic Review of 39 Benchmarks” is a 2025 review article by Eun Jeong Gong and 3 co-authors, published in Journal of Medical Internet Research. OpenAlex counts 90 citations as of 17 November 2026. It is open access under a CC BY licence: the full text is free to read at Journal of Medical Internet Research.

Citations
90
Most cited in
2026 (87)
Field-weighted impact
3.08
References
57
Authors
4

Abstract

BACKGROUND: The evaluation of large language models (LLMs) in medicine has undergone a shift from knowledge-based testing to practice-based assessment, representing an evolution in how we measure artificial intelligence readiness for clinical deployment. While LLMs now routinely exceed human performance on medical licensing examinations, their translation to clinical practice remains poorly characterized. OBJECTIVE: This systematic review aims to categorize and analyze medical LLM benchmarks, examining performance patterns across different evaluation paradigms and identifying gaps in current assessment methodologies. METHODS: The protocol was registered at PROSPERO (CRD420251139729). Four databases (MEDLINE/PubMed, Embase/Ovid, Cochrane Library, and arXiv) were searched from inception to August 31, 2025, using keywords related to clinical medicine benchmarks in LLMs. Studies were included if they (1) investigated clinical medicine benchmarks in LLMs, (2) were published in English, and (3) were available in full-text. Studies were excluded if they evaluated nonmedical domains or lacked benchmark validation. Methodological quality was assessed using the Mixed Methods Appraisal Tool (version 2018) by 2 independent reviewers (κ=0.91). Due to heterogeneity in evaluation metrics preventing meta-analysis, narrative synthesis was conducted using structured categorization of benchmark types. RESULTS: From 3917 screened records, 39 medical LLM benchmarks were identified and categorized into 21 (54%) knowledge-based, 15 (38%) practice-based, and 3 (8%) hybrid frameworks. These benchmarks collectively encompass over 2.3 million questions across 45 languages and 172 medical specialties. Traditional knowledge-based benchmarks show saturation with leading models achieving 84%-90% accuracy on USMLE (United States Medical Licensing Examination)-style examinations, approaching or exceeding average physician performance. However, practice-based assessments reveal performance challenges, with specific benchmarks showing varied results: DiagnosisArena 45.82% (95% CI 42.9%-48.8%), MedAgentBench 69.67% (95% CI 64.2%-74.6%), and HealthBench 60% (95% CI 58.6%-61.3%) success rates, with practice-based benchmarks showing lower performance (45%-69%) compared to knowledge benchmarks (84%-90%). Task-specific analysis revealed differential performance patterns: factual retrieval maintained 85%-93% accuracy, clinical reasoning dropped to 50%-60%, diagnostic tasks achieved 45%-55% success, and safety assessment showed significant gaps at 40%-50% accuracy despite being life-critical. Geographic representation spans 6 continents with 18 (46%) benchmarks, incorporating non-English content. Quality assessment revealed 26% (10/39) of benchmarks had insufficient methodological reporting for complete evaluation. CONCLUSIONS: This systematic review provides the first comprehensive analysis quantifying the significant "knowledge-practice gap" in medical artificial intelligence: high performance on knowledge-based examinations (84%-90%) does not translate to clinical competence (45%-69%), with safety assessments at 40%-50%. Our findings provide quantitative evidence for regulators and health systems that examination scores are insufficient and misleading proxies for clinical readiness. This review concludes that autonomous deployment is not currently justifiable and that all evidence-based implementation strategies…

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Gong, E. J., Bang, C. S., Lee, J. J., & Baik, G. H. (2025). Knowledge-Practice Performance Gap in Clinical Large Language Models: Systematic Review of 39 Benchmarks. Journal of Medical Internet Research, 27, e84120. https://doi.org/10.2196/84120

MLA 9

Gong, Eun Jeong, et al. “Knowledge-Practice Performance Gap in Clinical Large Language Models: Systematic Review of 39 Benchmarks.” Journal of Medical Internet Research, vol. 27, 2025, p. e84120. https://doi.org/10.2196/84120.

Chicago

Gong, Eun Jeong, Chang Seok Bang, Jae Jun Lee, and Gwang Ho Baik. 2025. “Knowledge-Practice Performance Gap in Clinical Large Language Models: Systematic Review of 39 Benchmarks.” Journal of Medical Internet Research 27: e84120. https://doi.org/10.2196/84120.

Harvard

Gong, E. J. et al. (2025) ‘Knowledge-Practice Performance Gap in Clinical Large Language Models: Systematic Review of 39 Benchmarks’, Journal of Medical Internet Research, 27, p. e84120. doi:10.2196/84120.

IEEE

E. J. Gong, C. S. Bang, J. J. Lee, and G. H. Baik, “Knowledge-Practice Performance Gap in Clinical Large Language Models: Systematic Review of 39 Benchmarks,” Journal of Medical Internet Research, vol. 27, p. e84120, 2025, doi: 10.2196/84120.

BibTeX

@article{gong2025knowledge,
  title     = {{Knowledge-Practice Performance Gap in Clinical Large Language Models: Systematic Review of 39 Benchmarks}},
  author    = {Gong, Eun Jeong and Bang, Chang Seok and Lee, Jae Jun and Baik, Gwang Ho},
  journal   = {Journal of Medical Internet Research},
  year      = {2025},
  volume    = {27},
  pages     = {e84120},
  doi       = {10.2196/84120},
  url       = {https://doi.org/10.2196/84120}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education

  • clinical large language models
  • LLM benchmark
  • knowledge-practice gap
  • practice-based assessment
  • knowledge-based assessment
  • clinical reasoning
  • diagnosis
  • clinical decision support

Work with this paper

Most-cited papers in Medicine

All highly cited Medicine papers

Questions about this paper

How do I cite this paper in APA style?

Gong, E. J., Bang, C. S., Lee, J. J., & Baik, G. H. (2025). Knowledge-Practice Performance Gap in Clinical Large Language Models: Systematic Review of 39 Benchmarks. Journal of Medical Internet Research, 27, e84120. https://doi.org/10.2196/84120

How many times has this paper been cited?

90 times, according to OpenAlex on 17 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at Journal of Medical Internet Research: sign in and open the PDF from this page.

Who wrote this paper?

Eun Jeong Gong, Chang Seok Bang, Jae Jun Lee, Gwang Ho Baik.

What is the DOI of this paper?

10.2196/84120, permanently at https://doi.org/10.2196/84120. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 17 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.