Journal article · 2025 · Open access

MedAgentBench: A Virtual EHR Environment to Benchmark Medical LLM Agents

Yixing Jiang, Kameron Collin Black, Gloria Geng, Danny Park, James Zou, Andrew Y. Ng, Jonathan H. Chen

NEJM AI, 2(9), AIdbp2500144 · Published 14 August 2025 · DOI 10.1056/aidbp2500144

Open PDFView at the publisherCite this paper

In brief

“MedAgentBench: A Virtual EHR Environment to Benchmark Medical LLM Agents” is a 2025 journal article by Yixing Jiang and 6 co-authors, published in NEJM AI. OpenAlex counts 318 citations as of 22 November 2026, which puts it in the top 1% of works published in 2025 by citations. It is open access: the full text is free to read at NEJM AI.

Citations
318
Most cited in
2026 (285)
Field-weighted impact
45.59
References
30
Authors
7

Abstract

BACKGROUND: Recent large language models (LLMs) have demonstrated significant advancements, particularly in their ability to serve as agents, thereby surpassing their traditional role as chatbots. These agents can leverage their planning and tool utilization capabilities to address tasks specified at a high level. This suggests new potential to reduce the burden of administrative tasks and address current health care staff shortages. However, a standardized dataset to benchmark the agent capabilities of LLMs in medical applications is currently lacking, making it difficult to evaluate their performance on complex tasks in interactive health care environments. METHODS: To address this gap in the deployment of agentic artificial intelligence (AI) in health care, we introduce MedAgentBench, a broad evaluation suite designed to assess the agent capabilities of LLMs within medical records contexts. MedAgentBench encompasses 300 patient-specific clinically derived tasks from 10 categories written by human physicians, realistic profiles of 100 patients with over 700,000 data elements, a Fast Healthcare Interoperability Resources-compliant interactive environment, and an accompanying codebase. The environment uses standard application programming interfaces and communication infrastructure used in modern electronic health record (EHR) systems so that it can be easily migrated into live EHR systems. RESULTS: MedAgentBench presents an unsaturated agent-oriented benchmark at which current state-of-the-art LLMs exhibit some ability to succeed. The best model (Claude 3.5 Sonnet v2) achieves a success rate of 69.67%. However, there is still substantial room for improvement, which gives the community a clear direction for future optimization efforts. Furthermore, there is significant variation in performance across task categories. CONCLUSIONS: Agent-based task frameworks and benchmarks are the necessary next step to advance the potential and capabilities for effectively improving and integrating AI systems into clinical workflows. MedAgentBench establishes this and is publicly available at https://github.com/stanfordmlgroup/MedAgentBench, offering a valuable framework for model developers to track progress and drive continuous improvements in the agent capabilities of LLMs within the medical domain. (Funded by the NIH and Singapore's National Science Scholarship [PhD].).

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Jiang, Y., Black, K. C., Geng, G., Park, D., Zou, J., Ng, A. Y., & Chen, J. H. (2025). MedAgentBench: A Virtual EHR Environment to Benchmark Medical LLM Agents. NEJM AI, 2(9), AIdbp2500144. https://doi.org/10.1056/aidbp2500144

MLA 9

Jiang, Yixing, et al. “MedAgentBench: A Virtual EHR Environment to Benchmark Medical LLM Agents.” NEJM AI, vol. 2, no. 9, 2025, p. AIdbp2500144. https://doi.org/10.1056/aidbp2500144.

Chicago

Jiang, Yixing, Kameron Collin Black, Gloria Geng, Danny Park, James Zou, Andrew Y. Ng, and Jonathan H. Chen. 2025. “MedAgentBench: A Virtual EHR Environment to Benchmark Medical LLM Agents.” NEJM AI 2 (9): AIdbp2500144. https://doi.org/10.1056/aidbp2500144.

Harvard

Jiang, Y. et al. (2025) ‘MedAgentBench: A Virtual EHR Environment to Benchmark Medical LLM Agents’, NEJM AI, 2(9), p. AIdbp2500144. doi:10.1056/aidbp2500144.

IEEE

Y. Jiang et al., “MedAgentBench: A Virtual EHR Environment to Benchmark Medical LLM Agents,” NEJM AI, vol. 2, no. 9, p. AIdbp2500144, 2025, doi: 10.1056/aidbp2500144.

BibTeX

@article{jiang2025medagentbench,
  title     = {{MedAgentBench: A Virtual EHR Environment to Benchmark Medical LLM Agents}},
  author    = {Jiang, Yixing and Black, Kameron Collin and Geng, Gloria and Park, Danny and Zou, James and Ng, Andrew Y. and Chen, Jonathan H.},
  journal   = {NEJM AI},
  year      = {2025},
  volume    = {2},
  number    = {9},
  pages     = {AIdbp2500144},
  doi       = {10.1056/aidbp2500144},
  url       = {https://doi.org/10.1056/aidbp2500144}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education

  • LLM agents
  • benchmarking
  • electronic health records
  • agentic AI
  • clinical tasks
  • FHIR
  • clinical workflow
  • administrative burden

Work with this paper

Most-cited papers in Medicine

All highly cited Medicine papers

Questions about this paper

How do I cite this paper in APA style?

Jiang, Y., Black, K. C., Geng, G., Park, D., Zou, J., Ng, A. Y., & Chen, J. H. (2025). MedAgentBench: A Virtual EHR Environment to Benchmark Medical LLM Agents. NEJM AI, 2(9), AIdbp2500144. https://doi.org/10.1056/aidbp2500144

How many times has this paper been cited?

318 times, according to OpenAlex on 22 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at NEJM AI: sign in and open the PDF from this page.

Who wrote this paper?

Yixing Jiang, Kameron Collin Black, Gloria Geng, Danny Park, James Zou, Andrew Y. Ng, Jonathan H. Chen.

What is the DOI of this paper?

10.1056/aidbp2500144, permanently at https://doi.org/10.1056/aidbp2500144. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 22 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.