Conference paper · 2024 · Open access

MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Models

Tessa Han, Aounon Kumar, Chirag Agarwal, Himabindu Lakkaraju

neural information processing systems, 33423–33454 · Published 1 January 2024 · DOI 10.52202/079017-1054

Open PDFView at the publisherCite this paper

In brief

“MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Models” is a 2024 conference paper by Tessa Han and 3 co-authors, published in neural information processing systems. OpenAlex counts 158 citations as of 22 November 2026, which puts it in the top 2% of works published in 2024 by citations. It is open access under a CC BY-NC-SA licence: the full text is free to read at arXiv (Cornell University).

Citations
158
Most cited in
2026 (97)
Field-weighted impact
19.55
References
27
Authors
4

Abstract

As large language models (LLMs) develop increasingly sophisticated capabilities and find applications in medical settings, it becomes important to assess their medical safety due to their far-reaching implications for personal and public health, patient safety, and human rights. However, there is little to no understanding of the notion of medical safety in the context of LLMs, let alone how to evaluate and improve it. To address this gap, we first define the notion of medical safety in LLMs based on the Principles of Medical Ethics set forth by the American Medical Association. We then leverage this understanding to introduce MedSafetyBench, the first benchmark dataset designed to measure the medical safety of LLMs. We demonstrate the utility of MedSafetyBench by using it to evaluate and improve the medical safety of LLMs. Our results show that publicly-available medical LLMs do not meet standards of medical safety and that fine-tuning them using MedSafetyBench improves their medical safety while preserving their medical performance. By introducing this new benchmark dataset, our work enables a systematic study of the state of medical safety in LLMs and motivates future work in this area, paving the way to mitigate the safety risks of LLMs in medicine. The benchmark dataset and code are available at https://github.com/AI4LIFE-GROUP/med-safety-bench.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Han, T., Kumar, A., Agarwal, C., & Lakkaraju, H. (2024). MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Models. neural information processing systems, 33423–33454. https://doi.org/10.52202/079017-1054

MLA 9

Han, Tessa, et al. “MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Models.” neural information processing systems, 2024, pp. 33423–33454. https://doi.org/10.52202/079017-1054.

Chicago

Han, Tessa, Aounon Kumar, Chirag Agarwal, and Himabindu Lakkaraju. 2024. “MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Models.” neural information processing systems: 33423–33454. https://doi.org/10.52202/079017-1054.

Harvard

Han, T. et al. (2024) ‘MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Models’, neural information processing systems, pp. 33423–33454. doi:10.52202/079017-1054.

IEEE

T. Han, A. Kumar, C. Agarwal, and H. Lakkaraju, “MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Models,” neural information processing systems, pp. 33423–33454, 2024, doi: 10.52202/079017-1054.

BibTeX

@inproceedings{han2024medsafetybench,
  title     = {{MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Models}},
  author    = {Han, Tessa and Kumar, Aounon and Agarwal, Chirag and Lakkaraju, Himabindu},
  booktitle = {neural information processing systems},
  year      = {2024},
  pages     = {33423--33454},
  doi       = {10.52202/079017-1054},
  url       = {https://doi.org/10.52202/079017-1054}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Citations by year

YearCitationsShare of the best year
202499
20255252
20269797

Its field-weighted citation impact is 19.55: it has been cited 19.55 times as often as the average work of the same field, type and year (1.00 is the world average).

Topic and keywords

Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education

  • medical safety
  • large language models
  • benchmark dataset
  • medical ethics
  • fine-tuning
  • patient safety
  • safety assessment
  • medical LLMs

Work with this paper

Most-cited papers in Medicine

All highly cited Medicine papers

Questions about this paper

How do I cite this paper in APA style?

Han, T., Kumar, A., Agarwal, C., & Lakkaraju, H. (2024). MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Models. neural information processing systems, 33423–33454. https://doi.org/10.52202/079017-1054

How many times has this paper been cited?

158 times, according to OpenAlex on 22 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at arXiv (Cornell University): sign in and open the PDF from this page.

Who wrote this paper?

Tessa Han, Aounon Kumar, Chirag Agarwal, Himabindu Lakkaraju.

What is the DOI of this paper?

10.52202/079017-1054, permanently at https://doi.org/10.52202/079017-1054. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 22 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.