Conference paper · 2024 · Open access
MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Models
neural information processing systems, 33423–33454 · Published 1 January 2024 · DOI 10.52202/079017-1054
In brief
“MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Models” is a 2024 conference paper by Tessa Han and 3 co-authors, published in neural information processing systems. OpenAlex counts 158 citations as of 22 November 2026, which puts it in the top 2% of works published in 2024 by citations. It is open access under a CC BY-NC-SA licence: the full text is free to read at arXiv (Cornell University).
- Citations
- 158
- Most cited in
- 2026 (97)
- Field-weighted impact
- 19.55
- References
- 27
- Authors
- 4
Abstract
As large language models (LLMs) develop increasingly sophisticated capabilities and find applications in medical settings, it becomes important to assess their medical safety due to their far-reaching implications for personal and public health, patient safety, and human rights. However, there is little to no understanding of the notion of medical safety in the context of LLMs, let alone how to evaluate and improve it. To address this gap, we first define the notion of medical safety in LLMs based on the Principles of Medical Ethics set forth by the American Medical Association. We then leverage this understanding to introduce MedSafetyBench, the first benchmark dataset designed to measure the medical safety of LLMs. We demonstrate the utility of MedSafetyBench by using it to evaluate and improve the medical safety of LLMs. Our results show that publicly-available medical LLMs do not meet standards of medical safety and that fine-tuning them using MedSafetyBench improves their medical safety while preserving their medical performance. By introducing this new benchmark dataset, our work enables a systematic study of the state of medical safety in LLMs and motivates future work in this area, paving the way to mitigate the safety risks of LLMs in medicine. The benchmark dataset and code are available at https://github.com/AI4LIFE-GROUP/med-safety-bench.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Han, T., Kumar, A., Agarwal, C., & Lakkaraju, H. (2024). MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Models. neural information processing systems, 33423–33454. https://doi.org/10.52202/079017-1054
MLA 9
Han, Tessa, et al. “MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Models.” neural information processing systems, 2024, pp. 33423–33454. https://doi.org/10.52202/079017-1054.
Chicago
Han, Tessa, Aounon Kumar, Chirag Agarwal, and Himabindu Lakkaraju. 2024. “MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Models.” neural information processing systems: 33423–33454. https://doi.org/10.52202/079017-1054.
Harvard
Han, T. et al. (2024) ‘MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Models’, neural information processing systems, pp. 33423–33454. doi:10.52202/079017-1054.
IEEE
T. Han, A. Kumar, C. Agarwal, and H. Lakkaraju, “MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Models,” neural information processing systems, pp. 33423–33454, 2024, doi: 10.52202/079017-1054.
BibTeX
@inproceedings{han2024medsafetybench,
title = {{MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Models}},
author = {Han, Tessa and Kumar, Aounon and Agarwal, Chirag and Lakkaraju, Himabindu},
booktitle = {neural information processing systems},
year = {2024},
pages = {33423--33454},
doi = {10.52202/079017-1054},
url = {https://doi.org/10.52202/079017-1054}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Citations by year
| Year | Citations | Share of the best year |
|---|---|---|
| 2024 | 9 | |
| 2025 | 52 | |
| 2026 | 97 |
Its field-weighted citation impact is 19.55: it has been cited 19.55 times as often as the average work of the same field, type and year (1.00 is the world average).
Topic and keywords
Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- medical safety
- large language models
- benchmark dataset
- medical ethics
- fine-tuning
- patient safety
- safety assessment
- medical LLMs
Work with this paper
Most-cited papers in Medicine
- Yaşlı Kadınlarda Üreme SağlığıDemet Kokanalı, Yaprak Engin Üstün · Jinekoloji-Obstetrik ve Neonatoloji Tıp Dergisi · 2019 · 169,630 citations
- Global Cancer Statistics 2020: GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 CountriesHyuna Sung, Adalberto M. Filho, Mathieu Laversanne and 5 more · CA A Cancer Journal for Clinicians · 2021 · 119,885 citations
- “Mini-mental state”Marshal F. Folstein, Susan E. Folstein, Paul R. McHugh · Journal of Psychiatric Research · 1975 · 90,424 citations
- Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countriesFreddie Bray, Jacques Ferlay, Isabelle Soerjomataram and 3 more · CA A Cancer Journal for Clinicians · 2018 · 87,396 citations
- Hallmarks of Cancer: The Next GenerationDouglas Hanahan, Robert A. Weinberg · Cell · 2011 · 69,758 citations
Questions about this paper
How do I cite this paper in APA style?
Han, T., Kumar, A., Agarwal, C., & Lakkaraju, H. (2024). MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Models. neural information processing systems, 33423–33454. https://doi.org/10.52202/079017-1054
How many times has this paper been cited?
158 times, according to OpenAlex on 22 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at arXiv (Cornell University): sign in and open the PDF from this page.
Who wrote this paper?
Tessa Han, Aounon Kumar, Chirag Agarwal, Himabindu Lakkaraju.
What is the DOI of this paper?
10.52202/079017-1054, permanently at https://doi.org/10.52202/079017-1054. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 22 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.