Preprint · 2026 · Open access
CareGuardAI: Context-Aware Multi-Agent Guardrails for Clinical Safety & Hallucination Mitigation in Patient-Facing LLMs
arXiv (Cornell University) · Published 7 April 2026 · DOI 10.48550/arxiv.2604.26959
In brief
“CareGuardAI: Context-Aware Multi-Agent Guardrails for Clinical Safety & Hallucination Mitigation in Patient-Facing LLMs” is a 2026 preprint by Elham Nasarian and 3 co-authors, published in arXiv (Cornell University). OpenAlex counts 3 citations as of 19 November 2026. It is open access under a CC BY licence: the full text is free to read at arXiv (Cornell University).
- Citations
- 3
- Most cited in
- 2026 (3)
- References
- 25
- Authors
- 4
Abstract
Integrating large language models (LLMs) into patient-facing healthcare systems offers significant potential to improve access to medical information. However, ensuring clinical safety and factual reliability remains a critical challenge. In practice, AI-generated responses may be conditionally correct yet medically inappropriate, as models often fail to interpret patient context and tend to produce agreeable responses rather than challenge unsafe assumptions. Unlike clinicians, who infer risk from incomplete information, LLMs frequently lack contextual awareness. Moreover, real-world patient interactions are open-ended and underspecified, unlike structured benchmark settings. We present CareGuardAI, a risk-aware safety framework for patient-facing medical question answering that addresses two key failure modes: clinical safety risk and hallucination risk. The framework introduces Clinical Safety Risk Assessment (SRA), inspired by ISO 14971, and Hallucination Risk Assessment (HRA) to evaluate medical risk and factual reliability. At inference time, CareGuardAI employs a multi-stage pipeline consisting of a controller agent, safety-constrained generation, and dual risk evaluation, followed by iterative refinement when necessary. Responses are released only when both SRA and HRA are less than or equal to 2, ensuring clinically acceptable outputs with bounded latency. We evaluate CareGuardAI on PatientSafeBench, MedSafetyBench, and MedHallu, covering both safety and hallucination detection. Across these benchmarks, the framework consistently outperforms strong baseline models, including GPT-4o-mini, demonstrating the importance of context-aware, risk-based, inference-time safety mechanisms for reliable deployment in healthcare.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Nasarian, E., Neog, A., Tsui, K., & Hosseinichimeh, N. (2026). CareGuardAI: Context-Aware Multi-Agent Guardrails for Clinical Safety & Hallucination Mitigation in Patient-Facing LLMs. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2604.26959
MLA 9
Nasarian, Elham, et al. “CareGuardAI: Context-Aware Multi-Agent Guardrails for Clinical Safety & Hallucination Mitigation in Patient-Facing LLMs.” arXiv (Cornell University), 2026. https://doi.org/10.48550/arxiv.2604.26959.
Chicago
Nasarian, Elham, Abhilash Neog, Kwok‐Leung Tsui, and Niyousha Hosseinichimeh. 2026. “CareGuardAI: Context-Aware Multi-Agent Guardrails for Clinical Safety & Hallucination Mitigation in Patient-Facing LLMs.” arXiv (Cornell University). https://doi.org/10.48550/arxiv.2604.26959.
Harvard
Nasarian, E. et al. (2026) ‘CareGuardAI: Context-Aware Multi-Agent Guardrails for Clinical Safety & Hallucination Mitigation in Patient-Facing LLMs’, arXiv (Cornell University). doi:10.48550/arxiv.2604.26959.
IEEE
E. Nasarian, A. Neog, K. Tsui, and N. Hosseinichimeh, “CareGuardAI: Context-Aware Multi-Agent Guardrails for Clinical Safety & Hallucination Mitigation in Patient-Facing LLMs,” arXiv (Cornell University), 2026, doi: 10.48550/arxiv.2604.26959.
BibTeX
@misc{nasarian2026careguardai,
title = {{CareGuardAI: Context-Aware Multi-Agent Guardrails for Clinical Safety \& Hallucination Mitigation in Patient-Facing LLMs}},
author = {Nasarian, Elham and Neog, Abhilash and Tsui, Kwok‐Leung and Hosseinichimeh, Niyousha},
howpublished = {arXiv (Cornell University)},
year = {2026},
doi = {10.48550/arxiv.2604.26959},
url = {https://doi.org/10.48550/arxiv.2604.26959}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- large language models in healthcare
- clinical safety
- hallucination mitigation
- multi-agent systems
- safety guardrails
- medical question answering
Work with this paper
Most-cited papers in Medicine
- Yaşlı Kadınlarda Üreme SağlığıDemet Kokanalı, Yaprak Engin Üstün · Jinekoloji-Obstetrik ve Neonatoloji Tıp Dergisi · 2019 · 169,630 citations
- Global Cancer Statistics 2020: GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 CountriesHyuna Sung, Adalberto M. Filho, Mathieu Laversanne and 5 more · CA A Cancer Journal for Clinicians · 2021 · 119,885 citations
- “Mini-mental state”Marshal F. Folstein, Susan E. Folstein, Paul R. McHugh · Journal of Psychiatric Research · 1975 · 90,424 citations
- Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countriesFreddie Bray, Jacques Ferlay, Isabelle Soerjomataram and 3 more · CA A Cancer Journal for Clinicians · 2018 · 87,396 citations
- Hallmarks of Cancer: The Next GenerationDouglas Hanahan, Robert A. Weinberg · Cell · 2011 · 69,758 citations
Questions about this paper
How do I cite this paper in APA style?
Nasarian, E., Neog, A., Tsui, K., & Hosseinichimeh, N. (2026). CareGuardAI: Context-Aware Multi-Agent Guardrails for Clinical Safety & Hallucination Mitigation in Patient-Facing LLMs. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2604.26959
How many times has this paper been cited?
3 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at arXiv (Cornell University): sign in and open the PDF from this page.
Who wrote this paper?
Elham Nasarian, Abhilash Neog, Kwok‐Leung Tsui, Niyousha Hosseinichimeh.
What is the DOI of this paper?
10.48550/arxiv.2604.26959, permanently at https://doi.org/10.48550/arxiv.2604.26959. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.