Preprint · 2026 · Open access

Trust but Verify: Mitigating Medical Hallucinations via Post-Hoc Adversarial Auditing and Multi-Agent Feedback Loops

Muhammad Osama, Maheera Amjad, Zartasha Mustansar, Arslan Shaukat, Maham Khan

arXiv (Cornell University) · Published 12 June 2026 · DOI 10.48550/arxiv.2606.14149

Open PDFView at the publisherCite this paper

In brief

“Trust but Verify: Mitigating Medical Hallucinations via Post-Hoc Adversarial Auditing and Multi-Agent Feedback Loops” is a 2026 preprint by Muhammad Osama and 4 co-authors, published in arXiv (Cornell University). OpenAlex counts 3 citations as of 19 November 2026. It is open access under a CC BY licence: the full text is free to read at arXiv (Cornell University).

Citations
3
Most cited in
2026 (3)
References
22
Authors
5

Abstract

Large Language Models (LLMs) are increasingly deployed in healthcare settings, yet their tendency to hallucinate poses risks when clinical decisions are involved. This study examine whether LLMs recommend recently banned or withdrawn pharmaceuticals when answering clinical questions and tests an agent-based method for reducing such errors. We developed a five-agent "Trust but Verify" system using a single LLM backbone. To measure regulatory knowledge obsolescence, we created an adversarial dataset of 103 clinical MCQs where historically correct answers now refer to banned substances. This scale ensures statistical significance across various therapeutic classes. We evaluated three open-access model families (GPT-OSS, Llama-3, Falcon-3) under vanilla and agentic conditions. Performance was measured via pointwise score, label accuracy, Hallucination Error Rate (HER), and Component Fidelity (CF) score. We also observed clinical safety regression in proprietary models. In default configurations, all models showed high hallucination rates, consistently selecting banned drugs that matched training data patterns. Our proposed agentic architecture reduced HER by approximately 53% across models. Pointwise scores shifted from -0.25 (unsafe recommendation) toward 0.0 (appropriate refusal). The safety audit intercepted dangerous outputs even when models' parametric knowledge favored the banned substance. The proposed multi-agent framework offers a model-agnostic method for enforcing regulatory compliance that prioritizes patient safety over fluent text generation. Our work demonstrates a practical approach for deploying autonomous AI systems in safety-critical healthcare settings. It shows how real-time regulatory data can be integrated into LLM pipelines to support clinical decision-making.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Osama, M., Amjad, M., Mustansar, Z., Shaukat, A., & Khan, M. (2026). Trust but Verify: Mitigating Medical Hallucinations via Post-Hoc Adversarial Auditing and Multi-Agent Feedback Loops. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2606.14149

MLA 9

Osama, Muhammad, et al. “Trust but Verify: Mitigating Medical Hallucinations via Post-Hoc Adversarial Auditing and Multi-Agent Feedback Loops.” arXiv (Cornell University), 2026. https://doi.org/10.48550/arxiv.2606.14149.

Chicago

Osama, Muhammad, Maheera Amjad, Zartasha Mustansar, Arslan Shaukat, and Maham Khan. 2026. “Trust but Verify: Mitigating Medical Hallucinations via Post-Hoc Adversarial Auditing and Multi-Agent Feedback Loops.” arXiv (Cornell University). https://doi.org/10.48550/arxiv.2606.14149.

Harvard

Osama, M. et al. (2026) ‘Trust but Verify: Mitigating Medical Hallucinations via Post-Hoc Adversarial Auditing and Multi-Agent Feedback Loops’, arXiv (Cornell University). doi:10.48550/arxiv.2606.14149.

IEEE

M. Osama, M. Amjad, Z. Mustansar, A. Shaukat, and M. Khan, “Trust but Verify: Mitigating Medical Hallucinations via Post-Hoc Adversarial Auditing and Multi-Agent Feedback Loops,” arXiv (Cornell University), 2026, doi: 10.48550/arxiv.2606.14149.

BibTeX

@misc{osama2026trust,
  title     = {{Trust but Verify: Mitigating Medical Hallucinations via Post-Hoc Adversarial Auditing and Multi-Agent Feedback Loops}},
  author    = {Osama, Muhammad and Amjad, Maheera and Mustansar, Zartasha and Shaukat, Arslan and Khan, Maham},
  howpublished = {arXiv (Cornell University)},
  year      = {2026},
  doi       = {10.48550/arxiv.2606.14149},
  url       = {https://doi.org/10.48550/arxiv.2606.14149}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education

  • large language models in healthcare
  • multi-agent systems
  • clinical decision support
  • safety-critical AI

Work with this paper

Most-cited papers in Medicine

All highly cited Medicine papers

Questions about this paper

How do I cite this paper in APA style?

Osama, M., Amjad, M., Mustansar, Z., Shaukat, A., & Khan, M. (2026). Trust but Verify: Mitigating Medical Hallucinations via Post-Hoc Adversarial Auditing and Multi-Agent Feedback Loops. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2606.14149

How many times has this paper been cited?

3 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at arXiv (Cornell University): sign in and open the PDF from this page.

Who wrote this paper?

Muhammad Osama, Maheera Amjad, Zartasha Mustansar, Arslan Shaukat, Maham Khan.

What is the DOI of this paper?

10.48550/arxiv.2606.14149, permanently at https://doi.org/10.48550/arxiv.2606.14149. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.