Preprint · 2024 · Open access

REALM: RAG-Driven Enhancement of Multimodal Electronic Health Records Analysis via Large Language Models

Yinghao Zhu, Changyu Ren, Shiyun Xie, Shukai Liu, Hangyuan Ji, Zixiang Wang, Tao Sun, Long He, Zhoujun Li, Xi Zhu, Chengwei Pan

arXiv (Cornell University) · Published 10 February 2024 · DOI 10.48550/arxiv.2402.07016

Open PDFView at the publisherCite this paper

In brief

“REALM: RAG-Driven Enhancement of Multimodal Electronic Health Records Analysis via Large Language Models” is a 2024 preprint by Yinghao Zhu and 10 co-authors, published in arXiv (Cornell University). OpenAlex counts 98 citations as of 19 November 2026. It is open access: the full text is free to read at arXiv (Cornell University).

Citations
98
Most cited in
2025 (48)
References
40
Authors
11

Abstract

The integration of multimodal Electronic Health Records (EHR) data has significantly improved clinical predictive capabilities. Leveraging clinical notes and multivariate time-series EHR, existing models often lack the medical context relevent to clinical tasks, prompting the incorporation of external knowledge, particularly from the knowledge graph (KG). Previous approaches with KG knowledge have primarily focused on structured knowledge extraction, neglecting unstructured data modalities and semantic high dimensional medical knowledge. In response, we propose REALM, a Retrieval-Augmented Generation (RAG) driven framework to enhance multimodal EHR representations that address these limitations. Firstly, we apply Large Language Model (LLM) to encode long context clinical notes and GRU model to encode time-series EHR data. Secondly, we prompt LLM to extract task-relevant medical entities and match entities in professionally labeled external knowledge graph (PrimeKG) with corresponding medical knowledge. By matching and aligning with clinical standards, our framework eliminates hallucinations and ensures consistency. Lastly, we propose an adaptive multimodal fusion network to integrate extracted knowledge with multimodal EHR data. Our extensive experiments on MIMIC-III mortality and readmission tasks showcase the superior performance of our REALM framework over baselines, emphasizing the effectiveness of each module. REALM framework contributes to refining the use of multimodal EHR data in healthcare and bridging the gap with nuanced medical context essential for informed clinical predictions.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Zhu, Y., Ren, C., Xie, S., Liu, S., Ji, H., Wang, Z., Sun, T., He, L., Li, Z., Zhu, X., & Pan, C. (2024). REALM: RAG-Driven Enhancement of Multimodal Electronic Health Records Analysis via Large Language Models. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2402.07016

MLA 9

Zhu, Yinghao, et al. “REALM: RAG-Driven Enhancement of Multimodal Electronic Health Records Analysis via Large Language Models.” arXiv (Cornell University), 2024. https://doi.org/10.48550/arxiv.2402.07016.

Chicago

Zhu, Yinghao, Changyu Ren, Shiyun Xie, Shukai Liu, Hangyuan Ji, Zixiang Wang, Tao Sun, et al. 2024. “REALM: RAG-Driven Enhancement of Multimodal Electronic Health Records Analysis via Large Language Models.” arXiv (Cornell University). https://doi.org/10.48550/arxiv.2402.07016.

Harvard

Zhu, Y. et al. (2024) ‘REALM: RAG-Driven Enhancement of Multimodal Electronic Health Records Analysis via Large Language Models’, arXiv (Cornell University). doi:10.48550/arxiv.2402.07016.

IEEE

Y. Zhu et al., “REALM: RAG-Driven Enhancement of Multimodal Electronic Health Records Analysis via Large Language Models,” arXiv (Cornell University), 2024, doi: 10.48550/arxiv.2402.07016.

BibTeX

@misc{zhu2024realm,
  title     = {{REALM: RAG-Driven Enhancement of Multimodal Electronic Health Records Analysis via Large Language Models}},
  author    = {Zhu, Yinghao and Ren, Changyu and Xie, Shiyun and Liu, Shukai and Ji, Hangyuan and Wang, Zixiang and Sun, Tao and He, Long and Li, Zhoujun and Zhu, Xi and Pan, Chengwei},
  howpublished = {arXiv (Cornell University)},
  year      = {2024},
  doi       = {10.48550/arxiv.2402.07016},
  url       = {https://doi.org/10.48550/arxiv.2402.07016}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Citations by year

YearCitationsShare of the best year
202311
20241313
20254848
20263636

Topic and keywords

Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare

  • retrieval-augmented generation
  • large language models
  • knowledge graphs
  • clinical notes
  • medical entity extraction
  • multimodal fusion
  • hallucination mitigation
  • MIMIC-III

Work with this paper

Most-cited papers in Computer Science

All highly cited Computer Science papers

Questions about this paper

How do I cite this paper in APA style?

Zhu, Y., Ren, C., Xie, S., Liu, S., Ji, H., Wang, Z., Sun, T., He, L., Li, Z., Zhu, X., & Pan, C. (2024). REALM: RAG-Driven Enhancement of Multimodal Electronic Health Records Analysis via Large Language Models. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2402.07016

How many times has this paper been cited?

98 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at arXiv (Cornell University): sign in and open the PDF from this page.

Who wrote this paper?

Yinghao Zhu, Changyu Ren, Shiyun Xie, Shukai Liu, Hangyuan Ji, Zixiang Wang, Tao Sun, Long He, Zhoujun Li, Xi Zhu, Chengwei Pan.

What is the DOI of this paper?

10.48550/arxiv.2402.07016, permanently at https://doi.org/10.48550/arxiv.2402.07016. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.