Paper · 2025 · Open access
Evaluating Large Language Models for Privacy-Sensitive Healthcare Applications
Information Society 2025 - AI in Healthcare · Published 1 January 2025 · DOI 10.70314/is.gptzdravje.5
In brief
“Evaluating Large Language Models for Privacy-Sensitive Healthcare Applications” is a 2025 paper by Tadej Horvat and 4 co-authors, published in Information Society 2025 - AI in Healthcare. OpenAlex counts 0 citations as of 19 November 2026. It is open access: the full text is free to read.
- Citations
- 0
- References
- 8
- Authors
- 5
Abstract
Large language models (LLMs) are being systematically evaluated through accuracy for clinical use, yet privacy risks, limited transparency, and operational variability still complicate their adoption on sensitive health data. Motivated by an intended deployment in HomeDOCtor, a Slovenian medical platform, we present an agenda for evaluating LLMs in real-life privacy-sensitive healthcare applications. First, we map privacy risks: training-data extraction, input leakage, and output re-identification; and outline concrete mitigations (red-teaming, canary strings, differential privacy, filtering, and structured prompts). Second, we propose a lightweight, reproducible evaluation protocol that pairs model-side privacy checks with clinician-in-the-loop utility and safety assessments on de-identified data, aligned with EU GDPR expectations. Third, using small, domain-specific, clinically grounded benchmarks, we compare frontier, commercial, and open-weight models and analyze trade-offs among utility, privacy, and maintainability in the HomeDOCtor context. Finally, we discuss deployment and governance patterns for healthcare operators (access control, audit logging, data minimization, incident response). Our results suggest that (i) focused, task-specific evaluations are more informative than generic world-wide benchmarks for patient-facing use; (ii) suitably hardened and monitored open-weight models can be viable although their quality is not comparable to top commercial systems; and (iii) privacy risk cannot be eliminated but can be bounded and operationalized. We conclude with recommendations for ethics approvals, documentation, and reproducibility to support safe adoption in Slovenia and beyond.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Horvat, T., Roštan, Ž., Ljubljana, S. F. Z. R. I. I., Jaš, J., & Ljubljana, S. F. Z. R. I. I. (2025). Evaluating Large Language Models for Privacy-Sensitive Healthcare Applications. Information Society 2025 - AI in Healthcare. https://doi.org/10.70314/is.gptzdravje.5
MLA 9
Horvat, Tadej, et al. “Evaluating Large Language Models for Privacy-Sensitive Healthcare Applications.” Information Society 2025 - AI in Healthcare, 2025. https://doi.org/10.70314/is.gptzdravje.5.
Chicago
Horvat, Tadej, Žan Roštan, Slovenija Fakulteta za računalništvo in informatiko Ljubljana, Jakob Jaš, and Slovenija Fakulteta za računalništvo in informatiko Ljubljana. 2025. “Evaluating Large Language Models for Privacy-Sensitive Healthcare Applications.” Information Society 2025 - AI in Healthcare. https://doi.org/10.70314/is.gptzdravje.5.
Harvard
Horvat, T. et al. (2025) ‘Evaluating Large Language Models for Privacy-Sensitive Healthcare Applications’, Information Society 2025 - AI in Healthcare. doi:10.70314/is.gptzdravje.5.
IEEE
T. Horvat, Ž. Roštan, S. F. Z. R. I. I. Ljubljana, J. Jaš, and S. F. Z. R. I. I. Ljubljana, “Evaluating Large Language Models for Privacy-Sensitive Healthcare Applications,” Information Society 2025 - AI in Healthcare, 2025, doi: 10.70314/is.gptzdravje.5.
BibTeX
@article{horvat2025evaluating,
title = {{Evaluating Large Language Models for Privacy-Sensitive Healthcare Applications}},
author = {Horvat, Tadej and Roštan, Žan and Ljubljana, Slovenija Fakulteta za računalništvo in informatiko and Jaš, Jakob and Ljubljana, Slovenija Fakulteta za računalništvo in informatiko},
journal = {Information Society 2025 - AI in Healthcare},
year = {2025},
doi = {10.70314/is.gptzdravje.5},
url = {https://doi.org/10.70314/is.gptzdravje.5}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- large language models
- privacy risk assessment
- healthcare data privacy
- re-identification
- training data extraction
- differential privacy
- red teaming
- prompt engineering
Work with this paper
Most-cited papers in Medicine
- Yaşlı Kadınlarda Üreme SağlığıDemet Kokanalı, Yaprak Engin Üstün · Jinekoloji-Obstetrik ve Neonatoloji Tıp Dergisi · 2019 · 169,630 citations
- Global Cancer Statistics 2020: GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 CountriesHyuna Sung, Adalberto M. Filho, Mathieu Laversanne and 5 more · CA A Cancer Journal for Clinicians · 2021 · 119,885 citations
- “Mini-mental state”Marshal F. Folstein, Susan E. Folstein, Paul R. McHugh · Journal of Psychiatric Research · 1975 · 90,424 citations
- Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countriesFreddie Bray, Jacques Ferlay, Isabelle Soerjomataram and 3 more · CA A Cancer Journal for Clinicians · 2018 · 87,396 citations
- Hallmarks of Cancer: The Next GenerationDouglas Hanahan, Robert A. Weinberg · Cell · 2011 · 69,758 citations
Questions about this paper
How do I cite this paper in APA style?
Horvat, T., Roštan, Ž., Ljubljana, S. F. Z. R. I. I., Jaš, J., & Ljubljana, S. F. Z. R. I. I. (2025). Evaluating Large Language Models for Privacy-Sensitive Healthcare Applications. Information Society 2025 - AI in Healthcare. https://doi.org/10.70314/is.gptzdravje.5
How many times has this paper been cited?
0 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access: sign in and open the PDF from this page.
Who wrote this paper?
Tadej Horvat, Žan Roštan, Slovenija Fakulteta za računalništvo in informatiko Ljubljana, Jakob Jaš, Slovenija Fakulteta za računalništvo in informatiko Ljubljana.
What is the DOI of this paper?
10.70314/is.gptzdravje.5, permanently at https://doi.org/10.70314/is.gptzdravje.5. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.