Journal article · 2025 · Open access
Large language models-in-the-loop: Leveraging expert small artificial intelligence models for multilingual anonymization and de-identification of protected health information
Artificial Intelligence in Health, 3(1), 138 · Published 19 September 2025 · DOI 10.36922/aih025120021
In brief
“Large language models-in-the-loop: Leveraging expert small artificial intelligence models for multilingual anonymization and de-identification of protected health information” is a 2025 journal article by Murat Gunay and 2 co-authors, published in Artificial Intelligence in Health. OpenAlex counts 7 citations as of 19 November 2026, which puts it in the top 5% of works published in 2025 by citations. It is open access under a CC BY licence: the full text is free to read at Artificial Intelligence in Health.
- Citations
- 7
- Most cited in
- 2026 (4)
- Field-weighted impact
- 1.33
- References
- 40
- Authors
- 3
Abstract
The rise of chronic diseases and pandemics, such as COVID-19 has emphasized the need for effective patient data processing while ensuring privacy through anonymization and de-identification of protected health information. Anonymized data facilitates research without compromising patient confidentiality. This paper introduces expert small artificial intelligence (AI) models developed using the large language model (LLM)-in-the-loop methodology to meet the demand for domain-specific de-identification of named entity recognition (NER) models. These models overcome the privacy risks associated with LLMs used through application programming interfaces by eliminating the need to transmit or store sensitive data. More importantly, they consistently outperform LLMs in de-identification tasks, offering superior performance and reliability. Our de-identification NER models, developed in eight languages—English, German, Italian, French, Romanian, Turkish, Spanish, and Arabic—achieved F1-macro score averages of 0.931, 0.960, 0.955, 0.937, 0.930, 0.963, 0.957, and 0.922, respectively. These results establish our de-identification NER models as the most accurate healthcare anonymization solutions, surpassing existing small models and even general-purpose LLMs, such as GPT-4o. While Part I of this series introduced the LLM-in-the-loop methodology for biomedical document translation, this second paper showcases its success in developing cost-effective expert small NER models in de-identification tasks. Our findings lay the groundwork for future healthcare AI innovations, including biomedical entity and relation extraction, demonstrating the value of specialized models for domain-specific challenges.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Gunay, M., Keles, B., & Hizlan, R. (2025). Large language models-in-the-loop: Leveraging expert small artificial intelligence models for multilingual anonymization and de-identification of protected health information. Artificial Intelligence in Health, 3(1), 138. https://doi.org/10.36922/aih025120021
MLA 9
Gunay, Murat, et al. “Large language models-in-the-loop: Leveraging expert small artificial intelligence models for multilingual anonymization and de-identification of protected health information.” Artificial Intelligence in Health, vol. 3, no. 1, 2025, p. 138. https://doi.org/10.36922/aih025120021.
Chicago
Gunay, Murat, Bunyamin Keles, and Raife Hizlan. 2025. “Large language models-in-the-loop: Leveraging expert small artificial intelligence models for multilingual anonymization and de-identification of protected health information.” Artificial Intelligence in Health 3 (1): 138. https://doi.org/10.36922/aih025120021.
Harvard
Gunay, M., Keles, B. and Hizlan, R. (2025) ‘Large language models-in-the-loop: Leveraging expert small artificial intelligence models for multilingual anonymization and de-identification of protected health information’, Artificial Intelligence in Health, 3(1), p. 138. doi:10.36922/aih025120021.
IEEE
M. Gunay, B. Keles, and R. Hizlan, “Large language models-in-the-loop: Leveraging expert small artificial intelligence models for multilingual anonymization and de-identification of protected health information,” Artificial Intelligence in Health, vol. 3, no. 1, p. 138, 2025, doi: 10.36922/aih025120021.
BibTeX
@article{gunay2025large,
title = {{Large language models-in-the-loop: Leveraging expert small artificial intelligence models for multilingual anonymization and de-identification of protected health information}},
author = {Gunay, Murat and Keles, Bunyamin and Hizlan, Raife},
journal = {Artificial Intelligence in Health},
year = {2025},
volume = {3},
number = {1},
pages = {138},
doi = {10.36922/aih025120021},
url = {https://doi.org/10.36922/aih025120021}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- named entity recognition
- de-identification
- data anonymization
- protected health information
- multilingual NER
- healthcare data privacy
- biomedical text processing
Work with this paper
Most-cited papers in Computer Science
- ImageNet classification with deep convolutional neural networksAlex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton · Communications of the ACM · 2017 · 114,726 citations
- Long Short-Term MemorySepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 109,097 citations
- A Mathematical Theory of CommunicationClaude E. Shannon · Bell System Technical Journal · 1948 · 91,736 citations
- Deep learningYann LeCun, Yoshua Bengio, Geoffrey E. Hinton · Nature · 2015 · 90,226 citations
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksShaoqing Ren, Kaiming He, Ross Girshick and 1 more · IEEE Transactions on Pattern Analysis and Machine Intelligence · 2016 · 74,143 citations
Questions about this paper
How do I cite this paper in APA style?
Gunay, M., Keles, B., & Hizlan, R. (2025). Large language models-in-the-loop: Leveraging expert small artificial intelligence models for multilingual anonymization and de-identification of protected health information. Artificial Intelligence in Health, 3(1), 138. https://doi.org/10.36922/aih025120021
How many times has this paper been cited?
7 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at Artificial Intelligence in Health: sign in and open the full text from this page.
Who wrote this paper?
Murat Gunay, Bunyamin Keles, Raife Hizlan.
What is the DOI of this paper?
10.36922/aih025120021, permanently at https://doi.org/10.36922/aih025120021. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.