Preprint · 2024 · Open access
Comparative evaluation of large-language models and purpose-built software for medical record de-identification
Research Square · Published 14 October 2024 · DOI 10.21203/rs.3.rs-4870585/v1
In brief
“Comparative evaluation of large-language models and purpose-built software for medical record de-identification” is a 2024 preprint by Rachel Yi Ling Kuo and 7 co-authors, published in Research Square. OpenAlex counts 1 citations as of 19 November 2026. It is open access under a CC BY licence: the full text is free to read at Research Square.
- Citations
- 1
- Most cited in
- 2025 (1)
- References
- 24
- Authors
- 8
Abstract
Background: Robust de-identification is necessary to preserve patient confidentiality and maintain public acceptability for electronic health record (EHR) research. Manual redaction of personally identifiable information (PII) is time-consuming and expensive, limiting the scale of data-sharing. Automated de-identification could alleviate this burden, but the best strategy is not clear. Advances in natural language processing (NLP) and the emergence of foundational large language models (LLMs) show promise in performing clinical NLP tasks with no, or limited training. Methods: We evaluated two task-specific (Microsoft Azure de-identification service, AnonCAT) and five general LLMs (Gemma-7b-IT, Llama-3-8B-Instruct, Phi-3-mini-128k-instruct, GPT3.5-turbo-base, GPT-4-0125) in de-identifying 3650 medical records from a UK hospital group, split into general and specialised datasets. Records were dual-annotated by clinicians for PII. Inter-annotator reliability was used to benchmark performance. The primary outcome was F1, precision (positive predictive value) and recall (sensitivity) for each comparator in classifying words as PII vs. non-PII. The secondary outcomes were performance per-PII-subtype, per-dataset, and the presence of LLM hallucinations. We report outcomes at zero- and few-shot learning for LLMs, and with/without fine-tuning for AnonCAT. Results: 17496/479760 (3.65%) words were PII. Inter-annotator F1 for word-level PII/non-PII was 0.977 (95%CI 0.957-0.991), precision 0.967 (0.923-0.993), and recall 0.986 (0.971-0.997). The best performing redaction tool was the Microsoft Azure de-identification service: F1 0.933 (0.928-0.938), precision 0.916 (0.930-0.922), recall 0.950 (0.942-0.957). The next-best were fine-tuned-AnonCAT: F1 0.873 (0.864-0.882), precision 0.981 (0.977-0.985), recall 0.787 (0.773-0.800), and GPT-4-0125 (ten-shots): F1 0.898 (0.876-0.915), precision 0.924 (0.914-0.933), recall 0.874 (0.834-0.905). There was hallucinatory output in Phi-3-mini-128k-instruct and Llama-3-8B-Instruct at zero-, one-, and five-shots, and universally for Gemma-7b-IT. Names/dates were consistently redacted by all comparators; there was variable performance for other categories. Fine-tuned-AnonCAT demonstrated the least performance shift across datasets. Conclusion: Automated EHR de-identification could facilitate large-scale, domain-agnostic record sharing for medical research, alongside other safeguards to prevent patient reidentification.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Kuo, R. Y. L., Soltan, A. A. S., O’Hanlon, C., Hasanic, A., Clifton, D. A., Collins, G. S., Furniss, D., & Eyre, D. W. (2024). Comparative evaluation of large-language models and purpose-built software for medical record de-identification. Research Square. https://doi.org/10.21203/rs.3.rs-4870585/v1
MLA 9
Kuo, Rachel Yi Ling, et al. “Comparative evaluation of large-language models and purpose-built software for medical record de-identification.” Research Square, 2024. https://doi.org/10.21203/rs.3.rs-4870585/v1.
Chicago
Kuo, Rachel Yi Ling, Andrew A. S. Soltan, Ciaran O’Hanlon, Alan Hasanic, David Andrew Clifton, Gary S. Collins, Dominic Furniss, and David W. Eyre. 2024. “Comparative evaluation of large-language models and purpose-built software for medical record de-identification.” Research Square. https://doi.org/10.21203/rs.3.rs-4870585/v1.
Harvard
Kuo, R. Y. L. et al. (2024) ‘Comparative evaluation of large-language models and purpose-built software for medical record de-identification’, Research Square. doi:10.21203/rs.3.rs-4870585/v1.
IEEE
R. Y. L. Kuo et al., “Comparative evaluation of large-language models and purpose-built software for medical record de-identification,” Research Square, 2024, doi: 10.21203/rs.3.rs-4870585/v1.
BibTeX
@misc{kuo2024comparative,
title = {{Comparative evaluation of large-language models and purpose-built software for medical record de-identification}},
author = {Kuo, Rachel Yi Ling and Soltan, Andrew A. S. and O’Hanlon, Ciaran and Hasanic, Alan and Clifton, David Andrew and Collins, Gary S. and Furniss, Dominic and Eyre, David W.},
howpublished = {Research Square},
year = {2024},
doi = {10.21203/rs.3.rs-4870585/v1},
url = {https://doi.org/10.21203/rs.3.rs-4870585/v1}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- large language models
- personally identifiable information
- electronic health records
- LLM hallucination
- few-shot learning
- fine-tuning
- clinical natural language processing
- inter-annotator reliability
Work with this paper
Most-cited papers in Computer Science
- ImageNet classification with deep convolutional neural networksAlex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton · Communications of the ACM · 2017 · 114,726 citations
- Long Short-Term MemorySepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 109,097 citations
- A Mathematical Theory of CommunicationClaude E. Shannon · Bell System Technical Journal · 1948 · 91,736 citations
- Deep learningYann LeCun, Yoshua Bengio, Geoffrey E. Hinton · Nature · 2015 · 90,226 citations
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksShaoqing Ren, Kaiming He, Ross Girshick and 1 more · IEEE Transactions on Pattern Analysis and Machine Intelligence · 2016 · 74,143 citations
Questions about this paper
How do I cite this paper in APA style?
Kuo, R. Y. L., Soltan, A. A. S., O’Hanlon, C., Hasanic, A., Clifton, D. A., Collins, G. S., Furniss, D., & Eyre, D. W. (2024). Comparative evaluation of large-language models and purpose-built software for medical record de-identification. Research Square. https://doi.org/10.21203/rs.3.rs-4870585/v1
How many times has this paper been cited?
1 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at Research Square: sign in and open the full text from this page.
Who wrote this paper?
Rachel Yi Ling Kuo, Andrew A. S. Soltan, Ciaran O’Hanlon, Alan Hasanic, David Andrew Clifton, Gary S. Collins, Dominic Furniss, David W. Eyre.
What is the DOI of this paper?
10.21203/rs.3.rs-4870585/v1, permanently at https://doi.org/10.21203/rs.3.rs-4870585/v1. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.