Dissertation · 2025 · Open access
Preserving the Privacy of Language Models : Experiments in Clinical NLP
DiVA (Digitala Vetenskapliga Arkivet) · Published 1 January 2025
In brief
“Preserving the Privacy of Language Models : Experiments in Clinical NLP” is a 2025 dissertation by Vakili, Thomas, published in DiVA (Digitala Vetenskapliga Arkivet). OpenAlex counts 1 citations as of 19 November 2026. It is open access: the full text is free to read at DiVA (Digitala Vetenskapliga Arkivet).
- Citations
- 1
- Most cited in
- 2026 (1)
- Authors
- 1
Abstract
State-of-the-art methods in natural language processing (NLP) increasingly rely on large pre-trained language models. The strength of these models stems from their large number of parameters and the enormous amounts of data used to train them. The datasets are of a scale that makes it difficult, if not impossible, to audit them manually. When unwieldy amounts of potentially sensitive data are used to train large models, an important problem arises: unwelcome memorization of the training data. All datasets—including those based on publicly available data—can contain personally identifiable information (PII). When models memorize sensitive data, they become vulnerable to privacy attacks. Very few datasets for NLP can be guaranteed to be free of sensitive data. Consequently, most NLP models are susceptible to privacy leakage. This susceptibility is especially concerning in clinical NLP, where the data typically consist of electronic health records (EHRs). Leaking data from EHRs is never acceptable from a privacy perspective. This doctoral thesis investigates the privacy risks of using sensitive data and how they can be mitigated—while maintaining their utility as training data. A BERT model pre-trained using clinical data is subjected to a training data extraction attack. The same model is used to evaluate a membership inference attack that has been proposed to quantify the privacy risks of masked language models. Multiple experiments assess the performance gains from adapting pre-trained models to the clinical domain. Then, the impact of automatic de-identification on the performance of BERT models is evaluated for both pre-training and fine-tuning data. The final experiments of the thesis explore how synthetic training corpora can be generated while limiting the use of sensitive data, and working under computational constraints. The quality of these corpora, and the factors affecting their utility, are explored by training and evaluating BERT models. The results show that domain adaptation leads to significantly better performance on clinical NLP tasks. They also show that extracting training data from BERT models is difficult and suggest that the risks can be further decreased by automatically de-identifying the training data. Automatic de-identification is found to preserve the utility of the data used for pre-training and fine-tuning BERT models. However, we also find that contemporary membership inference attacks are unable to quantify the privacy benefits of this technique. Similarly, high-quality synthetic corpora can be generated using limited resources, but further research is needed to determine the privacy gains from using them. The results show that automatic de-identification and training data synthesis reduce the privacy risks of using sensitive data for NLP while preserving the utility of…
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Thomas, V. (2025). Preserving the Privacy of Language Models : Experiments in Clinical NLP. DiVA (Digitala Vetenskapliga Arkivet). http://urn.kb.se/resolve?urn=urn:nbn:se:su:diva-250015
MLA 9
Thomas, Vakili,. “Preserving the Privacy of Language Models : Experiments in Clinical NLP.” DiVA (Digitala Vetenskapliga Arkivet), 2025. http://urn.kb.se/resolve?urn=urn:nbn:se:su:diva-250015.
Chicago
Thomas, Vakili,. 2025. “Preserving the Privacy of Language Models : Experiments in Clinical NLP.” DiVA (Digitala Vetenskapliga Arkivet). http://urn.kb.se/resolve?urn=urn:nbn:se:su:diva-250015.
Harvard
Thomas, V. (2025) ‘Preserving the Privacy of Language Models : Experiments in Clinical NLP’, DiVA (Digitala Vetenskapliga Arkivet). Available at: http://urn.kb.se/resolve?urn=urn:nbn:se:su:diva-250015.
IEEE
V. Thomas, “Preserving the Privacy of Language Models : Experiments in Clinical NLP,” DiVA (Digitala Vetenskapliga Arkivet), 2025.
BibTeX
@phdthesis{thomas2025preserving,
title = {{Preserving the Privacy of Language Models : Experiments in Clinical NLP}},
author = {Thomas, Vakili,},
year = {2025},
url = {http://urn.kb.se/resolve?urn=urn:nbn:se:su:diva-250015}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- privacy leakage
- training data memorization
- membership inference attacks
- training data extraction
- synthetic training data
- clinical natural language processing
- electronic health records
- BERT
Work with this paper
Most-cited papers in Computer Science
- ImageNet classification with deep convolutional neural networksAlex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton · Communications of the ACM · 2017 · 114,726 citations
- Long Short-Term MemorySepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 109,097 citations
- A Mathematical Theory of CommunicationClaude E. Shannon · Bell System Technical Journal · 1948 · 91,736 citations
- Deep learningYann LeCun, Yoshua Bengio, Geoffrey E. Hinton · Nature · 2015 · 90,226 citations
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksShaoqing Ren, Kaiming He, Ross Girshick and 1 more · IEEE Transactions on Pattern Analysis and Machine Intelligence · 2016 · 74,143 citations
Questions about this paper
How do I cite this paper in APA style?
Thomas, V. (2025). Preserving the Privacy of Language Models : Experiments in Clinical NLP. DiVA (Digitala Vetenskapliga Arkivet). http://urn.kb.se/resolve?urn=urn:nbn:se:su:diva-250015
How many times has this paper been cited?
1 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at DiVA (Digitala Vetenskapliga Arkivet): sign in and open the full text from this page.
Who wrote this paper?
Vakili, Thomas.
Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.