Journal article · 2012 · Open access
Large-scale evaluation of automated clinical note de-identification and its impact on information extraction
Journal of the American Medical Informatics Association, 20(1), 84–94 · Published 3 August 2012 · DOI 10.1136/amiajnl-2012-001012
In brief
“Large-scale evaluation of automated clinical note de-identification and its impact on information extraction” is a 2012 journal article by Louise Deléger and 10 co-authors, published in Journal of the American Medical Informatics Association. OpenAlex counts 111 citations as of 19 November 2026, which puts it in the top 8% of works published in 2012 by citations. It is open access under a CC BY-NC licence: the full text is free to read at Journal of the American Medical Informatics Association.
- Citations
- 111
- Most cited in
- 2017 (12)
- Field-weighted impact
- 4.01
- References
- 45
- Authors
- 11
Abstract
OBJECTIVE: (1) To evaluate a state-of-the-art natural language processing (NLP)-based approach to automatically de-identify a large set of diverse clinical notes. (2) To measure the impact of de-identification on the performance of information extraction algorithms on the de-identified documents. MATERIAL AND METHODS: A cross-sectional study that included 3503 stratified, randomly selected clinical notes (over 22 note types) from five million documents produced at one of the largest US pediatric hospitals. Sensitivity, precision, F value of two automated de-identification systems for removing all 18 HIPAA-defined protected health information elements were computed. Performance was assessed against a manually generated 'gold standard'. Statistical significance was tested. The automated de-identification performance was also compared with that of two humans on a 10% subsample of the gold standard. The effect of de-identification on the performance of subsequent medication extraction was measured. RESULTS: The gold standard included 30 815 protected health information elements and more than one million tokens. The most accurate NLP method had 91.92% sensitivity (R) and 95.08% precision (P) overall. The performance of the system was indistinguishable from that of human annotators (annotators' performance was 92.15%(R)/93.95%(P) and 94.55%(R)/88.45%(P) overall while the best system obtained 92.91%(R)/95.73%(P) on same text). The impact of automated de-identification was minimal on the utility of the narrative notes for subsequent information extraction as measured by the sensitivity and precision of medication name extraction. DISCUSSION AND CONCLUSION: NLP-based de-identification shows excellent performance that rivals the performance of human annotators. Furthermore, unlike manual de-identification, the automated approach scales up to millions of documents quickly and inexpensively.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Deléger, L., Molnar, K., Savova, G., Xia, F., Lingren, T., Li, Q., Marsolo, K. A., Jegga, A. G., Kaiser, M., Stoutenborough, L., & Solti, I. (2012). Large-scale evaluation of automated clinical note de-identification and its impact on information extraction. Journal of the American Medical Informatics Association, 20(1), 84–94. https://doi.org/10.1136/amiajnl-2012-001012
MLA 9
Deléger, Louise, et al. “Large-scale evaluation of automated clinical note de-identification and its impact on information extraction.” Journal of the American Medical Informatics Association, vol. 20, no. 1, 2012, pp. 84–94. https://doi.org/10.1136/amiajnl-2012-001012.
Chicago
Deléger, Louise, K. Molnar, Guergana Savova, F. Xia, Todd Lingren, Qian Li, Keith A. Marsolo, et al. 2012. “Large-scale evaluation of automated clinical note de-identification and its impact on information extraction.” Journal of the American Medical Informatics Association 20 (1): 84–94. https://doi.org/10.1136/amiajnl-2012-001012.
Harvard
Deléger, L. et al. (2012) ‘Large-scale evaluation of automated clinical note de-identification and its impact on information extraction’, Journal of the American Medical Informatics Association, 20(1), pp. 84–94. doi:10.1136/amiajnl-2012-001012.
IEEE
L. Deléger et al., “Large-scale evaluation of automated clinical note de-identification and its impact on information extraction,” Journal of the American Medical Informatics Association, vol. 20, no. 1, pp. 84–94, 2012, doi: 10.1136/amiajnl-2012-001012.
BibTeX
@article{deleger2012large,
title = {{Large-scale evaluation of automated clinical note de-identification and its impact on information extraction}},
author = {Deléger, Louise and Molnar, K. and Savova, Guergana and Xia, F. and Lingren, Todd and Li, Qian and Marsolo, Keith A. and Jegga, Anil Goud and Kaiser, Megan and Stoutenborough, Laura and Solti, Imre},
journal = {Journal of the American Medical Informatics Association},
year = {2012},
volume = {20},
number = {1},
pages = {84--94},
doi = {10.1136/amiajnl-2012-001012},
url = {https://doi.org/10.1136/amiajnl-2012-001012}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Citations by year
| Year | Citations | Share of the best year |
|---|---|---|
| 2017 | 12 | |
| 2018 | 6 | |
| 2019 | 8 | |
| 2020 | 4 | |
| 2021 | 4 | |
| 2022 | 9 | |
| 2023 | 8 | |
| 2024 | 8 | |
| 2025 | 4 | |
| 2026 | 5 |
Its field-weighted citation impact is 4.01: it has been cited 4.01 times as often as the average work of the same field, type and year (1.00 is the world average).
Topic and keywords
Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- natural language processing
- information extraction
- protected health information
- HIPAA
Work with this paper
Most-cited papers in Medicine
- Yaşlı Kadınlarda Üreme SağlığıDemet Kokanalı, Yaprak Engin Üstün · Jinekoloji-Obstetrik ve Neonatoloji Tıp Dergisi · 2019 · 169,630 citations
- Global Cancer Statistics 2020: GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 CountriesHyuna Sung, Adalberto M. Filho, Mathieu Laversanne and 5 more · CA A Cancer Journal for Clinicians · 2021 · 119,885 citations
- “Mini-mental state”Marshal F. Folstein, Susan E. Folstein, Paul R. McHugh · Journal of Psychiatric Research · 1975 · 90,424 citations
- Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countriesFreddie Bray, Jacques Ferlay, Isabelle Soerjomataram and 3 more · CA A Cancer Journal for Clinicians · 2018 · 87,396 citations
- Hallmarks of Cancer: The Next GenerationDouglas Hanahan, Robert A. Weinberg · Cell · 2011 · 69,758 citations
Questions about this paper
How do I cite this paper in APA style?
Deléger, L., Molnar, K., Savova, G., Xia, F., Lingren, T., Li, Q., Marsolo, K. A., Jegga, A. G., Kaiser, M., Stoutenborough, L., & Solti, I. (2012). Large-scale evaluation of automated clinical note de-identification and its impact on information extraction. Journal of the American Medical Informatics Association, 20(1), 84–94. https://doi.org/10.1136/amiajnl-2012-001012
How many times has this paper been cited?
111 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at Journal of the American Medical Informatics Association: sign in and open the PDF from this page.
Who wrote this paper?
Louise Deléger, K. Molnar, Guergana Savova, F. Xia, Todd Lingren, Qian Li, Keith A. Marsolo, Anil Goud Jegga, Megan Kaiser, Laura Stoutenborough, Imre Solti.
What is the DOI of this paper?
10.1136/amiajnl-2012-001012, permanently at https://doi.org/10.1136/amiajnl-2012-001012. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.