Journal article · 2012 · Open access

Large-scale evaluation of automated clinical note de-identification and its impact on information extraction

Louise Deléger, K. Molnar, Guergana Savova, F. Xia, Todd Lingren, Qian Li, Keith A. Marsolo, Anil Goud Jegga, Megan Kaiser, Laura Stoutenborough, Imre Solti

Journal of the American Medical Informatics Association, 20(1), 84–94 · Published 3 August 2012 · DOI 10.1136/amiajnl-2012-001012

Open PDFView at the publisherCite this paper

In brief

“Large-scale evaluation of automated clinical note de-identification and its impact on information extraction” is a 2012 journal article by Louise Deléger and 10 co-authors, published in Journal of the American Medical Informatics Association. OpenAlex counts 111 citations as of 19 November 2026, which puts it in the top 8% of works published in 2012 by citations. It is open access under a CC BY-NC licence: the full text is free to read at Journal of the American Medical Informatics Association.

Citations
111
Most cited in
2017 (12)
Field-weighted impact
4.01
References
45
Authors
11

Abstract

OBJECTIVE: (1) To evaluate a state-of-the-art natural language processing (NLP)-based approach to automatically de-identify a large set of diverse clinical notes. (2) To measure the impact of de-identification on the performance of information extraction algorithms on the de-identified documents. MATERIAL AND METHODS: A cross-sectional study that included 3503 stratified, randomly selected clinical notes (over 22 note types) from five million documents produced at one of the largest US pediatric hospitals. Sensitivity, precision, F value of two automated de-identification systems for removing all 18 HIPAA-defined protected health information elements were computed. Performance was assessed against a manually generated 'gold standard'. Statistical significance was tested. The automated de-identification performance was also compared with that of two humans on a 10% subsample of the gold standard. The effect of de-identification on the performance of subsequent medication extraction was measured. RESULTS: The gold standard included 30 815 protected health information elements and more than one million tokens. The most accurate NLP method had 91.92% sensitivity (R) and 95.08% precision (P) overall. The performance of the system was indistinguishable from that of human annotators (annotators' performance was 92.15%(R)/93.95%(P) and 94.55%(R)/88.45%(P) overall while the best system obtained 92.91%(R)/95.73%(P) on same text). The impact of automated de-identification was minimal on the utility of the narrative notes for subsequent information extraction as measured by the sensitivity and precision of medication name extraction. DISCUSSION AND CONCLUSION: NLP-based de-identification shows excellent performance that rivals the performance of human annotators. Furthermore, unlike manual de-identification, the automated approach scales up to millions of documents quickly and inexpensively.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Deléger, L., Molnar, K., Savova, G., Xia, F., Lingren, T., Li, Q., Marsolo, K. A., Jegga, A. G., Kaiser, M., Stoutenborough, L., & Solti, I. (2012). Large-scale evaluation of automated clinical note de-identification and its impact on information extraction. Journal of the American Medical Informatics Association, 20(1), 84–94. https://doi.org/10.1136/amiajnl-2012-001012

MLA 9

Deléger, Louise, et al. “Large-scale evaluation of automated clinical note de-identification and its impact on information extraction.” Journal of the American Medical Informatics Association, vol. 20, no. 1, 2012, pp. 84–94. https://doi.org/10.1136/amiajnl-2012-001012.

Chicago

Deléger, Louise, K. Molnar, Guergana Savova, F. Xia, Todd Lingren, Qian Li, Keith A. Marsolo, et al. 2012. “Large-scale evaluation of automated clinical note de-identification and its impact on information extraction.” Journal of the American Medical Informatics Association 20 (1): 84–94. https://doi.org/10.1136/amiajnl-2012-001012.

Harvard

Deléger, L. et al. (2012) ‘Large-scale evaluation of automated clinical note de-identification and its impact on information extraction’, Journal of the American Medical Informatics Association, 20(1), pp. 84–94. doi:10.1136/amiajnl-2012-001012.

IEEE

L. Deléger et al., “Large-scale evaluation of automated clinical note de-identification and its impact on information extraction,” Journal of the American Medical Informatics Association, vol. 20, no. 1, pp. 84–94, 2012, doi: 10.1136/amiajnl-2012-001012.

BibTeX

@article{deleger2012large,
  title     = {{Large-scale evaluation of automated clinical note de-identification and its impact on information extraction}},
  author    = {Deléger, Louise and Molnar, K. and Savova, Guergana and Xia, F. and Lingren, Todd and Li, Qian and Marsolo, Keith A. and Jegga, Anil Goud and Kaiser, Megan and Stoutenborough, Laura and Solti, Imre},
  journal   = {Journal of the American Medical Informatics Association},
  year      = {2012},
  volume    = {20},
  number    = {1},
  pages     = {84--94},
  doi       = {10.1136/amiajnl-2012-001012},
  url       = {https://doi.org/10.1136/amiajnl-2012-001012}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Citations by year

YearCitationsShare of the best year
20171212
201866
201988
202044
202144
202299
202388
202488
202544
202655

Its field-weighted citation impact is 4.01: it has been cited 4.01 times as often as the average work of the same field, type and year (1.00 is the world average).

Topic and keywords

Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies

  • natural language processing
  • information extraction
  • protected health information
  • HIPAA

Work with this paper

Most-cited papers in Medicine

All highly cited Medicine papers

Questions about this paper

How do I cite this paper in APA style?

Deléger, L., Molnar, K., Savova, G., Xia, F., Lingren, T., Li, Q., Marsolo, K. A., Jegga, A. G., Kaiser, M., Stoutenborough, L., & Solti, I. (2012). Large-scale evaluation of automated clinical note de-identification and its impact on information extraction. Journal of the American Medical Informatics Association, 20(1), 84–94. https://doi.org/10.1136/amiajnl-2012-001012

How many times has this paper been cited?

111 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at Journal of the American Medical Informatics Association: sign in and open the PDF from this page.

Who wrote this paper?

Louise Deléger, K. Molnar, Guergana Savova, F. Xia, Todd Lingren, Qian Li, Keith A. Marsolo, Anil Goud Jegga, Megan Kaiser, Laura Stoutenborough, Imre Solti.

What is the DOI of this paper?

10.1136/amiajnl-2012-001012, permanently at https://doi.org/10.1136/amiajnl-2012-001012. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.