Journal article · 2026 · Open access
ASQ-PHI: An adversarial synthetic data benchmark for clinical de-identification and search utility
Data in Brief, 65, 112586 · Published 11 February 2026 · DOI 10.1016/j.dib.2026.112586
In brief
“ASQ-PHI: An adversarial synthetic data benchmark for clinical de-identification and search utility” is a 2026 journal article by James Weatherhead and 2 co-authors, published in Data in Brief. OpenAlex counts 4 citations as of 19 November 2026. It is open access under a CC BY licence: the full text is free to read at Data in Brief.
- Citations
- 4
- Most cited in
- 2026 (4)
- References
- 11
- Authors
- 3
Abstract
Hospitals and vendors now run HIPAA-compliant Business Associate Agreement (BAA) large language models (LLMs) for clinical work. These systems do not use input data for further training, so clinicians can enter Protected Health Information (PHI) into them. LLMs are trained on a fixed corpus with a historical cutoff, therefore their answers often need to be supplemented with more recent clinical evidence from external sources such as live web search or other tools that are often not covered by a BAA. This creates a "safe handoff" point where a clinician's PHI-containing query must be transformed into a HIPAA Safe Harbor compliant version before leaving the protected environment. However, publicly shareable datasets for this setting are scarce; this article describes PHI-rich clinician-style questions paired with HIPAA Safe Harbor annotations at the point where an external tool is called. Existing de-identification benchmarks are typically built from long electronic health record narratives such as discharge summaries and clinic notes, rather than from short, compressed search-style queries such as those that might be used in chat-based clinical LLM interfaces. ASQ-PHI (Adversarial Synthetic Queries for Protected Health Information de-identification) is a fully synthetic benchmark dataset designed for this safe handoff setting; no real patient data, electronic health records, or protected health information were accessed, used, or referenced during dataset creation. It contains 1051 single-turn clinical search queries that are designed to resemble prompts that clinicians might enter into HIPAA-compliant LLMs. Each record uses machine-parsable delimiters to separate the free text query from PHI annotations, which are provided as one JSON object per element specifying the HIPAA Safe Harbor identifier category and exact string value. The corpus includes 832 PHI-positive queries (79.2%) and 219 hard negatives (20.8%) engineered to mimic PHI-like syntax while containing only non-identifying clinical information such as ages under 90 years, diagnoses, medications, and symptoms. Across the dataset, there are 2973 PHI elements labeled from 13 textual HIPAA Safe Harbor identifier types that can be represented as short alphanumeric strings in single-line clinical questions, supporting the measurement of both PHI removal and over-redaction on PHI-free queries. All queries were generated with an adversarial few-shot prompting pipeline using Azure OpenAI GPT-4o. The associated Mendeley Data repository provides the complete dataset file, a Jupyter notebook that implements the generation pipeline, summary statistics, baseline metrics for a commercial PHI detection service, and six figures that describe the dataset. ASQ-PHI is released under an MIT license.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Weatherhead, J., Golovko, G., & McCaffrey, P. (2026). ASQ-PHI: An adversarial synthetic data benchmark for clinical de-identification and search utility. Data in Brief, 65, 112586. https://doi.org/10.1016/j.dib.2026.112586
MLA 9
Weatherhead, James, et al. “ASQ-PHI: An adversarial synthetic data benchmark for clinical de-identification and search utility.” Data in Brief, vol. 65, 2026, p. 112586. https://doi.org/10.1016/j.dib.2026.112586.
Chicago
Weatherhead, James, George Golovko, and Peter McCaffrey. 2026. “ASQ-PHI: An adversarial synthetic data benchmark for clinical de-identification and search utility.” Data in Brief 65: 112586. https://doi.org/10.1016/j.dib.2026.112586.
Harvard
Weatherhead, J., Golovko, G. and McCaffrey, P. (2026) ‘ASQ-PHI: An adversarial synthetic data benchmark for clinical de-identification and search utility’, Data in Brief, 65, p. 112586. doi:10.1016/j.dib.2026.112586.
IEEE
J. Weatherhead, G. Golovko, and P. McCaffrey, “ASQ-PHI: An adversarial synthetic data benchmark for clinical de-identification and search utility,” Data in Brief, vol. 65, p. 112586, 2026, doi: 10.1016/j.dib.2026.112586.
BibTeX
@article{weatherhead2026adversarial,
title = {{ASQ-PHI: An adversarial synthetic data benchmark for clinical de-identification and search utility}},
author = {Weatherhead, James and Golovko, George and McCaffrey, Peter},
journal = {Data in Brief},
year = {2026},
volume = {65},
pages = {112586},
doi = {10.1016/j.dib.2026.112586},
url = {https://doi.org/10.1016/j.dib.2026.112586}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- protected health information
- synthetic benchmark dataset
- adversarial prompting
- large language models
- business associate agreements
- hard negative examples
- GPT-4o
- data anonymization
Work with this paper
Most-cited papers in Computer Science
- ImageNet classification with deep convolutional neural networksAlex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton · Communications of the ACM · 2017 · 114,726 citations
- Long Short-Term MemorySepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 109,097 citations
- A Mathematical Theory of CommunicationClaude E. Shannon · Bell System Technical Journal · 1948 · 91,736 citations
- Deep learningYann LeCun, Yoshua Bengio, Geoffrey E. Hinton · Nature · 2015 · 90,226 citations
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksShaoqing Ren, Kaiming He, Ross Girshick and 1 more · IEEE Transactions on Pattern Analysis and Machine Intelligence · 2016 · 74,143 citations
Questions about this paper
How do I cite this paper in APA style?
Weatherhead, J., Golovko, G., & McCaffrey, P. (2026). ASQ-PHI: An adversarial synthetic data benchmark for clinical de-identification and search utility. Data in Brief, 65, 112586. https://doi.org/10.1016/j.dib.2026.112586
How many times has this paper been cited?
4 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at Data in Brief: sign in and open the full text from this page.
Who wrote this paper?
James Weatherhead, George Golovko, Peter McCaffrey.
What is the DOI of this paper?
10.1016/j.dib.2026.112586, permanently at https://doi.org/10.1016/j.dib.2026.112586. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.