Preprint · 2025 · Open access

MAGPIE: A dataset for Multi-AGent contextual PrIvacy Evaluation

Gurusha Juneja, Alon Albalak, Wenyue Hua, William Yang Wang

arXiv (Cornell University) · Published 25 June 2025 · DOI 10.48550/arxiv.2506.20737

Open PDFView at the publisherCite this paper

In brief

“MAGPIE: A dataset for Multi-AGent contextual PrIvacy Evaluation” is a 2025 preprint by Gurusha Juneja and 3 co-authors, published in arXiv (Cornell University). OpenAlex counts 10 citations as of 19 November 2026. It is open access under a CC BY licence: the full text is free to read at arXiv (Cornell University).

Citations
10
Most cited in
2026 (7)
References
26
Authors
4

Abstract

The proliferation of LLM-based agents has led to increasing deployment of inter-agent collaboration for tasks like scheduling, negotiation, resource allocation etc. In such systems, privacy is critical, as agents often access proprietary tools and domain-specific databases requiring strict confidentiality. This paper examines whether LLM-based agents demonstrate an understanding of contextual privacy. And, if instructed, do these systems preserve inference time user privacy in non-adversarial multi-turn conversation. Existing benchmarks to evaluate contextual privacy in LLM-agents primarily assess single-turn, low-complexity tasks where private information can be easily excluded. We first present a benchmark - MAGPIE comprising 158 real-life high-stakes scenarios across 15 domains. These scenarios are designed such that complete exclusion of private data impedes task completion yet unrestricted information sharing could lead to substantial losses. We then evaluate the current state-of-the-art LLMs on (a) their understanding of contextually private data and (b) their ability to collaborate without violating user privacy. Empirical experiments demonstrate that current models, including GPT-4o and Claude-2.7-Sonnet, lack robust understanding of contextual privacy, misclassifying private data as shareable 25.2\% and 43.6\% of the time. In multi-turn conversations, these models disclose private information in 59.9\% and 50.5\% of cases even under explicit privacy instructions. Furthermore, multi-agent systems fail to complete tasks in 71\% of scenarios. These results underscore that current models are not aligned towards both contextual privacy preservation and collaborative task-solving.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Juneja, G., Albalak, A., Hua, W., & Wang, W. Y. (2025). MAGPIE: A dataset for Multi-AGent contextual PrIvacy Evaluation. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2506.20737

MLA 9

Juneja, Gurusha, et al. “MAGPIE: A dataset for Multi-AGent contextual PrIvacy Evaluation.” arXiv (Cornell University), 2025. https://doi.org/10.48550/arxiv.2506.20737.

Chicago

Juneja, Gurusha, Alon Albalak, Wenyue Hua, and William Yang Wang. 2025. “MAGPIE: A dataset for Multi-AGent contextual PrIvacy Evaluation.” arXiv (Cornell University). https://doi.org/10.48550/arxiv.2506.20737.

Harvard

Juneja, G. et al. (2025) ‘MAGPIE: A dataset for Multi-AGent contextual PrIvacy Evaluation’, arXiv (Cornell University). doi:10.48550/arxiv.2506.20737.

IEEE

G. Juneja, A. Albalak, W. Hua, and W. Y. Wang, “MAGPIE: A dataset for Multi-AGent contextual PrIvacy Evaluation,” arXiv (Cornell University), 2025, doi: 10.48550/arxiv.2506.20737.

BibTeX

@misc{juneja2025magpie,
  title     = {{MAGPIE: A dataset for Multi-AGent contextual PrIvacy Evaluation}},
  author    = {Juneja, Gurusha and Albalak, Alon and Hua, Wenyue and Wang, William Yang},
  howpublished = {arXiv (Cornell University)},
  year      = {2025},
  doi       = {10.48550/arxiv.2506.20737},
  url       = {https://doi.org/10.48550/arxiv.2506.20737}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data

  • contextual privacy
  • LLM agents
  • multi-agent systems
  • multi-turn dialogue
  • privacy preservation
  • large language models

Work with this paper

Most-cited papers in Computer Science

All highly cited Computer Science papers

Questions about this paper

How do I cite this paper in APA style?

Juneja, G., Albalak, A., Hua, W., & Wang, W. Y. (2025). MAGPIE: A dataset for Multi-AGent contextual PrIvacy Evaluation. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2506.20737

How many times has this paper been cited?

10 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at arXiv (Cornell University): sign in and open the PDF from this page.

Who wrote this paper?

Gurusha Juneja, Alon Albalak, Wenyue Hua, William Yang Wang.

What is the DOI of this paper?

10.48550/arxiv.2506.20737, permanently at https://doi.org/10.48550/arxiv.2506.20737. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.