Preprint · 2026 · Open access

GroupMemBench: Benchmarking LLM Agent Memory in Multi-Party Conversations

Jingbo Yang, Kwei-Herng Lai, Xiaowen Wang, Shiyu Chang, Yaar Harari, Evgeniy Gabrilovich

arXiv (Cornell University) · Published 14 May 2026 · DOI 10.48550/arxiv.2605.14498

Open PDFView at the publisherCite this paper

In brief

“GroupMemBench: Benchmarking LLM Agent Memory in Multi-Party Conversations” is a 2026 preprint by Jingbo Yang and 5 co-authors, published in arXiv (Cornell University). OpenAlex counts 24 citations as of 19 November 2026. It is open access under a CC BY licence: the full text is free to read at arXiv (Cornell University).

Citations
24
Most cited in
2026 (24)
References
32
Authors
6

Abstract

Large Language Model (LLM) agents increasingly serve as personal assistants and workplace collaborators, where their utility depends on memory systems that extract, retrieve, and apply information across long-running conversations. However, both existing memory systems and benchmarks are built around the dyadic, single-user setup, even though real deployments routinely span groups and channels with multiple users interacting with the agent and with each other. This mismatch leaves three properties of group memory unmeasured: (i) group dynamics that go beyond concatenated one-on-one chats, (ii) speaker-grounded belief tracking, where the per-user memory modeling is needed, and (iii) audience-adapted language, where Theory-of-Mind shifts produce role-specific vocabulary. We introduce GroupMemBench, a benchmark that exposes all three. A graph-grounded synthesis pipeline produces multi-party conversations with controllable reply structure and conditions each message on per-user personas and target audiences. An adversarial query pipeline then binds every question to a specific asker across six categories, spanning multi-hop reasoning, knowledge update, term ambiguity, user-implicit reasoning, temporal reasoning, and abstention, and iteratively searches challenging, realistic queries that reflect comprehensive memory capability. Benchmarking leading memory systems exposes a sharp collapse: the strongest one reaches only 46.0% average accuracy, with knowledge update at 27.1% and term ambiguity at 37.7%, while a simple BM25 baseline matches or exceeds most agent memory systems. This indicates current memory ingestion erases the structural and lexical features group memory depends on, leaving multi-user memory far from solved.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Yang, J., Lai, K.-H., Wang, X., Chang, S., Harari, Y., & Gabrilovich, E. (2026). GroupMemBench: Benchmarking LLM Agent Memory in Multi-Party Conversations. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2605.14498

MLA 9

Yang, Jingbo, et al. “GroupMemBench: Benchmarking LLM Agent Memory in Multi-Party Conversations.” arXiv (Cornell University), 2026. https://doi.org/10.48550/arxiv.2605.14498.

Chicago

Yang, Jingbo, Kwei-Herng Lai, Xiaowen Wang, Shiyu Chang, Yaar Harari, and Evgeniy Gabrilovich. 2026. “GroupMemBench: Benchmarking LLM Agent Memory in Multi-Party Conversations.” arXiv (Cornell University). https://doi.org/10.48550/arxiv.2605.14498.

Harvard

Yang, J. et al. (2026) ‘GroupMemBench: Benchmarking LLM Agent Memory in Multi-Party Conversations’, arXiv (Cornell University). doi:10.48550/arxiv.2605.14498.

IEEE

J. Yang, K.-H. Lai, X. Wang, S. Chang, Y. Harari, and E. Gabrilovich, “GroupMemBench: Benchmarking LLM Agent Memory in Multi-Party Conversations,” arXiv (Cornell University), 2026, doi: 10.48550/arxiv.2605.14498.

BibTeX

@misc{yang2026groupmembench,
  title     = {{GroupMemBench: Benchmarking LLM Agent Memory in Multi-Party Conversations}},
  author    = {Yang, Jingbo and Lai, Kwei-Herng and Wang, Xiaowen and Chang, Shiyu and Harari, Yaar and Gabrilovich, Evgeniy},
  howpublished = {arXiv (Cornell University)},
  year      = {2026},
  doi       = {10.48550/arxiv.2605.14498},
  url       = {https://doi.org/10.48550/arxiv.2605.14498}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems

  • group memory
  • multi-party conversation
  • benchmarking
  • belief tracking
  • theory of mind
  • personas
  • knowledge updating
  • term ambiguity

Work with this paper

Most-cited papers in Computer Science

All highly cited Computer Science papers

Questions about this paper

How do I cite this paper in APA style?

Yang, J., Lai, K.-H., Wang, X., Chang, S., Harari, Y., & Gabrilovich, E. (2026). GroupMemBench: Benchmarking LLM Agent Memory in Multi-Party Conversations. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2605.14498

How many times has this paper been cited?

24 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at arXiv (Cornell University): sign in and open the PDF from this page.

Who wrote this paper?

Jingbo Yang, Kwei-Herng Lai, Xiaowen Wang, Shiyu Chang, Yaar Harari, Evgeniy Gabrilovich.

What is the DOI of this paper?

10.48550/arxiv.2605.14498, permanently at https://doi.org/10.48550/arxiv.2605.14498. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.