Conference paper · 2023 · Open access
Understanding Social Reasoning in Language Models with Language Models
neural information processing systems, 13518–13529 · Published 1 January 2023 · DOI 10.52202/075280-0595
In brief
“Understanding Social Reasoning in Language Models with Language Models” is a 2023 conference paper by Kanishk Gandhi and 3 co-authors, published in neural information processing systems. OpenAlex counts 112 citations as of 16 November 2026. It is open access under a CC BY licence: the full text is free to read at arXiv (Cornell University).
- Citations
- 112
- Most cited in
- 2024 (36)
- Field-weighted impact
- 84.71
- References
- 20
- Authors
- 4
Abstract
As Large Language Models (LLMs) become increasingly integrated into our everyday lives, understanding their ability to comprehend human mental states becomes critical for ensuring effective interactions. However, despite the recent attempts to assess the Theory-of-Mind (ToM) reasoning capabilities of LLMs, the degree to which these models can align with human ToM remains a nuanced topic of exploration. This is primarily due to two distinct challenges: (1) the presence of inconsistent results from previous evaluations, and (2) concerns surrounding the validity of existing evaluation methodologies. To address these challenges, we present a novel framework for procedurally generating evaluations with LLMs by populating causal templates. Using our framework, we create a new social reasoning benchmark (BigToM) for LLMs which consists of 25 controls and 5,000 model-written evaluations. We find that human participants rate the quality of our benchmark higher than previous crowd-sourced evaluations and comparable to expert-written evaluations. Using BigToM, we evaluate the social reasoning capabilities of a variety of LLMs and compare model performances with human performance. Our results suggest that GPT4 has ToM capabilities that mirror human inference patterns, though less reliable, while other LLMs struggle.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Gandhi, K., Fränken, J.-P., Gerstenberg, T., & Goodman, N. D. (2023). Understanding Social Reasoning in Language Models with Language Models. neural information processing systems, 13518–13529. https://doi.org/10.52202/075280-0595
MLA 9
Gandhi, Kanishk, et al. “Understanding Social Reasoning in Language Models with Language Models.” neural information processing systems, 2023, pp. 13518–13529. https://doi.org/10.52202/075280-0595.
Chicago
Gandhi, Kanishk, Jan-Philipp Fränken, Tobias Gerstenberg, and Noah D. Goodman. 2023. “Understanding Social Reasoning in Language Models with Language Models.” neural information processing systems: 13518–13529. https://doi.org/10.52202/075280-0595.
Harvard
Gandhi, K. et al. (2023) ‘Understanding Social Reasoning in Language Models with Language Models’, neural information processing systems, pp. 13518–13529. doi:10.52202/075280-0595.
IEEE
K. Gandhi, J.-P. Fränken, T. Gerstenberg, and N. D. Goodman, “Understanding Social Reasoning in Language Models with Language Models,” neural information processing systems, pp. 13518–13529, 2023, doi: 10.52202/075280-0595.
BibTeX
@inproceedings{gandhi2023understanding,
title = {{Understanding Social Reasoning in Language Models with Language Models}},
author = {Gandhi, Kanishk and Fränken, Jan-Philipp and Gerstenberg, Tobias and Goodman, Noah D.},
booktitle = {neural information processing systems},
year = {2023},
pages = {13518--13529},
doi = {10.52202/075280-0595},
url = {https://doi.org/10.52202/075280-0595}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Citations by year
| Year | Citations | Share of the best year |
|---|---|---|
| 2021 | 1 | |
| 2023 | 6 | |
| 2024 | 36 | |
| 2025 | 35 | |
| 2026 | 34 |
Its field-weighted citation impact is 84.71: it has been cited 84.71 times as often as the average work of the same field, type and year (1.00 is the world average).
Topic and keywords
Social Sciences › Psychology › Developmental and Educational Psychology › Child and Animal Learning Development
- theory of mind
- social reasoning
- large language models
- automated assessment
- benchmark generation
- GPT-4
- human-LLM comparison
Work with this paper
Most-cited papers in Psychology
- Diagnostic and Statistical Manual of Mental DisordersAndrés Santiago González Moreno · American Psychiatric Association Publishing eBooks · 2013 · 123,932 citations
- The theory of planned behaviorIcek Ajzen · Organizational Behavior and Human Decision Processes · 1991 · 106,482 citations
- Self-efficacy: Toward a unifying theory of behavioral change.Albert Bandura · Psychological Review · 1977 · 61,923 citations
- Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being.Richard M. Ryan, Edward L. Deci · American Psychologist · 2000 · 55,457 citations
- Mind in Society: The Development of Higher Psychological ProcessesL. S. Vygotsky · 기초학문자료센터(KRM) (Korea Research Foundation) · 1978 · 53,340 citations
Questions about this paper
How do I cite this paper in APA style?
Gandhi, K., Fränken, J.-P., Gerstenberg, T., & Goodman, N. D. (2023). Understanding Social Reasoning in Language Models with Language Models. neural information processing systems, 13518–13529. https://doi.org/10.52202/075280-0595
How many times has this paper been cited?
112 times, according to OpenAlex on 16 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access: read the free full text at arXiv (Cornell University).
Who wrote this paper?
Kanishk Gandhi, Jan-Philipp Fränken, Tobias Gerstenberg, Noah D. Goodman.
What is the DOI of this paper?
10.52202/075280-0595. The permanent link is https://doi.org/10.52202/075280-0595.
Bibliographic data from OpenAlex (CC0), last checked 16 November 2026. ResearchGuru does not host this paper; the links go to the publisher or the repository that does.