Journal article · 2026 · Open access
When large language models are reliable for judging empathic communication
Nature Machine Intelligence, 8(2), 173–185 · Published 11 February 2026 · DOI 10.1038/s42256-025-01169-6
In brief
“When large language models are reliable for judging empathic communication” is a 2026 journal article by Aakriti Kumar and 5 co-authors, published in Nature Machine Intelligence. OpenAlex counts 24 citations as of 16 November 2026, which puts it in the top 1% of works published in 2026 by citations. It is open access under a CC BY licence: the full text is free to read at Nature Machine Intelligence.
- Citations
- 24
- Most cited in
- 2026 (24)
- Field-weighted impact
- 33.53
- References
- 76
- Authors
- 6
Abstract
Large language models (LLMs) excel at generating empathic responses in text-based conversations. But, how reliably do they judge the nuances of empathic communication? Here we investigate this question by comparing how experts, crowdworkers and LLMs annotate empathic communication across four evaluative frameworks drawn from psychology, natural language processing and communications applied to 200 real-world conversations where one speaker shares a personal problem and the other offers support. Drawing on 3,150 expert annotations, 2,844 crowd annotations and 3,150 LLM annotations, we assess interrater reliability between these three annotator groups. We find that expert agreement is high but varies across the frameworks’ subcomponents depending on their clarity, complexity and subjectivity. We show that expert agreement offers a more informative benchmark for contextualizing LLM performance than standard classification metrics. Across all four frameworks, LLMs consistently approach this expert level benchmark and exceed the reliability of crowdworkers. These results demonstrate how LLMs, when validated on specific tasks with appropriate benchmarks, can support transparency and oversight in emotionally sensitive applications including their use as conversational companions.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Kumar, A., Poungpeth, N., Yang, D., Farrell, E. L., Lambert, B. L., & Groh, M. (2026). When large language models are reliable for judging empathic communication. Nature Machine Intelligence, 8(2), 173–185. https://doi.org/10.1038/s42256-025-01169-6
MLA 9
Kumar, Aakriti, et al. “When large language models are reliable for judging empathic communication.” Nature Machine Intelligence, vol. 8, no. 2, 2026, pp. 173–185. https://doi.org/10.1038/s42256-025-01169-6.
Chicago
Kumar, Aakriti, Nalin Poungpeth, Diyi Yang, Erina L. Farrell, Bruce L Lambert, and Matthew Groh. 2026. “When large language models are reliable for judging empathic communication.” Nature Machine Intelligence 8 (2): 173–185. https://doi.org/10.1038/s42256-025-01169-6.
Harvard
Kumar, A. et al. (2026) ‘When large language models are reliable for judging empathic communication’, Nature Machine Intelligence, 8(2), pp. 173–185. doi:10.1038/s42256-025-01169-6.
IEEE
A. Kumar, N. Poungpeth, D. Yang, E. L. Farrell, B. L. Lambert, and M. Groh, “When large language models are reliable for judging empathic communication,” Nature Machine Intelligence, vol. 8, no. 2, pp. 173–185, 2026, doi: 10.1038/s42256-025-01169-6.
BibTeX
@article{kumar2026when,
title = {{When large language models are reliable for judging empathic communication}},
author = {Kumar, Aakriti and Poungpeth, Nalin and Yang, Diyi and Farrell, Erina L. and Lambert, Bruce L and Groh, Matthew},
journal = {Nature Machine Intelligence},
year = {2026},
volume = {8},
number = {2},
pages = {173--185},
doi = {10.1038/s42256-025-01169-6},
url = {https://doi.org/10.1038/s42256-025-01169-6}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- large language models
- empathic communication
- inter-rater reliability
- expert annotation
- crowd annotation
- LLM annotation
- emotional support
Work with this paper
Most-cited papers in Computer Science
- ImageNet classification with deep convolutional neural networksAlex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton · Communications of the ACM · 2017 · 114,726 citations
- Long Short-Term MemorySepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 109,097 citations
- A Mathematical Theory of CommunicationClaude E. Shannon · Bell System Technical Journal · 1948 · 91,736 citations
- Deep learningYann LeCun, Yoshua Bengio, Geoffrey E. Hinton · Nature · 2015 · 90,226 citations
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksShaoqing Ren, Kaiming He, Ross Girshick and 1 more · IEEE Transactions on Pattern Analysis and Machine Intelligence · 2016 · 74,143 citations
Questions about this paper
How do I cite this paper in APA style?
Kumar, A., Poungpeth, N., Yang, D., Farrell, E. L., Lambert, B. L., & Groh, M. (2026). When large language models are reliable for judging empathic communication. Nature Machine Intelligence, 8(2), 173–185. https://doi.org/10.1038/s42256-025-01169-6
How many times has this paper been cited?
24 times, according to OpenAlex on 16 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at Nature Machine Intelligence: sign in and open the PDF from this page.
Who wrote this paper?
Aakriti Kumar, Nalin Poungpeth, Diyi Yang, Erina L. Farrell, Bruce L Lambert, Matthew Groh.
What is the DOI of this paper?
10.1038/s42256-025-01169-6, permanently at https://doi.org/10.1038/s42256-025-01169-6. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 16 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.