Journal article · 2024 · Open access
A Survey on Evaluation of Large Language Models
ACM Transactions on Intelligent Systems and Technology, 15(3), 1–45 · Published 23 January 2024 · DOI 10.1145/3641289
In brief
“A Survey on Evaluation of Large Language Models” is a 2024 journal article by Yupeng Chang and 15 co-authors, published in ACM Transactions on Intelligent Systems and Technology. OpenAlex counts 4,362 citations as of 15 November 2026. It is open access under a CC BY licence: the full text is free to read at arXiv (Cornell University).
- Citations
- 4,362
- Most cited in
- 2025 (1,840)
- Field-weighted impact
- 440.35
- References
- 264
- Authors
- 16
Abstract
Large language models (LLMs) are gaining increasing popularity in both academia and industry, owing to their unprecedented performance in various applications. As LLMs continue to play a vital role in both research and daily use, their evaluation becomes increasingly critical, not only at the task level, but also at the society level for better understanding of their potential risks. Over the past years, significant efforts have been made to examine LLMs from various perspectives. This paper presents a comprehensive review of these evaluation methods for LLMs, focusing on three key dimensions: what to evaluate , where to evaluate , and how to evaluate . Firstly, we provide an overview from the perspective of evaluation tasks, encompassing general natural language processing tasks, reasoning, medical usage, ethics, education, natural and social sciences, agent applications, and other areas. Secondly, we answer the ‘where’ and ‘how’ questions by diving into the evaluation methods and benchmarks, which serve as crucial components in assessing the performance of LLMs. Then, we summarize the success and failure cases of LLMs in different tasks. Finally, we shed light on several future challenges that lie ahead in LLMs evaluation. Our aim is to offer invaluable insights to researchers in the realm of LLMs evaluation, thereby aiding the development of more proficient LLMs. Our key point is that evaluation should be treated as an essential discipline to better assist the development of LLMs. We consistently maintain the related open-source materials at: https://github.com/MLGroupJLU/LLM-eval-survey
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., Ye, W., Zhang, Y., Chang, Y., Yu, P. S., Yang, Q., & Xie, X. (2024). A Survey on Evaluation of Large Language Models. ACM Transactions on Intelligent Systems and Technology, 15(3), 1–45. https://doi.org/10.1145/3641289
MLA 9
Chang, Yupeng, et al. “A Survey on Evaluation of Large Language Models.” ACM Transactions on Intelligent Systems and Technology, vol. 15, no. 3, 2024, pp. 1–45. https://doi.org/10.1145/3641289.
Chicago
Chang, Yupeng, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, et al. 2024. “A Survey on Evaluation of Large Language Models.” ACM Transactions on Intelligent Systems and Technology 15 (3): 1–45. https://doi.org/10.1145/3641289.
Harvard
Chang, Y. et al. (2024) ‘A Survey on Evaluation of Large Language Models’, ACM Transactions on Intelligent Systems and Technology, 15(3), pp. 1–45. doi:10.1145/3641289.
IEEE
Y. Chang et al., “A Survey on Evaluation of Large Language Models,” ACM Transactions on Intelligent Systems and Technology, vol. 15, no. 3, pp. 1–45, 2024, doi: 10.1145/3641289.
BibTeX
@article{chang2024survey,
title = {{A Survey on Evaluation of Large Language Models}},
author = {Chang, Yupeng and Wang, Xu and Wang, Jindong and Wu, Yuan and Yang, Linyi and Zhu, Kaijie and Chen, Hao and Yi, Xiaoyuan and Wang, Cunxiang and Wang, Yidong and Ye, Wei and Zhang, Yue and Chang, Yi and Yu, Philip S. and Yang, Qiang and Xie, Xing},
journal = {ACM Transactions on Intelligent Systems and Technology},
year = {2024},
volume = {15},
number = {3},
pages = {1--45},
doi = {10.1145/3641289},
url = {https://doi.org/10.1145/3641289}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Citations by year
| Year | Citations | Share of the best year |
|---|---|---|
| 2021 | 1 | |
| 2022 | 2 | |
| 2023 | 124 | |
| 2024 | 1,028 | |
| 2025 | 1,840 | |
| 2026 | 1,367 |
Its field-weighted citation impact is 440.35: it has been cited 440.35 times as often as the average work of the same field, type and year (1.00 is the world average).
Topic and keywords
Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- large language models
- LLM evaluation
- evaluation benchmark
- assessment tasks
- reasoning
- medical applications
- ethics
- AI agents
Work with this paper
Most-cited papers in Computer Science
- ImageNet classification with deep convolutional neural networksAlex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton · Communications of the ACM · 2017 · 114,726 citations
- Long Short-Term MemorySepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 109,097 citations
- A Mathematical Theory of CommunicationClaude E. Shannon · Bell System Technical Journal · 1948 · 91,736 citations
- Deep learningYann LeCun, Yoshua Bengio, Geoffrey E. Hinton · Nature · 2015 · 90,226 citations
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksShaoqing Ren, Kaiming He, Ross Girshick and 1 more · IEEE Transactions on Pattern Analysis and Machine Intelligence · 2016 · 74,143 citations
Questions about this paper
How do I cite this paper in APA style?
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., Ye, W., Zhang, Y., Chang, Y., Yu, P. S., Yang, Q., & Xie, X. (2024). A Survey on Evaluation of Large Language Models. ACM Transactions on Intelligent Systems and Technology, 15(3), 1–45. https://doi.org/10.1145/3641289
How many times has this paper been cited?
4,362 times, according to OpenAlex on 15 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access: read the free full text at arXiv (Cornell University).
Who wrote this paper?
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang and 4 more.
What is the DOI of this paper?
10.1145/3641289. The permanent link is https://doi.org/10.1145/3641289.
Bibliographic data from OpenAlex (CC0), last checked 15 November 2026. ResearchGuru does not host this paper; the links go to the publisher or the repository that does.