Journal article · 2024 · Open access

A Survey on Evaluation of Large Language Models

Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang and 4 more

ACM Transactions on Intelligent Systems and Technology, 15(3), 1–45 · Published 23 January 2024 · DOI 10.1145/3641289

Free PDFView at the publisherCite this paper

In brief

“A Survey on Evaluation of Large Language Models” is a 2024 journal article by Yupeng Chang and 15 co-authors, published in ACM Transactions on Intelligent Systems and Technology. OpenAlex counts 4,362 citations as of 15 November 2026. It is open access under a CC BY licence: the full text is free to read at arXiv (Cornell University).

Citations
4,362
Most cited in
2025 (1,840)
Field-weighted impact
440.35
References
264
Authors
16

Abstract

Large language models (LLMs) are gaining increasing popularity in both academia and industry, owing to their unprecedented performance in various applications. As LLMs continue to play a vital role in both research and daily use, their evaluation becomes increasingly critical, not only at the task level, but also at the society level for better understanding of their potential risks. Over the past years, significant efforts have been made to examine LLMs from various perspectives. This paper presents a comprehensive review of these evaluation methods for LLMs, focusing on three key dimensions: what to evaluate , where to evaluate , and how to evaluate . Firstly, we provide an overview from the perspective of evaluation tasks, encompassing general natural language processing tasks, reasoning, medical usage, ethics, education, natural and social sciences, agent applications, and other areas. Secondly, we answer the ‘where’ and ‘how’ questions by diving into the evaluation methods and benchmarks, which serve as crucial components in assessing the performance of LLMs. Then, we summarize the success and failure cases of LLMs in different tasks. Finally, we shed light on several future challenges that lie ahead in LLMs evaluation. Our aim is to offer invaluable insights to researchers in the realm of LLMs evaluation, thereby aiding the development of more proficient LLMs. Our key point is that evaluation should be treated as an essential discipline to better assist the development of LLMs. We consistently maintain the related open-source materials at: https://github.com/MLGroupJLU/LLM-eval-survey

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., Ye, W., Zhang, Y., Chang, Y., Yu, P. S., Yang, Q., & Xie, X. (2024). A Survey on Evaluation of Large Language Models. ACM Transactions on Intelligent Systems and Technology, 15(3), 1–45. https://doi.org/10.1145/3641289

MLA 9

Chang, Yupeng, et al. “A Survey on Evaluation of Large Language Models.” ACM Transactions on Intelligent Systems and Technology, vol. 15, no. 3, 2024, pp. 1–45. https://doi.org/10.1145/3641289.

Chicago

Chang, Yupeng, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, et al. 2024. “A Survey on Evaluation of Large Language Models.” ACM Transactions on Intelligent Systems and Technology 15 (3): 1–45. https://doi.org/10.1145/3641289.

Harvard

Chang, Y. et al. (2024) ‘A Survey on Evaluation of Large Language Models’, ACM Transactions on Intelligent Systems and Technology, 15(3), pp. 1–45. doi:10.1145/3641289.

IEEE

Y. Chang et al., “A Survey on Evaluation of Large Language Models,” ACM Transactions on Intelligent Systems and Technology, vol. 15, no. 3, pp. 1–45, 2024, doi: 10.1145/3641289.

BibTeX

@article{chang2024survey,
  title     = {{A Survey on Evaluation of Large Language Models}},
  author    = {Chang, Yupeng and Wang, Xu and Wang, Jindong and Wu, Yuan and Yang, Linyi and Zhu, Kaijie and Chen, Hao and Yi, Xiaoyuan and Wang, Cunxiang and Wang, Yidong and Ye, Wei and Zhang, Yue and Chang, Yi and Yu, Philip S. and Yang, Qiang and Xie, Xing},
  journal   = {ACM Transactions on Intelligent Systems and Technology},
  year      = {2024},
  volume    = {15},
  number    = {3},
  pages     = {1--45},
  doi       = {10.1145/3641289},
  url       = {https://doi.org/10.1145/3641289}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Citations by year

YearCitationsShare of the best year
202111
202222
2023124124
20241,0281028
20251,8401840
20261,3671367

Its field-weighted citation impact is 440.35: it has been cited 440.35 times as often as the average work of the same field, type and year (1.00 is the world average).

Topic and keywords

Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling

  • large language models
  • LLM evaluation
  • evaluation benchmark
  • assessment tasks
  • reasoning
  • medical applications
  • ethics
  • AI agents

Work with this paper

Most-cited papers in Computer Science

All highly cited Computer Science papers

Questions about this paper

How do I cite this paper in APA style?

Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., Ye, W., Zhang, Y., Chang, Y., Yu, P. S., Yang, Q., & Xie, X. (2024). A Survey on Evaluation of Large Language Models. ACM Transactions on Intelligent Systems and Technology, 15(3), 1–45. https://doi.org/10.1145/3641289

How many times has this paper been cited?

4,362 times, according to OpenAlex on 15 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access: read the free full text at arXiv (Cornell University).

Who wrote this paper?

Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang and 4 more.

What is the DOI of this paper?

10.1145/3641289. The permanent link is https://doi.org/10.1145/3641289.

Bibliographic data from OpenAlex (CC0), last checked 15 November 2026. ResearchGuru does not host this paper; the links go to the publisher or the repository that does.