Preprint · 2024 · Open access

A Survey on Evaluating Large Language Models in Code Generation Tasks

Liguo Chen, Qi Guo, Hongrui Jia, Zhengran Zeng, Xin Wang, Yijiang Xu, Wu, Jian, Yidong Wang, Qing Zhong Gao, Jindong Wang, Wei Ye, Shikun Zhang

arXiv (Cornell University) · Published 29 August 2024 · DOI 10.48550/arxiv.2408.16498

Open PDFView at the publisherCite this paper

In brief

“A Survey on Evaluating Large Language Models in Code Generation Tasks” is a 2024 preprint by Liguo Chen and 11 co-authors, published in arXiv (Cornell University). OpenAlex counts 147 citations as of 19 November 2026. It is open access under a CC BY licence: the full text is free to read at arXiv (Cornell University).

Citations
147
Most cited in
2025 (78)
References
43
Authors
12

Abstract

This paper provides a comprehensive review of the current methods and metrics used to evaluate the performance of Large Language Models (LLMs) in code generation tasks. With the rapid growth in demand for automated software development, LLMs have demonstrated significant potential in the field of code generation. The paper begins by reviewing the historical development of LLMs and their applications in code generation. Next, it details various methods and metrics for assessing the code generation capabilities of LLMs, including code correctness, efficiency, readability, and evaluation methods based on expert review and user experience. The paper also evaluates the widely used benchmark datasets, identifying their limitations and proposing directions for future improvements. Specifically, the paper analyzes the performance of code generation models across different tasks by combining multiple evaluation metrics, such as code compilation/interpretation success rates, unit test pass rates, and performance and efficiency metrics, to comprehensively assess the practical application of LLMs in code generation. Finally, the paper discusses the challenges faced in evaluating LLMs in code generation, particularly how to ensure the comprehensiveness and accuracy of evaluation methods and how to adapt to the evolving practices of software development. These analyses and discussions provide valuable insights for further optimizing and improving the application of LLMs in code generation tasks.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Chen, L., Guo, Q., Jia, H., Zeng, Z., Wang, X., Xu, Y., Jian, W., Wang, Y., Gao, Q. Z., Wang, J., Ye, W., & Zhang, S. (2024). A Survey on Evaluating Large Language Models in Code Generation Tasks. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2408.16498

MLA 9

Chen, Liguo, et al. “A Survey on Evaluating Large Language Models in Code Generation Tasks.” arXiv (Cornell University), 2024. https://doi.org/10.48550/arxiv.2408.16498.

Chicago

Chen, Liguo, Qi Guo, Hongrui Jia, Zhengran Zeng, Xin Wang, Yijiang Xu, Wu, Jian, et al. 2024. “A Survey on Evaluating Large Language Models in Code Generation Tasks.” arXiv (Cornell University). https://doi.org/10.48550/arxiv.2408.16498.

Harvard

Chen, L. et al. (2024) ‘A Survey on Evaluating Large Language Models in Code Generation Tasks’, arXiv (Cornell University). doi:10.48550/arxiv.2408.16498.

IEEE

L. Chen et al., “A Survey on Evaluating Large Language Models in Code Generation Tasks,” arXiv (Cornell University), 2024, doi: 10.48550/arxiv.2408.16498.

BibTeX

@misc{chen2024survey,
  title     = {{A Survey on Evaluating Large Language Models in Code Generation Tasks}},
  author    = {Chen, Liguo and Guo, Qi and Jia, Hongrui and Zeng, Zhengran and Wang, Xin and Xu, Yijiang and Jian, Wu, and Wang, Yidong and Gao, Qing Zhong and Wang, Jindong and Ye, Wei and Zhang, Shikun},
  howpublished = {arXiv (Cornell University)},
  year      = {2024},
  doi       = {10.48550/arxiv.2408.16498},
  url       = {https://doi.org/10.48550/arxiv.2408.16498}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Citations by year

YearCitationsShare of the best year
202011
202444
20257878
20266464

Topic and keywords

Physical Sciences › Computer Science › Information Systems › Software Engineering Research

  • large language models
  • code generation
  • evaluation metrics
  • program correctness
  • code readability
  • benchmark dataset
  • software development automation
  • expert review

Work with this paper

Most-cited papers in Computer Science

All highly cited Computer Science papers

Questions about this paper

How do I cite this paper in APA style?

Chen, L., Guo, Q., Jia, H., Zeng, Z., Wang, X., Xu, Y., Jian, W., Wang, Y., Gao, Q. Z., Wang, J., Ye, W., & Zhang, S. (2024). A Survey on Evaluating Large Language Models in Code Generation Tasks. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2408.16498

How many times has this paper been cited?

147 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at arXiv (Cornell University): sign in and open the PDF from this page.

Who wrote this paper?

Liguo Chen, Qi Guo, Hongrui Jia, Zhengran Zeng, Xin Wang, Yijiang Xu, Wu, Jian, Yidong Wang, Qing Zhong Gao, Jindong Wang, Wei Ye, Shikun Zhang.

What is the DOI of this paper?

10.48550/arxiv.2408.16498, permanently at https://doi.org/10.48550/arxiv.2408.16498. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.