Preprint · 2024 · Open access
A Comprehensive Framework for Evaluating API-oriented Code Generation in Large Language Models
arXiv (Cornell University) · Published 23 September 2024 · DOI 10.48550/arxiv.2409.15228
In brief
“A Comprehensive Framework for Evaluating API-oriented Code Generation in Large Language Models” is a 2024 preprint by Ye Wu and 5 co-authors, published in arXiv (Cornell University). OpenAlex counts 12 citations as of 19 November 2026. It is open access: the full text is free to read at arXiv (Cornell University).
- Citations
- 12
- Most cited in
- 2025 (6)
- References
- 68
- Authors
- 6
Abstract
Large language models (LLMs) like GitHub Copilot and ChatGPT have emerged as powerful tools for code generation, significantly enhancing productivity and accelerating software development. However, existing benchmarks primarily focus on general code generation without considering API-oriented code generation, i.e., generating code that invokes APIs from specific libraries. Given the growing demand for API-oriented code generation, there is a pressing need for a systematic and automated approach to evaluate LLM on API-oriented code generation. To address this gap, we propose AutoAPIEval, a lightweight and automated framework designed to evaluate the capabilities of LLMs in API-oriented code generation. Our framework works with any library that provides API documentation and focuses on two unit tasks: API recommendation and code example generation, along with four metrics to evaluate the generated APIs and code examples, such as the proportion of incorrect API recommendations for Task 1, and the proportion of code examples where no specific API is invoked and uncompilable/unexecutable code examples for Task 2. In addition, we conducted a case study on three LLMs (ChatGPT, MagiCoder, and DeepSeek Coder) and Java Runtime Environment 8 to demonstrate the framework's effectiveness. Our findings reveal substantial variability in LLM performance across tasks, with ChatGPT adhering better to instructions, while sharing similar effectiveness in code example generation with its counterparts (i.e., MagiCoder and DeekSeek Coder). We also identify key factors associated with code quality, such as API popularity and model confidence, and build classifiers that achieve high accuracy in detecting incorrect API recommendations and erroneous code examples. Retrieval-augmented generation enhances the quality of code generated by LLMs, though its effectiveness varies across different LLMs.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Wu, Y., He, P., Wang, Z., Wang, S., Tian, Y., & Chen, T.-H. P. (2024). A Comprehensive Framework for Evaluating API-oriented Code Generation in Large Language Models. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2409.15228
MLA 9
Wu, Ye, et al. “A Comprehensive Framework for Evaluating API-oriented Code Generation in Large Language Models.” arXiv (Cornell University), 2024. https://doi.org/10.48550/arxiv.2409.15228.
Chicago
Wu, Ye, Peng He, Zehao Wang, Shaowei Wang, Yuan Tian, and Tse-Hsun Peter Chen. 2024. “A Comprehensive Framework for Evaluating API-oriented Code Generation in Large Language Models.” arXiv (Cornell University). https://doi.org/10.48550/arxiv.2409.15228.
Harvard
Wu, Y. et al. (2024) ‘A Comprehensive Framework for Evaluating API-oriented Code Generation in Large Language Models’, arXiv (Cornell University). doi:10.48550/arxiv.2409.15228.
IEEE
Y. Wu, P. He, Z. Wang, S. Wang, Y. Tian, and T.-H. P. Chen, “A Comprehensive Framework for Evaluating API-oriented Code Generation in Large Language Models,” arXiv (Cornell University), 2024, doi: 10.48550/arxiv.2409.15228.
BibTeX
@misc{wu2024comprehensive,
title = {{A Comprehensive Framework for Evaluating API-oriented Code Generation in Large Language Models}},
author = {Wu, Ye and He, Peng and Wang, Zehao and Wang, Shaowei and Tian, Yuan and Chen, Tse-Hsun Peter},
howpublished = {arXiv (Cornell University)},
year = {2024},
doi = {10.48550/arxiv.2409.15228},
url = {https://doi.org/10.48550/arxiv.2409.15228}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- large language models
- API recommendation
- API documentation
- code quality metrics
- retrieval-augmented generation
- ChatGPT
- DeepSeek-Coder
- Java
Work with this paper
Most-cited papers in Computer Science
- ImageNet classification with deep convolutional neural networksAlex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton · Communications of the ACM · 2017 · 114,726 citations
- Long Short-Term MemorySepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 109,097 citations
- A Mathematical Theory of CommunicationClaude E. Shannon · Bell System Technical Journal · 1948 · 91,736 citations
- Deep learningYann LeCun, Yoshua Bengio, Geoffrey E. Hinton · Nature · 2015 · 90,226 citations
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksShaoqing Ren, Kaiming He, Ross Girshick and 1 more · IEEE Transactions on Pattern Analysis and Machine Intelligence · 2016 · 74,143 citations
Questions about this paper
How do I cite this paper in APA style?
Wu, Y., He, P., Wang, Z., Wang, S., Tian, Y., & Chen, T.-H. P. (2024). A Comprehensive Framework for Evaluating API-oriented Code Generation in Large Language Models. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2409.15228
How many times has this paper been cited?
12 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at arXiv (Cornell University): sign in and open the PDF from this page.
Who wrote this paper?
Ye Wu, Peng He, Zehao Wang, Shaowei Wang, Yuan Tian, Tse-Hsun Peter Chen.
What is the DOI of this paper?
10.48550/arxiv.2409.15228, permanently at https://doi.org/10.48550/arxiv.2409.15228. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.