Preprint · 2025 · Open access
RustEvo^2: An Evolving Benchmark for API Evolution in LLM-based Rust Code Generation
arXiv (Cornell University) · Published 21 March 2025 · DOI 10.48550/arxiv.2503.16922
In brief
“RustEvo^2: An Evolving Benchmark for API Evolution in LLM-based Rust Code Generation” is a 2025 preprint by Linxi Liang and 7 co-authors, published in arXiv (Cornell University). OpenAlex counts 22 citations as of 19 November 2026. It is open access: the full text is free to read at arXiv (Cornell University).
- Citations
- 22
- Most cited in
- 2026 (19)
- References
- 45
- Authors
- 8
Abstract
Large Language Models (LLMs) have become pivotal tools for automating code generation in software development. However, these models face significant challenges in producing version-aware code for rapidly evolving languages like Rust, where frequent Application Programming Interfaces (API) changes across versions lead to compatibility issues and correctness errors. Existing benchmarks lack systematic evaluation of how models navigate API transitions, relying on labor-intensive manual curation and offering limited version-specific insights. To address this gap, we present RustEvo, a novel framework for constructing dynamic benchmarks that evaluate the ability of LLMs to adapt to evolving Rust APIs. RustEvo automates dataset creation by synthesizing 588 API changes (380 from Rust standard libraries, 208 from 15 third-party crates) into programming tasks mirroring real-world challenges. These tasks cover four API evolution categories: Stabilizations, Signature Changes, Behavioral Changes, and Deprecations, reflecting their actual distribution in the Rust ecosystem. Experiments on state-of-the-art (SOTA) LLMs reveal significant performance variations: models achieve a 65.8% average success rate on stabilized APIs but only 38.0% on behavioral changes, highlighting difficulties in detecting semantic shifts without signature alterations. Knowledge cutoff dates strongly influence performance, with models scoring 56.1% on before-cutoff APIs versus 32.5% on after-cutoff tasks. Retrieval-Augmented Generation (RAG) mitigates this gap, improving success rates by 13.5% on average for APIs released after model training. Our findings underscore the necessity of our evolution-aware benchmarks to advance the adaptability of LLMs in fast-paced software ecosystems. The framework and the benchmarks are publicly released at https://github.com/SYSUSELab/RustEvo.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Liang, L., Jing, G., Liu, M., Wang, C., Guangsheng, O., Wang, Y., Peng, X., & Zheng, Z. (2025). RustEvo^2: An Evolving Benchmark for API Evolution in LLM-based Rust Code Generation. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2503.16922
MLA 9
Liang, Linxi, et al. “RustEvo^2: An Evolving Benchmark for API Evolution in LLM-based Rust Code Generation.” arXiv (Cornell University), 2025. https://doi.org/10.48550/arxiv.2503.16922.
Chicago
Liang, Linxi, Gong, Jing, Mingwei Liu, Chong Wang, Ou, Guangsheng, Yanlin Wang, Xin Peng, and Zibin Zheng. 2025. “RustEvo^2: An Evolving Benchmark for API Evolution in LLM-based Rust Code Generation.” arXiv (Cornell University). https://doi.org/10.48550/arxiv.2503.16922.
Harvard
Liang, L. et al. (2025) ‘RustEvo^2: An Evolving Benchmark for API Evolution in LLM-based Rust Code Generation’, arXiv (Cornell University). doi:10.48550/arxiv.2503.16922.
IEEE
L. Liang et al., “RustEvo^2: An Evolving Benchmark for API Evolution in LLM-based Rust Code Generation,” arXiv (Cornell University), 2025, doi: 10.48550/arxiv.2503.16922.
BibTeX
@misc{liang2025rustevo,
title = {{RustEvo^2: An Evolving Benchmark for API Evolution in LLM-based Rust Code Generation}},
author = {Liang, Linxi and Jing, Gong, and Liu, Mingwei and Wang, Chong and Guangsheng, Ou, and Wang, Yanlin and Peng, Xin and Zheng, Zibin},
howpublished = {arXiv (Cornell University)},
year = {2025},
doi = {10.48550/arxiv.2503.16922},
url = {https://doi.org/10.48550/arxiv.2503.16922}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- API evolution
- Rust
- LLM-based code generation
- dynamic benchmarking
- API deprecation
- knowledge cutoff
- retrieval-augmented generation
Work with this paper
Most-cited papers in Computer Science
- ImageNet classification with deep convolutional neural networksAlex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton · Communications of the ACM · 2017 · 114,726 citations
- Long Short-Term MemorySepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 109,097 citations
- A Mathematical Theory of CommunicationClaude E. Shannon · Bell System Technical Journal · 1948 · 91,736 citations
- Deep learningYann LeCun, Yoshua Bengio, Geoffrey E. Hinton · Nature · 2015 · 90,226 citations
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksShaoqing Ren, Kaiming He, Ross Girshick and 1 more · IEEE Transactions on Pattern Analysis and Machine Intelligence · 2016 · 74,143 citations
Questions about this paper
How do I cite this paper in APA style?
Liang, L., Jing, G., Liu, M., Wang, C., Guangsheng, O., Wang, Y., Peng, X., & Zheng, Z. (2025). RustEvo^2: An Evolving Benchmark for API Evolution in LLM-based Rust Code Generation. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2503.16922
How many times has this paper been cited?
22 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at arXiv (Cornell University): sign in and open the PDF from this page.
Who wrote this paper?
Linxi Liang, Gong, Jing, Mingwei Liu, Chong Wang, Ou, Guangsheng, Yanlin Wang, Xin Peng, Zibin Zheng.
What is the DOI of this paper?
10.48550/arxiv.2503.16922, permanently at https://doi.org/10.48550/arxiv.2503.16922. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.