Journal article · 2025 · Open access
RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation
IEEE Transactions on Software Engineering, 52(2), 675–690 · Published 17 December 2025 · DOI 10.1109/tse.2025.3645056
In brief
“RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation” is a 2025 journal article by Yanli Wang and 10 co-authors, published in IEEE Transactions on Software Engineering. OpenAlex counts 70 citations as of 17 November 2026, which puts it in the top 1% of works published in 2025 by citations. It is open access: the full text is free to read at arXiv (Cornell University).
- Citations
- 70
- Most cited in
- 2026 (51)
- Field-weighted impact
- 35.76
- References
- 71
- Authors
- 11
Abstract
Repository-level code translation refers to translating an entire code repository from one programming language to another while preserving the functionality of the source repository. Many benchmarks have been proposed to evaluate the performance of such code translators. However, previous benchmarks mostly provide fine-grained samples, focusing at either code snippet, function, or file-level code translation. Such benchmarks do not accurately reflect real-world demands, where entire repositories often need to be translated, involving longer code length and more complex functionalities. To address this gap, we propose a new benchmark, named RepoTransBench, which is a real-world multilingual repository-level code translation benchmark featuring 1,897 real-world repository samples across 13 language pairs with automatically executable test suites. Besides, we introduce RepoTransAgent, a general agent framework to perform repository-level code translation. We evaluate both our benchmark’s challenges and agent’s effectiveness using several methods and backbone LLMs, revealing that repository-level translation remains challenging, where the best-performing method achieves only a 32.8% success rate. Furthermore, our analysis reveals that translation difficulty varies significantly by language pair direction, with dynamic-to-static language translation being much more challenging than the reverse direction (achieving below 10% vs. static-to-dynamic at 45-63%). Finally, we conduct a detailed error analysis and highlight current LLMs’ deficiencies in repository-level code translation, which could provide a reference for further improvements. We provide the code and data athttps://github.com/DeepSoftwareAnalytics/RepoTransBench.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Wang, Y., Wang, Y., Wang, S., Guo, D., Chen, J., Grundy, J., Liu, X., Ma, Y., Mao, M., Zhang, H., & Zheng, Z. (2025). RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation. IEEE Transactions on Software Engineering, 52(2), 675–690. https://doi.org/10.1109/tse.2025.3645056
MLA 9
Wang, Yanli, et al. “RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation.” IEEE Transactions on Software Engineering, vol. 52, no. 2, 2025, pp. 675–690. https://doi.org/10.1109/tse.2025.3645056.
Chicago
Wang, Yanli, Yanlin Wang, Suiquan Wang, Daya Guo, Jiachi Chen, John Grundy, Xilin Liu, et al. 2025. “RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation.” IEEE Transactions on Software Engineering 52 (2): 675–690. https://doi.org/10.1109/tse.2025.3645056.
Harvard
Wang, Y. et al. (2025) ‘RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation’, IEEE Transactions on Software Engineering, 52(2), pp. 675–690. doi:10.1109/tse.2025.3645056.
IEEE
Y. Wang et al., “RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation,” IEEE Transactions on Software Engineering, vol. 52, no. 2, pp. 675–690, 2025, doi: 10.1109/tse.2025.3645056.
BibTeX
@article{wang2025repotransbench,
title = {{RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation}},
author = {Wang, Yanli and Wang, Yanlin and Wang, Suiquan and Guo, Daya and Chen, Jiachi and Grundy, John and Liu, Xilin and Ma, Yuchi and Mao, Mingzhi and Zhang, Hongyu and Zheng, Zibin},
journal = {IEEE Transactions on Software Engineering},
year = {2025},
volume = {52},
number = {2},
pages = {675--690},
doi = {10.1109/tse.2025.3645056},
url = {https://doi.org/10.1109/tse.2025.3645056}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- large language models
- test suite execution
- error analysis
Work with this paper
Most-cited papers in Computer Science
- ImageNet classification with deep convolutional neural networksAlex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton · Communications of the ACM · 2017 · 114,726 citations
- Long Short-Term MemorySepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 109,097 citations
- A Mathematical Theory of CommunicationClaude E. Shannon · Bell System Technical Journal · 1948 · 91,736 citations
- Deep learningYann LeCun, Yoshua Bengio, Geoffrey E. Hinton · Nature · 2015 · 90,226 citations
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksShaoqing Ren, Kaiming He, Ross Girshick and 1 more · IEEE Transactions on Pattern Analysis and Machine Intelligence · 2016 · 74,143 citations
Questions about this paper
How do I cite this paper in APA style?
Wang, Y., Wang, Y., Wang, S., Guo, D., Chen, J., Grundy, J., Liu, X., Ma, Y., Mao, M., Zhang, H., & Zheng, Z. (2025). RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation. IEEE Transactions on Software Engineering, 52(2), 675–690. https://doi.org/10.1109/tse.2025.3645056
How many times has this paper been cited?
70 times, according to OpenAlex on 17 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at arXiv (Cornell University): sign in and open the PDF from this page.
Who wrote this paper?
Yanli Wang, Yanlin Wang, Suiquan Wang, Daya Guo, Jiachi Chen, John Grundy, Xilin Liu, Yuchi Ma, Mingzhi Mao, Hongyu Zhang, Zibin Zheng.
What is the DOI of this paper?
10.1109/tse.2025.3645056, permanently at https://doi.org/10.1109/tse.2025.3645056. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 17 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.