Conference paper · 2023 · Open access

Impact of Code Language Models on Automated Program Repair

Nan Jiang, Kevin Liu, Thibaud Lutellier, Lin Tan

ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), 1430–1442 · Published 1 May 2023 · DOI 10.1109/icse48619.2023.00125

Open PDFView at the publisherCite this paper

In brief

“Impact of Code Language Models on Automated Program Repair” is a 2023 conference paper by Nan Jiang and 3 co-authors, published in ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). OpenAlex counts 416 citations as of 19 November 2026. It is open access under a CC BY-NC-ND licence: the full text is free to read at arXiv (Cornell University).

Citations
416
Most cited in
2025 (149)
Field-weighted impact
53.97
References
68
Authors
4

Abstract

Automated program repair (APR) aims to help developers improve software reliability by generating patches for buggy programs. Although many code language models (CLM) are developed and effective in many software tasks such as code completion, there has been little comprehensive, in-depth work to evaluate CLMs' fixing capabilities and to fine-tune CLMs for the APR task. Firstly, this work is the first to evaluate ten CLMs on four APR benchmarks, which shows that surprisingly, the best CLM, as is, fixes 72% more bugs than the state-of-the-art deep-learning (DL)-based APR techniques. Secondly, one of the four APR benchmarks was created by us in this paper to avoid data leaking for a fair evaluation. Thirdly, it is the first work to fine-tune CLMs with APR training data, which shows that fine-tuning brings 31%-1,267% improvement to CLMs and enables them to fix 46%-164 % more bugs than existing DL-based APR techniques. Fourthly, this work studies the impact of buggy lines, showing that CLMs, as is, cannot make good use of the buggy lines to fix bugs, yet fine-tuned CLMs could potentially over-rely on buggy lines. Lastly, this work analyzes the size, time, and memory efficiency of different CLMs. This work shows promising directions for the APR domain, such as fine-tuning CLMs with APR-specific designs, and also raises awareness of fair and comprehensive evaluations of CLMs and calls for more transparent reporting of open-source repositories used in the pre-training data to address the data leaking problem.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Jiang, N., Liu, K., Lutellier, T., & Tan, L. (2023). Impact of Code Language Models on Automated Program Repair. ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), 1430–1442. https://doi.org/10.1109/icse48619.2023.00125

MLA 9

Jiang, Nan, et al. “Impact of Code Language Models on Automated Program Repair.” ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), 2023, pp. 1430–1442. https://doi.org/10.1109/icse48619.2023.00125.

Chicago

Jiang, Nan, Kevin Liu, Thibaud Lutellier, and Lin Tan. 2023. “Impact of Code Language Models on Automated Program Repair.” ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion): 1430–1442. https://doi.org/10.1109/icse48619.2023.00125.

Harvard

Jiang, N. et al. (2023) ‘Impact of Code Language Models on Automated Program Repair’, ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), pp. 1430–1442. doi:10.1109/icse48619.2023.00125.

IEEE

N. Jiang, K. Liu, T. Lutellier, and L. Tan, “Impact of Code Language Models on Automated Program Repair,” ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), pp. 1430–1442, 2023, doi: 10.1109/icse48619.2023.00125.

BibTeX

@inproceedings{jiang2023impact,
  title     = {{Impact of Code Language Models on Automated Program Repair}},
  author    = {Jiang, Nan and Liu, Kevin and Lutellier, Thibaud and Tan, Lin},
  booktitle = {ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion)},
  year      = {2023},
  pages     = {1430--1442},
  doi       = {10.1109/icse48619.2023.00125},
  url       = {https://doi.org/10.1109/icse48619.2023.00125}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Citations by year

YearCitationsShare of the best year
201311
20233636
2024108108
2025149149
2026122122

Its field-weighted citation impact is 53.97: it has been cited 53.97 times as often as the average work of the same field, type and year (1.00 is the world average).

Topic and keywords

Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques

  • automated program repair
  • code language models
  • fine-tuning
  • bug fixes
  • benchmark evaluation
  • data leakage
  • model efficiency
  • software reliability

Work with this paper

Most-cited papers in Computer Science

All highly cited Computer Science papers

Questions about this paper

How do I cite this paper in APA style?

Jiang, N., Liu, K., Lutellier, T., & Tan, L. (2023). Impact of Code Language Models on Automated Program Repair. ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), 1430–1442. https://doi.org/10.1109/icse48619.2023.00125

How many times has this paper been cited?

416 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at arXiv (Cornell University): sign in and open the PDF from this page.

Who wrote this paper?

Nan Jiang, Kevin Liu, Thibaud Lutellier, Lin Tan.

What is the DOI of this paper?

10.1109/icse48619.2023.00125, permanently at https://doi.org/10.1109/icse48619.2023.00125. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.