Journal article · 2026 · Open access

Does AI Code Review Lead to Code Changes? A Case Study of GitHub Actions

Kexin Sun, Hongyu Kuang, Sebastian Baltes, Xin Zhou, He Zhang, Xiaoxing Ma, Guoping Rong, Dong Shao, Christoph Treude

IEEE Transactions on Software Engineering, 52(7), 2111–2126 · Published 29 April 2026 · DOI 10.1109/tse.2026.3688237

Open PDFView at the publisherCite this paper

In brief

“Does AI Code Review Lead to Code Changes? A Case Study of GitHub Actions” is a 2026 journal article by Kexin Sun and 8 co-authors, published in IEEE Transactions on Software Engineering. OpenAlex counts 24 citations as of 17 November 2026, which puts it in the top 7% of works published in 2026 by citations. It is open access: the full text is free to read at arXiv (Cornell University).

Citations
24
Most cited in
2026 (22)
Field-weighted impact
56.98
References
48
Authors
9

Abstract

AI-based code review tools automatically review and comment on pull requests to improve code quality. Despite their growing presence, little is known about their actual impact. We present a large-scale empirical study of 16 popular AI-based code review actions for GitHub workflows, analyzing more than 22,000 review comments in 178 repositories. We investigate (1) how these tools are adopted and configured, (2) whether their comments lead to code changes, and (3) which factors influence their effectiveness. We develop a two-stage LLM-assisted framework to determine whether review comments are addressed, and use interpretable machine learning to identify influencing factors. Our findings show that, while adoption is growing, effectiveness varies widely. Comments that are concise, contain code snippets, and are manually triggered, particularly those from hunk-level review tools, are more likely to result in code changes. These results highlight the importance of careful tool design and suggest directions for improving AI-based code review systems.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Sun, K., Kuang, H., Baltes, S., Zhou, X., Zhang, H., Ma, X., Rong, G., Shao, D., & Treude, C. (2026). Does AI Code Review Lead to Code Changes? A Case Study of GitHub Actions. IEEE Transactions on Software Engineering, 52(7), 2111–2126. https://doi.org/10.1109/tse.2026.3688237

MLA 9

Sun, Kexin, et al. “Does AI Code Review Lead to Code Changes? A Case Study of GitHub Actions.” IEEE Transactions on Software Engineering, vol. 52, no. 7, 2026, pp. 2111–2126. https://doi.org/10.1109/tse.2026.3688237.

Chicago

Sun, Kexin, Hongyu Kuang, Sebastian Baltes, Xin Zhou, He Zhang, Xiaoxing Ma, Guoping Rong, Dong Shao, and Christoph Treude. 2026. “Does AI Code Review Lead to Code Changes? A Case Study of GitHub Actions.” IEEE Transactions on Software Engineering 52 (7): 2111–2126. https://doi.org/10.1109/tse.2026.3688237.

Harvard

Sun, K. et al. (2026) ‘Does AI Code Review Lead to Code Changes? A Case Study of GitHub Actions’, IEEE Transactions on Software Engineering, 52(7), pp. 2111–2126. doi:10.1109/tse.2026.3688237.

IEEE

K. Sun et al., “Does AI Code Review Lead to Code Changes? A Case Study of GitHub Actions,” IEEE Transactions on Software Engineering, vol. 52, no. 7, pp. 2111–2126, 2026, doi: 10.1109/tse.2026.3688237.

BibTeX

@article{sun2026does,
  title     = {{Does AI Code Review Lead to Code Changes? A Case Study of GitHub Actions}},
  author    = {Sun, Kexin and Kuang, Hongyu and Baltes, Sebastian and Zhou, Xin and Zhang, He and Ma, Xiaoxing and Rong, Guoping and Shao, Dong and Treude, Christoph},
  journal   = {IEEE Transactions on Software Engineering},
  year      = {2026},
  volume    = {52},
  number    = {7},
  pages     = {2111--2126},
  doi       = {10.1109/tse.2026.3688237},
  url       = {https://doi.org/10.1109/tse.2026.3688237}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Physical Sciences › Computer Science › Information Systems › Software Engineering Research

  • GitHub Actions
  • code amendments
  • model interpretability
  • tool adoption
  • code quality
  • tool effectiveness

Work with this paper

Most-cited papers in Computer Science

All highly cited Computer Science papers

Questions about this paper

How do I cite this paper in APA style?

Sun, K., Kuang, H., Baltes, S., Zhou, X., Zhang, H., Ma, X., Rong, G., Shao, D., & Treude, C. (2026). Does AI Code Review Lead to Code Changes? A Case Study of GitHub Actions. IEEE Transactions on Software Engineering, 52(7), 2111–2126. https://doi.org/10.1109/tse.2026.3688237

How many times has this paper been cited?

24 times, according to OpenAlex on 17 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at arXiv (Cornell University): sign in and open the PDF from this page.

Who wrote this paper?

Kexin Sun, Hongyu Kuang, Sebastian Baltes, Xin Zhou, He Zhang, Xiaoxing Ma, Guoping Rong, Dong Shao, Christoph Treude.

What is the DOI of this paper?

10.1109/tse.2026.3688237, permanently at https://doi.org/10.1109/tse.2026.3688237. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 17 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.