Journal article · 2026 · Open access
Defining and Detecting the Defects of Large Language Model-Based Autonomous Agents
IEEE Transactions on Software Engineering, 52(3), 1074–1093 · Published 28 January 2026 · DOI 10.1109/tse.2026.3658554
In brief
“Defining and Detecting the Defects of Large Language Model-Based Autonomous Agents” is a 2026 journal article by Kaiwen Ning and 7 co-authors, published in IEEE Transactions on Software Engineering. OpenAlex counts 24 citations as of 17 November 2026, which puts it in the top 4% of works published in 2026 by citations. It is open access: the full text is free to read at arXiv (Cornell University).
- Citations
- 24
- Most cited in
- 2026 (20)
- Field-weighted impact
- 49.86
- References
- 71
- Authors
- 8
Abstract
Artificial intelligence (AI) agents are systems capable of perceiving their environment, autonomously planning and executing tasks. Recent advancements in Large Language Models (LLMs) have introduced a transformative paradigm for AI agents, enabling them to interact with external resources and tools through prompt techniques. This advancement has significantly extended the capabilities of LLMs, positioning LLM-based AI Agents as an important research area. In such agents, the workflow integrates developer-written code, which manages framework construction and logic control, with LLM-generated natural language that enhances dynamic decision-making and interaction. However, inconsistencies between LLM outputs and developer logic can lead to defects, such as tool invocation failures. These issues introduce specific risks, leading to various defects in LLM-based AI Agents, including service interruptions and incorrect output. Despite the importance of these issues, there is a lack of systematic work that focuses on analyzing LLM-based AI Agents to uncover defects in their code. To address this gap, we present the first study focused on identifying and detecting defects in LLM Agents. We collected and analyzed 14,754 relevant developer reports from StackOverflow and GitHub. We further filtered 2,604 valid posts to define and classify eight types of agent code defects. Then, we designed a static analysis tool, named Agentable, to detect these defects. Agentable leverages Code Property Graphs (CPGs) and LLMs to analyze Agent workflows by efficiently identifying specific code patterns and analyzing natural language descriptions. To evaluate Agentable, we constructed two datasets: AgentSet, which consists of 84 real world Agent projects, and AgentTest, which contains 78 Agent projects specifically designed to include various types of defects. Our evaluation shows that Agentable achieves a precision of 88.79% on the real-world agent dataset and a recall of 91.03% on the manually labeled defect dataset. Furthermore, our analysis identifies 889 defects in real-world agent projects, highlighting the prevalence of these issues in practice.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Ning, K., Chen, J., Zhang, J., Li, W., Wang, Z., Feng, Y., Zhang, W., & Zheng, Z. (2026). Defining and Detecting the Defects of Large Language Model-Based Autonomous Agents. IEEE Transactions on Software Engineering, 52(3), 1074–1093. https://doi.org/10.1109/tse.2026.3658554
MLA 9
Ning, Kaiwen, et al. “Defining and Detecting the Defects of Large Language Model-Based Autonomous Agents.” IEEE Transactions on Software Engineering, vol. 52, no. 3, 2026, pp. 1074–1093. https://doi.org/10.1109/tse.2026.3658554.
Chicago
Ning, Kaiwen, Jiachi Chen, Jingwen Zhang, Wei Li, Zexu Wang, Yuming Feng, Weizhe Zhang, and Zibin Zheng. 2026. “Defining and Detecting the Defects of Large Language Model-Based Autonomous Agents.” IEEE Transactions on Software Engineering 52 (3): 1074–1093. https://doi.org/10.1109/tse.2026.3658554.
Harvard
Ning, K. et al. (2026) ‘Defining and Detecting the Defects of Large Language Model-Based Autonomous Agents’, IEEE Transactions on Software Engineering, 52(3), pp. 1074–1093. doi:10.1109/tse.2026.3658554.
IEEE
K. Ning et al., “Defining and Detecting the Defects of Large Language Model-Based Autonomous Agents,” IEEE Transactions on Software Engineering, vol. 52, no. 3, pp. 1074–1093, 2026, doi: 10.1109/tse.2026.3658554.
BibTeX
@article{ning2026defining,
title = {{Defining and Detecting the Defects of Large Language Model-Based Autonomous Agents}},
author = {Ning, Kaiwen and Chen, Jiachi and Zhang, Jingwen and Li, Wei and Wang, Zexu and Feng, Yuming and Zhang, Weizhe and Zheng, Zibin},
journal = {IEEE Transactions on Software Engineering},
year = {2026},
volume = {52},
number = {3},
pages = {1074--1093},
doi = {10.1109/tse.2026.3658554},
url = {https://doi.org/10.1109/tse.2026.3658554}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- large language models
- defect detection
- static analysis
- code property graph
- defect classification
- LLM-generated code
- agent workflows
Work with this paper
Most-cited papers in Computer Science
- ImageNet classification with deep convolutional neural networksAlex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton · Communications of the ACM · 2017 · 114,726 citations
- Long Short-Term MemorySepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 109,097 citations
- A Mathematical Theory of CommunicationClaude E. Shannon · Bell System Technical Journal · 1948 · 91,736 citations
- Deep learningYann LeCun, Yoshua Bengio, Geoffrey E. Hinton · Nature · 2015 · 90,226 citations
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksShaoqing Ren, Kaiming He, Ross Girshick and 1 more · IEEE Transactions on Pattern Analysis and Machine Intelligence · 2016 · 74,143 citations
Questions about this paper
How do I cite this paper in APA style?
Ning, K., Chen, J., Zhang, J., Li, W., Wang, Z., Feng, Y., Zhang, W., & Zheng, Z. (2026). Defining and Detecting the Defects of Large Language Model-Based Autonomous Agents. IEEE Transactions on Software Engineering, 52(3), 1074–1093. https://doi.org/10.1109/tse.2026.3658554
How many times has this paper been cited?
24 times, according to OpenAlex on 17 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at arXiv (Cornell University): sign in and open the PDF from this page.
Who wrote this paper?
Kaiwen Ning, Jiachi Chen, Jingwen Zhang, Wei Li, Zexu Wang, Yuming Feng, Weizhe Zhang, Zibin Zheng.
What is the DOI of this paper?
10.1109/tse.2026.3658554, permanently at https://doi.org/10.1109/tse.2026.3658554. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 17 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.