Preprint · 2026 · Open access

Why LLMs Fail: A Failure Analysis and Partial Success Measurement for Automated Security Patch Generation

Amir Al-Maamari

arXiv (Cornell University) · Published 10 March 2026 · DOI 10.48550/arxiv.2603.10072

Open PDFView at the publisherCite this paper

In brief

“Why LLMs Fail: A Failure Analysis and Partial Success Measurement for Automated Security Patch Generation” is a 2026 preprint by Amir Al-Maamari, published in arXiv (Cornell University). OpenAlex counts 2 citations as of 19 November 2026. It is open access: the full text is free to read at arXiv (Cornell University).

Citations
2
Most cited in
2026 (2)
References
12
Authors
1

Abstract

Large Language Models (LLMs) show promise for Automated Program Repair (APR), yet their effectiveness on security vulnerabilities remains poorly characterized. This study analyzes 319 LLM-generated security patchesacross 64 Java vulnerabilities from the Vul4J benchmark. Using tri-axis evaluation (compilation, security via PoV tests, functionality via test suites), the analysis reveals that only 24.8% of patches achieve full correctness, while 51.4% fail both security and functionality. The dominant failure mode is semantic misunderstanding: LLMs produce syntactically valid code but apply incorrect repair strategies. The proposed Security Repair Score (SRS) quantifies this gap, showing LLMs preserve functionality (mean 0.832) but struggle with security (mean 0.251). Vulnerability type strongly predicts difficulty, with fix rates ranging from 0% (input validation) to 45% (infinite loop). These findings demonstrate that LLM security patches require rigorous validation before deployment.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Al-Maamari, A. (2026). Why LLMs Fail: A Failure Analysis and Partial Success Measurement for Automated Security Patch Generation. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2603.10072

MLA 9

Al-Maamari, Amir. “Why LLMs Fail: A Failure Analysis and Partial Success Measurement for Automated Security Patch Generation.” arXiv (Cornell University), 2026. https://doi.org/10.48550/arxiv.2603.10072.

Chicago

Al-Maamari, Amir. 2026. “Why LLMs Fail: A Failure Analysis and Partial Success Measurement for Automated Security Patch Generation.” arXiv (Cornell University). https://doi.org/10.48550/arxiv.2603.10072.

Harvard

Al-Maamari, A. (2026) ‘Why LLMs Fail: A Failure Analysis and Partial Success Measurement for Automated Security Patch Generation’, arXiv (Cornell University). doi:10.48550/arxiv.2603.10072.

IEEE

A. Al-Maamari, “Why LLMs Fail: A Failure Analysis and Partial Success Measurement for Automated Security Patch Generation,” arXiv (Cornell University), 2026, doi: 10.48550/arxiv.2603.10072.

BibTeX

@misc{almaamari2026llms,
  title     = {{Why LLMs Fail: A Failure Analysis and Partial Success Measurement for Automated Security Patch Generation}},
  author    = {Al-Maamari, Amir},
  howpublished = {arXiv (Cornell University)},
  year      = {2026},
  doi       = {10.48550/arxiv.2603.10072},
  url       = {https://doi.org/10.48550/arxiv.2603.10072}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques

  • large language models
  • automated program repair
  • security vulnerabilities
  • Java vulnerabilities
  • failure analysis
  • vulnerability types
  • input validation

Work with this paper

Most-cited papers in Computer Science

All highly cited Computer Science papers

Questions about this paper

How do I cite this paper in APA style?

Al-Maamari, A. (2026). Why LLMs Fail: A Failure Analysis and Partial Success Measurement for Automated Security Patch Generation. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2603.10072

How many times has this paper been cited?

2 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at arXiv (Cornell University): sign in and open the PDF from this page.

Who wrote this paper?

Amir Al-Maamari.

What is the DOI of this paper?

10.48550/arxiv.2603.10072, permanently at https://doi.org/10.48550/arxiv.2603.10072. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.