Conference paper · 2024 · Open access

Can LLM Replace Stack Overflow? A Study on Robustness and Reliability of Large Language Model Code Generation

Li Zhong, Zilong Wang

Proceedings of the AAAI Conference on Artificial Intelligence, 38(19), 21841–21849 · Published 24 March 2024 · DOI 10.1609/aaai.v38i19.30185

Open PDFView at the publisherCite this paper

In brief

“Can LLM Replace Stack Overflow? A Study on Robustness and Reliability of Large Language Model Code Generation” is a 2024 conference paper by Li Zhong and Zilong Wang, published in Proceedings of the AAAI Conference on Artificial Intelligence. OpenAlex counts 215 citations as of 19 November 2026, which puts it in the top 5% of works published in 2024 by citations. It is open access: the full text is free to read at Proceedings of the AAAI Conference on Artificial Intelligence.

Citations
215
Most cited in
2026 (86)
Field-weighted impact
32.54
References
33
Authors
2

Abstract

Recently, large language models (LLMs) have shown an extraordinary ability to understand natural language and generate programming code. It has been a common practice for software engineers to consult LLMs when encountering coding questions. Although efforts have been made to avoid syntax errors and align the code with the intended semantics, the reliability, and robustness of the code generation from LLMs have not yet been thoroughly studied. The executable code is not equivalent to reliable and robust code, especially in the context of real-world software development. For example, the misuse of APIs in the generated code could lead to severe problems, such as resource leaks, program crashes, etc. Existing code evaluation benchmarks and datasets focus on crafting small tasks such as programming questions in coding interviews, which, however, deviates from the problem that developers would ask LLM for real-world coding help. To fill the missing piece, in this work, we propose a dataset RobustAPI for evaluating the reliability and robustness of code generated by LLMs. We collect 1208 coding questions from Stack Overflow on 18 representative Java APIs. We summarize the common misuse patterns of these APIs and evaluate them on current popular LLMs. The evaluation results show that even for GPT-4, 62% of the generated code contains API misuses, which would cause unexpected consequences if the code is introduced into real-world software.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Zhong, L., & Wang, Z. (2024). Can LLM Replace Stack Overflow? A Study on Robustness and Reliability of Large Language Model Code Generation. Proceedings of the AAAI Conference on Artificial Intelligence, 38(19), 21841–21849. https://doi.org/10.1609/aaai.v38i19.30185

MLA 9

Zhong, Li, and Zilong Wang. “Can LLM Replace Stack Overflow? A Study on Robustness and Reliability of Large Language Model Code Generation.” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 19, 2024, pp. 21841–21849. https://doi.org/10.1609/aaai.v38i19.30185.

Chicago

Zhong, Li and Zilong Wang. 2024. “Can LLM Replace Stack Overflow? A Study on Robustness and Reliability of Large Language Model Code Generation.” Proceedings of the AAAI Conference on Artificial Intelligence 38 (19): 21841–21849. https://doi.org/10.1609/aaai.v38i19.30185.

Harvard

Zhong, L. and Wang, Z. (2024) ‘Can LLM Replace Stack Overflow? A Study on Robustness and Reliability of Large Language Model Code Generation’, Proceedings of the AAAI Conference on Artificial Intelligence, 38(19), pp. 21841–21849. doi:10.1609/aaai.v38i19.30185.

IEEE

L. Zhong and Z. Wang, “Can LLM Replace Stack Overflow? A Study on Robustness and Reliability of Large Language Model Code Generation,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 19, pp. 21841–21849, 2024, doi: 10.1609/aaai.v38i19.30185.

BibTeX

@inproceedings{zhong2024replace,
  title     = {{Can LLM Replace Stack Overflow? A Study on Robustness and Reliability of Large Language Model Code Generation}},
  author    = {Zhong, Li and Wang, Zilong},
  booktitle = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year      = {2024},
  volume    = {38},
  number    = {19},
  pages     = {21841--21849},
  doi       = {10.1609/aaai.v38i19.30185},
  url       = {https://doi.org/10.1609/aaai.v38i19.30185}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Citations by year

YearCitationsShare of the best year
202333
20244646
20257979
20268686

Its field-weighted citation impact is 32.54: it has been cited 32.54 times as often as the average work of the same field, type and year (1.00 is the world average).

Topic and keywords

Physical Sciences › Computer Science › Information Systems › Software Engineering Research

  • API misuse
  • code reliability
  • code robustness
  • Java API
  • GPT-4

Work with this paper

Most-cited papers in Computer Science

All highly cited Computer Science papers

Questions about this paper

How do I cite this paper in APA style?

Zhong, L., & Wang, Z. (2024). Can LLM Replace Stack Overflow? A Study on Robustness and Reliability of Large Language Model Code Generation. Proceedings of the AAAI Conference on Artificial Intelligence, 38(19), 21841–21849. https://doi.org/10.1609/aaai.v38i19.30185

How many times has this paper been cited?

215 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at Proceedings of the AAAI Conference on Artificial Intelligence: sign in and open the PDF from this page.

Who wrote this paper?

Li Zhong, Zilong Wang.

What is the DOI of this paper?

10.1609/aaai.v38i19.30185, permanently at https://doi.org/10.1609/aaai.v38i19.30185. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.