Journal article · 2025 · Open access
LlamaRestTest: Effective REST API Testing with Small Language Models
Proceedings of the ACM on software engineering., 2(FSE), 465–488 · Published 19 June 2025 · DOI 10.1145/3715737
In brief
“LlamaRestTest: Effective REST API Testing with Small Language Models” is a 2025 journal article by Myeongsoo Kim and 2 co-authors, published in Proceedings of the ACM on software engineering.. OpenAlex counts 65 citations as of 19 November 2026, which puts it in the top 1% of works published in 2025 by citations. It is open access under a CC BY licence: the full text is free to read at Proceedings of the ACM on software engineering..
- Citations
- 65
- Most cited in
- 2026 (49)
- Field-weighted impact
- 32.95
- References
- 55
- Authors
- 3
Abstract
Modern web services rely heavily on REST APIs, typically documented using the OpenAPI specification. The widespread adoption of this standard has resulted in the development of many black-box testing tools that generate tests based on OpenAPI specifications. Although Large Language Models (LLMs) have shown promising test-generation abilities, their application to REST API testing remains mostly unexplored. We present LlamaRestTest, a novel approach that employs two custom LLMs—created by fine-tuning and quantizing the Llama3-8B model using mined datasets of REST API example values and inter-parameter dependencies—to generate realistic test inputs and uncover inter-parameter dependencies during the testing process by analyzing server responses. We evaluated LlamaRestTest on 12 real-world services (including popular services such as Spotify), comparing it against RESTGPT, a GPT-powered specification-enhancement tool, as well as several state-of-the-art REST API testing tools, including RESTler, MoRest, EvoMaster, and ARAT-RL. Our results demonstrate that fine-tuning enables smaller models to outperform much larger models in detecting actionable parameter-dependency rules and generating valid inputs for REST API testing. We also evaluated different tool configurations, ranging from the base Llama3-8B model to fine-tuned versions, and explored multiple quantization techniques, including 2-bit, 4-bit, and 8-bit integer formats. Our study shows that small language models can perform as well as, or better than, large language models in REST API testing, balancing effectiveness and efficiency. Furthermore, LlamaRestTest outperforms state-of-the-art REST API testing tools in code coverage achieved and internal server errors identified, even when those tools use RESTGPT-enhanced specifications. Finally, through an ablation study, we show that each component of LlamaRestTest contributes to its overall performance.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Kim, M., Sinha, S., & Orso, A. (2025). LlamaRestTest: Effective REST API Testing with Small Language Models. Proceedings of the ACM on software engineering., 2(FSE), 465–488. https://doi.org/10.1145/3715737
MLA 9
Kim, Myeongsoo, et al. “LlamaRestTest: Effective REST API Testing with Small Language Models.” Proceedings of the ACM on software engineering., vol. 2, no. FSE, 2025, pp. 465–488. https://doi.org/10.1145/3715737.
Chicago
Kim, Myeongsoo, Saurabh Sinha, and Alessandro Orso. 2025. “LlamaRestTest: Effective REST API Testing with Small Language Models.” Proceedings of the ACM on software engineering. 2 (FSE): 465–488. https://doi.org/10.1145/3715737.
Harvard
Kim, M., Sinha, S. and Orso, A. (2025) ‘LlamaRestTest: Effective REST API Testing with Small Language Models’, Proceedings of the ACM on software engineering., 2(FSE), pp. 465–488. doi:10.1145/3715737.
IEEE
M. Kim, S. Sinha, and A. Orso, “LlamaRestTest: Effective REST API Testing with Small Language Models,” Proceedings of the ACM on software engineering., vol. 2, no. FSE, pp. 465–488, 2025, doi: 10.1145/3715737.
BibTeX
@article{kim2025llamaresttest,
title = {{LlamaRestTest: Effective REST API Testing with Small Language Models}},
author = {Kim, Myeongsoo and Sinha, Saurabh and Orso, Alessandro},
journal = {Proceedings of the ACM on software engineering.},
year = {2025},
volume = {2},
number = {FSE},
pages = {465--488},
doi = {10.1145/3715737},
url = {https://doi.org/10.1145/3715737}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Citations by year
| Year | Citations | Share of the best year |
|---|---|---|
| 2024 | 1 | |
| 2025 | 15 | |
| 2026 | 49 |
Its field-weighted citation impact is 32.95: it has been cited 32.95 times as often as the average work of the same field, type and year (1.00 is the world average).
Topic and keywords
Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- REST API testing
- test data generation
- small language models
- Llama-3-8B
- fine-tuning
- model quantization
- OpenAPI
- parameter dependencies
Work with this paper
Most-cited papers in Computer Science
- ImageNet classification with deep convolutional neural networksAlex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton · Communications of the ACM · 2017 · 114,726 citations
- Long Short-Term MemorySepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 109,097 citations
- A Mathematical Theory of CommunicationClaude E. Shannon · Bell System Technical Journal · 1948 · 91,736 citations
- Deep learningYann LeCun, Yoshua Bengio, Geoffrey E. Hinton · Nature · 2015 · 90,226 citations
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksShaoqing Ren, Kaiming He, Ross Girshick and 1 more · IEEE Transactions on Pattern Analysis and Machine Intelligence · 2016 · 74,143 citations
Questions about this paper
How do I cite this paper in APA style?
Kim, M., Sinha, S., & Orso, A. (2025). LlamaRestTest: Effective REST API Testing with Small Language Models. Proceedings of the ACM on software engineering., 2(FSE), 465–488. https://doi.org/10.1145/3715737
How many times has this paper been cited?
65 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at Proceedings of the ACM on software engineering.: sign in and open the full text from this page.
Who wrote this paper?
Myeongsoo Kim, Saurabh Sinha, Alessandro Orso.
What is the DOI of this paper?
10.1145/3715737, permanently at https://doi.org/10.1145/3715737. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.