Review article · 2025 · Open access
Assessing the methodologic quality of systematic reviews using generative large language models
Canadian Urological Association Journal, 19(12), E427–E433 · Published 28 August 2025 · DOI 10.5489/cuaj.9243
In brief
“Assessing the methodologic quality of systematic reviews using generative large language models” is a 2025 review article by Bowen Yao and 5 co-authors, published in Canadian Urological Association Journal. OpenAlex counts 2 citations as of 18 November 2026, which puts it in the top 6% of works published in 2025 by citations. It is open access: the full text is free to read at Canadian Urological Association Journal.
- Citations
- 2
- Most cited in
- 2026 (2)
- Field-weighted impact
- 0.17
- References
- 29
- Authors
- 6
Abstract
INTRODUCTION: We aimed to evaluate whether generative large language models (LLMs) can accurately assess the methodologic quality of systematic reviews (SRs). METHODS: A total of 114 SRs from five leading urology journals were included in the study. Human reviewers graded each of the SRs in duplicate, with differences adjudicated by a third expert. We created a customized generative artificial intelligence (generative pre-trained transformer [GPT]), "Urology AMSTAR 2 Quality Assessor," and graded the 114 SRs in three iterations using a zero-shot method. We performed an enhanced trial focusing on critical criteria by giving GPT detailed, step-by-step instructions for each of the SRs using chain-of-thought method. Accuracy, sensitivity, specificity, and F1 score for each GPT trial were calculated against human results. Internal validity among three trials were computed. RESULTS: GPT had an overall congruence of 75%, with 77% in critical criteria and 73% in non-critical criteria when compared to human results. The average F1 score was 0.66. There was a high internal validity at 85% among three iterations. GPT accurately assigned 89% of studies into the correct overall category. When given specific, step-by-step instructions, congruence of critical criteria improved to 91%, and overall quality assessment accuracy to 93%. CONCLUSIONS: GPT showed promising ability to efficiently and accurately assess the quality of SRs in urology.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Yao, B., Ergun, O., Ding, M., Miller, C. D., Narayan, V. M., & Dahm, P. (2025). Assessing the methodologic quality of systematic reviews using generative large language models. Canadian Urological Association Journal, 19(12), E427–E433. https://doi.org/10.5489/cuaj.9243
MLA 9
Yao, Bowen, et al. “Assessing the methodologic quality of systematic reviews using generative large language models.” Canadian Urological Association Journal, vol. 19, no. 12, 2025, pp. E427–E433. https://doi.org/10.5489/cuaj.9243.
Chicago
Yao, Bowen, Onuralp Ergun, Maylynn Ding, Carly D. Miller, Vikram M. Narayan, and Philipp Dahm. 2025. “Assessing the methodologic quality of systematic reviews using generative large language models.” Canadian Urological Association Journal 19 (12): E427–E433. https://doi.org/10.5489/cuaj.9243.
Harvard
Yao, B. et al. (2025) ‘Assessing the methodologic quality of systematic reviews using generative large language models’, Canadian Urological Association Journal, 19(12), pp. E427–E433. doi:10.5489/cuaj.9243.
IEEE
B. Yao, O. Ergun, M. Ding, C. D. Miller, V. M. Narayan, and P. Dahm, “Assessing the methodologic quality of systematic reviews using generative large language models,” Canadian Urological Association Journal, vol. 19, no. 12, pp. E427–E433, 2025, doi: 10.5489/cuaj.9243.
BibTeX
@article{yao2025assessing,
title = {{Assessing the methodologic quality of systematic reviews using generative large language models}},
author = {Yao, Bowen and Ergun, Onuralp and Ding, Maylynn and Miller, Carly D. and Narayan, Vikram M. and Dahm, Philipp},
journal = {Canadian Urological Association Journal},
year = {2025},
volume = {19},
number = {12},
pages = {E427--E433},
doi = {10.5489/cuaj.9243},
url = {https://doi.org/10.5489/cuaj.9243}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- systematic reviews
- generative large language models
- AMSTAR 2
- urology
- chain-of-thought prompting
- zero-shot learning
- human-AI agreement
- internal validity
Work with this paper
Most-cited papers in Decision Sciences
- The PRISMA 2020 statement: an updated guideline for reporting systematic reviewsMatthew James Page, Joanne E. McKenzie, Patrick M. Bossuyt and 23 more · BMJ · 2021 · 144,919 citations
- Evaluating Structural Equation Models with Unobservable Variables and Measurement ErrorClaes Fornell, David F. Larcker · Journal of Marketing Research · 1981 · 118,336 citations
- Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternativesLi‐tze Hu, Peter M. Bentler · Structural Equation Modeling A Multidisciplinary Journal · 1999 · 117,461 citations
- Common method biases in behavioral research: A critical review of the literature and recommended remedies.Philip M. Podsakoff, SCOTT B. MACKENZIE, Jeong Yeon Lee and 1 more · Journal of Applied Psychology · 2003 · 89,248 citations
- Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statementDavid Moher, Alessandro Liberati, Jennifer Tetzlaff and 2 more · BMJ · 2009 · 85,524 citations
Questions about this paper
How do I cite this paper in APA style?
Yao, B., Ergun, O., Ding, M., Miller, C. D., Narayan, V. M., & Dahm, P. (2025). Assessing the methodologic quality of systematic reviews using generative large language models. Canadian Urological Association Journal, 19(12), E427–E433. https://doi.org/10.5489/cuaj.9243
How many times has this paper been cited?
2 times, according to OpenAlex on 18 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at Canadian Urological Association Journal: sign in and open the PDF from this page.
Who wrote this paper?
Bowen Yao, Onuralp Ergun, Maylynn Ding, Carly D. Miller, Vikram M. Narayan, Philipp Dahm.
What is the DOI of this paper?
10.5489/cuaj.9243, permanently at https://doi.org/10.5489/cuaj.9243. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 18 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.