Review article · 2025 · Open access

Assessing the methodologic quality of systematic reviews using generative large language models

Bowen Yao, Onuralp Ergun, Maylynn Ding, Carly D. Miller, Vikram M. Narayan, Philipp Dahm

Canadian Urological Association Journal, 19(12), E427–E433 · Published 28 August 2025 · DOI 10.5489/cuaj.9243

Open PDFView at the publisherCite this paper

In brief

“Assessing the methodologic quality of systematic reviews using generative large language models” is a 2025 review article by Bowen Yao and 5 co-authors, published in Canadian Urological Association Journal. OpenAlex counts 2 citations as of 18 November 2026, which puts it in the top 6% of works published in 2025 by citations. It is open access: the full text is free to read at Canadian Urological Association Journal.

Citations
2
Most cited in
2026 (2)
Field-weighted impact
0.17
References
29
Authors
6

Abstract

INTRODUCTION: We aimed to evaluate whether generative large language models (LLMs) can accurately assess the methodologic quality of systematic reviews (SRs). METHODS: A total of 114 SRs from five leading urology journals were included in the study. Human reviewers graded each of the SRs in duplicate, with differences adjudicated by a third expert. We created a customized generative artificial intelligence (generative pre-trained transformer [GPT]), "Urology AMSTAR 2 Quality Assessor," and graded the 114 SRs in three iterations using a zero-shot method. We performed an enhanced trial focusing on critical criteria by giving GPT detailed, step-by-step instructions for each of the SRs using chain-of-thought method. Accuracy, sensitivity, specificity, and F1 score for each GPT trial were calculated against human results. Internal validity among three trials were computed. RESULTS: GPT had an overall congruence of 75%, with 77% in critical criteria and 73% in non-critical criteria when compared to human results. The average F1 score was 0.66. There was a high internal validity at 85% among three iterations. GPT accurately assigned 89% of studies into the correct overall category. When given specific, step-by-step instructions, congruence of critical criteria improved to 91%, and overall quality assessment accuracy to 93%. CONCLUSIONS: GPT showed promising ability to efficiently and accurately assess the quality of SRs in urology.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Yao, B., Ergun, O., Ding, M., Miller, C. D., Narayan, V. M., & Dahm, P. (2025). Assessing the methodologic quality of systematic reviews using generative large language models. Canadian Urological Association Journal, 19(12), E427–E433. https://doi.org/10.5489/cuaj.9243

MLA 9

Yao, Bowen, et al. “Assessing the methodologic quality of systematic reviews using generative large language models.” Canadian Urological Association Journal, vol. 19, no. 12, 2025, pp. E427–E433. https://doi.org/10.5489/cuaj.9243.

Chicago

Yao, Bowen, Onuralp Ergun, Maylynn Ding, Carly D. Miller, Vikram M. Narayan, and Philipp Dahm. 2025. “Assessing the methodologic quality of systematic reviews using generative large language models.” Canadian Urological Association Journal 19 (12): E427–E433. https://doi.org/10.5489/cuaj.9243.

Harvard

Yao, B. et al. (2025) ‘Assessing the methodologic quality of systematic reviews using generative large language models’, Canadian Urological Association Journal, 19(12), pp. E427–E433. doi:10.5489/cuaj.9243.

IEEE

B. Yao, O. Ergun, M. Ding, C. D. Miller, V. M. Narayan, and P. Dahm, “Assessing the methodologic quality of systematic reviews using generative large language models,” Canadian Urological Association Journal, vol. 19, no. 12, pp. E427–E433, 2025, doi: 10.5489/cuaj.9243.

BibTeX

@article{yao2025assessing,
  title     = {{Assessing the methodologic quality of systematic reviews using generative large language models}},
  author    = {Yao, Bowen and Ergun, Onuralp and Ding, Maylynn and Miller, Carly D. and Narayan, Vikram M. and Dahm, Philipp},
  journal   = {Canadian Urological Association Journal},
  year      = {2025},
  volume    = {19},
  number    = {12},
  pages     = {E427--E433},
  doi       = {10.5489/cuaj.9243},
  url       = {https://doi.org/10.5489/cuaj.9243}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews

  • systematic reviews
  • generative large language models
  • AMSTAR 2
  • urology
  • chain-of-thought prompting
  • zero-shot learning
  • human-AI agreement
  • internal validity

Work with this paper

Most-cited papers in Decision Sciences

All highly cited Decision Sciences papers

Questions about this paper

How do I cite this paper in APA style?

Yao, B., Ergun, O., Ding, M., Miller, C. D., Narayan, V. M., & Dahm, P. (2025). Assessing the methodologic quality of systematic reviews using generative large language models. Canadian Urological Association Journal, 19(12), E427–E433. https://doi.org/10.5489/cuaj.9243

How many times has this paper been cited?

2 times, according to OpenAlex on 18 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at Canadian Urological Association Journal: sign in and open the PDF from this page.

Who wrote this paper?

Bowen Yao, Onuralp Ergun, Maylynn Ding, Carly D. Miller, Vikram M. Narayan, Philipp Dahm.

What is the DOI of this paper?

10.5489/cuaj.9243, permanently at https://doi.org/10.5489/cuaj.9243. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 18 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.