Review article · 2025 · Open access
Use of artificial intelligence to support the assessment of the methodological quality of systematic reviews
Journal of Clinical Epidemiology, 187, 111944 · Published 25 August 2025 · DOI 10.1016/j.jclinepi.2025.111944
In brief
“Use of artificial intelligence to support the assessment of the methodological quality of systematic reviews” is a 2025 review article by Manuel Marques‐Cruz and 22 co-authors, published in Journal of Clinical Epidemiology. OpenAlex counts 12 citations as of 18 November 2026, which puts it in the top 7% of works published in 2025 by citations. It is open access under a CC BY-NC-ND licence: the full text is free to read at Journal of Clinical Epidemiology.
- Citations
- 12
- Most cited in
- 2026 (9)
- Field-weighted impact
- 0.98
- References
- 11
- Authors
- 23
Abstract
OBJECTIVES: Published systematic reviews display a heterogeneous methodological quality, which can impact decision-making. Large language models (LLMs) can support and make the assessment of the methodological quality of systematic reviews more efficient, aiding in the incorporation of their evidence in guideline recommendations. We aimed to develop an LLM-based tool for supporting the assessment of the methodological quality of systematic reviews. METHODS: We assessed the performance of 8 LLMs in evaluating the methodological quality of systematic reviews. In particular, we provided 100 systematic reviews for eight LLMs (five base models and three fine-tuned models) to evaluate their methodological quality based on a 27-item validated tool (Reported Methodological Quality (ReMarQ)). The fine-tuned models had been trained with a different sample of 300 manually assessed systematic reviews. We compared the answers provided by LLMs with those independently provided by human reviewers, computing the accuracy, kappa coefficient and F1-score for this comparison. RESULTS: The best performing LLM was a fine-tuned GPT-3.5 model (mean accuracy = 96.5% [95% CI = 89.9%-100%]; mean kappa coefficient = 0.90 [95% CI = 0.71-1.00]; mean F1-score = 0.91 [95% CI = 0.83-1.00]). This model displayed an accuracy >80% and a kappa coefficient >0.60 for all individual items. When we made this LLM assess 60 times the same set of systematic reviews, answers to 18 of 27 items were always consistent (ie, were always the same) and only 11% of assessed systematic reviews showed inconsistency. CONCLUSION: Overall, LLMs have the potential to accurately support the assessment of the methodological quality of systematic reviews based on a validated tool comprising dichotomous items.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Marques‐Cruz, M., Pinto, F., Vieira, R. J., Bognanni, A., Perestrelo, P., Gil‐Mata, S., Duarte, V. H., Barbosa, J. P., Cardoso‐Fernandes, A., Martinho-Dias, D., Franco-Pêgo, F., Germini, F., Arienti, C., Chu, A. W. L., Serra, P. R., Jemioło, P., Rodrigues, P. P., Fonseca, J., Azevedo, L. F., . . . Sousa‐Pinto, B. (2025). Use of artificial intelligence to support the assessment of the methodological quality of systematic reviews. Journal of Clinical Epidemiology, 187, 111944. https://doi.org/10.1016/j.jclinepi.2025.111944
MLA 9
Marques‐Cruz, Manuel, et al. “Use of artificial intelligence to support the assessment of the methodological quality of systematic reviews.” Journal of Clinical Epidemiology, vol. 187, 2025, p. 111944. https://doi.org/10.1016/j.jclinepi.2025.111944.
Chicago
Marques‐Cruz, Manuel, Filipe Pinto, Rafael José Vieira, Antonio Bognanni, Paula Perestrelo, Sara Gil‐Mata, Vítor Henrique Duarte, et al. 2025. “Use of artificial intelligence to support the assessment of the methodological quality of systematic reviews.” Journal of Clinical Epidemiology 187: 111944. https://doi.org/10.1016/j.jclinepi.2025.111944.
Harvard
Marques‐Cruz, M. et al. (2025) ‘Use of artificial intelligence to support the assessment of the methodological quality of systematic reviews’, Journal of Clinical Epidemiology, 187, p. 111944. doi:10.1016/j.jclinepi.2025.111944.
IEEE
M. Marques‐Cruz et al., “Use of artificial intelligence to support the assessment of the methodological quality of systematic reviews,” Journal of Clinical Epidemiology, vol. 187, p. 111944, 2025, doi: 10.1016/j.jclinepi.2025.111944.
BibTeX
@article{marquescruz2025artificial,
title = {{Use of artificial intelligence to support the assessment of the methodological quality of systematic reviews}},
author = {Marques‐Cruz, Manuel and Pinto, Filipe and Vieira, Rafael José and Bognanni, Antonio and Perestrelo, Paula and Gil‐Mata, Sara and Duarte, Vítor Henrique and Barbosa, José Pedro and Cardoso‐Fernandes, António and Martinho-Dias, Daniel and Franco-Pêgo, Francisco and Germini, Federico and Arienti, Chiara and Chu, Alexandro W. L. and Serra, Pau Riera and Jemioło, Paweł and Rodrigues, Pedro Pereira and Fonseca, João and Azevedo, Luís Filipe and Schünemann, Holger Jens and Cruz-Correia, Ricardo Joao and Mikhaylov, Slava and Sousa‐Pinto, Bernardo},
journal = {Journal of Clinical Epidemiology},
year = {2025},
volume = {187},
pages = {111944},
doi = {10.1016/j.jclinepi.2025.111944},
url = {https://doi.org/10.1016/j.jclinepi.2025.111944}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- large language models
- methodological quality assessment
- systematic reviews
- GPT-3.5
- fine-tuned models
- interobserver agreement
- guideline development
- evidence-based decision making
Work with this paper
Most-cited papers in Decision Sciences
- The PRISMA 2020 statement: an updated guideline for reporting systematic reviewsMatthew James Page, Joanne E. McKenzie, Patrick M. Bossuyt and 23 more · BMJ · 2021 · 144,919 citations
- Evaluating Structural Equation Models with Unobservable Variables and Measurement ErrorClaes Fornell, David F. Larcker · Journal of Marketing Research · 1981 · 118,336 citations
- Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternativesLi‐tze Hu, Peter M. Bentler · Structural Equation Modeling A Multidisciplinary Journal · 1999 · 117,461 citations
- Common method biases in behavioral research: A critical review of the literature and recommended remedies.Philip M. Podsakoff, SCOTT B. MACKENZIE, Jeong Yeon Lee and 1 more · Journal of Applied Psychology · 2003 · 89,248 citations
- Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statementDavid Moher, Alessandro Liberati, Jennifer Tetzlaff and 2 more · BMJ · 2009 · 85,524 citations
Questions about this paper
How do I cite this paper in APA style?
Marques‐Cruz, M., Pinto, F., Vieira, R. J., Bognanni, A., Perestrelo, P., Gil‐Mata, S., Duarte, V. H., Barbosa, J. P., Cardoso‐Fernandes, A., Martinho-Dias, D., Franco-Pêgo, F., Germini, F., Arienti, C., Chu, A. W. L., Serra, P. R., Jemioło, P., Rodrigues, P. P., Fonseca, J., Azevedo, L. F., . . . Sousa‐Pinto, B. (2025). Use of artificial intelligence to support the assessment of the methodological quality of systematic reviews. Journal of Clinical Epidemiology, 187, 111944. https://doi.org/10.1016/j.jclinepi.2025.111944
How many times has this paper been cited?
12 times, according to OpenAlex on 18 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at Journal of Clinical Epidemiology: sign in and open the PDF from this page.
Who wrote this paper?
Manuel Marques‐Cruz, Filipe Pinto, Rafael José Vieira, Antonio Bognanni, Paula Perestrelo, Sara Gil‐Mata, Vítor Henrique Duarte, José Pedro Barbosa, António Cardoso‐Fernandes, Daniel Martinho-Dias, Francisco Franco-Pêgo, Federico Germini and 11 more.
What is the DOI of this paper?
10.1016/j.jclinepi.2025.111944, permanently at https://doi.org/10.1016/j.jclinepi.2025.111944. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 18 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.