Review article · 2025 · Open access

Use of artificial intelligence to support the assessment of the methodological quality of systematic reviews

Manuel Marques‐Cruz, Filipe Pinto, Rafael José Vieira, Antonio Bognanni, Paula Perestrelo, Sara Gil‐Mata, Vítor Henrique Duarte, José Pedro Barbosa, António Cardoso‐Fernandes, Daniel Martinho-Dias, Francisco Franco-Pêgo, Federico Germini and 11 more

Journal of Clinical Epidemiology, 187, 111944 · Published 25 August 2025 · DOI 10.1016/j.jclinepi.2025.111944

Open PDFView at the publisherCite this paper

In brief

“Use of artificial intelligence to support the assessment of the methodological quality of systematic reviews” is a 2025 review article by Manuel Marques‐Cruz and 22 co-authors, published in Journal of Clinical Epidemiology. OpenAlex counts 12 citations as of 18 November 2026, which puts it in the top 7% of works published in 2025 by citations. It is open access under a CC BY-NC-ND licence: the full text is free to read at Journal of Clinical Epidemiology.

Citations
12
Most cited in
2026 (9)
Field-weighted impact
0.98
References
11
Authors
23

Abstract

OBJECTIVES: Published systematic reviews display a heterogeneous methodological quality, which can impact decision-making. Large language models (LLMs) can support and make the assessment of the methodological quality of systematic reviews more efficient, aiding in the incorporation of their evidence in guideline recommendations. We aimed to develop an LLM-based tool for supporting the assessment of the methodological quality of systematic reviews. METHODS: We assessed the performance of 8 LLMs in evaluating the methodological quality of systematic reviews. In particular, we provided 100 systematic reviews for eight LLMs (five base models and three fine-tuned models) to evaluate their methodological quality based on a 27-item validated tool (Reported Methodological Quality (ReMarQ)). The fine-tuned models had been trained with a different sample of 300 manually assessed systematic reviews. We compared the answers provided by LLMs with those independently provided by human reviewers, computing the accuracy, kappa coefficient and F1-score for this comparison. RESULTS: The best performing LLM was a fine-tuned GPT-3.5 model (mean accuracy = 96.5% [95% CI = 89.9%-100%]; mean kappa coefficient = 0.90 [95% CI = 0.71-1.00]; mean F1-score = 0.91 [95% CI = 0.83-1.00]). This model displayed an accuracy >80% and a kappa coefficient >0.60 for all individual items. When we made this LLM assess 60 times the same set of systematic reviews, answers to 18 of 27 items were always consistent (ie, were always the same) and only 11% of assessed systematic reviews showed inconsistency. CONCLUSION: Overall, LLMs have the potential to accurately support the assessment of the methodological quality of systematic reviews based on a validated tool comprising dichotomous items.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Marques‐Cruz, M., Pinto, F., Vieira, R. J., Bognanni, A., Perestrelo, P., Gil‐Mata, S., Duarte, V. H., Barbosa, J. P., Cardoso‐Fernandes, A., Martinho-Dias, D., Franco-Pêgo, F., Germini, F., Arienti, C., Chu, A. W. L., Serra, P. R., Jemioło, P., Rodrigues, P. P., Fonseca, J., Azevedo, L. F., . . . Sousa‐Pinto, B. (2025). Use of artificial intelligence to support the assessment of the methodological quality of systematic reviews. Journal of Clinical Epidemiology, 187, 111944. https://doi.org/10.1016/j.jclinepi.2025.111944

MLA 9

Marques‐Cruz, Manuel, et al. “Use of artificial intelligence to support the assessment of the methodological quality of systematic reviews.” Journal of Clinical Epidemiology, vol. 187, 2025, p. 111944. https://doi.org/10.1016/j.jclinepi.2025.111944.

Chicago

Marques‐Cruz, Manuel, Filipe Pinto, Rafael José Vieira, Antonio Bognanni, Paula Perestrelo, Sara Gil‐Mata, Vítor Henrique Duarte, et al. 2025. “Use of artificial intelligence to support the assessment of the methodological quality of systematic reviews.” Journal of Clinical Epidemiology 187: 111944. https://doi.org/10.1016/j.jclinepi.2025.111944.

Harvard

Marques‐Cruz, M. et al. (2025) ‘Use of artificial intelligence to support the assessment of the methodological quality of systematic reviews’, Journal of Clinical Epidemiology, 187, p. 111944. doi:10.1016/j.jclinepi.2025.111944.

IEEE

M. Marques‐Cruz et al., “Use of artificial intelligence to support the assessment of the methodological quality of systematic reviews,” Journal of Clinical Epidemiology, vol. 187, p. 111944, 2025, doi: 10.1016/j.jclinepi.2025.111944.

BibTeX

@article{marquescruz2025artificial,
  title     = {{Use of artificial intelligence to support the assessment of the methodological quality of systematic reviews}},
  author    = {Marques‐Cruz, Manuel and Pinto, Filipe and Vieira, Rafael José and Bognanni, Antonio and Perestrelo, Paula and Gil‐Mata, Sara and Duarte, Vítor Henrique and Barbosa, José Pedro and Cardoso‐Fernandes, António and Martinho-Dias, Daniel and Franco-Pêgo, Francisco and Germini, Federico and Arienti, Chiara and Chu, Alexandro W. L. and Serra, Pau Riera and Jemioło, Paweł and Rodrigues, Pedro Pereira and Fonseca, João and Azevedo, Luís Filipe and Schünemann, Holger Jens and Cruz-Correia, Ricardo Joao and Mikhaylov, Slava and Sousa‐Pinto, Bernardo},
  journal   = {Journal of Clinical Epidemiology},
  year      = {2025},
  volume    = {187},
  pages     = {111944},
  doi       = {10.1016/j.jclinepi.2025.111944},
  url       = {https://doi.org/10.1016/j.jclinepi.2025.111944}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews

  • large language models
  • methodological quality assessment
  • systematic reviews
  • GPT-3.5
  • fine-tuned models
  • interobserver agreement
  • guideline development
  • evidence-based decision making

Work with this paper

Most-cited papers in Decision Sciences

All highly cited Decision Sciences papers

Questions about this paper

How do I cite this paper in APA style?

Marques‐Cruz, M., Pinto, F., Vieira, R. J., Bognanni, A., Perestrelo, P., Gil‐Mata, S., Duarte, V. H., Barbosa, J. P., Cardoso‐Fernandes, A., Martinho-Dias, D., Franco-Pêgo, F., Germini, F., Arienti, C., Chu, A. W. L., Serra, P. R., Jemioło, P., Rodrigues, P. P., Fonseca, J., Azevedo, L. F., . . . Sousa‐Pinto, B. (2025). Use of artificial intelligence to support the assessment of the methodological quality of systematic reviews. Journal of Clinical Epidemiology, 187, 111944. https://doi.org/10.1016/j.jclinepi.2025.111944

How many times has this paper been cited?

12 times, according to OpenAlex on 18 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at Journal of Clinical Epidemiology: sign in and open the PDF from this page.

Who wrote this paper?

Manuel Marques‐Cruz, Filipe Pinto, Rafael José Vieira, Antonio Bognanni, Paula Perestrelo, Sara Gil‐Mata, Vítor Henrique Duarte, José Pedro Barbosa, António Cardoso‐Fernandes, Daniel Martinho-Dias, Francisco Franco-Pêgo, Federico Germini and 11 more.

What is the DOI of this paper?

10.1016/j.jclinepi.2025.111944, permanently at https://doi.org/10.1016/j.jclinepi.2025.111944. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 18 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.