Preprint · 2026 · Open access
Evaluation of Systematic Review Methodological Quality with Large Language Models Versus Human Reviewers
Preprints.org · Published 22 September 2026 · DOI 10.20944/preprints202609.1913.v1
In brief
“Evaluation of Systematic Review Methodological Quality with Large Language Models Versus Human Reviewers” is a 2026 preprint by Sarah Alexandra Elliott and 3 co-authors, published in Preprints.org. OpenAlex counts 0 citations as of 18 November 2026. It is open access under a CC BY licence: the full text is free to read at Preprints.org.
- Citations
- 0
- Authors
- 4
Abstract
Background: There continues to be a growing number of systematic reviews with variable methodological quality. Using a systematic review to inform decision making relies on the quality of methods used to synthesize evidence. Tools that have been developed, such as AMSTAR 2, aim to assess the methodological quality and validity of systematic reviews of healthcare interventions. As this task takes time, use of large language models (LLMs) may help expedite the process. Objectives: We aim to test the ability of two LLMs, Claude Opus 5 and ChatGPT 5.5, to perform AMSTAR 2 assessments on a set of systematic reviews and to compare the accuracy to standard methods using human reviewers. As a secondary objective, we aim to explore the temporal consistency of the AMSTAR 2 assessments from each LLM over time (baseline and one month later).Methods: Our study sample includes open-access systematic reviews of interventions in pediatric populations. Our primary outcome is accuracy of each LLM’s AMSTAR 2 assessments as compared to a human reviewer process as the reference standard. Accuracy will be calculated as the total number of assessments of AMSTAR 2 items that correctly match the human reviewer team. Two humans will independently perform AMSTAR 2 assessments and come to consensus through discussion, or involvement with a third senior reviewer when required. A randomly selected subset of the systematic reviews will be used for the secondary objective exploring temporal consistency of the same LLM model versions over time. Results: AMSTAR 2 assessments will be performed on all 45 systematic reviews by each LLM and two human reviewers. Accuracy of the LLMs versus the human reference standard will be recorded for each AMSTAR 2 item. We will also identify whether there are common discrepancies for specific AMSTAR 2 items, and possible reasons (e.g., items requiring methodological judgement, vague methods reported in the systematic reviews, etc.). The temporal consistency for each model over time will be summarized narratively with descriptive statistics.Conclusion: Evaluating the methodological quality of systematic reviews with LLMs has the potential to more efficiently respond to the needs of decision-makers, by offsetting time spent performing AMSTAR 2 assessments, while maintaining accuracy. Our study will provide evidence to support or reject the use of LLMs to replace human reviewers in this process based on the accuracy of assessments, as well as whether responses are consistent over time. We will also identify AMSTAR 2 items that appear consistently problematic, where decision rules or adequate prompts are needed.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Elliott, S. A., Guitard, S., Ramos, A., & Hartling, L. (2026). Evaluation of Systematic Review Methodological Quality with Large Language Models Versus Human Reviewers. Preprints.org. https://doi.org/10.20944/preprints202609.1913.v1
MLA 9
Elliott, Sarah Alexandra, et al. “Evaluation of Systematic Review Methodological Quality with Large Language Models Versus Human Reviewers.” Preprints.org, 2026. https://doi.org/10.20944/preprints202609.1913.v1.
Chicago
Elliott, Sarah Alexandra, Samantha Guitard, Angelika Ramos, and Lisa Hartling. 2026. “Evaluation of Systematic Review Methodological Quality with Large Language Models Versus Human Reviewers.” Preprints.org. https://doi.org/10.20944/preprints202609.1913.v1.
Harvard
Elliott, S. A. et al. (2026) ‘Evaluation of Systematic Review Methodological Quality with Large Language Models Versus Human Reviewers’, Preprints.org. doi:10.20944/preprints202609.1913.v1.
IEEE
S. A. Elliott, S. Guitard, A. Ramos, and L. Hartling, “Evaluation of Systematic Review Methodological Quality with Large Language Models Versus Human Reviewers,” Preprints.org, 2026, doi: 10.20944/preprints202609.1913.v1.
BibTeX
@misc{elliott2026evaluation,
title = {{Evaluation of Systematic Review Methodological Quality with Large Language Models Versus Human Reviewers}},
author = {Elliott, Sarah Alexandra and Guitard, Samantha and Ramos, Angelika and Hartling, Lisa},
howpublished = {Preprints.org},
year = {2026},
doi = {10.20944/preprints202609.1913.v1},
url = {https://doi.org/10.20944/preprints202609.1913.v1}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- large language models
- AMSTAR 2
- automated quality assessment
- temporal coherence
- ChatGPT
- healthcare decision-making
Work with this paper
Most-cited papers in Decision Sciences
- The PRISMA 2020 statement: an updated guideline for reporting systematic reviewsMatthew James Page, Joanne E. McKenzie, Patrick M. Bossuyt and 23 more · BMJ · 2021 · 144,919 citations
- Evaluating Structural Equation Models with Unobservable Variables and Measurement ErrorClaes Fornell, David F. Larcker · Journal of Marketing Research · 1981 · 118,336 citations
- Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternativesLi‐tze Hu, Peter M. Bentler · Structural Equation Modeling A Multidisciplinary Journal · 1999 · 117,461 citations
- Common method biases in behavioral research: A critical review of the literature and recommended remedies.Philip M. Podsakoff, SCOTT B. MACKENZIE, Jeong Yeon Lee and 1 more · Journal of Applied Psychology · 2003 · 89,248 citations
- Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statementDavid Moher, Alessandro Liberati, Jennifer Tetzlaff and 2 more · BMJ · 2009 · 85,524 citations
Questions about this paper
How do I cite this paper in APA style?
Elliott, S. A., Guitard, S., Ramos, A., & Hartling, L. (2026). Evaluation of Systematic Review Methodological Quality with Large Language Models Versus Human Reviewers. Preprints.org. https://doi.org/10.20944/preprints202609.1913.v1
How many times has this paper been cited?
0 times, according to OpenAlex on 18 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at Preprints.org: sign in and open the full text from this page.
Who wrote this paper?
Sarah Alexandra Elliott, Samantha Guitard, Angelika Ramos, Lisa Hartling.
What is the DOI of this paper?
10.20944/preprints202609.1913.v1, permanently at https://doi.org/10.20944/preprints202609.1913.v1. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 18 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.