Journal article · 2026 · Open access
Evaluating large language model performance in Risk of Bias assessments: A cross-sectional validation study
PLoS ONE, 21(7), e0353155 · Published 9 July 2026 · DOI 10.1371/journal.pone.0353155
In brief
“Evaluating large language model performance in Risk of Bias assessments: A cross-sectional validation study” is a 2026 journal article by Siddharth Gandhi and 3 co-authors, published in PLoS ONE. OpenAlex counts 1 citations as of 19 November 2026. It is open access under a CC BY licence: the full text is free to read at PLoS ONE.
- Citations
- 1
- Most cited in
- 2026 (1)
- Field-weighted impact
- 1.53
- References
- 23
- Authors
- 4
Abstract
OBJECTIVE: To evaluate the reliability and diagnostic performance of ChatGPT-o3 in conducting Risk of Bias (RoB) assessments of randomized clinical trials (RCTs) using the Cochrane RoB 2.0 tool. MATERIALS AND METHODS: This methodological validation study analyzed 50 RCTs sampled from 50 published meta-analyses. Each trial was independently assessed by the original systematic review authors (OSRAs), our masked human panel, and ChatGPT-o3. Structured prompts based on RoB 2.0 guidelines were used to elicit ChatGPT-o3 assessments. Agreement was evaluated using weighted Cohen's kappa and Gwet's AC2. Diagnostic performance was measured by sensitivity, specificity, and balanced accuracy, with human ratings as the reference. RESULTS: ChatGPT-o3 classified 34% of trials as high risk, compared with 22% by our panel, and 12% by the OSRAs. Agreement was modest (median κ: 0.33 with our panel; 0.14 with OSRAs). Overall Gwet's AC2 was 0.30. For detecting high-risk trials, ChatGPT-o3 achieved a sensitivity of 0.46, specificity of 0.69, and balanced accuracy of 0.57. For low-risk trials, its sensitivity was 0.47, specificity was 0.86, and balanced accuracy was 0.66. DISCUSSION: The results indicate that ChatGPT-o3 produced more conservative RoB ratings than human reviewers, identifying a greater percentage of trials as having a high RoB. CONCLUSION: While unsuitable to be used as a sole assessor, ChatGPT-o3 may serve as an adjunct tool to enhance the efficiency and consistency of RoB assessments in systematic reviews.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Gandhi, S., Shokravi, A., Chelliahpillai, Y., & Balas, M. (2026). Evaluating large language model performance in Risk of Bias assessments: A cross-sectional validation study. PLoS ONE, 21(7), e0353155. https://doi.org/10.1371/journal.pone.0353155
MLA 9
Gandhi, Siddharth, et al. “Evaluating large language model performance in Risk of Bias assessments: A cross-sectional validation study.” PLoS ONE, vol. 21, no. 7, 2026, p. e0353155. https://doi.org/10.1371/journal.pone.0353155.
Chicago
Gandhi, Siddharth, Arveen Shokravi, Yashan Chelliahpillai, and Michael Balas. 2026. “Evaluating large language model performance in Risk of Bias assessments: A cross-sectional validation study.” PLoS ONE 21 (7): e0353155. https://doi.org/10.1371/journal.pone.0353155.
Harvard
Gandhi, S. et al. (2026) ‘Evaluating large language model performance in Risk of Bias assessments: A cross-sectional validation study’, PLoS ONE, 21(7), p. e0353155. doi:10.1371/journal.pone.0353155.
IEEE
S. Gandhi, A. Shokravi, Y. Chelliahpillai, and M. Balas, “Evaluating large language model performance in Risk of Bias assessments: A cross-sectional validation study,” PLoS ONE, vol. 21, no. 7, p. e0353155, 2026, doi: 10.1371/journal.pone.0353155.
BibTeX
@article{gandhi2026evaluating,
title = {{Evaluating large language model performance in Risk of Bias assessments: A cross-sectional validation study}},
author = {Gandhi, Siddharth and Shokravi, Arveen and Chelliahpillai, Yashan and Balas, Michael},
journal = {PLoS ONE},
year = {2026},
volume = {21},
number = {7},
pages = {e0353155},
doi = {10.1371/journal.pone.0353155},
url = {https://doi.org/10.1371/journal.pone.0353155}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- large language models
- ChatGPT
- risk of bias assessment
- randomized controlled trials
- diagnostic accuracy
- automated risk assessment
- systematic review methodology
- meta-analysis
Work with this paper
Most-cited papers in Decision Sciences
- The PRISMA 2020 statement: an updated guideline for reporting systematic reviewsMatthew James Page, Joanne E. McKenzie, Patrick M. Bossuyt and 23 more · BMJ · 2021 · 144,919 citations
- Evaluating Structural Equation Models with Unobservable Variables and Measurement ErrorClaes Fornell, David F. Larcker · Journal of Marketing Research · 1981 · 118,336 citations
- Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternativesLi‐tze Hu, Peter M. Bentler · Structural Equation Modeling A Multidisciplinary Journal · 1999 · 117,461 citations
- Common method biases in behavioral research: A critical review of the literature and recommended remedies.Philip M. Podsakoff, SCOTT B. MACKENZIE, Jeong Yeon Lee and 1 more · Journal of Applied Psychology · 2003 · 89,248 citations
- Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statementDavid Moher, Alessandro Liberati, Jennifer Tetzlaff and 2 more · BMJ · 2009 · 85,524 citations
Questions about this paper
How do I cite this paper in APA style?
Gandhi, S., Shokravi, A., Chelliahpillai, Y., & Balas, M. (2026). Evaluating large language model performance in Risk of Bias assessments: A cross-sectional validation study. PLoS ONE, 21(7), e0353155. https://doi.org/10.1371/journal.pone.0353155
How many times has this paper been cited?
1 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at PLoS ONE: sign in and open the PDF from this page.
Who wrote this paper?
Siddharth Gandhi, Arveen Shokravi, Yashan Chelliahpillai, Michael Balas.
What is the DOI of this paper?
10.1371/journal.pone.0353155, permanently at https://doi.org/10.1371/journal.pone.0353155. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.