Journal article · 2026 · Open access

Evaluating large language model performance in Risk of Bias assessments: A cross-sectional validation study

Siddharth Gandhi, Arveen Shokravi, Yashan Chelliahpillai, Michael Balas

PLoS ONE, 21(7), e0353155 · Published 9 July 2026 · DOI 10.1371/journal.pone.0353155

Open PDFView at the publisherCite this paper

In brief

“Evaluating large language model performance in Risk of Bias assessments: A cross-sectional validation study” is a 2026 journal article by Siddharth Gandhi and 3 co-authors, published in PLoS ONE. OpenAlex counts 1 citations as of 19 November 2026. It is open access under a CC BY licence: the full text is free to read at PLoS ONE.

Citations
1
Most cited in
2026 (1)
Field-weighted impact
1.53
References
23
Authors
4

Abstract

OBJECTIVE: To evaluate the reliability and diagnostic performance of ChatGPT-o3 in conducting Risk of Bias (RoB) assessments of randomized clinical trials (RCTs) using the Cochrane RoB 2.0 tool. MATERIALS AND METHODS: This methodological validation study analyzed 50 RCTs sampled from 50 published meta-analyses. Each trial was independently assessed by the original systematic review authors (OSRAs), our masked human panel, and ChatGPT-o3. Structured prompts based on RoB 2.0 guidelines were used to elicit ChatGPT-o3 assessments. Agreement was evaluated using weighted Cohen's kappa and Gwet's AC2. Diagnostic performance was measured by sensitivity, specificity, and balanced accuracy, with human ratings as the reference. RESULTS: ChatGPT-o3 classified 34% of trials as high risk, compared with 22% by our panel, and 12% by the OSRAs. Agreement was modest (median κ: 0.33 with our panel; 0.14 with OSRAs). Overall Gwet's AC2 was 0.30. For detecting high-risk trials, ChatGPT-o3 achieved a sensitivity of 0.46, specificity of 0.69, and balanced accuracy of 0.57. For low-risk trials, its sensitivity was 0.47, specificity was 0.86, and balanced accuracy was 0.66. DISCUSSION: The results indicate that ChatGPT-o3 produced more conservative RoB ratings than human reviewers, identifying a greater percentage of trials as having a high RoB. CONCLUSION: While unsuitable to be used as a sole assessor, ChatGPT-o3 may serve as an adjunct tool to enhance the efficiency and consistency of RoB assessments in systematic reviews.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Gandhi, S., Shokravi, A., Chelliahpillai, Y., & Balas, M. (2026). Evaluating large language model performance in Risk of Bias assessments: A cross-sectional validation study. PLoS ONE, 21(7), e0353155. https://doi.org/10.1371/journal.pone.0353155

MLA 9

Gandhi, Siddharth, et al. “Evaluating large language model performance in Risk of Bias assessments: A cross-sectional validation study.” PLoS ONE, vol. 21, no. 7, 2026, p. e0353155. https://doi.org/10.1371/journal.pone.0353155.

Chicago

Gandhi, Siddharth, Arveen Shokravi, Yashan Chelliahpillai, and Michael Balas. 2026. “Evaluating large language model performance in Risk of Bias assessments: A cross-sectional validation study.” PLoS ONE 21 (7): e0353155. https://doi.org/10.1371/journal.pone.0353155.

Harvard

Gandhi, S. et al. (2026) ‘Evaluating large language model performance in Risk of Bias assessments: A cross-sectional validation study’, PLoS ONE, 21(7), p. e0353155. doi:10.1371/journal.pone.0353155.

IEEE

S. Gandhi, A. Shokravi, Y. Chelliahpillai, and M. Balas, “Evaluating large language model performance in Risk of Bias assessments: A cross-sectional validation study,” PLoS ONE, vol. 21, no. 7, p. e0353155, 2026, doi: 10.1371/journal.pone.0353155.

BibTeX

@article{gandhi2026evaluating,
  title     = {{Evaluating large language model performance in Risk of Bias assessments: A cross-sectional validation study}},
  author    = {Gandhi, Siddharth and Shokravi, Arveen and Chelliahpillai, Yashan and Balas, Michael},
  journal   = {PLoS ONE},
  year      = {2026},
  volume    = {21},
  number    = {7},
  pages     = {e0353155},
  doi       = {10.1371/journal.pone.0353155},
  url       = {https://doi.org/10.1371/journal.pone.0353155}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews

  • large language models
  • ChatGPT
  • risk of bias assessment
  • randomized controlled trials
  • diagnostic accuracy
  • automated risk assessment
  • systematic review methodology
  • meta-analysis

Work with this paper

Most-cited papers in Decision Sciences

All highly cited Decision Sciences papers

Questions about this paper

How do I cite this paper in APA style?

Gandhi, S., Shokravi, A., Chelliahpillai, Y., & Balas, M. (2026). Evaluating large language model performance in Risk of Bias assessments: A cross-sectional validation study. PLoS ONE, 21(7), e0353155. https://doi.org/10.1371/journal.pone.0353155

How many times has this paper been cited?

1 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at PLoS ONE: sign in and open the PDF from this page.

Who wrote this paper?

Siddharth Gandhi, Arveen Shokravi, Yashan Chelliahpillai, Michael Balas.

What is the DOI of this paper?

10.1371/journal.pone.0353155, permanently at https://doi.org/10.1371/journal.pone.0353155. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.