Journal article · 2026 · Open access

Evaluating the Accuracy of Large Language Models in Risk-of-Bias Assessment Using Version 2 of the Cochrane Risk-of-Bias Tool for Randomized Trials: Exploratory Feasibility Study

Yu-Ju Lai, Shen-Hua Lin, Jen-Wei Liu

Journal of Medical Internet Research, 28, e84915 · Published 25 September 2026 · DOI 10.2196/84915

Open full textCite this paper

In brief

“Evaluating the Accuracy of Large Language Models in Risk-of-Bias Assessment Using Version 2 of the Cochrane Risk-of-Bias Tool for Randomized Trials: Exploratory Feasibility Study” is a 2026 journal article by Yu-Ju Lai and 2 co-authors, published in Journal of Medical Internet Research. OpenAlex counts 0 citations as of 19 November 2026. It is open access under a CC BY licence: the full text is free to read at Journal of Medical Internet Research.

Citations
0
References
55
Authors
3

Abstract

Background Large language models (LLMs) have the potential to improve the efficiency of evidence synthesis, but their reliability in performing complex tasks such as risk-of-bias (ROB) assessment in randomized controlled trials (RCTs) remains unclear. Objective This study aimed to evaluate whether LLMs can reliably assess ROB in RCTs using version 2 of the Cochrane ROB tool for randomized trials (ROB 2). Methods This study was conducted between December 28, 2024, and February 28, 2025, in adherence to American Association for Public Opinion Research reporting guidelines. Twenty-nine RCTs were selected from published Cochrane systematic reviews across diverse medical fields. We developed a structured prompt engineering framework that transformed ROB 2 decision trees into logical rules for the LLM. Each RCT was independently evaluated twice by ChatGPT, with Cochrane review authors’ assessments serving as the reference standard for comparison. The main outcomes were the accuracy and consistency of ROB 2 assessments at both the domain and trial levels, evaluated using accuracy, sensitivity, specificity, and F 1 -score. Consistency between the repeated assessments was quantified using the Cohen κ and prevalence-adjusted, bias-adjusted κ. Results The LLM demonstrated a moderate aggregate domain accuracy of 73.1% (95% CI 64.7%-81.5%) in the first assessment and 75.9% (95% CI 66.3%-85.4%) in the second assessment. Domain-averaged sensitivity decreased from 61.4% (95% CI 48.1%‐74.7%) to 53.4% (95% CI 41.7%‐65.0%), whereas domain-averaged specificity increased from 75.8% (95% CI 65.3%-86.3%) to 81.1% (95%CI 67.1%-95%), indicating a conservative tendency in identifying a high ROB. Domain-level accuracy ranged from 62.1% to 87.9%, with the lowest accuracy observed in domain 1 and the lowest F 1 -scores observed in domain 2. Consistency between repeated assessments was high, with a mean agreement of 89.0% (SD 7.5%), and Cohen κ values were 0.86, 0.39, 0.56, 0.84, and 0.85 in domains 1 to 5, respectively. Conclusions In this exploratory study, ChatGPT demonstrated moderate accuracy and high consistency in assessing ROB in RCTs using the ROB 2. However, its reliability diminished in complex scenarios requiring interpretation of implicit narratives or behavioral nuance. These findings suggest that LLMs may support methodological evaluations in systematic reviews by acting as automated screeners to reduce reviewer burden, but current implementation still requires expert oversight, particularly for trials involving subjective outcomes or nonstandard reporting.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Lai, Y.-J., Lin, S.-H., & Liu, J.-W. (2026). Evaluating the Accuracy of Large Language Models in Risk-of-Bias Assessment Using Version 2 of the Cochrane Risk-of-Bias Tool for Randomized Trials: Exploratory Feasibility Study. Journal of Medical Internet Research, 28, e84915. https://doi.org/10.2196/84915

MLA 9

Lai, Yu-Ju, et al. “Evaluating the Accuracy of Large Language Models in Risk-of-Bias Assessment Using Version 2 of the Cochrane Risk-of-Bias Tool for Randomized Trials: Exploratory Feasibility Study.” Journal of Medical Internet Research, vol. 28, 2026, p. e84915. https://doi.org/10.2196/84915.

Chicago

Lai, Yu-Ju, Shen-Hua Lin, and Jen-Wei Liu. 2026. “Evaluating the Accuracy of Large Language Models in Risk-of-Bias Assessment Using Version 2 of the Cochrane Risk-of-Bias Tool for Randomized Trials: Exploratory Feasibility Study.” Journal of Medical Internet Research 28: e84915. https://doi.org/10.2196/84915.

Harvard

Lai, Y.-J., Lin, S.-H. and Liu, J.-W. (2026) ‘Evaluating the Accuracy of Large Language Models in Risk-of-Bias Assessment Using Version 2 of the Cochrane Risk-of-Bias Tool for Randomized Trials: Exploratory Feasibility Study’, Journal of Medical Internet Research, 28, p. e84915. doi:10.2196/84915.

IEEE

Y.-J. Lai, S.-H. Lin, and J.-W. Liu, “Evaluating the Accuracy of Large Language Models in Risk-of-Bias Assessment Using Version 2 of the Cochrane Risk-of-Bias Tool for Randomized Trials: Exploratory Feasibility Study,” Journal of Medical Internet Research, vol. 28, p. e84915, 2026, doi: 10.2196/84915.

BibTeX

@article{lai2026evaluating,
  title     = {{Evaluating the Accuracy of Large Language Models in Risk-of-Bias Assessment Using Version 2 of the Cochrane Risk-of-Bias Tool for Randomized Trials: Exploratory Feasibility Study}},
  author    = {Lai, Yu-Ju and Lin, Shen-Hua and Liu, Jen-Wei},
  journal   = {Journal of Medical Internet Research},
  year      = {2026},
  volume    = {28},
  pages     = {e84915},
  doi       = {10.2196/84915},
  url       = {https://doi.org/10.2196/84915}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews

  • large language models
  • risk of bias assessment
  • randomized controlled trials
  • ChatGPT
  • prompt engineering
  • inter-assessor agreement
  • evidence synthesis
  • sensitivity and specificity

Work with this paper

Most-cited papers in Decision Sciences

All highly cited Decision Sciences papers

Questions about this paper

How do I cite this paper in APA style?

Lai, Y.-J., Lin, S.-H., & Liu, J.-W. (2026). Evaluating the Accuracy of Large Language Models in Risk-of-Bias Assessment Using Version 2 of the Cochrane Risk-of-Bias Tool for Randomized Trials: Exploratory Feasibility Study. Journal of Medical Internet Research, 28, e84915. https://doi.org/10.2196/84915

How many times has this paper been cited?

0 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at Journal of Medical Internet Research: sign in and open the full text from this page.

Who wrote this paper?

Yu-Ju Lai, Shen-Hua Lin, Jen-Wei Liu.

What is the DOI of this paper?

10.2196/84915, permanently at https://doi.org/10.2196/84915. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.