Journal article · 2025 · Open access

Large Language Model–Assisted Risk-of-Bias Assessment in Randomized Controlled Trials Using the Revised Risk-of-Bias Tool: Evaluation Study

Jiajie Huang, Honghao Lai, Weilong Zhao, Danni Xia, Chunyang Bai, Mingyao Sun, Jianing Liu, Jiayi Liu, Bei Pan, Jinhui Tian, Long Ge

Journal of Medical Internet Research, 27, e70450 · Published 5 May 2025 · DOI 10.2196/70450

Open full textCite this paper

In brief

“Large Language Model–Assisted Risk-of-Bias Assessment in Randomized Controlled Trials Using the Revised Risk-of-Bias Tool: Evaluation Study” is a 2025 journal article by Jiajie Huang and 10 co-authors, published in Journal of Medical Internet Research. OpenAlex counts 31 citations as of 19 November 2026, which puts it in the top 2% of works published in 2025 by citations. It is open access under a CC BY licence: the full text is free to read at Journal of Medical Internet Research.

Citations
31
Most cited in
2026 (25)
Field-weighted impact
5.66
References
58
Authors
11

Abstract

BACKGROUND: The revised Risk-of-Bias tool (RoB2) overcomes the limitations of its predecessor but introduces new implementation challenges. Studies demonstrate low interrater reliability and substantial time requirements for RoB2 implementation. Large language models (LLMs) may assist in RoB2 implementation, although their effectiveness remains uncertain. OBJECTIVE: This study aims to evaluate the accuracy of LLMs in RoB2 assessments to explore their potential as research assistants for bias evaluation. METHODS: We systematically searched the Cochrane Library (through October 2023) for reviews using RoB2, categorized by interest in adhering or assignment. From 86 eligible reviews of randomized controlled trials (covering 1399 RCTs), we randomly selected 46 RCTs (23 per category). In addition, 3 experienced reviewers independently assessed all 46 RCTs using RoB2, recording assessment time for each trial. Reviewer judgments were reconciled through consensus. Furthermore, 6 RCTs (3 from each category) were randomly selected for prompt development and optimization. The remaining 40 trials established the internal validation standard, while Cochrane Reviews judgments served as external validation. Primary outcomes were extracted as reported in corresponding Cochrane Reviews. We calculated accuracy rates, Cohen κ, and time differentials. RESULTS: We identified significant differences between Cochrane and reviewer judgments, particularly in domains 1, 4, and 5, likely due to different standards in assessing randomization and blinding. Among the 20 articles focusing on adhering, 18 Cochrane Reviews and 19 reviewer judgments classified them as "High risk," while assignment-focused RCTs showed more heterogeneous risk distribution. Compared with Cochrane Reviews, LLMs demonstrated accuracy rates of 57.5% and 70% for overall (assignment) and overall (adhering), respectively. When compared with reviewer judgments, LLMs' accuracy rates were 65% and 70% for these domains. The average accuracy rates for the remaining 6 domains were 65.2% (95% CI 57.6-72.7) against Cochrane Reviews and 74.2% (95% CI 64.7-83.9) against reviewers. At the signaling question level, LLMs achieved 83.2% average accuracy (95% CI 77.5-88.9), with accuracy exceeding 70% for most questions except 2.4 (assignment), 2.5 (assignment), 3.3, and 3.4. When domain judgments were derived from LLM-generated signaling questions using the RoB2 algorithm rather than direct LLM domain judgments, accuracy improved substantially for Domain 2 (adhering; 55-95) and overall (adhering; 70-90). LLMs demonstrated high consistency between iterations (average 85.2%, 95% CI 85.15-88.79) and completed assessments in 1.9 minutes versus 31.5 minutes for human reviewers (mean difference 29.6, 95% CI 25.6-33.6 minutes). CONCLUSIONS: LLMs achieved commendable accuracy when guided by structured prompts, particularly through processing methodological details through structured reasoning. While not replacing human assessment, LLMs demonstrate strong potential for assisting RoB2 evaluations. Larger studies with improved prompting could enhance…

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Huang, J., Lai, H., Zhao, W., Xia, D., Bai, C., Sun, M., Liu, J., Liu, J., Pan, B., Tian, J., & Ge, L. (2025). Large Language Model–Assisted Risk-of-Bias Assessment in Randomized Controlled Trials Using the Revised Risk-of-Bias Tool: Evaluation Study. Journal of Medical Internet Research, 27, e70450. https://doi.org/10.2196/70450

MLA 9

Huang, Jiajie, et al. “Large Language Model–Assisted Risk-of-Bias Assessment in Randomized Controlled Trials Using the Revised Risk-of-Bias Tool: Evaluation Study.” Journal of Medical Internet Research, vol. 27, 2025, p. e70450. https://doi.org/10.2196/70450.

Chicago

Huang, Jiajie, Honghao Lai, Weilong Zhao, Danni Xia, Chunyang Bai, Mingyao Sun, Jianing Liu, et al. 2025. “Large Language Model–Assisted Risk-of-Bias Assessment in Randomized Controlled Trials Using the Revised Risk-of-Bias Tool: Evaluation Study.” Journal of Medical Internet Research 27: e70450. https://doi.org/10.2196/70450.

Harvard

Huang, J. et al. (2025) ‘Large Language Model–Assisted Risk-of-Bias Assessment in Randomized Controlled Trials Using the Revised Risk-of-Bias Tool: Evaluation Study’, Journal of Medical Internet Research, 27, p. e70450. doi:10.2196/70450.

IEEE

J. Huang et al., “Large Language Model–Assisted Risk-of-Bias Assessment in Randomized Controlled Trials Using the Revised Risk-of-Bias Tool: Evaluation Study,” Journal of Medical Internet Research, vol. 27, p. e70450, 2025, doi: 10.2196/70450.

BibTeX

@article{huang2025large,
  title     = {{Large Language Model–Assisted Risk-of-Bias Assessment in Randomized Controlled Trials Using the Revised Risk-of-Bias Tool: Evaluation Study}},
  author    = {Huang, Jiajie and Lai, Honghao and Zhao, Weilong and Xia, Danni and Bai, Chunyang and Sun, Mingyao and Liu, Jianing and Liu, Jiayi and Pan, Bei and Tian, Jinhui and Ge, Long},
  journal   = {Journal of Medical Internet Research},
  year      = {2025},
  volume    = {27},
  pages     = {e70450},
  doi       = {10.2196/70450},
  url       = {https://doi.org/10.2196/70450}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews

  • large language models
  • risk of bias assessment
  • RoB 2
  • randomized controlled trials
  • research assistants
  • inter-rater reliability
  • blinding
  • assessment time

Work with this paper

Most-cited papers in Decision Sciences

All highly cited Decision Sciences papers

Questions about this paper

How do I cite this paper in APA style?

Huang, J., Lai, H., Zhao, W., Xia, D., Bai, C., Sun, M., Liu, J., Liu, J., Pan, B., Tian, J., & Ge, L. (2025). Large Language Model–Assisted Risk-of-Bias Assessment in Randomized Controlled Trials Using the Revised Risk-of-Bias Tool: Evaluation Study. Journal of Medical Internet Research, 27, e70450. https://doi.org/10.2196/70450

How many times has this paper been cited?

31 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at Journal of Medical Internet Research: sign in and open the full text from this page.

Who wrote this paper?

Jiajie Huang, Honghao Lai, Weilong Zhao, Danni Xia, Chunyang Bai, Mingyao Sun, Jianing Liu, Jiayi Liu, Bei Pan, Jinhui Tian, Long Ge.

What is the DOI of this paper?

10.2196/70450, permanently at https://doi.org/10.2196/70450. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.