Journal article · 2024 · Open access

Assessing the Risk of Bias in Randomized Clinical Trials With Large Language Models

Honghao Lai, Long Ge, Mingyao Sun, Bei Pan, Jiajie Huang, Liangying Hou, Qiuyu Yang, Jiayi Liu, Jianing Liu, Ziying Ye, Danni Xia, Weilong Zhao and 6 more

JAMA Network Open, 7(5), e2412687 · Published 22 May 2024 · DOI 10.1001/jamanetworkopen.2024.12687

Open PDFView at the publisherCite this paper

In brief

“Assessing the Risk of Bias in Randomized Clinical Trials With Large Language Models” is a 2024 journal article by Honghao Lai and 17 co-authors, published in JAMA Network Open. OpenAlex counts 136 citations as of 19 November 2026. It is open access under a CC BY licence: the full text is free to read at JAMA Network Open.

Citations
136
Most cited in
2025 (65)
Field-weighted impact
13.01
References
60
Authors
18

Abstract

Importance: Large language models (LLMs) may facilitate the labor-intensive process of systematic reviews. However, the exact methods and reliability remain uncertain. Objective: To explore the feasibility and reliability of using LLMs to assess risk of bias (ROB) in randomized clinical trials (RCTs). Design, Setting, and Participants: A survey study was conducted between August 10, 2023, and October 30, 2023. Thirty RCTs were selected from published systematic reviews. Main Outcomes and Measures: A structured prompt was developed to guide ChatGPT (LLM 1) and Claude (LLM 2) in assessing the ROB in these RCTs using a modified version of the Cochrane ROB tool developed by the CLARITY group at McMaster University. Each RCT was assessed twice by both models, and the results were documented. The results were compared with an assessment by 3 experts, which was considered a criterion standard. Correct assessment rates, sensitivity, specificity, and F1 scores were calculated to reflect accuracy, both overall and for each domain of the Cochrane ROB tool; consistent assessment rates and Cohen κ were calculated to gauge consistency; and assessment time was calculated to measure efficiency. Performance between the 2 models was compared using risk differences. Results: Both models demonstrated high correct assessment rates. LLM 1 reached a mean correct assessment rate of 84.5% (95% CI, 81.5%-87.3%), and LLM 2 reached a significantly higher rate of 89.5% (95% CI, 87.0%-91.8%). The risk difference between the 2 models was 0.05 (95% CI, 0.01-0.09). In most domains, domain-specific correct rates were around 80% to 90%; however, sensitivity below 0.80 was observed in domains 1 (random sequence generation), 2 (allocation concealment), and 6 (other concerns). Domains 4 (missing outcome data), 5 (selective outcome reporting), and 6 had F1 scores below 0.50. The consistent rates between the 2 assessments were 84.0% for LLM 1 and 87.3% for LLM 2. LLM 1's κ exceeded 0.80 in 7 and LLM 2's in 8 domains. The mean (SD) time needed for assessment was 77 (16) seconds for LLM 1 and 53 (12) seconds for LLM 2. Conclusions: In this survey study of applying LLMs for ROB assessment, LLM 1 and LLM 2 demonstrated substantial accuracy and consistency in evaluating RCTs, suggesting their potential as supportive tools in systematic review processes.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Lai, H., Ge, L., Sun, M., Pan, B., Huang, J., Hou, L., Yang, Q., Liu, J., Liu, J., Ye, Z., Xia, D., Zhao, W., Wang, X., Liu, M., Talukdar, J. R., Tian, J., Yang, K., & Estill, J. A. M. (2024). Assessing the Risk of Bias in Randomized Clinical Trials With Large Language Models. JAMA Network Open, 7(5), e2412687. https://doi.org/10.1001/jamanetworkopen.2024.12687

MLA 9

Lai, Honghao, et al. “Assessing the Risk of Bias in Randomized Clinical Trials With Large Language Models.” JAMA Network Open, vol. 7, no. 5, 2024, p. e2412687. https://doi.org/10.1001/jamanetworkopen.2024.12687.

Chicago

Lai, Honghao, Long Ge, Mingyao Sun, Bei Pan, Jiajie Huang, Liangying Hou, Qiuyu Yang, et al. 2024. “Assessing the Risk of Bias in Randomized Clinical Trials With Large Language Models.” JAMA Network Open 7 (5): e2412687. https://doi.org/10.1001/jamanetworkopen.2024.12687.

Harvard

Lai, H. et al. (2024) ‘Assessing the Risk of Bias in Randomized Clinical Trials With Large Language Models’, JAMA Network Open, 7(5), p. e2412687. doi:10.1001/jamanetworkopen.2024.12687.

IEEE

H. Lai et al., “Assessing the Risk of Bias in Randomized Clinical Trials With Large Language Models,” JAMA Network Open, vol. 7, no. 5, p. e2412687, 2024, doi: 10.1001/jamanetworkopen.2024.12687.

BibTeX

@article{lai2024assessing,
  title     = {{Assessing the Risk of Bias in Randomized Clinical Trials With Large Language Models}},
  author    = {Lai, Honghao and Ge, Long and Sun, Mingyao and Pan, Bei and Huang, Jiajie and Hou, Liangying and Yang, Qiuyu and Liu, Jiayi and Liu, Jianing and Ye, Ziying and Xia, Danni and Zhao, Weilong and Wang, Xiaoman and Liu, Ming and Talukdar, Jhalok Ronjan and Tian, Jinhui and Yang, Kehu and Estill, Janne Anton Markus},
  journal   = {JAMA Network Open},
  year      = {2024},
  volume    = {7},
  number    = {5},
  pages     = {e2412687},
  doi       = {10.1001/jamanetworkopen.2024.12687},
  url       = {https://doi.org/10.1001/jamanetworkopen.2024.12687}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Citations by year

YearCitationsShare of the best year
202311
202455
20256565
20266565

Its field-weighted citation impact is 13.01: it has been cited 13.01 times as often as the average work of the same field, type and year (1.00 is the world average).

Topic and keywords

Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews

  • risk of bias assessment
  • large language models
  • ChatGPT
  • Claude
  • randomized controlled trials
  • Cochrane risk of bias tool
  • systematic review methodology
  • model comparison

Work with this paper

Most-cited papers in Decision Sciences

All highly cited Decision Sciences papers

Questions about this paper

How do I cite this paper in APA style?

Lai, H., Ge, L., Sun, M., Pan, B., Huang, J., Hou, L., Yang, Q., Liu, J., Liu, J., Ye, Z., Xia, D., Zhao, W., Wang, X., Liu, M., Talukdar, J. R., Tian, J., Yang, K., & Estill, J. A. M. (2024). Assessing the Risk of Bias in Randomized Clinical Trials With Large Language Models. JAMA Network Open, 7(5), e2412687. https://doi.org/10.1001/jamanetworkopen.2024.12687

How many times has this paper been cited?

136 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at JAMA Network Open: sign in and open the PDF from this page.

Who wrote this paper?

Honghao Lai, Long Ge, Mingyao Sun, Bei Pan, Jiajie Huang, Liangying Hou, Qiuyu Yang, Jiayi Liu, Jianing Liu, Ziying Ye, Danni Xia, Weilong Zhao and 6 more.

What is the DOI of this paper?

10.1001/jamanetworkopen.2024.12687, permanently at https://doi.org/10.1001/jamanetworkopen.2024.12687. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.