Journal article · 2024 · Open access
Assessing the Risk of Bias in Randomized Clinical Trials With Large Language Models
JAMA Network Open, 7(5), e2412687 · Published 22 May 2024 · DOI 10.1001/jamanetworkopen.2024.12687
In brief
“Assessing the Risk of Bias in Randomized Clinical Trials With Large Language Models” is a 2024 journal article by Honghao Lai and 17 co-authors, published in JAMA Network Open. OpenAlex counts 136 citations as of 19 November 2026. It is open access under a CC BY licence: the full text is free to read at JAMA Network Open.
- Citations
- 136
- Most cited in
- 2025 (65)
- Field-weighted impact
- 13.01
- References
- 60
- Authors
- 18
Abstract
Importance: Large language models (LLMs) may facilitate the labor-intensive process of systematic reviews. However, the exact methods and reliability remain uncertain. Objective: To explore the feasibility and reliability of using LLMs to assess risk of bias (ROB) in randomized clinical trials (RCTs). Design, Setting, and Participants: A survey study was conducted between August 10, 2023, and October 30, 2023. Thirty RCTs were selected from published systematic reviews. Main Outcomes and Measures: A structured prompt was developed to guide ChatGPT (LLM 1) and Claude (LLM 2) in assessing the ROB in these RCTs using a modified version of the Cochrane ROB tool developed by the CLARITY group at McMaster University. Each RCT was assessed twice by both models, and the results were documented. The results were compared with an assessment by 3 experts, which was considered a criterion standard. Correct assessment rates, sensitivity, specificity, and F1 scores were calculated to reflect accuracy, both overall and for each domain of the Cochrane ROB tool; consistent assessment rates and Cohen κ were calculated to gauge consistency; and assessment time was calculated to measure efficiency. Performance between the 2 models was compared using risk differences. Results: Both models demonstrated high correct assessment rates. LLM 1 reached a mean correct assessment rate of 84.5% (95% CI, 81.5%-87.3%), and LLM 2 reached a significantly higher rate of 89.5% (95% CI, 87.0%-91.8%). The risk difference between the 2 models was 0.05 (95% CI, 0.01-0.09). In most domains, domain-specific correct rates were around 80% to 90%; however, sensitivity below 0.80 was observed in domains 1 (random sequence generation), 2 (allocation concealment), and 6 (other concerns). Domains 4 (missing outcome data), 5 (selective outcome reporting), and 6 had F1 scores below 0.50. The consistent rates between the 2 assessments were 84.0% for LLM 1 and 87.3% for LLM 2. LLM 1's κ exceeded 0.80 in 7 and LLM 2's in 8 domains. The mean (SD) time needed for assessment was 77 (16) seconds for LLM 1 and 53 (12) seconds for LLM 2. Conclusions: In this survey study of applying LLMs for ROB assessment, LLM 1 and LLM 2 demonstrated substantial accuracy and consistency in evaluating RCTs, suggesting their potential as supportive tools in systematic review processes.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Lai, H., Ge, L., Sun, M., Pan, B., Huang, J., Hou, L., Yang, Q., Liu, J., Liu, J., Ye, Z., Xia, D., Zhao, W., Wang, X., Liu, M., Talukdar, J. R., Tian, J., Yang, K., & Estill, J. A. M. (2024). Assessing the Risk of Bias in Randomized Clinical Trials With Large Language Models. JAMA Network Open, 7(5), e2412687. https://doi.org/10.1001/jamanetworkopen.2024.12687
MLA 9
Lai, Honghao, et al. “Assessing the Risk of Bias in Randomized Clinical Trials With Large Language Models.” JAMA Network Open, vol. 7, no. 5, 2024, p. e2412687. https://doi.org/10.1001/jamanetworkopen.2024.12687.
Chicago
Lai, Honghao, Long Ge, Mingyao Sun, Bei Pan, Jiajie Huang, Liangying Hou, Qiuyu Yang, et al. 2024. “Assessing the Risk of Bias in Randomized Clinical Trials With Large Language Models.” JAMA Network Open 7 (5): e2412687. https://doi.org/10.1001/jamanetworkopen.2024.12687.
Harvard
Lai, H. et al. (2024) ‘Assessing the Risk of Bias in Randomized Clinical Trials With Large Language Models’, JAMA Network Open, 7(5), p. e2412687. doi:10.1001/jamanetworkopen.2024.12687.
IEEE
H. Lai et al., “Assessing the Risk of Bias in Randomized Clinical Trials With Large Language Models,” JAMA Network Open, vol. 7, no. 5, p. e2412687, 2024, doi: 10.1001/jamanetworkopen.2024.12687.
BibTeX
@article{lai2024assessing,
title = {{Assessing the Risk of Bias in Randomized Clinical Trials With Large Language Models}},
author = {Lai, Honghao and Ge, Long and Sun, Mingyao and Pan, Bei and Huang, Jiajie and Hou, Liangying and Yang, Qiuyu and Liu, Jiayi and Liu, Jianing and Ye, Ziying and Xia, Danni and Zhao, Weilong and Wang, Xiaoman and Liu, Ming and Talukdar, Jhalok Ronjan and Tian, Jinhui and Yang, Kehu and Estill, Janne Anton Markus},
journal = {JAMA Network Open},
year = {2024},
volume = {7},
number = {5},
pages = {e2412687},
doi = {10.1001/jamanetworkopen.2024.12687},
url = {https://doi.org/10.1001/jamanetworkopen.2024.12687}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Citations by year
| Year | Citations | Share of the best year |
|---|---|---|
| 2023 | 1 | |
| 2024 | 5 | |
| 2025 | 65 | |
| 2026 | 65 |
Its field-weighted citation impact is 13.01: it has been cited 13.01 times as often as the average work of the same field, type and year (1.00 is the world average).
Topic and keywords
Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- risk of bias assessment
- large language models
- ChatGPT
- Claude
- randomized controlled trials
- Cochrane risk of bias tool
- systematic review methodology
- model comparison
Work with this paper
Most-cited papers in Decision Sciences
- The PRISMA 2020 statement: an updated guideline for reporting systematic reviewsMatthew James Page, Joanne E. McKenzie, Patrick M. Bossuyt and 23 more · BMJ · 2021 · 144,919 citations
- Evaluating Structural Equation Models with Unobservable Variables and Measurement ErrorClaes Fornell, David F. Larcker · Journal of Marketing Research · 1981 · 118,336 citations
- Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternativesLi‐tze Hu, Peter M. Bentler · Structural Equation Modeling A Multidisciplinary Journal · 1999 · 117,461 citations
- Common method biases in behavioral research: A critical review of the literature and recommended remedies.Philip M. Podsakoff, SCOTT B. MACKENZIE, Jeong Yeon Lee and 1 more · Journal of Applied Psychology · 2003 · 89,248 citations
- Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statementDavid Moher, Alessandro Liberati, Jennifer Tetzlaff and 2 more · BMJ · 2009 · 85,524 citations
Questions about this paper
How do I cite this paper in APA style?
Lai, H., Ge, L., Sun, M., Pan, B., Huang, J., Hou, L., Yang, Q., Liu, J., Liu, J., Ye, Z., Xia, D., Zhao, W., Wang, X., Liu, M., Talukdar, J. R., Tian, J., Yang, K., & Estill, J. A. M. (2024). Assessing the Risk of Bias in Randomized Clinical Trials With Large Language Models. JAMA Network Open, 7(5), e2412687. https://doi.org/10.1001/jamanetworkopen.2024.12687
How many times has this paper been cited?
136 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at JAMA Network Open: sign in and open the PDF from this page.
Who wrote this paper?
Honghao Lai, Long Ge, Mingyao Sun, Bei Pan, Jiajie Huang, Liangying Hou, Qiuyu Yang, Jiayi Liu, Jianing Liu, Ziying Ye, Danni Xia, Weilong Zhao and 6 more.
What is the DOI of this paper?
10.1001/jamanetworkopen.2024.12687, permanently at https://doi.org/10.1001/jamanetworkopen.2024.12687. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.