Journal article · 2025 · Open access
Using a Large Language Model (ChatGPT‐4o) to Assess the Risk of Bias in Randomized Controlled Trials of Medical Interventions: Interrater Agreement With Human Reviewers
Cochrane Evidence Synthesis and Methods, 3(5), e70048 · Published 1 September 2025 · DOI 10.1002/cesm.70048
In brief
“Using a Large Language Model (ChatGPT‐4o) to Assess the Risk of Bias in Randomized Controlled Trials of Medical Interventions: Interrater Agreement With Human Reviewers” is a 2025 journal article by Christopher James Rose and 9 co-authors, published in Cochrane Evidence Synthesis and Methods. OpenAlex counts 11 citations as of 19 November 2026, which puts it in the top 2% of works published in 2025 by citations. It is open access under a CC BY licence: the full text is free to read at Cochrane Evidence Synthesis and Methods.
- Citations
- 11
- Most cited in
- 2026 (11)
- Field-weighted impact
- 1.85
- References
- 24
- Authors
- 10
Abstract
Background Risk of bias (RoB) assessment is a highly skilled task that is time‐consuming and subject to human error. RoB automation tools have previously used machine learning models built using relatively small task‐specific training sets. Large language models (LLMs; e.g., ChatGPT) are complex models built using non‐task‐specific Internet‐scale training sets. They demonstrate human‐like abilities and might be able to support tasks like RoB assessment. Methods Following a published peer‐reviewed protocol, we randomly sampled 100 Cochrane reviews. New or updated reviews that evaluated medical interventions, included ≥ 1 eligible trial, and presented human consensus assessments using Cochrane RoB1 or RoB2 were eligible. We excluded reviews performed under emergency conditions (e.g., COVID‐19), and those on public health or welfare. We randomly sampled one trial from each review. Trials using individual‐ or cluster‐randomized designs were eligible. We extracted human consensus RoB assessments of the trials from the reviews, and methods texts from the trials. We used 25 review‐trial pairs to develop a ChatGPT prompt to assess RoB using trial methods text. We used the prompt and the remaining 75 review‐trial pairs to estimate human‐ChatGPT agreement for “Overall RoB” (primary outcome) and “RoB due to the randomization process”, and ChatGPT‐ChatGPT (intrarater) agreement for “Overall RoB”. We used ChatGPT‐4o (February 2025) throughout. Results The 75 reviews were sampled from 35 Cochrane review groups, and all used RoB1. The 75 trials spanned five decades, and all but one were published in English. Human‐ChatGPT agreement for “Overall RoB” assessment was 50.7% (95% CI 39.3%–62.0%), substantially higher than expected by chance ( p = 0.0015). Human‐ChatGPT agreement for “RoB due to the randomization process” was 78.7% (95% CI 69.4%–88.0%; p < 0.001). ChatGPT‐ChatGPT agreement was 74.7% (95% CI 64.8%–84.6%; p < 0.001). Conclusions ChatGPT appears to have some ability to assess RoB and is unlikely to be guessing or “hallucinating”. The estimated agreement for “Overall RoB” is well above estimates of agreement reported for some human reviewers, but below the highest estimates. LLM‐based systems for assessing RoB may be able to help streamline and improve evidence synthesis production.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Rose, C. J., Bidonde, J., Ringsten, M., Glanville, J., Potrebny, T., Cooper, C., Muller, A. E., Bergsund, H. B., Meneses-Echavez, J. F., & Berg, R. C. (2025). Using a Large Language Model (ChatGPT‐4o) to Assess the Risk of Bias in Randomized Controlled Trials of Medical Interventions: Interrater Agreement With Human Reviewers. Cochrane Evidence Synthesis and Methods, 3(5), e70048. https://doi.org/10.1002/cesm.70048
MLA 9
Rose, Christopher James, et al. “Using a Large Language Model (ChatGPT‐4o) to Assess the Risk of Bias in Randomized Controlled Trials of Medical Interventions: Interrater Agreement With Human Reviewers.” Cochrane Evidence Synthesis and Methods, vol. 3, no. 5, 2025, p. e70048. https://doi.org/10.1002/cesm.70048.
Chicago
Rose, Christopher James, Julia Bidonde, Martin Ringsten, Julie Glanville, Thomas Potrebny, Chris Cooper, Ashley Elizabeth Muller, Hans Bugge Bergsund, Jose Francisco Meneses-Echavez, and Rigmor C. Berg. 2025. “Using a Large Language Model (ChatGPT‐4o) to Assess the Risk of Bias in Randomized Controlled Trials of Medical Interventions: Interrater Agreement With Human Reviewers.” Cochrane Evidence Synthesis and Methods 3 (5): e70048. https://doi.org/10.1002/cesm.70048.
Harvard
Rose, C. J. et al. (2025) ‘Using a Large Language Model (ChatGPT‐4o) to Assess the Risk of Bias in Randomized Controlled Trials of Medical Interventions: Interrater Agreement With Human Reviewers’, Cochrane Evidence Synthesis and Methods, 3(5), p. e70048. doi:10.1002/cesm.70048.
IEEE
C. J. Rose et al., “Using a Large Language Model (ChatGPT‐4o) to Assess the Risk of Bias in Randomized Controlled Trials of Medical Interventions: Interrater Agreement With Human Reviewers,” Cochrane Evidence Synthesis and Methods, vol. 3, no. 5, p. e70048, 2025, doi: 10.1002/cesm.70048.
BibTeX
@article{rose2025using,
title = {{Using a Large Language Model (ChatGPT‐4o) to Assess the Risk of Bias in Randomized Controlled Trials of Medical Interventions: Interrater Agreement With Human Reviewers}},
author = {Rose, Christopher James and Bidonde, Julia and Ringsten, Martin and Glanville, Julie and Potrebny, Thomas and Cooper, Chris and Muller, Ashley Elizabeth and Bergsund, Hans Bugge and Meneses-Echavez, Jose Francisco and Berg, Rigmor C.},
journal = {Cochrane Evidence Synthesis and Methods},
year = {2025},
volume = {3},
number = {5},
pages = {e70048},
doi = {10.1002/cesm.70048},
url = {https://doi.org/10.1002/cesm.70048}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- risk of bias assessment
- GPT-4o
- large language models
- interobserver agreement
- intraobserver agreement
- randomized controlled trials
- medical interventions
Work with this paper
Most-cited papers in Decision Sciences
- The PRISMA 2020 statement: an updated guideline for reporting systematic reviewsMatthew James Page, Joanne E. McKenzie, Patrick M. Bossuyt and 23 more · BMJ · 2021 · 144,919 citations
- Evaluating Structural Equation Models with Unobservable Variables and Measurement ErrorClaes Fornell, David F. Larcker · Journal of Marketing Research · 1981 · 118,336 citations
- Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternativesLi‐tze Hu, Peter M. Bentler · Structural Equation Modeling A Multidisciplinary Journal · 1999 · 117,461 citations
- Common method biases in behavioral research: A critical review of the literature and recommended remedies.Philip M. Podsakoff, SCOTT B. MACKENZIE, Jeong Yeon Lee and 1 more · Journal of Applied Psychology · 2003 · 89,248 citations
- Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statementDavid Moher, Alessandro Liberati, Jennifer Tetzlaff and 2 more · BMJ · 2009 · 85,524 citations
Questions about this paper
How do I cite this paper in APA style?
Rose, C. J., Bidonde, J., Ringsten, M., Glanville, J., Potrebny, T., Cooper, C., Muller, A. E., Bergsund, H. B., Meneses-Echavez, J. F., & Berg, R. C. (2025). Using a Large Language Model (ChatGPT‐4o) to Assess the Risk of Bias in Randomized Controlled Trials of Medical Interventions: Interrater Agreement With Human Reviewers. Cochrane Evidence Synthesis and Methods, 3(5), e70048. https://doi.org/10.1002/cesm.70048
How many times has this paper been cited?
11 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at Cochrane Evidence Synthesis and Methods: sign in and open the PDF from this page.
Who wrote this paper?
Christopher James Rose, Julia Bidonde, Martin Ringsten, Julie Glanville, Thomas Potrebny, Chris Cooper, Ashley Elizabeth Muller, Hans Bugge Bergsund, Jose Francisco Meneses-Echavez, Rigmor C. Berg.
What is the DOI of this paper?
10.1002/cesm.70048, permanently at https://doi.org/10.1002/cesm.70048. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.