Journal article · 2025 · Open access

Using a Large Language Model (ChatGPT‐4o) to Assess the Risk of Bias in Randomized Controlled Trials of Medical Interventions: Interrater Agreement With Human Reviewers

Christopher James Rose, Julia Bidonde, Martin Ringsten, Julie Glanville, Thomas Potrebny, Chris Cooper, Ashley Elizabeth Muller, Hans Bugge Bergsund, Jose Francisco Meneses-Echavez, Rigmor C. Berg

Cochrane Evidence Synthesis and Methods, 3(5), e70048 · Published 1 September 2025 · DOI 10.1002/cesm.70048

Open PDFView at the publisherCite this paper

In brief

“Using a Large Language Model (ChatGPT‐4o) to Assess the Risk of Bias in Randomized Controlled Trials of Medical Interventions: Interrater Agreement With Human Reviewers” is a 2025 journal article by Christopher James Rose and 9 co-authors, published in Cochrane Evidence Synthesis and Methods. OpenAlex counts 11 citations as of 19 November 2026, which puts it in the top 2% of works published in 2025 by citations. It is open access under a CC BY licence: the full text is free to read at Cochrane Evidence Synthesis and Methods.

Citations
11
Most cited in
2026 (11)
Field-weighted impact
1.85
References
24
Authors
10

Abstract

Background Risk of bias (RoB) assessment is a highly skilled task that is time‐consuming and subject to human error. RoB automation tools have previously used machine learning models built using relatively small task‐specific training sets. Large language models (LLMs; e.g., ChatGPT) are complex models built using non‐task‐specific Internet‐scale training sets. They demonstrate human‐like abilities and might be able to support tasks like RoB assessment. Methods Following a published peer‐reviewed protocol, we randomly sampled 100 Cochrane reviews. New or updated reviews that evaluated medical interventions, included ≥ 1 eligible trial, and presented human consensus assessments using Cochrane RoB1 or RoB2 were eligible. We excluded reviews performed under emergency conditions (e.g., COVID‐19), and those on public health or welfare. We randomly sampled one trial from each review. Trials using individual‐ or cluster‐randomized designs were eligible. We extracted human consensus RoB assessments of the trials from the reviews, and methods texts from the trials. We used 25 review‐trial pairs to develop a ChatGPT prompt to assess RoB using trial methods text. We used the prompt and the remaining 75 review‐trial pairs to estimate human‐ChatGPT agreement for “Overall RoB” (primary outcome) and “RoB due to the randomization process”, and ChatGPT‐ChatGPT (intrarater) agreement for “Overall RoB”. We used ChatGPT‐4o (February 2025) throughout. Results The 75 reviews were sampled from 35 Cochrane review groups, and all used RoB1. The 75 trials spanned five decades, and all but one were published in English. Human‐ChatGPT agreement for “Overall RoB” assessment was 50.7% (95% CI 39.3%–62.0%), substantially higher than expected by chance ( p = 0.0015). Human‐ChatGPT agreement for “RoB due to the randomization process” was 78.7% (95% CI 69.4%–88.0%; p < 0.001). ChatGPT‐ChatGPT agreement was 74.7% (95% CI 64.8%–84.6%; p < 0.001). Conclusions ChatGPT appears to have some ability to assess RoB and is unlikely to be guessing or “hallucinating”. The estimated agreement for “Overall RoB” is well above estimates of agreement reported for some human reviewers, but below the highest estimates. LLM‐based systems for assessing RoB may be able to help streamline and improve evidence synthesis production.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Rose, C. J., Bidonde, J., Ringsten, M., Glanville, J., Potrebny, T., Cooper, C., Muller, A. E., Bergsund, H. B., Meneses-Echavez, J. F., & Berg, R. C. (2025). Using a Large Language Model (ChatGPT‐4o) to Assess the Risk of Bias in Randomized Controlled Trials of Medical Interventions: Interrater Agreement With Human Reviewers. Cochrane Evidence Synthesis and Methods, 3(5), e70048. https://doi.org/10.1002/cesm.70048

MLA 9

Rose, Christopher James, et al. “Using a Large Language Model (ChatGPT‐4o) to Assess the Risk of Bias in Randomized Controlled Trials of Medical Interventions: Interrater Agreement With Human Reviewers.” Cochrane Evidence Synthesis and Methods, vol. 3, no. 5, 2025, p. e70048. https://doi.org/10.1002/cesm.70048.

Chicago

Rose, Christopher James, Julia Bidonde, Martin Ringsten, Julie Glanville, Thomas Potrebny, Chris Cooper, Ashley Elizabeth Muller, Hans Bugge Bergsund, Jose Francisco Meneses-Echavez, and Rigmor C. Berg. 2025. “Using a Large Language Model (ChatGPT‐4o) to Assess the Risk of Bias in Randomized Controlled Trials of Medical Interventions: Interrater Agreement With Human Reviewers.” Cochrane Evidence Synthesis and Methods 3 (5): e70048. https://doi.org/10.1002/cesm.70048.

Harvard

Rose, C. J. et al. (2025) ‘Using a Large Language Model (ChatGPT‐4o) to Assess the Risk of Bias in Randomized Controlled Trials of Medical Interventions: Interrater Agreement With Human Reviewers’, Cochrane Evidence Synthesis and Methods, 3(5), p. e70048. doi:10.1002/cesm.70048.

IEEE

C. J. Rose et al., “Using a Large Language Model (ChatGPT‐4o) to Assess the Risk of Bias in Randomized Controlled Trials of Medical Interventions: Interrater Agreement With Human Reviewers,” Cochrane Evidence Synthesis and Methods, vol. 3, no. 5, p. e70048, 2025, doi: 10.1002/cesm.70048.

BibTeX

@article{rose2025using,
  title     = {{Using a Large Language Model (ChatGPT‐4o) to Assess the Risk of Bias in Randomized Controlled Trials of Medical Interventions: Interrater Agreement With Human Reviewers}},
  author    = {Rose, Christopher James and Bidonde, Julia and Ringsten, Martin and Glanville, Julie and Potrebny, Thomas and Cooper, Chris and Muller, Ashley Elizabeth and Bergsund, Hans Bugge and Meneses-Echavez, Jose Francisco and Berg, Rigmor C.},
  journal   = {Cochrane Evidence Synthesis and Methods},
  year      = {2025},
  volume    = {3},
  number    = {5},
  pages     = {e70048},
  doi       = {10.1002/cesm.70048},
  url       = {https://doi.org/10.1002/cesm.70048}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews

  • risk of bias assessment
  • GPT-4o
  • large language models
  • interobserver agreement
  • intraobserver agreement
  • randomized controlled trials
  • medical interventions

Work with this paper

Most-cited papers in Decision Sciences

All highly cited Decision Sciences papers

Questions about this paper

How do I cite this paper in APA style?

Rose, C. J., Bidonde, J., Ringsten, M., Glanville, J., Potrebny, T., Cooper, C., Muller, A. E., Bergsund, H. B., Meneses-Echavez, J. F., & Berg, R. C. (2025). Using a Large Language Model (ChatGPT‐4o) to Assess the Risk of Bias in Randomized Controlled Trials of Medical Interventions: Interrater Agreement With Human Reviewers. Cochrane Evidence Synthesis and Methods, 3(5), e70048. https://doi.org/10.1002/cesm.70048

How many times has this paper been cited?

11 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at Cochrane Evidence Synthesis and Methods: sign in and open the PDF from this page.

Who wrote this paper?

Christopher James Rose, Julia Bidonde, Martin Ringsten, Julie Glanville, Thomas Potrebny, Chris Cooper, Ashley Elizabeth Muller, Hans Bugge Bergsund, Jose Francisco Meneses-Echavez, Rigmor C. Berg.

What is the DOI of this paper?

10.1002/cesm.70048, permanently at https://doi.org/10.1002/cesm.70048. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.