Preprint · 2023 · Open access

ChatGPT for assessing risk of bias of randomized trials using the RoB 2.0 tool: A methods study

Tyler Pitre, Tanvir Jassal, Jhalok Ronjan Talukdar, Mahnoor Shahab, Michael Ling, Dena Zeraatkar

medRxiv · Published 22 November 2023 · DOI 10.1101/2023.11.19.23298727

Open PDFView at the publisherCite this paper

In brief

“ChatGPT for assessing risk of bias of randomized trials using the RoB 2.0 tool: A methods study” is a 2023 preprint by Tyler Pitre and 5 co-authors, published in medRxiv. OpenAlex counts 27 citations as of 19 November 2026. It is open access under a CC BY licence: the full text is free to read at medRxiv.

Citations
27
Most cited in
2025 (12)
References
47
Authors
6

Abstract

Background Internationally accepted standards for systematic reviews necessitate assessment of the risk of bias of primary studies. Assessing risk of bias, however, can be time- and resource-intensive. AI-based solutions may increase efficiency and reduce burden. Objective To evaluate the reliability of OpenAI’s ChatGPT for performing risk of bias assessments of randomized trials using the revised risk of bias tool for randomized trials (RoB 2.0). Methods We sampled recently published Cochrane systematic reviews of medical interventions (up to October 2023) that included randomized controlled trials and assessed risk of bias using the Cochrane-endorsed RoB 2.0. From each eligible review, we collected data on the risk of bias assessments for the first three reported outcomes. Using ChatGPT-4, we assessed the risk of bias for the same outcomes using three different prompts: a minimal prompt including limited instructions, a maximal prompt with extensive instructions, and an optimized prompt that was designed to yield the best risk of bias judgements. The agreement between ChatGPT’s assessments and those of Cochrane systematic reviewers was quantified using weighted kappa statistics. Results We included 34 systematic reviews with 157 unique trials. We found the agreement between ChatGPT and systematic review authors for assessment of overall risk of bias to be 0.16 (95% CI: 0.01 to 0.3) for the maximal ChatGPT prompt, 0.17 (95% CI: 0.02 to 0.32) for the optimized prompt, and 0.11 (95% CI: −0.04 to 0.27) for the minimal prompt. For the optimized prompt, agreement ranged between 0.11 (95% CI: −0.11 to 0.33) to 0.29 (95% CI: 0.14 to 0.44) across risk of bias domains, with the lowest agreement for the deviations from the intended intervention domain and the highest agreement for the missing outcome data domain. Conclusion Our results suggest that ChatGPT and systematic reviewers only have “slight” to “fair” agreement in risk of bias judgements for randomized trials. ChatGPT-4 cannot be relied upon to judge risk of bias and should not be used for this purpose. Since this study was completed, ChatGPT has advanced and OpenAI has released newer models with different capabilities. These may prove more adept at risk of bias evaluation. There may also be opportunities to use ChatGPT to streamline other aspects of systematic reviews, such as screening of search records or collection of data.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Pitre, T., Jassal, T., Talukdar, J. R., Shahab, M., Ling, M., & Zeraatkar, D. (2023). ChatGPT for assessing risk of bias of randomized trials using the RoB 2.0 tool: A methods study. medRxiv. https://doi.org/10.1101/2023.11.19.23298727

MLA 9

Pitre, Tyler, et al. “ChatGPT for assessing risk of bias of randomized trials using the RoB 2.0 tool: A methods study.” medRxiv, 2023. https://doi.org/10.1101/2023.11.19.23298727.

Chicago

Pitre, Tyler, Tanvir Jassal, Jhalok Ronjan Talukdar, Mahnoor Shahab, Michael Ling, and Dena Zeraatkar. 2023. “ChatGPT for assessing risk of bias of randomized trials using the RoB 2.0 tool: A methods study.” medRxiv. https://doi.org/10.1101/2023.11.19.23298727.

Harvard

Pitre, T. et al. (2023) ‘ChatGPT for assessing risk of bias of randomized trials using the RoB 2.0 tool: A methods study’, medRxiv. doi:10.1101/2023.11.19.23298727.

IEEE

T. Pitre, T. Jassal, J. R. Talukdar, M. Shahab, M. Ling, and D. Zeraatkar, “ChatGPT for assessing risk of bias of randomized trials using the RoB 2.0 tool: A methods study,” medRxiv, 2023, doi: 10.1101/2023.11.19.23298727.

BibTeX

@misc{pitre2023chatgpt,
  title     = {{ChatGPT for assessing risk of bias of randomized trials using the RoB 2.0 tool: A methods study}},
  author    = {Pitre, Tyler and Jassal, Tanvir and Talukdar, Jhalok Ronjan and Shahab, Mahnoor and Ling, Michael and Zeraatkar, Dena},
  howpublished = {medRxiv},
  year      = {2023},
  doi       = {10.1101/2023.11.19.23298727},
  url       = {https://doi.org/10.1101/2023.11.19.23298727}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Citations by year

YearCitationsShare of the best year
202477
20251212
202688

Topic and keywords

Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews

  • ChatGPT
  • risk of bias assessment
  • large language models
  • prompt engineering
  • AI reliability
  • missing outcome data
  • attrition bias
  • selective reporting

Work with this paper

Most-cited papers in Decision Sciences

All highly cited Decision Sciences papers

Questions about this paper

How do I cite this paper in APA style?

Pitre, T., Jassal, T., Talukdar, J. R., Shahab, M., Ling, M., & Zeraatkar, D. (2023). ChatGPT for assessing risk of bias of randomized trials using the RoB 2.0 tool: A methods study. medRxiv. https://doi.org/10.1101/2023.11.19.23298727

How many times has this paper been cited?

27 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at medRxiv: sign in and open the PDF from this page.

Who wrote this paper?

Tyler Pitre, Tanvir Jassal, Jhalok Ronjan Talukdar, Mahnoor Shahab, Michael Ling, Dena Zeraatkar.

What is the DOI of this paper?

10.1101/2023.11.19.23298727, permanently at https://doi.org/10.1101/2023.11.19.23298727. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.