Journal article · 2025 · Open access
Exploring the potential of Claude 2 for risk of bias assessment: Using a large language model to assess randomized controlled trials with RoB 2
Research Synthesis Methods, 16(3), 491–508 · Published 12 March 2025 · DOI 10.1017/rsm.2025.12
In brief
“Exploring the potential of Claude 2 for risk of bias assessment: Using a large language model to assess randomized controlled trials with RoB 2” is a 2025 journal article by Angelika Eisele‐Metzger and 7 co-authors, published in Research Synthesis Methods. OpenAlex counts 32 citations as of 19 November 2026. It is open access under a CC BY licence: the full text is free to read at Research Synthesis Methods.
- Citations
- 32
- Most cited in
- 2026 (22)
- Field-weighted impact
- 5.20
- References
- 61
- Authors
- 8
Abstract
Systematic reviews are essential for evidence-based health care, but conducting them is time- and resource-consuming. To date, efforts have been made to accelerate and (semi-)automate various steps of systematic reviews through the use of artificial intelligence (AI) and the emergence of large language models (LLMs) promises further opportunities. One crucial but complex task within systematic review conduct is assessing the risk of bias (RoB) of included studies. Therefore, the aim of this study was to test the LLM Claude 2 for RoB assessment of 100 randomized controlled trials, published in English language from 2013 onwards, using the revised Cochrane risk of bias tool ('RoB 2'; involving judgements for five specific domains and an overall judgement). We assessed the agreement of RoB judgements by Claude with human judgements published in Cochrane reviews. The observed agreement between Claude and Cochrane authors ranged from 41% for the overall judgement to 71% for domain 4 ('outcome measurement'). Cohen's κ was lowest for domain 5 ('selective reporting'; 0.10 (95% confidence interval (CI): -0.10-0.31)) and highest for domain 3 ('missing data'; 0.31 (95% CI: 0.10-0.52)), indicating slight to fair agreement. Fair agreement was found for the overall judgement (Cohen's κ: 0.22 (95% CI: 0.06-0.38)). Sensitivity analyses using alternative prompting techniques or the more recent version Claude 3 did not result in substantial changes. Currently, Claude's RoB 2 judgements cannot replace human RoB assessment. However, the potential of LLMs to support RoB assessment should be further explored.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Eisele‐Metzger, A., Lieberum, J.-L., Toews, M., Siemens, W., Heilmeyer, F., Haverkamp, C., Böhringer, D., & Meerpohl, J. J. (2025). Exploring the potential of Claude 2 for risk of bias assessment: Using a large language model to assess randomized controlled trials with RoB 2. Research Synthesis Methods, 16(3), 491–508. https://doi.org/10.1017/rsm.2025.12
MLA 9
Eisele‐Metzger, Angelika, et al. “Exploring the potential of Claude 2 for risk of bias assessment: Using a large language model to assess randomized controlled trials with RoB 2.” Research Synthesis Methods, vol. 16, no. 3, 2025, pp. 491–508. https://doi.org/10.1017/rsm.2025.12.
Chicago
Eisele‐Metzger, Angelika, Judith-Lisa Lieberum, Markus Toews, Waldemar Siemens, Felix Heilmeyer, Christian Haverkamp, Daniel Böhringer, and Joerg J Meerpohl. 2025. “Exploring the potential of Claude 2 for risk of bias assessment: Using a large language model to assess randomized controlled trials with RoB 2.” Research Synthesis Methods 16 (3): 491–508. https://doi.org/10.1017/rsm.2025.12.
Harvard
Eisele‐Metzger, A. et al. (2025) ‘Exploring the potential of Claude 2 for risk of bias assessment: Using a large language model to assess randomized controlled trials with RoB 2’, Research Synthesis Methods, 16(3), pp. 491–508. doi:10.1017/rsm.2025.12.
IEEE
A. Eisele‐Metzger et al., “Exploring the potential of Claude 2 for risk of bias assessment: Using a large language model to assess randomized controlled trials with RoB 2,” Research Synthesis Methods, vol. 16, no. 3, pp. 491–508, 2025, doi: 10.1017/rsm.2025.12.
BibTeX
@article{eiselemetzger2025exploring,
title = {{Exploring the potential of Claude 2 for risk of bias assessment: Using a large language model to assess randomized controlled trials with RoB 2}},
author = {Eisele‐Metzger, Angelika and Lieberum, Judith-Lisa and Toews, Markus and Siemens, Waldemar and Heilmeyer, Felix and Haverkamp, Christian and Böhringer, Daniel and Meerpohl, Joerg J},
journal = {Research Synthesis Methods},
year = {2025},
volume = {16},
number = {3},
pages = {491--508},
doi = {10.1017/rsm.2025.12},
url = {https://doi.org/10.1017/rsm.2025.12}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Citations by year
| Year | Citations | Share of the best year |
|---|---|---|
| 2024 | 1 | |
| 2025 | 9 | |
| 2026 | 22 |
Its field-weighted citation impact is 5.20: it has been cited 5.20 times as often as the average work of the same field, type and year (1.00 is the world average).
Topic and keywords
Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Claude 2
- large language models
- risk of bias assessment
- RoB 2
- randomized controlled trials
- systematic reviews
- selective reporting
- missing data
Work with this paper
Most-cited papers in Decision Sciences
- The PRISMA 2020 statement: an updated guideline for reporting systematic reviewsMatthew James Page, Joanne E. McKenzie, Patrick M. Bossuyt and 23 more · BMJ · 2021 · 144,919 citations
- Evaluating Structural Equation Models with Unobservable Variables and Measurement ErrorClaes Fornell, David F. Larcker · Journal of Marketing Research · 1981 · 118,336 citations
- Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternativesLi‐tze Hu, Peter M. Bentler · Structural Equation Modeling A Multidisciplinary Journal · 1999 · 117,461 citations
- Common method biases in behavioral research: A critical review of the literature and recommended remedies.Philip M. Podsakoff, SCOTT B. MACKENZIE, Jeong Yeon Lee and 1 more · Journal of Applied Psychology · 2003 · 89,248 citations
- Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statementDavid Moher, Alessandro Liberati, Jennifer Tetzlaff and 2 more · BMJ · 2009 · 85,524 citations
Questions about this paper
How do I cite this paper in APA style?
Eisele‐Metzger, A., Lieberum, J.-L., Toews, M., Siemens, W., Heilmeyer, F., Haverkamp, C., Böhringer, D., & Meerpohl, J. J. (2025). Exploring the potential of Claude 2 for risk of bias assessment: Using a large language model to assess randomized controlled trials with RoB 2. Research Synthesis Methods, 16(3), 491–508. https://doi.org/10.1017/rsm.2025.12
How many times has this paper been cited?
32 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at Research Synthesis Methods: sign in and open the full text from this page.
Who wrote this paper?
Angelika Eisele‐Metzger, Judith-Lisa Lieberum, Markus Toews, Waldemar Siemens, Felix Heilmeyer, Christian Haverkamp, Daniel Böhringer, Joerg J Meerpohl.
What is the DOI of this paper?
10.1017/rsm.2025.12, permanently at https://doi.org/10.1017/rsm.2025.12. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.