Journal article · 2026 · Open access
Large language models for risk-of-bias assessment in randomised clinical trials—a comparative validation study
EBioMedicine, 126, 106238 · Published 28 March 2026 · DOI 10.1016/j.ebiom.2026.106238
In brief
“Large language models for risk-of-bias assessment in randomised clinical trials—a comparative validation study” is a 2026 journal article by Lauri Nyrhi and 7 co-authors, published in EBioMedicine. OpenAlex counts 6 citations as of 19 November 2026, which puts it in the top 3% of works published in 2026 by citations. It is open access under a CC BY licence: the full text is free to read at EBioMedicine.
- Citations
- 6
- Most cited in
- 2026 (6)
- Field-weighted impact
- 9.19
- References
- 39
- Authors
- 8
Abstract
Background Large language models (LLMs) are emerging tools for evidence synthesis. Risk of bias (RoB) assessment of trials remains an essential but time-consuming step inconsistent even amongst experts. Early LLM studies showed mixed reliability. Advances in reasoning-enabled models warrant evaluation of their accuracy and consistency for RoB screening across randomised trials to reduce reviewer workload. Methods We conducted a preregistered comparative validation study (March 11–May 19, 2025) of four LLMs—ChatGPT o3, DeepSeek v3, Google Gemini Flash 2.0, and Grok 3—prompted with full-text randomised clinical trial articles and protocols. Two corpora were analysed: 100 RCTs from recent Cochrane reviews (RoB 1) and 100 RCTs from meta-analyses in high-impact journals (RoB 2). The reference standard was published human RoB judgements. The primary outcome was interobserver reliability (Cohen κ, 95% CI); secondary outcomes were intraobserver agreement and diagnostic accuracy (sensitivity, specificity, predictive values, F 1 -score). Findings For RoB 1, interobserver agreement ranged from κ 0.0.27 (95% CI 0.07–0.46) with Gemini Flash 2.0 to κ 0.39 (0.20–0.59) with DeepSeek v3. For RoB 2, agreement was lower, from κ 0.06 (−0.07 to 0.18) with ChatGPT o3 to κ 0.13 (−0.04 to 0.31) with Gemini. Diagnostic performance was limited with sensitivity ranging 0.05–0.55, specificity 0.78–0.99, PPV 0.31–0.50, and NPV 0.48–0.61 across models, with models consistently over-flagging concerns. Interpretation None of the evaluated LLMs were sufficiently reliable for fully autonomous RoB assessment. DeepSeek v3 and ChatGPT o3 approximated human performance best on RoB 1, but RoB 2 rule-in and rule-out performance remained modest. Current use should be supervised, with possible application of LLMs for triage or as a second assessor. Major improvements in protocol retrieval, task-specific tuning, and calibrated thresholds, prospectively validated, are needed for safe stand-alone deployment. Funding This study received no financial support.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Nyrhi, L., Ponkilainen, V., Laaksonen, J., Kuikka, L., Paljakka, L., Karjalainen, T., Mattila, V. M., & Kuitunen, I. (2026). Large language models for risk-of-bias assessment in randomised clinical trials—a comparative validation study. EBioMedicine, 126, 106238. https://doi.org/10.1016/j.ebiom.2026.106238
MLA 9
Nyrhi, Lauri, et al. “Large language models for risk-of-bias assessment in randomised clinical trials—a comparative validation study.” EBioMedicine, vol. 126, 2026, p. 106238. https://doi.org/10.1016/j.ebiom.2026.106238.
Chicago
Nyrhi, Lauri, Ville Ponkilainen, Juho Laaksonen, Lauri Kuikka, Lauri Paljakka, Teemu Karjalainen, Ville Matti Mattila, and Ilari Kuitunen. 2026. “Large language models for risk-of-bias assessment in randomised clinical trials—a comparative validation study.” EBioMedicine 126: 106238. https://doi.org/10.1016/j.ebiom.2026.106238.
Harvard
Nyrhi, L. et al. (2026) ‘Large language models for risk-of-bias assessment in randomised clinical trials—a comparative validation study’, EBioMedicine, 126, p. 106238. doi:10.1016/j.ebiom.2026.106238.
IEEE
L. Nyrhi et al., “Large language models for risk-of-bias assessment in randomised clinical trials—a comparative validation study,” EBioMedicine, vol. 126, p. 106238, 2026, doi: 10.1016/j.ebiom.2026.106238.
BibTeX
@article{nyrhi2026large,
title = {{Large language models for risk-of-bias assessment in randomised clinical trials—a comparative validation study}},
author = {Nyrhi, Lauri and Ponkilainen, Ville and Laaksonen, Juho and Kuikka, Lauri and Paljakka, Lauri and Karjalainen, Teemu and Mattila, Ville Matti and Kuitunen, Ilari},
journal = {EBioMedicine},
year = {2026},
volume = {126},
pages = {106238},
doi = {10.1016/j.ebiom.2026.106238},
url = {https://doi.org/10.1016/j.ebiom.2026.106238}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- large language models
- risk of bias assessment
- randomized controlled trials
- evidence synthesis
- inter-rater reliability
- diagnostic accuracy
- ChatGPT
- DeepSeek
Work with this paper
Most-cited papers in Decision Sciences
- The PRISMA 2020 statement: an updated guideline for reporting systematic reviewsMatthew James Page, Joanne E. McKenzie, Patrick M. Bossuyt and 23 more · BMJ · 2021 · 144,919 citations
- Evaluating Structural Equation Models with Unobservable Variables and Measurement ErrorClaes Fornell, David F. Larcker · Journal of Marketing Research · 1981 · 118,336 citations
- Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternativesLi‐tze Hu, Peter M. Bentler · Structural Equation Modeling A Multidisciplinary Journal · 1999 · 117,461 citations
- Common method biases in behavioral research: A critical review of the literature and recommended remedies.Philip M. Podsakoff, SCOTT B. MACKENZIE, Jeong Yeon Lee and 1 more · Journal of Applied Psychology · 2003 · 89,248 citations
- Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statementDavid Moher, Alessandro Liberati, Jennifer Tetzlaff and 2 more · BMJ · 2009 · 85,524 citations
Questions about this paper
How do I cite this paper in APA style?
Nyrhi, L., Ponkilainen, V., Laaksonen, J., Kuikka, L., Paljakka, L., Karjalainen, T., Mattila, V. M., & Kuitunen, I. (2026). Large language models for risk-of-bias assessment in randomised clinical trials—a comparative validation study. EBioMedicine, 126, 106238. https://doi.org/10.1016/j.ebiom.2026.106238
How many times has this paper been cited?
6 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at EBioMedicine: sign in and open the full text from this page.
Who wrote this paper?
Lauri Nyrhi, Ville Ponkilainen, Juho Laaksonen, Lauri Kuikka, Lauri Paljakka, Teemu Karjalainen, Ville Matti Mattila, Ilari Kuitunen.
What is the DOI of this paper?
10.1016/j.ebiom.2026.106238, permanently at https://doi.org/10.1016/j.ebiom.2026.106238. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.