Journal article · 2026 · Open access

Large language models for risk-of-bias assessment in randomised clinical trials—a comparative validation study

Lauri Nyrhi, Ville Ponkilainen, Juho Laaksonen, Lauri Kuikka, Lauri Paljakka, Teemu Karjalainen, Ville Matti Mattila, Ilari Kuitunen

EBioMedicine, 126, 106238 · Published 28 March 2026 · DOI 10.1016/j.ebiom.2026.106238

Open full textCite this paper

In brief

“Large language models for risk-of-bias assessment in randomised clinical trials—a comparative validation study” is a 2026 journal article by Lauri Nyrhi and 7 co-authors, published in EBioMedicine. OpenAlex counts 6 citations as of 19 November 2026, which puts it in the top 3% of works published in 2026 by citations. It is open access under a CC BY licence: the full text is free to read at EBioMedicine.

Citations
6
Most cited in
2026 (6)
Field-weighted impact
9.19
References
39
Authors
8

Abstract

Background Large language models (LLMs) are emerging tools for evidence synthesis. Risk of bias (RoB) assessment of trials remains an essential but time-consuming step inconsistent even amongst experts. Early LLM studies showed mixed reliability. Advances in reasoning-enabled models warrant evaluation of their accuracy and consistency for RoB screening across randomised trials to reduce reviewer workload. Methods We conducted a preregistered comparative validation study (March 11–May 19, 2025) of four LLMs—ChatGPT o3, DeepSeek v3, Google Gemini Flash 2.0, and Grok 3—prompted with full-text randomised clinical trial articles and protocols. Two corpora were analysed: 100 RCTs from recent Cochrane reviews (RoB 1) and 100 RCTs from meta-analyses in high-impact journals (RoB 2). The reference standard was published human RoB judgements. The primary outcome was interobserver reliability (Cohen κ, 95% CI); secondary outcomes were intraobserver agreement and diagnostic accuracy (sensitivity, specificity, predictive values, F 1 -score). Findings For RoB 1, interobserver agreement ranged from κ 0.0.27 (95% CI 0.07–0.46) with Gemini Flash 2.0 to κ 0.39 (0.20–0.59) with DeepSeek v3. For RoB 2, agreement was lower, from κ 0.06 (−0.07 to 0.18) with ChatGPT o3 to κ 0.13 (−0.04 to 0.31) with Gemini. Diagnostic performance was limited with sensitivity ranging 0.05–0.55, specificity 0.78–0.99, PPV 0.31–0.50, and NPV 0.48–0.61 across models, with models consistently over-flagging concerns. Interpretation None of the evaluated LLMs were sufficiently reliable for fully autonomous RoB assessment. DeepSeek v3 and ChatGPT o3 approximated human performance best on RoB 1, but RoB 2 rule-in and rule-out performance remained modest. Current use should be supervised, with possible application of LLMs for triage or as a second assessor. Major improvements in protocol retrieval, task-specific tuning, and calibrated thresholds, prospectively validated, are needed for safe stand-alone deployment. Funding This study received no financial support.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Nyrhi, L., Ponkilainen, V., Laaksonen, J., Kuikka, L., Paljakka, L., Karjalainen, T., Mattila, V. M., & Kuitunen, I. (2026). Large language models for risk-of-bias assessment in randomised clinical trials—a comparative validation study. EBioMedicine, 126, 106238. https://doi.org/10.1016/j.ebiom.2026.106238

MLA 9

Nyrhi, Lauri, et al. “Large language models for risk-of-bias assessment in randomised clinical trials—a comparative validation study.” EBioMedicine, vol. 126, 2026, p. 106238. https://doi.org/10.1016/j.ebiom.2026.106238.

Chicago

Nyrhi, Lauri, Ville Ponkilainen, Juho Laaksonen, Lauri Kuikka, Lauri Paljakka, Teemu Karjalainen, Ville Matti Mattila, and Ilari Kuitunen. 2026. “Large language models for risk-of-bias assessment in randomised clinical trials—a comparative validation study.” EBioMedicine 126: 106238. https://doi.org/10.1016/j.ebiom.2026.106238.

Harvard

Nyrhi, L. et al. (2026) ‘Large language models for risk-of-bias assessment in randomised clinical trials—a comparative validation study’, EBioMedicine, 126, p. 106238. doi:10.1016/j.ebiom.2026.106238.

IEEE

L. Nyrhi et al., “Large language models for risk-of-bias assessment in randomised clinical trials—a comparative validation study,” EBioMedicine, vol. 126, p. 106238, 2026, doi: 10.1016/j.ebiom.2026.106238.

BibTeX

@article{nyrhi2026large,
  title     = {{Large language models for risk-of-bias assessment in randomised clinical trials—a comparative validation study}},
  author    = {Nyrhi, Lauri and Ponkilainen, Ville and Laaksonen, Juho and Kuikka, Lauri and Paljakka, Lauri and Karjalainen, Teemu and Mattila, Ville Matti and Kuitunen, Ilari},
  journal   = {EBioMedicine},
  year      = {2026},
  volume    = {126},
  pages     = {106238},
  doi       = {10.1016/j.ebiom.2026.106238},
  url       = {https://doi.org/10.1016/j.ebiom.2026.106238}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews

  • large language models
  • risk of bias assessment
  • randomized controlled trials
  • evidence synthesis
  • inter-rater reliability
  • diagnostic accuracy
  • ChatGPT
  • DeepSeek

Work with this paper

Most-cited papers in Decision Sciences

All highly cited Decision Sciences papers

Questions about this paper

How do I cite this paper in APA style?

Nyrhi, L., Ponkilainen, V., Laaksonen, J., Kuikka, L., Paljakka, L., Karjalainen, T., Mattila, V. M., & Kuitunen, I. (2026). Large language models for risk-of-bias assessment in randomised clinical trials—a comparative validation study. EBioMedicine, 126, 106238. https://doi.org/10.1016/j.ebiom.2026.106238

How many times has this paper been cited?

6 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at EBioMedicine: sign in and open the full text from this page.

Who wrote this paper?

Lauri Nyrhi, Ville Ponkilainen, Juho Laaksonen, Lauri Kuikka, Lauri Paljakka, Teemu Karjalainen, Ville Matti Mattila, Ilari Kuitunen.

What is the DOI of this paper?

10.1016/j.ebiom.2026.106238, permanently at https://doi.org/10.1016/j.ebiom.2026.106238. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.