Journal article · 2026 · Open access

Head-to-head evaluation of large language models for automated risk of bias assessment in randomized controlled trials: A two-phase diagnostic study with meta-reanalysis

Ji Cheng, Fuqiang Wang, Zhangyi Dai, Zhiyang Li, Xiangchao Kong, Wang Xinwei, Hanlu Zhang, Yun Wang

Digital Health, 12, 20552076261487043 · Published 1 February 2026 · DOI 10.1177/20552076261487043

Open full textCite this paper

In brief

“Head-to-head evaluation of large language models for automated risk of bias assessment in randomized controlled trials: A two-phase diagnostic study with meta-reanalysis” is a 2026 journal article by Ji Cheng and 7 co-authors, published in Digital Health. OpenAlex counts 0 citations as of 19 November 2026. It is open access under a CC BY-NC licence: the full text is free to read at Digital Health.

Citations
0
References
37
Authors
8

Abstract

Objective: Large language models (LLMs) are emerging as promising tools in medical research, yet their feasibility for automating risk-of-bias assessment in systematic reviews and meta-analyses (SR/MAs) remains underexplored. To this end, we systematically evaluated the accuracy, consistency, and efficiency of two representative LLMs in conducting Cochrane Risk of Bias 2 (RoB-2) assessments, through a head-to-head comparison integrated with a two-phase analytical design that introduced post-hoc chain-of-thought prompt optimization to mirror the human adjudication process. A subsequent meta-reanalysis further traced the downstream impact of LLM-flagged high-risk-of-bias judgments on the robustness of pooled evidence. Methods: In a two-phase diagnostic study design, two representative LLMs (LLM-1, open-source; LLM-2, commercial) assessed 56 randomized controlled trials (RCTs) from five systematic reviews and meta-analyses. Standard RoB-2based prompts were refined post hoc through chain-of-thought prompting to mitigate overly risk-averse ratings. LLM assessments were compared against human reference standards from the original reviews. Performance was quantified via confusion matrices, accuracy, F1-score, Matthews Correlation Coefficient (MCC), and Cohen's kappa. Results: In post-hoc analysis, binary classification accuracy reached 86.8% (LLM-1) and 89.3% (LLM-2); ternary accuracy reached 85.7% and 87.5%, respectively. MCC exceeded 0.6 for both models, and inter-model kappa improved to 0.601, indicating moderate to high consistency. LLMs reduced assessment time by approximately 16 minutes per RCT compared with manual evaluators. Excluding LLM identified high risk RCTs in meta reanalysis strengthened evidence robustness without altering effect directions. Conclusion: LLMs demonstrated high diagnostic accuracy (binary accuracy: 89.3%; ternary accuracy: 87.5%) and reduced assessment time by approximately 16 minutes per RCT compared with manual evaluation. However, both models showed limited ability to reliably distinguish "Some concerns" from "High risk" judgments, and performance depended on post-hoc chain-of-thought prompt optimization. These findings indicate that LLMs are not yet suitable for fully autonomous bias assessment and should be deployed as adjunctive tools, with human oversight retained for final judgments.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Cheng, J., Wang, F., Dai, Z., Li, Z., Kong, X., Xinwei, W., Zhang, H., & Wang, Y. (2026). Head-to-head evaluation of large language models for automated risk of bias assessment in randomized controlled trials: A two-phase diagnostic study with meta-reanalysis. Digital Health, 12, 20552076261487043. https://doi.org/10.1177/20552076261487043

MLA 9

Cheng, Ji, et al. “Head-to-head evaluation of large language models for automated risk of bias assessment in randomized controlled trials: A two-phase diagnostic study with meta-reanalysis.” Digital Health, vol. 12, 2026, p. 20552076261487043. https://doi.org/10.1177/20552076261487043.

Chicago

Cheng, Ji, Fuqiang Wang, Zhangyi Dai, Zhiyang Li, Xiangchao Kong, Wang Xinwei, Hanlu Zhang, and Yun Wang. 2026. “Head-to-head evaluation of large language models for automated risk of bias assessment in randomized controlled trials: A two-phase diagnostic study with meta-reanalysis.” Digital Health 12: 20552076261487043. https://doi.org/10.1177/20552076261487043.

Harvard

Cheng, J. et al. (2026) ‘Head-to-head evaluation of large language models for automated risk of bias assessment in randomized controlled trials: A two-phase diagnostic study with meta-reanalysis’, Digital Health, 12, p. 20552076261487043. doi:10.1177/20552076261487043.

IEEE

J. Cheng et al., “Head-to-head evaluation of large language models for automated risk of bias assessment in randomized controlled trials: A two-phase diagnostic study with meta-reanalysis,” Digital Health, vol. 12, p. 20552076261487043, 2026, doi: 10.1177/20552076261487043.

BibTeX

@article{cheng2026head,
  title     = {{Head-to-head evaluation of large language models for automated risk of bias assessment in randomized controlled trials: A two-phase diagnostic study with meta-reanalysis}},
  author    = {Cheng, Ji and Wang, Fuqiang and Dai, Zhangyi and Li, Zhiyang and Kong, Xiangchao and Xinwei, Wang and Zhang, Hanlu and Wang, Yun},
  journal   = {Digital Health},
  year      = {2026},
  volume    = {12},
  pages     = {20552076261487043},
  doi       = {10.1177/20552076261487043},
  url       = {https://doi.org/10.1177/20552076261487043}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews

  • large language models
  • risk of bias assessment
  • RoB 2
  • randomized controlled trials
  • chain-of-thought prompting
  • systematic reviews
  • meta-analysis
  • human oversight

Work with this paper

Most-cited papers in Decision Sciences

All highly cited Decision Sciences papers

Questions about this paper

How do I cite this paper in APA style?

Cheng, J., Wang, F., Dai, Z., Li, Z., Kong, X., Xinwei, W., Zhang, H., & Wang, Y. (2026). Head-to-head evaluation of large language models for automated risk of bias assessment in randomized controlled trials: A two-phase diagnostic study with meta-reanalysis. Digital Health, 12, 20552076261487043. https://doi.org/10.1177/20552076261487043

How many times has this paper been cited?

0 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at Digital Health: sign in and open the full text from this page.

Who wrote this paper?

Ji Cheng, Fuqiang Wang, Zhangyi Dai, Zhiyang Li, Xiangchao Kong, Wang Xinwei, Hanlu Zhang, Yun Wang.

What is the DOI of this paper?

10.1177/20552076261487043, permanently at https://doi.org/10.1177/20552076261487043. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.