Journal article · 2026 · Open access

A Fine-Tuned Large Language Model Workflow to Support Risk of Bias Assessment: Development, External Validation, and Reviewer Verification Study

Yitian Li, Jingyu Chen, Liting Liang, Xingyun Geng, Guohua Wang, Jinsong Geng

Journal of Medical Internet Research, 28, e108001 · Published 13 November 2026 · DOI 10.2196/108001

Open full textCite this paper

In brief

“A Fine-Tuned Large Language Model Workflow to Support Risk of Bias Assessment: Development, External Validation, and Reviewer Verification Study” is a 2026 journal article by Yitian Li and 5 co-authors, published in Journal of Medical Internet Research. OpenAlex counts 0 citations as of 18 November 2026. It is open access under a CC BY licence: the full text is free to read at Journal of Medical Internet Research.

Citations
0
References
40
Authors
6

Abstract

Background Given that randomized controlled trials (RCTs) are the gold standard for effectiveness research, assessing their risk of bias (RoB) becomes an indispensable step in systematic reviews. Yet this task remains time-consuming and prone to reviewer subjectivity. While large language models may accelerate the process, they lack transparent, traceable judgments linked to trial reports. Objective To bridge methodological gaps and improve both efficiency and consistency, this study developed and externally validated TRoB-1, a fine-tuned open-weight large language model workflow for RoB assessment of RCTs. Methods TRoB-1 was developed specifically for the Cochrane risk of bias tool version 1 (RoB 1), which is the most widely adopted instrument for critical appraisal of RCTs. The workflow generates preliminary judgments and supporting quotations from reviewer-selected passages for the target RoB domains. The workflow was built on 2-stage supervised fine-tuning of Gemma 2-9B-Instruct using 4-bit quantized low-rank adaptation. The training set comprised 4321 RoB records extracted from systematic reviews of chronic obstructive pulmonary disease, diabetes, and hypertension. The validation set consisted of 161 records from recently published RCTs covering a broader disease area. The reference standard was established by 2 independent reviewers. Two clinicians blinded to the reference standard verified TRoB-1-assisted assessments through a locally hosted web interface. Using the same reference standard, the performance of TRoB-1 raw outputs was compared with that of the Gemma 2 base workflow, and clinician-verified outputs were compared with raw TRoB-1 outputs. Results TRoB-1 successfully generated structured outputs for all 161 validation records. In independent clinician-assisted assessments using the same workflow, raw TRoB-1 accuracy ranged from 73.3% (n=118) to 73.9% (n=119), compared with 52.2% (n=84) to 60.2% (n=97) for the Gemma 2 base model. After verification, accuracy ranged from 82.0% (n=132) to 84.5% (n=136) and macro- F 1 from 0.709 to 0.776. Accuracy remained higher for raw TRoB-1 after excluding invalid base-model outputs, and verification did not change any correct judgment to an incorrect one. Clinician 1 corrected 13 initially incorrect judgments and clinician 2 corrected 18. The F 1 -score for judgments of high-RoB increased from 0.600 to 0.784 for clinician 1 and from 0.571 to 0.821 for clinician 2. Conclusions TRoB-1 combined task-specific fine-tuning, evidence-linked outputs, and clinician verification in a transparent workflow for reviewer-supported RoB assessment. The TRoB-1 web interface provides a practical platform for integrating RoB into evidence synthesis, particularly for living systematic reviews where timely updates are critical.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Li, Y., Chen, J., Liang, L., Geng, X., Wang, G., & Geng, J. (2026). A Fine-Tuned Large Language Model Workflow to Support Risk of Bias Assessment: Development, External Validation, and Reviewer Verification Study. Journal of Medical Internet Research, 28, e108001. https://doi.org/10.2196/108001

MLA 9

Li, Yitian, et al. “A Fine-Tuned Large Language Model Workflow to Support Risk of Bias Assessment: Development, External Validation, and Reviewer Verification Study.” Journal of Medical Internet Research, vol. 28, 2026, p. e108001. https://doi.org/10.2196/108001.

Chicago

Li, Yitian, Jingyu Chen, Liting Liang, Xingyun Geng, Guohua Wang, and Jinsong Geng. 2026. “A Fine-Tuned Large Language Model Workflow to Support Risk of Bias Assessment: Development, External Validation, and Reviewer Verification Study.” Journal of Medical Internet Research 28: e108001. https://doi.org/10.2196/108001.

Harvard

Li, Y. et al. (2026) ‘A Fine-Tuned Large Language Model Workflow to Support Risk of Bias Assessment: Development, External Validation, and Reviewer Verification Study’, Journal of Medical Internet Research, 28, p. e108001. doi:10.2196/108001.

IEEE

Y. Li, J. Chen, L. Liang, X. Geng, G. Wang, and J. Geng, “A Fine-Tuned Large Language Model Workflow to Support Risk of Bias Assessment: Development, External Validation, and Reviewer Verification Study,” Journal of Medical Internet Research, vol. 28, p. e108001, 2026, doi: 10.2196/108001.

BibTeX

@article{li2026fine,
  title     = {{A Fine-Tuned Large Language Model Workflow to Support Risk of Bias Assessment: Development, External Validation, and Reviewer Verification Study}},
  author    = {Li, Yitian and Chen, Jingyu and Liang, Liting and Geng, Xingyun and Wang, Guohua and Geng, Jinsong},
  journal   = {Journal of Medical Internet Research},
  year      = {2026},
  volume    = {28},
  pages     = {e108001},
  doi       = {10.2196/108001},
  url       = {https://doi.org/10.2196/108001}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews

  • risk of bias assessment
  • large language models
  • randomized controlled trials
  • fine-tuning
  • low-rank adaptation
  • Gemma 2
  • external validation
  • chronic obstructive pulmonary disease

Work with this paper

Most-cited papers in Decision Sciences

All highly cited Decision Sciences papers

Questions about this paper

How do I cite this paper in APA style?

Li, Y., Chen, J., Liang, L., Geng, X., Wang, G., & Geng, J. (2026). A Fine-Tuned Large Language Model Workflow to Support Risk of Bias Assessment: Development, External Validation, and Reviewer Verification Study. Journal of Medical Internet Research, 28, e108001. https://doi.org/10.2196/108001

How many times has this paper been cited?

0 times, according to OpenAlex on 18 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at Journal of Medical Internet Research: sign in and open the full text from this page.

Who wrote this paper?

Yitian Li, Jingyu Chen, Liting Liang, Xingyun Geng, Guohua Wang, Jinsong Geng.

What is the DOI of this paper?

10.2196/108001, permanently at https://doi.org/10.2196/108001. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 18 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.