Journal article · 2026 · Open access
What level of automation is “good enough”? A benchmark of large language models for meta-analysis data extraction
Research Synthesis Methods, 17(4), 671–692 · Published 26 January 2026 · DOI 10.1017/rsm.2025.10066
In brief
“What level of automation is “good enough”? A benchmark of large language models for meta-analysis data extraction” is a 2026 journal article by Lingbo Li and 2 co-authors, published in Research Synthesis Methods. OpenAlex counts 17 citations as of 18 November 2026, which puts it in the top 7% of works published in 2026 by citations. It is open access under a CC BY licence: the full text is free to read at Research Synthesis Methods.
- Citations
- 17
- Most cited in
- 2026 (15)
- Field-weighted impact
- 25.82
- References
- 44
- Authors
- 3
Abstract
Automating data extraction from full-text randomized controlled trials for meta-analysis remains a significant challenge. This study evaluates the practical performance of three large language models (LLMs) (Gemini-2.0-flash, Grok-3, and GPT-4o-mini) across tasks involving statistical results, risk-of-bias assessments, and study-level characteristics in three medical domains: hypertension, diabetes, and orthopaedics. We tested four distinct prompting strategies (basic prompting, self-reflective prompting, model ensemble, and customized prompts) to determine how to improve extraction quality. All models demonstrate high precision but consistently suffer from poor recall by omitting key information. We found that customized prompts were the most effective, boosting recall by up to 15%. Based on this analysis, we propose a three-tiered set of guidelines for using LLMs in data extraction, matching data types to appropriate levels of automation based on task complexity and risk. Our study offers practical advice for automating data extraction in real-world meta-analyses, balancing LLM efficiency with expert oversight through targeted, task-specific automation.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Li, L., Mathrani, A., & Sušnjak, T. (2026). What level of automation is “good enough”? A benchmark of large language models for meta-analysis data extraction. Research Synthesis Methods, 17(4), 671–692. https://doi.org/10.1017/rsm.2025.10066
MLA 9
Li, Lingbo, et al. “What level of automation is “good enough”? A benchmark of large language models for meta-analysis data extraction.” Research Synthesis Methods, vol. 17, no. 4, 2026, pp. 671–692. https://doi.org/10.1017/rsm.2025.10066.
Chicago
Li, Lingbo, Anuradha Mathrani, and Teo Sušnjak. 2026. “What level of automation is “good enough”? A benchmark of large language models for meta-analysis data extraction.” Research Synthesis Methods 17 (4): 671–692. https://doi.org/10.1017/rsm.2025.10066.
Harvard
Li, L., Mathrani, A. and Sušnjak, T. (2026) ‘What level of automation is “good enough”? A benchmark of large language models for meta-analysis data extraction’, Research Synthesis Methods, 17(4), pp. 671–692. doi:10.1017/rsm.2025.10066.
IEEE
L. Li, A. Mathrani, and T. Sušnjak, “What level of automation is “good enough”? A benchmark of large language models for meta-analysis data extraction,” Research Synthesis Methods, vol. 17, no. 4, pp. 671–692, 2026, doi: 10.1017/rsm.2025.10066.
BibTeX
@article{li2026what,
title = {{What level of automation is “good enough”? A benchmark of large language models for meta-analysis data extraction}},
author = {Li, Lingbo and Mathrani, Anuradha and Sušnjak, Teo},
journal = {Research Synthesis Methods},
year = {2026},
volume = {17},
number = {4},
pages = {671--692},
doi = {10.1017/rsm.2025.10066},
url = {https://doi.org/10.1017/rsm.2025.10066}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- large language models
- automated data extraction
- meta-analysis
- randomized controlled trials
- prompting strategies
- ensemble learning
- risk of bias assessment
- GPT-4o-mini
Work with this paper
Most-cited papers in Decision Sciences
- The PRISMA 2020 statement: an updated guideline for reporting systematic reviewsMatthew James Page, Joanne E. McKenzie, Patrick M. Bossuyt and 23 more · BMJ · 2021 · 144,919 citations
- Evaluating Structural Equation Models with Unobservable Variables and Measurement ErrorClaes Fornell, David F. Larcker · Journal of Marketing Research · 1981 · 118,336 citations
- Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternativesLi‐tze Hu, Peter M. Bentler · Structural Equation Modeling A Multidisciplinary Journal · 1999 · 117,461 citations
- Common method biases in behavioral research: A critical review of the literature and recommended remedies.Philip M. Podsakoff, SCOTT B. MACKENZIE, Jeong Yeon Lee and 1 more · Journal of Applied Psychology · 2003 · 89,248 citations
- Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statementDavid Moher, Alessandro Liberati, Jennifer Tetzlaff and 2 more · BMJ · 2009 · 85,524 citations
Questions about this paper
How do I cite this paper in APA style?
Li, L., Mathrani, A., & Sušnjak, T. (2026). What level of automation is “good enough”? A benchmark of large language models for meta-analysis data extraction. Research Synthesis Methods, 17(4), 671–692. https://doi.org/10.1017/rsm.2025.10066
How many times has this paper been cited?
17 times, according to OpenAlex on 18 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at Research Synthesis Methods: sign in and open the PDF from this page.
Who wrote this paper?
Lingbo Li, Anuradha Mathrani, Teo Sušnjak.
What is the DOI of this paper?
10.1017/rsm.2025.10066, permanently at https://doi.org/10.1017/rsm.2025.10066. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 18 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.