Journal article · 2026 · Open access

What level of automation is “good enough”? A benchmark of large language models for meta-analysis data extraction

Lingbo Li, Anuradha Mathrani, Teo Sušnjak

Research Synthesis Methods, 17(4), 671–692 · Published 26 January 2026 · DOI 10.1017/rsm.2025.10066

Open PDFView at the publisherCite this paper

In brief

“What level of automation is “good enough”? A benchmark of large language models for meta-analysis data extraction” is a 2026 journal article by Lingbo Li and 2 co-authors, published in Research Synthesis Methods. OpenAlex counts 17 citations as of 18 November 2026, which puts it in the top 7% of works published in 2026 by citations. It is open access under a CC BY licence: the full text is free to read at Research Synthesis Methods.

Citations
17
Most cited in
2026 (15)
Field-weighted impact
25.82
References
44
Authors
3

Abstract

Automating data extraction from full-text randomized controlled trials for meta-analysis remains a significant challenge. This study evaluates the practical performance of three large language models (LLMs) (Gemini-2.0-flash, Grok-3, and GPT-4o-mini) across tasks involving statistical results, risk-of-bias assessments, and study-level characteristics in three medical domains: hypertension, diabetes, and orthopaedics. We tested four distinct prompting strategies (basic prompting, self-reflective prompting, model ensemble, and customized prompts) to determine how to improve extraction quality. All models demonstrate high precision but consistently suffer from poor recall by omitting key information. We found that customized prompts were the most effective, boosting recall by up to 15%. Based on this analysis, we propose a three-tiered set of guidelines for using LLMs in data extraction, matching data types to appropriate levels of automation based on task complexity and risk. Our study offers practical advice for automating data extraction in real-world meta-analyses, balancing LLM efficiency with expert oversight through targeted, task-specific automation.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Li, L., Mathrani, A., & Sušnjak, T. (2026). What level of automation is “good enough”? A benchmark of large language models for meta-analysis data extraction. Research Synthesis Methods, 17(4), 671–692. https://doi.org/10.1017/rsm.2025.10066

MLA 9

Li, Lingbo, et al. “What level of automation is “good enough”? A benchmark of large language models for meta-analysis data extraction.” Research Synthesis Methods, vol. 17, no. 4, 2026, pp. 671–692. https://doi.org/10.1017/rsm.2025.10066.

Chicago

Li, Lingbo, Anuradha Mathrani, and Teo Sušnjak. 2026. “What level of automation is “good enough”? A benchmark of large language models for meta-analysis data extraction.” Research Synthesis Methods 17 (4): 671–692. https://doi.org/10.1017/rsm.2025.10066.

Harvard

Li, L., Mathrani, A. and Sušnjak, T. (2026) ‘What level of automation is “good enough”? A benchmark of large language models for meta-analysis data extraction’, Research Synthesis Methods, 17(4), pp. 671–692. doi:10.1017/rsm.2025.10066.

IEEE

L. Li, A. Mathrani, and T. Sušnjak, “What level of automation is “good enough”? A benchmark of large language models for meta-analysis data extraction,” Research Synthesis Methods, vol. 17, no. 4, pp. 671–692, 2026, doi: 10.1017/rsm.2025.10066.

BibTeX

@article{li2026what,
  title     = {{What level of automation is “good enough”? A benchmark of large language models for meta-analysis data extraction}},
  author    = {Li, Lingbo and Mathrani, Anuradha and Sušnjak, Teo},
  journal   = {Research Synthesis Methods},
  year      = {2026},
  volume    = {17},
  number    = {4},
  pages     = {671--692},
  doi       = {10.1017/rsm.2025.10066},
  url       = {https://doi.org/10.1017/rsm.2025.10066}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews

  • large language models
  • automated data extraction
  • meta-analysis
  • randomized controlled trials
  • prompting strategies
  • ensemble learning
  • risk of bias assessment
  • GPT-4o-mini

Work with this paper

Most-cited papers in Decision Sciences

All highly cited Decision Sciences papers

Questions about this paper

How do I cite this paper in APA style?

Li, L., Mathrani, A., & Sušnjak, T. (2026). What level of automation is “good enough”? A benchmark of large language models for meta-analysis data extraction. Research Synthesis Methods, 17(4), 671–692. https://doi.org/10.1017/rsm.2025.10066

How many times has this paper been cited?

17 times, according to OpenAlex on 18 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at Research Synthesis Methods: sign in and open the PDF from this page.

Who wrote this paper?

Lingbo Li, Anuradha Mathrani, Teo Sušnjak.

What is the DOI of this paper?

10.1017/rsm.2025.10066, permanently at https://doi.org/10.1017/rsm.2025.10066. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 18 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.