Preprint · 2026 · Open access

High Average Accuracy Coexists with Substantial Between-Setting Variability: A Systematic Review and Meta-analysis of Large Language Models for Data Extraction in Evidence Synthesis

Carlos Darcy Alves Bersot

Research Square · Published 13 October 2026 · DOI 10.21203/rs.3.rs-11332938/v1

Open PDFView at the publisherCite this paper

In brief

“High Average Accuracy Coexists with Substantial Between-Setting Variability: A Systematic Review and Meta-analysis of Large Language Models for Data Extraction in Evidence Synthesis” is a 2026 preprint by Carlos Darcy Alves Bersot, published in Research Square. OpenAlex counts 0 citations as of 18 November 2026. It is open access under a CC BY licence: the full text is free to read at Research Square.

Citations
0
References
56
Authors
1

Abstract

Large language models (LLMs) are increasingly used for data extraction in systematic reviews, but reported performance varies across extraction tasks, models, domains, reference standards, and definitions of correctness. We systematically reviewed empirical studies of automated or AI-assisted data extraction for evidence synthesis and performed an estimand-driven meta-analysis of uncorrected LLM extraction against human or otherwise validated reference standards. Forty-six reports representing 45 independent study units were included in the systematic review. Eight independent studies contributed 10 model-level estimates and 29,643 item-level evaluations to the broad primary quantitative synthesis. Proportions were analyzed on the logit scale using a multilevel random-effects model with model-level estimates nested within study and cluster-robust inference. Pooled item-level accuracy was 86.4% (cluster-robust 95% CI, 62.6%–96.0%), with generalized I² of 99.6% and a model-based 95% prediction interval of 19.2%–99.4%, indicating marked between-setting variability. In a construct-restricted sensitivity analysis excluding an open-ended, recall-intensive extraction setting, seven studies contributed nine model-level estimates and 29,343 item-level evaluations; pooled accuracy was 91.2% (95% CI, 83.1%–95.6%) and the prediction interval narrowed to 66.7%–98.2%. Leave-one-study-out analyses showed that omission of individual studies other than this methodologically distinct setting had comparatively limited influence on the pooled mean, whereas heterogeneity remained substantial. Across the broader evidence base, performance varied according to extraction task, data type, workflow design, and definition of correctness. These findings indicate that average LLM extraction accuracy can be high, but performance cannot be assumed to generalize across evidence-synthesis settings. Task-specific validation and human verification remain important, particularly for ambiguous, missing, derived, numerical, or otherwise consequential information

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Bersot, C. D. A. (2026). High Average Accuracy Coexists with Substantial Between-Setting Variability: A Systematic Review and Meta-analysis of Large Language Models for Data Extraction in Evidence Synthesis. Research Square. https://doi.org/10.21203/rs.3.rs-11332938/v1

MLA 9

Bersot, Carlos Darcy Alves. “High Average Accuracy Coexists with Substantial Between-Setting Variability: A Systematic Review and Meta-analysis of Large Language Models for Data Extraction in Evidence Synthesis.” Research Square, 2026. https://doi.org/10.21203/rs.3.rs-11332938/v1.

Chicago

Bersot, Carlos Darcy Alves. 2026. “High Average Accuracy Coexists with Substantial Between-Setting Variability: A Systematic Review and Meta-analysis of Large Language Models for Data Extraction in Evidence Synthesis.” Research Square. https://doi.org/10.21203/rs.3.rs-11332938/v1.

Harvard

Bersot, C. D. A. (2026) ‘High Average Accuracy Coexists with Substantial Between-Setting Variability: A Systematic Review and Meta-analysis of Large Language Models for Data Extraction in Evidence Synthesis’, Research Square. doi:10.21203/rs.3.rs-11332938/v1.

IEEE

C. D. A. Bersot, “High Average Accuracy Coexists with Substantial Between-Setting Variability: A Systematic Review and Meta-analysis of Large Language Models for Data Extraction in Evidence Synthesis,” Research Square, 2026, doi: 10.21203/rs.3.rs-11332938/v1.

BibTeX

@misc{bersot2026high,
  title     = {{High Average Accuracy Coexists with Substantial Between-Setting Variability: A Systematic Review and Meta-analysis of Large Language Models for Data Extraction in Evidence Synthesis}},
  author    = {Bersot, Carlos Darcy Alves},
  howpublished = {Research Square},
  year      = {2026},
  doi       = {10.21203/rs.3.rs-11332938/v1},
  url       = {https://doi.org/10.21203/rs.3.rs-11332938/v1}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews

  • large language models
  • automated data extraction
  • evidence synthesis
  • systematic reviews
  • extraction accuracy
  • AI-assisted data extraction
  • human verification
  • heterogeneity

Work with this paper

Most-cited papers in Decision Sciences

All highly cited Decision Sciences papers

Questions about this paper

How do I cite this paper in APA style?

Bersot, C. D. A. (2026). High Average Accuracy Coexists with Substantial Between-Setting Variability: A Systematic Review and Meta-analysis of Large Language Models for Data Extraction in Evidence Synthesis. Research Square. https://doi.org/10.21203/rs.3.rs-11332938/v1

How many times has this paper been cited?

0 times, according to OpenAlex on 18 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at Research Square: sign in and open the PDF from this page.

Who wrote this paper?

Carlos Darcy Alves Bersot.

What is the DOI of this paper?

10.21203/rs.3.rs-11332938/v1, permanently at https://doi.org/10.21203/rs.3.rs-11332938/v1. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 18 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.