Journal article · 2026 · Open access

AI-assisted teams outperform AI-led teams but not human-only teams in assessing research reproducibility in quantitative social science

Abel Brodeur, David Valenta, Alexandru Marcoci, Juan P. Aparicio, Derek Mikola, Bruno Barbarioli, Rohan Alexander, Lachlan Deer, Tom Stafford, Lars Vilhuber, Gunther Bensch, Fabio Yoshio Suguri Motoki and 260 more

Proceedings of the National Academy of Sciences, 123(22), e2524747123 · Published 28 May 2026 · DOI 10.1073/pnas.2524747123

Open full textCite this paper

In brief

“AI-assisted teams outperform AI-led teams but not human-only teams in assessing research reproducibility in quantitative social science” is a 2026 journal article by Abel Brodeur and 271 co-authors, published in Proceedings of the National Academy of Sciences. OpenAlex counts 16 citations as of 18 November 2026, which puts it in the top 7% of works published in 2026 by citations. It is open access under a CC BY-NC-ND licence: the full text is free to read at Proceedings of the National Academy of Sciences.

Citations
16
Most cited in
2026 (14)
Field-weighted impact
44.40
References
61
Authors
272

Abstract

Large Language Models (LLMs) such as ChatGPT are transforming how scientists conduct and validate research, offering promise as tools to improve scientific reproducibility. However, computational reproducibility and error detection remain expensive and labor-intensive. We experimentally test how collaboration between researchers and LLM assistants influences the reproduction of quantitative social science findings across different levels of AI autonomy. We randomly assigned 288 researchers to 103 teams working under three conditions: human-only, AI-assisted (using ChatGPT as a collaborative tool), or AI-led (ChatGPT operating with minimal human oversight). Teams reproduced published results from leading social science journals, detected coding errors, and proposed robustness checks. Human-only and AI-assisted teams achieved comparable reproduction rates (94% vs. 91%) and performed similarly on most outcomes, except human-only teams identified significantly more major coding errors. Both substantially outperformed AI-led teams, which achieved only a 37% reproduction rate, detected fewer errors across all categories, proposed weaker robustness checks, and required more time. This autonomous approach, however, likely represents only a lower bound of AI capabilities. Despite rapid model advances, expert human judgment currently remains indispensable for reliable empirical verification. While AI assistance did not degrade most outcomes, it provided no measurable advantages and was associated with reduced detection of major errors. However, the 37% autonomous reproduction rate indicates that AI could provide value in settings where scale or cost constraints preclude human review of papers, even though general-purpose LLMs offer no immediate advantages for human-supervised verification.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Brodeur, A., Valenta, D., Marcoci, A., Aparicio, J. P., Mikola, D., Barbarioli, B., Alexander, R., Deer, L., Stafford, T., Vilhuber, L., Bensch, G., Motoki, F. Y. S., Abdelhady, M., Abdelmoula, Y., Baki, G. A., Aguirre, T., Aiyer, S., Akhtar, S., Akhtar, F., . . . Zozoungbo, C. (2026). AI-assisted teams outperform AI-led teams but not human-only teams in assessing research reproducibility in quantitative social science. Proceedings of the National Academy of Sciences, 123(22), e2524747123. https://doi.org/10.1073/pnas.2524747123

MLA 9

Brodeur, Abel, et al. “AI-assisted teams outperform AI-led teams but not human-only teams in assessing research reproducibility in quantitative social science.” Proceedings of the National Academy of Sciences, vol. 123, no. 22, 2026, p. e2524747123. https://doi.org/10.1073/pnas.2524747123.

Chicago

Brodeur, Abel, David Valenta, Alexandru Marcoci, Juan P. Aparicio, Derek Mikola, Bruno Barbarioli, Rohan Alexander, et al. 2026. “AI-assisted teams outperform AI-led teams but not human-only teams in assessing research reproducibility in quantitative social science.” Proceedings of the National Academy of Sciences 123 (22): e2524747123. https://doi.org/10.1073/pnas.2524747123.

Harvard

Brodeur, A. et al. (2026) ‘AI-assisted teams outperform AI-led teams but not human-only teams in assessing research reproducibility in quantitative social science’, Proceedings of the National Academy of Sciences, 123(22), p. e2524747123. doi:10.1073/pnas.2524747123.

IEEE

A. Brodeur et al., “AI-assisted teams outperform AI-led teams but not human-only teams in assessing research reproducibility in quantitative social science,” Proceedings of the National Academy of Sciences, vol. 123, no. 22, p. e2524747123, 2026, doi: 10.1073/pnas.2524747123.

BibTeX

@article{brodeur2026assisted,
  title     = {{AI-assisted teams outperform AI-led teams but not human-only teams in assessing research reproducibility in quantitative social science}},
  author    = {Brodeur, Abel and Valenta, David and Marcoci, Alexandru and Aparicio, Juan P. and Mikola, Derek and Barbarioli, Bruno and Alexander, Rohan and Deer, Lachlan and Stafford, Tom and Vilhuber, Lars and Bensch, Gunther and Motoki, Fabio Yoshio Suguri and Abdelhady, Mohamed and Abdelmoula, Yousra and Baki, Ghina Abdul and Aguirre, Tomás and Aiyer, Sriraj and Akhtar, Shumi and Akhtar, Farida and Albada, Melle R. and Altman, Micah and Angenendt, David and Lari, Zahra Arjmandi and Tejada, Jorge Armando De León and Arana, David Rodriguez and Asanov, Igor and Noha, Anastasiya-Mariya Asanov and Ashong, Rebecca and Auer, Tobias and Bahamonde-Birke, Francisco J. and others},
  journal   = {Proceedings of the National Academy of Sciences},
  year      = {2026},
  volume    = {123},
  number    = {22},
  pages     = {e2524747123},
  doi       = {10.1073/pnas.2524747123},
  url       = {https://doi.org/10.1073/pnas.2524747123}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Social Sciences › Social Sciences › General Social Sciences › Computational and Text Analysis Methods

  • large language models
  • ChatGPT
  • research reproducibility
  • computational reproducibility
  • AI-assisted research
  • human-AI collaboration
  • robustness testing
  • quantitative social science

Work with this paper

Most-cited papers in Social Sciences

All highly cited Social Sciences papers

Questions about this paper

How do I cite this paper in APA style?

Brodeur, A., Valenta, D., Marcoci, A., Aparicio, J. P., Mikola, D., Barbarioli, B., Alexander, R., Deer, L., Stafford, T., Vilhuber, L., Bensch, G., Motoki, F. Y. S., Abdelhady, M., Abdelmoula, Y., Baki, G. A., Aguirre, T., Aiyer, S., Akhtar, S., Akhtar, F., . . . Zozoungbo, C. (2026). AI-assisted teams outperform AI-led teams but not human-only teams in assessing research reproducibility in quantitative social science. Proceedings of the National Academy of Sciences, 123(22), e2524747123. https://doi.org/10.1073/pnas.2524747123

How many times has this paper been cited?

16 times, according to OpenAlex on 18 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at Proceedings of the National Academy of Sciences: sign in and open the full text from this page.

Who wrote this paper?

Abel Brodeur, David Valenta, Alexandru Marcoci, Juan P. Aparicio, Derek Mikola, Bruno Barbarioli, Rohan Alexander, Lachlan Deer, Tom Stafford, Lars Vilhuber, Gunther Bensch, Fabio Yoshio Suguri Motoki and 260 more.

What is the DOI of this paper?

10.1073/pnas.2524747123, permanently at https://doi.org/10.1073/pnas.2524747123. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 18 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.