Preprint · 2026 · Open access
AI Agent Prevalence and Data Quality Across Multiple Online Sample Providers
PsyArXiv (OSF Preprints) · Published 1 April 2026 · DOI 10.31234/osf.io/pvdjr_v1
In brief
“AI Agent Prevalence and Data Quality Across Multiple Online Sample Providers” is a 2026 preprint by Andrew Gordon and 6 co-authors, published in PsyArXiv (OSF Preprints). OpenAlex counts 7 citations as of 18 November 2026. It is open access under a public domain licence: the full text is free to read at PsyArXiv (OSF Preprints).
- Citations
- 7
- Most cited in
- 2026 (7)
- Authors
- 7
Abstract
Online recruitment platforms have become the dominant infrastructure for behavioral research, yet data quality concerns have acquired new urgency with the emergence of large language models (LLMs). Recent work showing that LLM-based agents can complete surveys while evading standard quality checks has prompted alarm about synthetic respondents infiltrating samples at scale. However, demonstrating agent capability is not equivalent to demonstrating ecosystem-level deployment, and variation in quality among human respondents across platform types may be a more consequential threat. We address both questions in a single pre-registered study: (1) what is the actual prevalence of AI agents across platforms, and (2) how does human data quality vary across structural market segments? We recruited 5,200 respondents across 13 conditions from 10 platforms spanning direct first-party panels, hybrid networks, and marketplace aggregators. Agent detection employed an automated environment check achieving perfect discrimination in pilot testing, plus a secondary battery of six behavioral indicators. Human quality was assessed across seven behavioral dimensions alongside metadata including device type, ecosystem activity, and cost efficiency. Agent detections were concentrated almost exclusively on Amazon MTurk (11–16%), with all other platforms at or below 1%; detected responses showed profiles more consistent with traditional bots than LLM-based agents. Evidence of humans using LLMs to augment answers, particularly on open-ended or difficult items, was consistent with recent work assuming no deployed mitigation. Human data quality varied substantially by platform type, with direct panels outperforming hybrid platforms, which outperformed marketplace platforms, across nearly all measures, an effect several times larger than that of agents or LLM-augmentation. Cost-efficiency analyses revealed direct panels, despite higher nominal costs, were most economical once quality thresholds were applied. The field’s most pressing data quality challenge remains systematic variation in human respondent quality by platform type, not AI agent infiltration.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Gordon, A., Rothschild, D. M., Affonso, F. M., Sulik, J., Hauser, D., Pepin, K., & Jones, S. (2026). AI Agent Prevalence and Data Quality Across Multiple Online Sample Providers. PsyArXiv (OSF Preprints). https://doi.org/10.31234/osf.io/pvdjr_v1
MLA 9
Gordon, Andrew, et al. “AI Agent Prevalence and Data Quality Across Multiple Online Sample Providers.” PsyArXiv (OSF Preprints), 2026. https://doi.org/10.31234/osf.io/pvdjr_v1.
Chicago
Gordon, Andrew, David M. Rothschild, Felipe M. Affonso, Justin Sulik, David Hauser, Karine Pepin, and Simon Jones. 2026. “AI Agent Prevalence and Data Quality Across Multiple Online Sample Providers.” PsyArXiv (OSF Preprints). https://doi.org/10.31234/osf.io/pvdjr_v1.
Harvard
Gordon, A. et al. (2026) ‘AI Agent Prevalence and Data Quality Across Multiple Online Sample Providers’, PsyArXiv (OSF Preprints). doi:10.31234/osf.io/pvdjr_v1.
IEEE
A. Gordon et al., “AI Agent Prevalence and Data Quality Across Multiple Online Sample Providers,” PsyArXiv (OSF Preprints), 2026, doi: 10.31234/osf.io/pvdjr_v1.
BibTeX
@misc{gordon2026agent,
title = {{AI Agent Prevalence and Data Quality Across Multiple Online Sample Providers}},
author = {Gordon, Andrew and Rothschild, David M. and Affonso, Felipe M. and Sulik, Justin and Hauser, David and Pepin, Karine and Jones, Simon},
howpublished = {PsyArXiv (OSF Preprints)},
year = {2026},
doi = {10.31234/osf.io/pvdjr_v1},
url = {https://doi.org/10.31234/osf.io/pvdjr_v1}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Social Sciences › Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- survey data quality
- recruitment websites
- large language models
- bot detection
- Amazon Mechanical Turk
- cost efficiency
- behavioral indicators
Work with this paper
Most-cited papers in Social Sciences
- Using thematic analysis in psychologyVirginia Braun, Victoria Clarke · Qualitative Research in Psychology · 2006 · 240,222 citations
- Case Study Research: Design and MethodsRobert K. Yin · 1984 · 81,951 citations
- Three Approaches to Qualitative Content AnalysisHsiu-Fang Hsieh, Sarah Elizabeth Shannon · Qualitative Health Research · 2005 · 49,623 citations
- Qualitative Data Analysis: An Expanded SourcebookMatthew B. Miles, A Michael Huberman · Journal of Environmental Psychology · 1994 · 48,311 citations
- Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groupsAllison Tong, Peter Sainsbury, Jonathan C. Craig · International Journal for Quality in Health Care · 2007 · 45,353 citations
Questions about this paper
How do I cite this paper in APA style?
Gordon, A., Rothschild, D. M., Affonso, F. M., Sulik, J., Hauser, D., Pepin, K., & Jones, S. (2026). AI Agent Prevalence and Data Quality Across Multiple Online Sample Providers. PsyArXiv (OSF Preprints). https://doi.org/10.31234/osf.io/pvdjr_v1
How many times has this paper been cited?
7 times, according to OpenAlex on 18 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at PsyArXiv (OSF Preprints): sign in and open the full text from this page.
Who wrote this paper?
Andrew Gordon, David M. Rothschild, Felipe M. Affonso, Justin Sulik, David Hauser, Karine Pepin, Simon Jones.
What is the DOI of this paper?
10.31234/osf.io/pvdjr_v1, permanently at https://doi.org/10.31234/osf.io/pvdjr_v1. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 18 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.