Preprint · 2026 · Open access

AI Agent Prevalence and Data Quality Across Multiple Online Sample Providers

Andrew Gordon, David M. Rothschild, Felipe M. Affonso, Justin Sulik, David Hauser, Karine Pepin, Simon Jones

PsyArXiv (OSF Preprints) · Published 1 April 2026 · DOI 10.31234/osf.io/pvdjr_v1

Open full textCite this paper

In brief

“AI Agent Prevalence and Data Quality Across Multiple Online Sample Providers” is a 2026 preprint by Andrew Gordon and 6 co-authors, published in PsyArXiv (OSF Preprints). OpenAlex counts 7 citations as of 18 November 2026. It is open access under a public domain licence: the full text is free to read at PsyArXiv (OSF Preprints).

Citations
7
Most cited in
2026 (7)
Authors
7

Abstract

Online recruitment platforms have become the dominant infrastructure for behavioral research, yet data quality concerns have acquired new urgency with the emergence of large language models (LLMs). Recent work showing that LLM-based agents can complete surveys while evading standard quality checks has prompted alarm about synthetic respondents infiltrating samples at scale. However, demonstrating agent capability is not equivalent to demonstrating ecosystem-level deployment, and variation in quality among human respondents across platform types may be a more consequential threat. We address both questions in a single pre-registered study: (1) what is the actual prevalence of AI agents across platforms, and (2) how does human data quality vary across structural market segments? We recruited 5,200 respondents across 13 conditions from 10 platforms spanning direct first-party panels, hybrid networks, and marketplace aggregators. Agent detection employed an automated environment check achieving perfect discrimination in pilot testing, plus a secondary battery of six behavioral indicators. Human quality was assessed across seven behavioral dimensions alongside metadata including device type, ecosystem activity, and cost efficiency. Agent detections were concentrated almost exclusively on Amazon MTurk (11–16%), with all other platforms at or below 1%; detected responses showed profiles more consistent with traditional bots than LLM-based agents. Evidence of humans using LLMs to augment answers, particularly on open-ended or difficult items, was consistent with recent work assuming no deployed mitigation. Human data quality varied substantially by platform type, with direct panels outperforming hybrid platforms, which outperformed marketplace platforms, across nearly all measures, an effect several times larger than that of agents or LLM-augmentation. Cost-efficiency analyses revealed direct panels, despite higher nominal costs, were most economical once quality thresholds were applied. The field’s most pressing data quality challenge remains systematic variation in human respondent quality by platform type, not AI agent infiltration.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Gordon, A., Rothschild, D. M., Affonso, F. M., Sulik, J., Hauser, D., Pepin, K., & Jones, S. (2026). AI Agent Prevalence and Data Quality Across Multiple Online Sample Providers. PsyArXiv (OSF Preprints). https://doi.org/10.31234/osf.io/pvdjr_v1

MLA 9

Gordon, Andrew, et al. “AI Agent Prevalence and Data Quality Across Multiple Online Sample Providers.” PsyArXiv (OSF Preprints), 2026. https://doi.org/10.31234/osf.io/pvdjr_v1.

Chicago

Gordon, Andrew, David M. Rothschild, Felipe M. Affonso, Justin Sulik, David Hauser, Karine Pepin, and Simon Jones. 2026. “AI Agent Prevalence and Data Quality Across Multiple Online Sample Providers.” PsyArXiv (OSF Preprints). https://doi.org/10.31234/osf.io/pvdjr_v1.

Harvard

Gordon, A. et al. (2026) ‘AI Agent Prevalence and Data Quality Across Multiple Online Sample Providers’, PsyArXiv (OSF Preprints). doi:10.31234/osf.io/pvdjr_v1.

IEEE

A. Gordon et al., “AI Agent Prevalence and Data Quality Across Multiple Online Sample Providers,” PsyArXiv (OSF Preprints), 2026, doi: 10.31234/osf.io/pvdjr_v1.

BibTeX

@misc{gordon2026agent,
  title     = {{AI Agent Prevalence and Data Quality Across Multiple Online Sample Providers}},
  author    = {Gordon, Andrew and Rothschild, David M. and Affonso, Felipe M. and Sulik, Justin and Hauser, David and Pepin, Karine and Jones, Simon},
  howpublished = {PsyArXiv (OSF Preprints)},
  year      = {2026},
  doi       = {10.31234/osf.io/pvdjr_v1},
  url       = {https://doi.org/10.31234/osf.io/pvdjr_v1}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Social Sciences › Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse

  • survey data quality
  • recruitment websites
  • large language models
  • bot detection
  • Amazon Mechanical Turk
  • cost efficiency
  • behavioral indicators

Work with this paper

Most-cited papers in Social Sciences

All highly cited Social Sciences papers

Questions about this paper

How do I cite this paper in APA style?

Gordon, A., Rothschild, D. M., Affonso, F. M., Sulik, J., Hauser, D., Pepin, K., & Jones, S. (2026). AI Agent Prevalence and Data Quality Across Multiple Online Sample Providers. PsyArXiv (OSF Preprints). https://doi.org/10.31234/osf.io/pvdjr_v1

How many times has this paper been cited?

7 times, according to OpenAlex on 18 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at PsyArXiv (OSF Preprints): sign in and open the full text from this page.

Who wrote this paper?

Andrew Gordon, David M. Rothschild, Felipe M. Affonso, Justin Sulik, David Hauser, Karine Pepin, Simon Jones.

What is the DOI of this paper?

10.31234/osf.io/pvdjr_v1, permanently at https://doi.org/10.31234/osf.io/pvdjr_v1. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 18 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.