Preprint · 2026 · Open access
Stochastic Parrots or Singing in Harmony? Testing Five Leading LLMs for their Ability to Replicate a Human Survey with Synthetic Data
arXiv (Cornell University) · Published 10 February 2026 · DOI 10.48550/arxiv.2603.00059
In brief
“Stochastic Parrots or Singing in Harmony? Testing Five Leading LLMs for their Ability to Replicate a Human Survey with Synthetic Data” is a 2026 preprint by Jason Miklian and 2 co-authors, published in arXiv (Cornell University). OpenAlex counts 5 citations as of 18 November 2026. It is open access under a CC BY-NC-SA licence: the full text is free to read at arXiv (Cornell University).
- Citations
- 5
- Most cited in
- 2026 (5)
- References
- 31
- Authors
- 3
Abstract
How well can AI-derived synthetic research data replicate the responses of human participants? An emerging literature has begun to engage with this question, which carries deep implications for organizational research practice. This article presents a comparison between a human-respondent survey of 420 Silicon Valley coders and developers and synthetic survey data designed to simulate real survey takers generated by five leading Generative AI Large Language Models: ChatGPT Thinking 5 Pro, Claude Sonnet 4.5 Pro plus Claude CoWork 1.123, Gemini Advanced 2.5 Pro, Incredible 1.0, and DeepSeek 3.2. Our findings reveal that while AI agents produced technically plausible results that lean more towards replicability and harmonization than assumed, none were able to capture the counterintuitive insights that made the human survey valuable. Moreover, deviations grouped together for all models, leaving the real data as the outlier. Our key finding is that while leading LLMs are increasingly being used to scale, replicate and replace human survey responses in research, these advances only show an increased capacity to parrot conventional wisdom in harmony with each other rather than revealing novel findings. If synthetic respondents are used in future research, we need more replicable validation protocols and reporting standards for when and where synthetic survey data can be used responsibly, a gap that this paper fills. Our results suggest that synthetic survey responses cannot meaningfully model real human social beliefs within organizations, particularly in contexts lacking previously documented evidence. We conclude that synthetic survey-based research should be cast not as a substitute for rigorous survey methods, but as an increasingly reliable pre- or post-fieldwork instrument for identifying societal assumptions, conventional wisdoms, and other expectations about research populations.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Miklian, J., Hoelscher, K., & Katsos, J. E. (2026). Stochastic Parrots or Singing in Harmony? Testing Five Leading LLMs for their Ability to Replicate a Human Survey with Synthetic Data. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2603.00059
MLA 9
Miklian, Jason, et al. “Stochastic Parrots or Singing in Harmony? Testing Five Leading LLMs for their Ability to Replicate a Human Survey with Synthetic Data.” arXiv (Cornell University), 2026. https://doi.org/10.48550/arxiv.2603.00059.
Chicago
Miklian, Jason, Kristian Hoelscher, and John E. Katsos. 2026. “Stochastic Parrots or Singing in Harmony? Testing Five Leading LLMs for their Ability to Replicate a Human Survey with Synthetic Data.” arXiv (Cornell University). https://doi.org/10.48550/arxiv.2603.00059.
Harvard
Miklian, J., Hoelscher, K. and Katsos, J. E. (2026) ‘Stochastic Parrots or Singing in Harmony? Testing Five Leading LLMs for their Ability to Replicate a Human Survey with Synthetic Data’, arXiv (Cornell University). doi:10.48550/arxiv.2603.00059.
IEEE
J. Miklian, K. Hoelscher, and J. E. Katsos, “Stochastic Parrots or Singing in Harmony? Testing Five Leading LLMs for their Ability to Replicate a Human Survey with Synthetic Data,” arXiv (Cornell University), 2026, doi: 10.48550/arxiv.2603.00059.
BibTeX
@misc{miklian2026stochastic,
title = {{Stochastic Parrots or Singing in Harmony? Testing Five Leading LLMs for their Ability to Replicate a Human Survey with Synthetic Data}},
author = {Miklian, Jason and Hoelscher, Kristian and Katsos, John E.},
howpublished = {arXiv (Cornell University)},
year = {2026},
doi = {10.48550/arxiv.2603.00059},
url = {https://doi.org/10.48550/arxiv.2603.00059}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Social Sciences › Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- synthetic survey data
- large language models
- generative AI
- survey replication
- organizational analysis
- conventional wisdom
- reporting guidelines
- survey harmonization
Work with this paper
Most-cited papers in Social Sciences
- Using thematic analysis in psychologyVirginia Braun, Victoria Clarke · Qualitative Research in Psychology · 2006 · 240,222 citations
- Case Study Research: Design and MethodsRobert K. Yin · 1984 · 81,951 citations
- Three Approaches to Qualitative Content AnalysisHsiu-Fang Hsieh, Sarah Elizabeth Shannon · Qualitative Health Research · 2005 · 49,623 citations
- Qualitative Data Analysis: An Expanded SourcebookMatthew B. Miles, A Michael Huberman · Journal of Environmental Psychology · 1994 · 48,311 citations
- Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groupsAllison Tong, Peter Sainsbury, Jonathan C. Craig · International Journal for Quality in Health Care · 2007 · 45,353 citations
Questions about this paper
How do I cite this paper in APA style?
Miklian, J., Hoelscher, K., & Katsos, J. E. (2026). Stochastic Parrots or Singing in Harmony? Testing Five Leading LLMs for their Ability to Replicate a Human Survey with Synthetic Data. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2603.00059
How many times has this paper been cited?
5 times, according to OpenAlex on 18 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at arXiv (Cornell University): sign in and open the PDF from this page.
Who wrote this paper?
Jason Miklian, Kristian Hoelscher, John E. Katsos.
What is the DOI of this paper?
10.48550/arxiv.2603.00059, permanently at https://doi.org/10.48550/arxiv.2603.00059. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 18 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.