Preprint · 2026 · Open access

Stochastic Parrots or Singing in Harmony? Testing Five Leading LLMs for their Ability to Replicate a Human Survey with Synthetic Data

Jason Miklian, Kristian Hoelscher, John E. Katsos

arXiv (Cornell University) · Published 10 February 2026 · DOI 10.48550/arxiv.2603.00059

Open PDFView at the publisherCite this paper

In brief

“Stochastic Parrots or Singing in Harmony? Testing Five Leading LLMs for their Ability to Replicate a Human Survey with Synthetic Data” is a 2026 preprint by Jason Miklian and 2 co-authors, published in arXiv (Cornell University). OpenAlex counts 5 citations as of 18 November 2026. It is open access under a CC BY-NC-SA licence: the full text is free to read at arXiv (Cornell University).

Citations
5
Most cited in
2026 (5)
References
31
Authors
3

Abstract

How well can AI-derived synthetic research data replicate the responses of human participants? An emerging literature has begun to engage with this question, which carries deep implications for organizational research practice. This article presents a comparison between a human-respondent survey of 420 Silicon Valley coders and developers and synthetic survey data designed to simulate real survey takers generated by five leading Generative AI Large Language Models: ChatGPT Thinking 5 Pro, Claude Sonnet 4.5 Pro plus Claude CoWork 1.123, Gemini Advanced 2.5 Pro, Incredible 1.0, and DeepSeek 3.2. Our findings reveal that while AI agents produced technically plausible results that lean more towards replicability and harmonization than assumed, none were able to capture the counterintuitive insights that made the human survey valuable. Moreover, deviations grouped together for all models, leaving the real data as the outlier. Our key finding is that while leading LLMs are increasingly being used to scale, replicate and replace human survey responses in research, these advances only show an increased capacity to parrot conventional wisdom in harmony with each other rather than revealing novel findings. If synthetic respondents are used in future research, we need more replicable validation protocols and reporting standards for when and where synthetic survey data can be used responsibly, a gap that this paper fills. Our results suggest that synthetic survey responses cannot meaningfully model real human social beliefs within organizations, particularly in contexts lacking previously documented evidence. We conclude that synthetic survey-based research should be cast not as a substitute for rigorous survey methods, but as an increasingly reliable pre- or post-fieldwork instrument for identifying societal assumptions, conventional wisdoms, and other expectations about research populations.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Miklian, J., Hoelscher, K., & Katsos, J. E. (2026). Stochastic Parrots or Singing in Harmony? Testing Five Leading LLMs for their Ability to Replicate a Human Survey with Synthetic Data. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2603.00059

MLA 9

Miklian, Jason, et al. “Stochastic Parrots or Singing in Harmony? Testing Five Leading LLMs for their Ability to Replicate a Human Survey with Synthetic Data.” arXiv (Cornell University), 2026. https://doi.org/10.48550/arxiv.2603.00059.

Chicago

Miklian, Jason, Kristian Hoelscher, and John E. Katsos. 2026. “Stochastic Parrots or Singing in Harmony? Testing Five Leading LLMs for their Ability to Replicate a Human Survey with Synthetic Data.” arXiv (Cornell University). https://doi.org/10.48550/arxiv.2603.00059.

Harvard

Miklian, J., Hoelscher, K. and Katsos, J. E. (2026) ‘Stochastic Parrots or Singing in Harmony? Testing Five Leading LLMs for their Ability to Replicate a Human Survey with Synthetic Data’, arXiv (Cornell University). doi:10.48550/arxiv.2603.00059.

IEEE

J. Miklian, K. Hoelscher, and J. E. Katsos, “Stochastic Parrots or Singing in Harmony? Testing Five Leading LLMs for their Ability to Replicate a Human Survey with Synthetic Data,” arXiv (Cornell University), 2026, doi: 10.48550/arxiv.2603.00059.

BibTeX

@misc{miklian2026stochastic,
  title     = {{Stochastic Parrots or Singing in Harmony? Testing Five Leading LLMs for their Ability to Replicate a Human Survey with Synthetic Data}},
  author    = {Miklian, Jason and Hoelscher, Kristian and Katsos, John E.},
  howpublished = {arXiv (Cornell University)},
  year      = {2026},
  doi       = {10.48550/arxiv.2603.00059},
  url       = {https://doi.org/10.48550/arxiv.2603.00059}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Social Sciences › Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse

  • synthetic survey data
  • large language models
  • generative AI
  • survey replication
  • organizational analysis
  • conventional wisdom
  • reporting guidelines
  • survey harmonization

Work with this paper

Most-cited papers in Social Sciences

All highly cited Social Sciences papers

Questions about this paper

How do I cite this paper in APA style?

Miklian, J., Hoelscher, K., & Katsos, J. E. (2026). Stochastic Parrots or Singing in Harmony? Testing Five Leading LLMs for their Ability to Replicate a Human Survey with Synthetic Data. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2603.00059

How many times has this paper been cited?

5 times, according to OpenAlex on 18 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at arXiv (Cornell University): sign in and open the PDF from this page.

Who wrote this paper?

Jason Miklian, Kristian Hoelscher, John E. Katsos.

What is the DOI of this paper?

10.48550/arxiv.2603.00059, permanently at https://doi.org/10.48550/arxiv.2603.00059. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 18 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.