Journal article · 2024 · Open access

Synthetic Replacements for Human Survey Data? The Perils of Large Language Models

James H. Bisbee, Joshua D. Clinton, Cassy Dorff, Brenton Kenkel, Jennifer Larson

Political Analysis, 32(4), 401–416 · Published 17 May 2024 · DOI 10.1017/pan.2024.5

Open full textCite this paper

In brief

“Synthetic Replacements for Human Survey Data? The Perils of Large Language Models” is a 2024 journal article by James H. Bisbee and 4 co-authors, published in Political Analysis. OpenAlex counts 522 citations as of 18 November 2026, which puts it in the top 1% of works published in 2024 by citations. It is open access under a CC BY licence: the full text is free to read at Political Analysis.

Citations
522
Most cited in
2026 (336)
Field-weighted impact
176.29
References
54
Authors
5

Abstract

Large language models (LLMs) offer new research possibilities for social scientists, but their potential as “synthetic data” is still largely unknown. In this paper, we investigate how accurately the popular LLM ChatGPT can recover public opinion, prompting the LLM to adopt different “personas” and then provide feeling thermometer scores for 11 sociopolitical groups. The average scores generated by ChatGPT correspond closely to the averages in our baseline survey, the 2016–2020 American National Election Study (ANES). Nevertheless, sampling by ChatGPT is not reliable for statistical inference: there is less variation in responses than in the real surveys, and regression coefficients often differ significantly from equivalent estimates obtained using ANES data. We also document how the distribution of synthetic responses varies with minor changes in prompt wording, and we show how the same prompt yields significantly different results over a 3-month period. Altogether, our findings raise serious concerns about the quality, reliability, and reproducibility of synthetic survey data generated by LLMs.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Bisbee, J. H., Clinton, J. D., Dorff, C., Kenkel, B., & Larson, J. (2024). Synthetic Replacements for Human Survey Data? The Perils of Large Language Models. Political Analysis, 32(4), 401–416. https://doi.org/10.1017/pan.2024.5

MLA 9

Bisbee, James H., et al. “Synthetic Replacements for Human Survey Data? The Perils of Large Language Models.” Political Analysis, vol. 32, no. 4, 2024, pp. 401–416. https://doi.org/10.1017/pan.2024.5.

Chicago

Bisbee, James H., Joshua D. Clinton, Cassy Dorff, Brenton Kenkel, and Jennifer Larson. 2024. “Synthetic Replacements for Human Survey Data? The Perils of Large Language Models.” Political Analysis 32 (4): 401–416. https://doi.org/10.1017/pan.2024.5.

Harvard

Bisbee, J. H. et al. (2024) ‘Synthetic Replacements for Human Survey Data? The Perils of Large Language Models’, Political Analysis, 32(4), pp. 401–416. doi:10.1017/pan.2024.5.

IEEE

J. H. Bisbee, J. D. Clinton, C. Dorff, B. Kenkel, and J. Larson, “Synthetic Replacements for Human Survey Data? The Perils of Large Language Models,” Political Analysis, vol. 32, no. 4, pp. 401–416, 2024, doi: 10.1017/pan.2024.5.

BibTeX

@article{bisbee2024synthetic,
  title     = {{Synthetic Replacements for Human Survey Data? The Perils of Large Language Models}},
  author    = {Bisbee, James H. and Clinton, Joshua D. and Dorff, Cassy and Kenkel, Brenton and Larson, Jennifer},
  journal   = {Political Analysis},
  year      = {2024},
  volume    = {32},
  number    = {4},
  pages     = {401--416},
  doi       = {10.1017/pan.2024.5},
  url       = {https://doi.org/10.1017/pan.2024.5}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Citations by year

YearCitationsShare of the best year
20231111
20243333
2025142142
2026336336

Its field-weighted citation impact is 176.29: it has been cited 176.29 times as often as the average work of the same field, type and year (1.00 is the world average).

Topic and keywords

Social Sciences › Social Sciences › General Social Sciences › Computational and Text Analysis Methods

  • large language models
  • synthetic survey data
  • ChatGPT
  • public opinion
  • feeling thermometers
  • sociopolitical attitudes
  • American National Election Studies
  • sampling reliability

Work with this paper

Most-cited papers in Social Sciences

All highly cited Social Sciences papers

Questions about this paper

How do I cite this paper in APA style?

Bisbee, J. H., Clinton, J. D., Dorff, C., Kenkel, B., & Larson, J. (2024). Synthetic Replacements for Human Survey Data? The Perils of Large Language Models. Political Analysis, 32(4), 401–416. https://doi.org/10.1017/pan.2024.5

How many times has this paper been cited?

522 times, according to OpenAlex on 18 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at Political Analysis: sign in and open the full text from this page.

Who wrote this paper?

James H. Bisbee, Joshua D. Clinton, Cassy Dorff, Brenton Kenkel, Jennifer Larson.

What is the DOI of this paper?

10.1017/pan.2024.5, permanently at https://doi.org/10.1017/pan.2024.5. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 18 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.