Journal article · 2024 · Open access
Synthetic Replacements for Human Survey Data? The Perils of Large Language Models
Political Analysis, 32(4), 401–416 · Published 17 May 2024 · DOI 10.1017/pan.2024.5
In brief
“Synthetic Replacements for Human Survey Data? The Perils of Large Language Models” is a 2024 journal article by James H. Bisbee and 4 co-authors, published in Political Analysis. OpenAlex counts 522 citations as of 18 November 2026, which puts it in the top 1% of works published in 2024 by citations. It is open access under a CC BY licence: the full text is free to read at Political Analysis.
- Citations
- 522
- Most cited in
- 2026 (336)
- Field-weighted impact
- 176.29
- References
- 54
- Authors
- 5
Abstract
Large language models (LLMs) offer new research possibilities for social scientists, but their potential as “synthetic data” is still largely unknown. In this paper, we investigate how accurately the popular LLM ChatGPT can recover public opinion, prompting the LLM to adopt different “personas” and then provide feeling thermometer scores for 11 sociopolitical groups. The average scores generated by ChatGPT correspond closely to the averages in our baseline survey, the 2016–2020 American National Election Study (ANES). Nevertheless, sampling by ChatGPT is not reliable for statistical inference: there is less variation in responses than in the real surveys, and regression coefficients often differ significantly from equivalent estimates obtained using ANES data. We also document how the distribution of synthetic responses varies with minor changes in prompt wording, and we show how the same prompt yields significantly different results over a 3-month period. Altogether, our findings raise serious concerns about the quality, reliability, and reproducibility of synthetic survey data generated by LLMs.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Bisbee, J. H., Clinton, J. D., Dorff, C., Kenkel, B., & Larson, J. (2024). Synthetic Replacements for Human Survey Data? The Perils of Large Language Models. Political Analysis, 32(4), 401–416. https://doi.org/10.1017/pan.2024.5
MLA 9
Bisbee, James H., et al. “Synthetic Replacements for Human Survey Data? The Perils of Large Language Models.” Political Analysis, vol. 32, no. 4, 2024, pp. 401–416. https://doi.org/10.1017/pan.2024.5.
Chicago
Bisbee, James H., Joshua D. Clinton, Cassy Dorff, Brenton Kenkel, and Jennifer Larson. 2024. “Synthetic Replacements for Human Survey Data? The Perils of Large Language Models.” Political Analysis 32 (4): 401–416. https://doi.org/10.1017/pan.2024.5.
Harvard
Bisbee, J. H. et al. (2024) ‘Synthetic Replacements for Human Survey Data? The Perils of Large Language Models’, Political Analysis, 32(4), pp. 401–416. doi:10.1017/pan.2024.5.
IEEE
J. H. Bisbee, J. D. Clinton, C. Dorff, B. Kenkel, and J. Larson, “Synthetic Replacements for Human Survey Data? The Perils of Large Language Models,” Political Analysis, vol. 32, no. 4, pp. 401–416, 2024, doi: 10.1017/pan.2024.5.
BibTeX
@article{bisbee2024synthetic,
title = {{Synthetic Replacements for Human Survey Data? The Perils of Large Language Models}},
author = {Bisbee, James H. and Clinton, Joshua D. and Dorff, Cassy and Kenkel, Brenton and Larson, Jennifer},
journal = {Political Analysis},
year = {2024},
volume = {32},
number = {4},
pages = {401--416},
doi = {10.1017/pan.2024.5},
url = {https://doi.org/10.1017/pan.2024.5}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Citations by year
| Year | Citations | Share of the best year |
|---|---|---|
| 2023 | 11 | |
| 2024 | 33 | |
| 2025 | 142 | |
| 2026 | 336 |
Its field-weighted citation impact is 176.29: it has been cited 176.29 times as often as the average work of the same field, type and year (1.00 is the world average).
Topic and keywords
Social Sciences › Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- large language models
- synthetic survey data
- ChatGPT
- public opinion
- feeling thermometers
- sociopolitical attitudes
- American National Election Studies
- sampling reliability
Work with this paper
Most-cited papers in Social Sciences
- Using thematic analysis in psychologyVirginia Braun, Victoria Clarke · Qualitative Research in Psychology · 2006 · 240,222 citations
- Case Study Research: Design and MethodsRobert K. Yin · 1984 · 81,951 citations
- Three Approaches to Qualitative Content AnalysisHsiu-Fang Hsieh, Sarah Elizabeth Shannon · Qualitative Health Research · 2005 · 49,623 citations
- Qualitative Data Analysis: An Expanded SourcebookMatthew B. Miles, A Michael Huberman · Journal of Environmental Psychology · 1994 · 48,311 citations
- Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groupsAllison Tong, Peter Sainsbury, Jonathan C. Craig · International Journal for Quality in Health Care · 2007 · 45,353 citations
Questions about this paper
How do I cite this paper in APA style?
Bisbee, J. H., Clinton, J. D., Dorff, C., Kenkel, B., & Larson, J. (2024). Synthetic Replacements for Human Survey Data? The Perils of Large Language Models. Political Analysis, 32(4), 401–416. https://doi.org/10.1017/pan.2024.5
How many times has this paper been cited?
522 times, according to OpenAlex on 18 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at Political Analysis: sign in and open the full text from this page.
Who wrote this paper?
James H. Bisbee, Joshua D. Clinton, Cassy Dorff, Brenton Kenkel, Jennifer Larson.
What is the DOI of this paper?
10.1017/pan.2024.5, permanently at https://doi.org/10.1017/pan.2024.5. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 18 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.