Conference paper · 2024 · Open access

Is Machine Psychology here? On Requirements for Using Human Psychological Tests on Large Language Models

Lea Löhn, Niklas Kiehne, Alexander Ljapunov, Wolf‐Tilo Balke

International Conference on Natural Language Generation, 230–242 · Published 1 January 2024 · DOI 10.18653/v1/2024.inlg-main.19

Open PDFView at the publisherCite this paper

In brief

“Is Machine Psychology here? On Requirements for Using Human Psychological Tests on Large Language Models” is a 2024 conference paper by Lea Löhn and 3 co-authors, published in International Conference on Natural Language Generation. OpenAlex counts 27 citations as of 19 November 2026. It is open access under a CC BY licence: the full text is free to read at International Conference on Natural Language Generation.

Citations
27
Most cited in
2026 (16)
Field-weighted impact
2.34
References
43
Authors
4

Abstract

In an effort to better understand the behavior of large language models (LLM), researchers recently turned to conducting psychological assessments on them.Several studies diagnose various psychological concepts in LLMs, such as psychopathological symptoms, personality traits, and intellectual functioning, aiming to unravel their black-box characteristics.But can we safely assess LLMs with tests that were originally designed for humans?The psychology domain looks back on decades of developing standards of appropriate testing procedures to ensure reliable and valid measures.We argue that analogous standardization processes are required for LLM assessments, given their differential functioning as compared to humans.In this paper, we propose seven requirements necessary for testing LLMs.Based on these, we critically reflect a sample of 25 recent machine psychology studies.Our analysis reveals (1) the lack of appropriate methods to assess test reliability and construct validity, (2) the unknown strength of construct-irrelevant influences, such as the contamination of pre-training corpora with test material, and (3) the pervasive issue of non-reproducibility of many studies.The results underscore the lack of a general methodology for the implementation of psychological assessments of LLMs and the need to redefine psychological constructs specifically for large language models rather than adopting them from human psychology.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Löhn, L., Kiehne, N., Ljapunov, A., & Balke, W. (2024). Is Machine Psychology here? On Requirements for Using Human Psychological Tests on Large Language Models. International Conference on Natural Language Generation, 230–242. https://doi.org/10.18653/v1/2024.inlg-main.19

MLA 9

Löhn, Lea, et al. “Is Machine Psychology here? On Requirements for Using Human Psychological Tests on Large Language Models.” International Conference on Natural Language Generation, 2024, pp. 230–242. https://doi.org/10.18653/v1/2024.inlg-main.19.

Chicago

Löhn, Lea, Niklas Kiehne, Alexander Ljapunov, and Wolf‐Tilo Balke. 2024. “Is Machine Psychology here? On Requirements for Using Human Psychological Tests on Large Language Models.” International Conference on Natural Language Generation: 230–242. https://doi.org/10.18653/v1/2024.inlg-main.19.

Harvard

Löhn, L. et al. (2024) ‘Is Machine Psychology here? On Requirements for Using Human Psychological Tests on Large Language Models’, International Conference on Natural Language Generation, pp. 230–242. doi:10.18653/v1/2024.inlg-main.19.

IEEE

L. Löhn, N. Kiehne, A. Ljapunov, and W. Balke, “Is Machine Psychology here? On Requirements for Using Human Psychological Tests on Large Language Models,” International Conference on Natural Language Generation, pp. 230–242, 2024, doi: 10.18653/v1/2024.inlg-main.19.

BibTeX

@inproceedings{lohn2024machine,
  title     = {{Is Machine Psychology here? On Requirements for Using Human Psychological Tests on Large Language Models}},
  author    = {Löhn, Lea and Kiehne, Niklas and Ljapunov, Alexander and Balke, Wolf‐Tilo},
  booktitle = {International Conference on Natural Language Generation},
  year      = {2024},
  pages     = {230--242},
  doi       = {10.18653/v1/2024.inlg-main.19},
  url       = {https://doi.org/10.18653/v1/2024.inlg-main.19}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Citations by year

YearCitationsShare of the best year
202411
20251010
20261616

Its field-weighted citation impact is 2.34: it has been cited 2.34 times as often as the average work of the same field, type and year (1.00 is the world average).

Topic and keywords

Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI

  • large language models
  • test reliability
  • construct validity
  • psychiatric symptoms
  • personality traits
  • intellectual functioning
  • reproducibility

Work with this paper

Most-cited papers in Computer Science

All highly cited Computer Science papers

Questions about this paper

How do I cite this paper in APA style?

Löhn, L., Kiehne, N., Ljapunov, A., & Balke, W. (2024). Is Machine Psychology here? On Requirements for Using Human Psychological Tests on Large Language Models. International Conference on Natural Language Generation, 230–242. https://doi.org/10.18653/v1/2024.inlg-main.19

How many times has this paper been cited?

27 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.

Is this paper free to read?

Yes. It is open access at International Conference on Natural Language Generation: sign in and open the PDF from this page.

Who wrote this paper?

Lea Löhn, Niklas Kiehne, Alexander Ljapunov, Wolf‐Tilo Balke.

What is the DOI of this paper?

10.18653/v1/2024.inlg-main.19, permanently at https://doi.org/10.18653/v1/2024.inlg-main.19. Sign in and open it from this page.

Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.