Journal article · 2026 · Open access
A survey on LLM-as-a-judge
The Innovation, 7(6), 101253 · Published 10 January 2026 · DOI 10.1016/j.xinn.2025.101253
In brief
“A survey on LLM-as-a-judge” is a 2026 journal article by Jiawei Gu and 17 co-authors, published in The Innovation. OpenAlex counts 2,136 citations as of 19 November 2026. It is open access under a CC BY-NC-ND licence: the full text is free to read at The Innovation.
- Citations
- 2,136
- Most cited in
- 2026 (1,391)
- Field-weighted impact
- 8329.38
- References
- 229
- Authors
- 18
Abstract
Accurate and consistent evaluation is crucial for decision-making across numerous fields, yet it remains challenging due to inherent subjectivity, variability, and scale. Large language models (LLMs) have achieved remarkable success, leading to "LLM-as-a-judge," where LLMs serve as evaluators for complex tasks. With their ability to process diverse data types and provide scalable assessments, LLMs present a compelling alternative to traditional expert-driven evaluations. However, ensuring the reliability of LLM-as-a-judge systems remains a significant challenge requiring careful design and standardization. This paper provides a comprehensive survey of LLM-as-a-judge, offering a formal definition and detailed classification while addressing the core question of how to build reliable LLM-as-a-judge systems. We explore strategies to enhance reliability, including improving consistency, mitigating biases, and adapting to diverse scenarios. We propose methodologies for evaluating reliability, supported by a novel benchmark. To advance development and deployment, we discuss practical applications, challenges, and future directions. Our contributions span multiple levels: we establish conceptual boundaries, reorganize fragmented literature into a unified framework, and propose a reliability-oriented benchmark. We articulate a forward-looking research agenda, offering theoretical foundations and practical guidance for constructing reliable and trustworthy LLM-as-a-judge systems.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Gu, J., Jiang, X., Shi, Z., Tan, H., Zhai, X., Xu, C., Li, W., Shen, Y., Ma, S., Liu, H.-Y., Wang, S., Zhang, K., Lin, Z., Zhang, B., Ni, L. M.-S., Gao, W., Wang, Y., & Guo, J. (2026). A survey on LLM-as-a-judge. The Innovation, 7(6), 101253. https://doi.org/10.1016/j.xinn.2025.101253
MLA 9
Gu, Jiawei, et al. “A survey on LLM-as-a-judge.” The Innovation, vol. 7, no. 6, 2026, p. 101253. https://doi.org/10.1016/j.xinn.2025.101253.
Chicago
Gu, Jiawei, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, et al. 2026. “A survey on LLM-as-a-judge.” The Innovation 7 (6): 101253. https://doi.org/10.1016/j.xinn.2025.101253.
Harvard
Gu, J. et al. (2026) ‘A survey on LLM-as-a-judge’, The Innovation, 7(6), p. 101253. doi:10.1016/j.xinn.2025.101253.
IEEE
J. Gu et al., “A survey on LLM-as-a-judge,” The Innovation, vol. 7, no. 6, p. 101253, 2026, doi: 10.1016/j.xinn.2025.101253.
BibTeX
@article{gu2026survey,
title = {{A survey on LLM-as-a-judge}},
author = {Gu, Jiawei and Jiang, Xuhui and Shi, Zhichao and Tan, Hexiang and Zhai, Xuehao and Xu, Chengjin and Li, Wei and Shen, Yinghan and Ma, Shengjie and Liu, H.-Y. and Wang, Saizhuo and Zhang, Kun and Lin, Zhouchi and Zhang, Bowen and Ni, Lionel Ming-shuan and Gao, Wen and Wang, Yuanzhuo and Guo, Jian},
journal = {The Innovation},
year = {2026},
volume = {7},
number = {6},
pages = {101253},
doi = {10.1016/j.xinn.2025.101253},
url = {https://doi.org/10.1016/j.xinn.2025.101253}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Citations by year
| Year | Citations | Share of the best year |
|---|---|---|
| 2023 | 1 | |
| 2024 | 12 | |
| 2025 | 732 | |
| 2026 | 1,391 |
Its field-weighted citation impact is 8329.38: it has been cited 8329.38 times as often as the average work of the same field, type and year (1.00 is the world average).
Topic and keywords
Social Sciences › Social Sciences › Law › Artificial Intelligence in Law
- LLM-as-a-judge
- large language models
- automated assessment
- reliability statistics
- consistency
- bias mitigation
- evaluation benchmark
- trustworthy AI
Work with this paper
Most-cited papers in Social Sciences
- Using thematic analysis in psychologyVirginia Braun, Victoria Clarke · Qualitative Research in Psychology · 2006 · 240,222 citations
- Case Study Research: Design and MethodsRobert K. Yin · 1984 · 81,951 citations
- Three Approaches to Qualitative Content AnalysisHsiu-Fang Hsieh, Sarah Elizabeth Shannon · Qualitative Health Research · 2005 · 49,623 citations
- Qualitative Data Analysis: An Expanded SourcebookMatthew B. Miles, A Michael Huberman · Journal of Environmental Psychology · 1994 · 48,311 citations
- Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groupsAllison Tong, Peter Sainsbury, Jonathan C. Craig · International Journal for Quality in Health Care · 2007 · 45,353 citations
Questions about this paper
How do I cite this paper in APA style?
Gu, J., Jiang, X., Shi, Z., Tan, H., Zhai, X., Xu, C., Li, W., Shen, Y., Ma, S., Liu, H.-Y., Wang, S., Zhang, K., Lin, Z., Zhang, B., Ni, L. M.-S., Gao, W., Wang, Y., & Guo, J. (2026). A survey on LLM-as-a-judge. The Innovation, 7(6), 101253. https://doi.org/10.1016/j.xinn.2025.101253
How many times has this paper been cited?
2,136 times, according to OpenAlex on 19 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at The Innovation: sign in and open the PDF from this page.
Who wrote this paper?
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, H.-Y. Liu, Saizhuo Wang, Kun Zhang and 6 more.
What is the DOI of this paper?
10.1016/j.xinn.2025.101253, permanently at https://doi.org/10.1016/j.xinn.2025.101253. Sign in and open it from this page.
Bibliographic data from OpenAlex (CC0), last checked 19 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.