A survey on LLM-as-a-judge
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, H.-Y. Liu, Saizhuo Wang, Kun Zhang, Zhouchi Lin, Bowen Zhang, Lionel Ming-shuan Ni, Wen Gao, Yuanzhuo Wang, Jian Guo
The Innovation · 2026 · 1,829 citationsOpen access
No retraction on record in OpenAlex.Check Crossref now
Cite this paper
Gu, J., Jiang, X., Shi, Z., Tan, H., Zhai, X., Xu, C., Li, W., Shen, Y., Ma, S., Liu, H.-Y., Wang, S., Zhang, K., Lin, Z., Zhang, B., Ni, L. M.- shuan, Gao, W., Wang, Y., & Guo, J. (2026). A survey on LLM-as-a-judge. The Innovation, 7(6), 101253. https://doi.org/10.1016/j.xinn.2025.101253
In text: (Gu et al., 2026)
Ask a question about this research
One question, free, no account. Anchorcite finds the papers, checks each against Crossref and links every sentence of the answer to the passage it comes from.
Abstract
Accurate and consistent evaluation is crucial for decision-making across numerous fields, yet it remains challenging due to inherent subjectivity, variability, and scale. Large language models (LLMs) have achieved remarkable success, leading to "LLM-as-a-judge," where LLMs serve as evaluators for complex tasks. With their ability to process diverse data types and provide scalable assessments, LLMs present a compelling alternative to traditional expert-driven evaluations. However, ensuring the reliability of LLM-as-a-judge systems remains a significant challenge requiring careful design and standardization. This paper provides a comprehensive survey of LLM-as-a-judge, offering a formal definition and detailed classification while addressing the core question of how to build reliable LLM-as-a-judge systems. We explore strategies to enhance reliability, including improving consistency, mitigating biases, and adapting to diverse scenarios. We propose methodologies for evaluating reliability, supported by a novel benchmark. To advance development and deployment, we discuss practical applications, challenges, and future directions. Our contributions span multiple levels: we establish conceptual boundaries, reorganize fragmented literature into a unified framework, and propose a reliability-oriented benchmark. We articulate a forward-looking research agenda, offering theoretical foundations and practical guidance for constructing reliable and trustworthy LLM-as-a-judge systems.
Related papers
- Dissecting racial bias in an algorithm used to manage the health of populations2019
- A Survey on Bias and Fairness in Machine Learning2021
- Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)2018
- Opinion Paper: “So what if ChatGPT wrote it?” Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy2023
- AI4People—An Ethical Framework for a Good AI Society: Opportunities, Risks, Principles, and Recommendations2018
Metadata from OpenAlex (CC0). Citations are generated from the published record.