Skip to content

A survey on LLM-as-a-judge

Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, H.-Y. Liu, Saizhuo Wang, Kun Zhang, Zhouchi Lin, Bowen Zhang, Lionel Ming-shuan Ni, Wen Gao, Yuanzhuo Wang, Jian Guo

The Innovation · 2026 · 1,829 citationsOpen access

No retraction on record in OpenAlex.Check Crossref now

Cite this paper

Gu, J., Jiang, X., Shi, Z., Tan, H., Zhai, X., Xu, C., Li, W., Shen, Y., Ma, S., Liu, H.-Y., Wang, S., Zhang, K., Lin, Z., Zhang, B., Ni, L. M.- shuan, Gao, W., Wang, Y., & Guo, J. (2026). A survey on LLM-as-a-judge. The Innovation, 7(6), 101253. https://doi.org/10.1016/j.xinn.2025.101253

In text: (Gu et al., 2026)

Ask a question about this research

One question, free, no account. Anchorcite finds the papers, checks each against Crossref and links every sentence of the answer to the passage it comes from.

Abstract

Accurate and consistent evaluation is crucial for decision-making across numerous fields, yet it remains challenging due to inherent subjectivity, variability, and scale. Large language models (LLMs) have achieved remarkable success, leading to "LLM-as-a-judge," where LLMs serve as evaluators for complex tasks. With their ability to process diverse data types and provide scalable assessments, LLMs present a compelling alternative to traditional expert-driven evaluations. However, ensuring the reliability of LLM-as-a-judge systems remains a significant challenge requiring careful design and standardization. This paper provides a comprehensive survey of LLM-as-a-judge, offering a formal definition and detailed classification while addressing the core question of how to build reliable LLM-as-a-judge systems. We explore strategies to enhance reliability, including improving consistency, mitigating biases, and adapting to diverse scenarios. We propose methodologies for evaluating reliability, supported by a novel benchmark. To advance development and deployment, we discuss practical applications, challenges, and future directions. Our contributions span multiple levels: we establish conceptual boundaries, reorganize fragmented literature into a unified framework, and propose a reliability-oriented benchmark. We articulate a forward-looking research agenda, offering theoretical foundations and practical guidance for constructing reliable and trustworthy LLM-as-a-judge systems.

  1. Dissecting racial bias in an algorithm used to manage the health of populations2019
  2. A Survey on Bias and Fairness in Machine Learning2021
  3. Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)2018
  4. Opinion Paper: “So what if ChatGPT wrote it?” Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy2023
  5. AI4People—An Ethical Framework for a Good AI Society: Opportunities, Risks, Principles, and Recommendations2018

Metadata from OpenAlex (CC0). Citations are generated from the published record.