Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

Authors: Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, Ion Stoica (2023)

arXiv: 2306.05685

Domains

Evaluation

TLDR (English)

Proposes GPT-4-as-judge + human preference crowdsourcing (Chatbot Arena) for evaluating dialogue capability. MT-Bench and Arena ELO remain community's de facto dual standards for comparing model "dialogue capability" today.

TLDR（中文）

提出 GPT-4-as-judge + 人类偏好众包 (Chatbot Arena) 评测对话能力。MT-Bench 和 Arena ELO 至今是社区比较模型"对话能力"的事实双标准。

Appears in These Articles

• 评估与基准：如何判断模型好坏
• Evaluation and Benchmarks: Judging Model Quality

Co-cited Papers

These papers appear in the same articles as this one

• Evaluating Large Language Models Trained on Code Mark Chen et al. (2021)
• Measuring Massive Multitask Language Understanding Dan Hendrycks et al. (2020)
• Stop Uploading Test Data in Plain Text: New Protocols for Dataset Release Alon Jacovi et al. (2023)
• Holistic Evaluation of Language Models Percy Liang et al. (2022)

Related Papers

Other papers in the same domain

• SWE-bench: Can Language Models Resolve Real-World GitHub Issues? Carlos E. Jimenez et al. (2024)
  EvaluationApplications
• Stop Uploading Test Data in Plain Text: New Protocols for Dataset Release Alon Jacovi et al. (2023)
  Evaluation
• Needle in a Haystack — Pressure Testing LLMs Greg Kamradt (2023)
  Long ContextEvaluation
• Let's Verify Step by Step Hunter Lightman et al. (2023)
  ReasoningEvaluation
• Large Language Models are not Fair Evaluators Peiyi Wang et al. (2023)
  Evaluation
• Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned Deep Ganguli et al. (2022)
  SafetyEvaluation