Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

作者： Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, Ion Stoica (2023)

arXiv： 2306.05685

领域

评估

TLDR（中文）

提出 GPT-4-as-judge + 人类偏好众包 (Chatbot Arena) 评测对话能力。MT-Bench 和 Arena ELO 至今是社区比较模型"对话能力"的事实双标准。

TLDR (English)

Proposes GPT-4-as-judge + human preference crowdsourcing (Chatbot Arena) for evaluating dialogue capability. MT-Bench and Arena ELO remain community's de facto dual standards for comparing model "dialogue capability" today.

出现在这些文章里

• 评估与基准：如何判断模型好坏
• Evaluation and Benchmarks: Judging Model Quality

同被引用

这些论文与本文出现在同一篇文章中

• Evaluating Large Language Models Trained on Code Mark Chen et al. (2021)
• Measuring Massive Multitask Language Understanding Dan Hendrycks et al. (2020)
• Stop Uploading Test Data in Plain Text: New Protocols for Dataset Release Alon Jacovi et al. (2023)
• Holistic Evaluation of Language Models Percy Liang et al. (2022)

相关论文

同一领域的其他论文

• SWE-bench: Can Language Models Resolve Real-World GitHub Issues? Carlos E. Jimenez et al. (2024)
  评估应用
• Stop Uploading Test Data in Plain Text: New Protocols for Dataset Release Alon Jacovi et al. (2023)
  评估
• Needle in a Haystack — Pressure Testing LLMs Greg Kamradt (2023)
  长上下文评估
• Let's Verify Step by Step Hunter Lightman et al. (2023)
  推理能力评估
• Large Language Models are not Fair Evaluators Peiyi Wang et al. (2023)
  评估
• Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned Deep Ganguli et al. (2022)
  安全评估