Measuring Massive Multitask Language Understanding

作者： Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, Jacob Steinhardt (2020)

arXiv： 2009.03300

领域

评估

TLDR（中文）

57 学科 1.4 万道考题，从此"刷 MMLU"成为衡量 LLM 通用能力的事实标准。即使在 2025 年仍是模型卡里第一行的指标；另见后续 MMLU-Pro。

TLDR (English)

57 subjects with 14K exam questions, since then "grinding MMLU" became de facto standard for measuring LLM general capability. Still first-line metric in model cards even in 2025; see also later MMLU-Pro.

出现在这些文章里

• 评估与基准：如何判断模型好坏
• Evaluation and Benchmarks: Judging Model Quality

同被引用

这些论文与本文出现在同一篇文章中

• Evaluating Large Language Models Trained on Code Mark Chen et al. (2021)
• Stop Uploading Test Data in Plain Text: New Protocols for Dataset Release Alon Jacovi et al. (2023)
• Holistic Evaluation of Language Models Percy Liang et al. (2022)
• Large Language Models are not Fair Evaluators Peiyi Wang et al. (2023)

相关论文

同一领域的其他论文

• SWE-bench: Can Language Models Resolve Real-World GitHub Issues? Carlos E. Jimenez et al. (2024)
  评估应用
• Stop Uploading Test Data in Plain Text: New Protocols for Dataset Release Alon Jacovi et al. (2023)
  评估
• Needle in a Haystack — Pressure Testing LLMs Greg Kamradt (2023)
  长上下文评估
• Let's Verify Step by Step Hunter Lightman et al. (2023)
  推理能力评估
• Large Language Models are not Fair Evaluators Peiyi Wang et al. (2023)
  评估
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena Lianmin Zheng et al. (2023)
  评估