Measuring Massive Multitask Language Understanding

Authors: Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, Jacob Steinhardt (2020)

arXiv: 2009.03300

Domains

Evaluation

TLDR (English)

57 subjects with 14K exam questions, since then "grinding MMLU" became de facto standard for measuring LLM general capability. Still first-line metric in model cards even in 2025; see also later MMLU-Pro.

TLDR（中文）

57 学科 1.4 万道考题，从此"刷 MMLU"成为衡量 LLM 通用能力的事实标准。即使在 2025 年仍是模型卡里第一行的指标；另见后续 MMLU-Pro。

Appears in These Articles

• 评估与基准：如何判断模型好坏
• Evaluation and Benchmarks: Judging Model Quality

Co-cited Papers

These papers appear in the same articles as this one

• Evaluating Large Language Models Trained on Code Mark Chen et al. (2021)
• Stop Uploading Test Data in Plain Text: New Protocols for Dataset Release Alon Jacovi et al. (2023)
• Holistic Evaluation of Language Models Percy Liang et al. (2022)
• Large Language Models are not Fair Evaluators Peiyi Wang et al. (2023)

Related Papers

Other papers in the same domain

• SWE-bench: Can Language Models Resolve Real-World GitHub Issues? Carlos E. Jimenez et al. (2024)
  EvaluationApplications
• Stop Uploading Test Data in Plain Text: New Protocols for Dataset Release Alon Jacovi et al. (2023)
  Evaluation
• Needle in a Haystack — Pressure Testing LLMs Greg Kamradt (2023)
  Long ContextEvaluation
• Let's Verify Step by Step Hunter Lightman et al. (2023)
  ReasoningEvaluation
• Large Language Models are not Fair Evaluators Peiyi Wang et al. (2023)
  Evaluation
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena Lianmin Zheng et al. (2023)
  Evaluation