Evaluating Large Language Models Trained on Code

Authors: Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, Wojciech Zaremba (2021)

arXiv: 2107.03374

Domains

EvaluationApplications

TLDR (English)

Proposes Codex model + HumanEval benchmark (164 programming problems). HumanEval remains "ECG metric" for coding models today; this paper is also root of GitHub Copilot.

TLDR（中文）

提出 Codex 模型 + HumanEval 基准（164 道编程题）。HumanEval 至今是 coding 模型的"心电图指标"；这篇论文也是 GitHub Copilot 的根。

Appears in These Articles

• 代码生成：模型如何写程序
• 评估与基准：如何判断模型好坏
• Code Generation: How Models Write Programs
• Evaluation and Benchmarks: Judging Model Quality

Co-cited Papers

These papers appear in the same articles as this one

• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning DeepSeek-AI (2025)
• Measuring Massive Multitask Language Understanding Dan Hendrycks et al. (2020)
• Stop Uploading Test Data in Plain Text: New Protocols for Dataset Release Alon Jacovi et al. (2023)
• SWE-bench: Can Language Models Resolve Real-World GitHub Issues? Carlos E. Jimenez et al. (2024)

Related Papers

Other papers in the same domain

• Model Context Protocol (MCP) Anthropic (2024)
  Applications
• SWE-bench: Can Language Models Resolve Real-World GitHub Issues? Carlos E. Jimenez et al. (2024)
  EvaluationApplications
• SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering John Yang et al. (2024)
  Applications
• Stop Uploading Test Data in Plain Text: New Protocols for Dataset Release Alon Jacovi et al. (2023)
  Evaluation
• Needle in a Haystack — Pressure Testing LLMs Greg Kamradt (2023)
  Long ContextEvaluation
• Let's Verify Step by Step Hunter Lightman et al. (2023)
  ReasoningEvaluation