SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

作者： Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, Karthik Narasimhan (2024)

arXiv： 2310.06770

领域

评估应用

TLDR（中文）

用 12 个真实 Python 仓库 2294 个 issue 评测代码模型"端到端解决 bug"的能力。一夜成为 coding agent 行业标准评测，几乎每篇 coding agent 论文都报 SWE-bench 分数。

TLDR (English)

Uses 12 real Python repos with 2294 issues to evaluate code models' "end-to-end bug solving" capability. Overnight became coding agent industry standard benchmark; almost every coding agent paper reports SWE-bench scores.

出现在这些文章里

• 代码生成：模型如何写程序
• Code Generation: How Models Write Programs

同被引用

这些论文与本文出现在同一篇文章中

• Evaluating Large Language Models Trained on Code Mark Chen et al. (2021)
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning DeepSeek-AI (2025)
• OpenAI o1 System Card OpenAI (2024)
• SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering John Yang et al. (2024)

相关论文

同一领域的其他论文

• Model Context Protocol (MCP) Anthropic (2024)
  应用
• SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering John Yang et al. (2024)
  应用
• Stop Uploading Test Data in Plain Text: New Protocols for Dataset Release Alon Jacovi et al. (2023)
  评估
• Needle in a Haystack — Pressure Testing LLMs Greg Kamradt (2023)
  长上下文评估
• Let's Verify Step by Step Hunter Lightman et al. (2023)
  推理能力评估
• Toolformer: Language Models Can Teach Themselves to Use Tools Timo Schick et al. (2023)
  应用