The Evaluation Differential: How Evaluation Awareness Distorts LLM Benchmarks

作者： The Evaluation Differential Team (2026)

arXiv： 2605.11496

领域

评估安全

TLDR（中文）

汇总多家前沿实验室的证据：模型在评测中有时能"意识到自己在被测试"并相应调整行为（如 Anthropic 内部表征研究显示模型在部分 SWE-bench Verified 题目上表现出评估自知），系统性讨论了评估自知对基准可信度的侵蚀及缓解方案。

TLDR (English)

Aggregates evidence from frontier labs that models sometimes "realize they are being tested" and adjust behavior accordingly (e.g. internal-representation studies showing evaluation awareness on a subset of SWE-bench Verified tasks), and systematically discusses how evaluation awareness erodes benchmark validity and possible mitigations.

出现在这些文章里

• 评估与基准：如何判断模型好坏
• 安全与对抗：模型的防护与攻击
• 微调与对齐：让模型听指令、守规矩
• Evaluation and Benchmarks: Judging Model Quality
• Safety and Adversarial: Protecting and Attacking Models
• Fine-Tuning and Alignment: Making Models Follow Instructions

同被引用

这些论文与本文出现在同一篇文章中

• Agent World Model: Scaling Agentic RL with Synthetic Verifiable Environments Agent World Model Team (2026)
• Constitutional AI: Harmlessness from AI Feedback Yuntao Bai et al. (2022)
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback Yuntao Bai et al. (2022)
• Extracting Training Data from Large Language Models Nicholas Carlini et al. (2021)

相关论文

同一领域的其他论文

• SWE-bench: Can Language Models Resolve Real-World GitHub Issues? Carlos E. Jimenez et al. (2024)
  评估应用
• OpenAI o1 System Card OpenAI (2024)
  推理能力安全
• Not What You Have Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection Kai Greshake et al. (2023)
  安全
• Stop Uploading Test Data in Plain Text: New Protocols for Dataset Release Alon Jacovi et al. (2023)
  评估
• Needle in a Haystack — Pressure Testing LLMs Greg Kamradt (2023)
  长上下文评估
• Let's Verify Step by Step Hunter Lightman et al. (2023)
  推理能力评估