Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

作者： Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, Jared Kaplan (2022)

arXiv： 2204.05862

领域

对齐安全

TLDR（中文）

Anthropic 早期 RLHF 论文，HH-RLHF 数据集自此成为开源对齐研究的"MNIST"。是理解 helpful vs harmless 张力的最早系统化工作。

TLDR (English)

Anthropic's early RLHF paper, HH-RLHF dataset since then became "MNIST" of open-source alignment research. Earliest systematic work understanding helpful vs harmless tension.

出现在这些文章里

• 安全与对抗：模型的防护与攻击
• Safety and Adversarial: Protecting and Attacking Models

同被引用

这些论文与本文出现在同一篇文章中

• Extracting Training Data from Large Language Models Nicholas Carlini et al. (2021)
• Not What You Have Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection Kai Greshake et al. (2023)
• Red Teaming Language Models with Language Models Ethan Perez et al. (2022)
• Jailbroken: How Does LLM Safety Training Fail? Alexander Wei et al. (2023)

相关论文

同一领域的其他论文

• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning DeepSeek-AI (2025)
  推理能力对齐
• KTO: Model Alignment as Prospect Theoretic Optimization Kawin Ethayarajh et al. (2024)
  对齐
• OpenAI o1 System Card OpenAI (2024)
  推理能力安全
• A General Theoretical Paradigm to Understand Learning from Human Preferences Mohammad Gheshlaghi Azar et al. (2023)
  对齐
• LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models Yukang Chen et al. (2023)
  长上下文对齐
• QLoRA: Efficient Finetuning of Quantized LLMs Tim Dettmers et al. (2023)
  对齐推理