Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

Authors: Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, Jared Kaplan (2022)

arXiv: 2204.05862

Domains

AlignmentSafety

TLDR (English)

Anthropic's early RLHF paper, HH-RLHF dataset since then became "MNIST" of open-source alignment research. Earliest systematic work understanding helpful vs harmless tension.

TLDR（中文）

Anthropic 早期 RLHF 论文，HH-RLHF 数据集自此成为开源对齐研究的"MNIST"。是理解 helpful vs harmless 张力的最早系统化工作。

Appears in These Articles

• 安全与对抗：模型的防护与攻击
• Safety and Adversarial: Protecting and Attacking Models

Co-cited Papers

These papers appear in the same articles as this one

• Extracting Training Data from Large Language Models Nicholas Carlini et al. (2021)
• Not What You Have Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection Kai Greshake et al. (2023)
• Red Teaming Language Models with Language Models Ethan Perez et al. (2022)
• Jailbroken: How Does LLM Safety Training Fail? Alexander Wei et al. (2023)

Related Papers

Other papers in the same domain

• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning DeepSeek-AI (2025)
  ReasoningAlignment
• KTO: Model Alignment as Prospect Theoretic Optimization Kawin Ethayarajh et al. (2024)
  Alignment
• OpenAI o1 System Card OpenAI (2024)
  ReasoningSafety
• A General Theoretical Paradigm to Understand Learning from Human Preferences Mohammad Gheshlaghi Azar et al. (2023)
  Alignment
• LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models Yukang Chen et al. (2023)
  Long ContextAlignment
• QLoRA: Efficient Finetuning of Quantized LLMs Tim Dettmers et al. (2023)
  AlignmentInference