Learning to summarize from human feedback

Authors: Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, Paul Christiano (2020)

arXiv: 2009.01325

Domains

Alignment

TLDR (English)

OpenAI's first application of RLHF to large language models (summarization), proving RLHF systematically better than SFT/MLE on human preferences. Direct predecessor to InstructGPT.

TLDR（中文）

OpenAI 把 RLHF 第一次用到大型语言模型（摘要），证明 RLHF 比 SFT/MLE 在人类偏好上系统性更好。是 InstructGPT 的直接前身。

Related Papers

Other papers in the same domain

• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning DeepSeek-AI (2025)
  ReasoningAlignment
• KTO: Model Alignment as Prospect Theoretic Optimization Kawin Ethayarajh et al. (2024)
  Alignment
• A General Theoretical Paradigm to Understand Learning from Human Preferences Mohammad Gheshlaghi Azar et al. (2023)
  Alignment
• LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models Yukang Chen et al. (2023)
  Long ContextAlignment
• QLoRA: Efficient Finetuning of Quantized LLMs Tim Dettmers et al. (2023)
  AlignmentInference
• RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback Harrison Lee et al. (2023)
  Alignment