Learning to summarize from human feedback

作者： Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, Paul Christiano (2020)

arXiv： 2009.01325

领域

对齐

TLDR（中文）

OpenAI 把 RLHF 第一次用到大型语言模型（摘要），证明 RLHF 比 SFT/MLE 在人类偏好上系统性更好。是 InstructGPT 的直接前身。

TLDR (English)

OpenAI's first application of RLHF to large language models (summarization), proving RLHF systematically better than SFT/MLE on human preferences. Direct predecessor to InstructGPT.

相关论文

同一领域的其他论文

• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning DeepSeek-AI (2025)
  推理能力对齐
• KTO: Model Alignment as Prospect Theoretic Optimization Kawin Ethayarajh et al. (2024)
  对齐
• A General Theoretical Paradigm to Understand Learning from Human Preferences Mohammad Gheshlaghi Azar et al. (2023)
  对齐
• LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models Yukang Chen et al. (2023)
  长上下文对齐
• QLoRA: Efficient Finetuning of Quantized LLMs Tim Dettmers et al. (2023)
  对齐推理
• RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback Harrison Lee et al. (2023)
  对齐