Deep Reinforcement Learning from Human Preferences

Authors: Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, Dario Amodei (2017)

arXiv: 1706.03741

Domains

Alignment

TLDR (English)

The foundational RLHF paper. The authors show that training a reward model from human pairwise preferences, then using it to guide reinforcement learning, enables agents to learn complex behaviors that are difficult to specify with explicit reward functions. This framework was directly adopted by InstructGPT/ChatGPT.

TLDR（中文）

RLHF（人类反馈强化学习）的奠基论文。作者展示了通过人类对比偏好来训练奖励模型， 再用该奖励模型指导强化学习，可以让 agent 学会难以用奖励函数显式描述的复杂行为。 这个框架后来被 InstructGPT/ChatGPT 直接采用。

Appears in These Articles

• 微调与对齐：让模型听指令、守规矩
• Fine-Tuning and Alignment: Making Models Follow Instructions

Co-cited Papers

These papers appear in the same articles as this one

• Constitutional AI: Harmlessness from AI Feedback Yuntao Bai et al. (2022)
• LoRA: Low-Rank Adaptation of Large Language Models Edward J. Hu et al. (2021)
• RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback Harrison Lee et al. (2023)
• Training language models to follow instructions with human feedback Long Ouyang et al. (2022)

Related Papers

Other papers in the same domain

• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning DeepSeek-AI (2025)
  ReasoningAlignment
• KTO: Model Alignment as Prospect Theoretic Optimization Kawin Ethayarajh et al. (2024)
  Alignment
• A General Theoretical Paradigm to Understand Learning from Human Preferences Mohammad Gheshlaghi Azar et al. (2023)
  Alignment
• LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models Yukang Chen et al. (2023)
  Long ContextAlignment
• QLoRA: Efficient Finetuning of Quantized LLMs Tim Dettmers et al. (2023)
  AlignmentInference
• RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback Harrison Lee et al. (2023)
  Alignment