Training language models to follow instructions with human feedback

Authors: Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, Ryan Lowe (2022)

arXiv: 2203.02155

Domains

Alignment

TLDR (English)

InstructGPT introduces the three-stage RLHF pipeline (SFT → reward model → PPO) that transforms language models from "predict next token" to "follow human intent." This is the direct blueprint for ChatGPT and established the dominant approach to LLM alignment.

TLDR（中文）

InstructGPT 论文，提出了 RLHF 三阶段训练方法（SFT → 奖励模型 → PPO 强化学习）， 让语言模型从"预测下一个词"转变为"按人类意图回答问题"。这是 ChatGPT 的直接前身， 开创了对齐技术的主流路线。

Appears in These Articles

• 微调与对齐：让模型听指令、守规矩
• Fine-Tuning and Alignment: Making Models Follow Instructions

Co-cited Papers

These papers appear in the same articles as this one

• Constitutional AI: Harmlessness from AI Feedback Yuntao Bai et al. (2022)
• Deep Reinforcement Learning from Human Preferences Paul Christiano et al. (2017)
• LoRA: Low-Rank Adaptation of Large Language Models Edward J. Hu et al. (2021)
• RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback Harrison Lee et al. (2023)

Related Papers

Other papers in the same domain

• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning DeepSeek-AI (2025)
  ReasoningAlignment
• KTO: Model Alignment as Prospect Theoretic Optimization Kawin Ethayarajh et al. (2024)
  Alignment
• A General Theoretical Paradigm to Understand Learning from Human Preferences Mohammad Gheshlaghi Azar et al. (2023)
  Alignment
• LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models Yukang Chen et al. (2023)
  Long ContextAlignment
• QLoRA: Efficient Finetuning of Quantized LLMs Tim Dettmers et al. (2023)
  AlignmentInference
• RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback Harrison Lee et al. (2023)
  Alignment