Training language models to follow instructions with human feedback

作者： Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, Ryan Lowe (2022)

arXiv： 2203.02155

领域

对齐

TLDR（中文）

InstructGPT 论文，提出了 RLHF 三阶段训练方法（SFT → 奖励模型 → PPO 强化学习）， 让语言模型从"预测下一个词"转变为"按人类意图回答问题"。这是 ChatGPT 的直接前身， 开创了对齐技术的主流路线。

TLDR (English)

InstructGPT introduces the three-stage RLHF pipeline (SFT → reward model → PPO) that transforms language models from "predict next token" to "follow human intent." This is the direct blueprint for ChatGPT and established the dominant approach to LLM alignment.

出现在这些文章里

• 微调与对齐：让模型听指令、守规矩
• Fine-Tuning and Alignment: Making Models Follow Instructions

同被引用

这些论文与本文出现在同一篇文章中

• Constitutional AI: Harmlessness from AI Feedback Yuntao Bai et al. (2022)
• Deep Reinforcement Learning from Human Preferences Paul Christiano et al. (2017)
• LoRA: Low-Rank Adaptation of Large Language Models Edward J. Hu et al. (2021)
• RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback Harrison Lee et al. (2023)

相关论文

同一领域的其他论文

• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning DeepSeek-AI (2025)
  推理能力对齐
• KTO: Model Alignment as Prospect Theoretic Optimization Kawin Ethayarajh et al. (2024)
  对齐
• A General Theoretical Paradigm to Understand Learning from Human Preferences Mohammad Gheshlaghi Azar et al. (2023)
  对齐
• LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models Yukang Chen et al. (2023)
  长上下文对齐
• QLoRA: Efficient Finetuning of Quantized LLMs Tim Dettmers et al. (2023)
  对齐推理
• RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback Harrison Lee et al. (2023)
  对齐