RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback

作者： Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, Sushant Prakash (2023)

arXiv： 2309.00267

领域

对齐

TLDR（中文）

Google 系统性地证明 RLAIF 在多种任务上能匹敌 RLHF，把"AI 反馈替代人工"作为可扩展的对齐方案给出工程证据。

TLDR (English)

Google systematically proves RLAIF can match RLHF on various tasks, providing engineering evidence for "AI feedback replacing human" as scalable alignment solution.

出现在这些文章里

• 微调与对齐：让模型听指令、守规矩
• Fine-Tuning and Alignment: Making Models Follow Instructions

同被引用

这些论文与本文出现在同一篇文章中

• Constitutional AI: Harmlessness from AI Feedback Yuntao Bai et al. (2022)
• Deep Reinforcement Learning from Human Preferences Paul Christiano et al. (2017)
• LoRA: Low-Rank Adaptation of Large Language Models Edward J. Hu et al. (2021)
• Training language models to follow instructions with human feedback Long Ouyang et al. (2022)

相关论文

同一领域的其他论文

• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning DeepSeek-AI (2025)
  推理能力对齐
• KTO: Model Alignment as Prospect Theoretic Optimization Kawin Ethayarajh et al. (2024)
  对齐
• A General Theoretical Paradigm to Understand Learning from Human Preferences Mohammad Gheshlaghi Azar et al. (2023)
  对齐
• LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models Yukang Chen et al. (2023)
  长上下文对齐
• QLoRA: Efficient Finetuning of Quantized LLMs Tim Dettmers et al. (2023)
  对齐推理
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model Rafael Rafailov et al. (2023)
  对齐