RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback

Authors: Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, Sushant Prakash (2023)

arXiv: 2309.00267

Domains

Alignment

TLDR (English)

Google systematically proves RLAIF can match RLHF on various tasks, providing engineering evidence for "AI feedback replacing human" as scalable alignment solution.

TLDR（中文）

Google 系统性地证明 RLAIF 在多种任务上能匹敌 RLHF，把"AI 反馈替代人工"作为可扩展的对齐方案给出工程证据。

Appears in These Articles

• 微调与对齐：让模型听指令、守规矩
• Fine-Tuning and Alignment: Making Models Follow Instructions

Co-cited Papers

These papers appear in the same articles as this one

• Constitutional AI: Harmlessness from AI Feedback Yuntao Bai et al. (2022)
• Deep Reinforcement Learning from Human Preferences Paul Christiano et al. (2017)
• LoRA: Low-Rank Adaptation of Large Language Models Edward J. Hu et al. (2021)
• Training language models to follow instructions with human feedback Long Ouyang et al. (2022)

Related Papers

Other papers in the same domain

• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning DeepSeek-AI (2025)
  ReasoningAlignment
• KTO: Model Alignment as Prospect Theoretic Optimization Kawin Ethayarajh et al. (2024)
  Alignment
• A General Theoretical Paradigm to Understand Learning from Human Preferences Mohammad Gheshlaghi Azar et al. (2023)
  Alignment
• LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models Yukang Chen et al. (2023)
  Long ContextAlignment
• QLoRA: Efficient Finetuning of Quantized LLMs Tim Dettmers et al. (2023)
  AlignmentInference
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model Rafael Rafailov et al. (2023)
  Alignment