A General Theoretical Paradigm to Understand Learning from Human Preferences

作者： Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, Rémi Munos (2023)

arXiv： 2310.12036

领域

对齐

TLDR（中文）

用 Ψ-PO 框架统一 RLHF/DPO，并指出 DPO 在 BT 假设下会过拟合；提出 IPO 损失更稳健。是理解"为什么 DPO 不总是 work"的理论必读；另见 KTO、SimPO。

TLDR (English)

Unifies RLHF/DPO with Ψ-PO framework, points out DPO overfits under BT assumption; proposes more robust IPO loss. Theoretical must-read for understanding "why DPO doesn't always work"; see also KTO, SimPO.

相关论文

同一领域的其他论文

• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning DeepSeek-AI (2025)
  推理能力对齐
• KTO: Model Alignment as Prospect Theoretic Optimization Kawin Ethayarajh et al. (2024)
  对齐
• LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models Yukang Chen et al. (2023)
  长上下文对齐
• QLoRA: Efficient Finetuning of Quantized LLMs Tim Dettmers et al. (2023)
  对齐推理
• RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback Harrison Lee et al. (2023)
  对齐
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model Rafael Rafailov et al. (2023)
  对齐