KTO: Model Alignment as Prospect Theoretic Optimization

Authors: Kawin Ethayarajh, Yejin Choi, Swabha Swayamdipta (2024)

arXiv: 2402.01306

Domains

Alignment

TLDR (English)

Proposes KTO (Kahneman-Tversky Optimization), which aligns models using only binary feedback (good/bad) without requiring paired preference data like DPO. Introduces prospect theory into alignment optimization, proving that knowing whether a single output is desirable is sufficient to learn human preferences.

TLDR（中文）

提出 KTO（Kahneman-Tversky Optimization），仅需二元反馈（好/坏）即可对齐模型，无需像 DPO 那样需要成对偏好数据。将前景理论引入对齐优化，证明单条输出是否被喜欢的信号足以学习人类偏好。

Related Papers

Other papers in the same domain

• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning DeepSeek-AI (2025)
  ReasoningAlignment
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model Rafael Rafailov et al. (2023)
  Alignment
• Alpaca: A Strong, Replicable Instruction-Following Model Rohan Taori et al. (2023)
  Alignment
• Constitutional AI: Harmlessness from AI Feedback Yuntao Bai et al. (2022)
  AlignmentSafety
• Training language models to follow instructions with human feedback Long Ouyang et al. (2022)
  Alignment
• LoRA: Low-Rank Adaptation of Large Language Models Edward J. Hu et al. (2021)
  Alignment