On-Policy Self-Distillation for Reasoning Models

Authors: Self-Distilled Reasoner Team (2026)

arXiv: 2601.18734

Domains

AlignmentReasoning

TLDR (English)

A representative work of the early-2026 "self-distillation" trend in post-training: the model generates data on its own policy distribution and distills it back into itself, removing the need for an external strong teacher and closing a self-improvement loop with verifiable rewards, reducing post-training dependence on distillation pipelines.

TLDR（中文）

2026 开年后训练热点"自蒸馏"的代表工作：模型在自身策略分布上生成数据并蒸馏回自身，不再依赖外部强教师模型，配合可验证奖励实现自我改进闭环，显著降低后训练对蒸馏管线的依赖。

Appears in These Articles

• 微调与对齐：让模型听指令、守规矩
• Fine-Tuning and Alignment: Making Models Follow Instructions

Co-cited Papers

These papers appear in the same articles as this one

• Agent World Model: Scaling Agentic RL with Synthetic Verifiable Environments Agent World Model Team (2026)
• Constitutional AI: Harmlessness from AI Feedback Yuntao Bai et al. (2022)
• Deep Reinforcement Learning from Human Preferences Paul Christiano et al. (2017)
• The Evaluation Differential: How Evaluation Awareness Distorts LLM Benchmarks The Evaluation Differential Team (2026)

Related Papers

Other papers in the same domain

• Agent World Model: Scaling Agentic RL with Synthetic Verifiable Environments Agent World Model Team (2026)
  ApplicationsReasoning
• Adaptive Parallel Reasoning (UC Berkeley Technical Report EECS-2026-213) UC Berkeley (2026)
  ReasoningInference
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning DeepSeek-AI (2025)
  ReasoningAlignment
• KTO: Model Alignment as Prospect Theoretic Optimization Kawin Ethayarajh et al. (2024)
  Alignment
• OpenAI o1 System Card OpenAI (2024)
  ReasoningSafety
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters Charlie Snell et al. (2024)
  ReasoningInference