Kimi K2: Open Agentic Intelligence

作者： Kimi Team (2025)

arXiv： 2507.20534

领域

预训练

TLDR（中文）

Kimi K2 技术报告：1T 总参 / 32B 激活的 MoE，15.5T tokens 预训练。优化系统上把 Muon 推到万亿规模并给出 MuonClip（QK-clip：当注意力 logits 前瞻性地超过阈值 100 时提前裁剪），配合 WSD 学习率调度，全程零 loss spike、零训练回滚；报告同时开源权重与详细 infra 细节。

TLDR (English)

The Kimi K2 technical report: a 1T-total / 32B-active MoE pretrained on 15.5T tokens. On the optimization side it pushes Muon to trillion-parameter scale with MuonClip (QK-clip: anticipatorily clip attention logits once they exceed a threshold of 100) together with a WSD learning-rate schedule, achieving zero loss spikes and zero training rollbacks; weights and detailed infra notes are open-sourced.

出现在这些文章里

• 权重更新与优化器：参数如何动起来
• Weight Update & Optimizers: How Parameters Move

同被引用

这些论文与本文出现在同一篇文章中

• Symbolic Discovery of Optimization Algorithms Xiangning Chen et al. (2023)
• MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies Shengding Hu et al. (2024)
• Muon: An Optimizer for Hidden Layers Keller Jordan et al. (2024)
• Adam: A Method for Stochastic Optimization Diederik P. Kingma et al. (2015)

相关论文

同一领域的其他论文

• Muon is Scalable for LLM Training Moonshot AI (2025)
  预训练
• Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone Microsoft (2024)
  预训练
• Yi: Open Foundation Models by 01.AI 01. AI et al. (2024)
  预训练
• The Llama 3 Herd of Models Meta AI (2024)
  预训练
• MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies Shengding Hu et al. (2024)
  预训练
• Muon: An Optimizer for Hidden Layers Keller Jordan et al. (2024)
  预训练