Muon is Scalable for LLM Training
arXiv: 2502.16982
Domains
TLDR (English)
The Moonlight engineering report: adds weight decay to Muon and rescales updates by matrix shape (0.2*sqrt(max(m,n))) so learning rates transfer from AdamW recipes. Validates Muon at production scale on a 16B MoE trained on 5.7T tokens with roughly 2x the token efficiency of AdamW, proving Muon can move from speedrun tracks to real pretraining.
TLDR（中文）
Moonlight 工程报告：给 Muon 补上权重衰减并按矩阵维度做 RMS 匹配（乘 0.2*sqrt(max(m,n))），使其学习率可与 AdamW 配方复用。在 16B MoE、5.7T tokens 的真实规模上验证 Muon 稳定且达到约 2 倍于 AdamW 的 token 效率，证明 Muon 可以从速通赛道走向生产预训练。
Appears in These Articles
Co-cited Papers
These papers appear in the same articles as this one
Related Papers
Other papers in the same domain