Muon: An Optimizer for Hidden Layers
arXiv: 2410.05258
Domains
TLDR (English)
Introduces Muon: orthogonalize the momentum buffer (Newton-Schulz iteration approximating UV^T) before applying it to hidden-layer matrices, interpretable as steepest descent under the spectral norm so every layer update has consistent spectral magnitude. Over 1.35x faster than a tuned AdamW on the NanoGPT speedrun at the cost of one momentum orthogonalization, making it the most watched post-Adam optimizer.
TLDR（中文）
提出 Muon：对动量做正交化（Newton-Schulz 迭代近似 UV^T）后再更新隐藏层矩阵，可理解为谱范数意义下的最速下降，让每层的更新「谱」大小一致。在 NanoGPT speedrun 上比调优后的 AdamW 快约 1.35 倍以上，成本仅为一次动量正交化，成为后 Adam 时代最受关注的优化器。
Appears in These Articles
Co-cited Papers
These papers appear in the same articles as this one
Related Papers
Other papers in the same domain