Symbolic Discovery of Optimization Algorithms
arXiv: 2302.06675
Domains
TLDR (English)
Discovers optimizers via AutoML program search, yielding Lion: it compares the sign of momentum with the sign of the gradient and emits constant-magnitude updates without any second moment, saving memory and matching or beating AdamW on vision and language tasks. Showed that optimizers can be searched for, inspiring a wave of sign-based update rules.
TLDR（中文）
用 AutoML 程序搜索自动发现优化器，产出 Lion：只取动量的符号与梯度符号比较生成更新，更新幅值恒定、不需要二阶动量、显存更省，在视觉和语言任务上匹配或超过 AdamW。证明了「优化器可以被搜出来」，启发了后续一系列符号化更新规则。
Appears in These Articles
Co-cited Papers
These papers appear in the same articles as this one
Related Papers
Other papers in the same domain