Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
arXiv: 1804.04235
Domains
TLDR (English)
Factorizes the second-moment matrix of Adam into low-rank terms, cutting optimizer state memory from O(mn) to O(m+n) with little loss in convergence. Used to train T5; a classic memory-conscious alternative to Adam and an early hint that optimizer states must slim down for large models.
TLDR（中文）
把 Adam 的二阶动量矩阵做低秩分解，优化器状态从 O(mn) 降到 O(m+n)，几乎不损失收敛质量。T5 用它训练；是显存紧张场景下替代 Adam 的经典方案，也预示了「优化器状态必须为大模型而瘦身」这条线。
Appears in These Articles
Co-cited Papers
These papers appear in the same articles as this one
Related Papers
Other papers in the same domain