Adam: A Method for Stochastic Optimization
arXiv: 1412.6980
Domains
TLDR (English)
Introduces Adam, which keeps exponential moving averages of gradient first and second moments and rescales each parameter learning rate by the square root of the second moment. Fast and robust, it dominated deep learning for a decade; its decoupled variant AdamW remains the default optimizer for LLM training.
TLDR（中文）
提出 Adam：为梯度一阶矩和二阶矩各维护一个指数移动平均，用二阶矩的平方根缩放每个参数的学习率。收敛快、对超参稳健，统治深度学习训练近十年；LLM 时代它的解耦变体 AdamW 至今仍是默认优化器。
Appears in These Articles
Co-cited Papers
These papers appear in the same articles as this one
Related Papers
Other papers in the same domain