Decoupled Weight Decay Regularization
arXiv: 1711.05101
Domains
TLDR (English)
Shows that L2 regularization inside Adam gets rescaled by the adaptive denominator, so effective weight decay drifts with gradient magnitude, and proposes AdamW with decoupled weight decay. Nearly all LLM pretraining today uses AdamW with weight decay around 0.1.
TLDR（中文）
指出 Adam 里传统的 L2 正则会被自适应分母重新缩放、导致权重衰减强度随梯度尺度漂移，提出把权重衰减从梯度中解耦出来的 AdamW。如今几乎所有 LLM 预训练都使用 AdamW 加约 0.1 权重衰减的配置。
Appears in These Articles
Co-cited Papers
These papers appear in the same articles as this one
Related Papers
Other papers in the same domain