Mixed Precision Training
arXiv: 1710.03740
Domains
TLDR (English)
Systematically validates mixed precision training: FP16 storage and compute with FP32 master weights and accumulators plus dynamic loss scaling to prevent gradient underflow. Nearly doubles throughput without quality loss; BF16 later removes the need for loss scaling and becomes the LLM default, and FP8 doubles throughput again.
TLDR（中文）
系统论证了混合精度训练 recipe：FP16 存储与计算、FP32 主权重与累加、动态损失缩放防止梯度下溢。吞吐接近翻倍且精度无损，是大模型训练的效率基石；后来 BF16 免去损失缩放成为 LLM 标配，FP8 又在此基础上把吞吐再翻一倍。
Appears in These Articles
Co-cited Papers
These papers appear in the same articles as this one
Related Papers
Other papers in the same domain