ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

作者： Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong He (2020)

arXiv： 1910.02054

领域

架构

TLDR（中文）

不再让每个数据并行副本都完整持有优化器状态：ZeRO-1/2/3 分别把优化器状态、梯度、参数切分到 DP 组内各卡，用通信换显存，使单卡显存随卡数近似线性下降。ZeRO-3 的完全分片思想就是后来 PyTorch FSDP 的源头，让单机装不下的模型训练成为可能。

TLDR (English)

Stops replicating optimizer state across every data-parallel copy: ZeRO-1/2/3 shard optimizer states, gradients, and parameters respectively across the DP group, trading communication for memory so per-GPU memory drops roughly linearly with GPU count. The full sharding idea of ZeRO-3 became PyTorch FSDP, making models too big for one machine trainable.

出现在这些文章里

• 并行策略与性能优化：把 GPU 榨干
• Parallelism & Performance: Squeezing the GPUs

同被引用

这些论文与本文出现在同一篇文章中

• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning Tri Dao (2023)
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness Tri Dao et al. (2022)
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
• GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism Yanping Huang et al. (2019)

相关论文

同一领域的其他论文

• DeepSeek-V4 Technical Report DeepSeek-AI (2026)
  架构长上下文混合专家
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI (2024)
  架构推理
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
  架构混合专家
• MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs Ziheng Jiang et al. (2024)
  架构
• Mixtral of Experts Albert Q. Jiang et al. (2024)
  混合专家架构
• Zero Bubble Pipeline Parallelism Penghui Qi et al. (2024)
  架构