ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

Authors: Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong He (2020)

arXiv: 1910.02054

Domains

Architecture

TLDR (English)

Stops replicating optimizer state across every data-parallel copy: ZeRO-1/2/3 shard optimizer states, gradients, and parameters respectively across the DP group, trading communication for memory so per-GPU memory drops roughly linearly with GPU count. The full sharding idea of ZeRO-3 became PyTorch FSDP, making models too big for one machine trainable.

TLDR（中文）

不再让每个数据并行副本都完整持有优化器状态：ZeRO-1/2/3 分别把优化器状态、梯度、参数切分到 DP 组内各卡，用通信换显存，使单卡显存随卡数近似线性下降。ZeRO-3 的完全分片思想就是后来 PyTorch FSDP 的源头，让单机装不下的模型训练成为可能。

Appears in These Articles

• 并行策略与性能优化：把 GPU 榨干
• Parallelism & Performance: Squeezing the GPUs

Co-cited Papers

These papers appear in the same articles as this one

• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning Tri Dao (2023)
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness Tri Dao et al. (2022)
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
• GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism Yanping Huang et al. (2019)

Related Papers

Other papers in the same domain

• DeepSeek-V4 Technical Report DeepSeek-AI (2026)
  ArchitectureLong ContextMixture of Experts
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI (2024)
  ArchitectureInference
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
  ArchitectureMixture of Experts
• MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs Ziheng Jiang et al. (2024)
  Architecture
• Mixtral of Experts Albert Q. Jiang et al. (2024)
  Mixture of ExpertsArchitecture
• Zero Bubble Pipeline Parallelism Penghui Qi et al. (2024)
  Architecture