Zero Bubble Pipeline Parallelism

Authors: Penghui Qi, Xingcheng Zhang, Zewei Yao, Zhewei Yao, Shang Yang, Meng Chen, Jiaxiang Wu, Xin Luo, Dongqi Cai, Zhipeng Zhang, Xiaokang Wang, Jianjian Sun, Zhan Ma, Xiaoyun Li, Jie Ren, Yong Guo, Mingyi Li, Sunghun Kim, Zhengzhong Liu, Linji Yang, Minjia Zhang, Xinle Ma, Zhiqiang Shen, Mingxuan Wang, Qipeng Guo, Guangting Mai, Dahua Lin, Chao Zhang (2024)

arXiv: 2401.10241

Domains

Architecture

TLDR (English)

Splits backward into activation-gradient (B) and weight-gradient (W) computations and reschedules them so W fills pipeline bubbles, achieving near-zero-bubble pipeline parallelism with better throughput than 1F1B under the same memory budget, proving there is still structural headroom in pipeline schedules.

TLDR（中文）

把 backward 拆成「计算激活梯度（B）」和「计算参数梯度（W）」两步重新调度，让 W 的计算去填流水线气泡，实现接近零气泡的流水线并行，且在同等显存预算下吞吐优于 1F1B。证明流水线调度仍有结构性红利可挖。

Appears in These Articles

• 并行策略与性能优化：把 GPU 榨干
• Parallelism & Performance: Squeezing the GPUs

Co-cited Papers

These papers appear in the same articles as this one

• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning Tri Dao (2023)
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness Tri Dao et al. (2022)
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
• GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism Yanping Huang et al. (2019)

Related Papers

Other papers in the same domain

• DeepSeek-V4 Technical Report DeepSeek-AI (2026)
  ArchitectureLong ContextMixture of Experts
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI (2024)
  ArchitectureInference
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
  ArchitectureMixture of Experts
• MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs Ziheng Jiang et al. (2024)
  Architecture
• Mixtral of Experts Albert Q. Jiang et al. (2024)
  Mixture of ExpertsArchitecture
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Joshua Ainslie et al. (2023)
  InferenceArchitecture