Zero Bubble Pipeline Parallelism

作者： Penghui Qi, Xingcheng Zhang, Zewei Yao, Zhewei Yao, Shang Yang, Meng Chen, Jiaxiang Wu, Xin Luo, Dongqi Cai, Zhipeng Zhang, Xiaokang Wang, Jianjian Sun, Zhan Ma, Xiaoyun Li, Jie Ren, Yong Guo, Mingyi Li, Sunghun Kim, Zhengzhong Liu, Linji Yang, Minjia Zhang, Xinle Ma, Zhiqiang Shen, Mingxuan Wang, Qipeng Guo, Guangting Mai, Dahua Lin, Chao Zhang (2024)

arXiv： 2401.10241

领域

架构

TLDR（中文）

把 backward 拆成「计算激活梯度（B）」和「计算参数梯度（W）」两步重新调度，让 W 的计算去填流水线气泡，实现接近零气泡的流水线并行，且在同等显存预算下吞吐优于 1F1B。证明流水线调度仍有结构性红利可挖。

TLDR (English)

Splits backward into activation-gradient (B) and weight-gradient (W) computations and reschedules them so W fills pipeline bubbles, achieving near-zero-bubble pipeline parallelism with better throughput than 1F1B under the same memory budget, proving there is still structural headroom in pipeline schedules.

出现在这些文章里

• 并行策略与性能优化：把 GPU 榨干
• Parallelism & Performance: Squeezing the GPUs

同被引用

这些论文与本文出现在同一篇文章中

• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning Tri Dao (2023)
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness Tri Dao et al. (2022)
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
• GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism Yanping Huang et al. (2019)

相关论文

同一领域的其他论文

• DeepSeek-V4 Technical Report DeepSeek-AI (2026)
  架构长上下文混合专家
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI (2024)
  架构推理
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
  架构混合专家
• MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs Ziheng Jiang et al. (2024)
  架构
• Mixtral of Experts Albert Q. Jiang et al. (2024)
  混合专家架构
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Joshua Ainslie et al. (2023)
  推理架构