GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism

作者： Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Mia Xu Chen, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, Zhifeng Chen (2019)

arXiv： 1811.06965

领域

架构

TLDR（中文）

把模型按层切成多个 stage 流水执行，用 micro-batch 把流水线气泡摊薄（气泡占比约 (p-1)/(m+p-1)），并配合激活重计算控制显存。流水线并行的开山之作，后续 1F1B、交错式调度、零气泡都是在它的气泡模型上做文章。

TLDR (English)

Splits a model into layer-wise stages executed in a pipeline and uses micro-batches to amortize pipeline bubbles (bubble fraction about (p-1)/(m+p-1)), combined with activation recomputation to control memory. The founding work of pipeline parallelism; later 1F1B, interleaved, and zero-bubble schedules all refine its bubble model.

出现在这些文章里

• 并行策略与性能优化：把 GPU 榨干
• Parallelism & Performance: Squeezing the GPUs

同被引用

这些论文与本文出现在同一篇文章中

• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning Tri Dao (2023)
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness Tri Dao et al. (2022)
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
• Reducing Activation Recomputation in Large Transformer Models Vijay Anand Korthikanti et al. (2023)

相关论文

同一领域的其他论文

• DeepSeek-V4 Technical Report DeepSeek-AI (2026)
  架构长上下文混合专家
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI (2024)
  架构推理
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
  架构混合专家
• MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs Ziheng Jiang et al. (2024)
  架构
• Mixtral of Experts Albert Q. Jiang et al. (2024)
  混合专家架构
• Zero Bubble Pipeline Parallelism Penghui Qi et al. (2024)
  架构