GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism

Authors: Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Mia Xu Chen, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, Zhifeng Chen (2019)

arXiv: 1811.06965

Domains

Architecture

TLDR (English)

Splits a model into layer-wise stages executed in a pipeline and uses micro-batches to amortize pipeline bubbles (bubble fraction about (p-1)/(m+p-1)), combined with activation recomputation to control memory. The founding work of pipeline parallelism; later 1F1B, interleaved, and zero-bubble schedules all refine its bubble model.

TLDR（中文）

把模型按层切成多个 stage 流水执行，用 micro-batch 把流水线气泡摊薄（气泡占比约 (p-1)/(m+p-1)），并配合激活重计算控制显存。流水线并行的开山之作，后续 1F1B、交错式调度、零气泡都是在它的气泡模型上做文章。

Appears in These Articles

• 并行策略与性能优化：把 GPU 榨干
• Parallelism & Performance: Squeezing the GPUs

Co-cited Papers

These papers appear in the same articles as this one

• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning Tri Dao (2023)
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness Tri Dao et al. (2022)
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
• Reducing Activation Recomputation in Large Transformer Models Vijay Anand Korthikanti et al. (2023)

Related Papers

Other papers in the same domain

• DeepSeek-V4 Technical Report DeepSeek-AI (2026)
  ArchitectureLong ContextMixture of Experts
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI (2024)
  ArchitectureInference
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
  ArchitectureMixture of Experts
• MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs Ziheng Jiang et al. (2024)
  Architecture
• Mixtral of Experts Albert Q. Jiang et al. (2024)
  Mixture of ExpertsArchitecture
• Zero Bubble Pipeline Parallelism Penghui Qi et al. (2024)
  Architecture