Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning

Authors: Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric P. Xing, Joseph E. Gonzalez, Ion Stoica (2022)

arXiv: 2201.12023

Domains

Architecture

TLDR (English)

The first system to jointly automate intra-operator parallelism (tensor-parallel-like) and inter-operator parallelism (pipeline-parallel-like): it hierarchically partitions the enormous strategy space and solves for the optimal split with dynamic programming. Matches or beats hand-tuned expert 3D-parallel configurations at GPT-3 scale, turning manual parallelism tuning into a compiler problem.

TLDR（中文）

第一个自动联合搜索算子内并行（类张量并行）与算子间并行（类流水线并行）的系统：把庞大的并行策略空间做层次化划分，用动态规划求解最优切分。在 GPT-3 规模模型上追平甚至超过专家手工调制的 3D 并行配置，把「人肉调并行」变成了一个编译器问题。

Appears in These Articles

• 并行策略与性能优化：把 GPU 榨干
• Parallelism & Performance: Squeezing the GPUs

Co-cited Papers

These papers appear in the same articles as this one

• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning Tri Dao (2023)
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness Tri Dao et al. (2022)
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
• GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism Yanping Huang et al. (2019)

Related Papers

Other papers in the same domain

• DeepSeek-V4 Technical Report DeepSeek-AI (2026)
  ArchitectureLong ContextMixture of Experts
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI (2024)
  ArchitectureInference
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
  ArchitectureMixture of Experts
• MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs Ziheng Jiang et al. (2024)
  Architecture
• Mixtral of Experts Albert Q. Jiang et al. (2024)
  Mixture of ExpertsArchitecture
• Zero Bubble Pipeline Parallelism Penghui Qi et al. (2024)
  Architecture