Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning

作者： Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric P. Xing, Joseph E. Gonzalez, Ion Stoica (2022)

arXiv： 2201.12023

领域

架构

TLDR（中文）

第一个自动联合搜索算子内并行（类张量并行）与算子间并行（类流水线并行）的系统：把庞大的并行策略空间做层次化划分，用动态规划求解最优切分。在 GPT-3 规模模型上追平甚至超过专家手工调制的 3D 并行配置，把「人肉调并行」变成了一个编译器问题。

TLDR (English)

The first system to jointly automate intra-operator parallelism (tensor-parallel-like) and inter-operator parallelism (pipeline-parallel-like): it hierarchically partitions the enormous strategy space and solves for the optimal split with dynamic programming. Matches or beats hand-tuned expert 3D-parallel configurations at GPT-3 scale, turning manual parallelism tuning into a compiler problem.

出现在这些文章里

• 并行策略与性能优化：把 GPU 榨干
• Parallelism & Performance: Squeezing the GPUs

同被引用

这些论文与本文出现在同一篇文章中

• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning Tri Dao (2023)
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness Tri Dao et al. (2022)
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
• GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism Yanping Huang et al. (2019)

相关论文

同一领域的其他论文

• DeepSeek-V4 Technical Report DeepSeek-AI (2026)
  架构长上下文混合专家
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI (2024)
  架构推理
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
  架构混合专家
• MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs Ziheng Jiang et al. (2024)
  架构
• Mixtral of Experts Albert Q. Jiang et al. (2024)
  混合专家架构
• Zero Bubble Pipeline Parallelism Penghui Qi et al. (2024)
  架构