Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM

作者： Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, Matei Zaharia (2021)

arXiv： 2104.04473

领域

架构

TLDR（中文）

Megatron-LM 系统论文：把 TP、流水线 1F1B、数据并行组合成完整配方，给出 3D 并行的放置原则（TP 绑在 NVLink 域内、PP 跨节点、DP 兜底）、交错式 1F1B 调度与流水线气泡分析，并在 3072 张 A100 上以 52.8% 峰值利用率训练 GPT-3 量级模型，定义了大集群训练的效率基线。

TLDR (English)

The Megatron-LM systems paper: composes TP, pipelined 1F1B, and data parallelism into a full recipe, giving placement rules for 3D parallelism (TP inside the NVLink domain, PP across nodes, DP everywhere), the interleaved 1F1B schedule, and pipeline bubble analysis; trains GPT-3-scale models on 3072 A100 GPUs at 52.8% of peak FLOPs, setting the efficiency baseline for large-cluster training.

出现在这些文章里

• 集群网络与通信：GPU 之间的对话
• 并行策略与性能优化：把 GPU 榨干
• Cluster Networking & Communication: How GPUs Talk
• Parallelism & Performance: Squeezing the GPUs

同被引用

这些论文与本文出现在同一篇文章中

• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning Tri Dao (2023)
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness Tri Dao et al. (2022)
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
• GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism Yanping Huang et al. (2019)

相关论文

同一领域的其他论文

• DeepSeek-V4 Technical Report DeepSeek-AI (2026)
  架构长上下文混合专家
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI (2024)
  架构推理
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
  架构混合专家
• MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs Ziheng Jiang et al. (2024)
  架构
• Mixtral of Experts Albert Q. Jiang et al. (2024)
  混合专家架构
• Zero Bubble Pipeline Parallelism Penghui Qi et al. (2024)
  架构