Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM

Authors: Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, Matei Zaharia (2021)

arXiv: 2104.04473

Domains

Architecture

TLDR (English)

The Megatron-LM systems paper: composes TP, pipelined 1F1B, and data parallelism into a full recipe, giving placement rules for 3D parallelism (TP inside the NVLink domain, PP across nodes, DP everywhere), the interleaved 1F1B schedule, and pipeline bubble analysis; trains GPT-3-scale models on 3072 A100 GPUs at 52.8% of peak FLOPs, setting the efficiency baseline for large-cluster training.

TLDR（中文）

Megatron-LM 系统论文：把 TP、流水线 1F1B、数据并行组合成完整配方，给出 3D 并行的放置原则（TP 绑在 NVLink 域内、PP 跨节点、DP 兜底）、交错式 1F1B 调度与流水线气泡分析，并在 3072 张 A100 上以 52.8% 峰值利用率训练 GPT-3 量级模型，定义了大集群训练的效率基线。

Appears in These Articles

• 集群网络与通信：GPU 之间的对话
• 并行策略与性能优化：把 GPU 榨干
• Cluster Networking & Communication: How GPUs Talk
• Parallelism & Performance: Squeezing the GPUs

Co-cited Papers

These papers appear in the same articles as this one

• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning Tri Dao (2023)
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness Tri Dao et al. (2022)
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
• GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism Yanping Huang et al. (2019)

Related Papers

Other papers in the same domain

• DeepSeek-V4 Technical Report DeepSeek-AI (2026)
  ArchitectureLong ContextMixture of Experts
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI (2024)
  ArchitectureInference
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
  ArchitectureMixture of Experts
• MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs Ziheng Jiang et al. (2024)
  Architecture
• Mixtral of Experts Albert Q. Jiang et al. (2024)
  Mixture of ExpertsArchitecture
• Zero Bubble Pipeline Parallelism Penghui Qi et al. (2024)
  Architecture