Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

作者： Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, Bryan Catanzaro (2019)

arXiv： 1909.08053

领域

架构

TLDR（中文）

提出张量并行（TP）：把 self-attention 和 MLP 的矩阵乘按列/行切分到多张 GPU，层内并行、层间保持全精度激活，单卡只需承载 1/N 的权重与梯度。显式、可控、通信模式规则，与现代流水线、数据并行组合成 3D 并行范式，是大模型并行训练的奠基工作。

TLDR (English)

Introduces tensor parallelism (TP): split the matrix multiplications of self-attention and MLPs column-wise and row-wise across GPUs, parallelizing within layers while keeping inter-layer activations full precision, so each card holds 1/N of weights and gradients. Explicit, controllable, and regular in its communication pattern, it composes with pipeline and data parallelism into the 3D-parallel paradigm and founded large-model parallel training.

出现在这些文章里

• 并行策略与性能优化：把 GPU 榨干
• Parallelism & Performance: Squeezing the GPUs

同被引用

这些论文与本文出现在同一篇文章中

• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning Tri Dao (2023)
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness Tri Dao et al. (2022)
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
• GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism Yanping Huang et al. (2019)

相关论文

同一领域的其他论文

• DeepSeek-V4 Technical Report DeepSeek-AI (2026)
  架构长上下文混合专家
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI (2024)
  架构推理
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
  架构混合专家
• MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs Ziheng Jiang et al. (2024)
  架构
• Mixtral of Experts Albert Q. Jiang et al. (2024)
  混合专家架构
• Zero Bubble Pipeline Parallelism Penghui Qi et al. (2024)
  架构