Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
arXiv: 1909.08053
Domains
TLDR (English)
Introduces tensor parallelism (TP): split the matrix multiplications of self-attention and MLPs column-wise and row-wise across GPUs, parallelizing within layers while keeping inter-layer activations full precision, so each card holds 1/N of weights and gradients. Explicit, controllable, and regular in its communication pattern, it composes with pipeline and data parallelism into the 3D-parallel paradigm and founded large-model parallel training.
TLDR（中文）
提出张量并行（TP）：把 self-attention 和 MLP 的矩阵乘按列/行切分到多张 GPU，层内并行、层间保持全精度激活，单卡只需承载 1/N 的权重与梯度。显式、可控、通信模式规则，与现代流水线、数据并行组合成 3D 并行范式，是大模型并行训练的奠基工作。
Appears in These Articles
Co-cited Papers
These papers appear in the same articles as this one
Related Papers
Other papers in the same domain