MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs

作者： Ziheng Jiang, Haibin Lin, Zhijian Liu, Qi Huang, Yu Chen, Xiaodong Lv, Yudong Wang, Ying Xu, Shang Yang, He Zhang, Jiamang Wang, Tiezhen Wang, Wenjie Liu, Bin Cui, Bo Zheng (2024)

arXiv： 2402.15627

领域

架构

TLDR（中文）

字节跳动 12288 卡训练大模型的生产实践全记录：双平面胖树网络设计、把故障当作常态的容错体系（自动诊断、快速回滚、冗余专家）、通信/计算/数据加载全链路 profiling。把 MFU 推到 55% 以上，首次把「万卡训练」的完整工程 SOP 公开成论文。

TLDR (English)

A full production record of training large models on 12288 GPUs at ByteDance: dual-plane fat-tree network design, a fault-tolerance system that treats failures as routine (auto diagnosis, fast rollback, redundant experts), and end-to-end profiling of communication, compute, and data loading. Pushes MFU above 55% and is the first paper to publish a complete engineering SOP for 10k-GPU training.

出现在这些文章里

• 集群网络与通信：GPU 之间的对话
• Cluster Networking & Communication: How GPUs Talk

同被引用

这些论文与本文出现在同一篇文章中

• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
• TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings Norman P. Jouppi et al. (2023)
• Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM Deepak Narayanan et al. (2021)
• GLM-130B: An Open Bilingual Pre-trained Model Aohan Zeng et al. (2022)

相关论文

同一领域的其他论文

• DeepSeek-V4 Technical Report DeepSeek-AI (2026)
  架构长上下文混合专家
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI (2024)
  架构推理
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
  架构混合专家
• Mixtral of Experts Albert Q. Jiang et al. (2024)
  混合专家架构
• Zero Bubble Pipeline Parallelism Penghui Qi et al. (2024)
  架构
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Joshua Ainslie et al. (2023)
  推理架构