MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs

Authors: Ziheng Jiang, Haibin Lin, Zhijian Liu, Qi Huang, Yu Chen, Xiaodong Lv, Yudong Wang, Ying Xu, Shang Yang, He Zhang, Jiamang Wang, Tiezhen Wang, Wenjie Liu, Bin Cui, Bo Zheng (2024)

arXiv: 2402.15627

Domains

Architecture

TLDR (English)

A full production record of training large models on 12288 GPUs at ByteDance: dual-plane fat-tree network design, a fault-tolerance system that treats failures as routine (auto diagnosis, fast rollback, redundant experts), and end-to-end profiling of communication, compute, and data loading. Pushes MFU above 55% and is the first paper to publish a complete engineering SOP for 10k-GPU training.

TLDR（中文）

字节跳动 12288 卡训练大模型的生产实践全记录：双平面胖树网络设计、把故障当作常态的容错体系（自动诊断、快速回滚、冗余专家）、通信/计算/数据加载全链路 profiling。把 MFU 推到 55% 以上，首次把「万卡训练」的完整工程 SOP 公开成论文。

Appears in These Articles

• 集群网络与通信：GPU 之间的对话
• Cluster Networking & Communication: How GPUs Talk

Co-cited Papers

These papers appear in the same articles as this one

• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
• TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings Norman P. Jouppi et al. (2023)
• Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM Deepak Narayanan et al. (2021)
• GLM-130B: An Open Bilingual Pre-trained Model Aohan Zeng et al. (2022)

Related Papers

Other papers in the same domain

• DeepSeek-V4 Technical Report DeepSeek-AI (2026)
  ArchitectureLong ContextMixture of Experts
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI (2024)
  ArchitectureInference
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
  ArchitectureMixture of Experts
• Mixtral of Experts Albert Q. Jiang et al. (2024)
  Mixture of ExpertsArchitecture
• Zero Bubble Pipeline Parallelism Penghui Qi et al. (2024)
  Architecture
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Joshua Ainslie et al. (2023)
  InferenceArchitecture