Reducing Activation Recomputation in Large Transformer Models

Authors: Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Matei Zaharia, Jonathan Ragan-Kelley, Amar Phanishayee (2023)

arXiv: 2205.05198

Domains

Architecture

TLDR (English)

The Megatron-Core memory engineering paper: classifies activations into must-store and recomputable, proposing selective recomputation (recompute only the attention core, keep MLP activations) plus sequence parallelism to capture most memory savings with far fewer extra FLOPs than full recomputation; it also derives the per-layer activation formulas for transformers, becoming the reference for memory planning in large-model training.

TLDR（中文）

Megatron-Core 的显存工程论文：把激活值分成「必存」与「可重算」两类，提出选择性重计算（只重算 attention 核心、MLP 激活整段存）加序列并行，用远小于全量重计算的额外 FLOPs 拿到大部分显存收益；还系统分析了 transformer 各层激活的构成公式，是大模型训练显存规划的参考标准。

Appears in These Articles

• 并行策略与性能优化：把 GPU 榨干
• Parallelism & Performance: Squeezing the GPUs

Co-cited Papers

These papers appear in the same articles as this one

• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning Tri Dao (2023)
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness Tri Dao et al. (2022)
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
• GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism Yanping Huang et al. (2019)

Related Papers

Other papers in the same domain

• DeepSeek-V4 Technical Report DeepSeek-AI (2026)
  ArchitectureLong ContextMixture of Experts
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI (2024)
  ArchitectureInference
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
  ArchitectureMixture of Experts
• MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs Ziheng Jiang et al. (2024)
  Architecture
• Mixtral of Experts Albert Q. Jiang et al. (2024)
  Mixture of ExpertsArchitecture
• Zero Bubble Pipeline Parallelism Penghui Qi et al. (2024)
  Architecture