Reducing Activation Recomputation in Large Transformer Models

作者： Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Matei Zaharia, Jonathan Ragan-Kelley, Amar Phanishayee (2023)

arXiv： 2205.05198

领域

架构

TLDR（中文）

Megatron-Core 的显存工程论文：把激活值分成「必存」与「可重算」两类，提出选择性重计算（只重算 attention 核心、MLP 激活整段存）加序列并行，用远小于全量重计算的额外 FLOPs 拿到大部分显存收益；还系统分析了 transformer 各层激活的构成公式，是大模型训练显存规划的参考标准。

TLDR (English)

The Megatron-Core memory engineering paper: classifies activations into must-store and recomputable, proposing selective recomputation (recompute only the attention core, keep MLP activations) plus sequence parallelism to capture most memory savings with far fewer extra FLOPs than full recomputation; it also derives the per-layer activation formulas for transformers, becoming the reference for memory planning in large-model training.

出现在这些文章里

• 并行策略与性能优化：把 GPU 榨干
• Parallelism & Performance: Squeezing the GPUs

同被引用

这些论文与本文出现在同一篇文章中

• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning Tri Dao (2023)
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness Tri Dao et al. (2022)
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
• GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism Yanping Huang et al. (2019)

相关论文

同一领域的其他论文

• DeepSeek-V4 Technical Report DeepSeek-AI (2026)
  架构长上下文混合专家
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI (2024)
  架构推理
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
  架构混合专家
• MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs Ziheng Jiang et al. (2024)
  架构
• Mixtral of Experts Albert Q. Jiang et al. (2024)
  混合专家架构
• Zero Bubble Pipeline Parallelism Penghui Qi et al. (2024)
  架构