Parallelism & Performance: Squeezing the GPUs
One GPU can’t hold it — so slice the problem
Section titled “One GPU can’t hold it — so slice the problem”All large-model training engineering answers one question: model, data, and compute — at least one must be split. Where do you cut, and how?
Three classic cuts, classified by “what is sliced”:
| Parallelism | What is sliced | Analogy | Communication pattern |
|---|---|---|---|
| Data parallel (DP) | the data | 100 copies of the homework, each person does one page, then pool results | one gradient all-reduce per step |
| Tensor parallel (TP) | matrix multiplications inside a layer | a big problem solved relay-style by four people | 2 all-reduces per layer, densest |
| Pipeline parallel (PP) | the model’s layers | a factory assembly line, one station per worker | point-to-point activations between adjacent stages |
Rule of thumb: more cuts → lighter per-GPU load, but more chit-chat between GPUs. The art of performance is finding the sweet spot between “computing fast” and “chatting too much.”
First, the memory ledger: 16 bytes per parameter
Section titled “First, the memory ledger: 16 bytes per parameter”Mixed-precision AdamW training carries this static overhead per parameter:
- BF16 weight copy 2B + BF16 gradient 2B + FP32 master weight 4B + optimizer states 8B ≈ 16 bytes
A 70B model therefore needs 1.12TB — an 80GB H100 cannot even hold the weights. This is the first driving force behind every parallel strategy.
Three kinds of parallelism + ZeRO: composing the recipe
Section titled “Three kinds of parallelism + ZeRO: composing the recipe”The evolution of DP: ZeRO/FSDP. Naive DP stores a full model replica on every GPU — wasteful. ZeRO shards the redundant parts across the DP group:
- ZeRO-1: shard optimizer states (saves 4/16, ~2× effective DP)
- ZeRO-2: also shard gradients (another 2/16)
- ZeRO-3 / FSDP: shard parameters too (almost everything; only transient copies during communication)
PyTorch FSDP is ZeRO-3 in official clothing. Scaling logic is simple: double the GPUs, halve the per-GPU static memory (at slightly higher communication).
TP: fast but expensive. Megatron-LM splits attention’s QKV projection column-wise and the output projection row-wise; same for MLP. Two all-reduces per layer in each direction — extremely dense communication, so TP must stay inside the NVLink domain (within a node or an NVL72 rack). Crossing nodes is a disaster.
PP: slow but frugal. GPipe splits layers into stages with micro-batched flow. Bubble fraction ≈ (p−1)/(m+p−1): 8 micro-batches across 4 stages wastes ~27% of time. 1F1B (one-forward-one-backward) first stabilizes memory; interleaved 1F1B splits each stage into v chunks and divides the bubble by v.
Parallelism Explorer: partition your model
Fits, with headroom for activations and buffers.
Tip: larger TP means denser communication (2 all-reduces per layer), keep it inside the NVLink domain; PP adds pipeline bubbles; DP+ZeRO is the cheapest axis to scale.
Coarse estimate: mixed-precision AdamW (~16 bytes/param), activations via selective-recompute + sequence-parallel heuristic; verify with profilers.
A modern recipe (DeepSeek-V3): a 671B model uses 16-way PP × 64-way EP (expert parallelism) × DP, with no TP at all — MLA attention is memory-light, and MoE’s all-to-all over IB beats TP’s all-reduce. EPLB balances experts; DualPipe overlaps communication with compute. The 3D recipe is a toolbox, not a doctrine.
Activations: the second memory black hole
Section titled “Activations: the second memory black hole”Beyond static states, activations (intermediate results kept for the backward pass) grow with batch×seq×layers. Three countermeasures:
- Activation recomputation (gradient checkpointing): store only each layer’s input; recompute intermediates during backward. Full recompute ≈ one extra forward (+33% FLOPs); selective recomputation recomputes only the attention core and keeps MLP activations — trading far fewer extra FLOPs for most of the memory savings. Megatron-Core’s default
- Sequence parallelism (SP): slice per-token activations (LayerNorm/dropout) along the sequence dimension, complementing TP (korthikanti2023)
- Context parallelism (CP) / Ring Attention: distribute an ultra-long context as blocks across devices; KV blocks circulate around a ring, overlapping communication with compute
Computing faster: FlashAttention and kernel fusion
Section titled “Computing faster: FlashAttention and kernel fusion”- FlashAttention (dao2022): never materialize the N×N attention matrix in HBM; compute blockwise in SRAM. Memory drops from O(N²) to O(N) — and it is faster while staying exact. FA2 improves work partitioning across warpgroups toward hardware limits; FA3 adds TMA, warp specialization, and FP8 on Hopper
- Kernel fusion: RMSNorm, RoPE, loss computation fused into single kernels — fewer launches, fewer HBM round-trips;
torch.compileautomates a batch of these - MoE all-to-all optimization: DeepEP turns dispatch/combine into standalone kernels that share SMs with compute, so expert parallelism stops dragging down MFU
MFU: the final report card
Section titled “MFU: the final report card”MFU (Model FLOPs Utilization) = actual throughput ÷ hardware peak. Formula: training FLOPs ≈ 6·N·D (N params, D tokens), divided by “peak FLOPS × time.”
Industry watermarks: average teams 30%–40%; the Megatron-LM paper hit 52.8% on 3072 A100s; ByteDance’s MegaScale pushed past 55% on 12288 GPUs. At MLPerf Training v5.0, 2496 GB200 GPUs delivered 1.8 PFLOPS of application throughput at 90% scaling efficiency — whether ten-thousand-card runs stay linear is the verdict on your networking and scheduling.
Debugging order (from experience): first check whether the dataloader starves the GPUs (waiting on data) → then the communication share (the prime suspect in low MFU) → then fragmented kernels (fusor/compiler) → and only last, suspect the hardware.
Auto-parallelism: hand the recipe to a compiler
Section titled “Auto-parallelism: hand the recipe to a compiler”Hand-tuning TP×PP×DP×ZeRO is combinatorial search by hand. Alpa was first to treat “intra-operator parallelism (TP-like) × inter-operator parallelism (PP-like)” as a compilation problem: dynamic programming over a hierarchical strategy space, matching or beating expert-tuned recipes at GPT-3 scale. Follow-ups (Galvatron, Oases, and the pipeline-compiler line) push into MoE and heterogeneous clusters. The 2026 consensus: auto tools are great baselines, but the last 5–10% still comes from humans tuning to the hardware topology.
The limits of pipeline bubbles
Section titled “The limits of pipeline bubbles”Zero Bubble Pipeline Parallelism splits backward into B (activation gradients) and W (weight gradients) tasks and reschedules them so W fills the bubbles — near-zero bubbles with no extra memory. Going further is DeepSeek’s DualPipe: a bidirectional pipeline interleaving neighboring stages’ forward and backward passes, hiding almost all communication. Bubble overhead has been squeezed from GPipe’s ~27% down to single digits — the pipeline-scheduling dividend is not exhausted.
RL training: the new frontier of performance
Section titled “RL training: the new frontier of performance”Agentic RL reframes everything: rollout (trajectory sampling) is bandwidth/latency-bound while training is FLOPs-bound — completely different resource profiles. Disaggregated architectures (RollArt, AgentRL) split the two phases onto heterogeneous hardware, absorbing rate differences with bounded queues and version management; in async pipelines, the staleness budget directly determines convergence. If you are building agentic-RL infrastructure, every point here is real money — and the optimization target is no longer pure throughput as in classic pretraining.
🔬 Open Research Questions
Key questions and research directions in this area:
- Can automatic parallelism search reliably beat human experts on MoE + heterogeneous clusters? What telemetry (live topology, jitter, failure rates) is still missing?
- After Zero Bubble, where is the theoretical floor of pipeline bubbles? Can the memory–bubble–communication three-way tradeoff be characterized as an operable Pareto frontier?
- For ultra-long-context training (100K+), where do Ring Attention / CP and sequence parallelism diverge? Will the memory wall hit position encodings first?
- In agentic RL's rollout–train disaggregation, can the tolerable staleness budget be derived analytically instead of tuned empirically?
References
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Introduces tensor parallelism (TP): split the matrix multiplications of self-attention and MLPs column-wise and row-wise across GPUs, parallelizing within layers while keeping inter-layer activations full precision, so each card holds 1/N of weights and gradients. Explicit, controllable, and regular in its communication pattern, it composes with pipeline and data parallelism into the 3D-parallel paradigm and founded large-model parallel training.
- GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
Splits a model into layer-wise stages executed in a pipeline and uses micro-batches to amortize pipeline bubbles (bubble fraction about (p-1)/(m+p-1)), combined with activation recomputation to control memory. The founding work of pipeline parallelism; later 1F1B, interleaved, and zero-bubble schedules all refine its bubble model.
- ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
Stops replicating optimizer state across every data-parallel copy: ZeRO-1/2/3 shard optimizer states, gradients, and parameters respectively across the DP group, trading communication for memory so per-GPU memory drops roughly linearly with GPU count. The full sharding idea of ZeRO-3 became PyTorch FSDP, making models too big for one machine trainable.
- Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
The Megatron-LM systems paper: composes TP, pipelined 1F1B, and data parallelism into a full recipe, giving placement rules for 3D parallelism (TP inside the NVLink domain, PP across nodes, DP everywhere), the interleaved 1F1B schedule, and pipeline bubble analysis; trains GPT-3-scale models on 3072 A100 GPUs at 52.8% of peak FLOPs, setting the efficiency baseline for large-cluster training.
- Reducing Activation Recomputation in Large Transformer Models
The Megatron-Core memory engineering paper: classifies activations into must-store and recomputable, proposing selective recomputation (recompute only the attention core, keep MLP activations) plus sequence parallelism to capture most memory savings with far fewer extra FLOPs than full recomputation; it also derives the per-layer activation formulas for transformers, becoming the reference for memory planning in large-model training.
- Ring Attention with Blockwise Transformers for Near-Infinite Context
Distributes an ultra-long context as blocks across devices for blockwise attention: each device keeps only its own KV block while blocks circulate around a ring of devices for reuse, overlapping communication with compute. Context length can scale to near-infinity bounded by the sum of cluster memory, founding the context-parallelism line of work.
- Zero Bubble Pipeline Parallelism
Splits backward into activation-gradient (B) and weight-gradient (W) computations and reschedules them so W fills pipeline bubbles, achieving near-zero-bubble pipeline parallelism with better throughput than 1F1B under the same memory budget, proving there is still structural headroom in pipeline schedules.
- Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning
The first system to jointly automate intra-operator parallelism (tensor-parallel-like) and inter-operator parallelism (pipeline-parallel-like): it hierarchically partitions the enormous strategy space and solves for the optimal split with dynamic programming. Matches or beats hand-tuned expert 3D-parallel configurations at GPT-3 scale, turning manual parallelism tuning into a compiler problem.
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
FlashAttention uses IO-aware tiled computation to reduce attention memory from O(N²) to O(N) without losing precision, achieving 2-4x speedup. It fundamentally changed what's feasible for long-context training and is now an indispensable optimization in modern LLM training and inference.
- FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
Uses more aggressive warp-level parallelism and work partitioning to double FlashAttention performance. Today vLLM/SGLang/Megatron training backends have all upgraded to FA-2.
- FlashAttention-3: Fast and Accurate Attention with Asympotic IO Complexity
Leverages H100's async TMA and FP8 to push attention to 1.2 PFLOPs while maintaining numerical precision. Key dependency for long-context + FP8 training on Hopper architecture.
- DeepSeek-V3 Technical Report
671B parameters (37B activated) MoE, 14.8T token training; first large-scale production LLM to run FP8 training + Multi-Token Prediction, compressing training cost to $5.6M. Shook entire industry.