Cluster Networking & Communication: How GPUs Talk
Networking is the invisible protagonist of training systems
Section titled “Networking is the invisible protagonist of training systems”A brutal ratio: a GPU performs tens of thousands of multiply-adds per second, while handing one byte to the neighboring node is two orders of magnitude slower than a local memory read. In every training step, gradients, activations, and parameter copies flow between cards — with a weak network, even the most expensive GPUs spend their time waiting.
Learn networking with a two-layer mental model:
- Scale-up: inside a server or rack, NVLink + NVSwitch glue dozens of GPUs into one “super GPU.” Extremely high bandwidth (1.8TB/s class) and low latency, but the domain is limited (currently capped at 72 GPUs)
- Scale-out: between racks, InfiniBand or RoCE Ethernet links thousands to tens of thousands of nodes. One order of magnitude lower bandwidth and switch hops, but nearly unbounded
Mnemonic: NVLink domains do the fine work (tensor parallelism); Ethernet/IB does the coarse work (gradient sync of data parallelism).
The bandwidth ladder: an order-of-magnitude worldview
Section titled “The bandwidth ladder: an order-of-magnitude worldview”| Interconnect | Per-GPU bandwidth | Typical use |
|---|---|---|
| NVLink 3 (A100 gen) | 600 GB/s | intra-node TP |
| NVLink 4 (H100 gen) | 900 GB/s | intra-node TP |
| NVLink 5 (B200 gen) | 1.8 TB/s | TP/EP inside NVL72 racks |
| PCIe Gen5 ×16 | ~64 GB/s | host interconnect, peripherals |
| InfiniBand NDR 400G | ~50 GB/s per port | cross-node DP/EP |
| RoCE 800G Ethernet | ~100 GB/s per port | cross-node DP/EP (cheaper) |
NVLink 5 is ~28× PCIe and ~18× IB. That is why “never run TP across the NVLink domain” is iron law, and why GB200 NVL72 — 72 GPUs in one NVLink domain (130TB/s aggregate, 9 NVSwitch trays, 5000+ copper cables, fully liquid-cooled) — is a milestone: it moves traffic that used to cross IB into the domain, and MoE’s all-to-all benefits directly.
Scale-out’s two camps: IB vs RoCE
Section titled “Scale-out’s two camps: IB vs RoCE”- InfiniBand: a natively lossless network designed for supercomputing from day one; credit-based flow control, good congestion control, low latency; SHARP supports in-switch reduction. Expensive, vendor-locked (NVIDIA/Mellanox)
- RoCE (RDMA over Converged Ethernet): RDMA on standard Ethernet. Cheaper, open ecosystem, but you must hand-build losslessness: PFC to prevent drops + ECN/DCQCN for congestion control — misconfigured, you get PFC storms and head-of-line blocking. NVIDIA Spectrum-X’s significance is turning RoCE congestion management into a turnkey solution; Meta’s large clusters already use it to match IB
Post-2025 middle roads are proliferating: dual-plane fat-trees, rail-optimized topologies (one NIC plane per GPU so all-to-all traffic aligns with rails, minimizing cross-plane hops).
RDMA and collectives: NCCL is the de facto standard
Section titled “RDMA and collectives: NCCL is the de facto standard”RDMA lets a NIC read/write remote GPU memory directly (GPUDirect RDMA), turning transport from “software stack relocation” into “hardware DMA.”
The four collective primitives used in training (topology-aware, algorithm auto-selected by NCCL):
- AllReduce: sum everyone’s gradients (naive DP). Ring algorithm is bandwidth-optimal (2(N−1)/N × message size per GPU); tree is latency-optimal
- ReduceScatter + AllGather: the two-stage pattern of ZeRO/FSDP; same total traffic as AllReduce but memory-friendlier
- AllToAll: MoE’s dispatch/combine; every GPU sends distinct blocks to all others — bandwidth-hungry and jitter-sensitive
- Broadcast / Reduce: parameter distribution, metrics aggregation
Advanced: in-network reduction. IB SHARP and NVLS (NVLink SHARP) reduce data inside the switch/switch silicon, saving a round trip; a real ~2× class speedup for all-reduce. NCCL 2.17+ uses NVLS automatically when NVSwitch is present.
10k-GPU engineering: failures as routine
Section titled “10k-GPU engineering: failures as routine”ByteDance’s MegaScale (12288 GPUs) numbers tell the story best: MFU above 55%, yet hardware is failing at every moment — fault tolerance must be a system capability, not an ops reaction:
- Dual-plane networking: training survives maintenance on either plane
- Second-level fault localization: link flaps and slow nodes (stragglers) auto-isolated
- Hot redundancy: redundant experts, redundant parameter replicas
- Minute-level checkpoints + fast rollback; choosing the rollback point is itself a policy question (too far = wasted compute, too shallow = carrying the disease)
The DeepSeek route is another path: custom communication software (DeepEP for all-to-all, DualPipe scheduling, the 3FS filesystem), squeezing IB utilization dry for their specific model shape — 5.58M GPU-hours, 2 months for 671B, with the network not the bottleneck.
Validation checklist (self-built clusters)
Section titled “Validation checklist (self-built clusters)”nccl-tests/build/all_reduce_perf -b 128M -e 4G -f 2 -g 8 # does bandwidth meet spec?nvidia-smi topo -m # GPU-NIC affinity topologyib_write_bw / ib_read_bw # single-link RDMA baselineThree metrics to watch in training: communication share (>20% warrants investigation), step-time p99/p50 ratio (jitter means stragglers), and checkpoint duration (it sets the rollback granularity).
The other path: TPU’s topology philosophy
Section titled “The other path: TPU’s topology philosophy”The GPU camp uses “small fast domain + fat-tree between domains.” The TPU camp took a completely different road: a 3D torus of inter-chip links plus optically circuit-switched (OCS) uplinks. TPU v4 (jouppi2023) uses OCS to reconfigure a 4096-chip pod’s topology at runtime — carve high-bandwidth blocks for training-communication-heavy jobs, or split into independent partitions for batch inference. OCS also routes around failed links, turning “network topology” from a hardware constant into a software variable. Implication for the GPU world: is reconfigurable optical interconnect the only answer at 100k-GPU scale?
In-network computing and CPO
Section titled “In-network computing and CPO”Moving reduction from endpoints into the network (SHARP/NVLS is generation one), the next step is co-packaged optics (CPO) and memory-semantic interconnects — another step up in bandwidth×energy efficiency. Academia is also asking: can collective algorithms be co-designed with topology (hierarchical all-reduce on torus, rail-aware all-to-all scheduling)? As single links reach 1.6T/3.2T, will congestion-control algorithms (HPCC, successors to DCQCN) become the new research main battlefield?
🔬 Open Research Questions
Key questions and research directions in this area:
- Can the RoCE/Ethernet camp (including UEC) catch up to IB at 10k-GPU scale? Will the decider be congestion control, in-network computing, or operational ecosystem?
- Will OCS-style reconfigurable topology enter GPU clusters? How much end-to-end gain from switching topology per training phase (DP-sync-heavy vs EP-all-to-all-heavy)?
- Can all-to-all (MoE) reach a lossless experience on IB via in-network computing or switch scheduling? Where is the next order of magnitude after DeepEP?
- In 100k-GPU clusters, 'slow nodes' become a distribution, not an anomaly — can schedulers use network telemetry (per-flow latency, retransmit rate) to move jobs away from hotspots in real time?
References
- Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
The Megatron-LM systems paper: composes TP, pipelined 1F1B, and data parallelism into a full recipe, giving placement rules for 3D parallelism (TP inside the NVLink domain, PP across nodes, DP everywhere), the interleaved 1F1B schedule, and pipeline bubble analysis; trains GPT-3-scale models on 3072 A100 GPUs at 52.8% of peak FLOPs, setting the efficiency baseline for large-cluster training.
- TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings
Reveals the TPU v4 interconnect design: a 3D torus of electrical inter-chip links plus optically circuit-switched (OCS) reconfigurable uplinks in 4096-chip pods. A striking contrast to the GPU camp fat-tree plus NVLink approach: OCS can switch topology at runtime to fit different model shapes, representing a philosophy where the network defines the topology.
- MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
A full production record of training large models on 12288 GPUs at ByteDance: dual-plane fat-tree network design, a fault-tolerance system that treats failures as routine (auto diagnosis, fast rollback, redundant experts), and end-to-end profiling of communication, compute, and data loading. Pushes MFU above 55% and is the first paper to publish a complete engineering SOP for 10k-GPU training.
- DeepSeek-V3 Technical Report
671B parameters (37B activated) MoE, 14.8T token training; first large-scale production LLM to run FP8 training + Multi-Token Prediction, compressing training cost to $5.6M. Shook entire industry.
- GLM-130B: An Open Bilingual Pre-trained Model
Tsinghua + Zhipu's open Chinese-English bilingual 130B model, earliest representative technical report of Chinese LLM industrialization. Subsequent ChatGLM-6B/9B pushed open-source Chinese dialogue to mass adoption.