Training
Training
From pretraining to alignment
5 Articles
43 Papers Referenced
~40 min Reading Time
Recommended Reading Order
1
Pretraining and Scaling Law: How Models Learn
Pretraining objectives, data engineering, scaling laws, and compute-optimal training.
2
Fine-Tuning and Alignment: Making Models Follow Instructions
SFT, RLHF, DPO, and preference optimization methods.
3
Weight Update & Optimizers: How Parameters Move
From AdamW to Muon: first/second moments, learning-rate schedules, mixed precision, gradient clipping, and training-stability engineering.
4
Parallelism & Performance: Squeezing the GPUs
Data/tensor/pipeline parallelism, ZeRO and FSDP, activation recomputation, FlashAttention, MFU, and pipeline bubbles.
5
Cluster Networking & Communication: How GPUs Talk
NVLink and InfiniBand, RDMA and collective communication, scale-up vs scale-out, and the networking engineering of 10k-GPU clusters.