Skip to content
LLM Primer

MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies

Authors: Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li, Zhiyuan Liu, Maosong Sun (2024)

arXiv: 2404.06395

Domains

Pretraining

TLDR (English)

Systematically validates the Warmup-Stable-Decay (WSD) learning-rate schedule on small models: a long stable phase allows flexible continual training and mid-flight branching, while a short final decay recovers cosine-level quality. The recipe was adopted by the MiniCPM family and Kimi K2, becoming a mainstream schedule for long-horizon training.

TLDR（中文）

在小模型上系统验证了 WSD（Warmup-Stable-Decay）学习率调度：中段保持恒定学习率便于随时续训与灵活实验，末尾短程衰减即可恢复到 cosine 调度的效果。该 recipe 被 MiniCPM 系列与 Kimi K2 等大模型沿用，成为长程训练的主流调度之一。

Appears in These Articles

Co-cited Papers

These papers appear in the same articles as this one

Related Papers

Other papers in the same domain