跳转到内容
LLM Primer

MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies

作者： Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li, Zhiyuan Liu, Maosong Sun (2024)

arXiv： 2404.06395

领域

预训练

TLDR（中文）

在小模型上系统验证了 WSD（Warmup-Stable-Decay）学习率调度：中段保持恒定学习率便于随时续训与灵活实验，末尾短程衰减即可恢复到 cosine 调度的效果。该 recipe 被 MiniCPM 系列与 Kimi K2 等大模型沿用，成为长程训练的主流调度之一。

TLDR (English)

Systematically validates the Warmup-Stable-Decay (WSD) learning-rate schedule on small models: a long stable phase allows flexible continual training and mid-flight branching, while a short final decay recovers cosine-level quality. The recipe was adopted by the MiniCPM family and Kimi K2, becoming a mainstream schedule for long-horizon training.

出现在这些文章里

同被引用

这些论文与本文出现在同一篇文章中

相关论文

同一领域的其他论文