GLaM: Efficient Scaling of Language Models with Mixture-of-Experts

作者： Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V Le, Yonghui Wu, Zhifeng Chen, Claire Cui (2021)

arXiv： 2112.06905

领域

混合专家预训练

TLDR（中文）

1.2T 参数 MoE 在 1/3 训练算力下达到 GPT-3 同等质量，是 MoE 路线"性价比胜出"的早期代表。Mixtral / DeepSeek-V2/V3 都是它的精神后裔。

TLDR (English)

1.2T parameter MoE achieves GPT-3 quality with 1/3 training compute, early representative of MoE "cost-effectiveness wins". Mixtral/DeepSeek-V2/V3 are its spiritual descendants.

出现在这些文章里

• Transformer Architecture：现代 LLM 的骨架
• Transformer Architecture: The Skeleton of Modern LLMs

同被引用

这些论文与本文出现在同一篇文章中

• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin et al. (2018)
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity William Fedus et al. (2021)
• Mixtral of Experts Albert Q. Jiang et al. (2024)
• Language Models are Unsupervised Multitask Learners Alec Radford et al. (2019)

相关论文

同一领域的其他论文

• Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone Microsoft (2024)
  预训练
• Yi: Open Foundation Models by 01.AI 01. AI et al. (2024)
  预训练
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
  架构混合专家
• The Llama 3 Herd of Models Meta AI (2024)
  预训练
• Mixtral of Experts Albert Q. Jiang et al. (2024)
  混合专家架构
• Qwen2.5 Technical Report Qwen et al. (2024)
  预训练