GLaM: Efficient Scaling of Language Models with Mixture-of-Experts

Authors: Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V Le, Yonghui Wu, Zhifeng Chen, Claire Cui (2021)

arXiv: 2112.06905

Domains

Mixture of ExpertsPretraining

TLDR (English)

1.2T parameter MoE achieves GPT-3 quality with 1/3 training compute, early representative of MoE "cost-effectiveness wins". Mixtral/DeepSeek-V2/V3 are its spiritual descendants.

TLDR（中文）

1.2T 参数 MoE 在 1/3 训练算力下达到 GPT-3 同等质量，是 MoE 路线"性价比胜出"的早期代表。Mixtral / DeepSeek-V2/V3 都是它的精神后裔。

Appears in These Articles

• Transformer Architecture：现代 LLM 的骨架
• Transformer Architecture: The Skeleton of Modern LLMs

Co-cited Papers

These papers appear in the same articles as this one

• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin et al. (2018)
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity William Fedus et al. (2021)
• Mixtral of Experts Albert Q. Jiang et al. (2024)
• Language Models are Unsupervised Multitask Learners Alec Radford et al. (2019)

Related Papers

Other papers in the same domain

• Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone Microsoft (2024)
  Pretraining
• Yi: Open Foundation Models by 01.AI 01. AI et al. (2024)
  Pretraining
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
  ArchitectureMixture of Experts
• The Llama 3 Herd of Models Meta AI (2024)
  Pretraining
• Mixtral of Experts Albert Q. Jiang et al. (2024)
  Mixture of ExpertsArchitecture
• Qwen2.5 Technical Report Qwen et al. (2024)
  Pretraining