Textbooks Are All You Need

作者： Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, Yuanzhi Li (2023)

arXiv： 2306.11644

领域

预训练

TLDR（中文）

微软用 7B token 高质量"教科书级"合成数据训出 1.3B 在 HumanEval 上接近 GPT-3.5。把"数据质量 >> 数据规模"的故事讲到极致，开启 Phi 系列。

TLDR (English)

Microsoft uses 7B tokens of high-quality "textbook-level" synthetic data to train 1.3B model approaching GPT-3.5 on HumanEval. Takes "data quality >> data scale" story to extreme, launching Phi series.

出现在这些文章里

• 为什么 LLM 会涌现能力
• Why LLMs Emerge Abilities

同被引用

这些论文与本文出现在同一篇文章中

• Language Models are Few-Shot Learners Tom Brown et al. (2020)
• Training Compute-Optimal Large Language Models Jordan Hoffmann et al. (2022)
• Scaling Laws for Neural Language Models Jared Kaplan et al. (2020)
• GPT-4 Technical Report OpenAI (2023)

相关论文

同一领域的其他论文

• Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone Microsoft (2024)
  预训练
• Yi: Open Foundation Models by 01.AI 01. AI et al. (2024)
  预训练
• The Llama 3 Herd of Models Meta AI (2024)
  预训练
• Qwen2.5 Technical Report Qwen et al. (2024)
  预训练
• Qwen Technical Report Jinze Bai et al. (2023)
  预训练
• Gemini: A Family of Highly Capable Multimodal Models Gemini Team et al. (2023)
  预训练多模态