Textbooks Are All You Need

Authors: Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, Yuanzhi Li (2023)

arXiv: 2306.11644

Domains

Pretraining

TLDR (English)

Microsoft uses 7B tokens of high-quality "textbook-level" synthetic data to train 1.3B model approaching GPT-3.5 on HumanEval. Takes "data quality >> data scale" story to extreme, launching Phi series.

TLDR（中文）

微软用 7B token 高质量"教科书级"合成数据训出 1.3B 在 HumanEval 上接近 GPT-3.5。把"数据质量 >> 数据规模"的故事讲到极致，开启 Phi 系列。

Appears in These Articles

• 为什么 LLM 会涌现能力
• Why LLMs Emerge Abilities

Co-cited Papers

These papers appear in the same articles as this one

• Language Models are Few-Shot Learners Tom Brown et al. (2020)
• Training Compute-Optimal Large Language Models Jordan Hoffmann et al. (2022)
• Scaling Laws for Neural Language Models Jared Kaplan et al. (2020)
• GPT-4 Technical Report OpenAI (2023)

Related Papers

Other papers in the same domain

• Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone Microsoft (2024)
  Pretraining
• Yi: Open Foundation Models by 01.AI 01. AI et al. (2024)
  Pretraining
• The Llama 3 Herd of Models Meta AI (2024)
  Pretraining
• Qwen2.5 Technical Report Qwen et al. (2024)
  Pretraining
• Qwen Technical Report Jinze Bai et al. (2023)
  Pretraining
• Gemini: A Family of Highly Capable Multimodal Models Gemini Team et al. (2023)
  PretrainingMultimodal