Deduplicating Training Data Makes Language Models Better

Authors: Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, Nicholas Carlini (2022)

arXiv: 2107.06499

Domains

Pretraining

TLDR (English)

Systematically demonstrates that deduplicating training data significantly improves language model performance and reduces memorization. By removing near-duplicate and exact-duplicate examples from C4 and RealNews, models perform better on downstream tasks and are far less likely to emit training data verbatim.

TLDR（中文）

系统证明训练数据去重能显著提升语言模型性能并减少记忆效应。通过在 C4 和 RealNews 数据集上去除近似重复和精确重复，模型在下游任务上表现更好，且生成训练数据副本的概率大幅降低。

Related Papers

Other papers in the same domain

• The Llama 3 Herd of Models Meta AI (2024)
  Pretraining
• GPT-4 Technical Report OpenAI (2023)
  Pretraining
• PaLM: Scaling Language Modeling with Pathways Aakanksha Chowdhery et al. (2022)
  Pretraining
• Training Compute-Optimal Large Language Models Jordan Hoffmann et al. (2022)
  Pretraining
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity William Fedus et al. (2021)
  Mixture of ExpertsPretraining
• Language Models are Few-Shot Learners Tom Brown et al. (2020)
  PretrainingReasoning