RoBERTa: A Robustly Optimized BERT Pretraining Approach

作者： Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov (2019)

arXiv： 1907.11692

领域

预训练

TLDR（中文）

用更多数据、更长训练、去掉 NSP，证明 BERT 远未训练充分。重要意义不只是更强的模型，而是首次清晰展示"训练配方"本身就是核心研究问题。

TLDR (English)

Uses more data, longer training, removes NSP to prove BERT was far from fully trained. Important not just for stronger model, but for first clearly demonstrating that "training recipe" itself is a core research question.

出现在这些文章里

• 预训练与 Scaling Law：模型怎么学
• Pretraining and Scaling Law: How Models Learn

同被引用

这些论文与本文出现在同一篇文章中

• Language Models are Few-Shot Learners Tom Brown et al. (2020)
• PaLM: Scaling Language Modeling with Pathways Aakanksha Chowdhery et al. (2022)
• Training Compute-Optimal Large Language Models Jordan Hoffmann et al. (2022)
• Scaling Laws for Neural Language Models Jared Kaplan et al. (2020)

相关论文

同一领域的其他论文

• Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone Microsoft (2024)
  预训练
• Yi: Open Foundation Models by 01.AI 01. AI et al. (2024)
  预训练
• The Llama 3 Herd of Models Meta AI (2024)
  预训练
• Qwen2.5 Technical Report Qwen et al. (2024)
  预训练
• Qwen Technical Report Jinze Bai et al. (2023)
  预训练
• Gemini: A Family of Highly Capable Multimodal Models Gemini Team et al. (2023)
  预训练多模态