RoBERTa: A Robustly Optimized BERT Pretraining Approach

Authors: Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov (2019)

arXiv: 1907.11692

Domains

Pretraining

TLDR (English)

Uses more data, longer training, removes NSP to prove BERT was far from fully trained. Important not just for stronger model, but for first clearly demonstrating that "training recipe" itself is a core research question.

TLDR（中文）

用更多数据、更长训练、去掉 NSP，证明 BERT 远未训练充分。重要意义不只是更强的模型，而是首次清晰展示"训练配方"本身就是核心研究问题。

Appears in These Articles

• 预训练与 Scaling Law：模型怎么学
• Pretraining and Scaling Law: How Models Learn

Co-cited Papers

These papers appear in the same articles as this one

• Language Models are Few-Shot Learners Tom Brown et al. (2020)
• PaLM: Scaling Language Modeling with Pathways Aakanksha Chowdhery et al. (2022)
• Training Compute-Optimal Large Language Models Jordan Hoffmann et al. (2022)
• Scaling Laws for Neural Language Models Jared Kaplan et al. (2020)

Related Papers

Other papers in the same domain

• Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone Microsoft (2024)
  Pretraining
• Yi: Open Foundation Models by 01.AI 01. AI et al. (2024)
  Pretraining
• The Llama 3 Herd of Models Meta AI (2024)
  Pretraining
• Qwen2.5 Technical Report Qwen et al. (2024)
  Pretraining
• Qwen Technical Report Jinze Bai et al. (2023)
  Pretraining
• Gemini: A Family of Highly Capable Multimodal Models Gemini Team et al. (2023)
  PretrainingMultimodal