ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators

Authors: Kevin Clark, Minh-Thang Luong, Quoc V. Le, Christopher D. Manning (2020)

arXiv: 2003.10555

Domains

Pretraining

TLDR (English)

Uses replaced token detection instead of MLM, allowing small models to achieve BERT-large level performance. Representative work on "pre-training objective determines sample efficiency".

TLDR（中文）

用 replaced token detection 替代 MLM，让小模型也能拿到 BERT-large 级表现。是"预训练目标决定样本效率"这条线索的代表作。

Appears in These Articles

• Embeddings：把离散符号放进连续空间
• Embeddings: Putting Discrete Symbols into Continuous Space

Co-cited Papers

These papers appear in the same articles as this one

• Convolutional Neural Networks for Sentence Classification Yoon Kim (2014)
• Distributed Representations of Words and Phrases and their Compositionality Tomas Mikolov et al. (2013)
• Efficient Estimation of Word Representations in Vector Space Tomas Mikolov et al. (2013)
• GloVe: Global Vectors for Word Representation Jeffrey Pennington et al. (2014)

Related Papers

Other papers in the same domain

• Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone Microsoft (2024)
  Pretraining
• Yi: Open Foundation Models by 01.AI 01. AI et al. (2024)
  Pretraining
• The Llama 3 Herd of Models Meta AI (2024)
  Pretraining
• Qwen2.5 Technical Report Qwen et al. (2024)
  Pretraining
• Qwen Technical Report Jinze Bai et al. (2023)
  Pretraining
• Gemini: A Family of Highly Capable Multimodal Models Gemini Team et al. (2023)
  PretrainingMultimodal