ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators

作者： Kevin Clark, Minh-Thang Luong, Quoc V. Le, Christopher D. Manning (2020)

arXiv： 2003.10555

领域

预训练

TLDR（中文）

用 replaced token detection 替代 MLM，让小模型也能拿到 BERT-large 级表现。是"预训练目标决定样本效率"这条线索的代表作。

TLDR (English)

Uses replaced token detection instead of MLM, allowing small models to achieve BERT-large level performance. Representative work on "pre-training objective determines sample efficiency".

出现在这些文章里

• Embeddings：把离散符号放进连续空间
• Embeddings: Putting Discrete Symbols into Continuous Space

同被引用

这些论文与本文出现在同一篇文章中

• Convolutional Neural Networks for Sentence Classification Yoon Kim (2014)
• Distributed Representations of Words and Phrases and their Compositionality Tomas Mikolov et al. (2013)
• Efficient Estimation of Word Representations in Vector Space Tomas Mikolov et al. (2013)
• GloVe: Global Vectors for Word Representation Jeffrey Pennington et al. (2014)

相关论文

同一领域的其他论文

• Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone Microsoft (2024)
  预训练
• Yi: Open Foundation Models by 01.AI 01. AI et al. (2024)
  预训练
• The Llama 3 Herd of Models Meta AI (2024)
  预训练
• Qwen2.5 Technical Report Qwen et al. (2024)
  预训练
• Qwen Technical Report Jinze Bai et al. (2023)
  预训练
• Gemini: A Family of Highly Capable Multimodal Models Gemini Team et al. (2023)
  预训练多模态