XLNet: Generalized Autoregressive Pretraining for Language Understanding

作者： Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, Quoc V. Le (2019)

arXiv： 1906.08237

领域

架构

TLDR（中文）

提出 Permutation LM 把 AR 和 AE 的好处合并，配合 Transformer-XL 长序列；展示"预训练目标"本身仍然是开放问题，是 BERT 之后最有想象力的替代品。

TLDR (English)

Proposes Permutation LM to merge benefits of AR and AE, combined with Transformer-XL for long sequences. Shows "pre-training objective" is still an open question, most imaginative alternative after BERT.

出现在这些文章里

• Positional Encoding：顺序从哪里来
• Positional Encoding: Where Does Order Come From

同被引用

这些论文与本文出现在同一篇文章中

• Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press et al. (2021)
• RoFormer: Enhanced Transformer with Rotary Position Embedding Jianlin Su et al. (2021)
• Attention Is All You Need Ashish Vaswani et al. (2017)
• GLM-130B: An Open Bilingual Pre-trained Model Aohan Zeng et al. (2022)

相关论文

同一领域的其他论文

• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI (2024)
  架构推理
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
  架构混合专家
• Mixtral of Experts Albert Q. Jiang et al. (2024)
  混合专家架构
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Joshua Ainslie et al. (2023)
  推理架构
• GLM-130B: An Open Bilingual Pre-trained Model Aohan Zeng et al. (2022)
  架构
• Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press et al. (2021)
  架构长上下文