Attention Is All You Need

作者： Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin (2017)

arXiv： 1706.03762

领域

架构

TLDR（中文）

Transformer 架构的奠基之作。作者完全用注意力机制替代了 RNN/CNN，提出多头自注意力与位置编码， 在机器翻译任务上大幅超越此前所有模型。今天所有主流 LLM 的底层架构都源于此论文。

TLDR (English)

The foundational paper that introduced the Transformer architecture. The authors replaced RNNs and CNNs entirely with attention mechanisms, proposing multi-head self-attention and positional encoding. It dramatically outperformed prior models on machine translation. Every major LLM today is built on this architecture.

出现在这些文章里

• Attention：让每个位置选择上下文
• Positional Encoding：顺序从哪里来
• Transformer Architecture：现代 LLM 的骨架
• Attention: Choosing the Relevant Context
• Positional Encoding: Where Does Order Come From
• Transformer Architecture: The Skeleton of Modern LLMs

同被引用

这些论文与本文出现在同一篇文章中

• Neural Machine Translation by Jointly Learning to Align and Translate Dzmitry Bahdanau et al. (2014)
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin et al. (2018)
• GLaM: Efficient Scaling of Language Models with Mixture-of-Experts Nan Du et al. (2021)
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity William Fedus et al. (2021)

相关论文

同一领域的其他论文

• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI (2024)
  架构推理
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
  架构混合专家
• Mixtral of Experts Albert Q. Jiang et al. (2024)
  混合专家架构
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Joshua Ainslie et al. (2023)
  推理架构
• GLM-130B: An Open Bilingual Pre-trained Model Aohan Zeng et al. (2022)
  架构
• Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press et al. (2021)
  架构长上下文