Neural Machine Translation by Jointly Learning to Align and Translate

作者： Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio (2014)

arXiv： 1409.0473

领域

架构

TLDR（中文）

注意力机制的开山之作（在 Transformer 之前）。作者发现 Seq2Seq 的固定长度瓶颈向量限制了翻译 质量，提出让解码器在生成每个词时都能"回顾"编码器的所有隐状态，动态分配注意力权重。 这个思想直接演化为 Transformer 的自注意力。

TLDR (English)

The seminal attention mechanism paper (pre-Transformer). The authors found that seq2seq's fixed-length bottleneck vector limited translation quality, and proposed letting the decoder dynamically attend to all encoder hidden states when generating each word. This idea directly evolved into Transformer self-attention.

出现在这些文章里

• Attention：让每个位置选择上下文
• Attention: Choosing the Relevant Context

同被引用

这些论文与本文出现在同一篇文章中

• Neural Machine Translation in Linear Time Nal Kalchbrenner et al. (2016)
• Effective Approaches to Attention-based Neural Machine Translation Minh-Thang Luong et al. (2015)
• Sequence to Sequence Learning with Neural Networks Ilya Sutskever et al. (2014)
• Attention Is All You Need Ashish Vaswani et al. (2017)

相关论文

同一领域的其他论文

• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI (2024)
  架构推理
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
  架构混合专家
• Mixtral of Experts Albert Q. Jiang et al. (2024)
  混合专家架构
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Joshua Ainslie et al. (2023)
  推理架构
• GLM-130B: An Open Bilingual Pre-trained Model Aohan Zeng et al. (2022)
  架构
• Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press et al. (2021)
  架构长上下文