Effective Approaches to Attention-based Neural Machine Translation

Authors: Minh-Thang Luong, Hieu Pham, Christopher D. Manning (2015)

arXiv: 1508.04025

Domains

Architecture

TLDR (English)

Systematically compares global vs local attention and different scoring functions (dot/general/concat). The most commonly cited engineering reference when explaining "how attention scores are computed".

TLDR（中文）

系统化地比较 global vs local attention、不同打分函数（dot / general / concat），是后人讲 "attention score 是怎么算的" 时最常引用的工程化版本。

Appears in These Articles

• Attention：让每个位置选择上下文
• Attention: Choosing the Relevant Context

Co-cited Papers

These papers appear in the same articles as this one

• Neural Machine Translation by Jointly Learning to Align and Translate Dzmitry Bahdanau et al. (2014)
• Neural Machine Translation in Linear Time Nal Kalchbrenner et al. (2016)
• Sequence to Sequence Learning with Neural Networks Ilya Sutskever et al. (2014)
• Attention Is All You Need Ashish Vaswani et al. (2017)

Related Papers

Other papers in the same domain

• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI (2024)
  ArchitectureInference
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
  ArchitectureMixture of Experts
• Mixtral of Experts Albert Q. Jiang et al. (2024)
  Mixture of ExpertsArchitecture
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Joshua Ainslie et al. (2023)
  InferenceArchitecture
• GLM-130B: An Open Bilingual Pre-trained Model Aohan Zeng et al. (2022)
  Architecture
• Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press et al. (2021)
  ArchitectureLong Context