Neural Machine Translation in Linear Time

Authors: Nal Kalchbrenner, Lasse Espeholt, Karen Simonyan, Aaron van den Oord, Alex Graves, Koray Kavukcuoglu (2016)

arXiv: 1610.10099

Domains

Architecture

TLDR (English)

Uses dilated convolutions for seq2seq, liberating sequence modeling from "must use RNN sequential computation". Together with ConvS2S, represents the strongest attempt at parallel sequence modeling before Transformer.

TLDR（中文）

用扩张卷积做 seq2seq，把序列建模从"必须 RNN 顺序计算"中解放出来；和同期 ConvS2S 一起是 Transformer 之前"并行序列建模"的最强尝试。

Appears in These Articles

• Attention：让每个位置选择上下文
• Attention: Choosing the Relevant Context

Co-cited Papers

These papers appear in the same articles as this one

• Neural Machine Translation by Jointly Learning to Align and Translate Dzmitry Bahdanau et al. (2014)
• Effective Approaches to Attention-based Neural Machine Translation Minh-Thang Luong et al. (2015)
• Sequence to Sequence Learning with Neural Networks Ilya Sutskever et al. (2014)
• Attention Is All You Need Ashish Vaswani et al. (2017)

Related Papers

Other papers in the same domain

• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI (2024)
  ArchitectureInference
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
  ArchitectureMixture of Experts
• Mixtral of Experts Albert Q. Jiang et al. (2024)
  Mixture of ExpertsArchitecture
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Joshua Ainslie et al. (2023)
  InferenceArchitecture
• GLM-130B: An Open Bilingual Pre-trained Model Aohan Zeng et al. (2022)
  Architecture
• Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press et al. (2021)
  ArchitectureLong Context