Fast Transformer Decoding: One Write-Head is All You Need

Authors: Noam Shazeer (2019)

arXiv: 1911.02150

Domains

InferenceArchitecture

TLDR (English)

Proposes Multi-Query Attention: all heads share the same K/V, reducing KV cache usage to 1/h. All modern KV cache optimization and long-context inference stories start from this 5-page paper.

TLDR（中文）

提出 Multi-Query Attention：所有 head 共享同一份 K/V，把 KV cache 占用打到 1/h。今天 KV cache 优化、长上下文推理的故事都从这篇 5 页短文开始。

Appears in These Articles

• 高效注意力：突破序列长度平方瓶颈
• Efficient Attention: Breaking the Quadratic Sequence Bottleneck

Co-cited Papers

These papers appear in the same articles as this one

• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Joshua Ainslie et al. (2023)
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness Tri Dao et al. (2022)
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning Tri Dao (2023)
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI (2024)

Related Papers

Other papers in the same domain

• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI (2024)
  ArchitectureInference
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
  ArchitectureMixture of Experts
• Mixtral of Experts Albert Q. Jiang et al. (2024)
  Mixture of ExpertsArchitecture
• FlashAttention-3: Fast and Accurate Attention with Asympotic IO Complexity Jay Shah et al. (2024)
  Inference
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters Charlie Snell et al. (2024)
  ReasoningInference
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Joshua Ainslie et al. (2023)
  InferenceArchitecture