FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

作者： Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré (2022)

arXiv： 2205.14135

领域

推理

TLDR（中文）

FlashAttention 通过 IO-aware 的分块计算，在不牺牲精度的前提下，将注意力计算的内存 从 O(N²) 降至 O(N)，速度提升 2-4 倍。它改变了长上下文训练的可行性边界， 是现代高效 LLM 训练和推理不可或缺的底层优化。

TLDR (English)

FlashAttention uses IO-aware tiled computation to reduce attention memory from O(N²) to O(N) without losing precision, achieving 2-4x speedup. It fundamentally changed what's feasible for long-context training and is now an indispensable optimization in modern LLM training and inference.

出现在这些文章里

• 高效注意力：突破序列长度平方瓶颈
• Efficient Attention: Breaking the Quadratic Sequence Bottleneck

同被引用

这些论文与本文出现在同一篇文章中

• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Joshua Ainslie et al. (2023)
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning Tri Dao (2023)
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI (2024)
• FlashAttention-3: Fast and Accurate Attention with Asympotic IO Complexity Jay Shah et al. (2024)

相关论文

同一领域的其他论文

• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI (2024)
  架构推理
• FlashAttention-3: Fast and Accurate Attention with Asympotic IO Complexity Jay Shah et al. (2024)
  推理
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters Charlie Snell et al. (2024)
  推理能力推理
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Joshua Ainslie et al. (2023)
  推理架构
• Accelerating Large Language Model Decoding with Speculative Sampling Charlie Chen et al. (2023)
  推理
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning Tri Dao (2023)
  推理