Efficient Streaming Language Models with Attention Sinks

作者： Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis (2023)

arXiv： 2309.17453

领域

推理长上下文

TLDR（中文）

提出 Attention Sink 现象：在自回归生成中，模型始终关注开头的几个初始 token。利用这一发现，StreamingLLM 可以在不重新计算的情况下处理无限长输入流，同时保持性能稳定。

TLDR (English)

Discovers the Attention Sink phenomenon: in autoregressive generation, models consistently attend to a few initial tokens. StreamingLLM leverages this to handle infinite-length input streams without recomputation while maintaining stable performance.

相关论文

同一领域的其他论文

• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Joshua Ainslie et al. (2023)
  推理架构
• Needle in a Haystack — Pressure Testing LLMs Greg Kamradt (2023)
  长上下文评估
• H2O: Heavy-Hitter Oracle for Accurate KV Cache Compression Zichang Liu et al. (2023)
  推理
• Efficient Guided Generation for Large Language Models Brandon T. Willard et al. (2023)
  推理应用
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness Tri Dao et al. (2022)
  推理
• RoFormer: Enhanced Transformer with Rotary Position Embedding Jianlin Su et al. (2021)
  架构长上下文