MISA: Mixture of Indexer Sparse Attention

作者： MISA Team (2026)

arXiv： 2605.07363

领域

长上下文推理

TLDR（中文）

2026 年稀疏注意力的代表工作：用多个可学习的"索引器"（indexer）混合地选择每个查询需要关注的 token 子集，把长上下文注意力从静态模式（滑动窗口等）推向动态学习路由，显著降低长序列推理成本。

TLDR (English)

A representative 2026 sparse-attention method: multiple learnable "indexers" mix to select the token subset each query attends to, moving long-context attention from static patterns (e.g. sliding windows) to learned dynamic routing, substantially cutting long-sequence inference cost.

出现在这些文章里

• 高效注意力：突破序列长度平方瓶颈
• 长上下文：让模型读得更远
• Efficient Attention: Breaking the Quadratic Sequence Bottleneck
• Long Context: Helping Models Read Farther

同被引用

这些论文与本文出现在同一篇文章中

• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Joshua Ainslie et al. (2023)
• LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models Yukang Chen et al. (2023)
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness Tri Dao et al. (2022)
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning Tri Dao (2023)

相关论文

同一领域的其他论文

• Adaptive Parallel Reasoning (UC Berkeley Technical Report EECS-2026-213) UC Berkeley (2026)
  推理能力推理
• DeepSeek-V4 Technical Report DeepSeek-AI (2026)
  架构长上下文混合专家
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI (2024)
  架构推理
• FlashAttention-3: Fast and Accurate Attention with Asympotic IO Complexity Jay Shah et al. (2024)
  推理
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters Charlie Snell et al. (2024)
  推理能力推理
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Joshua Ainslie et al. (2023)
  推理架构