MISA: Mixture of Indexer Sparse Attention

Authors: MISA Team (2026)

arXiv: 2605.07363

Domains

Long ContextInference

TLDR (English)

A representative 2026 sparse-attention method: multiple learnable "indexers" mix to select the token subset each query attends to, moving long-context attention from static patterns (e.g. sliding windows) to learned dynamic routing, substantially cutting long-sequence inference cost.

TLDR（中文）

2026 年稀疏注意力的代表工作：用多个可学习的"索引器"（indexer）混合地选择每个查询需要关注的 token 子集，把长上下文注意力从静态模式（滑动窗口等）推向动态学习路由，显著降低长序列推理成本。

Appears in These Articles

• 高效注意力：突破序列长度平方瓶颈
• 长上下文：让模型读得更远
• Efficient Attention: Breaking the Quadratic Sequence Bottleneck
• Long Context: Helping Models Read Farther

Co-cited Papers

These papers appear in the same articles as this one

• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Joshua Ainslie et al. (2023)
• LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models Yukang Chen et al. (2023)
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness Tri Dao et al. (2022)
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning Tri Dao (2023)

Related Papers

Other papers in the same domain

• Adaptive Parallel Reasoning (UC Berkeley Technical Report EECS-2026-213) UC Berkeley (2026)
  ReasoningInference
• DeepSeek-V4 Technical Report DeepSeek-AI (2026)
  ArchitectureLong ContextMixture of Experts
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI (2024)
  ArchitectureInference
• FlashAttention-3: Fast and Accurate Attention with Asympotic IO Complexity Jay Shah et al. (2024)
  Inference
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters Charlie Snell et al. (2024)
  ReasoningInference
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Joshua Ainslie et al. (2023)
  InferenceArchitecture