Ring Attention with Blockwise Transformers for Near-Infinite Context

作者： Hao Liu, Matei Zaharia, Pieter Abbeel (2023)

arXiv： 2302.14045

领域

长上下文

TLDR（中文）

把超长上下文切成 block 分布到多台设备上做分块注意力：每卡只保存自己那段 KV block，block 沿设备环逐个轮转复用，通信与计算重叠进行。在集群显存总和允许的范围内上下文长度可近乎无限扩展，是上下文并行（context parallelism）的奠基工作。

TLDR (English)

Distributes an ultra-long context as blocks across devices for blockwise attention: each device keeps only its own KV block while blocks circulate around a ring of devices for reuse, overlapping communication with compute. Context length can scale to near-infinity bounded by the sum of cluster memory, founding the context-parallelism line of work.

出现在这些文章里

• 并行策略与性能优化：把 GPU 榨干
• Parallelism & Performance: Squeezing the GPUs

同被引用

这些论文与本文出现在同一篇文章中

• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning Tri Dao (2023)
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness Tri Dao et al. (2022)
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
• GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism Yanping Huang et al. (2019)

相关论文

同一领域的其他论文

• DeepSeek-V4 Technical Report DeepSeek-AI (2026)
  架构长上下文混合专家
• MISA: Mixture of Indexer Sparse Attention MISA Team (2026)
  长上下文推理
• LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models Yukang Chen et al. (2023)
  长上下文对齐
• Needle in a Haystack — Pressure Testing LLMs Greg Kamradt (2023)
  长上下文评估
• YaRN: Efficient Context Window Extension of Large Language Models Bowen Peng et al. (2023)
  长上下文
• Efficient Streaming Language Models with Attention Sinks Guangxuan Xiao et al. (2023)
  推理长上下文