Ring Attention with Blockwise Transformers for Near-Infinite Context

Authors: Hao Liu, Matei Zaharia, Pieter Abbeel (2023)

arXiv: 2302.14045

Domains

Long Context

TLDR (English)

Distributes an ultra-long context as blocks across devices for blockwise attention: each device keeps only its own KV block while blocks circulate around a ring of devices for reuse, overlapping communication with compute. Context length can scale to near-infinity bounded by the sum of cluster memory, founding the context-parallelism line of work.

TLDR（中文）

把超长上下文切成 block 分布到多台设备上做分块注意力：每卡只保存自己那段 KV block，block 沿设备环逐个轮转复用，通信与计算重叠进行。在集群显存总和允许的范围内上下文长度可近乎无限扩展，是上下文并行（context parallelism）的奠基工作。

Appears in These Articles

• 并行策略与性能优化：把 GPU 榨干
• Parallelism & Performance: Squeezing the GPUs

Co-cited Papers

These papers appear in the same articles as this one

• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning Tri Dao (2023)
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness Tri Dao et al. (2022)
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
• GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism Yanping Huang et al. (2019)

Related Papers

Other papers in the same domain

• DeepSeek-V4 Technical Report DeepSeek-AI (2026)
  ArchitectureLong ContextMixture of Experts
• MISA: Mixture of Indexer Sparse Attention MISA Team (2026)
  Long ContextInference
• LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models Yukang Chen et al. (2023)
  Long ContextAlignment
• Needle in a Haystack — Pressure Testing LLMs Greg Kamradt (2023)
  Long ContextEvaluation
• YaRN: Efficient Context Window Extension of Large Language Models Bowen Peng et al. (2023)
  Long Context
• Efficient Streaming Language Models with Attention Sinks Guangxuan Xiao et al. (2023)
  InferenceLong Context