Efficient Memory Management for Large Language Model Serving with PagedAttention

作者： Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica (2023)

arXiv： 2309.06180

领域

推理

TLDR（中文）

把操作系统的"分页内存"思想引入 KV cache，几乎消灭 OOM 浪费，让吞吐量翻 2-4 倍。vLLM 由此成为开源推理引擎事实标准；MCP/Agent 时代的算力底座。

TLDR (English)

Introduces OS "paged memory" concept to KV cache, virtually eliminating OOM waste and multiplying throughput 2-4x. vLLM thereby becomes de facto standard open-source inference engine; compute foundation for MCP/Agent era.

出现在这些文章里

• KV Cache 与量化：让大模型跑得更快
• KV Cache and Quantization: Making Large Models Faster

同被引用

这些论文与本文出现在同一篇文章中

• LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale Tim Dettmers et al. (2022)
• QLoRA: Efficient Finetuning of Quantized LLMs Tim Dettmers et al. (2023)
• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers Elias Frantar et al. (2022)
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration Ji Lin et al. (2023)

相关论文

同一领域的其他论文

• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI (2024)
  架构推理
• FlashAttention-3: Fast and Accurate Attention with Asympotic IO Complexity Jay Shah et al. (2024)
  推理
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters Charlie Snell et al. (2024)
  推理能力推理
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Joshua Ainslie et al. (2023)
  推理架构
• Accelerating Large Language Model Decoding with Speculative Sampling Charlie Chen et al. (2023)
  推理
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning Tri Dao (2023)
  推理