DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

作者： DeepSeek-AI (2024)

arXiv： 2405.04434

领域

架构推理

TLDR（中文）

引入 Multi-head Latent Attention (MLA) 把 KV cache 砍到 1/13，让 236B MoE 推理价格碾压同档闭源。MLA 是 V3/R1 推理性价比的核心来源。

TLDR (English)

Introduces Multi-head Latent Attention (MLA) reducing KV cache to 1/13, making 236B MoE inference price crush same-tier closed-source. MLA is core source of V3/R1 inference cost-effectiveness.

出现在这些文章里

• 高效注意力：突破序列长度平方瓶颈
• Efficient Attention: Breaking the Quadratic Sequence Bottleneck

同被引用

这些论文与本文出现在同一篇文章中

• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Joshua Ainslie et al. (2023)
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness Tri Dao et al. (2022)
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning Tri Dao (2023)
• FlashAttention-3: Fast and Accurate Attention with Asympotic IO Complexity Jay Shah et al. (2024)

相关论文

同一领域的其他论文

• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
  架构混合专家
• Mixtral of Experts Albert Q. Jiang et al. (2024)
  混合专家架构
• FlashAttention-3: Fast and Accurate Attention with Asympotic IO Complexity Jay Shah et al. (2024)
  推理
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters Charlie Snell et al. (2024)
  推理能力推理
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Joshua Ainslie et al. (2023)
  推理架构
• Accelerating Large Language Model Decoding with Speculative Sampling Charlie Chen et al. (2023)
  推理