DeepSeek-V4 Technical Report

作者： DeepSeek-AI (2026)

arXiv： 2606.19348

领域

架构长上下文混合专家

TLDR（中文）

万亿参数级 MoE 旗舰，采用稀疏注意力与混合注意力架构，原生支持约 1M token 上下文；技术报告称长上下文推理 FLOPs 与 KV cache 占用相比 V3.2 下降约一个数量级，权重以 MIT 协议开放。

TLDR (English)

A trillion-parameter-class MoE flagship using sparse and hybrid attention with native ~1M-token context. The report claims roughly an order-of-magnitude reduction in long-context inference FLOPs and KV cache footprint versus V3.2, with weights released under MIT.

出现在这些文章里

• 高效注意力：突破序列长度平方瓶颈
• 长上下文：让模型读得更远
• KV Cache 与量化：让大模型跑得更快
• 预训练与 Scaling Law：模型怎么学
• Efficient Attention: Breaking the Quadratic Sequence Bottleneck
• KV Cache and Quantization: Making Large Models Faster
• Long Context: Helping Models Read Farther
• Pretraining and Scaling Law: How Models Learn

同被引用

这些论文与本文出现在同一篇文章中

• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Joshua Ainslie et al. (2023)
• Language Models are Few-Shot Learners Tom Brown et al. (2020)
• LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models Yukang Chen et al. (2023)
• PaLM: Scaling Language Modeling with Pathways Aakanksha Chowdhery et al. (2022)

相关论文

同一领域的其他论文

• MISA: Mixture of Indexer Sparse Attention MISA Team (2026)
  长上下文推理
• Kimi K3: Open-Weight Trillion-Scale MoE for Agentic Intelligence Moonshot AI (2026)
  混合专家多模态应用
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI (2024)
  架构推理
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
  架构混合专家
• Mixtral of Experts Albert Q. Jiang et al. (2024)
  混合专家架构
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Joshua Ainslie et al. (2023)
  推理架构