DeepSeek-V4 Technical Report

Authors: DeepSeek-AI (2026)

arXiv: 2606.19348

Domains

ArchitectureLong ContextMixture of Experts

TLDR (English)

A trillion-parameter-class MoE flagship using sparse and hybrid attention with native ~1M-token context. The report claims roughly an order-of-magnitude reduction in long-context inference FLOPs and KV cache footprint versus V3.2, with weights released under MIT.

TLDR（中文）

万亿参数级 MoE 旗舰，采用稀疏注意力与混合注意力架构，原生支持约 1M token 上下文；技术报告称长上下文推理 FLOPs 与 KV cache 占用相比 V3.2 下降约一个数量级，权重以 MIT 协议开放。

Appears in These Articles

• 高效注意力：突破序列长度平方瓶颈
• 长上下文：让模型读得更远
• KV Cache 与量化：让大模型跑得更快
• 预训练与 Scaling Law：模型怎么学
• Efficient Attention: Breaking the Quadratic Sequence Bottleneck
• KV Cache and Quantization: Making Large Models Faster
• Long Context: Helping Models Read Farther
• Pretraining and Scaling Law: How Models Learn

Co-cited Papers

These papers appear in the same articles as this one

• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Joshua Ainslie et al. (2023)
• Language Models are Few-Shot Learners Tom Brown et al. (2020)
• LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models Yukang Chen et al. (2023)
• PaLM: Scaling Language Modeling with Pathways Aakanksha Chowdhery et al. (2022)

Related Papers

Other papers in the same domain

• MISA: Mixture of Indexer Sparse Attention MISA Team (2026)
  Long ContextInference
• Kimi K3: Open-Weight Trillion-Scale MoE for Agentic Intelligence Moonshot AI (2026)
  Mixture of ExpertsMultimodalApplications
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI (2024)
  ArchitectureInference
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
  ArchitectureMixture of Experts
• Mixtral of Experts Albert Q. Jiang et al. (2024)
  Mixture of ExpertsArchitecture
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Joshua Ainslie et al. (2023)
  InferenceArchitecture