TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings

作者： Norman P. Jouppi, George Kurian, Sheng Li, Peter Ma, Ming-Xing Tan, David Patterson, David A. Patterson, Andy Swing, Yuan Xue, Brian Yuan, James Laudon, Harshit Khaitan, Matthew Morrow, Chris Peterson, Jonathan Ross, Amir Salek, Shreyas Saxena, Brian Servies, Ralph Wittig, Cliff Young, Hao Wu (2023)

arXiv： 2304.01433

领域

架构

TLDR（中文）

揭秘 TPU v4 的互联设计：芯片间 3D 环面（torus）电交换互连 + 光电路交换（OCS）可重构上联，一个 pod 4096 芯片。与 GPU 阵营的 fat-tree 加 NVLink 路线形成鲜明对照，OCS 允许在运行中切换拓扑以适配不同模型形状，代表了一种网络先于拓扑定义的系统设计哲学。

TLDR (English)

Reveals the TPU v4 interconnect design: a 3D torus of electrical inter-chip links plus optically circuit-switched (OCS) reconfigurable uplinks in 4096-chip pods. A striking contrast to the GPU camp fat-tree plus NVLink approach: OCS can switch topology at runtime to fit different model shapes, representing a philosophy where the network defines the topology.

出现在这些文章里

• 集群网络与通信：GPU 之间的对话
• Cluster Networking & Communication: How GPUs Talk

同被引用

这些论文与本文出现在同一篇文章中

• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
• MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs Ziheng Jiang et al. (2024)
• Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM Deepak Narayanan et al. (2021)
• GLM-130B: An Open Bilingual Pre-trained Model Aohan Zeng et al. (2022)

相关论文

同一领域的其他论文

• DeepSeek-V4 Technical Report DeepSeek-AI (2026)
  架构长上下文混合专家
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI (2024)
  架构推理
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
  架构混合专家
• MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs Ziheng Jiang et al. (2024)
  架构
• Mixtral of Experts Albert Q. Jiang et al. (2024)
  混合专家架构
• Zero Bubble Pipeline Parallelism Penghui Qi et al. (2024)
  架构