TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings

Authors: Norman P. Jouppi, George Kurian, Sheng Li, Peter Ma, Ming-Xing Tan, David Patterson, David A. Patterson, Andy Swing, Yuan Xue, Brian Yuan, James Laudon, Harshit Khaitan, Matthew Morrow, Chris Peterson, Jonathan Ross, Amir Salek, Shreyas Saxena, Brian Servies, Ralph Wittig, Cliff Young, Hao Wu (2023)

arXiv: 2304.01433

Domains

Architecture

TLDR (English)

Reveals the TPU v4 interconnect design: a 3D torus of electrical inter-chip links plus optically circuit-switched (OCS) reconfigurable uplinks in 4096-chip pods. A striking contrast to the GPU camp fat-tree plus NVLink approach: OCS can switch topology at runtime to fit different model shapes, representing a philosophy where the network defines the topology.

TLDR（中文）

揭秘 TPU v4 的互联设计：芯片间 3D 环面（torus）电交换互连 + 光电路交换（OCS）可重构上联，一个 pod 4096 芯片。与 GPU 阵营的 fat-tree 加 NVLink 路线形成鲜明对照，OCS 允许在运行中切换拓扑以适配不同模型形状，代表了一种网络先于拓扑定义的系统设计哲学。

Appears in These Articles

• 集群网络与通信：GPU 之间的对话
• Cluster Networking & Communication: How GPUs Talk

Co-cited Papers

These papers appear in the same articles as this one

• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
• MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs Ziheng Jiang et al. (2024)
• Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM Deepak Narayanan et al. (2021)
• GLM-130B: An Open Bilingual Pre-trained Model Aohan Zeng et al. (2022)

Related Papers

Other papers in the same domain

• DeepSeek-V4 Technical Report DeepSeek-AI (2026)
  ArchitectureLong ContextMixture of Experts
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI (2024)
  ArchitectureInference
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
  ArchitectureMixture of Experts
• MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs Ziheng Jiang et al. (2024)
  Architecture
• Mixtral of Experts Albert Q. Jiang et al. (2024)
  Mixture of ExpertsArchitecture
• Zero Bubble Pipeline Parallelism Penghui Qi et al. (2024)
  Architecture