GLM-130B: An Open Bilingual Pre-trained Model

Authors: Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Peng Zhang, Yuxiao Dong, Jie Tang (2022)

arXiv: 2210.02414

Domains

Architecture

TLDR (English)

Tsinghua + Zhipu's open Chinese-English bilingual 130B model, earliest representative technical report of Chinese LLM industrialization. Subsequent ChatGLM-6B/9B pushed open-source Chinese dialogue to mass adoption.

TLDR（中文）

清华+智谱开放的中英双语 130B 模型，是中国大模型工业化最早的代表性技术报告。后续 ChatGLM-6B/9B 把开源中文对话推到普及量级。

Appears in These Articles

• Positional Encoding：顺序从哪里来
• Positional Encoding: Where Does Order Come From

Co-cited Papers

These papers appear in the same articles as this one

• Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press et al. (2021)
• RoFormer: Enhanced Transformer with Rotary Position Embedding Jianlin Su et al. (2021)
• Attention Is All You Need Ashish Vaswani et al. (2017)
• XLNet: Generalized Autoregressive Pretraining for Language Understanding Zhilin Yang et al. (2019)

Related Papers

Other papers in the same domain

• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI (2024)
  ArchitectureInference
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
  ArchitectureMixture of Experts
• Mixtral of Experts Albert Q. Jiang et al. (2024)
  Mixture of ExpertsArchitecture
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Joshua Ainslie et al. (2023)
  InferenceArchitecture
• Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press et al. (2021)
  ArchitectureLong Context
• RoFormer: Enhanced Transformer with Rotary Position Embedding Jianlin Su et al. (2021)
  ArchitectureLong Context