GLM-130B: An Open Bilingual Pre-trained Model

作者： Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Peng Zhang, Yuxiao Dong, Jie Tang (2022)

arXiv： 2210.02414

领域

架构

TLDR（中文）

清华+智谱开放的中英双语 130B 模型，是中国大模型工业化最早的代表性技术报告。后续 ChatGLM-6B/9B 把开源中文对话推到普及量级。

TLDR (English)

Tsinghua + Zhipu's open Chinese-English bilingual 130B model, earliest representative technical report of Chinese LLM industrialization. Subsequent ChatGLM-6B/9B pushed open-source Chinese dialogue to mass adoption.

出现在这些文章里

• Positional Encoding：顺序从哪里来
• Positional Encoding: Where Does Order Come From

同被引用

这些论文与本文出现在同一篇文章中

• Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press et al. (2021)
• RoFormer: Enhanced Transformer with Rotary Position Embedding Jianlin Su et al. (2021)
• Attention Is All You Need Ashish Vaswani et al. (2017)
• XLNet: Generalized Autoregressive Pretraining for Language Understanding Zhilin Yang et al. (2019)

相关论文

同一领域的其他论文

• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI (2024)
  架构推理
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
  架构混合专家
• Mixtral of Experts Albert Q. Jiang et al. (2024)
  混合专家架构
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Joshua Ainslie et al. (2023)
  推理架构
• Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press et al. (2021)
  架构长上下文
• RoFormer: Enhanced Transformer with Rotary Position Embedding Jianlin Su et al. (2021)
  架构长上下文