RoFormer: Enhanced Transformer with Rotary Position Embedding

作者： Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, Yunfeng Liu (2021)

arXiv： 2104.09864

领域

架构长上下文

TLDR（中文）

RoPE（旋转位置编码）是目前主流 LLM（LLaMA、Mistral、Qwen 等）采用的位置编码方案。 通过将位置信息以旋转矩阵的形式融入注意力计算，它能优雅地处理相对位置关系， 且在上下文长度外推时表现比绝对位置编码好得多。

TLDR (English)

RoPE (Rotary Position Embedding) is the position encoding scheme used in most major LLMs today (LLaMA, Mistral, Qwen, etc.). By incorporating position information as rotation matrices in attention computation, it elegantly handles relative positions and generalizes much better than absolute position encoding when extrapolating to longer context lengths.

出现在这些文章里

• Positional Encoding：顺序从哪里来
• Positional Encoding: Where Does Order Come From

同被引用

这些论文与本文出现在同一篇文章中

• Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press et al. (2021)
• Attention Is All You Need Ashish Vaswani et al. (2017)
• XLNet: Generalized Autoregressive Pretraining for Language Understanding Zhilin Yang et al. (2019)
• GLM-130B: An Open Bilingual Pre-trained Model Aohan Zeng et al. (2022)

相关论文

同一领域的其他论文

• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI (2024)
  架构推理
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
  架构混合专家
• Mixtral of Experts Albert Q. Jiang et al. (2024)
  混合专家架构
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Joshua Ainslie et al. (2023)
  推理架构
• LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models Yukang Chen et al. (2023)
  长上下文对齐
• Needle in a Haystack — Pressure Testing LLMs Greg Kamradt (2023)
  长上下文评估