Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation

作者： Ofir Press, Noah A. Smith, Mike Lewis (2021)

arXiv： 2108.12409

领域

架构长上下文

TLDR（中文）

把位置信息变成 attention 上的线性偏置，零参数即可外推到训练长度数倍以上。是早期长上下文方案的代表，与 RoPE 形成两条路线之争。

TLDR (English)

Converts position information into linear bias on attention, enabling extrapolation to several times training length with zero parameters. Representative early long-context solution, competing with RoPE as two alternative approaches.

出现在这些文章里

• Positional Encoding：顺序从哪里来
• 长上下文：让模型读得更远
• Positional Encoding: Where Does Order Come From
• Long Context: Helping Models Read Farther

同被引用

这些论文与本文出现在同一篇文章中

• LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models Yukang Chen et al. (2023)
• Needle in a Haystack — Pressure Testing LLMs Greg Kamradt (2023)
• H2O: Heavy-Hitter Oracle for Accurate KV Cache Compression Zichang Liu et al. (2023)
• YaRN: Efficient Context Window Extension of Large Language Models Bowen Peng et al. (2023)

相关论文

同一领域的其他论文

• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI (2024)
  架构推理
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
  架构混合专家
• Mixtral of Experts Albert Q. Jiang et al. (2024)
  混合专家架构
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Joshua Ainslie et al. (2023)
  推理架构
• LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models Yukang Chen et al. (2023)
  长上下文对齐
• Needle in a Haystack — Pressure Testing LLMs Greg Kamradt (2023)
  长上下文评估