Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation

Authors: Ofir Press, Noah A. Smith, Mike Lewis (2021)

arXiv: 2108.12409

Domains

ArchitectureLong Context

TLDR (English)

Converts position information into linear bias on attention, enabling extrapolation to several times training length with zero parameters. Representative early long-context solution, competing with RoPE as two alternative approaches.

TLDR（中文）

把位置信息变成 attention 上的线性偏置，零参数即可外推到训练长度数倍以上。是早期长上下文方案的代表，与 RoPE 形成两条路线之争。

Appears in These Articles

• Positional Encoding：顺序从哪里来
• 长上下文：让模型读得更远
• Positional Encoding: Where Does Order Come From
• Long Context: Helping Models Read Farther

Co-cited Papers

These papers appear in the same articles as this one

• LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models Yukang Chen et al. (2023)
• Needle in a Haystack — Pressure Testing LLMs Greg Kamradt (2023)
• H2O: Heavy-Hitter Oracle for Accurate KV Cache Compression Zichang Liu et al. (2023)
• YaRN: Efficient Context Window Extension of Large Language Models Bowen Peng et al. (2023)

Related Papers

Other papers in the same domain

• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI (2024)
  ArchitectureInference
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
  ArchitectureMixture of Experts
• Mixtral of Experts Albert Q. Jiang et al. (2024)
  Mixture of ExpertsArchitecture
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Joshua Ainslie et al. (2023)
  InferenceArchitecture
• LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models Yukang Chen et al. (2023)
  Long ContextAlignment
• Needle in a Haystack — Pressure Testing LLMs Greg Kamradt (2023)
  Long ContextEvaluation