YaRN: Efficient Context Window Extension of Large Language Models

Authors: Bowen Peng, Jeffrey Quesnelle, Honglu Fan, Enrico Shippole (2023)

arXiv: 2309.00071

Domains

Long Context

TLDR (English)

Applies NTK-aware interpolation + temperature correction on RoPE, extending context to 64K-128K with minimal training. Most open-source models today use YaRN or variants for length extension.

TLDR（中文）

在 RoPE 上做 NTK-aware 插值 + 温度修正，少量训练即可把上下文扩到 64K-128K。当下大多数开源模型扩长基本走 YaRN 或其变体。

Appears in These Articles

• 长上下文：让模型读得更远
• Long Context: Helping Models Read Farther

Co-cited Papers

These papers appear in the same articles as this one

• LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models Yukang Chen et al. (2023)
• Needle in a Haystack — Pressure Testing LLMs Greg Kamradt (2023)
• H2O: Heavy-Hitter Oracle for Accurate KV Cache Compression Zichang Liu et al. (2023)
• Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press et al. (2021)

Related Papers

Other papers in the same domain

• LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models Yukang Chen et al. (2023)
  Long ContextAlignment
• Needle in a Haystack — Pressure Testing LLMs Greg Kamradt (2023)
  Long ContextEvaluation
• Efficient Streaming Language Models with Attention Sinks Guangxuan Xiao et al. (2023)
  InferenceLong Context
• Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press et al. (2021)
  ArchitectureLong Context
• RoFormer: Enhanced Transformer with Rotary Position Embedding Jianlin Su et al. (2021)
  ArchitectureLong Context