YaRN: Efficient Context Window Extension of Large Language Models

作者： Bowen Peng, Jeffrey Quesnelle, Honglu Fan, Enrico Shippole (2023)

arXiv： 2309.00071

领域

长上下文

TLDR（中文）

在 RoPE 上做 NTK-aware 插值 + 温度修正，少量训练即可把上下文扩到 64K-128K。当下大多数开源模型扩长基本走 YaRN 或其变体。

TLDR (English)

Applies NTK-aware interpolation + temperature correction on RoPE, extending context to 64K-128K with minimal training. Most open-source models today use YaRN or variants for length extension.

出现在这些文章里

• 长上下文：让模型读得更远
• Long Context: Helping Models Read Farther

同被引用

这些论文与本文出现在同一篇文章中

• LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models Yukang Chen et al. (2023)
• Needle in a Haystack — Pressure Testing LLMs Greg Kamradt (2023)
• H2O: Heavy-Hitter Oracle for Accurate KV Cache Compression Zichang Liu et al. (2023)
• Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press et al. (2021)

相关论文

同一领域的其他论文

• LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models Yukang Chen et al. (2023)
  长上下文对齐
• Needle in a Haystack — Pressure Testing LLMs Greg Kamradt (2023)
  长上下文评估
• Efficient Streaming Language Models with Attention Sinks Guangxuan Xiao et al. (2023)
  推理长上下文
• Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press et al. (2021)
  架构长上下文
• RoFormer: Enhanced Transformer with Rotary Position Embedding Jianlin Su et al. (2021)
  架构长上下文