LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models

作者： Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, Jiaya Jia (2023)

arXiv： 2309.12307

领域

长上下文对齐

TLDR（中文）

用 shifted sparse attention + LoRA 把 7B 模型扩到 100K 上下文，且只用一台 8xA100。是长上下文微调的工程标杆；另见 YaRN、PoSE。

TLDR (English)

Uses shifted sparse attention + LoRA to extend 7B model to 100K context with just one 8xA100 machine. Engineering benchmark for long-context fine-tuning; see also YaRN, PoSE.

出现在这些文章里

• 长上下文：让模型读得更远
• Long Context: Helping Models Read Farther

同被引用

这些论文与本文出现在同一篇文章中

• Needle in a Haystack — Pressure Testing LLMs Greg Kamradt (2023)
• H2O: Heavy-Hitter Oracle for Accurate KV Cache Compression Zichang Liu et al. (2023)
• YaRN: Efficient Context Window Extension of Large Language Models Bowen Peng et al. (2023)
• Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press et al. (2021)

相关论文

同一领域的其他论文

• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning DeepSeek-AI (2025)
  推理能力对齐
• KTO: Model Alignment as Prospect Theoretic Optimization Kawin Ethayarajh et al. (2024)
  对齐
• A General Theoretical Paradigm to Understand Learning from Human Preferences Mohammad Gheshlaghi Azar et al. (2023)
  对齐
• QLoRA: Efficient Finetuning of Quantized LLMs Tim Dettmers et al. (2023)
  对齐推理
• Needle in a Haystack — Pressure Testing LLMs Greg Kamradt (2023)
  长上下文评估
• RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback Harrison Lee et al. (2023)
  对齐