LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models

Authors: Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, Jiaya Jia (2023)

arXiv: 2309.12307

Domains

Long ContextAlignment

TLDR (English)

Uses shifted sparse attention + LoRA to extend 7B model to 100K context with just one 8xA100 machine. Engineering benchmark for long-context fine-tuning; see also YaRN, PoSE.

TLDR（中文）

用 shifted sparse attention + LoRA 把 7B 模型扩到 100K 上下文，且只用一台 8xA100。是长上下文微调的工程标杆；另见 YaRN、PoSE。

Appears in These Articles

• 长上下文：让模型读得更远
• Long Context: Helping Models Read Farther

Co-cited Papers

These papers appear in the same articles as this one

• Needle in a Haystack — Pressure Testing LLMs Greg Kamradt (2023)
• H2O: Heavy-Hitter Oracle for Accurate KV Cache Compression Zichang Liu et al. (2023)
• YaRN: Efficient Context Window Extension of Large Language Models Bowen Peng et al. (2023)
• Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press et al. (2021)

Related Papers

Other papers in the same domain

• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning DeepSeek-AI (2025)
  ReasoningAlignment
• KTO: Model Alignment as Prospect Theoretic Optimization Kawin Ethayarajh et al. (2024)
  Alignment
• A General Theoretical Paradigm to Understand Learning from Human Preferences Mohammad Gheshlaghi Azar et al. (2023)
  Alignment
• QLoRA: Efficient Finetuning of Quantized LLMs Tim Dettmers et al. (2023)
  AlignmentInference
• Needle in a Haystack — Pressure Testing LLMs Greg Kamradt (2023)
  Long ContextEvaluation
• RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback Harrison Lee et al. (2023)
  Alignment