The Right Tool for the Job: Matching Model and Instance Complexities

作者： Roy Schwartz, Gabriel Stanovsky, Shie Mannar, Jesse Dodge, Noah A. Smith (2020)

arXiv： 2003.03618

领域

推理

TLDR（中文）

提出自适应计算思想：不同输入实例需要的计算量不同。通过训练一个轻量级路由器将简单样本分配给较小模型、复杂样本分配给较大模型，可以在几乎不损失精度的情况下将平均推理成本降低 2-3 倍。

TLDR (English)

Proposes adaptive computation: different input instances require different amounts of computation. By training a lightweight router to assign simple samples to smaller models and complex samples to larger models, reduces average inference cost by 2-3x with minimal accuracy loss.

相关论文

同一领域的其他论文

• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Joshua Ainslie et al. (2023)
  推理架构
• H2O: Heavy-Hitter Oracle for Accurate KV Cache Compression Zichang Liu et al. (2023)
  推理
• Efficient Guided Generation for Large Language Models Brandon T. Willard et al. (2023)
  推理应用
• Efficient Streaming Language Models with Attention Sinks Guangxuan Xiao et al. (2023)
  推理长上下文
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness Tri Dao et al. (2022)
  推理
• chen2023-spec-sampling
  推理