Fast Inference from Transformers via Speculative Decoding

作者： Yaniv Leviathan, Matan Kalman, Yossi Matias (2023)

arXiv： 2211.17192

领域

推理

TLDR（中文）

用一个小 draft model 预测多个 token，再让大模型一次校验，几乎无损地获得 2-3x 加速。是当下所有推理引擎（vLLM、TensorRT-LLM）的标配技术之一。

TLDR (English)

Uses small draft model to predict multiple tokens, large model verifies in one pass, achieving nearly lossless 2-3x speedup. Standard technique in all inference engines (vLLM, TensorRT-LLM) today.

出现在这些文章里

• Sampling 与 Decoding：从概率到文字
• Sampling and Decoding: From Probabilities to Text

同被引用

这些论文与本文出现在同一篇文章中

• Language Models are Few-Shot Learners Tom Brown et al. (2020)
• Accelerating Large Language Model Decoding with Speculative Sampling Charlie Chen et al. (2023)
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters Charlie Snell et al. (2024)
• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason Wei et al. (2022)

相关论文

同一领域的其他论文

• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI (2024)
  架构推理
• FlashAttention-3: Fast and Accurate Attention with Asympotic IO Complexity Jay Shah et al. (2024)
  推理
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters Charlie Snell et al. (2024)
  推理能力推理
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Joshua Ainslie et al. (2023)
  推理架构
• Accelerating Large Language Model Decoding with Speculative Sampling Charlie Chen et al. (2023)
  推理
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning Tri Dao (2023)
  推理