Accelerating Large Language Model Decoding with Speculative Sampling

作者： Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, John Jumper (2023)

arXiv： 2302.01318

领域

推理

TLDR（中文）

DeepMind 同期独立提出 speculative sampling，理论上证明可在保持采样分布不变的前提下加速。和 Leviathan 一起为这条路线定调；另见 Medusa、EAGLE 等后续。

TLDR (English)

DeepMind's concurrent independent proposal of speculative sampling, theoretically proving acceleration while preserving sampling distribution. Together with Leviathan sets the direction; see also later Medusa, EAGLE.

出现在这些文章里

• Sampling 与 Decoding：从概率到文字
• Sampling and Decoding: From Probabilities to Text

同被引用

这些论文与本文出现在同一篇文章中

• Language Models are Few-Shot Learners Tom Brown et al. (2020)
• Fast Inference from Transformers via Speculative Decoding Yaniv Leviathan et al. (2023)
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters Charlie Snell et al. (2024)
• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason Wei et al. (2022)

相关论文

同一领域的其他论文

• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI (2024)
  架构推理
• FlashAttention-3: Fast and Accurate Attention with Asympotic IO Complexity Jay Shah et al. (2024)
  推理
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters Charlie Snell et al. (2024)
  推理能力推理
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Joshua Ainslie et al. (2023)
  推理架构
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning Tri Dao (2023)
  推理
• QLoRA: Efficient Finetuning of Quantized LLMs Tim Dettmers et al. (2023)
  对齐推理