Accelerating Large Language Model Decoding with Speculative Sampling

Authors: Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, John Jumper (2023)

arXiv: 2302.01318

Domains

Inference

TLDR (English)

DeepMind's concurrent independent proposal of speculative sampling, theoretically proving acceleration while preserving sampling distribution. Together with Leviathan sets the direction; see also later Medusa, EAGLE.

TLDR（中文）

DeepMind 同期独立提出 speculative sampling，理论上证明可在保持采样分布不变的前提下加速。和 Leviathan 一起为这条路线定调；另见 Medusa、EAGLE 等后续。

Appears in These Articles

• Sampling 与 Decoding：从概率到文字
• Sampling and Decoding: From Probabilities to Text

Co-cited Papers

These papers appear in the same articles as this one

• Language Models are Few-Shot Learners Tom Brown et al. (2020)
• Fast Inference from Transformers via Speculative Decoding Yaniv Leviathan et al. (2023)
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters Charlie Snell et al. (2024)
• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason Wei et al. (2022)

Related Papers

Other papers in the same domain

• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI (2024)
  ArchitectureInference
• FlashAttention-3: Fast and Accurate Attention with Asympotic IO Complexity Jay Shah et al. (2024)
  Inference
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters Charlie Snell et al. (2024)
  ReasoningInference
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Joshua Ainslie et al. (2023)
  InferenceArchitecture
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning Tri Dao (2023)
  Inference
• QLoRA: Efficient Finetuning of Quantized LLMs Tim Dettmers et al. (2023)
  AlignmentInference