Efficient Guided Generation for Large Language Models

Authors: Brandon T. Willard, Remi Louf (2023)

arXiv: 2307.09702

Domains

InferenceApplications

TLDR (English)

Proposes efficient constrained decoding that enforces JSON Schema, regular expressions, or context-free grammars during generation. Converts syntax constraints into finite-state automata, guaranteeing correct output format with minimal latency overhead.

TLDR（中文）

提出高效的约束解码方法，让大语言模型在生成过程中实时遵守 JSON Schema、正则表达式或上下文无关文法。通过将语法约束转化为有限状态自动机，在几乎不增加延迟的情况下保证输出格式正确。

Related Papers

Other papers in the same domain

• Model Context Protocol (MCP) Anthropic (2024)
  Applications
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Joshua Ainslie et al. (2023)
  InferenceArchitecture
• H2O: Heavy-Hitter Oracle for Accurate KV Cache Compression Zichang Liu et al. (2023)
  Inference
• Efficient Streaming Language Models with Attention Sinks Guangxuan Xiao et al. (2023)
  InferenceLong Context
• Tree of Thoughts: Deliberate Problem Solving with Large Language Models Shunyu Yao et al. (2023)
  ReasoningApplications
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness Tri Dao et al. (2022)
  Inference