GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

Authors: Elias Frantar, Saleh Ashkboos, Torsten Hoefler, Dan Alistarh (2022)

arXiv: 2210.17323

Domains

Inference

TLDR (English)

First to achieve "4-bit quantization of 175B model on single GPU with almost no accuracy loss". Lowered LLM inference hardware barrier from 8xA100 to single consumer GPU, popularizing "run open-source LLMs locally".

TLDR（中文）

第一次实现"在单卡上 4-bit 量化 175B 模型而几乎不掉精度"。把 LLM 推理硬件门槛从 8xA100 拉到一张消费级显卡，普及"开源大模型本地跑"。

Appears in These Articles

• KV Cache 与量化：让大模型跑得更快
• KV Cache and Quantization: Making Large Models Faster

Co-cited Papers

These papers appear in the same articles as this one

• LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale Tim Dettmers et al. (2022)
• QLoRA: Efficient Finetuning of Quantized LLMs Tim Dettmers et al. (2023)
• Efficient Memory Management for Large Language Model Serving with PagedAttention Woosuk Kwon et al. (2023)
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration Ji Lin et al. (2023)

Related Papers

Other papers in the same domain

• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI (2024)
  ArchitectureInference
• FlashAttention-3: Fast and Accurate Attention with Asympotic IO Complexity Jay Shah et al. (2024)
  Inference
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters Charlie Snell et al. (2024)
  ReasoningInference
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Joshua Ainslie et al. (2023)
  InferenceArchitecture
• Accelerating Large Language Model Decoding with Speculative Sampling Charlie Chen et al. (2023)
  Inference
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning Tri Dao (2023)
  Inference