KV Cache and Quantization: Making Large Models Faster

Intuitive Engineering Research

Intuition: remember what you already computed

LLMs generate text one token at a time. If each new token required recomputing attention over all previous tokens, it would waste enormous compute. The intuition behind KV Cache is simple: store the Key and Value vectors computed for previous tokens and reuse them directly, avoiding redundant computation.

Quantization is another cost-reduction idea: model weights are typically 32-bit floats. If reduced to 16-bit, 8-bit, or even 4-bit, memory footprint and compute drop significantly. Some precision is lost, but the trade-off is often fully acceptable in practice.

Engineering view: the memory wall and precision trade-offs

KV Cache is one of the main memory consumers during inference, especially in long-context scenarios. Optimization directions include:

• PagedAttention (vLLM): Manage KV Cache in pages to reduce memory fragmentation and improve batching efficiency.
• KV Cache compression: Reduce cache size through quantization, pruning, sliding windows, or eviction policies.
• GQA / MQA: Share Key/Value across multiple attention heads to reduce cache volume.

Quantization techniques by category:

• PTQ (Post-Training Quantization): GPTQ, AWQ, SmoothQuant, etc. Quantize an already-trained model without retraining.
• QAT (Quantization-Aware Training): Simulate low precision during training; usually better results but higher cost.
• GGML/GGUF: Community-standard 4-bit formats that let large models run on laptops.

In practice, evaluate perplexity, downstream task accuracy, and end-to-end latency after quantization—not just memory savings. Different layers have different sensitivity to precision; mixed precision or per-layer tuning often works best.

Research view: the boundary of precision and efficiency

Research questions include: what are the limits of quantization? Can 1-bit or ternary weights still preserve language ability? How can activation distribution analysis find optimal clipping thresholds and scaling factors?

Another frontier is speculative decoding: a small model rapidly generates candidate sequences, and the large model verifies and corrects them in parallel, achieving 2-3x speedup without quality loss. This is essentially a rebalancing between computation and memory.

🔬 Open Research Questions

Key questions and research directions in this area:

1. Can KV cache memory footprint be further compressed without significant precision loss? Which approach is better: sparsification, quantization, or distillation?
2. What is the optimal strategy for mixed-precision inference? Which layers and parameters are more suitable for low precision?
3. Is quantization-aware training necessary? Can post-training quantization achieve the same effect?

Related Reading

Efficient Attention: Breaking the Quadratic Sequence Bottleneck

FlashAttention, sparse attention, and long-context inference optimization.

Intro Engineer Research

Attention: Choosing the Relevant Context

Attention weights, Q/K/V, multi-head self-attention, and compute costs.

Intro Engineer Research

Pretraining and Scaling Law: How Models Learn

Pretraining objectives, data engineering, scaling laws, and compute-optimal training.

Intro Engineer Research

Long Context: Helping Models Read Farther

Context window extension, positional encoding extrapolation, and long-text evaluation.

Intro Engineer Research

References

• Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon et al. (2023)

  Introduces OS "paged memory" concept to KV cache, virtually eliminating OOM waste and multiplying throughput 2-4x. vLLM thereby becomes de facto standard open-source inference engine; compute foundation for MCP/Agent era.

• LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale — Tim Dettmers et al. (2022)

  Reveals "emergent outliers" in large model activations and proposes mixed-precision solution. Core work behind bitsandbytes library, first enabling 175B models to fit in 8 A100s.

• QLoRA: Efficient Finetuning of Quantized LLMs — Tim Dettmers et al. (2023)

  4-bit NF4 + LoRA + paged optimizer enables SFT of 65B on single 48GB GPU. Open-source community fine-tuning of LLaMA-2/3, Qwen uses this approach almost 100%.

• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — Elias Frantar et al. (2022)

  First to achieve "4-bit quantization of 175B model on single GPU with almost no accuracy loss". Lowered LLM inference hardware barrier from 8xA100 to single consumer GPU, popularizing "run open-source LLMs locally".

• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration — Ji Lin et al. (2023)

  Discovers "few critical weights correspond to large activations", applies per-channel scaling by importance. More robust and faster than GPTQ at 4-bit, one of mainstream INT4 deployment solutions today.

• SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models — Guangxuan Xiao et al. (2022)

  Moves activation outliers to weights through equivalent mathematical transformation, making INT8 inference feasible. Key engineering discovery enabling GPU FP8/INT8 deployment.