Quantization & Model Compression

Quantization & Model Compression

What it is

Quantization is the process of reducing the precision of a model's weights and activations to decrease its memory footprint. Instead of using high-precision floating-point numbers (like FP32), quantization maps these values to lower-bit formats, such as INT8 or INT4. Model compression is the broader goal of making a model smaller and faster without significantly degrading its intelligence.

Why it matters

Large Language Models (LLMs) are often too massive to fit into the VRAM of a single consumer GPU. By shrinking the model, you reduce the hardware requirements for deployment, lower the cost per inference request, and increase throughput (tokens per second). This makes AI systems scalable and economically viable for production.

Core Concepts

• Weight Precision: Moving from FP16 (16-bit) to INT8 or INT4 reduces memory usage by 2x to 4x. FP8 is an emerging middle ground that balances speed and accuracy.
• Post-Training Quantization (PTQ): Techniques like GPTQ and AWQ that compress a pre-trained model. AWQ (Activation-aware Weight Quantization) is particularly effective because it protects the most important weights based on activation patterns.
• Calibration: The process of passing a small sample dataset through the model to determine the optimal scaling factors for the lower-precision range.
• The Accuracy-Cost Tradeoff: Every bit reduced increases the risk of "quantization error," where the model loses nuance or coherence. The goal is to find the "knee of the curve" where size is minimized but performance remains acceptable.

Common Mistakes

• Ignoring Perplexity: Relying solely on speed benchmarks without measuring the actual drop in model accuracy or perplexity.
• Over-quantizing: Pushing a small model (e.g., 3B parameters) to INT4, which often causes more degradation than pushing a large model (e.g., 70B parameters) to the same precision.
• Neglecting Hardware Support: Using a quantization format that the target GPU kernel does not natively support, which can actually slow down inference.

Track Connection

Quantization sits at the intersection of model architecture and hardware deployment. It directly enables efficient Inference by optimizing how weights are loaded from VRAM to the GPU cores, serving as the final optimization step before a model is deployed into a production AI Infrastructure.