Build from first principles

Quantize a Language Model to 4 Bits

Store a language model's numbers in fewer bits with scales and zero points, group-wise int4, protected channels, SmoothQuant, GPTQ, NF4 and a quantized KV cache.

What is quantization?Bits, range, step, and why a few large values cause most of the trouble

1Quantize with a Zero PointNumber formats and zero points

2Group-Wise 4-Bit Weights with Protected ColumnsScales for small groups

3SmoothQuant Smoothing FactorsMove the difficulty from activations to weights

4GPTQ: Quantize Column by Column with Error FeedbackCorrect the rounding error as you go

5NF4: Map Weights to the Nearest Normal LevelLevels shaped like the weights

6Size a Quantized KV CacheQuantize the KV cache

From Formulas to a Real ModelSee how your functions will be used on a trained language model

Quantize a Real Language ModelBreak it at 4 bits, then bring it back with groups, AWQ, GPTQ and SmoothQuant