What is quantization?
Introduction
Project introduction
What is quantization?
A trained model stores every weight as a 32-bit or 16-bit float. A 7 billion parameter model in 16 bits takes about 13.5 GB, more than many GPUs hold. Quantization stores each number in fewer bits, 8 or 4, together with a few scales that say what the small integers mean.
A number format is a choice of how many bits each number gets, the range it can cover, and the step between neighbouring values. With 4 bits there are only 16 possible values, so the range and the step have to be chosen with care, or most weights end up rounded to the same few codes. The damage is measured with perplexity, which is e to the power of the model's average loss on text it never saw and roughly how many characters it is choosing between. Lower is better.
Why 4 bits is hard
Each scale is set by the largest value it covers. If one scale serves a whole matrix, its single largest weight decides the step for every other weight. Large language models make this worse, because a few hidden channels carry activations tens of times larger than the rest, so the weights they meet matter far more than the others. The model in this project is too small to grow such channels, so a few were planted in it, with a rescaling that leaves its outputs unchanged, to make the effect visible.
The fixes in this project all deal with those few large values. They are smaller groups with their own scales, keeping or scaling the salient channels (AWQ), correcting rounding errors as you go (GPTQ), moving the outliers from activations into weights (SmoothQuant), and levels shaped like the weights (NF4).
Perplexity, lower is better
32 bitsWhat you will build
01
Scales and zero points
Asymmetric quantization, and the bytes of a quantized KV cache.
02
Group-wise int4 and NF4
A scale per group with protected columns, and the nearest NormalFloat level.
03
SmoothQuant and GPTQ
Smoothing factors for W8A8, and column by column error feedback.
Start with a zero point
Map real numbers onto b-bit codes so that 0 comes back exactly.