What is quantization in LLMs?

Quantization is the process of representing a model's weights, and optionally its activations, with fewer bits than the format they were trained in. A typical pretrained LLM stores weights in FP16 or BF16 (2 bytes per parameter); a quantized model might use INT8 (1 byte) or INT4 (half a byte). Cutting the bit-width directly reduces the GPU memory footprint, lowers memory bandwidth pressure during inference, and lets larger models fit on smaller hardware, at the cost of some rounding error that has to be managed carefully.

Why does quantization work for neural networks?

Neural networks are surprisingly tolerant of small perturbations to their weights. Training already exposes the model to noise (dropout, data augmentation, optimizer noise), and gradient descent finds solutions that sit in relatively flat regions of the loss landscape. As a result, replacing FP16 weights with an INT8 approximation introduces a rounding error that lies inside the model's natural noise budget for most parameters. The interesting work in modern quantization is in handling the small subset of outlier weights and activations that do not fit inside that budget.

What is the difference between symmetric and asymmetric quantization?

Symmetric quantization maps a range [-A, A] symmetrically around zero to the signed integer range [-127, 127] using a single scale factor. Asymmetric quantization maps an arbitrary range [min, max] to the integer range [0, 255] using both a scale and a zero-point. Symmetric is simpler and faster (no zero-point arithmetic), but asymmetric uses the integer range more efficiently when the underlying distribution is skewed, which is common for activations that go through a ReLU or GELU.

What is the outlier problem in LLM quantization?

In a typical LLM, a small fraction of activations (often less than 1%) have magnitudes orders of magnitude larger than the rest. When you fit a single scale factor over a tensor that contains these outliers, the scale is dominated by the outliers and the typical values lose precision. This is why naive INT8 quantization can collapse the quality of models like OPT-175B, and it is the problem that LLM.int8(), SmoothQuant, GPTQ, and AWQ each address differently.

What is GPTQ and how does it quantize weights?

GPTQ (Frantar et al., 2022) is a post-training weight quantization algorithm that quantizes columns of a weight matrix one at a time and uses the inverse Hessian of the layer's reconstruction loss to redistribute the resulting error to the still-unquantized columns. Because the algorithm operates layer-by-layer with a single calibration pass, it scales to very large models and consistently outperforms naive round-to-nearest quantization at INT4 and below, with weight-only INT4 inference now standard for serving Llama, Mistral, and similar models.

What is AWQ and how does it differ from GPTQ?

AWQ (Activation-aware Weight Quantization, Lin et al., 2023) preserves a small percentage of weight channels that correspond to the most salient activations by scaling them up before quantization, so they survive the rounding step in higher precision. Unlike GPTQ, AWQ does not need a per-layer optimization loop, which makes it much faster to apply and easier to deploy. In practice, AWQ matches or exceeds GPTQ on most benchmarks at INT4, and it is widely supported in vLLM, TensorRT-LLM, and llama.cpp.

What is NF4 quantization and where is it used?

NF4 (4-bit NormalFloat) is a 4-bit quantization scheme introduced as part of QLoRA (Dettmers et al., 2023). Instead of distributing 16 quantization levels uniformly, NF4 places them at the quantiles of a standard normal distribution, which matches how weights are actually distributed after training. NF4 is the default 4-bit format in bitsandbytes and the standard choice for memory-efficient fine-tuning of large models on consumer GPUs.

What is the difference between weight-only and weight-plus-activation quantization?

Weight-only quantization stores model weights in low precision (INT8 or INT4) but keeps activations in higher precision (typically FP16). Matrix multiplications dequantize the weight tile, multiply in FP16, and the cost saving is purely in memory bandwidth. Weight-plus-activation quantization (W8A8 schemes like LLM.int8 and SmoothQuant) keeps activations in INT8 too, enabling integer matrix multiplications that are much faster on supported hardware, at the cost of much harder calibration and outlier handling.

Post-Training Quantization (PTQ) takes an already trained model and quantizes its weights without further training, using a small calibration dataset to compute scale factors. It is fast, cheap, and what GPTQ, AWQ, and NF4 fall under. Quantization-Aware Training (QAT) simulates the quantization noise during training (or fine-tuning) so the model can learn weights that survive the rounding step. QAT generally yields better accuracy at very low bit-widths but is rarely worth the training cost for LLMs at INT8 or INT4; PTQ is the dominant approach in practice.

Does quantization affect model accuracy?

INT8 weight-only quantization typically loses well under 1% on standard benchmarks. INT4 with a good algorithm (GPTQ, AWQ, NF4) loses 1-3% on most tasks but can degrade more on reasoning-heavy benchmarks and on very long context. Below 4 bits, accuracy starts to drop noticeably and the choice of algorithm and the per-group block size matter much more. For most production deployments, INT8 and well-tuned INT4 are accuracy-acceptable; sub-4-bit is still an active research area.

Quantization in LLMs Explained: INT8, INT4, GPTQ, AWQ, NF4

The GPU Memory Budget

A GPU has a fixed amount of memory, and an LLM uses it for two things. First, the model weights, which load once and stay put. Second, the KV cache, which grows with every user, every token they generate, and every layer of the model. Take LLaMA-3 70B in BF16: the weights alone take about 140 GB. Serving 8 users at 4K context adds another 11 GB for the KV cache (8 $\times$ 4096 tokens $\times$ 80 layers $\times$ 8 KV heads $\times$ 128 head_dim $\times$ 2 for K and V $\times$ 2 bytes $\approx$ 10.7 GB). A single H100 SXM only ships with 80 GB of HBM3. Whether the deployment runs at all, and how many users it can serve, comes down to how those two numbers add up against that 80.

GPU Memory Budget

MODEL

WEIGHTS

KV DTYPE

BATCH8

CONTEXT4096

GPU

Inside one transformer block, the FLOPs are dominated by linear projections. The MLP's gate, up, and down projections (in SwiGLU-style LLaMA, Mistral, and Qwen architectures the MLP holds roughly two thirds of the parameters) and the attention QKV-and-output projections (most of the remaining third) account for almost all of the arithmetic. A representative LLaMA-2 7B parameter breakdown is approximately 66% MLP, 31% attention, and 3% other (embedding, layernorms, and the output head). Layernorms, softmax, residual adds, and embeddings each contribute single-digit percentages. Almost every FLOP of a forward pass is some $y = Wx$ call on a weight matrix $W$ . That is why weight quantization is the highest-leverage lever: it shrinks the static memory footprint and speeds up the dominant compute kernel at the same time.

Cutting weight bit-width from FP16 to INT4 quarters the weight memory footprint and quarters the bytes-per-FLOP that the matmul has to fetch from HBM. Both numbers matter independently. Memory headroom unlocks larger batches or longer contexts on the same hardware. Bytes-per-FLOP matters because inference matmul is memory-bandwidth-bound rather than arithmetic-bound: typical arithmetic intensity sits at dozens of FLOPs per byte while the H100 roofline is around 250 FLOPs per byte at FP16, so most LLM matmuls finish faster purely from doing less HBM traffic per output element. KV cache quantization is a separate, complementary lever covered later in the article. Activation quantization is a third, harder lever covered in the outlier section.

Introduction

Inference of a large language model is bounded less by raw arithmetic than by memory and bandwidth. A 70B-parameter model stored in FP32 weights occupies around 280 GB of GPU memory, which exceeds the capacity of every single accelerator currently available. Even loading it in BF16, the standard training format, costs roughly 140 GB, which is more than an H100 can hold by itself. Quantization is the standard workaround: store the same weights in fewer bits, accept some rounding error, and recover most of the model's behaviour while cutting memory and memory-bandwidth pressure by a factor of two to eight.

What quantization actually does

Quantization represents each weight (and sometimes each activation) using fewer bits than the format the model was trained in, with a small scale factor recovering an approximation of the original value at compute time. The whole field is about deciding which approximations the model survives and which ones it does not.

The reason this works at all is that trained networks are unusually tolerant of small perturbations to their weights. Training noise (dropout, data augmentation, optimizer noise, finite-precision arithmetic) already exposes the model to constant low-level corruption, and gradient descent ends up in regions of the loss landscape that are flat in most directions. The rounding error introduced by replacing FP16 weights with an INT8 or INT4 approximation sits well inside that natural noise budget for the vast majority of parameters.

A small fraction of weights and activations are not tolerant at all, and a single one of them rounded badly can shift outputs noticeably. Most of the algorithmic content in modern quantization is about identifying those exceptions and handling them separately.

Memory reduction (FP32 to INT8)

2-4x

Inference speedup on INT8 hardware

<1%

Typical perplexity loss at INT8

Floating Point Formats

Three floating-point formats account for almost all modern deep learning numerics: FP32, FP16, and BF16. All three follow the same IEEE 754 layout of one sign bit, some exponent bits, and some mantissa bits, but they split the 16 or 32 bits between exponent and mantissa very differently, and those splits decide what range of values a quantization scheme has to compress later.

How a floating-point number is encoded

Each IEEE 754 value is the product of three pieces stored in separate bit fields:

\text{value} \;=\; (-1)^{s} \,\times\, (1.m) \,\times\, 2^{\,e \,-\, \text{bias}}

Sign bit $(s)$ : one bit that decides whether the number is positive or negative.
Exponent $(e)$ : a small block of bits that decides the magnitude (how large or small the number can be). More exponent bits buys more dynamic range.
Mantissa $(m)$ : the remaining bits that decide how precisely a value can be pinned down within the magnitude the exponent has chosen. More mantissa bits buys finer spacing between representable numbers.

The bias is a fixed constant per format (127 for FP32, 15 for FP16, 127 for BF16) used so the same bit pattern can represent both very small and very large numbers. The colors below match the bit-structure diagrams.

Bit Structure Comparison

FP3232 bits total

8 exponent

23 mantissa

Range: ±3.4×10³⁸ | Precision: ~7 decimal digits

BF1616 bits total

8 exponent

7 mantissa

Range: ±3.4×10³⁸ | Precision: ~2 decimal digits | Same exponent as FP32

FP1616 bits total

5 exp

10 mantissa

Range: ±65,504 | Precision: ~3.3 decimal digits

BF16 vs FP16 for training

FP16 tops out at 65,504, and gradients during large-model training routinely spike above that. Once an FP16 value overflows it becomes infinity and propagates everywhere downstream, which breaks the optimizer state and usually the run. BF16 keeps the same 8-bit exponent as FP32, so its representable range matches FP32 exactly; the cost is mantissa precision (7 bits versus FP16's 10).

Google introduced BF16 for TPU training in 2018, and it has since become the default storage format for large-scale training across NVIDIA, AMD, and TPU hardware. FP16 still appears where range can be controlled (mostly inference and small fine-tunes), often with loss-scaling to keep activations safely inside its range.

The Mantissa Determines Precision

The mantissa controls how finely adjacent values are spaced within a given range. FP32's 23 mantissa bits give a relative spacing of about $2^{-23} \approx 1.2 \times 10^{-7}$ , FP16's 10 bits give roughly $2^{-10} \approx 10^{-3}$ , and BF16's 7 bits give about $2^{-7} \approx 8 \times 10^{-3}$ .

The exponent controls the range itself. More exponent bits buys you a larger dynamic range, but the gaps between representable values scale up proportionally with the magnitudes they cover.

From floats to integers

Quantizing from BF16 to INT8 or INT4 replaces a floating-point representation with a fixed-point integer. The integer has no exponent component, only a linear range (-127 to 127 for signed INT8, 0 to 15 for unsigned INT4). To recover the original magnitudes during compute, every quantized tensor carries a small scale factor that maps the integer range back to the float range. How that scale is chosen, and how much of the tensor it has to cover at once, accounts for most of the difference between modern quantization algorithms.

INT8 vs FP8 — Where the Representable Values Sit

The Core Idea

Instead of storing weights as real numbers with floating-point representation:

w ∈ R (continuous, floating-point)

We store them as discrete integers drawn from a small finite set:

w_hat ∈ {q_1, q_2, ..., q_k} (discrete integer)

where k = 2^bits: 256 levels for 8-bit, 16 levels for 4-bit, 4 levels for 2-bit

The simplest form is uniform quantization: divide the weight range into equally spaced buckets and round each weight to the nearest bucket center.

q = round((x - zero_point) / scale)

quantize

↕

dequantize

x_hat = q * scale + zero_point

The scale maps the integer range to the float range. The zero_point shifts the mapping so the integer zero corresponds to the float zero (or some other reference). Choosing these two parameters well determines most of your quantization quality.

The quantization error for a single weight is bounded: |x - x_hat| ≤ scale/2. The error is small when scale is small, and scale is small when the weight range is narrow. Wide ranges force large scales, which force large errors. This is why outlier channels are catastrophic: they drag the scale up for the entire tensor.

Why Neural Networks Survive This

Training on diverse data forces the model to be robust to input perturbations. That same robustness extends to weight perturbations: quantization error rarely changes predictions because the model has already learned not to depend on precision it does not need.

1. Bits and Precision

The choice of bit width is the first decision in any quantization scheme. Each bit you remove cuts memory in half but also halves the number of representable values.

Format	Bits	Representable Values	Memory (70B model)	Practical Use
FP32	32	~4.3 billion	280 GB	Training baseline
BF16	16	65,536	140 GB	Training standard, inference baseline
FP16	16	65,536	140 GB	Inference, mixed precision
INT8	8	256	70 GB	Production inference
INT4	4	16	35 GB	Consumer GPU deployment
INT2	2	4	17.5 GB	Research frontier

The jump from 256 levels (INT8) to 16 levels (INT4) is severe. With only 16 buckets, quantization error becomes large enough to meaningfully change model outputs for sensitive layers. This is why INT4 requires smarter algorithms than naive rounding.

Group-Wise Quantization

Rather than computing one scale for an entire weight matrix (per-tensor) or one per output channel (per-channel), group-wise quantization computes a separate scale for every 128 weights. This dramatically improves accuracy at INT4, at the cost of storing extra scale values. For a weight matrix with N elements, the overhead is N/128 extra FP16 values, typically under 1% of total size.

Interactive: Bit Widths

Switch between bit widths and see how precision, representable levels, and memory change.

Bits and Precision

2. Quantization Schemes

The mapping from floats to integers can be set up in several ways. The choice depends on whether your distribution is symmetric and how wide it is.

Symmetric Quantization

s \;=\; \frac{\max(|W|)}{2^{\,b-1} - 1}

q \;=\; \mathrm{round}\!\left(\frac{W}{s}\right)

\hat{W} \;=\; q \cdot s

Float zero maps exactly to integer zero. The integer range is used symmetrically around zero.

Works well for weights, which are typically centered near zero after training. Simple to implement. The -128 slot in INT8 is often wasted (used range is -127 to 127).

Asymmetric Quantization

s \;=\; \frac{\max(W) - \min(W)}{2^{\,b} - 1}

z \;=\; -\,\mathrm{round}\!\left(\frac{\min(W)}{s}\right)

q \;=\; \mathrm{round}\!\left(\frac{W}{s}\right) + z

Uses the full integer range by introducing a zero-point offset. Better for distributions that are not centered at zero.

Activations after ReLU are always non-negative, so symmetric quantization wastes half the range. Asymmetric uses 0-255 fully. Slightly more overhead due to zero-point arithmetic.

Per-Tensor

One scale for the entire weight matrix. Minimal memory overhead. Most sensitive to outliers because a single large value forces a high scale for everything.

Per-Channel

One scale per output neuron (row of the weight matrix). Each row has its own dynamic range. Much better than per-tensor with modest overhead.

Per-Group (g=128)

One scale per 128 consecutive weights. Best accuracy at low bits. GPTQ and AWQ both use this. Industry standard for INT4 deployment.

The Practical Default

Most LLM quantization tools default to per-group symmetric quantization with group size 128. This is the setting used by GPTQ and AWQ in their published results. The extra scale values add roughly 0.4% to total model size at INT4, which is negligible.

Interactive: Schemes

Compare symmetric and asymmetric quantization. See how weights map to integer levels and how the scale changes with the distribution.

Quantization Schemes

levels:

Step Size

0.64

Total Error

1.67

Avg Error

0.139

3. Weight vs Activation Quantization

A transformer's forward pass involves two kinds of tensors: weights (fixed after training) and activations (different for every input). These have very different quantization properties.

Weight-Only Quantization (W4A16, W8A16)

Weights are stored in INT4 or INT8. They are dequantized to FP16 before the matrix multiply. Activations remain in FP16 throughout.

+ No calibration data required
+ Reduces memory bandwidth (the bottleneck)
+ Works well at INT4 with GPTQ/AWQ
- Matmul still runs in FP16 (no INT8 tensor core benefit)
- Dequantization overhead at inference

W+A

Weight + Activation Quantization (W8A8)

Both weights and activations are quantized to INT8. The matrix multiply runs entirely in INT8 using dedicated tensor cores.

+ Uses INT8 tensor cores (true 2x speedup)
+ Supported by TensorRT, vLLM
- Requires calibration data for activation scales
- Harder to handle outlier activations
- More accuracy loss without careful handling

Why Weight-Only Dominates LLM Inference

LLM inference during decoding is memory-bandwidth-bound, not compute-bound. At batch size 1, the GPU spends most of its time loading weight matrices from HBM, not multiplying. Weight-only quantization cuts that load by 4x (INT4) while keeping compute in FP16. The actual matmul is fast regardless. Adding activation quantization gains relatively little on this workload while introducing more complexity and accuracy risk.

4. PTQ vs QAT

There are two fundamentally different strategies for introducing quantization into a model. They differ in whether the model gets to adapt its weights to compensate for quantization error.

PTQ

Post-Training Quantization

Take a pretrained model and quantize its weights without any gradient updates. Use a small calibration dataset to compute scale factors. No retraining needed.

1. Load pretrained weights W

2. Run calibration data through model

3. Compute scale = f(activation statistics)

4. Apply: W_q = round(W / scale)

5. Done. No backprop.

GPTQ, AWQ, and LLM.int8() are all PTQ methods. PTQ is the only practical option for models above ~13B parameters, where retraining costs are prohibitive.

QAT

Quantization-Aware Training

Insert "fake quantization" operations into the forward pass during training. The model experiences quantization noise during every forward and backward step, and its weights adapt to minimize loss despite it.

FakeQuant(x) = round(x/scale)*scale

# forward: simulate quantization

dL/dx ≈ dL/dFakeQuant(x)

# backward: straight-through estimator

The gradient of rounding is zero everywhere except at discontinuities. The straight-through estimator approximates it as 1, which lets gradients flow through. QAT typically achieves better quality than PTQ at 1-4 bit widths, but requires the original training setup.

Property	PTQ	QAT
Training required	No (calibration only)	Yes (full or fine-tune)
GPU time	Hours (GPTQ) to minutes (AWQ)	Days to weeks
Quality at INT8	Near-lossless	Near-lossless
Quality at INT4	Small perplexity increase	Better than PTQ
Quality at 1-2 bit	Significant degradation	More viable with QAT
Practical for 70B+	Yes	Rarely

When Each Applies

For LLM deployment with public checkpoints (LLaMA, Mistral, etc.), PTQ is the only option since you do not have access to the training pipeline. For models you train yourself, QAT can recover accuracy at aggressive bit widths. The field is moving toward methods like LLM-QAT (Meta, 2023) that apply QAT to already-pretrained models with data-free or data-efficient fine-tuning.

5. Calibration

PTQ quantization requires calibration: a process of running a small dataset through the model to measure activation statistics and compute good scale factors. Calibration does not change model weights. It only measures them.

Calibration Steps

1Select 128-512 representative samples from your target domain (code, text, domain-specific documents)
2Run these samples through the full model in FP16, recording per-layer activation tensors
3Compute statistics per tensor: min, max, mean absolute, and sometimes percentiles (e.g., 99.9th)
4Compute scale and zero-point from these statistics to minimize quantization error
5Apply quantization and optionally validate perplexity on a held-out set

Min-Max Calibration

scale = (max - min) / 255. Simple but sensitive to outliers. A single large activation value forces a wide scale, degrading precision for all other values.

Percentile Calibration

Use 99.9th percentile instead of max. Clips extreme outliers and improves precision for the bulk of the distribution. TensorRT uses this by default.

Domain Mismatch Matters

Calibration data should match your deployment domain. A model calibrated on Wikipedia text will have poorly tuned activation scales for code. The calibration set does not need to be large (128-512 samples usually suffices), but it should be representative of real inputs.

Interactive: Calibration

Adjust the calibration dataset size and method. See how scale and zero-point are computed and how they affect quantization error.

Calibration

q = round(x / s) + z|x̂ = s × (q − z)

SAMPLES100

6. The Outlier Problem

In 2022, Tim Dettmers and colleagues published a paper that changed how the field thinks about LLM quantization. They discovered that as transformer models scale past 6.7 billion parameters, a systematic phenomenon emerges: a small number of activation channels develop extreme magnitudes.

The Discovery: Emergent Outliers (Dettmers et al., 2022)

In models larger than 6.7B parameters, a fixed set of hidden dimensions (roughly 1-5% of channels) carry activation values 10x to 100x larger than typical channels. These outlier channels are consistent: the same dimensions are outliers across different inputs and different layers.

# Typical channel

[-0.5, 0.3, -0.2, 0.1, 0.4]

range: ~[-0.5, 0.5]

# Outlier channel

[-68, 45, -72, 51, -60]

range: ~[-80, 60]

If you compute a per-tensor scale from this mixed distribution, the scale is set by the outlier range (say 80). The step size for INT8 becomes 80/127 ≈ 0.63. Normal channel values in the range [-0.5, 0.5] now quantize to just 1-2 distinct integers. You have effectively lost all precision for most of the model.

The Outlier Problem

LLM.int8(): Mixed-Precision Decomposition

Dettmers et al. also proposed the solution: instead of trying to quantize outlier channels, separate them and handle them in FP16. The algorithm has three steps.

LLM.int8() Algorithm

1
Identify outlier dimensions. Scan activation matrix X and find all feature dimensions (columns) where any value exceeds threshold |x| > 6.0. This threshold was found empirically to separate true outliers from normal large values.
2
Split into two sub-matrices. Extract the outlier columns from X (call them X_outlier) and the corresponding rows from W (call them W_outlier). Keep these in FP16. The remaining columns/rows form X_int8 and W_int8.
3
Compute two matmuls and combine. Run X_int8 @ W_int8 in INT8 (dequantize result to FP16), run X_outlier @ W_outlier in FP16, then add the results. The final output is in FP16 with full precision for the channels that need it.

# LLM.int8() decomposition

Y = X_outlier_fp16 @ W_outlier_fp16

+ dequant(quant(X_int8) @ quant(W_int8))

# Combined in FP16. No perplexity loss on 6.7B+ models.

LLM.int8()

Split computation: outlier columns in FP16, rest in INT8. Enables 8-bit serving without perplexity loss at any model size.

SmoothQuant

Migrate quantization difficulty from activations to weights by applying a per-channel scale. Both W and A then quantize cleanly.

AWQ / GPTQ

Weight-only quantization avoids the problem entirely by leaving activations in FP16. Outliers only affect weights indirectly via calibration.

7. GPTQ: Optimal Weight Quantization

GPTQ (Frantar et al., 2022) approaches quantization as an optimization problem. Rather than rounding each weight independently, it finds the INT4 or INT3 weights that minimize the error in the layer's output, given real calibration data. This requires second-order information about the loss landscape.

\min_{\hat{W}} \| WX - \hat{W}X \|^2_F

Find integer weights W_hat that minimize the Frobenius norm of the output difference, given calibration inputs X.

The Hessian Connection

GPTQ builds on the Optimal Brain Surgeon (OBS) framework from Hassibi and Stork (1993). OBS shows that the optimal update to remaining weights after perturbing weight w_j is determined by the inverse Hessian of the loss with respect to weights.

For a linear layer Y = WX, the Hessian of the output error with respect to W is:

H = 2XX^T

H is a d_in x d_in matrix. H_jj captures how much weight column j affects the output.

GPTQ Algorithm (Full Detail)

1.
Run calibration data (128 samples) through the layer and collect input activations $X$ . Compute $H = 2XX^{T}$ plus a small damping term $\lambda I$ for numerical stability.
2.
Compute the Cholesky decomposition of $H$ : $H = LL^{T}$ , then solve for $H^{-1}$ . This is done once per layer and reused for all columns.
3.
Process columns in order (or in lazy batches of 128). For each column $j$ : quantize $w_{j}$ to the nearest INT4 value, compute the quantization error $\delta_{j} = w_{j} - \mathrm{quant}(w_{j})$ .
4.
Propagate the error to all remaining columns: $W_{j+1:} \mathrel{-}= \delta_{j} \cdot \dfrac{H^{-1}_{j,\, j+1:}}{H^{-1}_{jj}}$ . This adjusts unquantized weights to compensate for quantizing column $j$ .
5.
Repeat for each column. The Cholesky structure allows efficient updates without recomputing $H^{-1}$ from scratch. Lazy batch updates $(B = 128)$ reduce memory bandwidth overhead on GPU.

GPTQ vs Naive RTN

column 0 / 8

Why Error Propagation Works

Independently rounding each weight ignores the correlations between weights captured in H. When you quantize column j and then adjust remaining columns, you are solving a local quadratic subproblem optimally. Each quantized weight is chosen to minimize output error given what has already been quantized.

Practical Results

LLaMA-2 7B at INT4: perplexity ~5.7 vs 5.4 FP16
LLaMA-2 65B at INT4: nearly identical to FP16
Quantization time: ~4 GPU hours for 175B model on A100
Memory: 70B model fits on single A100 80GB at INT4

Connection to OBS

The Optimal Brain Surgeon (1993) was originally designed for pruning: which weight can you remove with least damage, and how do you adjust other weights to compensate? GPTQ applies the same second-order update formula but constrains weights to a discrete set (INT4 levels) rather than setting them to zero. The core OBS insight transfers directly.

8. AWQ: Activation-Aware Quantization

AWQ (Lin et al., MIT Han Lab, 2023) takes a fundamentally different angle than GPTQ. Instead of minimizing quantization error after the fact via error propagation, AWQ asks: which weights matter most, and can we protect them?

The Key Observation

Not all weight channels have equal impact on the output. A weight w_i in channel i contributes to the output in proportion to both its own magnitude and the magnitude of the activation it multiplies. Large activation channels amplify any weight error in that channel.

Importance(w_i) = |w_i| × mean(|a_i|)

Where a_i is the activation associated with weight channel i, averaged over calibration samples.

The AWQ paper finds that only about 1% of weight channels have high importance by this metric. But protecting just that 1% preserves most of the model's output quality.

The Scaling Trick

AWQ cannot keep important channels in FP16 (that would require runtime conditional logic). Instead, it uses a mathematical equivalence: scaling a weight channel up before quantization, then scaling the corresponding activation down at runtime, leaves the output unchanged but improves quantization fidelity for important weights.

# Mathematical identity:

W @ X = (W * s) @ (X / s)

# AWQ applies this as:

Q(W * s) @ (X / s)

# Quantizing W*s instead of W gives better precision

# for channels where s > 1 (important channels)

The per-channel scale s_i for each weight channel is found by a grid search over the calibration data. For each candidate scale vector, AWQ evaluates the quantized model's output error on the calibration batch. The best scale is the one that minimizes this error. No backpropagation is needed.

AWQ: Activation-Aware Quantization

AWQ vs GPTQ: Practical Comparison

AWQ Strengths

+ Quantization in minutes (vs hours for GPTQ)
+ No per-column Hessian update computation
+ Works well at 3-4 bit with per-group scales
+ Inference kernels (AWQ-GEMM) are highly optimized

GPTQ Strengths

+ Theoretically optimal per-column updates
+ Slightly better perplexity at very low bits
+ More established tooling (AutoGPTQ)
+ Better studied with theoretical guarantees

Published benchmarks: LLaMA-7B W4A16 - GPTQ: 5.67 perplexity, AWQ: 5.78 perplexity, FP16: 5.47 perplexity (WikiText-2). The gap narrows at 13B+ where the model is more robust.

9. NF4 (4-bit NormalFloat)

GPTQ and AWQ both decide which levels to round to algorithmically, but they still place those levels on a uniform grid by default. NF4, introduced by Dettmers et al. as part of QLoRA (2023), changes the levels themselves. The 4-bit number system stops being uniformly spaced and instead places its sixteen quantization levels at the quantiles of a standard normal distribution.

The motivation is empirical and simple. Trained neural-network weights are approximately Gaussian after normalization: most values sit close to zero, and very few are out near the tails. Uniform 4-bit spends one-sixteenth of its precision near zero (where the bulk of weights live) and one-sixteenth out in the thin tails (where almost nothing lives). NF4 reallocates levels so the ones near zero are densely packed and the ones in the tails are sparse, matching the actual weight density.

How the NF4 levels are derived

Take a standard normal distribution N(0, 1) and split it into 16 equal-probability bins. The level for each bin is the expected value of a normal sample falling into that bin (the "quantile midpoint"). The 16 expected values are then rescaled so the extremes map to -1 and +1. Because the construction is symmetric, exactly one of the sixteen levels lands at zero, which matters because dropout and weight decay push many real weights to be exactly zero.

Quantizing a weight tensor with NF4 is then the same operation as any other 4-bit scheme: normalize each block of weights by their absolute maximum, round to the nearest NF4 level, store the four-bit index. The trick is purely in the level placement.

NF4 vs Uniform 4-bit

Where NF4 wins

Memory-efficient fine-tuning. The bitsandbytes library uses NF4 as the default 4-bit storage format for QLoRA, where the base model is frozen in NF4 and LoRA adapters are trained in BF16 on top. NF4 gives a slightly better quality / memory tradeoff than uniform INT4 because most rounding errors are smaller.

Where NF4 loses

Throughput-optimized inference serving. GPTQ and AWQ still beat NF4 on most accuracy benchmarks at INT4, because they pair their level grid with a per-tensor error-correction pass (GPTQ) or a salient-channel scaling step (AWQ). NF4's strength is being cheap to produce, not being optimal.

10. Error Propagation

Quantization error at each layer does not stay local. It propagates through subsequent layers and can compound. Understanding how error accumulates helps explain why some layers need higher precision.

\epsilon = x - \hat{x} = x - \text{round}(x/s) \cdot s

For uniform quantization, error is bounded: |ε| ≤ s/2 where s is the step size.

Smaller step size (more bits or narrower range) means smaller error. This is why group-wise quantization helps.

Not all layers are equally sensitive to this error:

High Sensitivity

Attention QKV projections. First and last transformer layers. Embedding and unembedding layers.

These layers set the representational basis. Error here affects everything downstream.

Medium Sensitivity

MLP up and down projections. Output projections in attention blocks.

Some redundancy. INT4 works with careful group-wise quantization.

Lower Sensitivity

Middle-layer gate projections. Layers after residual connections.

Residual connections preserve gradient signal even if the layer output is noisy.

Mixed-Precision Quantization

Rather than applying the same bit width uniformly, modern deployments often use mixed precision: 8-bit for the first and last layers, and the QKV projections; 4-bit for MLP weights in middle layers. Tools like AutoGPTQ and llama.cpp support per-layer bit-width settings. The accuracy gain from protecting sensitive layers is large relative to the small memory cost.

The standard evaluation metric for quantization quality is perplexity on WikiText-2. A well-implemented INT4 quantization of a 7B model should show perplexity within 0.3-0.5 points of FP16. If it is more than that, something went wrong in calibration, scheme choice, or outlier handling.

Interactive: Error

See how quantization error accumulates across layers and how bit width affects the total error at the model output.

Error Propagation

Per-Layer Error

2.0%

Layers

4

Accumulated

7.8%

LAYERS4

ERR/LAYER2.0%

Practical Deployment

The right quantization choice depends on your hardware, latency target, and accuracy requirements. Here is the practical decision tree.

Scenario	Recommended	Bits	Notes
Maximum accuracy	BF16	16	No quantization. Best for 7B models on A100.
Server (A100/H100), high throughput	W8A8 (SmoothQuant)	8	Uses INT8 tensor cores. 2x throughput gain.
Server, memory-constrained	GPTQ W4A16	4	Best accuracy at INT4. Slower to quantize but better quality.
Consumer GPU (RTX 4090, 24GB)	AWQ W4A16	4	Fast to quantize. Run 70B on 2x consumer cards.
CPU / Apple Silicon	GGUF Q4_K_M	4	llama.cpp format. Optimized for CPU inference.
Edge / mobile	INT8 PTQ (TensorRT)	8	Supported on many mobile NPUs. Proven for smaller models.

Contents