QLoRA in LLMs

The Problem QLoRA Solves

LoRA keeps the original model weights frozen. Here, frozen only means that training does not change them. The weights do not disappear, because the model still needs them to understand the input and produce an answer. They must remain in memory during every training step.

Consider a model with 7 billion weights. When each weight uses 16 bits, or 2 bytes, the original model weights alone need about 14 GB of memory. This estimate does not yet include the LoRA adapter, training data being processed, or other temporary memory. QLoRA focuses on making this large frozen part smaller.

1. Bits, Bytes, and Number Formats

A computer stores numbers as patterns of zeros and ones. Each position in that pattern is a bit, and eight bits together make one byte. A number stored with 8 bits occupies 1 byte; a number stored with 16 bits occupies 2 bytes; a number stored with 32 bits occupies 4 bytes.

Each extra bit doubles the number of possible patterns. One bit gives us two patterns, two bits give us four, and eight bits give us 256. A number format determines what those patterns mean, including whether they represent whole numbers or values with a fractional part.

Integers: INT8 and UINT8

An integer is a whole number, such as 12, 0, or -45. The name INT8 means an integer stored using eight bits. Signed INT8 can represent every whole number from -128 to 127. The unsigned version, UINT8, uses the same amount of space but represents values from 0 to 255.

Both formats have 256 possible values. The difference is how they interpret the bits: UINT8 uses all of its range for nonnegative numbers, while INT8 includes negative numbers. For example, the pattern 11111111 means 255 as UINT8 and -1 as INT8.

Signed integers commonly use two's complement. To represent -45 in eight bits, start with the binary pattern for +45, flip every zero and one, then add one. This representation lets the same binary addition rules work for positive and negative integers.

Integer formats also come in larger sizes, including INT16 and INT32. More bits allow a wider range of whole numbers, but an integer by itself still cannot express a weight such as 0.127. To store fractional values directly, we use floating point.

Floating point: sign, exponent, and mantissa

Scientific notation separates a number into significant digits and a power that sets its size. For example, 625 can be written as 6.25 × 10². Binary floating point uses the same principle with powers of two, which allows a fixed number of bits to represent both small fractions and large values.

Each floating-point number has three fields. The sign uses one bit: 0 for positive and 1 for negative. The exponent sets the power of two and controls the numerical range. The fraction field, often called the mantissa, stores the detail within that range.

For a normal binary floating-point number, the significant digits begin with 1. That leading 1 is implied, so only the digits after the binary point need to be stored. The leading 1 together with those fraction bits is the significand. More fraction bits let the format distinguish values that are closer together.

The exponent also has a storage convention called a bias, an offset added before storing it. FP32 uses a bias of 127, so an actual exponent of 2 is stored as 129. Subtracting the bias when reading the number recovers the actual exponent and allows the field to cover both positive and negative powers.

FP32, FP16, BF16, and FP8

These formats divide their available bits differently. FP32 has room for a wide range and fine detail. FP16 uses half as much storage, with fewer bits for both the exponent and fraction. BF16 also uses 16 bits, but allocates more of them to the exponent.

FP16 keeps 10 fraction bits, compared with 7 in BF16, so it can preserve finer differences between nearby values within its range. BF16 has a wider range because its exponent uses 8 bits instead of 5. During training, that range helps accommodate calculations that produce very large or very small numbers.

FP8 takes the same idea down to one byte. E4M3 has 4 exponent bits and 3 fraction bits, while E5M2 uses 5 and 2. FP8 and INT8 occupy the same space, but interpret it differently: FP8 has an exponent and a fraction, while INT8 represents whole numbers.

The storage cost adds up quickly across a model. Seven billion weights require about 28 GB in FP32 or 14 GB in FP16 or BF16, before any other memory is counted. Representing those weights with one byte each would reduce their raw storage to 7 GB. The next step is to understand how an integer can represent an approximate fractional weight.

2. How Quantization Works

A weight such as 0.314 fits naturally in a floating-point format. INT8 only stores whole numbers, so rounding it directly to 0 would lose too much information. Quantization first changes the scale of the numbers, allowing the available integers to represent much smaller steps.

For example, if one integer step represents 0.01, the stored integer 31 stands for a weight of 0.31. The integer takes one byte, and the scale tells us how to interpret it. This is how integer quantization can represent fractional weights while reducing their storage.

Choose a scale, then round

Suppose a group of weights ranges from -1.27 to 1.27. We can map that range to the integers -127 through 127, making each integer step worth 1.27 ÷ 127 = 0.01. This value is the scale, shared by all the weights in the group.

To store a weight, divide it by 0.01 and round to the nearest integer. To read it back, multiply the stored integer by 0.01. Follow a weight of -0.826 through those operations:

Multiplying the integer by its scale is called dequantization. The recovered value is -0.83, which differs from -0.826 by 0.004. That difference is the quantization error introduced when we rounded -82.6 to -83.

This example uses symmetric quantization: the chosen range extends equally in both directions from zero. INT8 supports -128 too, but this symmetric mapping leaves that value unused. The scale is the largest absolute weight divided by 127.

An asymmetric mapping also uses an integer offset called a zero point, which specifies where a weight of zero lands. This can use the integer range more effectively when the weights are unevenly distributed around zero. Both approaches share the same idea: a scale and a rounding step connect the original weights to the stored integers.

What determines the rounding error?

The scale sets the distance between values that can be recovered. With a scale of 0.01, the integers 30, 31, and 32 represent 0.30, 0.31, and 0.32. Values between those points must be rounded. A smaller step preserves more detail, provided the largest and smallest weights still fit within the available integer range.

A few unusually large weights can make that difficult. If the largest absolute weight grows from 1.27 to 12.7, the scale becomes 0.1, and the spacing between recoverable values becomes ten times larger. That loses more detail among the small weights. Such unusually large values are called outliers.

One way to limit that effect is to divide the weights into smaller groups and give each group its own scale. A large value then affects fewer weights. This balances two costs: smaller groups can reduce rounding error, while more groups require more scales to be stored. These choices become especially important as we reduce the number of bits further.

3. How LoRA and Quantization Fit Together

QLoRA applies quantization directly to the pretrained base model using four bits per quantized weight. Two codes fit into one byte, reducing raw weight storage to half that of INT8 and one quarter that of FP16. This is why four bits matters for QLoRA: it lets the frozen base model occupy substantially less memory during fine-tuning.

With only sixteen possible codes, the choice of values becomes more important. QLoRA uses a format called NormalFloat 4, or NF4. Each code selects a value from a shared list of sixteen entries called a codebook. A weight matrix is divided into small groups called blocks, and a scale for each block adjusts those values to the local weight range. Section 5 explains how NF4 chooses its sixteen entries.

The base weights stay frozen in this compact form. When a layer needs them for a calculation, they are dequantized into a format such as BF16. Alongside this base calculation, LoRA trains a small adapter that learns from the fine-tuning examples. The layer adds the base result and the adapter result together.

For an input, the base model produces its usual result using the reconstructed weights. The LoRA adapter produces a small learned correction, and the layer adds the two results together. Only the adapter is changed by training.

Here, $x$ is the input to the layer and $W_q$ is the quantized base weight. The matrices $A$ and $B$ form the trainable LoRA adapter, while $\alpha / r$ controls the strength of its correction.

The first term is the frozen base calculation and the second term is the trainable LoRA correction. This is the meaning of Quantized Low-Rank Adaptation: quantization reduces the memory used by the base model, while low-rank adaptation keeps the trainable part small.

4. One QLoRA Training Step, End to End

A training example still passes through the entire Transformer and produces the usual task loss. Inside each adapted linear layer, the frozen base path and the trainable adapter path handle the same hidden state differently. Use the two views below to follow the forward and backward calculations.

In the forward pass, a quantized base block is reconstructed only for computation. The resulting base projection is added to the LoRA correction. In the backward pass, gradients travel through the calculation so the adapter can learn, but the optimizer holds and updates state only for parameters marked as trainable.

The quantized base model is therefore part of the differentiable computation without becoming a trainable 4-bit model. QLoRA does not repeatedly round newly updated base weights because those base weights never receive optimizer updates.

5. NF4: A 4-Bit Format Designed for Model Weights

Four bits provide sixteen codes, but a quantizer still has to decide which numerical values those codes represent. Uniform 4-bit quantization spreads them evenly. This is not always the best use of a small codebook because pretrained weights within a normalized block are often concentrated near zero, with fewer values in the tails.

NormalFloat 4, usually shortened to NF4, places its representable values according to quantiles of a standard normal distribution. Under the normally distributed weight assumption in the QLoRA paper, this gives the dense part of the distribution more resolution and is information-theoretically optimal for the stated setting.

NF4 does not add more codes. It uses the same sixteen possibilities more deliberately. The QLoRA experiments found NF4 more accurate than FP4 and INT4 for the studied 4-bit fine-tuning configurations, which is why NF4 is the usual bitsandbytes choice for QLoRA training.

6. Double Quantization Compresses the Scales

Blockwise quantization needs a scale for every block. The weight codes are small, but millions of blocks can make their scale values noticeable. If each 64-weight block stores one 32-bit scale, those scales add half a bit per model parameter.

QLoRA calls this double quantization. The second stage quantizes the first stage's constants and stores a smaller set of higher-level constants for reconstruction. The original paper reported an average reduction from about 0.50 to 0.127 bits per parameter for the quantization constants, saving roughly 0.37 bits per parameter.

Double quantization is sometimes called nested quantization in libraries. It does not quantize the LoRA adapter twice, and it does not change the four-bit code assigned to each base weight. Its target is the metadata needed to interpret those codes.

7. Paged Optimizers Manage Temporary Memory Spikes

A training run can fit during ordinary steps and still fail when memory briefly rises. Long sequences, saved activations, and gradient checkpointing can create peaks that exceed the GPU's remaining capacity even though the average footprint is acceptable.

Paged optimizers use NVIDIA unified memory so optimizer pages can move between GPU and CPU memory when pressure rises. This gives the runtime a way to survive short peaks instead of requiring the GPU to reserve enough space for the worst moment throughout the run.

Paging is a safety mechanism rather than free memory. Moving pages has a transfer cost, and sustained over-allocation can still make training slow or fail. The practical goal is to handle brief spikes after the base model and adapter state have already been reduced.

8. Account for QLoRA Memory Correctly

The phrase “4-bit training” can suggest that the complete training process occupies one quarter of a 16-bit run. QLoRA applies 4-bit storage to the frozen base weights, while several other allocations remain. A realistic estimate keeps these categories separate.

For $P$ parameters, raw 16-bit base weights occupy roughly $2P$ bytes. Raw 4-bit codes occupy roughly $0.5P$ bytes before quantization metadata. This is close to a fourfold reduction for the weight values, not a promise of fourfold lower end-to-end training memory.

Batch size, sequence length, model width, activation checkpointing, optimizer choice, attention implementation, and device placement can each change the final peak. Measure the actual configuration rather than choosing a model only from its parameter count.

9. Choosing the QLoRA Configuration

A QLoRA setup combines choices from quantization, LoRA, and the training system. Each control has a different job, so tuning becomes clearer when the controls are grouped by what they affect.

10. Quality, Memory, and Speed Are Separate Questions

QLoRA's direct benefit is a smaller memory footprint for the frozen base model. Quantization is lossy, so the base computation uses approximated weights. The trainable adapters can compensate for task-specific behavior, but they do not reconstruct every original weight exactly.

In the original paper's 7B-to-65B instruction-tuning comparison, NF4 with double quantization recovered the tested 16-bit LoRA performance. Separate experiments on smaller RoBERTa and T5 models found 4-bit adapter tuning matched the full 16-bit baseline. These are results for particular models, datasets, tasks, and evaluations rather than guarantees for every application.

QLoRA also does not fix unrelated training problems. Poor examples, incorrect labels, unsuitable loss masking, unstable hyperparameters, or weak evaluation remain poor regardless of how efficiently the base model is stored.

11. Saving, Loading, and Merging QLoRA Adapters

A QLoRA training result is normally saved as a LoRA adapter rather than another full model. The checkpoint contains the learned adapter tensors and configuration, while the frozen base model remains a separate dependency.

Keeping the adapter separate allows several tasks to share one quantized base model. At load time, the same base and compatible quantization configuration must be reconstructed before attaching the adapter, otherwise the numerical reference that training used may differ.

Merging requires additional care. A common path dequantizes the base into a wider precision, adds the LoRA update, and optionally quantizes the merged result again. That process needs enough memory for the wider model and can introduce another round of quantization error, so merged and unmerged outputs should be compared before deployment.

12. Minimal QLoRA Setup with Hugging Face

This example shows the boundary between the four-bit base model and the trainable adapter. It uses the current Transformers quantization configuration with PEFT's k-bit preparation and all-linear target shortcut. The rank, alpha, dropout, model data type, and target coverage still need to be selected for the actual model and task.

import torch
from peft import (
    LoraConfig,
    get_peft_model,
    prepare_model_for_kbit_training,
)
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

model_id = "your-base-model"

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=quantization_config,
    dtype=torch.bfloat16,
)
model = prepare_model_for_kbit_training(model)

lora_config = LoraConfig(
    task_type="CAUSAL_LM",
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules="all-linear",
    bias="none",
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()