LoRA in LLMs

The Change Hidden Inside Fine-Tuning

Fine-tuning begins with a model whose weights have already learned broad patterns from language. Training on a narrower dataset changes those weights so the model becomes better at a particular task, domain, or style.

Pick any weight matrix inside the model. Its fine-tuned version can be understood as the original pretrained matrix plus a change learned from the new data. Full fine-tuning stores the combined result as another complete matrix. LoRA keeps the original matrix fixed and learns only the change, using a much smaller representation.

1. From Training Examples to a Weight Update

A fine-tuning example pairs an input with the answer the model should produce. The model makes a prediction, the training loss measures how far that prediction is from the desired answer, and backpropagation calculates which trainable weights should change.

Full fine-tuning allows that update to flow into every model weight. This gives training broad freedom, but it also requires gradient and optimizer state for the full model and produces another full-sized set of weights for the task.

LoRA changes only the destination of the update. The same examples, loss, and backpropagation are used, but the pretrained weights are marked as frozen. Gradients update the small LoRA parameters instead, so the task is stored as an adapter rather than another copy of the model.

2. The Transformer Layers LoRA Can Change

A Transformer carries each token as a list of learned features called a hidden state. Linear layers transform those features using weight matrices. They appear throughout every Transformer block, especially inside attention and the feed-forward network.

Attention uses query, key, value, and output projections to gather information from other tokens. The feed-forward network, or FFN, then transforms each token independently: it expands the hidden state into a wider feature space, applies a gated nonlinearity, and reduces it to the original model width. In many LLMs, these FFN projections contain a large share of the block's parameters.

LoRA can be attached to selected projections in attention, the FFN, or both. It adds a parallel weight update to those linear layers. The surrounding attention calculation, activation functions, normalization, and residual paths remain unchanged.

3. Matrix Rank and the LoRA Path

Before following the LoRA path, three matrix properties are useful. A matrix is a rectangular table of numbers. Its shape tells us how many input features it receives and how many output features it produces. A weight with shape $d_{\text{out}} \times d_{\text{in}}$ maps a hidden state with $d_{\text{in}}$ features to one with $d_{\text{out}}$ features.

The rank of a matrix describes how many independent directions its transformation can express. A full-rank matrix can use every available direction. A low-rank matrix is more restricted because some rows or columns can be produced from combinations of others. LoRA deliberately places this restriction on the task update, while leaving the pretrained matrix untouched.

Notice that $BA$ has the same outer shape as the pretrained weight, so it can be added to that weight. Its rank cannot exceed the narrow inner dimension $r$. This is the mathematical reason the two smaller matrices form a low-rank update.

An adapted linear layer computes two results from the same hidden state. The pretrained weight produces the result it would have produced before fine-tuning. Beside it, the LoRA adapter produces a correction learned from the new data. Adding the correction changes the layer's behavior without changing the pretrained weight itself.

The adapter creates its correction with two matrices, named $A$ and $B$. The first reduces the hidden state to a narrow intermediate width. The second restores the original output width. Select the merged view below to see how this separate path can disappear after training.

A LoRA adapter is therefore not a second language model. It is the collection of these small update matrices across the selected layers. Several adapters can share the same frozen base model because each one stores only its own corrections.

4. Why the Update Is Low-Rank

Consider a linear layer with 4,096 input features and 4,096 output features. A freely trainable update for that layer would contain 4,096 × 4,096 values, or 16,777,216 parameters. LoRA sends the update through a much narrower space instead.

The middle width is the LoRA rank, written as $r$. With rank 8, the two adapter matrices contain 65,536 parameters for this layer, about 0.39% as many as a full update matrix.

Written compactly, the task update is the product of the two trainable matrices:

For an input hidden state $x$, the frozen matrix $W_0$ produces the base result. The scaled LoRA correction is added beside it:

Interactive: How Rank Changes the Parameter Count

Select a typical Transformer projection and move the rank slider. The calculation covers one adapted weight matrix, making the relationship between matrix width, rank, and trainable parameter count explicit.

5. How LoRA Training Works

The task data, forward loss, backpropagation, and optimizer loop remain familiar. The important difference is which parameters are allowed to receive gradients and optimizer updates.

Freezing a weight means the optimizer does not change it. The frozen matrix still participates in the forward pass and in computing gradients for earlier activations, so the base model remains part of training computation.

6. Initialization, Scaling, and the First Update

In the original LoRA formulation, $A$ is initialized with small random values and $B$ is initialized to zero. The initial adapter product is therefore zero:

Fine-tuning begins with exactly the base layer's behavior rather than a random perturbation. On the first backward pass, $B$ can receive a gradient because $Ax$ is generally nonzero. The gradient for $A$ initially passes through zero-valued $B$, so $A$ starts changing after $B$ moves away from zero.

The scaling factor $\alpha/r$ separates the adapter's effective strength from the raw matrix values. LoRA dropout, when configured, is applied on the adapter path during training and disabled during evaluation.

7. Choosing Which Linear Layers to Adapt

LoRA is a method for adapting matrices, not a fixed list of layer names. The original paper studied attention projections extensively and commonly adapted query and value weights. Current model families use different module names and may place adapters on additional attention and FFN projections.

Adapting more module types increases the adapter's capacity and training cost. Adapting fewer modules creates a smaller checkpoint but imposes a tighter constraint. There is no architecture-independent target list because names, tensor shapes, grouped projections, and empirical behavior differ across model families.

A reliable configuration begins with the model implementation's actual named modules and a recipe validated for that architecture. Parameter counts should be inspected after adapter injection to confirm that only the intended weights are trainable.

8. Choosing Rank and Alpha

Rank controls the maximum dimensionality of each learned update. Alpha controls its scale. They affect different parts of the adapter and should be recorded as separate hyperparameters.

A larger rank is not automatically better. It can improve expressiveness while increasing memory, checkpoint size, and optimization freedom. Rank, learning rate, alpha, dropout, target modules, and dataset quality interact, so comparisons should change one controlled configuration at a time and evaluate on held-out data.

9. What a LoRA Checkpoint Must Save

A LoRA checkpoint is more than two anonymous tensors. It must identify how those tensors attach to the base model and how their output is scaled.

Loading an adapter onto a different base revision can silently produce incorrect behavior even when tensor shapes happen to match. The adapter and its base model form one versioned artifact relationship.

10. Merging and Serving Multiple Adapters

After training, the scaled update has the same shape as the original matrix. A serving system can therefore merge it into a copy of the base weight:

Merging does not discard the need for the original adapter artifact if future unmerging, auditing, or reuse is required. Production systems commonly preserve immutable base weights and create derived merged weights separately.

11. Account for LoRA Memory Correctly

LoRA's largest savings concern trainable parameter state and task-specific storage. It does not make every part of fine-tuning or inference small.

The original LoRA paper reported large reductions for its studied models and configurations, including a GPT-3 example with far fewer trainable parameters and smaller task checkpoints. Those figures are paper-specific results rather than universal guarantees. The actual reduction depends on selected modules, rank, optimizer, precision, and training system.

12. Important LoRA Variants

Later methods retain the idea of parameter-efficient updates while changing how rank is allocated, how weights are decomposed, or how the frozen base is stored.

13. Minimal LoRA Linear Layer

This compact PyTorch-style implementation shows the tensor shapes, frozen base path, zero-output initialization, scaling, dropout, and merge operation. A production library also handles module replacement, adapter naming, mixed precision, distributed training, serialization, bias policies, and safe merge tracking.

class LoRALinear(nn.Module):
    def __init__(self, base, rank=8, alpha=16, dropout=0.05):
        super().__init__()
        self.base = base
        self.base.weight.requires_grad_(False)
        if self.base.bias is not None:
            self.base.bias.requires_grad_(False)

        d_out, d_in = self.base.weight.shape
        self.A = nn.Parameter(torch.empty(rank, d_in))
        self.B = nn.Parameter(torch.zeros(d_out, rank))
        nn.init.normal_(self.A, mean=0.0, std=0.02)

        self.scale = alpha / rank
        self.dropout = nn.Dropout(dropout)

    def forward(self, x):
        base_output = self.base(x)
        rank_features = F.linear(self.dropout(x), self.A)
        adapter_output = F.linear(rank_features, self.B)
        return base_output + self.scale * adapter_output

    @torch.no_grad()
    def merge_into_base(self):
        self.base.weight.add_(self.scale * (self.B @ self.A))

Primary Sources and Further Reading

The factorization, initialization, scaling, parameter-efficiency claims, merge behavior, and variant descriptions in this guide are grounded in the original research papers below.

• LoRA: Low-Rank Adaptation of Large Language Models, Hu et al. (2022)
• LoRA Conceptual Guide, Hugging Face PEFT documentation
• Parameter-Efficient Fine-Tuning, Hugging Face Transformers documentation
• AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning, Zhang et al. (2023)
• QLoRA: Efficient Finetuning of Quantized LLMs, Dettmers et al. (2023)
• DoRA: Weight-Decomposed Low-Rank Adaptation, Liu et al. (2024)