How to Fine-Tune an LLM with LoRA and QLoRA, Step by Step
Fine-tune a pretrained language model in ten stages, from choosing a base model and formatting chat data to LoRA, QLoRA, evaluation, merging and DPO, with a lesson and a coding problem for each part.
How do you fine-tune an LLM?
You fine-tune an LLM by choosing a pretrained base model, building a dataset of example conversations for one task, and training the model on those examples with the next-token loss, usually through small LoRA adapters while the original weights stay frozen. You then compare the fine-tuned model with the base model on held-out examples, and either keep the adapter as a separate file or merge it into the weights for serving. TensorTonic has a lesson and a tested coding problem for each part of that process, and a notebook project where you fine-tune Qwen3-4B with QLoRA from start to finish.
This guide follows the order of a real fine-tuning run in ten stages. Each stage has a short explanation and a table that pairs every topic with a page to read and a problem or project to practise it. If you are preparing for interviews on the same material, the fine-tuning and alignment sections of the LLM interview questions guide give short model answers on supervised fine-tuning, LoRA, QLoRA, RLHF, DPO and GRPO, so this guide does not repeat them.
When should you fine-tune an LLM instead of prompting or using RAG?
You should fine-tune an LLM when you need the same behaviour across many requests, such as a fixed output format, a consistent tone or better accuracy on one repeated task, and a careful prompt has already fallen short. Retrieval-augmented generation is the better choice when the model needs knowledge it does not have, because documents can be updated without retraining. Prompting is the cheapest option of the three and should always be tried first.
| Approach | Use it when | What you gain |
|---|---|---|
| Better prompting | The model can already do the task when the instructions and a few examples are clear | Nothing to train, and changes take effect immediately |
| RAG | The model needs knowledge it does not have, or knowledge that changes often | Documents can be updated without retraining, and answers can cite their sources |
| Fine-tuning | The model needs a consistent format, tone or skill that prompting does not deliver reliably | The behaviour is built into the weights, so prompts can be shorter |
The approaches are often combined, with fine-tuning setting how the model answers and retrieval supplying what it answers from. The guide on how to build a RAG system from scratch covers the retrieval side, from chunking documents to evaluating answers.
What do you need before you fine-tune an LLM?
You need comfortable Python, a working knowledge of PyTorch, an understanding of how a transformer predicts the next token, and access to a GPU for the training run itself. You do not need to have trained a model from scratch, although knowing how pretraining works makes every fine-tuning setting easier to reason about.
- For PyTorch, you should be able to load a model, run a forward pass and read a training loop, which the guide on how to learn PyTorch covers.
- For transformers, you should know what attention, the residual stream and the output layer over the vocabulary do, and the guide on how to build an LLM from scratch walks through each component.
- For the loss, you should understand cross-entropy, since supervised fine-tuning uses the same next-token loss as pretraining.
- For hardware, you need one GPU with enough memory for the method you choose, and the section on fine-tuning without a big GPU explains how QLoRA lowers that requirement.
What are the steps to fine-tune an LLM?
The steps to fine-tune an LLM are choosing the base model and task, preparing the data, setting up the loss, choosing a training method and its settings, fitting the run into memory, training, evaluating, serving, and optionally tuning on preferences. The stages below follow that order, and the Fine-Tune Qwen3-4B for Medical Q&A project runs through most of them in one notebook.
Stage 1: How do you pick a base model and a task?
Pick a base model that is small enough to train on your hardware and already close to the task, and define the task as a clear input and the output you want for it. An instruct model that already follows chat turns is usually the better starting point for an assistant, while a base model without instruction tuning suits tasks where you want full control over the format. Write down how you will judge success before you train, because the same checking examples have to be scored before and after training.
| Topic | Read | Practice |
|---|---|---|
| What pretraining gives you | GPT Pretraining: How Next-Token Prediction Works | |
| Deciding whether to fine-tune | What Is LLM Fine-Tuning? | |
| Planning the training and checking examples | How to Fine-Tune an LLM | Fine-Tune Qwen3-4B for Medical Q&A |
The TensorTonic project uses Qwen3-4B, a model with about four billion parameters, and the task of answering medical questions concisely from the MedQuAD dataset. The model is small enough to fine-tune with QLoRA on a single GPU and large enough to give useful answers.
Stage 2: How do you build the dataset and format it with a chat template?
You build the dataset by collecting input and output pairs for the task, cleaning and filtering them, splitting off a held-out set the model never trains on, and rendering every example as a conversation with the model's own chat template. The chat template inserts the special tokens that mark where the system message, the user turn and the assistant turn begin and end. Training and inference must use the same template, and at inference you render only the system and user turns and add the generation prompt so that the model writes the assistant turn.
| Topic | Read | Practice |
|---|---|---|
| Removing duplicate examples | Perform Stable Exact Deduplication | |
| How text becomes tokens | Tokenization in the Transformer | Apply Ranked BPE Merges |
| Padding examples into a batch | Pad Sequences | |
| Rendering conversations with a chat template | Fine-Tune Qwen3-4B for Medical Q&A |
In the project you inspect the MedQuAD columns, calculate the median answer length, keep answers between 20 and 160 words, and build a training set of 96 examples and a separate evaluation set of 8. You then write each example as a system, user and assistant message and render it with the tokenizer's chat template, with Qwen3 thinking mode turned off.
Stage 3: What loss does supervised fine-tuning use, and why mask the prompt tokens?
Supervised fine-tuning uses the same next-token cross-entropy loss as pretraining, computed on the rendered conversation with the targets shifted by one position. The loss is often restricted to the assistant tokens by giving every prompt and system token an ignore label, so that the model learns to produce answers and is not trained to reproduce the questions. Masking matters most when prompts are long or repeated across many examples, because those tokens would otherwise make up a large share of the loss.
| Topic | Read | Practice |
|---|---|---|
| Next-token cross-entropy | Cross-Entropy Loss: Formula and BCE vs MSE | Cross-Entropy Loss |
| Masking prompt tokens out of the loss | Build an Assistant-Only SFT Loss Mask | |
| Scoring a response given its prompt | Perplexity in LLMs: Formula and Worked Example | Compute Conditional Perplexity |
The masking problem is part of the post-training section of the Build LLM from Scratch study plan. The project trains on the full rendered conversation to keep the setup short, so solving the masking problem is a good way to see what changes when only the answer counts.
Stage 4: Should you use full fine-tuning, LoRA or QLoRA?
Most people should use LoRA, or QLoRA when GPU memory is tight, because both train only a small fraction of the parameters and reach similar quality to full fine-tuning on many tasks. Full fine-tuning updates every weight, which needs memory for the weights, their gradients and two AdamW states for every parameter. LoRA freezes the base model and trains two small matrices beside selected layers, and QLoRA does the same with the frozen base model stored in 4 bits.
| Method | What trains | Base weights stored as | Rough memory for a 4B model |
|---|---|---|---|
| Full fine-tuning | Every weight | 16-bit, plus fp32 master weights and two AdamW states | About 16 bytes per parameter, or roughly 64 GB before activations |
| LoRA | Small adapter matrices only | 16-bit and frozen | About 2 bytes per parameter, or roughly 8 GB, plus the adapter |
| QLoRA | Small adapter matrices only | 4-bit NF4 and frozen | About half a byte per parameter, or roughly 2 GB plus scales, plus the adapter |
The memory figures above count weights, gradients and optimizer states only, and activations add more on top depending on the batch size and sequence length. The topics below explain where each byte goes.
| Topic | Read | Practice |
|---|---|---|
| Where training memory goes | Mixed Precision Training: FP16 vs BF16 | Mixed-Precision Training Memory Accountant |
| LoRA adapters | LoRA in LLMs: Beginner's Guide to Fine-Tuning | |
| QLoRA on a 4-bit base model | QLoRA Explained: 4-Bit LLM Fine-Tuning | |
| Saving and reusing adapters | LoRA and QLoRA | Fine-Tune Qwen3-4B for Medical Q&A |
Stage 5: How do you choose the LoRA rank, alpha and target modules?
Start with a rank of 8 or 16, set alpha to twice the rank, and attach adapters to all the linear layers in the attention and feed-forward blocks. LoRA replaces the output of a frozen layer W with Wx plus alpha divided by r times BAx, where A maps the input down to r dimensions and B maps it back up. B starts at zero, so the model begins training exactly where the base model is, and the alpha over r factor keeps the size of the update similar when you change the rank.
| Topic | Read | Practice |
|---|---|---|
| Matrix rank | Singular value decomposition | |
| Rank, alpha and initialization | LoRA in LLMs: Beginner's Guide to Fine-Tuning | |
| Choosing target modules | LoRA in LLMs: Beginner's Guide to Fine-Tuning | Fine-Tune Qwen3-4B for Medical Q&A |
The rank sets how many trainable parameters each adapter has. For a 4096 by 4096 weight matrix, a rank of 8 gives two adapter matrices with 65,536 parameters in total, about 0.39% of a full update. Adapting every linear layer usually works better than adapting only the query and value projections, and in the project you set a small dropout and the all-linear target in the LoRA configuration, with rank 4, 8 or 16 to choose from in the settings panel.
Stage 6: How do quantization, mixed precision and gradient accumulation reduce memory?
Quantization shrinks the frozen weights, mixed precision shrinks the activations and the arithmetic, and gradient accumulation lets a small batch stand in for a large one. QLoRA stores the base model in NF4, a 4-bit format whose levels are spaced to fit normally distributed weights, and double quantization also compresses the scale stored for each block of weights. Training then runs in bf16, which has the same range as fp32 and needs no loss scaling, and gradient accumulation sums gradients over several small micro-batches before each optimizer step.
| Topic | Read | Practice |
|---|---|---|
| 4-bit and 8-bit weights | LLM Quantization Explained: INT8, INT4, GPTQ and AWQ | Symmetric INT8 Quantization, Group-Wise INT4 Quantization |
| NF4 and double quantization | QLoRA Explained: 4-Bit LLM Fine-Tuning | Quantize a Language Model to 4 Bits |
| BF16 and FP16 | Mixed Precision Training: FP16 vs BF16 | Mixed Precision with Loss Scaling |
| Gradient accumulation | Gradient Accumulation in LLM Training | Gradient Accumulation Equivalence, Gradient Accumulation |
| Optimizer state memory across GPUs | Account for ZeRO Memory by Stage |
Gradient checkpointing is the other common saving. It stores only some activations during the forward pass and recomputes the rest during the backward pass, which trades extra compute for memory. The project combines all of these, loading Qwen3-4B with NF4, bf16 compute and double quantization, and training with a batch of 2, gradient accumulation over 4 steps for an effective batch of 8, and gradient checkpointing turned on.
Stage 7: Which training settings matter, and how do you avoid overfitting?
The settings that matter most are the learning rate, the number of epochs, the effective batch size and the maximum sequence length. LoRA usually trains with a learning rate around 1e-4 to 3e-4, which is higher than the rate used for full fine-tuning because only the small adapters change. On a small dataset one to three epochs is typical, and the main sign of overfitting is a held-out score that falls while the training loss keeps going down.
| Topic | Read | Practice |
|---|---|---|
| AdamW | Which optimizer should you use in deep learning? | AdamW (Decoupled Weight Decay) |
| Warmup and decay | Warmup + Linear Decay LR Schedule, Cosine Annealing LR Scheduler | |
| Gradient clipping | Gradient Clipping (Global Norm) | |
| Reading the loss curve | Why is my training loss not decreasing? |
Set the maximum sequence length from the token lengths of your examples, since examples longer than the limit are cut off and lose the end of their answers. In the project you can choose a learning rate of 1e-4, 2e-4 or 3e-4 and one or two epochs, and the training run uses a maximum length of 384 tokens with the loss logged at every step.
Stage 8: How do you evaluate a fine-tuned adapter?
You evaluate a fine-tuned adapter by generating answers for the held-out examples with both the base model and the fine-tuned model, using the same prompt and the same decoding settings, and scoring both with the same metric. A score change on its own says little, so read the weakest answers to see what the model still gets wrong. Also check a few general tasks the base model handled well, because fine-tuning on a narrow task can reduce ability elsewhere.
| Topic | Read | Practice |
|---|---|---|
| Choosing a metric for the task | Which evaluation metric fits which machine learning problem? | Fine-Tune Qwen3-4B for Medical Q&A |
| Judging open-ended answers | Evaluating generated answers | Fit Elo Ratings from Pairwise Votes, Remove Response-Length Bias from Judge Scores |
| Checking general ability with a benchmark | Score Multiple-Choice Continuations |
The project scores answers with token F1, the overlap of lowercase words between the generated answer and the reference, using greedy decoding so that runs are comparable. You implement the metric, measure the base model before training, measure the fine-tuned model afterwards, compute the score change and pull out the two weakest answers to inspect.
Stage 9: How do you merge a LoRA adapter and serve the model?
You merge a LoRA adapter by adding alpha over r times BA into each frozen weight matrix, which gives a model with exactly the original shape and no extra latency. Keeping the adapter separate is the other option, since an adapter file is far smaller than the model and one server can switch between several adapters on the same base model. For a QLoRA adapter, load the base model in 16 bits before merging, because adding the update into 4-bit weights rounds part of it away.
| Topic | Read | Practice |
|---|---|---|
| Merging an adapter into the weights | LoRA in LLMs: Beginner's Guide to Fine-Tuning | |
| Fast generation | KV Cache in LLMs: How It Makes Inference Fast | Build a Qwen3 Inference Server |
| Quantizing the model for inference | LLM Quantization Explained: INT8, INT4, GPTQ and AWQ | Quantize a Language Model to 4 Bits |
After merging, the model is served like any other, and you can quantize it again for inference if memory or speed matters. The guide to inference engineering covers serving in depth, from the KV cache to batching.
Stage 10: When should you add preference tuning with DPO?
Add preference tuning with DPO when the supervised model gives acceptable answers but you want it to prefer one kind of answer over another, such as more polite, more concise or safer replies. DPO trains on pairs of a chosen and a rejected response to the same prompt and raises the likelihood of the chosen one relative to a frozen reference model, with no separate reward model. The beta setting controls how far the model may move from the reference, and GRPO is the usual alternative when a program can check whether an answer is correct.
| Topic | Read | Practice |
|---|---|---|
| Reward models and RLHF | RLHF Explained: Reward Models and PPO | Bradley-Terry Reward Model Loss |
| Preference pairs | Canonicalize Pairwise Preference Data | |
| The DPO loss | DPO: Direct Preference Optimization Explained | Implement the DPO Loss |
| Staying close to the reference model | Score a KL-Regularized Policy Batch | |
| GRPO with checkable rewards | GRPO: Group Relative Policy Optimization | Normalize GRPO Group Advantages, Compute a Clipped GRPO Surrogate |
| A full DPO run | Align a Language Model with DPO |
The Align a Language Model with DPO project trains a rude help desk model into a polite one and measures how far it drifts when beta is too small. To test the same material under time pressure, the Post-Training and Alignment assessment asks you to implement PPO, DPO, GRPO and RLHF objectives in 45 minutes across 3 problems.
Can you fine-tune an LLM without a big GPU?
Yes, you can fine-tune an LLM without a big GPU by combining QLoRA, a small model, short sequences, a small batch with gradient accumulation and gradient checkpointing. With 4-bit base weights, a model with a few billion parameters fits on a single GPU with far less memory than full fine-tuning would need. The run will be slower than on a large GPU, and that is acceptable when the dataset is small.
Every coding problem linked in this guide runs in the browser on small arrays, so the loss mask, quantization, gradient accumulation and DPO loss need no GPU at all. The fine-tuning notebook runs in a managed GPU session after you sign in, and the guide on how to practise ML with no GPU lists other places to find compute.
What mistakes do people make when fine-tuning an LLM?
The most common mistakes in fine-tuning are using it for the wrong problem, mismatched chat templates, evaluating on data the model trained on, and training for too long on a small dataset. The list below gives each mistake with the check that prevents it.
- Fine-tuning before trying a careful prompt wastes a training run on behaviour that the base model could already produce.
- Using fine-tuning to add facts that change often gives a model that repeats old answers, and retrieval handles that job better.
- Formatting training examples with a different chat template from the one used at inference shows the model inputs it was never trained on.
- Leaving the prompt tokens in the loss spends training signal on reproducing questions, so mask them when the prompts are long or repetitive.
- Evaluating on examples that also appear in the training set reports memorization as improvement, so remove duplicates before you split.
- Comparing the base and fine-tuned models with different prompts or different decoding settings makes the score change meaningless.
- Training too many epochs on a small dataset makes the held-out score fall while the training loss keeps going down.
- Merging an adapter directly into 4-bit weights loses precision, so dequantize the base model to 16 bits before merging.
- Loading an adapter on a different base model or a different revision of it gives broken output, because the adapter only fits the weights it was trained on.
Practice this
- Build an Assistant-Only SFT Loss Mask makes you decide which tokens of a chat the supervised fine-tuning loss should train on.
- Group-Wise INT4 Quantization shows how a frozen base model is stored in 4 bits with a scale for each group of weights, which is the idea QLoRA builds on.
- Implement the DPO Loss is the preference loss you add once supervised fine-tuning works.
Then run a full fine-tune in the Fine-Tune Qwen3-4B for Medical Q&A project, or start with the short lessons in the Fine-Tune an LLM track.
What is the difference between LoRA and QLoRA?
LoRA and QLoRA train the same small low-rank adapters, and they differ in how the frozen base model is stored. LoRA keeps the base weights in 16-bit precision, while QLoRA stores them in a 4-bit format called NF4 and dequantizes each layer to 16 bits only when it is used. QLoRA therefore needs roughly a quarter of the memory for the base weights, at the cost of some extra computation and a small loss of precision in the frozen model.
How much data do you need to fine-tune an LLM?
The amount of data you need depends on how far the task is from what the base model already does, and clean, consistent examples matter more than a large number of them. A narrow change of format or tone can show up after a small set of examples, while a new skill needs many more. A practical approach is to start with a small, carefully checked dataset, measure the result on held-out examples, and add data where the model still fails.
Does fine-tuning teach an LLM new facts?
Fine-tuning is an unreliable way to teach a model new facts, and it works best for changing behaviour such as format, tone or accuracy on a repeated task. A model fine-tuned on documents may repeat their wording without being able to use the facts reliably, and it cannot cite where an answer came from. Retrieval-augmented generation is the better tool for knowledge that is large or changes often, and many systems fine-tune for behaviour and retrieve for knowledge.
What LoRA rank and alpha should you use?
A rank of 8 or 16 with alpha set to twice the rank is a common starting point for LoRA fine-tuning. The adapter update is scaled by alpha divided by the rank, so keeping that ratio fixed keeps the size of the update similar when you change the rank. Raise the rank when the task is far from what the base model already does and the held-out score stops improving, and lower it when the model overfits a small dataset.
Is fine-tuning the same as RLHF or DPO?
Supervised fine-tuning, RLHF and DPO are separate stages of post-training. Supervised fine-tuning trains the model on example responses with the next-token loss, while RLHF and DPO train it on comparisons between a preferred and a rejected response. Preference tuning usually starts from a model that has already been supervised fine-tuned, and both stages can be done with LoRA adapters.