Direct Preference Optimization

1. Learning from Answer Comparisons

Suppose we ask a model to explain why the sky looks blue in one sentence. Both answers below address the topic, but only one gives the correct explanation. A reviewer can express that preference without assigning a numerical quality score to either answer.

In supervised fine-tuning, or SFT, we can train on the preferred answer as a demonstration of the response we want. A preference pair supplies an additional piece of information: another answer was considered and judged worse. DPO makes that comparison part of the objective, encouraging the preferred answer to gain ground relative to the rejected one.

The classic RLHF pipeline first uses comparisons to train a separate reward model, then updates the language model using that reward. The original DPO paper introduced a way to train directly from the comparisons, without fitting that separate reward model or running a PPO-style reinforcement-learning loop.

2. What Goes into a Preference Pair

A training example has three parts: the prompt, the chosen response, and the rejected response. Chosen and rejected are dataset labels. The model does not decide which answer is the winner while calculating the loss. That judgment has already been supplied by a human reviewer, an automated evaluation rule, or another model acting as a judge.

Both answers must be judged against the same context. For a conversation, that includes the earlier messages and any instructions that affect what a good answer should contain. If a prompt asks for one sentence, a detailed five-paragraph explanation may be accurate while still failing the instruction. The preference tells us which response better satisfies the stated criteria.

Rejected does not necessarily mean entirely wrong, and chosen does not mean perfect. It is a relative judgment between two responses. Reviewers may disagree, and automated judges can make mistakes. If a dataset consistently rewards confident wording over correct content, the learning signal will reflect that preference too.

In the standard offline setup covered here, these pairs are collected before optimization and reused across training batches. Generating candidates can be part of building the dataset, but the model does not need to generate a fresh response for each DPO update. Online variants that refresh the data introduce an additional collection process.

3. How a Model Scores an Existing Answer

A language model predicts probabilities for the next token, where a token is a piece of text. We can use those predictions to score an answer that is already written. Supply the prompt and answer to the model, then inspect the probability assigned to each answer token given the text that comes before it.

For example, consider a short response represented by three tokens. If their conditional probabilities are 0.5, 0.4, and 0.2, the probability of that entire token sequence is 0.5 × 0.4 × 0.2 = 0.04. Each probability uses a different prefix: the second token is scored after the first, and the third after the first two. Multiplying them accounts for the complete response.

Long sequences can have extremely small probabilities, so implementations use log probabilities. Taking a logarithm turns the product into a sum. The same example has a natural log probability of about −3.22. A value closer to zero means the sequence is more likely; the negative sign does not indicate a bad answer. Likelihood describes the model's predictions, while the preference label describes the judgment in the dataset.

Scoring provided tokens is called teacher forcing. Because the entire response is available, a causal Transformer can score its positions in parallel while masking future tokens. There is no need to sample the answer one token at a time. This uses the same next-token prediction machinery introduced in GPT pretraining.

4. Why DPO Keeps a Reference Model

Before preference training starts, the model already assigns different probabilities to different answers. DPO uses a reference model to preserve that starting point. A common setup begins with an instruction-tuned model, makes one copy trainable, and keeps the other fixed. The trainable model is also called the policy because it determines which tokens can be generated.

Both models score both answers, giving us four numbers. For the chosen answer, subtract its reference log probability from its current log probability. Repeat for the rejected answer. A positive difference means that answer has become more likely than it was under the reference; a negative difference means it has become less likely.

DPO then compares those two changes. The chosen change minus the rejected change is the reference-adjusted margin. A positive margin means the chosen answer has gained relative to its competitor after accounting for their starting likelihoods. The visual below keeps the reference fixed so that this distinction is visible.

The fixed reference supplies the likelihood baseline, while the dataset label supplies the preference judgment. It is also different from PPO's periodically refreshed rollout policy: in standard DPO, the reference stays fixed for the training run. Although it anchors the comparison, the loss does not impose a hard limit on how much every possible response probability may change.

5. Turning the Comparison into a Loss

A loss is a number that training tries to reduce. For DPO, it should be large when the rejected answer gains more than the chosen answer, and smaller when the chosen answer gains more. The standard loss turns the margin into a modeled preference probability, then penalizes low probability for the observed preference label.

First, multiply the margin by a positive setting called beta, written β. Then apply a sigmoid, a smooth function that maps any real number to a value between zero and one. A zero input gives 0.5, a positive input gives more than 0.5, and a negative input gives less than 0.5. Finally, take the negative logarithm of that value to obtain the loss.

When the trainable model and reference start with identical weights, the margin is zero for every pair. The loss is then about 0.693. That does not mean training is stuck: the curve has a slope at zero, so the optimizer can reduce the loss by increasing the chosen-versus-rejected margin.

6. What Beta Controls

In the visual, beta changes how strongly a given margin affects the preference probability. With a margin of 1.5, β = 1 produces a preference probability of about 82%, while β = 0.1 produces about 54%. The data and margin have not changed; we have changed the scale used by the loss.

Beta also has a role in DPO's derivation. The starting objective balances earning reward against moving away from a reference distribution. The movement penalty uses KL divergence, a measure of how two probability distributions differ. For a fixed reward function in that objective, a larger beta penalizes deviation more strongly, and its optimal policy stays closer to the reference.

That theoretical role should not be confused with the size of one gradient step. At zero margin, the magnitude of the loss slope is β / 2, so increasing beta actually makes that initial slope larger. The learning rate, data, and later margins also affect the trained model. Beta is not a universal knob that makes every optimizer update smaller or guarantees a particular KL divergence.

7. Follow One Training Step

A batch begins with stored preference pairs. Each prompt is combined with its chosen response and, separately, its rejected response. Apply the model's conversation format consistently, tokenize the sequences, and mark which positions belong to the response. Padding makes batches rectangular; attention masks prevent padding from acting as real context, while loss masks exclude prompt and padding targets from the response score.

Run the trainable model on these sequences and sum the response-token log probabilities. Run the frozen reference on the same sequences without recording gradients, or load its precomputed scores. Chosen and rejected sequences can be combined into one larger batch per model, so four scores per pair do not require four separate forward calls.

The loss uses those four scores to calculate a margin for each pair and averages the resulting losses. Backpropagation differentiates through both response scores from the trainable model. The optimizer updates that model's trainable weights, while the reference remains unchanged. The next batch repeats the process using the updated policy and the same reference.

A fixed reference makes score caching possible when the tokenized examples and scoring conventions also stay fixed. DPO can also update LoRA adapters rather than every model weight. The Hugging Face DPO trainer supports reference-score precomputation and adapter training. When sharing a base model, the reference configuration must reproduce the intended starting policy, including any adapters it already had.

8. DPO, SFT, PPO, and GRPO

These methods differ in the feedback they use and the work performed during training. SFT learns from demonstrated answers. DPO adds an explicit comparison with rejected answers. PPO and GRPO use rewards on sampled responses to form policy updates. In language-model training, SFT commonly provides the starting checkpoint for subsequent preference or reward-based optimization.

The comparison describes standard offline DPO and the PPO and GRPO setups covered in this series. Reward signals for the latter methods can come from learned models or task-specific checks. DPO avoids their rollout-and-reward loop during optimization, but still needs preference data, model scoring, and backpropagation. Its total cost depends on those choices and on how the data was obtained.

9. The Core Calculation in PyTorch

The code below separates response scoring from the preference loss. The first function aligns each prediction with the next token, gathers the log probability of that token, and sums only response positions. The second function calculates the reference-adjusted margin and uses a numerically stable log-sigmoid to obtain the loss.

import math
import torch
import torch.nn.functional as F


def response_logps(logits, labels):
    """Sum response-token log probabilities for a causal language model.

    logits: [batch, sequence, vocabulary]
    labels: [batch, sequence], with prompt and padding set to -100
    """
    targets = labels[:, 1:]
    mask = targets.ne(-100)
    if not mask.any(dim=-1).all().item():
        raise ValueError("Each example must contain a response token.")

    safe_targets = targets.masked_fill(~mask, 0)
    token_logps = F.log_softmax(logits[:, :-1].float(), dim=-1)
    selected = token_logps.gather(-1, safe_targets.unsqueeze(-1)).squeeze(-1)
    return selected.masked_fill(~mask, 0.0).sum(-1)


def dpo_loss(chosen, rejected, ref_chosen, ref_rejected, beta=0.1):
    """Mean sigmoid DPO loss from four [batch] log-probability tensors."""
    if not math.isfinite(beta) or beta <= 0:
        raise ValueError("beta must be positive and finite.")
    if chosen.ndim != 1 or chosen.numel() == 0:
        raise ValueError("Expected a nonempty batch of sequence scores.")
    if any(t.shape != chosen.shape for t in (rejected, ref_chosen, ref_rejected)):
        raise ValueError("All four score tensors must have the same shape.")

    chosen_change = chosen.float() - ref_chosen.detach().float()
    rejected_change = rejected.float() - ref_rejected.detach().float()
    margin = chosen_change - rejected_change
    return -F.logsigmoid(beta * margin).mean()


chosen = torch.tensor([-7.0], requires_grad=True)
rejected = torch.tensor([-6.5], requires_grad=True)
reference_chosen = torch.tensor([-8.0])
reference_rejected = torch.tensor([-6.0])

loss = dpo_loss(chosen, rejected, reference_chosen, reference_rejected, beta=1.0)
loss.backward()
print(round(loss.item(), 3))  # 0.201

In a real batch, construct labels by copying the input token IDs and replacing prompt and padding positions with −100. Pass an appropriate attention mask to each model before calling the scoring function. Include an end-of-sequence token as a response target when your training format uses one, and preserve enough context to predict the first response token.

Standard sigmoid DPO uses summed response log probabilities. Averaging each response by its token count changes the objective. The authors' implementation uses the summed form, shifts causal labels, and computes reference scores without gradients. Detaching the reference tensors in this small function provides an additional safeguard, but a real trainer should avoid constructing their gradient graph in the first place.

The printed loss uses the earlier illustrative scores and β = 1 for easy comparison. The scores are independent tensors in this demonstration; a real model obtains them through a shared neural network. This example checks the calculation, not dataset preparation, distributed training, or the memory efficiency of a complete trainer.

10. What Changes After Training

After DPO training, the updated language model generates responses in the usual way. It does not need a chosen-and-rejected pair for each new user request, and the reference model is not required for ordinary inference. If training used adapters, those learned adapters remain part of the deployed model configuration or are merged where supported.

What the model learns depends on the comparisons it sees. A dataset dominated by longer chosen answers may entangle verbosity with quality. Missing context, inconsistent labels, and truncation that removes the distinguishing part of an answer can all distort the training signal. A chosen answer's absolute probability can also fall while its relative margin improves, as the reference visual showed, so a lower pairwise loss alone does not establish better generated answers.

Check the trained model on held-out prompts that represent the intended use, including factual accuracy and instruction following. Preference optimization does not independently verify facts or guarantee a particular reasoning strategy. For a worked reasoning response, the quality of the final answer and the validity of its explanation still require their own checks. DPO supplies a way to learn from comparisons; the quality and coverage of those comparisons determine what guidance the model receives.