Compute the Direct Preference Optimization loss for preferred and rejected response pairs. For pair i, let ℓi+ and ℓi− be the policy's supplied sequence log probabilities, and ℓref,i+ and ℓref,i− the reference model's. With scale β, define
zi=β[(ℓi+−ℓi−)−(ℓref,i+−ℓref,i−)].For label smoothing ϵ, the per-pair loss is
Li=−(1−ϵ)logσ(zi)−ϵlogσ(−zi).Here, σ is the logistic sigmoid and log is the natural logarithm. The implicit rewards are
ri+=β(ℓi+−ℓref,i+) ri−=β(ℓi−−ℓref,i−).The loss is the mean of Li. Preference accuracy is the fraction of pairs with strictly greater chosen reward; ties count as incorrect.
Return a Python dictionary of PyTorch tensors: loss and preference_accuracy are scalars; per_example_loss, chosen_rewards, and rejected_rewards each have shape (batch,). Every output preserves the input floating dtype and device.
Input: policy_chosen = [-1], policy_rejected = [-2], reference_chosen = [-1.5], reference_rejected = [-1.5], beta = 0.5, label_smoothing = 0
Output: {"loss":0.474077,"per_example_loss":[0.474077],"chosen_rewards":[0.25],"rejected_rewards":[-0.25],"preference_accuracy":1}
Explanation: The policy gives the preferred response a larger relative margin than the reference, producing a positive DPO logit.
Input: policy_chosen = [-2,-1], policy_rejected = [-1,-2], reference_chosen = [-1,-1], reference_rejected = [-1,-1], beta = 1, label_smoothing = 0
Output: {"loss":0.813262,"per_example_loss":[1.313262,0.313262],"chosen_rewards":[-1,0],"rejected_rewards":[0,-1],"preference_accuracy":0.5}
Input: policy_chosen = [0], policy_rejected = [0], reference_chosen = [0], reference_rejected = [0], beta = 2, label_smoothing = 0.1
Output: {"loss":0.693147,"per_example_loss":[0.693147],"chosen_rewards":[0],"rejected_rewards":[0],"preference_accuracy":0}
torch.nn.functional.logsigmoid evaluates log-sigmoid stably.
Tensor.mean reduces a batch vector to a scalar tensor.
A comparison produces a Boolean tensor that can be converted to the input dtype.
Accepts: array
Accepts: array
Accepts: array
Accepts: array
Accepts: number
Accepts: number
Compute the Direct Preference Optimization loss for preferred and rejected response pairs. For pair i, let ℓi+ and ℓi− be the policy's supplied sequence log probabilities, and ℓref,i+ and ℓref,i− the reference model's. With scale β, define
zi=β[(ℓi+−ℓi−)−(ℓref,i+−ℓref,i−)].For label smoothing ϵ, the per-pair loss is
Li=−(1−ϵ)logσ(zi)−ϵlogσ(−zi).Here, σ is the logistic sigmoid and log is the natural logarithm. The implicit rewards are
ri+=β(ℓi+−ℓref,i+) ri−=β(ℓi−−ℓref,i−).The loss is the mean of Li. Preference accuracy is the fraction of pairs with strictly greater chosen reward; ties count as incorrect.
Return a Python dictionary of PyTorch tensors: loss and preference_accuracy are scalars; per_example_loss, chosen_rewards, and rejected_rewards each have shape (batch,). Every output preserves the input floating dtype and device.
Input: policy_chosen = [-1], policy_rejected = [-2], reference_chosen = [-1.5], reference_rejected = [-1.5], beta = 0.5, label_smoothing = 0
Output: {"loss":0.474077,"per_example_loss":[0.474077],"chosen_rewards":[0.25],"rejected_rewards":[-0.25],"preference_accuracy":1}
Explanation: The policy gives the preferred response a larger relative margin than the reference, producing a positive DPO logit.
Input: policy_chosen = [-2,-1], policy_rejected = [-1,-2], reference_chosen = [-1,-1], reference_rejected = [-1,-1], beta = 1, label_smoothing = 0
Output: {"loss":0.813262,"per_example_loss":[1.313262,0.313262],"chosen_rewards":[-1,0],"rejected_rewards":[0,-1],"preference_accuracy":0.5}
Input: policy_chosen = [0], policy_rejected = [0], reference_chosen = [0], reference_rejected = [0], beta = 2, label_smoothing = 0.1
Output: {"loss":0.693147,"per_example_loss":[0.693147],"chosen_rewards":[0],"rejected_rewards":[0],"preference_accuracy":0}
torch.nn.functional.logsigmoid evaluates log-sigmoid stably.
Tensor.mean reduces a batch vector to a scalar tensor.
A comparison produces a Boolean tensor that can be converted to the input dtype.
Accepts: array
Accepts: array
Accepts: array
Accepts: array
Accepts: number
Accepts: number
