Build from first principles

Align a Language Model with DPO

Turn preference pairs into a loss, train a rude help desk model into a polite one without a reward model, and measure how far it drifts from where it started.

What is DPO?Understand preference pairs, the reference model, and why no reward model is needed

1Turn Two Scores into a Preference ProbabilityModel preferences between two replies

2Score a Reply with Its Log-ProbabilityScore whole replies with a language model

3Write the DPO Loss and Its Implicit RewardsTrain on preferences without a reward model

4Log Reward Accuracy and MarginWatch DPO train

5Write the IPO LossKeep the policy from running away

6Measure Drift with Per-Token KL DivergenceMeasure how far the policy has moved

From Pieces to DPOSee how your functions fit together before training the model

Align, Inspect, and Stress-Test a Language ModelMake a small model polite with DPO, see where the margin comes from, and sweep beta