What is DPO?
Introduction
Project introduction
What is DPO?
A language model trained on text from the internet learns to write, but not how people want it to behave. Alignment is the step that teaches it which of its possible replies people prefer. The usual data is a preference pair, made of one prompt, two replies, and a note saying which reply a person chose.
The classic recipe, RLHF (reinforcement learning from human feedback), first trains a separate reward model to score replies the way people would, then uses reinforcement learning to make the language model earn a high score without drifting far from where it started. It works, as InstructGPT showed in 2022, but it needs a second model and a delicate training loop.
Direct Preference Optimization (DPO, 2023) reaches the same goal with one ordinary loss. It measures each reply's reward by how much more likely the model now makes it than a frozen copy of the starting model, the reference, does. No reward model and no sampling during training are needed.
Why build it on a tiny model?
Every piece of DPO fits in a few lines of code, including summing token log-probabilities, the Bradley-Terry formula for preferences, and a loss on the gap between two rewards. A tiny model with a preference you can check by eye, polite replies versus rude ones, makes each piece visible and trains in about 2 seconds on a CPU.
The small scale also lets you test what is usually hard to see. You will measure where the reward margin comes from and how far the model moves away from the reference on text that has nothing to do with politeness, and find that one setting, beta, the size of the gap the loss asks for, decides whether that move stays small or wrecks the model in this setup. The effects are exaggerated at this tiny scale, and the notebook says where.
One preference pair, before and after DPO
training 0%The prompt is please fix the gate .
i will fix the gate now . happy to help .
i will fix the gate now . do not ask again .
What you will build
01
Score replies and preferences
Sum a reply's token log-probabilities and turn two scores into a preference probability.
02
The DPO loss
Freeze a reference model, write the loss and its implicit rewards, and log accuracy and margin.
03
Measure the drift
Compute KL divergence on neutral text, compare beta values, and try IPO, a loss with a fixed target.
Start with the Bradley-Terry model
Turn two scores into the probability that one reply is preferred.