From Pieces to DPO

After 100 short steps the model is polite on every one of 120 requests it never trained on, and it still carries out every request. Looking inside shows how it happened. The rude replies became thousands of times less likely, while the polite ones gained only a little. The loss asks for a gap, and pushing the rejected reply down is the easiest way to make one.

Then you change beta. A smaller beta still gives perfectly polite replies, so the preference numbers look just as good, but KL on neutral text grows about 16 times, and a very small beta breaks the model while the loss reports success. Beta sets how big a gap the loss asks for; the step size, scaled here so every beta starts at the same speed, sets how fast a run gets there, and the notebook shows that a fixed step size gives a different picture. In this tiny model the tone comes down to a single word, so these effects are larger and faster than in big models, which train with much smaller steps and usually start from beta 0.1.