Direct Preference Optimization (DPO) replaces the reward-model + PPO pipeline of RLHF with a single supervised loss.
