PPO with Reference-Policy KL

Compute a PPO policy objective with reference-policy KL regularization for stable RLHF optimization.