RLOO (REINFORCE Leave-One-Out)

RLOO (REINFORCE Leave-One-Out) is a policy gradient variance-reduction trick used in modern RLHF.