Compute the clipped GRPO surrogate for selected response tokens. For sequence i and token t, let ℓitnew and ℓitold be the supplied log probabilities, Ai the sequence advantage, and ϵ the clip width. The bounded probability ratio is
rit=exp(clip(ℓitnew−ℓitold,−60,60)).The token surrogate is
sit=min(ritAi,clip(rit,1−ϵ,1+ϵ)Ai).Here, exp is the exponential function and clip limits a value to the stated inclusive interval. The loss is the negative mean surrogate across all tokens selected by the mask. Report each sequence's selected-token mean and the fraction of selected ratios strictly outside the clip interval. A sequence without selected tokens has mean zero; when no tokens are selected, both scalar results are zero.
Return a Python dictionary of PyTorch tensors: loss and clipped_fraction are scalars, and per_sequence_surrogate has shape (batch,). All outputs preserve the log-probability dtype and device.
Input: new_log_probs = [[0.4054651081081644,-0.6931471805599453],[0,0]], old_log_probs = [[0,0],[0,0]], advantages = [2,-1], token_mask = [[true,true],[true,false]], clip_epsilon = 0.2
Output: {"loss":-0.8,"per_sequence_surrogate":[1.7,-1],"clipped_fraction":0.666667}
Explanation: Ratios 1.5 and 0.5 lie outside the interval, while the third active ratio equals one.
Input: new_log_probs = [[0.1823215567939546,-0.2231435513142097]], old_log_probs = [[0,0]], advantages = [1], token_mask = [[true,true]], clip_epsilon = 0.2
Output: {"loss":-1,"per_sequence_surrogate":[1],"clipped_fraction":0}
Input: new_log_probs = [[2,-2]], old_log_probs = [[0,0]], advantages = [3], token_mask = [[false,false]], clip_epsilon = 0.2
Output: {"loss":0,"per_sequence_surrogate":[0],"clipped_fraction":0}
torch.clamp bounds a tensor elementwise.
A singleton token axis broadcasts one advantage across its sequence.
torch.minimum compares the two signed surrogate tensors elementwise.
Accepts: array
Accepts: array
Accepts: array
Accepts: array
Accepts: number
Compute the clipped GRPO surrogate for selected response tokens. For sequence i and token t, let ℓitnew and ℓitold be the supplied log probabilities, Ai the sequence advantage, and ϵ the clip width. The bounded probability ratio is
rit=exp(clip(ℓitnew−ℓitold,−60,60)).The token surrogate is
sit=min(ritAi,clip(rit,1−ϵ,1+ϵ)Ai).Here, exp is the exponential function and clip limits a value to the stated inclusive interval. The loss is the negative mean surrogate across all tokens selected by the mask. Report each sequence's selected-token mean and the fraction of selected ratios strictly outside the clip interval. A sequence without selected tokens has mean zero; when no tokens are selected, both scalar results are zero.
Return a Python dictionary of PyTorch tensors: loss and clipped_fraction are scalars, and per_sequence_surrogate has shape (batch,). All outputs preserve the log-probability dtype and device.
Input: new_log_probs = [[0.4054651081081644,-0.6931471805599453],[0,0]], old_log_probs = [[0,0],[0,0]], advantages = [2,-1], token_mask = [[true,true],[true,false]], clip_epsilon = 0.2
Output: {"loss":-0.8,"per_sequence_surrogate":[1.7,-1],"clipped_fraction":0.666667}
Explanation: Ratios 1.5 and 0.5 lie outside the interval, while the third active ratio equals one.
Input: new_log_probs = [[0.1823215567939546,-0.2231435513142097]], old_log_probs = [[0,0]], advantages = [1], token_mask = [[true,true]], clip_epsilon = 0.2
Output: {"loss":-1,"per_sequence_surrogate":[1],"clipped_fraction":0}
Input: new_log_probs = [[2,-2]], old_log_probs = [[0,0]], advantages = [3], token_mask = [[false,false]], clip_epsilon = 0.2
Output: {"loss":0,"per_sequence_surrogate":[0],"clipped_fraction":0}
torch.clamp bounds a tensor elementwise.
A singleton token axis broadcasts one advantage across its sequence.
torch.minimum compares the two signed surrogate tensors elementwise.
Accepts: array
Accepts: array
Accepts: array
Accepts: array
Accepts: number
