Study Plan

Compare Sequence and Token Length Normalization

Language ModelsReinforcement LearningLoss Functions

Medium

Compare sequence-summed and token-averaged policy-gradient contributions. For sequence i and token t, let $A_i$ be its advantage, $\ell_{it}$ its token log probability, $m_{it}$ the Boolean selection mask, and $n_i$ its selected-token count. The two contributions are

g_i^{\mathrm{sum}}=A_i\sum_t m_{it}\ell_{it}

g_i^{\mathrm{avg}}=\begin{cases}g_i^{\mathrm{sum}}/n_i,&n_i>0\\0,&n_i=0.\end{cases}

Compare their total absolute magnitudes using

R=\frac{\sum_i|g_i^{\mathrm{sum}}|}{\sum_i|g_i^{\mathrm{avg}}|}.

Here, $R$ is the magnitude ratio and the sums run over sequences. Use ratio zero when its denominator is zero. A fully masked sequence contributes zero to both vectors.

Return a Python dictionary of PyTorch tensors with sequence_summed and token_averaged of shape (batch,), and magnitude_ratio as a scalar. All outputs preserve the log-probability dtype and device.

Loading visualization...

Examples

Input: token_log_probs = [[-1,-2],[-3,-4]], advantages = [2,-1], token_mask = [[true,true],[true,true]]

Output: {"sequence_summed":[-6,7],"token_averaged":[-3,3.5],"magnitude_ratio":2}

Explanation: Both rows have two active tokens, so each averaged contribution is half its summed contribution.

Input: token_log_probs = [[-2,-5,-8]], advantages = [3], token_mask = [[true,false,false]]

Output: {"sequence_summed":[-6],"token_averaged":[-6],"magnitude_ratio":1}

Input: token_log_probs = [[-1,-2],[-3,-4]], advantages = [2,-1], token_mask = [[false,false],[false,false]]

Output: {"sequence_summed":[0,0],"token_averaged":[0,0],"magnitude_ratio":0}

Hint 1

Tensor.sum reduces a selected axis.

Hint 2

torch.where selects a value for rows with no active tokens.

Hint 3

Tensor.abs prevents signed sequence contributions from canceling during aggregation.

Requirements

Use PyTorch and preserve the supplied floating dtype and device.
Exclude masked tokens from both normalization conventions.
Do not modify input tensors.

Constraints

token_log_probs is a finite rank-two PyTorch tensor of shape (batch, tokens).
advantages is a floating tensor of shape (batch,); token_mask is a Boolean tensor matching the token shape.
Floating tensors share dtype and device; editor inputs use float32, and the mask uses the same device.
Inputs may be noncontiguous.

Try Similar Problems

Compute a Clipped GRPO SurrogateHard Implement the DPO LossMedium Score a KL-Regularized Policy BatchMedium Normalize GRPO Group AdvantagesMedium Canonicalize Pairwise Preference DataMedium Implement Symmetric CLIP Contrastive LossMedium Implement the SigLIP Pairwise LossMedium Build an Assistant-Only SFT Loss MaskEasy Bradley-Terry Reward Model LossMedium REINFORCE with BaselineMedium PPO Clipped Surrogate ObjectiveMedium DQN Loss with Target NetworkMedium

Compare Sequence and Token Length Normalization

Language ModelsReinforcement LearningLoss Functions

Medium

g_i^{\mathrm{sum}}=A_i\sum_t m_{it}\ell_{it}

g_i^{\mathrm{avg}}=\begin{cases}g_i^{\mathrm{sum}}/n_i,&n_i>0\\0,&n_i=0.\end{cases}

Compare their total absolute magnitudes using

R=\frac{\sum_i|g_i^{\mathrm{sum}}|}{\sum_i|g_i^{\mathrm{avg}}|}.

Here, $R$ is the magnitude ratio and the sums run over sequences. Use ratio zero when its denominator is zero. A fully masked sequence contributes zero to both vectors.

Return a Python dictionary of PyTorch tensors with sequence_summed and token_averaged of shape (batch,), and magnitude_ratio as a scalar. All outputs preserve the log-probability dtype and device.

Loading visualization...

Examples

Input: token_log_probs = [[-1,-2],[-3,-4]], advantages = [2,-1], token_mask = [[true,true],[true,true]]

Output: {"sequence_summed":[-6,7],"token_averaged":[-3,3.5],"magnitude_ratio":2}

Explanation: Both rows have two active tokens, so each averaged contribution is half its summed contribution.

Input: token_log_probs = [[-2,-5,-8]], advantages = [3], token_mask = [[true,false,false]]

Output: {"sequence_summed":[-6],"token_averaged":[-6],"magnitude_ratio":1}

Input: token_log_probs = [[-1,-2],[-3,-4]], advantages = [2,-1], token_mask = [[false,false],[false,false]]

Output: {"sequence_summed":[0,0],"token_averaged":[0,0],"magnitude_ratio":0}

Hint 1

Tensor.sum reduces a selected axis.

Hint 2

torch.where selects a value for rows with no active tokens.

Hint 3

Tensor.abs prevents signed sequence contributions from canceling during aggregation.

Requirements

Use PyTorch and preserve the supplied floating dtype and device.
Exclude masked tokens from both normalization conventions.
Do not modify input tensors.

Constraints

token_log_probs is a finite rank-two PyTorch tensor of shape (batch, tokens).
advantages is a floating tensor of shape (batch,); token_mask is a Boolean tensor matching the token shape.
Floating tensors share dtype and device; editor inputs use float32, and the mask uses the same device.
Inputs may be noncontiguous.

Try Similar Problems

token_log_probs =

Accepts: array

advantages =

Accepts: array

token_mask =

Accepts: array