Study Plan

Implement the AlphaGo Policy-Value Loss

Loss FunctionsReinforcement Learning

Medium

Combine AlphaGo's policy, value, and L2 objectives. For batch size $B$ , action count $A$ , target search policy $\pi$ , and probabilities $p$ given by softmax of policy_logits across actions, compute

\ell_p=-\frac{1}{B}\sum_{i=1}^{B}\sum_{a=1}^{A}\pi_{i,a}\log p_{i,a}.

Here $i$ indexes batch items, $a$ indexes actions, and $\ell_p$ is policy loss. For predicted_values $v$ and target_values $z$ , the value loss is

\ell_v=\frac{1}{B}\sum_{i=1}^{B}(v_i-z_i)^2.

For parameter tensor $\theta_j$ at index $j$ in parameters and l2_coefficient $\lambda$ , the regularization loss is

\ell_r=\lambda\sum_j\lVert\theta_j\rVert_2^2.

The total loss is

\ell=\ell_p+\ell_v+\ell_r.

Return a four-item tuple of scalar PyTorch float32 tensors in this order: total loss, policy loss, value loss, regularization loss. Keep them on the input device; an empty parameter list contributes zero regularization.

Loading visualization...

Examples

Input: policy_logits = [[0,0]], predicted_values = [0], target_policies = [[1,0]], target_values = [1], parameters = [[1,-1]], l2_coefficient = 0.1

Output: (1.893147, 0.693147, 1, 0.2)

Explanation: The total adds log(2), unit value error, and the weighted squared parameter norm.

Input: policy_logits = [[2,-1,0.5],[-2,3,0]], predicted_values = [0.4,-0.2], target_policies = [[0.6,0.1,0.3],[0.2,0.5,0.3]], target_values = [0.5,-0.5], parameters = [], l2_coefficient = 0

Output: (1.523148, 1.473148, 0.05, 0)

Input: policy_logits = [[1000,999],[-1000,-999]], predicted_values = [1,-1], target_policies = [[0.25,0.75],[0.8,0.2]], target_values = [1,-1], parameters = [[0.5],[2,-3]], l2_coefficient = 0.01

Output: (1.220762, 1.088262, 0, 0.1325)

Hint 1

torch.log_softmax provides stable log probabilities across actions.

Hint 2

torch.square and torch.sum give a parameter tensor's squared norm.

Hint 3

Tensor.new_zeros can create a scalar on the input device and dtype.

Requirements

Use PyTorch float32 tensors; parameters is a list of tensors.
Preserve gradients from each nonconstant loss term to its contributing inputs.
Leave input values unchanged.

Constraints

All tensors are finite float32 tensors on the same device.
Policy logits and target policies have matching nonempty shape (B, A).
Predicted values and target values each have shape (B,).
Target-policy entries are nonnegative and each row sums to one.
Target values lie between -1 and 1; l2_coefficient is finite and nonnegative.
The parameter list may be empty.

Try Similar Problems

Turn Root Visits into a Search PolicyMedium Compare Sequence and Token Length NormalizationMedium Implement the DPO LossMedium Score a KL-Regularized Policy BatchMedium Mix Dirichlet Noise into Root PriorsEasy Compute a Clipped GRPO SurrogateHard Compute the Policy and Value HeadsMedium Apply an AlphaGo Residual BlockMedium Bradley-Terry Reward Model LossMedium REINFORCE with BaselineMedium PPO Clipped Surrogate ObjectiveMedium DQN Loss with Target NetworkMedium

Implement the AlphaGo Policy-Value Loss

Loss FunctionsReinforcement Learning

Medium

\ell_p=-\frac{1}{B}\sum_{i=1}^{B}\sum_{a=1}^{A}\pi_{i,a}\log p_{i,a}.

Here $i$ indexes batch items, $a$ indexes actions, and $\ell_p$ is policy loss. For predicted_values $v$ and target_values $z$ , the value loss is

\ell_v=\frac{1}{B}\sum_{i=1}^{B}(v_i-z_i)^2.

For parameter tensor $\theta_j$ at index $j$ in parameters and l2_coefficient $\lambda$ , the regularization loss is

\ell_r=\lambda\sum_j\lVert\theta_j\rVert_2^2.

The total loss is

\ell=\ell_p+\ell_v+\ell_r.

Loading visualization...

Examples

Input: policy_logits = [[0,0]], predicted_values = [0], target_policies = [[1,0]], target_values = [1], parameters = [[1,-1]], l2_coefficient = 0.1

Output: (1.893147, 0.693147, 1, 0.2)

Explanation: The total adds log(2), unit value error, and the weighted squared parameter norm.

Input: policy_logits = [[2,-1,0.5],[-2,3,0]], predicted_values = [0.4,-0.2], target_policies = [[0.6,0.1,0.3],[0.2,0.5,0.3]], target_values = [0.5,-0.5], parameters = [], l2_coefficient = 0

Output: (1.523148, 1.473148, 0.05, 0)

Input: policy_logits = [[1000,999],[-1000,-999]], predicted_values = [1,-1], target_policies = [[0.25,0.75],[0.8,0.2]], target_values = [1,-1], parameters = [[0.5],[2,-3]], l2_coefficient = 0.01

Output: (1.220762, 1.088262, 0, 0.1325)

Hint 1

torch.log_softmax provides stable log probabilities across actions.

Hint 2

torch.square and torch.sum give a parameter tensor's squared norm.

Hint 3

Tensor.new_zeros can create a scalar on the input device and dtype.

Requirements

Use PyTorch float32 tensors; parameters is a list of tensors.
Preserve gradients from each nonconstant loss term to its contributing inputs.
Leave input values unchanged.

Constraints

All tensors are finite float32 tensors on the same device.
Policy logits and target policies have matching nonempty shape (B, A).
Predicted values and target values each have shape (B,).
Target-policy entries are nonnegative and each row sums to one.
Target values lie between -1 and 1; l2_coefficient is finite and nonnegative.
The parameter list may be empty.

Try Similar Problems

policy_logits =

Accepts: array

predicted_values =

Accepts: array

target_policies =

Accepts: array

target_values =

Accepts: array

parameters =

Accepts: array

l2_coefficient =

Accepts: number