TensorTonic
Study Plans
Study Plan
Compute the Policy and Value Heads
Reinforcement LearningNeural Networks
Medium

Compute the policy and value heads from a shared PyTorch feature tensor F of shape (B, C, H, W), where B is batch size, C is channel count, and H and W are spatial dimensions.

L=Wpvec⁡(ReLU⁡(BN⁡(Conv⁡1×1(F;Wpc))))+bpL=W_p\operatorname{vec}(\operatorname{ReLU}(\operatorname{BN}(\operatorname{Conv}_{1\times1}(F;W_{pc}))))+b_pL=Wp​vec(ReLU(BN(Conv1×1​(F;Wpc​))))+bp​ v=tanh⁡(WvReLU⁡(Whvec⁡(ReLU⁡(BN⁡(Conv⁡1×1(F;Wvc))))+bh)+bv)v=\tanh(W_v\operatorname{ReLU}(W_h\operatorname{vec}(\operatorname{ReLU}(\operatorname{BN}(\operatorname{Conv}_{1\times1}(F;W_{vc}))))+b_h)+b_v)v=tanh(Wv​ReLU(Wh​vec(ReLU(BN(Conv1×1​(F;Wvc​))))+bh​)+bv​)

Here, Wpc and Wvc are policy_conv_weight and value_conv_weight. Wp and bp are policy_linear_weight and policy_linear_bias; Wh and bh are value_hidden_weight and value_hidden_bias; Wv and bv are value_output_weight and value_output_bias. L contains raw action logits and v contains current-player values. Each convolution uses stride 1 with no padding or bias. Flattening preserves the batch dimension and orders entries by channel, row, then column.

For an activation z in one channel, batch normalization is

BN⁡(z;γ,β)=γz−μσ2+ε+β\operatorname{BN}(z;\gamma,\beta)=\gamma\frac{z-\mu}{\sqrt{\sigma^2+\varepsilon}}+\betaBN(z;γ,β)=γσ2+ε​z−μ​+β

Here, mu and sigma squared are that channel's mean and population variance over the batch and spatial dimensions. Gamma and beta are its supplied scale and shift, and epsilon is the supplied positive normalization constant.

The policy branch uses policy_scale and policy_shift; the value branch uses value_scale and value_shift. Return a tuple of two PyTorch float32 tensors: policy logits of shape (B, H times W plus one) and values of shape (B,). Both outputs stay on the feature device; the final policy entry is pass.

Loading visualization...

Examples

Input: features = [[[[2]]]], policy_conv_weight = [[[[0]]],[[[0]]]], policy_scale = [1,1], policy_shift = [1,0], policy_linear_weight = [[1,0],[0,1]], policy_linear_bias = [0.5,-0.5], value_conv_weight = [[[[0]]]], value_scale = [1], value_shift = [1], value_hidden_weight = [[2]], value_hidden_bias = [0], value_output_weight = [[1]], value_output_bias = [0.5], epsilon = 0.00001

Output: ([[1.5,-0.5]], [0.986614])

Explanation: The policy branch returns one board-action logit plus pass, while tanh bounds the value branch.

Input: features shape = (2,1,2,2), policy_linear_weight shape = (5,8), value_hidden_weight shape = (2,4)

Output: (32-bit floating-point tensor with shape (2,5), 32-bit floating-point tensor with shape (2,))

Input: features shape = (2,2,1,2), policy_linear_weight shape = (3,4), value_hidden_weight shape = (3,2)

Output: (32-bit floating-point tensor with shape (2,3), 32-bit floating-point tensor with shape (2,))

Hint 1

flatten(1) keeps batch items separate.

Hint 2

F.linear uses weights shaped output features by input features.

Hint 3

squeeze(-1) removes only the final singleton value dimension.

Requirements

  • Use PyTorch float32 computation and outputs.
  • Return raw policy logits without softmax or legal masking.
  • Preserve every input tensor and the feature device.

Constraints

  • Features are a nonempty floating-point PyTorch tensor of shape (B, C, H, W); rectangular spatial dimensions are allowed.
  • All supplied tensors are floating-point tensors on the same device; their input dtypes may differ.
  • Policy convolution weights have shape (2, C, 1, 1); policy scale and shift each have length 2.
  • Policy linear weights have one row per action and 2 times H times W columns; the bias has one entry per action.
  • Value convolution weights have shape (1, C, 1, 1); value scale and shift each have length 1.
  • Value hidden weights have shape (D, H times W), with a length-D bias; D is the positive hidden width.
  • Value output weights have shape (1, D), with a length-1 bias. All linear weights store output features by input features.
  • Epsilon is a positive finite number.
Try Similar Problems
Apply an AlphaGo Residual BlockMediumAssemble a Compact Dual-Head NetworkHardEncode AlphaGo Zero Board HistoryMediumApply One Policy-Value Head Training StepHardTurn Root Visits into a Search PolicyMediumImplement the AlphaGo Policy-Value LossMediumApply a Dihedral Go SymmetryMediumScore a Terminal Position with Tromp-Taylor RulesMediumDQN Loss with Target NetworkMediumDouble DQN TargetEasyDueling DQN DecompositionEasyAutoencoder Forward PassMedium

Accepts: array

Accepts: array

Accepts: array

Accepts: array

Accepts: array

Accepts: array

Accepts: array

Accepts: array

Accepts: array

Accepts: array

Accepts: array

Accepts: array

Accepts: array

Accepts: number

You must run your code first.

Compute the Policy and Value Heads

Reinforcement LearningNeural Networks
Medium

Compute the policy and value heads from a shared PyTorch feature tensor F of shape (B, C, H, W), where B is batch size, C is channel count, and H and W are spatial dimensions.

L=Wpvec⁡(ReLU⁡(BN⁡(Conv⁡1×1(F;Wpc))))+bpL=W_p\operatorname{vec}(\operatorname{ReLU}(\operatorname{BN}(\operatorname{Conv}_{1\times1}(F;W_{pc}))))+b_pL=Wp​vec(ReLU(BN(Conv1×1​(F;Wpc​))))+bp​ v=tanh⁡(WvReLU⁡(Whvec⁡(ReLU⁡(BN⁡(Conv⁡1×1(F;Wvc))))+bh)+bv)v=\tanh(W_v\operatorname{ReLU}(W_h\operatorname{vec}(\operatorname{ReLU}(\operatorname{BN}(\operatorname{Conv}_{1\times1}(F;W_{vc}))))+b_h)+b_v)v=tanh(Wv​ReLU(Wh​vec(ReLU(BN(Conv1×1​(F;Wvc​))))+bh​)+bv​)

Here, Wpc and Wvc are policy_conv_weight and value_conv_weight. Wp and bp are policy_linear_weight and policy_linear_bias; Wh and bh are value_hidden_weight and value_hidden_bias; Wv and bv are value_output_weight and value_output_bias. L contains raw action logits and v contains current-player values. Each convolution uses stride 1 with no padding or bias. Flattening preserves the batch dimension and orders entries by channel, row, then column.

For an activation z in one channel, batch normalization is

BN⁡(z;γ,β)=γz−μσ2+ε+β\operatorname{BN}(z;\gamma,\beta)=\gamma\frac{z-\mu}{\sqrt{\sigma^2+\varepsilon}}+\betaBN(z;γ,β)=γσ2+ε​z−μ​+β

Here, mu and sigma squared are that channel's mean and population variance over the batch and spatial dimensions. Gamma and beta are its supplied scale and shift, and epsilon is the supplied positive normalization constant.

The policy branch uses policy_scale and policy_shift; the value branch uses value_scale and value_shift. Return a tuple of two PyTorch float32 tensors: policy logits of shape (B, H times W plus one) and values of shape (B,). Both outputs stay on the feature device; the final policy entry is pass.

Loading visualization...

Examples

Input: features = [[[[2]]]], policy_conv_weight = [[[[0]]],[[[0]]]], policy_scale = [1,1], policy_shift = [1,0], policy_linear_weight = [[1,0],[0,1]], policy_linear_bias = [0.5,-0.5], value_conv_weight = [[[[0]]]], value_scale = [1], value_shift = [1], value_hidden_weight = [[2]], value_hidden_bias = [0], value_output_weight = [[1]], value_output_bias = [0.5], epsilon = 0.00001

Output: ([[1.5,-0.5]], [0.986614])

Explanation: The policy branch returns one board-action logit plus pass, while tanh bounds the value branch.

Input: features shape = (2,1,2,2), policy_linear_weight shape = (5,8), value_hidden_weight shape = (2,4)

Output: (32-bit floating-point tensor with shape (2,5), 32-bit floating-point tensor with shape (2,))

Input: features shape = (2,2,1,2), policy_linear_weight shape = (3,4), value_hidden_weight shape = (3,2)

Output: (32-bit floating-point tensor with shape (2,3), 32-bit floating-point tensor with shape (2,))

Hint 1

flatten(1) keeps batch items separate.

Hint 2

F.linear uses weights shaped output features by input features.

Hint 3

squeeze(-1) removes only the final singleton value dimension.

Requirements

  • Use PyTorch float32 computation and outputs.
  • Return raw policy logits without softmax or legal masking.
  • Preserve every input tensor and the feature device.

Constraints

  • Features are a nonempty floating-point PyTorch tensor of shape (B, C, H, W); rectangular spatial dimensions are allowed.
  • All supplied tensors are floating-point tensors on the same device; their input dtypes may differ.
  • Policy convolution weights have shape (2, C, 1, 1); policy scale and shift each have length 2.
  • Policy linear weights have one row per action and 2 times H times W columns; the bias has one entry per action.
  • Value convolution weights have shape (1, C, 1, 1); value scale and shift each have length 1.
  • Value hidden weights have shape (D, H times W), with a length-D bias; D is the positive hidden width.
  • Value output weights have shape (1, D), with a length-1 bias. All linear weights store output features by input features.
  • Epsilon is a positive finite number.
Try Similar Problems
Apply an AlphaGo Residual BlockMediumAssemble a Compact Dual-Head NetworkHardEncode AlphaGo Zero Board HistoryMediumApply One Policy-Value Head Training StepHardTurn Root Visits into a Search PolicyMediumImplement the AlphaGo Policy-Value LossMediumApply a Dihedral Go SymmetryMediumScore a Terminal Position with Tromp-Taylor RulesMediumDQN Loss with Target NetworkMediumDouble DQN TargetEasyDueling DQN DecompositionEasyAutoencoder Forward PassMedium

Accepts: array

Accepts: array

Accepts: array

Accepts: array

Accepts: array

Accepts: array

Accepts: array

Accepts: array

Accepts: array

Accepts: array

Accepts: array

Accepts: array

Accepts: array

Accepts: number

You must run your code first.