TensorTonic
Study Plans
Study Plan
Turn Root Visits into a Search Policy
Reinforcement Learning
Medium

Convert root visit counts into the search policy used to select a self-play move. For positive temperature τ\tauτ, each legal action aaa receives

πa=Na1/τ∑b∈LNb1/τ.\pi_a=\frac{N_a^{1/\tau}}{\sum_{b\in\mathcal{L}}N_b^{1/\tau}}.πa​=∑b∈L​Nb1/τ​Na1/τ​​.

Here πa\pi_aπa​ is the search probability, NaN_aNa​ is the visit count, τ\tauτ is temperature, and L\mathcal{L}L is the legal-action set; bbb ranges over that set. At positive temperature, if every legal visit count is zero, use a uniform legal policy. At temperature zero, use a one-hot policy at the lowest-index legal action with the largest count. At positive temperature, select the first action whose cumulative probability is strictly greater than the supplied uniform random draw. Return a two-item tuple: the NumPy float64 policy array with the same shape as visits, followed by the selected action as a Python integer.

Loading visualization...

Examples

Input: visits = [1,3,0], legal_mask = [true,true,true], temperature = 1, uniform_draw = 0.2

Output: ([0.25,0.75,0], 0)

Explanation: Temperature one normalizes visits directly, and draw 0.2 lies in the first cumulative interval.

Input: visits = [4,4,2], legal_mask = [true,true,true], temperature = 0, uniform_draw = null

Output: ([1,0,0], 0)

Input: visits = [1,2,3], legal_mask = [true,true,true], temperature = 0.5, uniform_draw = 0.4

Output: ([0.071429,0.285714,0.642857], 2)

Hint 1

np.flatnonzero and np.argmax preserve the first-maximum tie break.

Hint 2

Logarithms can keep positive-count weights stable at small temperatures.

Hint 3

np.searchsorted with side="right" handles strict cumulative boundaries.

Requirements

  • Accept integer NumPy visit counts and a Boolean NumPy legal mask.
  • Assign exact zero probability to every illegal action.
  • Use the supplied uniform_draw, not a newly sampled value.

Constraints

  • Visit counts and the legal mask are aligned, nonempty one-dimensional arrays.
  • Visit counts are nonnegative integers and the mask contains at least one legal action.
  • Temperature is finite and nonnegative.
  • uniform_draw is None at temperature zero and lies from zero inclusive to one exclusive otherwise.
Try Similar Problems
Implement the AlphaGo Policy-Value LossMediumMix Dirichlet Noise into Root PriorsEasyCompute the Policy and Value HeadsMediumApply an AlphaGo Residual BlockMediumApply a Dihedral Go SymmetryMediumScore a Terminal Position with Tromp-Taylor RulesMediumBack Up an MCTS Simulation PathMediumEncode AlphaGo Zero Board HistoryMediumValue IterationMediumExpected SARSAMediumBradley-Terry Reward Model LossMediumRLOO (REINFORCE Leave-One-Out)Medium

Accepts: array

Accepts: array

Accepts: number

Accepts: any

You must run your code first.

Turn Root Visits into a Search Policy

Reinforcement Learning
Medium

Convert root visit counts into the search policy used to select a self-play move. For positive temperature τ\tauτ, each legal action aaa receives

πa=Na1/τ∑b∈LNb1/τ.\pi_a=\frac{N_a^{1/\tau}}{\sum_{b\in\mathcal{L}}N_b^{1/\tau}}.πa​=∑b∈L​Nb1/τ​Na1/τ​​.

Here πa\pi_aπa​ is the search probability, NaN_aNa​ is the visit count, τ\tauτ is temperature, and L\mathcal{L}L is the legal-action set; bbb ranges over that set. At positive temperature, if every legal visit count is zero, use a uniform legal policy. At temperature zero, use a one-hot policy at the lowest-index legal action with the largest count. At positive temperature, select the first action whose cumulative probability is strictly greater than the supplied uniform random draw. Return a two-item tuple: the NumPy float64 policy array with the same shape as visits, followed by the selected action as a Python integer.

Loading visualization...

Examples

Input: visits = [1,3,0], legal_mask = [true,true,true], temperature = 1, uniform_draw = 0.2

Output: ([0.25,0.75,0], 0)

Explanation: Temperature one normalizes visits directly, and draw 0.2 lies in the first cumulative interval.

Input: visits = [4,4,2], legal_mask = [true,true,true], temperature = 0, uniform_draw = null

Output: ([1,0,0], 0)

Input: visits = [1,2,3], legal_mask = [true,true,true], temperature = 0.5, uniform_draw = 0.4

Output: ([0.071429,0.285714,0.642857], 2)

Hint 1

np.flatnonzero and np.argmax preserve the first-maximum tie break.

Hint 2

Logarithms can keep positive-count weights stable at small temperatures.

Hint 3

np.searchsorted with side="right" handles strict cumulative boundaries.

Requirements

  • Accept integer NumPy visit counts and a Boolean NumPy legal mask.
  • Assign exact zero probability to every illegal action.
  • Use the supplied uniform_draw, not a newly sampled value.

Constraints

  • Visit counts and the legal mask are aligned, nonempty one-dimensional arrays.
  • Visit counts are nonnegative integers and the mask contains at least one legal action.
  • Temperature is finite and nonnegative.
  • uniform_draw is None at temperature zero and lies from zero inclusive to one exclusive otherwise.
Try Similar Problems
Implement the AlphaGo Policy-Value LossMediumMix Dirichlet Noise into Root PriorsEasyCompute the Policy and Value HeadsMediumApply an AlphaGo Residual BlockMediumApply a Dihedral Go SymmetryMediumScore a Terminal Position with Tromp-Taylor RulesMediumBack Up an MCTS Simulation PathMediumEncode AlphaGo Zero Board HistoryMediumValue IterationMediumExpected SARSAMediumBradley-Terry Reward Model LossMediumRLOO (REINFORCE Leave-One-Out)Medium

Accepts: array

Accepts: array

Accepts: number

Accepts: any

You must run your code first.