TensorTonic
Study Plans
Study Plan
Run Deterministic Neural-Guided MCTS
Reinforcement Learning
Hard

Run deterministic neural-guided MCTS on a supplied two-player tree, starting at root_state. Inputs are Python dictionaries keyed by state names:

  • transitions maps each nonterminal state to its integer-action-to-child dictionary.
  • legal_actions gives increasing action lists; policy_logits gives matching neural logits.
  • players gives every state's current player, 1 or -1.
  • value_predictions gives nonterminal neural values; terminal_values gives terminal outcomes. Both use that state's current-player perspective.

The root starts expanded; all edge visit counts and total values start at zero. Priors are softmax probabilities of the supplied logits. Run exactly simulations traversals. Selection maximizes

S(s,a)=Q(s,a)+cpuctP(s,a)∑bN(s,b)1+N(s,a).S(s,a)=Q(s,a)+c_{\mathrm{puct}}P(s,a)\frac{\sqrt{\sum_b N(s,b)}}{1+N(s,a)}.S(s,a)=Q(s,a)+cpuct​P(s,a)1+N(s,a)∑b​N(s,b)​​.

Here sss is a state, aaa and bbb are legal actions, SSS is the score, cpuctc_{\mathrm{puct}}cpuct​ is cpuct, PPP is the prior, NNN is the visit count, and QQQ is the mean value from the parent player's perspective, initially zero. Exact ties choose the lowest action.

Each traversal ends at the first terminal or unexpanded state. Use its supplied value, expanding it only if nonterminal. Each traversed edge gains one visit and the leaf value signed for its parent player; its mean is accumulated value divided by visits. At the root and positive temperature,

πa=Na1/τ∑bNb1/τ.\pi_a=\frac{N_a^{1/\tau}}{\sum_b N_b^{1/\tau}}.πa​=∑b​Nb1/τ​Na1/τ​​.

Here NaN_aNa​ is a root visit count, τ\tauτ is temperature, and πa\pi_aπa​ is the final policy. At zero temperature, assign probability one to the most-visited action, breaking ties by lowest action.

Return a three-item tuple of NumPy arrays in the root's legal-action order: int64 visit counts, float64 mean values, and float64 policy. Each has shape (A,), where A is the root's legal-action count.

Loading visualization...

Examples

Input: root_state="r", transitions={"r": {"0": "a", "1": "b"}}, legal_actions={"r": [0, 1]}, players={"r": 1, "a": -1, "b": -1}, policy_logits={"r": [0, 0]}, value_predictions={"r": 0}, terminal_values={"a": 1, "b": -1}, simulations=3, cpuct=1, temperature=1

Output: ([1, 2], [-1.0, 1.0], [0.333333, 0.666667])

Explanation: The first tie visits action 0, then the backed-up values make action 1 preferable for the remaining simulations.

Input: root_state="r", transitions={"r": {"2": "m", "5": "t5"}, "m": {"0": "t20", "1": "t21"}}, legal_actions={"r": [2, 5], "m": [0, 1]}, players={"r": 1, "m": -1, "t5": -1, "t20": 1, "t21": 1}, policy_logits={"r": [0, 0], "m": [0, 0]}, value_predictions={"r": 0.2, "m": 0.4}, terminal_values={"t5": -0.5, "t20": 1, "t21": -1}, simulations=5, cpuct=4, temperature=2

Output: ([2, 3], [0.3, 0.5], [0.44949, 0.55051])

Input: root_state="r", transitions={"r": {"3": "a", "7": "b"}}, legal_actions={"r": [3, 7]}, players={"r": -1, "a": 1, "b": 1}, policy_logits={"r": [0, 0]}, value_predictions={"r": 0}, terminal_values={"a": 0, "b": 0}, simulations=2, cpuct=1, temperature=0

Output: ([1, 1], [0.0, 0.0], [1.0, 0.0])

Hint 1

np.argmax returns the first maximum in legal-action order.

Hint 2

Comparing the parent and leaf player signs determines the backed-up value.

Hint 3

Logarithms can stabilize visit-count weights at small positive temperatures.

Requirements

  • Accept native Python dictionaries and lists; use NumPy float64 for priors, values, and policy calculations.
  • Use the supplied logits and values without sampling or calling a model.
  • Leave every input unchanged.

Constraints

  • The complete graph is a valid finite tree with a nonterminal root.
  • State names are nonempty strings; legal actions are strictly increasing nonnegative integers.
  • Each nonterminal has matching transitions, logits, and a value prediction.
  • Players alternate across each edge; neural and terminal values lie between -1 and 1.
  • simulations is a positive integer; cpuct and temperature are finite and nonnegative.
Try Similar Problems
Apply One Policy-Value Head Training StepHardApply a Dihedral Go SymmetryMediumSample a Recent Self-Play Replay BatchEasyApply a Go Action with Captures and RepetitionHardAssemble a Compact Dual-Head NetworkHardBuild the Legal Go Action MaskHardTurn Root Visits into a Search PolicyMediumCompute the Policy and Value HeadsMediumGRPO Group-Relative AdvantageHardPPO with Reference-Policy KLHardLinUCB Contextual BanditHardValue IterationMedium

Accepts: string

Accepts: object

Accepts: object

Accepts: object

Accepts: object

Accepts: object

Accepts: object

Accepts: number

Accepts: number

Accepts: number

You must run your code first.

Run Deterministic Neural-Guided MCTS

Reinforcement Learning
Hard

Run deterministic neural-guided MCTS on a supplied two-player tree, starting at root_state. Inputs are Python dictionaries keyed by state names:

  • transitions maps each nonterminal state to its integer-action-to-child dictionary.
  • legal_actions gives increasing action lists; policy_logits gives matching neural logits.
  • players gives every state's current player, 1 or -1.
  • value_predictions gives nonterminal neural values; terminal_values gives terminal outcomes. Both use that state's current-player perspective.

The root starts expanded; all edge visit counts and total values start at zero. Priors are softmax probabilities of the supplied logits. Run exactly simulations traversals. Selection maximizes

S(s,a)=Q(s,a)+cpuctP(s,a)∑bN(s,b)1+N(s,a).S(s,a)=Q(s,a)+c_{\mathrm{puct}}P(s,a)\frac{\sqrt{\sum_b N(s,b)}}{1+N(s,a)}.S(s,a)=Q(s,a)+cpuct​P(s,a)1+N(s,a)∑b​N(s,b)​​.

Here sss is a state, aaa and bbb are legal actions, SSS is the score, cpuctc_{\mathrm{puct}}cpuct​ is cpuct, PPP is the prior, NNN is the visit count, and QQQ is the mean value from the parent player's perspective, initially zero. Exact ties choose the lowest action.

Each traversal ends at the first terminal or unexpanded state. Use its supplied value, expanding it only if nonterminal. Each traversed edge gains one visit and the leaf value signed for its parent player; its mean is accumulated value divided by visits. At the root and positive temperature,

πa=Na1/τ∑bNb1/τ.\pi_a=\frac{N_a^{1/\tau}}{\sum_b N_b^{1/\tau}}.πa​=∑b​Nb1/τ​Na1/τ​​.

Here NaN_aNa​ is a root visit count, τ\tauτ is temperature, and πa\pi_aπa​ is the final policy. At zero temperature, assign probability one to the most-visited action, breaking ties by lowest action.

Return a three-item tuple of NumPy arrays in the root's legal-action order: int64 visit counts, float64 mean values, and float64 policy. Each has shape (A,), where A is the root's legal-action count.

Loading visualization...

Examples

Input: root_state="r", transitions={"r": {"0": "a", "1": "b"}}, legal_actions={"r": [0, 1]}, players={"r": 1, "a": -1, "b": -1}, policy_logits={"r": [0, 0]}, value_predictions={"r": 0}, terminal_values={"a": 1, "b": -1}, simulations=3, cpuct=1, temperature=1

Output: ([1, 2], [-1.0, 1.0], [0.333333, 0.666667])

Explanation: The first tie visits action 0, then the backed-up values make action 1 preferable for the remaining simulations.

Input: root_state="r", transitions={"r": {"2": "m", "5": "t5"}, "m": {"0": "t20", "1": "t21"}}, legal_actions={"r": [2, 5], "m": [0, 1]}, players={"r": 1, "m": -1, "t5": -1, "t20": 1, "t21": 1}, policy_logits={"r": [0, 0], "m": [0, 0]}, value_predictions={"r": 0.2, "m": 0.4}, terminal_values={"t5": -0.5, "t20": 1, "t21": -1}, simulations=5, cpuct=4, temperature=2

Output: ([2, 3], [0.3, 0.5], [0.44949, 0.55051])

Input: root_state="r", transitions={"r": {"3": "a", "7": "b"}}, legal_actions={"r": [3, 7]}, players={"r": -1, "a": 1, "b": 1}, policy_logits={"r": [0, 0]}, value_predictions={"r": 0}, terminal_values={"a": 0, "b": 0}, simulations=2, cpuct=1, temperature=0

Output: ([1, 1], [0.0, 0.0], [1.0, 0.0])

Hint 1

np.argmax returns the first maximum in legal-action order.

Hint 2

Comparing the parent and leaf player signs determines the backed-up value.

Hint 3

Logarithms can stabilize visit-count weights at small positive temperatures.

Requirements

  • Accept native Python dictionaries and lists; use NumPy float64 for priors, values, and policy calculations.
  • Use the supplied logits and values without sampling or calling a model.
  • Leave every input unchanged.

Constraints

  • The complete graph is a valid finite tree with a nonterminal root.
  • State names are nonempty strings; legal actions are strictly increasing nonnegative integers.
  • Each nonterminal has matching transitions, logits, and a value prediction.
  • Players alternate across each edge; neural and terminal values lie between -1 and 1.
  • simulations is a positive integer; cpuct and temperature are finite and nonnegative.
Try Similar Problems
Apply One Policy-Value Head Training StepHardApply a Dihedral Go SymmetryMediumSample a Recent Self-Play Replay BatchEasyApply a Go Action with Captures and RepetitionHardAssemble a Compact Dual-Head NetworkHardBuild the Legal Go Action MaskHardTurn Root Visits into a Search PolicyMediumCompute the Policy and Value HeadsMediumGRPO Group-Relative AdvantageHardPPO with Reference-Policy KLHardLinUCB Contextual BanditHardValue IterationMedium

Accepts: string

Accepts: object

Accepts: object

Accepts: object

Accepts: object

Accepts: object

Accepts: object

Accepts: number

Accepts: number

Accepts: number

You must run your code first.