The REINFORCE algorithm uses the policy gradient theorem to directly optimize a stochastic policy. The loss is constructed so that gradient descent increases the probability of actions that led to higher-than-average returns and decreases the probability of actions that led to lower-than-average returns.
Given the log-probabilities of the actions taken, the rewards at each timestep, and a discount factor gamma, compute the policy gradient loss with a mean-return baseline.
Return the policy-gradient loss as a float.
Input: log_probs = [-1, -2, -0.5], rewards = [1, 1, 1], gamma = 1
Output: 0.166667
Explanation: Returns [3, 2, 1] give centered advantages [1, 0, -1].
Input: log_probs = [-1, -0.5], rewards = [0, 10], gamma = 0
Output: -1.25
Build discounted returns from right to left before computing their mean.
Multiply each log probability by its centered return and negate the average.
Accepts: array
Accepts: array
Accepts: number
The REINFORCE algorithm uses the policy gradient theorem to directly optimize a stochastic policy. The loss is constructed so that gradient descent increases the probability of actions that led to higher-than-average returns and decreases the probability of actions that led to lower-than-average returns.
Given the log-probabilities of the actions taken, the rewards at each timestep, and a discount factor gamma, compute the policy gradient loss with a mean-return baseline.
Return the policy-gradient loss as a float.
Input: log_probs = [-1, -2, -0.5], rewards = [1, 1, 1], gamma = 1
Output: 0.166667
Explanation: Returns [3, 2, 1] give centered advantages [1, 0, -1].
Input: log_probs = [-1, -0.5], rewards = [0, 10], gamma = 0
Output: -1.25
Build discounted returns from right to left before computing their mean.
Multiply each log probability by its centered return and negate the average.
Accepts: array
Accepts: array
Accepts: number