Problems

1 / 1

Policy Gradient Loss

Reinforcement Learning

Medium

The REINFORCE algorithm uses the policy gradient theorem to directly optimize a stochastic policy. The loss is constructed so that gradient descent increases the probability of actions that led to higher-than-average returns and decreases the probability of actions that led to lower-than-average returns.

Given the log-probabilities of the actions taken, the rewards at each timestep, and a discount factor gamma, compute the policy gradient loss with a mean-return baseline.

Algorithm

Compute discounted returns backward:

G_{T-1} = r_{T-1}

G_t = r_t + \gamma G_{t+1}

Subtract the mean return as a baseline to reduce variance:

\bar{G} = \frac{1}{T} \sum_{t=0}^{T-1} G_t

A_t = G_t - \bar{G}

Compute the loss (negative because we want gradient ascent on expected return):

L = -\frac{1}{T} \sum_{t=0}^{T-1} \log \pi(a_t | s_t) \cdot A_t

Return the policy-gradient loss as a float.

Loading visualization...

Examples

Input: log_probs = [-1, -2, -0.5], rewards = [1, 1, 1], gamma = 1

Output: 0.166667

Explanation: Returns [3, 2, 1] give centered advantages [1, 0, -1].

Input: log_probs = [-1, -0.5], rewards = [0, 10], gamma = 0

Output: -1.25

Hint 1

Build discounted returns from right to left before computing their mean.

Hint 2

Multiply each log probability by its centered return and negate the average.

Requirements

Compute discounted returns using backward recursion
Subtract the mean return from each return to get advantages
Compute loss as the negative mean of (log_prob * advantage)
Return a single float

Constraints

log_probs and rewards have the same length (at least 1)
log_probs contains negative floats (log of probabilities)
0 <= gamma <= 1
Return a single float
Time limit: 300 ms

Try Similar Problems

Compute Advantage Gae Computation Discount Returns Q Learning Update Cross Entropy Loss

Prev Next

Policy Gradient Loss

Reinforcement Learning

Medium

Given the log-probabilities of the actions taken, the rewards at each timestep, and a discount factor gamma, compute the policy gradient loss with a mean-return baseline.

Algorithm

Compute discounted returns backward:

G_{T-1} = r_{T-1}

G_t = r_t + \gamma G_{t+1}

Subtract the mean return as a baseline to reduce variance:

\bar{G} = \frac{1}{T} \sum_{t=0}^{T-1} G_t

A_t = G_t - \bar{G}

Compute the loss (negative because we want gradient ascent on expected return):

L = -\frac{1}{T} \sum_{t=0}^{T-1} \log \pi(a_t | s_t) \cdot A_t

Return the policy-gradient loss as a float.

Loading visualization...

Examples

Input: log_probs = [-1, -2, -0.5], rewards = [1, 1, 1], gamma = 1

Output: 0.166667

Explanation: Returns [3, 2, 1] give centered advantages [1, 0, -1].

Input: log_probs = [-1, -0.5], rewards = [0, 10], gamma = 0

Output: -1.25

Hint 1

Build discounted returns from right to left before computing their mean.

Hint 2

Multiply each log probability by its centered return and negate the average.

Requirements

Compute discounted returns using backward recursion
Subtract the mean return from each return to get advantages
Compute loss as the negative mean of (log_prob * advantage)
Return a single float

Constraints

log_probs and rewards have the same length (at least 1)
log_probs contains negative floats (log of probabilities)
0 <= gamma <= 1
Return a single float
Time limit: 300 ms

Try Similar Problems

Compute Advantage Gae Computation Discount Returns Q Learning Update Cross Entropy Loss

Case 1

Case 2

log_probs =

Accepts: array

rewards =

Accepts: array

gamma =

Accepts: number

Given the log-probabilities of the actions taken, the rewards at each timestep, and a discount factor gamma, compute the policy gradient loss with a mean-return baseline.

Algorithm

Compute discounted returns backward:

G_{T-1} = r_{T-1}

G_t = r_t + \gamma G_{t+1}

Subtract the mean return as a baseline to reduce variance:

\bar{G} = \frac{1}{T} \sum_{t=0}^{T-1} G_t

A_t = G_t - \bar{G}

Compute the loss (negative because we want gradient ascent on expected return):

L = -\frac{1}{T} \sum_{t=0}^{T-1} \log \pi(a_t | s_t) \cdot A_t

Return the policy-gradient loss as a float.