Problems

1 / 1

Generalized Advantage Estimation

Reinforcement Learning

Medium

Generalized Advantage Estimation (GAE) provides a family of advantage estimators that trade off bias and variance using a parameter lambda. When lambda = 0 it reduces to the one-step TD error (low variance, high bias). When lambda = 1 it becomes equivalent to the full Monte Carlo advantage (high variance, low bias). In practice, lambda around 0.95 works well.

Given rewards, value estimates (including V(s_T) = 0 for the terminal state), gamma, and lambda, compute the GAE advantages for each timestep.

Algorithm

Compute the TD error (delta) at each timestep:

\delta_t = r_t + \gamma \cdot V(s_{t+1}) - V(s_t)

Compute advantages backward using the recursive formula:

A_{T-1} = \delta_{T-1}

A_t = \delta_t + \gamma \cdot \lambda \cdot A_{t+1}

Return the advantages as a list of floats with the same length as rewards.

Loading visualization...

Examples

Input: rewards = [1, 1, 1], values = [0, 0, 0, 0], gamma = 1, lam = 1

Output: [3.0, 2.0, 1.0]

Explanation: With gamma and lambda equal to 1, future TD errors accumulate without decay.

Input: rewards = [1, 0, 5], values = [1, 2, 3, 0], gamma = 0.9, lam = 0.95

Output: [3.86055, 2.41, 2.0]

Hint 1

Traverse timesteps backward while carrying the next advantage.

Hint 2

At each step, add the TD error to gamma times lambda times the carried advantage.

Requirements

Compute TD errors: delta_t = rewards[t] + gamma * values[t+1] - values[t]
Compute advantages backward: A[t] = delta_t + gamma * lambda * A[t+1]
The values list has length T+1 (includes the terminal value)
Return a list of floats with length T (same as rewards)

Constraints

rewards has at least one element
values has len(rewards) + 1 elements
0 <= gamma <= 1, 0 <= lam <= 1
Return a list of floats with the same length as rewards
Time limit: 300 ms

Try Similar Problems

Compute Advantage Policy Gradient Loss Discount Returns Td Value Update Value Iteration Step

Prev Next

Generalized Advantage Estimation

Reinforcement Learning

Medium

Given rewards, value estimates (including V(s_T) = 0 for the terminal state), gamma, and lambda, compute the GAE advantages for each timestep.

Algorithm

Compute the TD error (delta) at each timestep:

\delta_t = r_t + \gamma \cdot V(s_{t+1}) - V(s_t)

Compute advantages backward using the recursive formula:

A_{T-1} = \delta_{T-1}

A_t = \delta_t + \gamma \cdot \lambda \cdot A_{t+1}

Return the advantages as a list of floats with the same length as rewards.

Loading visualization...

Examples

Input: rewards = [1, 1, 1], values = [0, 0, 0, 0], gamma = 1, lam = 1

Output: [3.0, 2.0, 1.0]

Explanation: With gamma and lambda equal to 1, future TD errors accumulate without decay.

Input: rewards = [1, 0, 5], values = [1, 2, 3, 0], gamma = 0.9, lam = 0.95

Output: [3.86055, 2.41, 2.0]

Hint 1

Traverse timesteps backward while carrying the next advantage.

Hint 2

At each step, add the TD error to gamma times lambda times the carried advantage.

Requirements

Compute TD errors: delta_t = rewards[t] + gamma * values[t+1] - values[t]
Compute advantages backward: A[t] = delta_t + gamma * lambda * A[t+1]
The values list has length T+1 (includes the terminal value)
Return a list of floats with length T (same as rewards)

Constraints

rewards has at least one element
values has len(rewards) + 1 elements
0 <= gamma <= 1, 0 <= lam <= 1
Return a list of floats with the same length as rewards
Time limit: 300 ms

Try Similar Problems

Compute Advantage Policy Gradient Loss Discount Returns Td Value Update Value Iteration Step

Case 1

Case 2

rewards =

Accepts: array

values =

Accepts: array

gamma =

Accepts: number

lam =

Accepts: number

Given rewards, value estimates (including V(s_T) = 0 for the terminal state), gamma, and lambda, compute the GAE advantages for each timestep.

Algorithm

Compute the TD error (delta) at each timestep:

\delta_t = r_t + \gamma \cdot V(s_{t+1}) - V(s_t)

Compute advantages backward using the recursive formula:

A_{T-1} = \delta_{T-1}

A_t = \delta_t + \gamma \cdot \lambda \cdot A_{t+1}

Return the advantages as a list of floats with the same length as rewards.