1 / 1

Generalized Advantage Estimation

Reinforcement Learning

Medium

Generalized Advantage Estimation (GAE) provides a family of advantage estimators that trade off bias and variance using a parameter lambda. When lambda = 0 it reduces to the one-step TD error (low variance, high bias). When lambda = 1 it becomes equivalent to the full Monte Carlo advantage (high variance, low bias). In practice, lambda around 0.95 works well.

Given rewards, value estimates (including V(s_T) = 0 for the terminal state), gamma, and lambda, compute the GAE advantages for each timestep.

Algorithm

Compute the TD error (delta) at each timestep:

\delta_t = r_t + \gamma \cdot V(s_{t+1}) - V(s_t)

Compute advantages backward using the recursive formula:

A_{T-1} = \delta_{T-1}

A_t = \delta_t + \gamma \cdot \lambda \cdot A_{t+1}

Return the advantages as a list of floats with the same length as rewards.

Loading visualization...

Examples

Input: rewards = [1, 1, 1], values = [0, 0, 0, 0], gamma = 1, lam = 1

Output: [3.0, 2.0, 1.0]

Explanation: With gamma and lambda equal to 1, future TD errors accumulate without decay.

Input: rewards = [1, 0, 5], values = [1, 2, 3, 0], gamma = 0.9, lam = 0.95

Output: [3.86055, 2.41, 2.0]

Hint 1

Traverse timesteps backward while carrying the next advantage.

Hint 2

At each step, add the TD error to gamma times lambda times the carried advantage.

Requirements

Compute TD errors: delta_t = rewards[t] + gamma * values[t+1] - values[t]
Compute advantages backward: A[t] = delta_t + gamma * lambda * A[t+1]
The values list has length T+1 (includes the terminal value)
Return a list of floats with length T (same as rewards)

Constraints

rewards has at least one element
values has len(rewards) + 1 elements
0 <= gamma <= 1, 0 <= lam <= 1
Return a list of floats with the same length as rewards
Time limit: 300 ms

Prev Next