Problems

1 / 1

Advantage Computation

Reinforcement Learning

Easy

Compute the advantage for every time step in an episode.

Mathematical Definition

Return from time t:

G_t = r_t + \gamma r_{t+1} + \gamma^{2} r_{t+2} + \cdots

Advantage:

A_t = G_t - V(s_t)

where A_t measures how much better the actual return was compared to expected value.

Function Arguments

states: list of ints - state at each timestep
rewards: list of floats - reward at each timestep
V: NumPy array - state values
gamma: float ∈ [0,1] - discount factor

Loading visualization...

Examples

Input: states=[0,1,2], rewards=[1,2,3], V=[0.5,1,1.5], γ=1

Output: A = [5.5, 4.0, 1.5]

Input: states=[0,1,2], rewards=[1,2,3], V=[0,0,0], γ=0

Output: A = [1.0, 2.0, 3.0]

Hint 1

Compute returns backward: start from the last timestep where G_T=r_T then work backward.

Hint 2

Once you have all returns, compute advantage as $A_t = G_t - V(s_t)$ for each timestep.

Requirements

Compute returns backward for efficiency
Do not modify inputs
Return NumPy array of advantages

Constraints

Episode length ≤ 10,000
NumPy only
Time limit: 200ms

Try Similar Problems

Gae Computation Policy Gradient Loss Discount Returns Td Value Update Q Learning Update

Prev Next

Advantage Computation

Reinforcement Learning

Easy

Compute the advantage for every time step in an episode.

Mathematical Definition

Return from time t:

G_t = r_t + \gamma r_{t+1} + \gamma^{2} r_{t+2} + \cdots

Advantage:

A_t = G_t - V(s_t)

where A_t measures how much better the actual return was compared to expected value.

Function Arguments

states: list of ints - state at each timestep
rewards: list of floats - reward at each timestep
V: NumPy array - state values
gamma: float ∈ [0,1] - discount factor

Loading visualization...

Examples

Input: states=[0,1,2], rewards=[1,2,3], V=[0.5,1,1.5], γ=1

Output: A = [5.5, 4.0, 1.5]

Input: states=[0,1,2], rewards=[1,2,3], V=[0,0,0], γ=0

Output: A = [1.0, 2.0, 3.0]

Hint 1

Compute returns backward: start from the last timestep where G_T=r_T then work backward.

Hint 2

Once you have all returns, compute advantage as $A_t = G_t - V(s_t)$ for each timestep.

Requirements

Compute returns backward for efficiency
Do not modify inputs
Return NumPy array of advantages

Constraints

Episode length ≤ 10,000
NumPy only
Time limit: 200ms

Try Similar Problems

Gae Computation Policy Gradient Loss Discount Returns Td Value Update Q Learning Update

Case 1

Case 2

states =

Accepts: array

rewards =

Accepts: array

V =

Accepts: array

gamma =

Accepts: number