Implement a one-step TD(0) update for a state value function given a single transition sample.
TD Error:
δ=r+γV(snext)−V(s)Value Update:
V(s)←V(s)+αδwhere α is the learning rate and γ is the discount factor.
V: 1D array, shape (n_states,) - current value estimatess: int - current state indexr: float - reward observeds_next: int - next state indexalpha: float - learning rate (0<α≤1)gamma: float - discount factor (0≤γ≤1)Input: V=[0,0,0], s=0, r=1, s_next=1, α=0.5, γ=0.9
Output: V_new = [0.5, 0.0, 0.0]
Input: V=[0.2,0.8], s=0, r=0, s_next=1, α=1.0, γ=0.5
Output: V_new = [0.4, 0.8]
First compute the TD error δ=r+γV(snext)−V(s).
Use V.copy() to avoid modifying the input, then update V_new[s] with the TD error.
Sign in to take notes on this problem
Accepts: array
Accepts: number
Accepts: number
Accepts: number
Accepts: number
Accepts: number
Implement a one-step TD(0) update for a state value function given a single transition sample.
TD Error:
δ=r+γV(snext)−V(s)Value Update:
V(s)←V(s)+αδwhere α is the learning rate and γ is the discount factor.
V: 1D array, shape (n_states,) - current value estimatess: int - current state indexr: float - reward observeds_next: int - next state indexalpha: float - learning rate (0<α≤1)gamma: float - discount factor (0≤γ≤1)Input: V=[0,0,0], s=0, r=1, s_next=1, α=0.5, γ=0.9
Output: V_new = [0.5, 0.0, 0.0]
Input: V=[0.2,0.8], s=0, r=0, s_next=1, α=1.0, γ=0.5
Output: V_new = [0.4, 0.8]
First compute the TD error δ=r+γV(snext)−V(s).
Use V.copy() to avoid modifying the input, then update V_new[s] with the TD error.
Sign in to take notes on this problem
Accepts: array
Accepts: number
Accepts: number
Accepts: number
Accepts: number
Accepts: number