SARSA is an on-policy temporal difference (TD) learning algorithm. The name comes from the quintuple (State, Action, Reward, next State, next Action) used in each update. It learns Q-values (action-value estimates) by bootstrapping from the next state-action pair that the agent actually takes.
Given a Q-table and a single transition (s, a, r, s', a'), perform one SARSA update and return the updated Q-table.
Input:
q_table = [[0, 0], [0, 0]], state = 0, action = 1, reward = 1.0, next_state = 1, next_action = 0, alpha = 0.1, gamma = 0.9
Output:
[[0.0, 0.1], [0.0, 0.0]]
TD error = 1.0 + 0.9 * 0 - 0 = 1.0. Update: Q(0,1) = 0 + 0.1 * 1.0 = 0.1. All other Q-values stay the same.
Input:
q_table = [[1, 2], [3, 4]], state = 0, action = 0, reward = 5.0, next_state = 1, next_action = 1, alpha = 0.5, gamma = 0.9
Output:
[[4.8, 2.0], [3.0, 4.0]]
TD error = 5.0 + 0.9 * 4 - 1 = 7.6. Update: Q(0,0) = 1 + 0.5 * 7.6 = 4.8.
First make a deep copy of the Q-table (copy each row). Then compute td = reward + gamma * q_table[next_state][next_action] - q_table[state][action]. Finally update the copy: new_q[state][action] += alpha * td.
Make sure to use the original Q-table values when computing the TD error, not the copy. The copy is only for writing the update.
Sign in to take notes on this problem
Accepts: array
Accepts: number
Accepts: number
Accepts: number
Accepts: number
Accepts: number
Accepts: number
Accepts: number
SARSA is an on-policy temporal difference (TD) learning algorithm. The name comes from the quintuple (State, Action, Reward, next State, next Action) used in each update. It learns Q-values (action-value estimates) by bootstrapping from the next state-action pair that the agent actually takes.
Given a Q-table and a single transition (s, a, r, s', a'), perform one SARSA update and return the updated Q-table.
Input:
q_table = [[0, 0], [0, 0]], state = 0, action = 1, reward = 1.0, next_state = 1, next_action = 0, alpha = 0.1, gamma = 0.9
Output:
[[0.0, 0.1], [0.0, 0.0]]
TD error = 1.0 + 0.9 * 0 - 0 = 1.0. Update: Q(0,1) = 0 + 0.1 * 1.0 = 0.1. All other Q-values stay the same.
Input:
q_table = [[1, 2], [3, 4]], state = 0, action = 0, reward = 5.0, next_state = 1, next_action = 1, alpha = 0.5, gamma = 0.9
Output:
[[4.8, 2.0], [3.0, 4.0]]
TD error = 5.0 + 0.9 * 4 - 1 = 7.6. Update: Q(0,0) = 1 + 0.5 * 7.6 = 4.8.
First make a deep copy of the Q-table (copy each row). Then compute td = reward + gamma * q_table[next_state][next_action] - q_table[state][action]. Finally update the copy: new_q[state][action] += alpha * td.
Make sure to use the original Q-table values when computing the TD error, not the copy. The copy is only for writing the update.
Sign in to take notes on this problem
Accepts: array
Accepts: number
Accepts: number
Accepts: number
Accepts: number
Accepts: number
Accepts: number
Accepts: number