A replay buffer (or experience replay memory) stores past transitions that an agent has experienced. During training, a random batch of transitions is sampled from the buffer to break temporal correlations and stabilize learning. This technique is used in DQN, SAC, DDPG, and many other off-policy algorithms.
Given a buffer of transitions, a batch size, and a random seed, sample a batch of transitions uniformly at random without replacement.
Set the random seed for reproducibility
Sample batch_size transitions from the buffer uniformly at random without replacement
Input:
buffer = [[0, 0, 1.0, 1, 0], [1, 1, 0.5, 2, 0], [2, 0, -1.0, 3, 1], [3, 1, 2.0, 4, 0], [4, 0, 0.0, 0, 1]], batch_size = 3, seed = 42
Output:
[[4, 0, 0.0, 0, 1], [2, 0, -1.0, 3, 1], [3, 1, 2.0, 4, 0]]
With seed 42, random.sample selects indices 4, 2, and 3 from the buffer. The result is a list of 3 transitions with no duplicates.
Input:
buffer = [[0, 0, 1.0, 1, 0], [1, 1, 0.5, 2, 0], [2, 0, -1.0, 3, 1], [3, 1, 2.0, 4, 0], [4, 0, 0.0, 0, 1]], batch_size = 1, seed = 7
Output:
[[4, 0, 0.0, 0, 1]]
A single transition is sampled from the buffer.
np.random.RandomState lets you create a seeded random number generator. Look into its choice method for sampling.
Think about whether the order of your sampled transitions matters for consistent output.
Sign in to take notes on this problem
Accepts: array
Accepts: number
Accepts: number
A replay buffer (or experience replay memory) stores past transitions that an agent has experienced. During training, a random batch of transitions is sampled from the buffer to break temporal correlations and stabilize learning. This technique is used in DQN, SAC, DDPG, and many other off-policy algorithms.
Given a buffer of transitions, a batch size, and a random seed, sample a batch of transitions uniformly at random without replacement.
Set the random seed for reproducibility
Sample batch_size transitions from the buffer uniformly at random without replacement
Input:
buffer = [[0, 0, 1.0, 1, 0], [1, 1, 0.5, 2, 0], [2, 0, -1.0, 3, 1], [3, 1, 2.0, 4, 0], [4, 0, 0.0, 0, 1]], batch_size = 3, seed = 42
Output:
[[4, 0, 0.0, 0, 1], [2, 0, -1.0, 3, 1], [3, 1, 2.0, 4, 0]]
With seed 42, random.sample selects indices 4, 2, and 3 from the buffer. The result is a list of 3 transitions with no duplicates.
Input:
buffer = [[0, 0, 1.0, 1, 0], [1, 1, 0.5, 2, 0], [2, 0, -1.0, 3, 1], [3, 1, 2.0, 4, 0], [4, 0, 0.0, 0, 1]], batch_size = 1, seed = 7
Output:
[[4, 0, 0.0, 0, 1]]
A single transition is sampled from the buffer.
np.random.RandomState lets you create a seeded random number generator. Look into its choice method for sampling.
Think about whether the order of your sampled transitions matters for consistent output.
Sign in to take notes on this problem
Accepts: array
Accepts: number
Accepts: number