Problems

1 / 1

Prioritized Experience Replay

Reinforcement Learning

Easy

Prioritized Experience Replay (PER) improves on uniform replay by sampling transitions proportional to their TD error magnitude. Transitions where the agent was most "surprised" (large TD error) are replayed more often, leading to faster learning. To correct for the bias introduced by non-uniform sampling, importance sampling weights are applied.

Given a list of priority values, an alpha parameter (controls how much prioritization is used), and a beta parameter (controls importance sampling correction), compute the sampling probabilities and normalized importance sampling weights.

Algorithm

Compute powered priorities:

p_i^{\alpha}

Compute sampling probabilities:

P(i) = \frac{p_i^{\alpha}}{\sum_j p_j^{\alpha}}

Compute raw importance sampling weights:

w_i = (N \cdot P(i))^{-\beta}

Normalize weights by the maximum weight:

\hat{w}_i = \frac{w_i}{\max_j w_j}

Return a two-item list containing probabilities followed by normalized weights.

Loading visualization...

Examples

Input: priorities = [1, 2, 3], alpha = 1, beta = 1

Output: [[0.166667, 0.333333, 0.5], [1.0, 0.5, 0.333333]]

Explanation: Probabilities follow the priorities, while inverse-probability weights reduce the influence of frequently sampled items.

Input: priorities = [1, 1, 1], alpha = 1, beta = 1

Output: [[0.333333, 0.333333, 0.333333], [1.0, 1.0, 1.0]]

Hint 1

Normalize priorities after raising each one to alpha.

Hint 2

Compute each raw weight from its probability, then divide all weights by their maximum.

Requirements

Raise each priority to the power alpha to control prioritization strength
Compute sampling probabilities by normalizing the powered priorities
Compute importance sampling weights as (N * P(i))^(-beta)
Normalize weights by dividing by the maximum weight
Return a list of two lists: [probabilities, normalized_weights]

Constraints

priorities has at least one element with all positive values
alpha >= 0, beta >= 0
Return [probabilities, normalized_weights] where each is a list of floats
Time limit: 300 ms

Try Similar Problems

Replay Buffer Sample Q Learning Update Sarsa Update Epsilon Greedy Td Value Update

Prev Next

Prioritized Experience Replay

Reinforcement Learning

Easy

Algorithm

Compute powered priorities:

p_i^{\alpha}

Compute sampling probabilities:

P(i) = \frac{p_i^{\alpha}}{\sum_j p_j^{\alpha}}

Compute raw importance sampling weights:

w_i = (N \cdot P(i))^{-\beta}

Normalize weights by the maximum weight:

\hat{w}_i = \frac{w_i}{\max_j w_j}

Return a two-item list containing probabilities followed by normalized weights.

Loading visualization...

Examples

Input: priorities = [1, 2, 3], alpha = 1, beta = 1

Output: [[0.166667, 0.333333, 0.5], [1.0, 0.5, 0.333333]]

Explanation: Probabilities follow the priorities, while inverse-probability weights reduce the influence of frequently sampled items.

Input: priorities = [1, 1, 1], alpha = 1, beta = 1

Output: [[0.333333, 0.333333, 0.333333], [1.0, 1.0, 1.0]]

Hint 1

Normalize priorities after raising each one to alpha.

Hint 2

Compute each raw weight from its probability, then divide all weights by their maximum.

Requirements

Raise each priority to the power alpha to control prioritization strength
Compute sampling probabilities by normalizing the powered priorities
Compute importance sampling weights as (N * P(i))^(-beta)
Normalize weights by dividing by the maximum weight
Return a list of two lists: [probabilities, normalized_weights]

Constraints

priorities has at least one element with all positive values
alpha >= 0, beta >= 0
Return [probabilities, normalized_weights] where each is a list of floats
Time limit: 300 ms

Try Similar Problems

Replay Buffer Sample Q Learning Update Sarsa Update Epsilon Greedy Td Value Update

Case 1

Case 2

priorities =

Accepts: array

alpha =

Accepts: number

beta =

Accepts: number

Algorithm

Compute powered priorities:

p_i^{\alpha}

Compute sampling probabilities:

P(i) = \frac{p_i^{\alpha}}{\sum_j p_j^{\alpha}}

Compute raw importance sampling weights:

w_i = (N \cdot P(i))^{-\beta}

Normalize weights by the maximum weight:

\hat{w}_i = \frac{w_i}{\max_j w_j}

Return a two-item list containing probabilities followed by normalized weights.