TensorTonicTensorTonic
Problems
Study PlansProjectsNewInterviewPricingFeedback
Problems
Loading...
1 / 1

Implement AdamW (Decoupled Weight Decay)

Optimization
Easy

Perform one AdamW step without bias correction. Update the first and second moments:

mt=β1mt−1+(1−β1)gtm_t = \beta_1 m_{t-1} + (1-\beta_1)g_tmt​=β1​mt−1​+(1−β1​)gt​ vt=β2vt−1+(1−β2)gt2v_t = \beta_2 v_{t-1} + (1-\beta_2)g_t^2vt​=β2​vt−1​+(1−β2​)gt2​

Then apply the adaptive update and decoupled weight decay:

wt=wt−1−ηmtvt+ε−ηλwt−1w_t = w_{t-1} - \eta\frac{m_t}{\sqrt{v_t}+\varepsilon} - \eta\lambda w_{t-1}wt​=wt−1​−ηvt​​+εmt​​−ηλwt−1​

Here, www contains parameters, ggg contains gradients, mmm and vvv are moment arrays, η\etaη is lr, λ\lambdaλ is weight_decay, and ε\varepsilonε is eps. Return new_w, new_m, and new_v in a dictionary of NumPy arrays.

Loading visualization...

Examples

Input: w = [1.0, -2.0], m = [0.0, 0.0], v = [0.0, 0.0], grad = [0.3, -0.7], lr = 0.01, beta1 = 0.9, beta2 = 0.999, weight_decay = 0.1, eps = 1e-8

Output: {"new_w": [0.967377, -1.966377], "new_m": [0.03, -0.07], "new_v": [0.00009, 0.00049]}

Explanation: Adam's adaptive step and the independent decay term both change each parameter.

Input: w = [5.0], m = [0.1], v = [0.01], grad = [0.2], lr = 0.01, beta1 = 0.9, beta2 = 0.999, weight_decay = 0.05, eps = 1e-8

Output: {"new_w": [4.986516], "new_m": [0.11], "new_v": [0.01003]}

Input: w = [1.0, 2.0], m = [0.1, 0.2], v = [0.01, 0.04], grad = [0.0, 0.0], lr = 0.01, beta1 = 0.9, beta2 = 0.999, weight_decay = 0.1, eps = 1e-8

Output: {"new_w": [0.989995, 1.988995], "new_m": [0.09, 0.18], "new_v": [0.00999, 0.03996]}

Hint 1

Update new_m and new_v with their exponential moving averages.

Hint 2

Subtract both lr * new_m / (np.sqrt(new_v) + eps) and lr * weight_decay * w.

Requirements

  • Apply the stated AdamW equations without bias correction
  • Keep weight decay separate from the adaptive gradient term
  • Return exactly new_w, new_m, and new_v in a dictionary
  • Every returned value must be a NumPy array

Constraints

  • w, m, v, and grad have the same nonempty numeric shape
  • lr is positive and weight_decay is nonnegative
  • beta1 and beta2 are at least 0 and less than 1
  • eps is positive
  • Use NumPy only
Try Similar Problems
Adam OptimizerNesterov MomentumNadam OptimizerAdagrad OptimizerRmsprop Optimizer

Sign in to take notes on this problem

Case 1
Case 2
Case 3

Accepts: array

Accepts: array

Accepts: array

Accepts: array

Accepts: number

Accepts: number

Accepts: number

Accepts: number

Accepts: number

You must run your code first.
PrevNext

Implement AdamW (Decoupled Weight Decay)

Optimization
Easy

Perform one AdamW step without bias correction. Update the first and second moments:

mt=β1mt−1+(1−β1)gtm_t = \beta_1 m_{t-1} + (1-\beta_1)g_tmt​=β1​mt−1​+(1−β1​)gt​ vt=β2vt−1+(1−β2)gt2v_t = \beta_2 v_{t-1} + (1-\beta_2)g_t^2vt​=β2​vt−1​+(1−β2​)gt2​

Then apply the adaptive update and decoupled weight decay:

wt=wt−1−ηmtvt+ε−ηλwt−1w_t = w_{t-1} - \eta\frac{m_t}{\sqrt{v_t}+\varepsilon} - \eta\lambda w_{t-1}wt​=wt−1​−ηvt​​+εmt​​−ηλwt−1​

Here, www contains parameters, ggg contains gradients, mmm and vvv are moment arrays, η\etaη is lr, λ\lambdaλ is weight_decay, and ε\varepsilonε is eps. Return new_w, new_m, and new_v in a dictionary of NumPy arrays.

Loading visualization...

Examples

Input: w = [1.0, -2.0], m = [0.0, 0.0], v = [0.0, 0.0], grad = [0.3, -0.7], lr = 0.01, beta1 = 0.9, beta2 = 0.999, weight_decay = 0.1, eps = 1e-8

Output: {"new_w": [0.967377, -1.966377], "new_m": [0.03, -0.07], "new_v": [0.00009, 0.00049]}

Explanation: Adam's adaptive step and the independent decay term both change each parameter.

Input: w = [5.0], m = [0.1], v = [0.01], grad = [0.2], lr = 0.01, beta1 = 0.9, beta2 = 0.999, weight_decay = 0.05, eps = 1e-8

Output: {"new_w": [4.986516], "new_m": [0.11], "new_v": [0.01003]}

Input: w = [1.0, 2.0], m = [0.1, 0.2], v = [0.01, 0.04], grad = [0.0, 0.0], lr = 0.01, beta1 = 0.9, beta2 = 0.999, weight_decay = 0.1, eps = 1e-8

Output: {"new_w": [0.989995, 1.988995], "new_m": [0.09, 0.18], "new_v": [0.00999, 0.03996]}

Hint 1

Update new_m and new_v with their exponential moving averages.

Hint 2

Subtract both lr * new_m / (np.sqrt(new_v) + eps) and lr * weight_decay * w.

Requirements

  • Apply the stated AdamW equations without bias correction
  • Keep weight decay separate from the adaptive gradient term
  • Return exactly new_w, new_m, and new_v in a dictionary
  • Every returned value must be a NumPy array

Constraints

  • w, m, v, and grad have the same nonempty numeric shape
  • lr is positive and weight_decay is nonnegative
  • beta1 and beta2 are at least 0 and less than 1
  • eps is positive
  • Use NumPy only
Try Similar Problems
Adam OptimizerNesterov MomentumNadam OptimizerAdagrad OptimizerRmsprop Optimizer

Sign in to take notes on this problem

Case 1
Case 2
Case 3

Accepts: array

Accepts: array

Accepts: array

Accepts: array

Accepts: number

Accepts: number

Accepts: number

Accepts: number

Accepts: number

You must run your code first.