TensorTonicTensorTonic
Problems
Study PlansProjectsInterviewPricingFeedback
Problems
Loading...
1 / 1

Implement AdamW (Decoupled Weight Decay)

Optimization
Easy

Implement one update step of AdamW optimizer. Given current parameters, moments, and gradients, return updated parameters and moments using decoupled weight decay.

Step 1: Update First Moment

mt=β1mt−1+(1−β1)gtm_t = \beta_1 m_{t-1} + (1 - \beta_1) g_tmt​=β1​mt−1​+(1−β1​)gt​

Step 2: Update Second Moment

vt=β2vt−1+(1−β2)gt2v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2vt​=β2​vt−1​+(1−β2​)gt2​

Step 3: AdamW Parameter Update

wt=wt−1−η(decay⋅wt−1)−η⋅mtvt+εw_t = w_{t-1} - \eta(\text{decay} \cdot w_{t-1}) - \eta \cdot \frac{m_t}{\sqrt{v_t} + \varepsilon}wt​=wt−1​−η(decay⋅wt−1​)−η⋅vt​​+εmt​​

Where: w = parameters, m = first moment, v = second moment, g = gradients, \u03b7 = learning rate, decay = weight decay

Loading visualization...

Examples

Input: w=[1.0, -2.0], m=[0, 0], v=[0, 0], grad=[0.3, -0.7], lr=0.01, weight_decay=0.1

Output: ([0.967, -1.966], [0.03, -0.07], [0.00009, 0.00049])

Input: w=[1.0, 2.0], m=[0.1, 0.2], v=[0.01, 0.04], grad=[0, 0], lr=0.01, weight_decay=0.1

Output: ([0.99, 1.989], [0.09, 0.18], [0.00999, 0.03996])

Hint 1

Convert inputs to NumPy arrays first. Update moments using exponential moving averages.

Hint 2

The parameter update has two parts: weight decay term and adaptive gradient term. Apply both simultaneously.

Requirements

  • Return tuple (new_w, new_m, new_v) with same shapes as inputs
  • Vectorized implementation only (no Python loops)
  • Handle any array shape (1D, 2D, etc.)

Constraints

  • 0 < beta1, beta2 < 1
  • lr > 0, weight_decay ≥ 0
  • Libraries: NumPy only
Try Similar Problems
Adam OptimizerNesterov MomentumNadam OptimizerAdagrad OptimizerRmsprop Optimizer

Sign in to take notes on this problem

Case 1
Case 2
Case 3
Case 4

Accepts: array

Accepts: array

Accepts: array

Accepts: array

Accepts: number

Accepts: number

Accepts: number

Accepts: number

Accepts: number

You must run your code first.
PrevNext

Implement AdamW (Decoupled Weight Decay)

Optimization
Easy

Implement one update step of AdamW optimizer. Given current parameters, moments, and gradients, return updated parameters and moments using decoupled weight decay.

Step 1: Update First Moment

mt=β1mt−1+(1−β1)gtm_t = \beta_1 m_{t-1} + (1 - \beta_1) g_tmt​=β1​mt−1​+(1−β1​)gt​

Step 2: Update Second Moment

vt=β2vt−1+(1−β2)gt2v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2vt​=β2​vt−1​+(1−β2​)gt2​

Step 3: AdamW Parameter Update

wt=wt−1−η(decay⋅wt−1)−η⋅mtvt+εw_t = w_{t-1} - \eta(\text{decay} \cdot w_{t-1}) - \eta \cdot \frac{m_t}{\sqrt{v_t} + \varepsilon}wt​=wt−1​−η(decay⋅wt−1​)−η⋅vt​​+εmt​​

Where: w = parameters, m = first moment, v = second moment, g = gradients, \u03b7 = learning rate, decay = weight decay

Loading visualization...

Examples

Input: w=[1.0, -2.0], m=[0, 0], v=[0, 0], grad=[0.3, -0.7], lr=0.01, weight_decay=0.1

Output: ([0.967, -1.966], [0.03, -0.07], [0.00009, 0.00049])

Input: w=[1.0, 2.0], m=[0.1, 0.2], v=[0.01, 0.04], grad=[0, 0], lr=0.01, weight_decay=0.1

Output: ([0.99, 1.989], [0.09, 0.18], [0.00999, 0.03996])

Hint 1

Convert inputs to NumPy arrays first. Update moments using exponential moving averages.

Hint 2

The parameter update has two parts: weight decay term and adaptive gradient term. Apply both simultaneously.

Requirements

  • Return tuple (new_w, new_m, new_v) with same shapes as inputs
  • Vectorized implementation only (no Python loops)
  • Handle any array shape (1D, 2D, etc.)

Constraints

  • 0 < beta1, beta2 < 1
  • lr > 0, weight_decay ≥ 0
  • Libraries: NumPy only
Try Similar Problems
Adam OptimizerNesterov MomentumNadam OptimizerAdagrad OptimizerRmsprop Optimizer

Sign in to take notes on this problem

Case 1
Case 2
Case 3
Case 4

Accepts: array

Accepts: array

Accepts: array

Accepts: array

Accepts: number

Accepts: number

Accepts: number

Accepts: number

Accepts: number

You must run your code first.