Problems
Loading...
1 / 1

Implement Nadam (Nesterov + Adam)

Optimization
Medium

Implement one update step of Nadam optimizer. Given current parameters, moments, and gradients, return updated parameters and moments using Nesterov-accelerated Adam.

Step 1: Update First Moment

mt=β1mt−1+(1−β1)gtm_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t

Step 2: Update Second Moment

vt=β2vt−1+(1−β2)gt2v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2

Step 3: Nesterov-Adjusted Update

wt=wt−1−η⋅β1mt+(1−β1)gtvt+ϵw_t = w_{t-1} - \eta \cdot \frac{\beta_1 m_t + (1 - \beta_1) g_t}{\sqrt{v_t} + \epsilon}

Where: w = parameters, m = first moment, v = second moment, g = gradients, η = learning rate, β₁,β₂ = decay rates

Loading visualization...

Examples

Input: w=[1.0, -1.0], m=[0.1, -0.1], v=[0.01, 0.01], grad=[0.2, -0.3], lr=0.002

Output: ([0.998, -0.997], [0.11, -0.12], [0.01003, 0.01008])

Input: w=[1.0, 2.0], m=[0.1, 0.2], v=[0.01, 0.04], grad=[0, 0], lr=0.002

Output: ([0.998, 1.998], [0.09, 0.18], [0.00999, 0.03996])

Hint 1

Convert inputs to NumPy arrays first. Update moments using exponential moving averages like in Adam.

Hint 2

The Nesterov trick combines updated momentum with the current gradient for a look-ahead update in the numerator.

Requirements

  • Return tuple (w_new, m_new, v_new) with same shapes as inputs
  • Use the exact update formulas above (no bias correction)
  • Vectorized implementation only (no Python loops)
  • Handle any array shape (1D, 2D, etc.)

Constraints

  • 0 < beta1, beta2 < 1
  • lr > 0, eps > 0
  • Libraries: NumPy only
Try Similar Problems
Adam OptimizerAdamw OptimizerAdagrad OptimizerRmsprop OptimizerNesterov Momentum
PrevNext