Problems
Loading...
1 / 1

Implement Adam Optimizer Step

Optimization
Easy

Implement one update step of the Adam optimizer. Given current parameter(s), gradient(s), and running first/second moments, return the updated parameter(s) and updated moments.

Step 1: Update First Moment

mt=β1⋅mt−1+(1−β1)⋅gtm_t = \beta_1 \cdot m_{t-1} + (1 - \beta_1) \cdot g_t

Step 2: Update Second Moment

vt=β2⋅vt−1+(1−β2)⋅gt2v_t = \beta_2 \cdot v_{t-1} + (1 - \beta_2) \cdot g_t^2

Step 3: Bias Correction

m^t=mt1−β1t,v^t=vt1−β2t\hat{m}_t = \frac{m_t}{1 - \beta_1^t}, \quad \hat{v}_t = \frac{v_t}{1 - \beta_2^t}

Step 4: Parameter Update

θt=θt−1−α⋅m^tv^t+ϵ\theta_t = \theta_{t-1} - \alpha \cdot \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}

Where: \u03b8 = parameters, m = first moment, v = second moment, g = gradients, \u03b1 = learning rate, t = timestep (1-based)

Loading visualization...

Examples

Input: param=[1.0, 2.0], m=[0, 0], v=[0, 0], grad=[0, 0], t=1, lr=0.001

Output: ([1.0, 2.0], [0.0, 0.0], [0.0, 0.0])

Input: param=[0.0], m=[0], v=[0], grad=[0.1], t=1, lr=0.001

Output: ([-0.001], [0.01], [0.00001])

Hint 1

Update m and v first before computing bias-corrected m̂ and v̂.

Hint 2

Use 1-based t in bias correction (1 - β^t) and ensure ε is added in the denominator for numerical stability.

Requirements

  • Accept scalars or NumPy arrays; operations must be elementwise and vectorized
  • Do bias correction using the provided t (1-based)
  • Return a tuple (param_new, m_new, v_new) with the same shapes as param, m, v
  • No external ML libraries

Constraints

  • Inputs up to shape ~ (10⁵)
  • Time limit: 500 ms; Memory: 128 MB
  • Allowed library: NumPy only
Try Similar Problems
Adamw OptimizerNadam OptimizerRmsprop OptimizerAdagrad OptimizerNesterov Momentum
PrevNext