TensorTonicTensorTonic
Problems
Study PlansProjectsNewInterviewPricingFeedback
Problems
Loading...
1 / 1

Implement Nadam (Nesterov + Adam)

Optimization
Medium

Perform one Nadam update without bias correction. Update the moments:

mt=β1mt−1+(1−β1)gtm_t = \beta_1m_{t-1} + (1-\beta_1)g_tmt​=β1​mt−1​+(1−β1​)gt​ vt=β2vt−1+(1−β2)gt2v_t = \beta_2v_{t-1} + (1-\beta_2)g_t^2vt​=β2​vt−1​+(1−β2​)gt2​

Form the Nesterov-adjusted first moment:

m~t=β1mt+(1−β1)gt\widetilde{m}_t = \beta_1m_t + (1-\beta_1)g_tmt​=β1​mt​+(1−β1​)gt​

Then update the parameters:

wt=wt−1−ηm~tvt+εw_t = w_{t-1} - \eta\frac{\widetilde{m}_t}{\sqrt{v_t}+\varepsilon}wt​=wt−1​−ηvt​​+εmt​​

Here, www contains parameters, ggg contains gradients, mmm and vvv are moment arrays, and η\etaη is lr. Return new_w, new_m, and new_v in a dictionary of NumPy arrays.

Loading visualization...

Examples

Input: w = [1.0, -1.0], m = [0.1, -0.1], v = [0.01, 0.01], grad = [0.2, -0.3], lr = 0.002, beta1 = 0.9, beta2 = 0.999, eps = 1e-8

Output: {"new_w": [0.997624, -0.997251], "new_m": [0.11, -0.12], "new_v": [0.01003, 0.01008]}

Explanation: The updated first moment is combined with the current gradient before adaptive scaling.

Input: w = [1.0, 2.0], m = [0.1, 0.2], v = [0.01, 0.04], grad = [0.0, 0.0], lr = 0.002, beta1 = 0.9, beta2 = 0.999, eps = 1e-8

Output: {"new_w": [0.998379, 1.998379], "new_m": [0.09, 0.18], "new_v": [0.00999, 0.03996]}

Input: w = [1.0, 2.0], m = [0.0, 0.0], v = [0.0, 0.0], grad = [0.1, 0.2], lr = 0.002, beta1 = 0.9, beta2 = 0.999, eps = 1e-8

Output: {"new_w": [0.987983, 1.987983], "new_m": [0.01, 0.02], "new_v": [0.00001, 0.00004]}

Hint 1

Compute the ordinary first and second moments before nesterov_m.

Hint 2

Use beta1 * new_m + (1 - beta1) * grad in the numerator.

Requirements

  • Apply the stated Nadam equations without bias correction
  • Compute the Nesterov-adjusted moment before the parameter update
  • Return exactly new_w, new_m, and new_v in a dictionary
  • Every returned value must be a NumPy array

Constraints

  • w, m, v, and grad have the same nonempty numeric shape
  • lr is positive
  • beta1 and beta2 are at least 0 and less than 1
  • eps is positive
  • Use NumPy only
Try Similar Problems
Adam OptimizerAdamw OptimizerAdagrad OptimizerRmsprop OptimizerNesterov Momentum

Sign in to take notes on this problem

Case 1
Case 2
Case 3

Accepts: array

Accepts: array

Accepts: array

Accepts: array

Accepts: number

Accepts: number

Accepts: number

Accepts: number

You must run your code first.
PrevNext

Implement Nadam (Nesterov + Adam)

Optimization
Medium

Perform one Nadam update without bias correction. Update the moments:

mt=β1mt−1+(1−β1)gtm_t = \beta_1m_{t-1} + (1-\beta_1)g_tmt​=β1​mt−1​+(1−β1​)gt​ vt=β2vt−1+(1−β2)gt2v_t = \beta_2v_{t-1} + (1-\beta_2)g_t^2vt​=β2​vt−1​+(1−β2​)gt2​

Form the Nesterov-adjusted first moment:

m~t=β1mt+(1−β1)gt\widetilde{m}_t = \beta_1m_t + (1-\beta_1)g_tmt​=β1​mt​+(1−β1​)gt​

Then update the parameters:

wt=wt−1−ηm~tvt+εw_t = w_{t-1} - \eta\frac{\widetilde{m}_t}{\sqrt{v_t}+\varepsilon}wt​=wt−1​−ηvt​​+εmt​​

Here, www contains parameters, ggg contains gradients, mmm and vvv are moment arrays, and η\etaη is lr. Return new_w, new_m, and new_v in a dictionary of NumPy arrays.

Loading visualization...

Examples

Input: w = [1.0, -1.0], m = [0.1, -0.1], v = [0.01, 0.01], grad = [0.2, -0.3], lr = 0.002, beta1 = 0.9, beta2 = 0.999, eps = 1e-8

Output: {"new_w": [0.997624, -0.997251], "new_m": [0.11, -0.12], "new_v": [0.01003, 0.01008]}

Explanation: The updated first moment is combined with the current gradient before adaptive scaling.

Input: w = [1.0, 2.0], m = [0.1, 0.2], v = [0.01, 0.04], grad = [0.0, 0.0], lr = 0.002, beta1 = 0.9, beta2 = 0.999, eps = 1e-8

Output: {"new_w": [0.998379, 1.998379], "new_m": [0.09, 0.18], "new_v": [0.00999, 0.03996]}

Input: w = [1.0, 2.0], m = [0.0, 0.0], v = [0.0, 0.0], grad = [0.1, 0.2], lr = 0.002, beta1 = 0.9, beta2 = 0.999, eps = 1e-8

Output: {"new_w": [0.987983, 1.987983], "new_m": [0.01, 0.02], "new_v": [0.00001, 0.00004]}

Hint 1

Compute the ordinary first and second moments before nesterov_m.

Hint 2

Use beta1 * new_m + (1 - beta1) * grad in the numerator.

Requirements

  • Apply the stated Nadam equations without bias correction
  • Compute the Nesterov-adjusted moment before the parameter update
  • Return exactly new_w, new_m, and new_v in a dictionary
  • Every returned value must be a NumPy array

Constraints

  • w, m, v, and grad have the same nonempty numeric shape
  • lr is positive
  • beta1 and beta2 are at least 0 and less than 1
  • eps is positive
  • Use NumPy only
Try Similar Problems
Adam OptimizerAdamw OptimizerAdagrad OptimizerRmsprop OptimizerNesterov Momentum

Sign in to take notes on this problem

Case 1
Case 2
Case 3

Accepts: array

Accepts: array

Accepts: array

Accepts: array

Accepts: number

Accepts: number

Accepts: number

Accepts: number

You must run your code first.