TensorTonicTensorTonic
Problems
Study PlansProjectsNewInterviewPricingFeedback
Problems
Loading...
1 / 1

Implement AdaDelta Update Step

Optimization
Medium

Perform one AdaDelta update. First update the running squared-gradient average:

E[g2]t=ρE[g2]t−1+(1−ρ)gt2E[g^2]_t = \rho E[g^2]_{t-1} + (1-\rho)g_t^2E[g2]t​=ρE[g2]t−1​+(1−ρ)gt2​

Compute the parameter change:

Δwt=−E[Δw2]t−1+εE[g2]t+εgt\Delta w_t = -\frac{\sqrt{E[\Delta w^2]_{t-1}+\varepsilon}}{\sqrt{E[g^2]_t+\varepsilon}}g_tΔwt​=−E[g2]t​+ε​E[Δw2]t−1​+ε​​gt​

Update the running squared-change average:

E[Δw2]t=ρE[Δw2]t−1+(1−ρ)(Δwt)2E[\Delta w^2]_t = \rho E[\Delta w^2]_{t-1} + (1-\rho)(\Delta w_t)^2E[Δw2]t​=ρE[Δw2]t−1​+(1−ρ)(Δwt​)2

Finally update the parameters:

wt=wt−1+Δwtw_t = w_{t-1} + \Delta w_twt​=wt−1​+Δwt​

Here, www contains parameters, ggg contains gradients, ρ\rhoρ is the decay rate, and ε\varepsilonε is eps. Return new_w, new_E_grad_sq, and new_E_update_sq in a dictionary of NumPy arrays.

Loading visualization...

Examples

Input: w = [1.0, -1.0], grad = [0.1, -0.2], E_grad_sq = [0.01, 0.04], E_update_sq = [0.001, 0.004], rho = 0.9, eps = 1e-6

Output: {"new_w": [0.968363, -0.936747], "new_E_grad_sq": [0.01, 0.04], "new_E_update_sq": [0.001, 0.004]}

Explanation: The ratio of the two running root-mean-square values scales each gradient before the state is updated.

Input: w = [1.0, 2.0], grad = [0.0, 0.0], E_grad_sq = [0.01, 0.04], E_update_sq = [0.001, 0.004], rho = 0.9, eps = 1e-6

Output: {"new_w": [1.0, 2.0], "new_E_grad_sq": [0.009, 0.036], "new_E_update_sq": [0.0009, 0.0036]}

Input: w = [1.0, 2.0], grad = [0.1, 0.2], E_grad_sq = [0.0, 0.0], E_update_sq = [0.0, 0.0], rho = 0.9, eps = 1e-6

Output: {"new_w": [0.996839, 1.996838], "new_E_grad_sq": [0.001, 0.004], "new_E_update_sq": [0.000001, 0.000001]}

Hint 1

Update the squared-gradient average before computing the parameter change.

Hint 2

Use the new parameter change to update new_E_update_sq.

Requirements

  • Apply the four equations in the stated order
  • Do not introduce a separate learning-rate parameter
  • Return exactly new_w, new_E_grad_sq, and new_E_update_sq in a dictionary
  • Every returned value must be a NumPy array

Constraints

  • All four array inputs have the same nonempty numeric shape
  • rho is at least 0 and less than 1
  • eps is positive
  • Use NumPy only
Try Similar Problems
Adagrad OptimizerRmsprop OptimizerAdam OptimizerGradient Descent QuadraticAdamw Optimizer

Sign in to take notes on this problem

Case 1
Case 2
Case 3

Accepts: array

Accepts: array

Accepts: array

Accepts: array

Accepts: number

Accepts: number

You must run your code first.
PrevNext

Implement AdaDelta Update Step

Optimization
Medium

Perform one AdaDelta update. First update the running squared-gradient average:

E[g2]t=ρE[g2]t−1+(1−ρ)gt2E[g^2]_t = \rho E[g^2]_{t-1} + (1-\rho)g_t^2E[g2]t​=ρE[g2]t−1​+(1−ρ)gt2​

Compute the parameter change:

Δwt=−E[Δw2]t−1+εE[g2]t+εgt\Delta w_t = -\frac{\sqrt{E[\Delta w^2]_{t-1}+\varepsilon}}{\sqrt{E[g^2]_t+\varepsilon}}g_tΔwt​=−E[g2]t​+ε​E[Δw2]t−1​+ε​​gt​

Update the running squared-change average:

E[Δw2]t=ρE[Δw2]t−1+(1−ρ)(Δwt)2E[\Delta w^2]_t = \rho E[\Delta w^2]_{t-1} + (1-\rho)(\Delta w_t)^2E[Δw2]t​=ρE[Δw2]t−1​+(1−ρ)(Δwt​)2

Finally update the parameters:

wt=wt−1+Δwtw_t = w_{t-1} + \Delta w_twt​=wt−1​+Δwt​

Here, www contains parameters, ggg contains gradients, ρ\rhoρ is the decay rate, and ε\varepsilonε is eps. Return new_w, new_E_grad_sq, and new_E_update_sq in a dictionary of NumPy arrays.

Loading visualization...

Examples

Input: w = [1.0, -1.0], grad = [0.1, -0.2], E_grad_sq = [0.01, 0.04], E_update_sq = [0.001, 0.004], rho = 0.9, eps = 1e-6

Output: {"new_w": [0.968363, -0.936747], "new_E_grad_sq": [0.01, 0.04], "new_E_update_sq": [0.001, 0.004]}

Explanation: The ratio of the two running root-mean-square values scales each gradient before the state is updated.

Input: w = [1.0, 2.0], grad = [0.0, 0.0], E_grad_sq = [0.01, 0.04], E_update_sq = [0.001, 0.004], rho = 0.9, eps = 1e-6

Output: {"new_w": [1.0, 2.0], "new_E_grad_sq": [0.009, 0.036], "new_E_update_sq": [0.0009, 0.0036]}

Input: w = [1.0, 2.0], grad = [0.1, 0.2], E_grad_sq = [0.0, 0.0], E_update_sq = [0.0, 0.0], rho = 0.9, eps = 1e-6

Output: {"new_w": [0.996839, 1.996838], "new_E_grad_sq": [0.001, 0.004], "new_E_update_sq": [0.000001, 0.000001]}

Hint 1

Update the squared-gradient average before computing the parameter change.

Hint 2

Use the new parameter change to update new_E_update_sq.

Requirements

  • Apply the four equations in the stated order
  • Do not introduce a separate learning-rate parameter
  • Return exactly new_w, new_E_grad_sq, and new_E_update_sq in a dictionary
  • Every returned value must be a NumPy array

Constraints

  • All four array inputs have the same nonempty numeric shape
  • rho is at least 0 and less than 1
  • eps is positive
  • Use NumPy only
Try Similar Problems
Adagrad OptimizerRmsprop OptimizerAdam OptimizerGradient Descent QuadraticAdamw Optimizer

Sign in to take notes on this problem

Case 1
Case 2
Case 3

Accepts: array

Accepts: array

Accepts: array

Accepts: array

Accepts: number

Accepts: number

You must run your code first.