TensorTonicTensorTonic
Problems
Study PlansProjectsInterviewPricingFeedback
Problems
Loading...
1 / 1

Implement AdaDelta Update Step

Optimization
Medium

Implement one update step of AdaDelta optimizer. Given current parameters, gradients, and running averages, return updated parameters and averages using the adaptive learning rate approach.

Step 1: Update Squared Gradient Average

E[g2]t=ρE[g2]t−1+(1−ρ)gt2E[g^2]_t = \rho E[g^2]_{t-1} + (1 - \rho) g_t^2E[g2]t​=ρE[g2]t−1​+(1−ρ)gt2​

Step 2: Compute Parameter Update

Δwt=−E[Δw2]t−1+εE[g2]t+ε⋅gt\Delta w_t = -\frac{\sqrt{E[\Delta w^2]_{t-1} + \varepsilon}}{\sqrt{E[g^2]_t + \varepsilon}} \cdot g_tΔwt​=−E[g2]t​+ε​E[Δw2]t−1​+ε​​⋅gt​

Step 3: Update Squared Update Average

E[Δw2]t=ρE[Δw2]t−1+(1−ρ)(Δwt)2E[\Delta w^2]_t = \rho E[\Delta w^2]_{t-1} + (1 - \rho) (\Delta w_t)^2E[Δw2]t​=ρE[Δw2]t−1​+(1−ρ)(Δwt​)2

Step 4: Update Parameters

wt=wt−1+Δwtw_t = w_{t-1} + \Delta w_twt​=wt−1​+Δwt​

Where: w = parameters, g = gradients, E[g²] = running avg of squared gradients, E[Δw²] = running avg of squared updates, ρ = decay rate, ε = stability constant

Function Arguments

  • w: np.ndarray - Current parameters (any shape)
  • grad: np.ndarray - Current gradients (same shape as w)
  • E_grad_sq: np.ndarray - Running average of squared gradients (same shape as w)
  • E_update_sq: np.ndarray - Running average of squared updates (same shape as w)
  • rho: float = 0.9 - Decay rate for moving averages
  • eps: float = 1e-6 - Small constant for numerical stability
Loading visualization...

Examples

Input: w=[1.0, -1.0], grad=[0.1, -0.2], E_grad_sq=[0.01, 0.04], E_update_sq=[0.001, 0.004]

Output: ([0.968, -0.937], [0.01, 0.04], [0.001, 0.004])

Updates computed using ratio of RMS update to RMS gradient

Input: w=[1.0, 2.0], grad=[0.0, 0.0], E_grad_sq=[0.01, 0.04], E_update_sq=[0.001, 0.004]

Output: ([1.0, 2.0], [0.009, 0.036], [0.0009, 0.0036])

Zero gradient: no parameter update, but averages still decay

Input: w=[1.0, 2.0], grad=[0.1, 0.2], E_grad_sq=[0.0, 0.0], E_update_sq=[0.0, 0.0]

Output: ([0.997, 1.997], [0.001, 0.004], [0.000001, 0.000001])

First step: small updates due to near-zero accumulated update history

Hint 1

Convert inputs to NumPy arrays first. Update E_grad_sq using exponential moving average of squared gradients.

Hint 2

Compute delta_w using the ratio of RMS update to RMS gradient, then update E_update_sq with squared delta_w.

Requirements

  • Return tuple (new_w, new_E_grad_sq, new_E_update_sq) with same shapes as inputs
  • Use the exact update formulas above
  • No manual learning rate parameter (AdaDelta eliminates it)
  • Vectorized implementation only (no Python loops)
  • Handle any array shape (1D, 2D, etc.)

Constraints

  • 0 < rho < 1
  • eps > 0 (small positive float)
  • Libraries: NumPy only
Try Similar Problems
Adagrad OptimizerRmsprop OptimizerAdam OptimizerGradient Descent QuadraticAdamw Optimizer

Sign in to take notes on this problem

Case 1
Case 2
Case 3

Accepts: array

Accepts: array

Accepts: array

Accepts: array

Accepts: number

Accepts: number

You must run your code first.
PrevNext

Implement AdaDelta Update Step

Optimization
Medium

Implement one update step of AdaDelta optimizer. Given current parameters, gradients, and running averages, return updated parameters and averages using the adaptive learning rate approach.

Step 1: Update Squared Gradient Average

E[g2]t=ρE[g2]t−1+(1−ρ)gt2E[g^2]_t = \rho E[g^2]_{t-1} + (1 - \rho) g_t^2E[g2]t​=ρE[g2]t−1​+(1−ρ)gt2​

Step 2: Compute Parameter Update

Δwt=−E[Δw2]t−1+εE[g2]t+ε⋅gt\Delta w_t = -\frac{\sqrt{E[\Delta w^2]_{t-1} + \varepsilon}}{\sqrt{E[g^2]_t + \varepsilon}} \cdot g_tΔwt​=−E[g2]t​+ε​E[Δw2]t−1​+ε​​⋅gt​

Step 3: Update Squared Update Average

E[Δw2]t=ρE[Δw2]t−1+(1−ρ)(Δwt)2E[\Delta w^2]_t = \rho E[\Delta w^2]_{t-1} + (1 - \rho) (\Delta w_t)^2E[Δw2]t​=ρE[Δw2]t−1​+(1−ρ)(Δwt​)2

Step 4: Update Parameters

wt=wt−1+Δwtw_t = w_{t-1} + \Delta w_twt​=wt−1​+Δwt​

Where: w = parameters, g = gradients, E[g²] = running avg of squared gradients, E[Δw²] = running avg of squared updates, ρ = decay rate, ε = stability constant

Function Arguments

  • w: np.ndarray - Current parameters (any shape)
  • grad: np.ndarray - Current gradients (same shape as w)
  • E_grad_sq: np.ndarray - Running average of squared gradients (same shape as w)
  • E_update_sq: np.ndarray - Running average of squared updates (same shape as w)
  • rho: float = 0.9 - Decay rate for moving averages
  • eps: float = 1e-6 - Small constant for numerical stability
Loading visualization...

Examples

Input: w=[1.0, -1.0], grad=[0.1, -0.2], E_grad_sq=[0.01, 0.04], E_update_sq=[0.001, 0.004]

Output: ([0.968, -0.937], [0.01, 0.04], [0.001, 0.004])

Updates computed using ratio of RMS update to RMS gradient

Input: w=[1.0, 2.0], grad=[0.0, 0.0], E_grad_sq=[0.01, 0.04], E_update_sq=[0.001, 0.004]

Output: ([1.0, 2.0], [0.009, 0.036], [0.0009, 0.0036])

Zero gradient: no parameter update, but averages still decay

Input: w=[1.0, 2.0], grad=[0.1, 0.2], E_grad_sq=[0.0, 0.0], E_update_sq=[0.0, 0.0]

Output: ([0.997, 1.997], [0.001, 0.004], [0.000001, 0.000001])

First step: small updates due to near-zero accumulated update history

Hint 1

Convert inputs to NumPy arrays first. Update E_grad_sq using exponential moving average of squared gradients.

Hint 2

Compute delta_w using the ratio of RMS update to RMS gradient, then update E_update_sq with squared delta_w.

Requirements

  • Return tuple (new_w, new_E_grad_sq, new_E_update_sq) with same shapes as inputs
  • Use the exact update formulas above
  • No manual learning rate parameter (AdaDelta eliminates it)
  • Vectorized implementation only (no Python loops)
  • Handle any array shape (1D, 2D, etc.)

Constraints

  • 0 < rho < 1
  • eps > 0 (small positive float)
  • Libraries: NumPy only
Try Similar Problems
Adagrad OptimizerRmsprop OptimizerAdam OptimizerGradient Descent QuadraticAdamw Optimizer

Sign in to take notes on this problem

Case 1
Case 2
Case 3

Accepts: array

Accepts: array

Accepts: array

Accepts: array

Accepts: number

Accepts: number

You must run your code first.