Problems
Loading...
1 / 1

AdaGrad Optimizer

Optimization
Easy

Implement one update step of the AdaGrad optimizer. Given current parameters, gradients, and accumulated squared gradients, return updated parameters and accumulator.

Step 1: Accumulate Squared Gradients

Gt=Gt−1+gt2G_t = G_{t-1} + g_t^2

Step 2: Parameter Update

wt=wt−1−ηGt+ε⋅gtw_t = w_{t-1} - \frac{\eta}{\sqrt{G_t + \varepsilon}} \cdot g_t

Where: w = parameters, g = gradients, G = accumulated squared gradients, η = learning rate, ε = stability constant

Function Arguments

  • w: np.ndarray - Current parameters (any shape)
  • g: np.ndarray - Current gradients (same shape as w)
  • G: np.ndarray - Accumulated squared gradients (same shape as w)
  • lr: float = 0.01 - Learning rate
  • eps: float = 1e-8 - Small constant for numerical stability
Loading visualization...

Examples

Input: w=[1.0, 2.0], g=[0.1, -0.2], G=[0.0, 0.0], lr=0.1

Output: ([0.9, 2.1], [0.01, 0.04])

First step: G becomes [0.01, 0.04], then w updated with adaptive learning rates

Input: w=[1.0, 2.0], g=[0.0, 0.0], G=[0.1, 0.2], lr=0.1

Output: ([1.0, 2.0], [0.1, 0.2])

Zero gradient means no parameter update and G stays unchanged

Input: w=[0.0], g=[1.0], G=[100.0], lr=0.1

Output: ([-0.00995], [101.0])

Large accumulated G leads to very small effective learning rate (0.1/√101 ≈ 0.00995)

Hint 1

Convert inputs to NumPy arrays first. Update G by adding the squared gradients.

Hint 2

Use element-wise operations: np.sqrt() for the denominator and standard arithmetic for the parameter update.

Requirements

  • Return tuple (new_w, new_G) with same shapes as inputs
  • Use the exact update formulas above
  • Vectorized implementation only (no Python loops)
  • Handle any array shape (1D, 2D, etc.)

Constraints

  • Parameter dimension D: 1 ≤ D ≤ 10⁵
  • Learning rate lr > 0
  • Epsilon ε: small positive float
  • Libraries: NumPy only
Try Similar Problems
Rmsprop OptimizerAdam OptimizerAdadelta OptimizerAdamw OptimizerNadam Optimizer
PrevNext