Implement one update step of the RMSProp optimizer. Given current parameters, gradients, and running squared gradient accumulator, return updated parameters and accumulator.
Step 1: Update Running Average
st=β⋅st−1+(1−β)⋅gt2Step 2: Parameter Update
wt=wt−1−st+εη⋅gtWhere: w = parameters, g = gradients, s = squared gradient accumulator, η = learning rate, β = decay factor, ε = stability constant
Input: w = [1.0, 2.0], g = [0.2, -0.4], s = [0.0, 0.0], lr = 0.1, beta = 0.9, eps = 1e-8
Output: ([0.683773, 2.316228], [0.004, 0.016])
Explanation: The squared-gradient accumulator is updated first, then each parameter uses its own scaled step.
Input: w = [5.0], g = [0.0], s = [0.1], lr = 0.1, beta = 0.9, eps = 1e-8
Output: ([5.0], [0.09])
Input: w = [[1.0, 2.0]], g = [[0.1, 0.2]], s = [[0.01, 0.04]], lr = 0.1, beta = 0.9, eps = 1e-8
Output: ([[0.9, 1.9]], [[0.01, 0.04]])
Convert w, g, and s to NumPy arrays before computing the accumulator update.
Use g * g for squared gradients and np.sqrt(new_s) in the parameter update.
Sign in to take notes on this problem
Accepts: array
Accepts: array
Accepts: array
Accepts: number
Accepts: number
Accepts: number
Implement one update step of the RMSProp optimizer. Given current parameters, gradients, and running squared gradient accumulator, return updated parameters and accumulator.
Step 1: Update Running Average
st=β⋅st−1+(1−β)⋅gt2Step 2: Parameter Update
wt=wt−1−st+εη⋅gtWhere: w = parameters, g = gradients, s = squared gradient accumulator, η = learning rate, β = decay factor, ε = stability constant
Input: w = [1.0, 2.0], g = [0.2, -0.4], s = [0.0, 0.0], lr = 0.1, beta = 0.9, eps = 1e-8
Output: ([0.683773, 2.316228], [0.004, 0.016])
Explanation: The squared-gradient accumulator is updated first, then each parameter uses its own scaled step.
Input: w = [5.0], g = [0.0], s = [0.1], lr = 0.1, beta = 0.9, eps = 1e-8
Output: ([5.0], [0.09])
Input: w = [[1.0, 2.0]], g = [[0.1, 0.2]], s = [[0.01, 0.04]], lr = 0.1, beta = 0.9, eps = 1e-8
Output: ([[0.9, 1.9]], [[0.01, 0.04]])
Convert w, g, and s to NumPy arrays before computing the accumulator update.
Use g * g for squared gradients and np.sqrt(new_s) in the parameter update.
Sign in to take notes on this problem
Accepts: array
Accepts: array
Accepts: array
Accepts: number
Accepts: number
Accepts: number