Implement global norm gradient clipping to prevent exploding gradients in deep networks. Scale gradients proportionally when their norm exceeds a threshold.
Compute Gradient Norm:
∣∣g∣∣=i∑gi2Clipping Rule:
gclipped={gg⋅∣∣g∣∣max_normif ∣∣g∣∣≤max_normotherwiseWhere: g = gradient vector, ||g|| = L2 norm, max_norm = clipping threshold
g: np.ndarray - Gradient array (any shape)max_norm: float - Maximum allowed norm (positive)Input: g=[0.1, 0.2, 0.2], max_norm=1.0
Output: [0.1, 0.2, 0.2]
norm = 0.3 ≤ 1.0, so no clipping needed
Input: g=[6, 8], max_norm=5.0
Output: [3.0, 4.0]
norm = 10.0 > 5.0, so multiply by 5.0/10.0 = 0.5
Input: g=[[2, 2], [2, 2]], max_norm=2.0
Output: [[1.0, 1.0], [1.0, 1.0]]
norm = 4.0 > 2.0, so multiply by 2.0/4.0 = 0.5
Use np.asarray() to convert input and np.linalg.norm() to compute the L2 norm of the entire gradient array.
Check if norm <= max_norm to decide whether clipping is needed. Use g.copy() to avoid in-place modification.
Scale gradients with g * (max_norm / norm). Handle edge cases: zero norm or non-positive max_norm should return unchanged gradients.
np.ndarray with same shape as inputSign in to take notes on this problem
Accepts: array
Accepts: number
Implement global norm gradient clipping to prevent exploding gradients in deep networks. Scale gradients proportionally when their norm exceeds a threshold.
Compute Gradient Norm:
∣∣g∣∣=i∑gi2Clipping Rule:
gclipped={gg⋅∣∣g∣∣max_normif ∣∣g∣∣≤max_normotherwiseWhere: g = gradient vector, ||g|| = L2 norm, max_norm = clipping threshold
g: np.ndarray - Gradient array (any shape)max_norm: float - Maximum allowed norm (positive)Input: g=[0.1, 0.2, 0.2], max_norm=1.0
Output: [0.1, 0.2, 0.2]
norm = 0.3 ≤ 1.0, so no clipping needed
Input: g=[6, 8], max_norm=5.0
Output: [3.0, 4.0]
norm = 10.0 > 5.0, so multiply by 5.0/10.0 = 0.5
Input: g=[[2, 2], [2, 2]], max_norm=2.0
Output: [[1.0, 1.0], [1.0, 1.0]]
norm = 4.0 > 2.0, so multiply by 2.0/4.0 = 0.5
Use np.asarray() to convert input and np.linalg.norm() to compute the L2 norm of the entire gradient array.
Check if norm <= max_norm to decide whether clipping is needed. Use g.copy() to avoid in-place modification.
Scale gradients with g * (max_norm / norm). Handle edge cases: zero norm or non-positive max_norm should return unchanged gradients.
np.ndarray with same shape as inputSign in to take notes on this problem
Accepts: array
Accepts: number