RNN Step Backward (Vanilla RNN)
RNN Step Backward (Vanilla RNN)
Implement the backward pass for a single RNN time step (vanilla RNN with tanh activation).
Forward Pass Reference
RNN Forward Equation:
ht=tanh(Wxt+Uht−1+b)where xt∈RD, ht∈RH, W∈RH×D, U∈RH×H, b∈RH.
Backward Pass Gradients
Tanh Derivative:
∂z∂tanh(z)=1−tanh2(z)=1−ht2Chain Rule Application:
∂z∂L=∂ht∂L∘(1−ht2)Function Arguments
dh: array, shape (H,)- upstream gradient ∂L/∂h_tcache: list- cached values[x_t, h_prev, h_t, W, U, b]from forward pass
Returns
Tuple of 5 gradients: (dx_t, dh_prev, dW, dU, db)
Examples
Input: dh = [1, 1], cache = [[0.5, 0.3], [0.1, 0.2], [0.6, 0.4], [[0.1, 0.2], [0.3, 0.4]], [[0.5, 0.6], [0.7, 0.8]], [0, 0]]
Output: dx = [0.316, 0.464], dh_prev = [0.908, 1.056], dW = [[0.32, 0.192], [0.42, 0.252]], dU = [[0.064, 0.128], [0.084, 0.168]], db = [0.64, 0.84]
Input: dh = [0, 0], cache = [[1, 2], [0.5, 0.5], [0.9, 0.8], [[0.1, 0.2], [0.3, 0.4]], [[0.5, 0.6], [0.7, 0.8]], [0.1, 0.1]]
Output: dx = [0, 0], dh_prev = [0, 0], dW = [[0, 0], [0, 0]], dU = [[0, 0], [0, 0]], db = [0, 0]
Note: When dh is zero, all gradients are zero regardless of cache values.
Hint 1
First compute the gradient through ∂z∂L=∂ht∂L∘(1−ht2).
Hint 2
Use @ for matrix multiplication and np.outer() to compute ∂W∂Land∂U∂L.
Requirements
- Return tuple of 5 gradients:
(dx_t, dh_prev, dW, dU, db) - Correct derivative of tanh: 1−ht²
- Apply chain rule properly
- All gradients must match reference shapes
- NumPy only, no autograd
Constraints
- 2 ≤ D, H ≤ 128
- NumPy only
- Time limit: 300ms
Log in to take notes on this problem
Accepts: array
Accepts: array
RNN Step Backward (Vanilla RNN)
RNN Step Backward (Vanilla RNN)
Implement the backward pass for a single RNN time step (vanilla RNN with tanh activation).
Forward Pass Reference
RNN Forward Equation:
ht=tanh(Wxt+Uht−1+b)where xt∈RD, ht∈RH, W∈RH×D, U∈RH×H, b∈RH.
Backward Pass Gradients
Tanh Derivative:
∂z∂tanh(z)=1−tanh2(z)=1−ht2Chain Rule Application:
∂z∂L=∂ht∂L∘(1−ht2)Function Arguments
dh: array, shape (H,)- upstream gradient ∂L/∂h_tcache: list- cached values[x_t, h_prev, h_t, W, U, b]from forward pass
Returns
Tuple of 5 gradients: (dx_t, dh_prev, dW, dU, db)
Examples
Input: dh = [1, 1], cache = [[0.5, 0.3], [0.1, 0.2], [0.6, 0.4], [[0.1, 0.2], [0.3, 0.4]], [[0.5, 0.6], [0.7, 0.8]], [0, 0]]
Output: dx = [0.316, 0.464], dh_prev = [0.908, 1.056], dW = [[0.32, 0.192], [0.42, 0.252]], dU = [[0.064, 0.128], [0.084, 0.168]], db = [0.64, 0.84]
Input: dh = [0, 0], cache = [[1, 2], [0.5, 0.5], [0.9, 0.8], [[0.1, 0.2], [0.3, 0.4]], [[0.5, 0.6], [0.7, 0.8]], [0.1, 0.1]]
Output: dx = [0, 0], dh_prev = [0, 0], dW = [[0, 0], [0, 0]], dU = [[0, 0], [0, 0]], db = [0, 0]
Note: When dh is zero, all gradients are zero regardless of cache values.
Hint 1
First compute the gradient through ∂z∂L=∂ht∂L∘(1−ht2).
Hint 2
Use @ for matrix multiplication and np.outer() to compute ∂W∂Land∂U∂L.
Requirements
- Return tuple of 5 gradients:
(dx_t, dh_prev, dW, dU, db) - Correct derivative of tanh: 1−ht²
- Apply chain rule properly
- All gradients must match reference shapes
- NumPy only, no autograd
Constraints
- 2 ≤ D, H ≤ 128
- NumPy only
- Time limit: 300ms
Log in to take notes on this problem
Accepts: array
Accepts: array