Problems
Loading...

06/08Feed-Forward Network

Easy

Implement the position-wise feed-forward network used in the Transformer.

FFN⁡(x)=ReLU⁡(xW1+b1)W2+b2\operatorname{FFN}(x) = \operatorname{ReLU}(xW_1 + b_1)W_2 + b_2

Here, W1 expands from model width to hidden width, b1 is the hidden bias, W2 projects back to model width, and b2 is the output bias. Return a NumPy float64 array with the same leading dimensions and model width as x.

Loading visualization...

Examples

Input: x = [[[1.0, 0.5, -0.5, 2.0]]], W1: (4,8), b1: (8,), W2: (8,4), b2: (4,)

Output: [[[-0.0271, -0.1309, 0.3779, 0.9302]]]

Explanation: The input is projected to a higher-dimensional hidden space via W1+b1, ReLU clips negative activations to zero, then W2+b2 projects back to the original dimension.

Input: x = [[[1.0, -1.0, 0.5], [0.0, 2.0, -1.0]]], W1: (3,6), b1: (6,), W2: (6,3), b2: (3,)

Output: [[[-0.2502, -0.1218, -0.1188], [-0.0221, -0.2764, -0.5222]]]

Input: x = [[[3.0, 1.0]]], W1: (2,4), b1: (4,), W2: (4,2), b2: (2,)

Output: [[[0.0479, -0.6016]]]

Hint 1

Compute hidden = np.maximum(0, x @ W1 + b1).

Hint 2

NumPy broadcasts each bias over leading dimensions.

Hint 3

Project hidden with W2 and add b2.

Requirements

  • Use NumPy float64 array operations.
  • Apply W1 and b1 to the final axis.
  • Apply ReLU before W2.
  • Return the unrounded result.

Constraints

  • x has shape (..., d_model).
  • W1 has shape (d_model, d_ff).
  • W2 has shape (d_ff, d_model).
  • Bias lengths match their corresponding outputs.
Prev6 / 8Next