Problems
Loading...
Step 5 of 6
BERT Pooler
Easy

Implement BERT's pooler. Select the hidden state at sequence position zero, apply the supplied square projection, add its bias, and apply the hyperbolic tangent.

Hpooled=tanh⁡(H:,0,:Wpool+bpool)H_{\mathrm{pooled}} = \tanh\left(H_{:,0,:}W_{\mathrm{pool}} + b_{\mathrm{pool}}\right)

Here, H∈RB×S×DH \in \mathbb{R}^{B \times S \times D} contains encoder hidden states, position zero is the classification token, Wpool∈RD×DW_{\mathrm{pool}} \in \mathbb{R}^{D \times D} is the supplied projection, and bpool∈RDb_{\mathrm{pool}} \in \mathbb{R}^{D} is its bias. Return HpooledH_{\mathrm{pooled}} as a float64 NumPy array with shape (B,D)(B,D).

Loading visualization...

Examples

Input: hidden_states = [[[0.5,-0.3],[0.1,0.2],[-0.4,0.6]]], W_pool = [[0.3,-0.1],[0.2,0.4]], b_pool = [0.1,-0.1]

Output: [[0.187746,-0.263625]]

Explanation: The pooler transforms only the classification-token state at sequence position zero.

Input: hidden_states = [[[1,0,-1],[0,0,0]],[[-0.5,0.5,0.25],[1,1,1]]], W_pool = [[0.2,-0.1,0.3],[0.4,0.2,-0.2],[-0.3,0.5,0.1]], b_pool = [0,0.1,-0.1]

Output: [[0.462117,-0.462117,0.099668],[0.024995,0.358357,-0.314021]]

Input: hidden_states = [[[2,-1],[5,5]]], W_pool = [[0,0],[0,0]], b_pool = [0.2,-0.4]

Output: [[0.197375,-0.379949]]

Hint 1

Select the classification states with hidden_states[:, 0, :].

Hint 2

Apply np.tanh after the matrix multiplication and bias.

Requirements

  • Use NumPy.
  • Use only sequence position zero from each sample.
  • Apply the supplied affine projection.
  • Apply tanh elementwise.
  • Return a float64 NumPy array with shape (B, D).

Constraints

  • hidden_states has shape (B, S, D) and dtype float64.
  • W_pool has shape (D, D) and dtype float64.
  • b_pool has shape (D) and dtype float64.
  • B, S, and D are positive.
Prev5 / 6Next