Problems
Loading...
Step 5 of 6
Classification Head
Easy

Convert the encoder classification-token state into class logits. Select sequence position zero, apply LayerNorm over its embedding coordinates, and multiply by the supplied classification matrix.

μb=1D∑d=1Dhb,d.\mu_b = \frac{1}{D}\sum_{d=1}^{D} h_{b,d}. σb2=1D∑d=1D(hb,d−μb)2.\sigma_b^2 = \frac{1}{D}\sum_{d=1}^{D}(h_{b,d}-\mu_b)^2. h^b,d=hb,d−μbσb2+ε.\widehat{h}_{b,d} = \frac{h_{b,d}-\mu_b}{\sqrt{\sigma_b^2+\varepsilon}}. logits⁡=h^Whead.\operatorname{logits} = \widehat{h}W_{\mathrm{head}}.

Here, hh is the token at position zero, BB is batch size, DD is embedding width, WheadW_{\mathrm{head}} is W_head, and ε=10−6\varepsilon=10^{-6}. Return the logits as a float64 NumPy array with shape (B,C)(B,C), where CC is the second dimension of W_head.

Loading visualization...

Examples

Input: encoder_output = [[[4,2],[1,1],[0,0]]], W_head = [[1,0,1],[0,1,1]]

Output: [[1,-1,0]]

Explanation: LayerNorm is applied to token zero before W_head produces class logits.

Input: encoder_output = [[[2,2],[9,-9]]], W_head = [[1,0],[0,1]]

Output: [[0,0]]

Input: encoder_output = [[[6,2],[100,100]],[[1,3],[-100,-100]]], W_head = [[1,2],[-1,1]]

Output: [[2,1],[-1.999999,-1]]

Hint 1

Select encoder_output[:, 0, :] before normalizing.

Hint 2

Compute the mean and population variance along the last axis.

Hint 3

The number of output classes is W_head.shape[1].

Requirements

  • Use NumPy.
  • Select token position zero.
  • Apply LayerNorm with population variance and epsilon 1e-6.
  • Multiply the normalized state by W_head without an output activation.
  • Return a float64 NumPy array with shape (B, C).

Constraints

  • encoder_output has shape (B, N, D) and dtype float64.
  • W_head has shape (D, C) and dtype float64.
  • B, N, D, and C are positive.
Prev5 / 6Next