Problems
Loading...

03/08Positional Encoding

Medium

Implement the fixed sinusoidal positional encoding from the Transformer.

PE(pos,2i)=sin⁡ ⁣(pos100002i/dmodel)PE(pos,2i)=\sin\!\left(\frac{pos}{10000^{2i/d_{model}}}\right) PE(pos,2i+1)=cos⁡ ⁣(pos100002i/dmodel)PE(pos,2i+1)=\cos\!\left(\frac{pos}{10000^{2i/d_{model}}}\right)

Here, pos is a zero-based sequence position, i indexes a sine-cosine pair, and d_model is the model width. Return a NumPy float64 array with shape (seq_length, d_model).

Loading visualization...

Examples

Input: seq_length = 4, d_model = 6

Output: [[0.0, 1.0, 0.0, 1.0, 0.0, 1.0], [0.8415, 0.5403, 0.0464, 0.9989, 0.0022, 1.0], [0.9093, -0.4161, 0.0927, 0.9957, 0.0043, 1.0], [0.1411, -0.99, 0.1388, 0.9903, 0.0065, 1.0]]

Explanation: Row 0 is all sin(0)=0 and cos(0)=1. Each subsequent row encodes a new position with higher-frequency oscillations in earlier columns and lower-frequency oscillations in later columns.

Input: seq_length = 3, d_model = 4

Output: [[0.0, 1.0, 0.0, 1.0], [0.8415, 0.5403, 0.01, 1.0], [0.9093, -0.4161, 0.02, 0.9998]]

Input: seq_length = 2, d_model = 8

Output: [[0.0, 1.0, 0.0, 1.0, 0.0, 1.0, 0.0, 1.0], [0.8415, 0.5403, 0.0998, 0.995, 0.01, 1.0, 0.001, 1.0]]

Hint 1

Create positions with shape (seq_length, 1).

Hint 2

Compute one inverse frequency for each even column.

Hint 3

Assign sine and cosine results with 0::2 and 1::2 slices.

Requirements

  • Use NumPy float64 computations.
  • Place sine values in even columns and cosine values in odd columns.
  • Use base 10000 and the specified exponent.
  • Return the unrounded encoding matrix.

Constraints

  • seq_length is positive.
  • d_model is a positive even integer.
  • Valid inputs are guaranteed.
Prev3 / 8Next