Implement Positional Encoding (sin/cos)
Implement Positional Encoding (sin/cos)
Implement sinusoidal positional encodings as described in "Attention Is All You Need" to inject sequence order into token embeddings.
Given a sequence length and model dimension, compute the positional encoding matrix using the sin/cos formulation.
Mathematical Definition
For position pos and dimension index i:
PE(pos,2i)=sin(base2i/dmodelpos) PE(pos,2i+1)=cos(base2i/dmodelpos)Even-indexed columns use sine, odd-indexed columns use cosine, and the frequency decreases with dimension index.
Examples
Input:
seq_len = 3, d_model = 4
Output (approx):
[[0.0000, 1.0000, 0.0000, 1.0000], [0.8415, 0.5403, 0.0100, 0.9999], [0.9093, -0.4161, 0.0200, 0.9998]]
Columns: [sin(i=0), cos(i=0), sin(i=1), cos(i=1)] For i=1, divisor = 10000^(2/4) = 100, so angles are small (~0.01, 0.02).
Input:
seq_len = 2, d_model = 3 (odd)
Output (approx):
[[0.0000, 1.0000, 0.0000], [0.8415, 0.5403, 0.0022]]
Columns: [sin(i=0), cos(i=0), sin(i=1)] Odd d_model means the last column is a sin column without a paired cos. For i=1, divisor = 10000^(2/3) ~ 464.16, so angle ~ 0.00215, sin ~ 0.0022.
Hint 1
Build a column vector of positions (T,1) and a row vector of frequencies (1,⌈d/2⌉) for broadcasting.
Hint 2
Use pe[:, 0::2] = sin(...) and pe[:, 1::2] = cos(...) to fill alternating columns.
Requirements
- Fully vectorized (no Python loops over positions or dimensions)
- Support any seq_len ≥ 1 and d_model ≥ 1
- For odd d_model, the last column should be the sin column
- Return dtype=float for stable downstream use
Constraints
- Sequence lengths up to 10,000
- Model dimensions up to 2,048
- Use NumPy only
- Time limit: 300 ms
Try Similar Problems
Log in to take notes on this problem
Accepts: number
Accepts: number
Accepts: number
Implement Positional Encoding (sin/cos)
Implement Positional Encoding (sin/cos)
Implement sinusoidal positional encodings as described in "Attention Is All You Need" to inject sequence order into token embeddings.
Given a sequence length and model dimension, compute the positional encoding matrix using the sin/cos formulation.
Mathematical Definition
For position pos and dimension index i:
PE(pos,2i)=sin(base2i/dmodelpos) PE(pos,2i+1)=cos(base2i/dmodelpos)Even-indexed columns use sine, odd-indexed columns use cosine, and the frequency decreases with dimension index.
Examples
Input:
seq_len = 3, d_model = 4
Output (approx):
[[0.0000, 1.0000, 0.0000, 1.0000], [0.8415, 0.5403, 0.0100, 0.9999], [0.9093, -0.4161, 0.0200, 0.9998]]
Columns: [sin(i=0), cos(i=0), sin(i=1), cos(i=1)] For i=1, divisor = 10000^(2/4) = 100, so angles are small (~0.01, 0.02).
Input:
seq_len = 2, d_model = 3 (odd)
Output (approx):
[[0.0000, 1.0000, 0.0000], [0.8415, 0.5403, 0.0022]]
Columns: [sin(i=0), cos(i=0), sin(i=1)] Odd d_model means the last column is a sin column without a paired cos. For i=1, divisor = 10000^(2/3) ~ 464.16, so angle ~ 0.00215, sin ~ 0.0022.
Hint 1
Build a column vector of positions (T,1) and a row vector of frequencies (1,⌈d/2⌉) for broadcasting.
Hint 2
Use pe[:, 0::2] = sin(...) and pe[:, 1::2] = cos(...) to fill alternating columns.
Requirements
- Fully vectorized (no Python loops over positions or dimensions)
- Support any seq_len ≥ 1 and d_model ≥ 1
- For odd d_model, the last column should be the sin column
- Return dtype=float for stable downstream use
Constraints
- Sequence lengths up to 10,000
- Model dimensions up to 2,048
- Use NumPy only
- Time limit: 300 ms
Try Similar Problems
Log in to take notes on this problem
Accepts: number
Accepts: number
Accepts: number