Implement sinusoidal positional encodings as described in "Attention Is All You Need" to inject sequence order into token embeddings.
Given a sequence length and model dimension, compute the positional encoding matrix using the sin/cos formulation.
For position pos and dimension index i:
PE(pos,2i)=sin(base2i/dmodelpos) PE(pos,2i+1)=cos(base2i/dmodelpos)Even-indexed columns use sine, odd-indexed columns use cosine, and the frequency decreases with dimension index.
Input: seq_len = 3, d_model = 4
Output: [[0.0000, 1.0000, 0.0000, 1.0000], [0.8415, 0.5403, 0.0100, 0.9999], [0.9093, -0.4161, 0.0200, 0.9998]]
Explanation: Columns alternate sine and cosine values. For the second frequency, the divisor is 100, producing the smaller angles 0.01 and 0.02.
Input: seq_len = 5, d_model = 7, base = 10000.0
Output: NumPy array of shape (5, 7)
Build positions with shape (seq_len, 1) and frequencies with shape (1, ceil(d_model / 2)) for broadcasting.
Fill even columns with sine values and odd columns with the corresponding cosine values.
Sign in to take notes on this problem
Accepts: number
Accepts: number
Accepts: number
Implement sinusoidal positional encodings as described in "Attention Is All You Need" to inject sequence order into token embeddings.
Given a sequence length and model dimension, compute the positional encoding matrix using the sin/cos formulation.
For position pos and dimension index i:
PE(pos,2i)=sin(base2i/dmodelpos) PE(pos,2i+1)=cos(base2i/dmodelpos)Even-indexed columns use sine, odd-indexed columns use cosine, and the frequency decreases with dimension index.
Input: seq_len = 3, d_model = 4
Output: [[0.0000, 1.0000, 0.0000, 1.0000], [0.8415, 0.5403, 0.0100, 0.9999], [0.9093, -0.4161, 0.0200, 0.9998]]
Explanation: Columns alternate sine and cosine values. For the second frequency, the divisor is 100, producing the smaller angles 0.01 and 0.02.
Input: seq_len = 5, d_model = 7, base = 10000.0
Output: NumPy array of shape (5, 7)
Build positions with shape (seq_len, 1) and frequencies with shape (1, ceil(d_model / 2)) for broadcasting.
Fill even columns with sine values and odd columns with the corresponding cosine values.
Sign in to take notes on this problem
Accepts: number
Accepts: number
Accepts: number