02/08Embedding Layer
Implement the Transformer's token embedding step. create_embedding_layer must construct a torch.nn.Embedding with vocab_size rows and d_model columns. embed_tokens must look up the supplied token IDs and scale their vectors as
E=Embedding(tokens)dmodelHere, d_model is the embedding width. Return a PyTorch float32 tensor whose leading dimensions match tokens and whose last dimension is d_model.
Examples
Input: vocab_size = 10, d_model = 4, tokens = [1, 2, 3]
Output: [[1.3568, -2.469, -0.0862, -3.2094], [-1.5042, 3.2974, -0.785, -2.8072], [-1.4558, -1.1188, -1.5376, 1.5248]]
Explanation: The embedding layer maps each token index to a 4-dimensional vector, then multiplies by sqrt(4) = 2.0 to produce the final scaled embeddings.
Input: vocab_size = 10, d_model = 4, tokens = [0]
Output: [[3.8538, 2.9746, 1.8014, -4.211]]
Input: vocab_size = 10, d_model = 4, tokens = [5, 5]
Output: [[-1.5162, 2.1566, 1.6016, 3.3612], [-1.5162, 2.1566, 1.6016, 3.3612]]
Hint 1
Return nn.Embedding(vocab_size, d_model) from the factory.
Hint 2
Call the supplied embedding layer on tokens.
Hint 3
Multiply the lookup result by math.sqrt(d_model).
Requirements
- Implement both provided functions with PyTorch.
- Construct nn.Embedding(vocab_size, d_model).
- Preserve all token dimensions during lookup.
- Scale embeddings by the square root of d_model.
Constraints
- tokens has dtype torch.int64.
- Embedding weights have dtype torch.float32.
- Token IDs are valid vocabulary indices.
- Valid inputs are guaranteed.
Accepts: number
Accepts: number
Accepts: array
Accepts: array
02/08Embedding Layer
Implement the Transformer's token embedding step. create_embedding_layer must construct a torch.nn.Embedding with vocab_size rows and d_model columns. embed_tokens must look up the supplied token IDs and scale their vectors as
E=Embedding(tokens)dmodelHere, d_model is the embedding width. Return a PyTorch float32 tensor whose leading dimensions match tokens and whose last dimension is d_model.
Examples
Input: vocab_size = 10, d_model = 4, tokens = [1, 2, 3]
Output: [[1.3568, -2.469, -0.0862, -3.2094], [-1.5042, 3.2974, -0.785, -2.8072], [-1.4558, -1.1188, -1.5376, 1.5248]]
Explanation: The embedding layer maps each token index to a 4-dimensional vector, then multiplies by sqrt(4) = 2.0 to produce the final scaled embeddings.
Input: vocab_size = 10, d_model = 4, tokens = [0]
Output: [[3.8538, 2.9746, 1.8014, -4.211]]
Input: vocab_size = 10, d_model = 4, tokens = [5, 5]
Output: [[-1.5162, 2.1566, 1.6016, 3.3612], [-1.5162, 2.1566, 1.6016, 3.3612]]
Hint 1
Return nn.Embedding(vocab_size, d_model) from the factory.
Hint 2
Call the supplied embedding layer on tokens.
Hint 3
Multiply the lookup result by math.sqrt(d_model).
Requirements
- Implement both provided functions with PyTorch.
- Construct nn.Embedding(vocab_size, d_model).
- Preserve all token dimensions during lookup.
- Scale embeddings by the square root of d_model.
Constraints
- tokens has dtype torch.int64.
- Embedding weights have dtype torch.float32.
- Token IDs are valid vocabulary indices.
- Valid inputs are guaranteed.
Accepts: number
Accepts: number
Accepts: array
Accepts: array