Build BERT input embeddings by summing the token, position, and segment vectors at every sequence position.
Eb,s=Etoken[tb,s]+Eposition[s]+Esegment[gb,s]Here, b indexes the batch, s indexes sequence position, tb,s is a token ID, and gb,s is either zero for sentence A or one for sentence B. The supplied embedding tables contain the vectors to look up. Return E as a float64 NumPy array with shape (B,S,H).
Examples
Input: token_ids = [[0,1,2,3]], segment_ids = [[0,0,1,1]], token_embeddings = [[0.1,0.2],[0.3,-0.1],[-0.2,0.4],[0.5,0]], position_embeddings = [[0,0.1],[0.1,0],[0.2,-0.1],[-0.1,0.2]], segment_embeddings = [[0.01,0.02],[-0.03,0.04]]
Output: [[[0.11,0.32],[0.41,-0.08],[-0.03,0.34],[0.37,0.24]]]
Explanation: Each output vector is the sum of one row from each supplied embedding table.
Input: token_ids = [[1,0],[2,1]], segment_ids = [[0,0],[1,1]], token_embeddings = [[1,0,-1],[0.5,0.5,0.5],[-0.2,0.3,0.4]], position_embeddings = [[0.1,0.2,0.3],[-0.1,0,0.1]], segment_embeddings = [[0,0,0],[0.2,-0.2,0.1]]
Output: [[[0.6,0.7,0.8],[0.9,0,-0.9]],[[0.1,0.3,0.8],[0.6,0.3,0.7]]]
Input: token_ids = [[0]], segment_ids = [[1]], token_embeddings = [[0.4,-0.5]], position_embeddings = [[0.1,0.2]], segment_embeddings = [[0,0],[-0.2,0.3]]
Output: [[[0.3,0]]]
Hint 1
Index token_embeddings and segment_embeddings with the two ID arrays.
Hint 2
Create sequence positions with np.arange(token_ids.shape[1]).
Requirements
- Use NumPy.
- Look up token embeddings from token_ids.
- Use positions from zero through S minus one for every batch row.
- Look up segment embeddings from segment_ids.
- Return the elementwise sum as a float64 NumPy array.
Constraints
- token_ids and segment_ids have shape (B, S) and dtype int64.
- Every segment ID is zero or one.
- token_embeddings has shape (V, H).
- position_embeddings has at least S rows and shape (P, H).
- segment_embeddings has shape (2, H).
- All embedding tables have dtype float64.
All steps in BERT
Accepts: array
Accepts: array
Accepts: array
Accepts: array
Accepts: array
Segment Embeddings
Build BERT input embeddings by summing the token, position, and segment vectors at every sequence position.
Eb,s=Etoken[tb,s]+Eposition[s]+Esegment[gb,s]Here, b indexes the batch, s indexes sequence position, tb,s is a token ID, and gb,s is either zero for sentence A or one for sentence B. The supplied embedding tables contain the vectors to look up. Return E as a float64 NumPy array with shape (B,S,H).
Examples
Input: token_ids = [[0,1,2,3]], segment_ids = [[0,0,1,1]], token_embeddings = [[0.1,0.2],[0.3,-0.1],[-0.2,0.4],[0.5,0]], position_embeddings = [[0,0.1],[0.1,0],[0.2,-0.1],[-0.1,0.2]], segment_embeddings = [[0.01,0.02],[-0.03,0.04]]
Output: [[[0.11,0.32],[0.41,-0.08],[-0.03,0.34],[0.37,0.24]]]
Explanation: Each output vector is the sum of one row from each supplied embedding table.
Input: token_ids = [[1,0],[2,1]], segment_ids = [[0,0],[1,1]], token_embeddings = [[1,0,-1],[0.5,0.5,0.5],[-0.2,0.3,0.4]], position_embeddings = [[0.1,0.2,0.3],[-0.1,0,0.1]], segment_embeddings = [[0,0,0],[0.2,-0.2,0.1]]
Output: [[[0.6,0.7,0.8],[0.9,0,-0.9]],[[0.1,0.3,0.8],[0.6,0.3,0.7]]]
Input: token_ids = [[0]], segment_ids = [[1]], token_embeddings = [[0.4,-0.5]], position_embeddings = [[0.1,0.2]], segment_embeddings = [[0,0],[-0.2,0.3]]
Output: [[[0.3,0]]]
Hint 1
Index token_embeddings and segment_embeddings with the two ID arrays.
Hint 2
Create sequence positions with np.arange(token_ids.shape[1]).
Requirements
- Use NumPy.
- Look up token embeddings from token_ids.
- Use positions from zero through S minus one for every batch row.
- Look up segment embeddings from segment_ids.
- Return the elementwise sum as a float64 NumPy array.
Constraints
- token_ids and segment_ids have shape (B, S) and dtype int64.
- Every segment ID is zero or one.
- token_embeddings has shape (V, H).
- position_embeddings has at least S rows and shape (P, H).
- segment_embeddings has shape (2, H).
- All embedding tables have dtype float64.
All steps in BERT
Accepts: array
Accepts: array
Accepts: array
Accepts: array
Accepts: array