Problems
Loading...
Step 5 of 6
CBOW Forward
Medium

Implement one Continuous Bag-of-Words forward pass. CBOW predicts a center word from an unordered context by averaging the context embeddings and applying a full-vocabulary softmax.

h=1M∑j=1MWin[cj]h=\frac{1}{M}\sum_{j=1}^{M}W_{\mathrm{in}}[c_j] z=Wouthz=W_{\mathrm{out}}h L=−log⁡ ⁣(softmax⁡(z)t)\mathcal{L}=-\log\!\left(\operatorname{softmax}(z)_t\right)

Here, cjc_j are the M context token IDs, tt is target_id, and the two embedding matrices have shape (V,D)(V,D). Return the cross-entropy loss as a scalar float64 PyTorch tensor.

Loading visualization...

Examples

Input: context_ids = [0,2], target_id = 1, W_in = [[0.1,-0.2],[0.3,0.4],[-0.5,0.6],[0.7,-0.8]], W_out = [[0.2,0.1],[-0.3,0.5],[0.4,-0.6],[0.9,0]]

Output: 1.177028

Explanation: Averaging rows 0 and 2 gives the context representation used to score target word 1.

Input: context_ids = [1], target_id = 0, W_in = [[0.2,0.4],[-0.3,0.5],[0.1,-0.2]], W_out = [[0.4,-0.1],[0.2,0.3],[-0.5,0.2]]

Output: 1.33996

Input: context_ids = [0,0,2], target_id = 2, W_in = [[0.5],[-0.2],[0.1]], W_out = [[0.3],[-0.4],[0.2]]

Output: 1.043757

Hint 1

Use W_in[context_ids].mean(dim=0) for the bag-of-words representation.

Hint 2

Use F.log_softmax over the vocabulary logits.

Requirements

  • Use PyTorch.
  • Average the selected input embeddings rather than summing them.
  • Score every vocabulary entry with the output embedding matrix.
  • Compute the target loss with a numerically stable log-softmax.
  • Return a scalar float64 tensor.

Constraints

  • context_ids is a nonempty int64 tensor with shape (M).
  • target_id is a valid integer vocabulary index.
  • W_in and W_out have shape (V, D) and dtype torch.float64.
  • V, D, and M are positive.
Prev5 / 6Next