Problems
Loading...
Step 4 of 6
SGNS Loss
Medium

Implement the Skip-gram with Negative Sampling loss for one positive center-context pair and a set of sampled negative words.

L=−log⁡σ(uoTvc)−∑i=1Klog⁡σ(−uiTvc)\mathcal{L}=-\log\sigma(u_o^{\mathsf T}v_c)-\sum_{i=1}^{K}\log\sigma(-u_i^{\mathsf T}v_c)

Here, vcv_c is center_vec, uou_o is pos_vec, uiu_i are the rows of neg_vecs, KK is the number of negatives, and σ\sigma is the logistic sigmoid. Return the loss as a scalar float64 PyTorch tensor and keep it finite for large finite dot products.

Loading visualization...

Examples

Input: center_vec = [0.5,-0.2], pos_vec = [0.3,0.4], neg_vecs = [[-0.1,0.2],[0.4,-0.3]]

Output: 2.139492

Explanation: The loss combines one positive softplus term with the sum of the two negative terms.

Input: center_vec = [0.2,0.1,-0.4], pos_vec = [0.5,-0.2,0.3], neg_vecs = [[0.1,0.1,0.1]]

Output: 1.401507

Input: center_vec = [0.1,-0.2,0.3,-0.4], pos_vec = [0.2,0.2,-0.1,0.5], neg_vecs = [[0.1,0,-0.2,0.3],[-0.4,0.1,0.2,-0.1],[0,0.3,-0.3,0.2]]

Output: 2.735787

Hint 1

Use torch.dot for the positive pair and a matrix-vector product for all negatives.

Hint 2

The stable loss is F.softplus(-positive_score) plus F.softplus(negative_scores).sum().

Requirements

  • Use PyTorch.
  • Compute one positive score and all supplied negative scores.
  • Sum the negative terms rather than averaging them.
  • Use a numerically stable log-sigmoid or softplus formulation.
  • Return a scalar float64 tensor.

Constraints

  • center_vec and pos_vec have shape (D) and dtype torch.float64.
  • neg_vecs has shape (K, D) and dtype torch.float64.
  • D and K are positive.
  • All inputs contain finite values.
Prev4 / 6Next