Problems
Loading...
Step 6 of 6
SGNS Step
Medium

Implement one stochastic-gradient step for Skip-gram with Negative Sampling. Compute every gradient from the pre-update embeddings, then update the center row of the input matrix and the positive and negative rows of the output matrix.

∇uo=(σ(so)−1)vc\nabla u_o=(\sigma(s_o)-1)v_c ∇ui=σ(si)vc\nabla u_i=\sigma(s_i)v_c ∇vc=(σ(so)−1)uo+∑i=1Kσ(si)ui\nabla v_c=(\sigma(s_o)-1)u_o+\sum_{i=1}^{K}\sigma(s_i)u_i

Here, vcv_c is the center input embedding, uou_o is the positive output embedding, uiu_i are negative output embeddings, and each score is the corresponding dot product with vcv_c. Repeated negative IDs contribute repeatedly to the same row. Return a Python dictionary with exactly two keys, W_in and W_out, containing the updated float64 PyTorch tensors.

Loading visualization...

Examples

Input: W_in = [[0.1,0.2],[-0.3,0.4],[0.5,-0.1]], W_out = [[0.3,-0.2],[0.1,0.4],[-0.5,0.3]], center_id = 0, pos_id = 1, neg_ids = [2], lr = 0.1

Output: {"W_in":[[0.1299,0.204026],[-0.3,0.4],[0.5,-0.1]],"W_out":[[0.3,-0.2],[0.104775,0.40955],[-0.505025,0.28995]]}

Explanation: Only the center input row and the sampled output rows receive gradients from this training pair.

Input: W_in = [[0.64651,0.7142],[0.286,-0.22433],[0.31545,0.78284],[0.74624,0.31508]], W_out = [[-0.75734,0.69183],[-0.2522,0.24634],[-0.27092,-0.38388],[0.20596,0.70268]], center_id = 1, pos_id = 2, neg_ids = [3,1], lr = 0.05

Output: {"W_in":[[0.64651,0.7142],[0.280265,-0.256353],[0.31545,0.78284],[0.74624,0.31508]],"W_out":[[-0.75734,0.69183],[-0.258895,0.251592],[-0.263801,-0.389464],[0.199163,0.708012]]}

Input: W_in = [[0.2],[-0.1],[0.4]], W_out = [[0.3],[-0.2],[0.1]], center_id = 2, pos_id = 0, neg_ids = [1,1], lr = 0.05

Output: {"W_in":[[0.2],[-0.1],[0.416651]],"W_out":[[0.309401],[-0.2192],[0.1]]}

Hint 1

Clone the center vector and the original output rows before calculating any updates.

Hint 2

Accumulate output-row gradients in a dictionary keyed by token ID.

Hint 3

Subtract lr times each completed gradient after all scores are computed.

Requirements

  • Use PyTorch.
  • Compute all gradients from the original embeddings before applying updates.
  • Accumulate repeated negative IDs into the same output row.
  • Leave every untouched embedding row unchanged.
  • Return exactly the documented dictionary.

Constraints

  • W_in and W_out have shape (V, D) and dtype torch.float64.
  • center_id and pos_id are valid integer row indices.
  • neg_ids is a nonempty int64 tensor of valid row indices.
  • lr is positive.
  • V and D are positive.
Prev6 / 6Next