Problems
Loading...
Step 3 of 6
Masked Language Modeling
Medium

Apply BERT's deterministic 80-10-10 masking rule to positions selected for masked-language-model training. The supplied replace_probs value chooses the action at each selected position:

x~i={mask token,ri<0.8random token,0.8≤ri<0.9xi,ri≥0.9\widetilde{x}_{i} = \begin{cases} \text{mask token}, & r_i < 0.8 \\ \text{random token}, & 0.8 \le r_i < 0.9 \\ x_i, & r_i \ge 0.9 \end{cases}

Here, xix_i is the original token ID and rir_i is the supplied replacement probability. Unselected tokens remain unchanged. Labels contain the original token ID at selected positions and -100 elsewhere. Return a Python dictionary with exactly masked_ids and labels. Both values must be int64 NumPy arrays with the same shape as token_ids.

Loading visualization...

Examples

Input: token_ids = [[101,2003,1037,102]], mask_positions = [[false,true,true,false]], replace_probs = [[0,0.5,0.85,0]], random_tokens = [[0,500,200,0]], mask_token_id = 103

Output: {"masked_ids":[[101,103,200,102]],"labels":[[-100,2003,1037,-100]]}

Explanation: The first selected token uses the mask ID, while the second uses its supplied random token because its probability lies in [0.8, 0.9).

Input: token_ids = [[10,20,30]], mask_positions = [[true,true,true]], replace_probs = [[0.1,0.95,0.82]], random_tokens = [[99,99,55]], mask_token_id = 103

Output: {"masked_ids":[[103,20,55]],"labels":[[10,20,30]]}

Input: token_ids = [[5,10,15]], mask_positions = [[false,false,false]], replace_probs = [[0,0,0]], random_tokens = [[0,0,0]], mask_token_id = 103

Output: {"masked_ids":[[5,10,15]],"labels":[[-100,-100,-100]]}

Hint 1

Initialize labels with np.full_like(token_ids, -100).

Hint 2

Use boolean masks for the three replacement intervals.

Hint 3

Copy token_ids before changing selected positions.

Requirements

  • Use NumPy and the supplied replacement decisions.
  • Do not sample random values inside the function.
  • Apply the three replacement intervals only at selected positions.
  • Store original IDs in labels only at selected positions.
  • Return exactly the two documented int64 arrays.

Constraints

  • token_ids and random_tokens have the same shape (B, S) and dtype int64.
  • mask_positions has shape (B, S) and dtype boolean.
  • replace_probs has shape (B, S), dtype float64, and values in [0, 1].
  • mask_token_id is an integer.
  • B and S are positive.
Prev3 / 6Next