Implement BERT WordPiece tokenization with greedy longest-match-first segmentation. Lowercase the input and split it on whitespace. For each word, choose the longest vocabulary token beginning at the current character. A token after the first piece must use the continuation prefix ##. If a word cannot be fully segmented, or its length exceeds max_word_length, emit only unk_token for that word.
For example, when play and ##ing are in the vocabulary, playing becomes play followed by ##ing. Return all pieces from all words as a Python list of strings in their original word order.
Examples
Input: text = "Playing cats", vocab = {"play":1,"##ing":2,"cat":3,"##s":4,"[UNK]":0}, unk_token = "[UNK]", max_word_length = 20
Output: ["play","##ing","cat","##s"]
Explanation: The tokenizer chooses play first, then matches the remaining characters with the continuation token ##ing.
Input: text = "unaffable", vocab = {"un":1,"##aff":2,"##able":3,"##a":4,"##ff":5,"[UNK]":0}, unk_token = "[UNK]", max_word_length = 20
Output: ["un","##aff","##able"]
Input: text = "known mystery", vocab = {"known":1,"my":2,"[UNK]":0}, unk_token = "[UNK]", max_word_length = 20
Output: ["known","[UNK]"]
Hint 1
Move a start index through each word and shrink an end index until vocab contains the candidate.
Hint 2
Add ## to a candidate whenever start is greater than zero.
Hint 3
Discard partial pieces if any remaining character cannot be matched.
Requirements
- Use ordinary Python string and list operations.
- Lowercase the text before tokenizing.
- Use greedy longest-match-first segmentation for each word.
- Prefix continuation candidates with ##.
- Replace an entire unsegmentable or overlong word with unk_token.
- Return a Python list of strings.
Constraints
- text contains words separated by whitespace.
- vocab is a dictionary whose keys are token strings.
- unk_token is present in vocab.
- max_word_length is a positive integer.
- Continuation tokens in vocab include the ## prefix.
All steps in BERT
Related practice
Accepts: string
Accepts: object
Accepts: string
Accepts: number
WordPiece Tokenization
Implement BERT WordPiece tokenization with greedy longest-match-first segmentation. Lowercase the input and split it on whitespace. For each word, choose the longest vocabulary token beginning at the current character. A token after the first piece must use the continuation prefix ##. If a word cannot be fully segmented, or its length exceeds max_word_length, emit only unk_token for that word.
For example, when play and ##ing are in the vocabulary, playing becomes play followed by ##ing. Return all pieces from all words as a Python list of strings in their original word order.
Examples
Input: text = "Playing cats", vocab = {"play":1,"##ing":2,"cat":3,"##s":4,"[UNK]":0}, unk_token = "[UNK]", max_word_length = 20
Output: ["play","##ing","cat","##s"]
Explanation: The tokenizer chooses play first, then matches the remaining characters with the continuation token ##ing.
Input: text = "unaffable", vocab = {"un":1,"##aff":2,"##able":3,"##a":4,"##ff":5,"[UNK]":0}, unk_token = "[UNK]", max_word_length = 20
Output: ["un","##aff","##able"]
Input: text = "known mystery", vocab = {"known":1,"my":2,"[UNK]":0}, unk_token = "[UNK]", max_word_length = 20
Output: ["known","[UNK]"]
Hint 1
Move a start index through each word and shrink an end index until vocab contains the candidate.
Hint 2
Add ## to a candidate whenever start is greater than zero.
Hint 3
Discard partial pieces if any remaining character cannot be matched.
Requirements
- Use ordinary Python string and list operations.
- Lowercase the text before tokenizing.
- Use greedy longest-match-first segmentation for each word.
- Prefix continuation candidates with ##.
- Replace an entire unsegmentable or overlong word with unk_token.
- Return a Python list of strings.
Constraints
- text contains words separated by whitespace.
- vocab is a dictionary whose keys are token strings.
- unk_token is present in vocab.
- max_word_length is a positive integer.
- Continuation tokens in vocab include the ## prefix.
All steps in BERT
Related practice
Accepts: string
Accepts: object
Accepts: string
Accepts: number