01/08Tokenization
Implement a deterministic word-level tokenizer used as preparation for the Transformer exercises. This is a simplified tokenizer, not the subword system used in the original paper.
build_vocab must assign IDs 0 through 3 to PAD, UNK, BOS, and EOS, then assign IDs to lowercased training words in sorted order. encode lowercases and whitespace-splits text, using UNK for unseen words. decode maps IDs back to tokens, using UNK for unknown IDs. Return a list of integer IDs from encode and a space-separated string from decode.
Examples
Input: text = "hello world"
Output: [5, 9]
Explanation: Vocab from ["hello world", "this is a test", "hello test"]. After 4 special tokens, sorted words: a=4, hello=5, is=6, test=7, this=8, world=9
Input: text = "hello unknown"
Output: [5, 1]
Input: text = "foo bar baz"
Output: [1, 1, 1]
Hint 1
Insert the four special tokens before collecting ordinary words.
Hint 2
Use a set to collect lowercased whitespace-separated words, then sort it.
Hint 3
Use dictionary get with the UNK ID or token as the fallback.
Requirements
- Implement build_vocab, encode, and decode.
- Preserve the four fixed special-token IDs.
- Lowercase text before vocabulary collection and encoding.
- Add ordinary vocabulary words in sorted order.
- Set vocab_size after building.
Constraints
- texts is a list of strings.
- encode returns a list of integers.
- decode returns one space-separated string.
- Valid inputs are guaranteed.
Accepts: string
01/08Tokenization
Implement a deterministic word-level tokenizer used as preparation for the Transformer exercises. This is a simplified tokenizer, not the subword system used in the original paper.
build_vocab must assign IDs 0 through 3 to PAD, UNK, BOS, and EOS, then assign IDs to lowercased training words in sorted order. encode lowercases and whitespace-splits text, using UNK for unseen words. decode maps IDs back to tokens, using UNK for unknown IDs. Return a list of integer IDs from encode and a space-separated string from decode.
Examples
Input: text = "hello world"
Output: [5, 9]
Explanation: Vocab from ["hello world", "this is a test", "hello test"]. After 4 special tokens, sorted words: a=4, hello=5, is=6, test=7, this=8, world=9
Input: text = "hello unknown"
Output: [5, 1]
Input: text = "foo bar baz"
Output: [1, 1, 1]
Hint 1
Insert the four special tokens before collecting ordinary words.
Hint 2
Use a set to collect lowercased whitespace-separated words, then sort it.
Hint 3
Use dictionary get with the UNK ID or token as the fallback.
Requirements
- Implement build_vocab, encode, and decode.
- Preserve the four fixed special-token IDs.
- Lowercase text before vocabulary collection and encoding.
- Add ordinary vocabulary words in sorted order.
- Set vocab_size after building.
Constraints
- texts is a list of strings.
- encode returns a list of integers.
- decode returns one space-separated string.
- Valid inputs are guaranteed.
Accepts: string