TensorTonic
Study Plans
Study Plan
Train a Deterministic BPE Vocabulary
Language Models
Medium

Train a deterministic byte-pair encoding vocabulary from the UTF-8 bytes of the supplied strings. The initial vocabulary maps IDs 0 through 255 to their corresponding single bytes; vocab_size includes these initial tokens.

Each round merges the most frequent adjacent token pair across the corpus. Count overlapping pairs within each string, never across string boundaries. Frequency ties select the lexicographically greater pair of byte strings, not the greater token IDs. Replace matches without overlap, from left to right. New token IDs start at 256 and increase by one. Stop when the target size is reached or no pair remains; a target of 256 or less adds no tokens.

Return a Python dictionary with exactly two keys: vocab, a list of learned [token_id, byte_values] entries in ID order, and merges, a list of [left_id, right_id, new_id] entries in creation order. All IDs and byte values are Python integers; exclude the initial 256 tokens from vocab.

Loading visualization...

Examples

Input: corpus = ["aaab"], vocab_size = 257

Output: {"vocab":[[256,[97,97]]],"merges":[[97,97,256]]}

Explanation: The pair [97,97] occurs twice, so it becomes token 256 with byte values [97,97].

Input: corpus = ["ab","cd"], vocab_size = 257

Output: {"vocab":[[256,[99,100]]],"merges":[[99,100,256]]}

Input: corpus = ["abc"], vocab_size = 1000

Output: {"vocab":[[256,[98,99]],[257,[97,98,99]]],"merges":[[98,99,256],[97,256,257]]}

Hint 1

str.encode converts Unicode text to UTF-8 bytes.

Hint 2

dict.get can accumulate counts for tuple keys.

Hint 3

Python compares byte strings lexicographically.

Requirements

  • Use Python strings, lists, dictionaries, and integer token IDs; NumPy is not required.
  • Preserve the supplied corpus.

Constraints

  • corpus is a Python list containing 0 to 50 Unicode strings, each at most 200 characters long.
  • vocab_size is a Python integer from 0 to 2000.
Try Similar Problems
Apply Ranked BPE MergesMediumCanonicalize Pairwise Preference DataMediumCompute Temperature-Based Data MixturesMediumParallel and Recurrent Linear AttentionMediumMamba 2 Gated State ScanMediumParameter-Matched SwiGLU BlockMediumImplement Symmetric CLIP Contrastive LossMediumPlan KV-Cache CapacityMediumImplement Prefix Cache Matching and ReuseMediumN-gram Language ModelMediumBeam Search DecodingMediumText Generation SamplingMedium

Accepts: array

Accepts: number

You must run your code first.

Train a Deterministic BPE Vocabulary

Language Models
Medium

Train a deterministic byte-pair encoding vocabulary from the UTF-8 bytes of the supplied strings. The initial vocabulary maps IDs 0 through 255 to their corresponding single bytes; vocab_size includes these initial tokens.

Each round merges the most frequent adjacent token pair across the corpus. Count overlapping pairs within each string, never across string boundaries. Frequency ties select the lexicographically greater pair of byte strings, not the greater token IDs. Replace matches without overlap, from left to right. New token IDs start at 256 and increase by one. Stop when the target size is reached or no pair remains; a target of 256 or less adds no tokens.

Return a Python dictionary with exactly two keys: vocab, a list of learned [token_id, byte_values] entries in ID order, and merges, a list of [left_id, right_id, new_id] entries in creation order. All IDs and byte values are Python integers; exclude the initial 256 tokens from vocab.

Loading visualization...

Examples

Input: corpus = ["aaab"], vocab_size = 257

Output: {"vocab":[[256,[97,97]]],"merges":[[97,97,256]]}

Explanation: The pair [97,97] occurs twice, so it becomes token 256 with byte values [97,97].

Input: corpus = ["ab","cd"], vocab_size = 257

Output: {"vocab":[[256,[99,100]]],"merges":[[99,100,256]]}

Input: corpus = ["abc"], vocab_size = 1000

Output: {"vocab":[[256,[98,99]],[257,[97,98,99]]],"merges":[[98,99,256],[97,256,257]]}

Hint 1

str.encode converts Unicode text to UTF-8 bytes.

Hint 2

dict.get can accumulate counts for tuple keys.

Hint 3

Python compares byte strings lexicographically.

Requirements

  • Use Python strings, lists, dictionaries, and integer token IDs; NumPy is not required.
  • Preserve the supplied corpus.

Constraints

  • corpus is a Python list containing 0 to 50 Unicode strings, each at most 200 characters long.
  • vocab_size is a Python integer from 0 to 2000.
Try Similar Problems
Apply Ranked BPE MergesMediumCanonicalize Pairwise Preference DataMediumCompute Temperature-Based Data MixturesMediumParallel and Recurrent Linear AttentionMediumMamba 2 Gated State ScanMediumParameter-Matched SwiGLU BlockMediumImplement Symmetric CLIP Contrastive LossMediumPlan KV-Cache CapacityMediumImplement Prefix Cache Matching and ReuseMediumN-gram Language ModelMediumBeam Search DecodingMediumText Generation SamplingMedium

Accepts: array

Accepts: number

You must run your code first.