Train a deterministic byte-pair encoding vocabulary from the UTF-8 bytes of the supplied strings. The initial vocabulary maps IDs 0 through 255 to their corresponding single bytes; vocab_size includes these initial tokens.
Each round merges the most frequent adjacent token pair across the corpus. Count overlapping pairs within each string, never across string boundaries. Frequency ties select the lexicographically greater pair of byte strings, not the greater token IDs. Replace matches without overlap, from left to right. New token IDs start at 256 and increase by one. Stop when the target size is reached or no pair remains; a target of 256 or less adds no tokens.
Return a Python dictionary with exactly two keys: vocab, a list of learned [token_id, byte_values] entries in ID order, and merges, a list of [left_id, right_id, new_id] entries in creation order. All IDs and byte values are Python integers; exclude the initial 256 tokens from vocab.
Input: corpus = ["aaab"], vocab_size = 257
Output: {"vocab":[[256,[97,97]]],"merges":[[97,97,256]]}
Explanation: The pair [97,97] occurs twice, so it becomes token 256 with byte values [97,97].
Input: corpus = ["ab","cd"], vocab_size = 257
Output: {"vocab":[[256,[99,100]]],"merges":[[99,100,256]]}
Input: corpus = ["abc"], vocab_size = 1000
Output: {"vocab":[[256,[98,99]],[257,[97,98,99]]],"merges":[[98,99,256],[97,256,257]]}
str.encode converts Unicode text to UTF-8 bytes.
dict.get can accumulate counts for tuple keys.
Python compares byte strings lexicographically.
Accepts: array
Accepts: number
Train a deterministic byte-pair encoding vocabulary from the UTF-8 bytes of the supplied strings. The initial vocabulary maps IDs 0 through 255 to their corresponding single bytes; vocab_size includes these initial tokens.
Each round merges the most frequent adjacent token pair across the corpus. Count overlapping pairs within each string, never across string boundaries. Frequency ties select the lexicographically greater pair of byte strings, not the greater token IDs. Replace matches without overlap, from left to right. New token IDs start at 256 and increase by one. Stop when the target size is reached or no pair remains; a target of 256 or less adds no tokens.
Return a Python dictionary with exactly two keys: vocab, a list of learned [token_id, byte_values] entries in ID order, and merges, a list of [left_id, right_id, new_id] entries in creation order. All IDs and byte values are Python integers; exclude the initial 256 tokens from vocab.
Input: corpus = ["aaab"], vocab_size = 257
Output: {"vocab":[[256,[97,97]]],"merges":[[97,97,256]]}
Explanation: The pair [97,97] occurs twice, so it becomes token 256 with byte values [97,97].
Input: corpus = ["ab","cd"], vocab_size = 257
Output: {"vocab":[[256,[99,100]]],"merges":[[99,100,256]]}
Input: corpus = ["abc"], vocab_size = 1000
Output: {"vocab":[[256,[98,99]],[257,[97,98,99]]],"merges":[[98,99,256],[97,256,257]]}
str.encode converts Unicode text to UTF-8 bytes.
dict.get can accumulate counts for tuple keys.
Python compares byte strings lexicographically.
Accepts: array
Accepts: number