Implement encoding with ordered byte-pair merge rules and decoding with a supplied byte vocabulary. Encoding starts from the text's UTF-8 byte IDs. Apply each rule once in list order, replacing all non-overlapping matches from left to right. Later rules may use tokens produced by earlier rules.
(lk,rk)→nkHere k is the rule's position, l and r are its left and right token IDs, and n is the replacement ID. Decoding concatenates the vocabulary bytes in token order before interpreting the complete sequence as UTF-8. Empty inputs produce empty outputs.
Return a Python list of integer token IDs from encode and a Python string from decode.
Input: text = "low", merges = [[108,111,256],[256,119,257]]
Output: [257]
Explanation: The first rule changes [108,111,119] to [256,119], and the second changes it to [257].
Input: ids = [257], vocab = {256:[108,111],257:[108,111,119]}
Output: "low"
Input: text = "aaa", merges = [[97,97,256]]
Output: [256,97]
str.encode produces UTF-8 bytes for the initial token IDs.
A separate output list avoids shifting positions during a merge pass.
list.extend combines byte lists, and bytes.decode converts the complete sequence to text.
Accepts: string
Accepts: array
Implement encoding with ordered byte-pair merge rules and decoding with a supplied byte vocabulary. Encoding starts from the text's UTF-8 byte IDs. Apply each rule once in list order, replacing all non-overlapping matches from left to right. Later rules may use tokens produced by earlier rules.
(lk,rk)→nkHere k is the rule's position, l and r are its left and right token IDs, and n is the replacement ID. Decoding concatenates the vocabulary bytes in token order before interpreting the complete sequence as UTF-8. Empty inputs produce empty outputs.
Return a Python list of integer token IDs from encode and a Python string from decode.
Input: text = "low", merges = [[108,111,256],[256,119,257]]
Output: [257]
Explanation: The first rule changes [108,111,119] to [256,119], and the second changes it to [257].
Input: ids = [257], vocab = {256:[108,111],257:[108,111,119]}
Output: "low"
Input: text = "aaa", merges = [[97,97,256]]
Output: [256,97]
str.encode produces UTF-8 bytes for the initial token IDs.
A separate output list avoids shifting positions during a merge pass.
list.extend combines byte lists, and bytes.decode converts the complete sequence to text.
Accepts: string
Accepts: array
