Deduplicate documents while retaining the earliest occurrence of each normalized text. Apply Unicode NFKC normalization, then Unicode case folding when lowercase is true. When collapse_whitespace is true, also trim the text and replace each Unicode whitespace run with one ASCII space.
Bucket keys are the low-order hash_bits bits of the normalized text's SHA-256 digest, using UTF-8 encoding and a big-endian unsigned digest integer. Hash collisions do not imply equality: different normalized texts must remain separate. Document IDs are not normalized.
Return a Python dictionary with retained_ids, a list of retained string IDs in input order, and removed_to_retained, a dictionary mapping each removed string ID to its earliest retained matching ID.
Input: documents = [{"id":"a","text":" Hello World "},{"id":"b","text":"hello world"},{"id":"c","text":"HELLO WORLD!"}], lowercase = true, collapse_whitespace = true, hash_bits = 64
Output: {"retained_ids":["a","c"],"removed_to_retained":{"b":"a"}}
Explanation: The first two texts normalize identically, while punctuation keeps the third text distinct.
Input: documents = [{"id":"x","text":"Ｓtraße"},{"id":"y","text":"STRASSE"}], lowercase = true, collapse_whitespace = true, hash_bits = 64
Output: {"retained_ids":["x"],"removed_to_retained":{"y":"x"}}
Input: documents = [{"id":"a","text":"a"},{"id":"b","text":"b"}], lowercase = false, collapse_whitespace = false, hash_bits = 1
Output: {"retained_ids":["a","b"],"removed_to_retained":{}}
unicodedata.normalize supports Unicode compatibility normalization.
str.casefold handles Unicode caseless matching.
hashlib.sha256 accepts encoded bytes.
Accepts: array
Accepts: number
Deduplicate documents while retaining the earliest occurrence of each normalized text. Apply Unicode NFKC normalization, then Unicode case folding when lowercase is true. When collapse_whitespace is true, also trim the text and replace each Unicode whitespace run with one ASCII space.
Bucket keys are the low-order hash_bits bits of the normalized text's SHA-256 digest, using UTF-8 encoding and a big-endian unsigned digest integer. Hash collisions do not imply equality: different normalized texts must remain separate. Document IDs are not normalized.
Return a Python dictionary with retained_ids, a list of retained string IDs in input order, and removed_to_retained, a dictionary mapping each removed string ID to its earliest retained matching ID.
Input: documents = [{"id":"a","text":" Hello World "},{"id":"b","text":"hello world"},{"id":"c","text":"HELLO WORLD!"}], lowercase = true, collapse_whitespace = true, hash_bits = 64
Output: {"retained_ids":["a","c"],"removed_to_retained":{"b":"a"}}
Explanation: The first two texts normalize identically, while punctuation keeps the third text distinct.
Input: documents = [{"id":"x","text":"Ｓtraße"},{"id":"y","text":"STRASSE"}], lowercase = true, collapse_whitespace = true, hash_bits = 64
Output: {"retained_ids":["x"],"removed_to_retained":{"y":"x"}}
Input: documents = [{"id":"a","text":"a"},{"id":"b","text":"b"}], lowercase = false, collapse_whitespace = false, hash_bits = 1
Output: {"retained_ids":["a","b"],"removed_to_retained":{}}
unicodedata.normalize supports Unicode compatibility normalization.
str.casefold handles Unicode caseless matching.
hashlib.sha256 accepts encoded bytes.
Accepts: array
Accepts: number
