Build deterministic MinHash signatures for tokenized documents. A shingle is one consecutive token window, encoded as a JSON array with Unicode preserved and no separator spaces. Repeated shingles count once.
Hash columns are numbered from zero. A shingle's hash in column h is SHA-256 of the UTF-8 text seed:h: followed by its encoded shingle, using decimal seed and column values. Interpret the first four digest bytes as an unsigned big-endian integer. Each signature entry is the smallest hash for that document and column; a document without shingles uses 232−1.
For nonempty shingle sets, estimate similarity by
J(i,j)=H1h=0∑H−11[si,h=sj,h].Here, i and j index documents, H is the number of hash columns, s is the signature matrix, and the indicator is one when the two entries match. Diagonal entries and pairs of empty shingle sets have similarity 1; empty/nonempty pairs have similarity 0.
Return a Python dictionary with signatures, a uint32 NumPy array of shape (N, H), and similarities, a float64 NumPy array of shape (N, N), where N is the number of documents.
Input: documents = [["a","b","c"],["a","b","c"]], shingle_size = 2, num_hashes = 2, seed = 7
Output: {"signatures":[[1402394988,3748241178],[1402394988,3748241178]],"similarities":[[1,1],[1,1]]}
Explanation: Identical shingle sets choose the same minimum in every hash column.
Input: documents = [["a"],["a","b"]], shingle_size = 2, num_hashes = 2, seed = 3
Output: {"signatures":[[4294967295,4294967295],[936885737,1343921959]],"similarities":[[1,0],[0,1]]}
Input: documents = [["a","b","c"],["a","b","d"]], shingle_size = 2, num_hashes = 2, seed = 11
Output: {"signatures":[[1936632583,3094012498],[473285690,1512557262]],"similarities":[[1,0],[0,1]]}
json.dumps supports compact separators and unescaped Unicode.
int.from_bytes reads a byte sequence in a chosen byte order.
NumPy broadcasting compares signature rows across all columns.
Accepts: array
Accepts: number
Accepts: number
Accepts: number
Build deterministic MinHash signatures for tokenized documents. A shingle is one consecutive token window, encoded as a JSON array with Unicode preserved and no separator spaces. Repeated shingles count once.
Hash columns are numbered from zero. A shingle's hash in column h is SHA-256 of the UTF-8 text seed:h: followed by its encoded shingle, using decimal seed and column values. Interpret the first four digest bytes as an unsigned big-endian integer. Each signature entry is the smallest hash for that document and column; a document without shingles uses 232−1.
For nonempty shingle sets, estimate similarity by
J(i,j)=H1h=0∑H−11[si,h=sj,h].Here, i and j index documents, H is the number of hash columns, s is the signature matrix, and the indicator is one when the two entries match. Diagonal entries and pairs of empty shingle sets have similarity 1; empty/nonempty pairs have similarity 0.
Return a Python dictionary with signatures, a uint32 NumPy array of shape (N, H), and similarities, a float64 NumPy array of shape (N, N), where N is the number of documents.
Input: documents = [["a","b","c"],["a","b","c"]], shingle_size = 2, num_hashes = 2, seed = 7
Output: {"signatures":[[1402394988,3748241178],[1402394988,3748241178]],"similarities":[[1,1],[1,1]]}
Explanation: Identical shingle sets choose the same minimum in every hash column.
Input: documents = [["a"],["a","b"]], shingle_size = 2, num_hashes = 2, seed = 3
Output: {"signatures":[[4294967295,4294967295],[936885737,1343921959]],"similarities":[[1,0],[0,1]]}
Input: documents = [["a","b","c"],["a","b","d"]], shingle_size = 2, num_hashes = 2, seed = 11
Output: {"signatures":[[1936632583,3094012498],[473285690,1512557262]],"similarities":[[1,0],[0,1]]}
json.dumps supports compact separators and unescaped Unicode.
int.from_bytes reads a byte sequence in a chosen byte order.
NumPy broadcasting compares signature rows across all columns.
Accepts: array
Accepts: number
Accepts: number
Accepts: number
