Build a bigram language model with add-one smoothing. Sort the unique tokens to obtain vocabulary vocab. Row i represents context token vocab[i], and column j represents next token vocab[j].
P(wj∣wi)=∑u=1VCiu+VCij+1Here, Cij is the number of adjacent occurrences of wi followed by wj, and V is vocabulary size. Return vocab as a list, counts as an integer NumPy matrix, and probabilities as a floating-point NumPy matrix in a dictionary.
Input: tokens = ["a", "b", "a"]
Output: {"vocab": ["a", "b"], "counts": [[0, 1], [1, 0]], "probabilities": [[0.333333, 0.666667], [0.666667, 0.333333]]}
Explanation: The observed transitions are a to b and b to a; add-one smoothing also assigns probability to unseen pairs.
Input: tokens = ["i", "love", "ml", "love", "ml"]
Output: {"vocab": ["i", "love", "ml"], "counts": [[0, 1, 0], [0, 0, 2], [0, 1, 0]], "probabilities": [[0.25, 0.5, 0.25], [0.2, 0.2, 0.6], [0.25, 0.5, 0.25]]}
Create an index dictionary from sorted(set(tokens)).
Increment counts[index[first], index[second]] for adjacent pairs.
Divide counts + 1 by (counts.sum(axis=1, keepdims=True) + vocab_size).
Accepts: array
Build a bigram language model with add-one smoothing. Sort the unique tokens to obtain vocabulary vocab. Row i represents context token vocab[i], and column j represents next token vocab[j].
P(wj∣wi)=∑u=1VCiu+VCij+1Here, Cij is the number of adjacent occurrences of wi followed by wj, and V is vocabulary size. Return vocab as a list, counts as an integer NumPy matrix, and probabilities as a floating-point NumPy matrix in a dictionary.
Input: tokens = ["a", "b", "a"]
Output: {"vocab": ["a", "b"], "counts": [[0, 1], [1, 0]], "probabilities": [[0.333333, 0.666667], [0.666667, 0.333333]]}
Explanation: The observed transitions are a to b and b to a; add-one smoothing also assigns probability to unseen pairs.
Input: tokens = ["i", "love", "ml", "love", "ml"]
Output: {"vocab": ["i", "love", "ml"], "counts": [[0, 1, 0], [0, 0, 2], [0, 1, 0]], "probabilities": [[0.25, 0.5, 0.25], [0.2, 0.2, 0.6], [0.25, 0.5, 0.25]]}
Create an index dictionary from sorted(set(tokens)).
Increment counts[index[first], index[second]] for adjacent pairs.
Divide counts + 1 by (counts.sum(axis=1, keepdims=True) + vocab_size).
Accepts: array