Verify a sequence of proposed tokens using supplied draft and target probabilities and random values. For zero-based draft position i, let xi be the proposed token, qi(v) its draft probability for token v, and pi(v) its target probability. The acceptance probability is:
ai=min(1,qi(xi)pi(xi)).Accept when the supplied acceptance value at position i is strictly less than ai. When the draft probability is zero, use acceptance one if the target probability is positive and zero otherwise. At the first rejection, the replacement distribution is:
ri(v)=∑umax(pi(u)−qi(u),0)max(pi(v)−qi(v),0).Here, ri(v) is the replacement probability and u ranges over the vocabulary. Choose the smallest token index whose cumulative replacement probability is strictly greater than residual_random_value; if rounding leaves the cumulative sum too small, use the last index with positive replacement probability. Emit the accepted prefix followed by this replacement and stop. If all proposals are accepted, return the draft unchanged without an extra token.
Return a Python dictionary with tokens, an int64 PyTorch tensor on the draft-token device; accepted_count, a Python integer counting accepted proposals only; and rejected_at, the zero-based rejected position as a Python integer, or None when all proposals are accepted.
Input: draft_tokens = [0], draft_probabilities = [[0.75,0.25]], target_probabilities = [[0.25,0.75]], acceptance_random_values = [0.9], residual_random_value = 0.5
Output: {"tokens":[1],"accepted_count":0,"rejected_at":0}
Explanation: The proposal is rejected, and all positive replacement probability belongs to token 1.
Input: draft_tokens = [0,1], draft_probabilities = [[0.8,0.1,0.1],[0.2,0.7,0.1]], target_probabilities = [[0.2,0.5,0.3],[0.2,0.7,0.1]], acceptance_random_values = [0.9,0.1], residual_random_value = 0.4
Output: {"tokens":[1],"accepted_count":0,"rejected_at":0}
Input: draft_tokens = [0,1], draft_probabilities = [[0.5,0.3,0.2],[0.1,0.8,0.1]], target_probabilities = [[0.6,0.2,0.2],[0.5,0.2,0.3]], acceptance_random_values = [0.2,0.8], residual_random_value = 0.75
Output: {"tokens":[0,2],"accepted_count":1,"rejected_at":1}
torch.clamp can remove negative entries from a tensor.
torch.cumsum builds cumulative probabilities along an axis.
torch.searchsorted locates a value in a sorted tensor.
Accepts: array
Accepts: array
Accepts: array
Accepts: array
Accepts: number
Verify a sequence of proposed tokens using supplied draft and target probabilities and random values. For zero-based draft position i, let xi be the proposed token, qi(v) its draft probability for token v, and pi(v) its target probability. The acceptance probability is:
ai=min(1,qi(xi)pi(xi)).Accept when the supplied acceptance value at position i is strictly less than ai. When the draft probability is zero, use acceptance one if the target probability is positive and zero otherwise. At the first rejection, the replacement distribution is:
ri(v)=∑umax(pi(u)−qi(u),0)max(pi(v)−qi(v),0).Here, ri(v) is the replacement probability and u ranges over the vocabulary. Choose the smallest token index whose cumulative replacement probability is strictly greater than residual_random_value; if rounding leaves the cumulative sum too small, use the last index with positive replacement probability. Emit the accepted prefix followed by this replacement and stop. If all proposals are accepted, return the draft unchanged without an extra token.
Return a Python dictionary with tokens, an int64 PyTorch tensor on the draft-token device; accepted_count, a Python integer counting accepted proposals only; and rejected_at, the zero-based rejected position as a Python integer, or None when all proposals are accepted.
Input: draft_tokens = [0], draft_probabilities = [[0.75,0.25]], target_probabilities = [[0.25,0.75]], acceptance_random_values = [0.9], residual_random_value = 0.5
Output: {"tokens":[1],"accepted_count":0,"rejected_at":0}
Explanation: The proposal is rejected, and all positive replacement probability belongs to token 1.
Input: draft_tokens = [0,1], draft_probabilities = [[0.8,0.1,0.1],[0.2,0.7,0.1]], target_probabilities = [[0.2,0.5,0.3],[0.2,0.7,0.1]], acceptance_random_values = [0.9,0.1], residual_random_value = 0.4
Output: {"tokens":[1],"accepted_count":0,"rejected_at":0}
Input: draft_tokens = [0,1], draft_probabilities = [[0.5,0.3,0.2],[0.1,0.8,0.1]], target_probabilities = [[0.6,0.2,0.2],[0.5,0.2,0.3]], acceptance_random_values = [0.2,0.8], residual_random_value = 0.75
Output: {"tokens":[0,2],"accepted_count":1,"rejected_at":1}
torch.clamp can remove negative entries from a tensor.
torch.cumsum builds cumulative probabilities along an axis.
torch.searchsorted locates a value in a sorted tensor.
Accepts: array
Accepts: array
Accepts: array
Accepts: array
Accepts: number