Choose one token for each row of logits using temperature scaling, a top-k filter, a top-p (nucleus) filter and inverse-CDF sampling with the supplied uniform numbers.
For every row, apply these rules in order.
Here logits has shape (B, V) for B rows over a vocabulary of V tokens, uniform has shape (B,) with one number per row, temperature is the number the logits are divided by, top_k is how many tokens the top-k filter keeps, and top_p is the nucleus threshold.
Your function is called under jax.jit with logits and uniform traced and with temperature, top_k and top_p passed as static Python numbers, so every intermediate array needs a shape that does not depend on the data.
Return an int32 JAX array of shape (B,) holding the chosen token index for each row.
Input: logits = [[2.0,1.0,0.5,-1.0]], uniform = [0.8], temperature = 1.0, top_k = 4, top_p = 1.0
Output: [1]
Explanation: Nothing is filtered, so the probabilities are the softmax of [2, 1, 0.5, -1], about [0.6095, 0.2242, 0.136, 0.0303]. The running totals are about [0.6095, 0.8337, 0.9697, 1], and the first one greater than u = 0.8 is at index 1.
Input: logits = [[0.1,1.5,1.2,-0.3,0.9],[2.0,-1.0,0.0,1.0,0.5]], uniform = [0.9,0.3], temperature = 0.5, top_k = 2, top_p = 1.0
Output: [2,0]
Input: logits = [[1.0,3.0,2.5,0.0,-2.0,1.5]], uniform = [0.95], temperature = 1.0, top_k = 6, top_p = 0.6
Output: [2]
values, idx = jax.lax.top_k(logits, k); values[..., -1:] is the cut-off.
jnp.sort then jnp.cumsum along axis=-1 gives the mass ranked before each token.
jnp.argmax(cum > u, axis=-1) returns the first index where the condition is True.
Accepts: array
Accepts: array
Accepts: number
Accepts: integer
Accepts: number
Choose one token for each row of logits using temperature scaling, a top-k filter, a top-p (nucleus) filter and inverse-CDF sampling with the supplied uniform numbers.
For every row, apply these rules in order.
Here logits has shape (B, V) for B rows over a vocabulary of V tokens, uniform has shape (B,) with one number per row, temperature is the number the logits are divided by, top_k is how many tokens the top-k filter keeps, and top_p is the nucleus threshold.
Your function is called under jax.jit with logits and uniform traced and with temperature, top_k and top_p passed as static Python numbers, so every intermediate array needs a shape that does not depend on the data.
Return an int32 JAX array of shape (B,) holding the chosen token index for each row.
Input: logits = [[2.0,1.0,0.5,-1.0]], uniform = [0.8], temperature = 1.0, top_k = 4, top_p = 1.0
Output: [1]
Explanation: Nothing is filtered, so the probabilities are the softmax of [2, 1, 0.5, -1], about [0.6095, 0.2242, 0.136, 0.0303]. The running totals are about [0.6095, 0.8337, 0.9697, 1], and the first one greater than u = 0.8 is at index 1.
Input: logits = [[0.1,1.5,1.2,-0.3,0.9],[2.0,-1.0,0.0,1.0,0.5]], uniform = [0.9,0.3], temperature = 0.5, top_k = 2, top_p = 1.0
Output: [2,0]
Input: logits = [[1.0,3.0,2.5,0.0,-2.0,1.5]], uniform = [0.95], temperature = 1.0, top_k = 6, top_p = 0.6
Output: [2]
values, idx = jax.lax.top_k(logits, k); values[..., -1:] is the cut-off.
jnp.sort then jnp.cumsum along axis=-1 gives the mass ranked before each token.
jnp.argmax(cum > u, axis=-1) returns the first index where the condition is True.
Accepts: array
Accepts: array
Accepts: number
Accepts: integer
Accepts: number