Perplexity Computation
Perplexity Computation
Perplexity is the standard metric for evaluating language models. It measures how "surprised" a model is by a sequence of tokens. A lower perplexity means the model assigns higher probability to the observed sequence, indicating better predictions.
Given a list of probability distributions (one per position) and the actual token indices, compute the perplexity of the sequence.
Algorithm
- For each position i, extract the probability assigned to the actual token:
- Compute the average negative log-probability (cross-entropy):
- Perplexity is the exponential of the cross-entropy:
Examples
Input:
prob_distributions = [[0.5, 0.5], [0.5, 0.5]], actual_tokens = [0, 1]
Output:
2.0
Each token has probability 0.5. Cross-entropy H = -average(log(0.5), log(0.5)) = log(2). Perplexity = exp(log(2)) = 2.0. The model is as uncertain as a fair coin flip.
Input:
prob_distributions = [[1.0, 0.0], [0.0, 1.0]], actual_tokens = [0, 1]
Output:
1.0
The model assigns probability 1.0 to each correct token. Cross-entropy = 0. Perplexity = exp(0) = 1.0. A perfect model has perplexity 1.
Hint 1
For each position i, get p = prob_distributions[i][actual_tokens[i]]. Sum up log(p) for all positions. Divide by the number of tokens. Negate. Then exponentiate the result.
Hint 2
Use math.log for natural logarithm and math.exp for exponentiation. The formula is: exp(-1/N * sum(log(p_i))). Be careful with the sign: cross-entropy is the negative of the mean log-probability.
Requirements
- Extract the predicted probability for each actual token from the corresponding distribution
- Compute cross-entropy as the negative mean of log-probabilities
- Return perplexity as exp(cross-entropy)
- Return a float
Constraints
- prob_distributions is a list of N probability distributions (each a list of positive floats summing to 1)
- actual_tokens is a list of N valid indices into the corresponding distributions
- All relevant probabilities are positive (no log(0) issues)
- Return a positive float
- Time limit: 300 ms
Try Similar Problems
Log in to take notes on this problem
Accepts: array
Accepts: array
Perplexity Computation
Perplexity Computation
Perplexity is the standard metric for evaluating language models. It measures how "surprised" a model is by a sequence of tokens. A lower perplexity means the model assigns higher probability to the observed sequence, indicating better predictions.
Given a list of probability distributions (one per position) and the actual token indices, compute the perplexity of the sequence.
Algorithm
- For each position i, extract the probability assigned to the actual token:
- Compute the average negative log-probability (cross-entropy):
- Perplexity is the exponential of the cross-entropy:
Examples
Input:
prob_distributions = [[0.5, 0.5], [0.5, 0.5]], actual_tokens = [0, 1]
Output:
2.0
Each token has probability 0.5. Cross-entropy H = -average(log(0.5), log(0.5)) = log(2). Perplexity = exp(log(2)) = 2.0. The model is as uncertain as a fair coin flip.
Input:
prob_distributions = [[1.0, 0.0], [0.0, 1.0]], actual_tokens = [0, 1]
Output:
1.0
The model assigns probability 1.0 to each correct token. Cross-entropy = 0. Perplexity = exp(0) = 1.0. A perfect model has perplexity 1.
Hint 1
For each position i, get p = prob_distributions[i][actual_tokens[i]]. Sum up log(p) for all positions. Divide by the number of tokens. Negate. Then exponentiate the result.
Hint 2
Use math.log for natural logarithm and math.exp for exponentiation. The formula is: exp(-1/N * sum(log(p_i))). Be careful with the sign: cross-entropy is the negative of the mean log-probability.
Requirements
- Extract the predicted probability for each actual token from the corresponding distribution
- Compute cross-entropy as the negative mean of log-probabilities
- Return perplexity as exp(cross-entropy)
- Return a float
Constraints
- prob_distributions is a list of N probability distributions (each a list of positive floats summing to 1)
- actual_tokens is a list of N valid indices into the corresponding distributions
- All relevant probabilities are positive (no log(0) issues)
- Return a positive float
- Time limit: 300 ms
Try Similar Problems
Log in to take notes on this problem
Accepts: array
Accepts: array