Perplexity is the standard metric for evaluating language models. It measures how "surprised" a model is by a sequence of tokens. A lower perplexity means the model assigns higher probability to the observed sequence, indicating better predictions.
Given a list of probability distributions (one per position) and the actual token indices, compute the perplexity of the sequence.
Input:
prob_distributions = [[0.5, 0.5], [0.5, 0.5]], actual_tokens = [0, 1]
Output:
2.0
Each token has probability 0.5. Cross-entropy H = -average(log(0.5), log(0.5)) = log(2). Perplexity = exp(log(2)) = 2.0. The model is as uncertain as a fair coin flip.
Input:
prob_distributions = [[1.0, 0.0], [0.0, 1.0]], actual_tokens = [0, 1]
Output:
1.0
The model assigns probability 1.0 to each correct token. Cross-entropy = 0. Perplexity = exp(0) = 1.0. A perfect model has perplexity 1.
For each position i, get p = prob_distributions[i][actual_tokens[i]]. Sum up log(p) for all positions. Divide by the number of tokens. Negate. Then exponentiate the result.
Use math.log for natural logarithm and math.exp for exponentiation. The formula is: exp(-1/N * sum(log(p_i))). Be careful with the sign: cross-entropy is the negative of the mean log-probability.
Sign in to take notes on this problem
Accepts: array
Accepts: array
Perplexity is the standard metric for evaluating language models. It measures how "surprised" a model is by a sequence of tokens. A lower perplexity means the model assigns higher probability to the observed sequence, indicating better predictions.
Given a list of probability distributions (one per position) and the actual token indices, compute the perplexity of the sequence.
Input:
prob_distributions = [[0.5, 0.5], [0.5, 0.5]], actual_tokens = [0, 1]
Output:
2.0
Each token has probability 0.5. Cross-entropy H = -average(log(0.5), log(0.5)) = log(2). Perplexity = exp(log(2)) = 2.0. The model is as uncertain as a fair coin flip.
Input:
prob_distributions = [[1.0, 0.0], [0.0, 1.0]], actual_tokens = [0, 1]
Output:
1.0
The model assigns probability 1.0 to each correct token. Cross-entropy = 0. Perplexity = exp(0) = 1.0. A perfect model has perplexity 1.
For each position i, get p = prob_distributions[i][actual_tokens[i]]. Sum up log(p) for all positions. Divide by the number of tokens. Negate. Then exponentiate the result.
Use math.log for natural logarithm and math.exp for exponentiation. The formula is: exp(-1/N * sum(log(p_i))). Be careful with the sign: cross-entropy is the negative of the mean log-probability.
Sign in to take notes on this problem
Accepts: array
Accepts: array