Name: TensorTonic
Author: TensorTonic

Last updated 13 October 2026

What is perplexity?

Perplexity measures how well a language model predicts a text. It is the exponential of the average negative log-likelihood of each actual next token, which is the same as the exponential of the cross-entropy loss. A perplexity of N means the model is as uncertain as if it were choosing uniformly among N tokens. Lower is better.

When you read "The cat sat on the ___", your brain instantly predicts words like "mat", "floor", or "couch". You are confident about what comes next. But for "I really want to ___", many words could follow: eat, sleep, dance, leave, scream. You are less certain.

Perplexity measures exactly this: how "surprised" or "confused" a language model is when predicting the next word. It quantifies the model's uncertainty across an entire text.

Why Do We Need Perplexity?

Language models like GPT output probability distributions over thousands of possible next tokens. But how do we compare models? We need a single number that captures "how good are these predictions overall?"

We could use raw probability, but multiplying thousands of small probabilities gives tiny numbers like 0.0000000001. Perplexity converts this into an interpretable scale.

The connection to information theory: Entropy measures uncertainty in bits. Perplexity transforms entropy into "effective number of choices" by exponentiating: $\text{PPL} = 2^H$ . This gives us a more intuitive interpretation.

The Core Idea

Perplexity = "How many equally likely choices is the model effectively picking from?"

If a language model has perplexity 50, it is "as confused as if it had to choose uniformly among 50 options at each position." The model's complex probability distribution is equivalent to rolling a 50-sided die.

Lower perplexity = higher confidence = better predictions. A model that "knows" what comes next has low perplexity.

Simple Analogy: The Guessing Game

Imagine playing 20 questions, but instead of yes/no, you guess the next word in a sentence.

Expert Player (Low PPL)

"The capital of France is ___" - You immediately say "Paris" with 95% confidence. You only need to consider ~1-2 realistic options.

Confused Player (High PPL)

"She decided to ___" - Could be anything! eat, leave, stay, cry, laugh... You are effectively choosing from 100+ options.

Perplexity Scale Reference

Perfect Model

Always 100% sure

(Unrealistic for natural language)

~20

Good LLM

GPT-level on benchmarks

(State-of-the-art range)

Random Guess

V = vocab size (~50K)

(Uniform over all tokens)

Intuition: Effective Choices

Imagine a model predicting the next word in a sentence. At each position, it assigns probabilities to all possible next tokens.

Low Perplexity (Good)

"The capital of France is ___"

Model is 95% confident it's "Paris." $\text{PPL} \approx 1.2$

High Perplexity (Confused)

"I really like to ___"

Could be eat, sleep, dance, run, etc. $\text{PPL} \approx 50+$

Real models output complex probability distributions. Perplexity converts that distribution into an equivalent "uniform over N choices" number.

Perplexity as "Effective Choices"

Perplexity measures how confused the model is. It represents the number of "equally likely options" the model is choosing from.

Distribution Sharpness (Temperature)1.00

Sharp (Confident)Flat (Uncertain)

Actual Probabilities

10 possible tokens

Perplexity Equivalent

5.27 "effective" tokens

Entropy (bits)

2.40

Perplexity

5.27

\text{PPL} = 2^{\text{Entropy}} = 2^{2.40} \approx 5.27

The formula and its link to cross-entropy

For a single distribution with entropy $H$ :

\text{PPL} = 2^{H}

Perplexity is 2 raised to the power of entropy (in bits).

For a sequence of $N$ tokens, we use the average negative log-likelihood:

\text{PPL} = \exp\left(-\frac{1}{N}\sum_{i=1}^N \ln p_{model}(x_i | x_{<i})\right)

Bits vs Nats

Using $\log_2$ gives entropy in bits and $\text{PPL} = 2^H$ . Using $\ln$ gives entropy in nats (natural log) and $\text{PPL} = e^H$ . Deep learning frameworks typically use nats, but the concepts are identical.

Why Exponentiate?

Entropy is additive: $H(A, B) = H(A) + H(B)$ for independent events.

Perplexity is multiplicative: $\text{PPL}(A, B) = \text{PPL}(A) \times \text{PPL}(B)$ . This makes it easier to interpret as "number of choices."

Minimizing Perplexity is mathematically equivalent to minimizing Cross-Entropy loss.

Entropy

H

Bits needed

Cross-Entropy

H(p, q)

Training loss

Perplexity

2^H

Effective choices

Key Relationship

$\text{PPL} = \exp(\text{Cross-Entropy Loss})$ . When your training loss goes down by 0.1, your perplexity is multiplied by $e^{-0.1} \approx 0.9$ .

Interactive: Word Prediction

See how a language model assigns probabilities to next words. Different contexts lead to different levels of certainty.

Next Word Prediction

Simulating language model probability distribution

Select Input Context

$predict_next_token--context"The cat sat on the"

Input:The cat sat on the

Token Probabilities

mat

35.0%TOP

floor

20.0%

couch

15.0%

bed

10.0%

chair

8.0%

table

5.0%

ground

4.0%

roof

3.0%

Uncertainty

2.58bits

Perplexity

6.0

Equivalent to choosing from 6.0 options.

Worked calculation

Calculate perplexity for a simple 4-word sequence:

Sequence

"The cat sat down"

Step 1: Get Model Probabilities

P("The" | <BOS>) = 0.15
P("cat" | "The") = 0.05
P("sat" | "The cat") = 0.20
P("down" | "The cat sat") = 0.25

Step 2: Compute Log Probabilities

$\ln(0.15) = -1.897$
$\ln(0.05) = -2.996$
$\ln(0.20) = -1.609$
$\ln(0.25) = -1.386$

Step 3: Average Negative Log-Likelihood

\text{NLL} = \frac{1.897 + 2.996 + 1.609 + 1.386}{4} = 1.972 \text{ nats}

Step 4: Exponentiate

\text{PPL} = e^{1.972} \approx \mathbf{7.19}

The model is "as confused as choosing from 7 equally likely words" on average.

The same calculation in NumPy. In a real evaluation, token_probs comes from the model's softmax at each position, and you average over every token in the held-out set.

Python

import numpy as np

# Probability the model gave to each actual next token in "The cat sat down"
token_probs = np.array([0.15, 0.05, 0.20, 0.25])

nll = -np.mean(np.log(token_probs))   # average negative log-likelihood, in nats
ppl = np.exp(nll)

print(f"NLL = {nll:.3f} nats")
print(f"PPL = {ppl:.2f}")

Output

NLL = 1.972 nats
PPL = 7.19

Benchmark scores in context

These are zero-shot perplexities reported in the original papers. Lower is better, and each number only makes sense next to the dataset and tokenization it was measured with.

Model	Parameters	WikiText-103	Penn Treebank
GPT-2 Small	117M	37.50	65.85
GPT-2 Medium	345M	26.37	47.33
GPT-2 Large	762M	22.05	40.31
GPT-2 XL	1.5B	17.48	35.76
GPT-3	175B	Not reported	20.5

Sources: GPT-2 from Radford et al., “Language Models are Unsupervised Multitask Learners” (OpenAI, February 2019), Table 3. GPT-3 from Brown et al., “Language Models are Few-Shot Learners” (NeurIPS 2020), Table 3.1. The GPT-3 paper skipped WikiText-103 because of overlap with its training data.

Comparing scores fairly

Perplexity values are only comparable when using the same tokenizer and same test set. A model with BPE tokenization cannot be directly compared to one with word-level tokenization, and many newer model papers stop reporting perplexity in favour of task benchmarks.

Bits Per Character (BPC)

For character-level models, we often use Bits Per Character instead of perplexity. This is tokenizer-independent.

\text{BPC} = \frac{\text{Cross-Entropy Loss}}{\ln(2)}

BPC to PPL

\text{PPL} = 2^{\text{BPC}}

Typical Values

Good character LM:

1.0\text{-}1.5

BPC

Limitations & Caveats

1. Not Task-Specific

Low perplexity measures how well a model predicts the next token, but this does not directly translate to downstream task performance. A model can be excellent at predicting common word sequences yet fail at:

Reasoning tasks: Predicting "2" after "1 + 1 =" requires understanding, not just pattern matching
Factual accuracy: A model might confidently predict plausible but incorrect facts
Instruction following: Low PPL on text does not mean the model follows user instructions well
Safety: Fluent generation of harmful content would still show low perplexity

This is why modern LLM evaluation uses task-specific benchmarks (MMLU, HumanEval, etc.) alongside perplexity.

2. Tokenizer Dependent

Perplexity is computed per-token, so the choice of tokenizer fundamentally affects the score. Consider encoding "unhappiness":

Tokenizer A (3 tokens):

["un", "happi", "ness"]

PPL measured over 3 predictions

Tokenizer B (1 token):

["unhappiness"]

PPL measured over 1 prediction

A model using Tokenizer B appears to have lower perplexity because it makes fewer (but harder) predictions. Neither score is "wrong," but they cannot be directly compared.

Solution: Use bits-per-character (BPC) or bits-per-byte for tokenizer-agnostic comparison.

3. Domain Sensitivity

Language models learn the statistical patterns of their training data. When evaluated on a different domain, perplexity can change dramatically. The numbers below are illustrative, not measured:

Model trained on	News PPL	Code PPL	Medical PPL
News articles	~25	~150	~80
GitHub code	~90	~15	~200

A "good" perplexity on one domain means nothing for another. Rare terminology, different syntax patterns, and specialized jargon all increase perplexity.

Always evaluate on data representative of your target use case.

4. Memorization vs Understanding

A model that has seen the test set during training can achieve artificially low perplexity by memorizing rather than learning generalizable patterns:

PPL = 1.0on test set

Model assigns 100% probability to each correct token. Perfect score, but is it understanding or memorization?

Signs of memorization over understanding:

Large gap between train and test perplexity (overfitting)
Model reproduces training examples verbatim when prompted
Poor performance on paraphrased or novel formulations
Fails on out-of-distribution inputs despite low benchmark PPL

Mitigations: Use held-out test sets, check for data contamination, evaluate on diverse benchmarks.

ML Applications

LLM Evaluation

GPT, LLaMA, and other models are benchmarked on WikiText-103, PTB, and other datasets using perplexity. Lower is always better.

Machine Translation

Decoder perplexity measures fluency of generated translations. Often combined with BLEU for quality assessment.

t-SNE Hyperparameter

t-SNE uses "perplexity" to define effective neighborhood size. Similar concept: how many neighbors to consider.

Speech Recognition

Language model perplexity affects ASR accuracy. Lower LM perplexity typically improves word error rate (WER).

Practice this

On TensorTonic you implement this from scratch and test it in the browser. These problems go from the basic formula to the version used to evaluate fine-tuned LLMs.

Perplexity Computation: compute perplexity from token probability distributions and the observed tokens.
Perplexity from Logits: compute sequence perplexity from next-token logits with stable log probabilities.
Compute Conditional Perplexity: score only the response tokens selected by a mask, as in instruction tuning.
N-gram Language Model: build a smoothed n-gram model and score test sequences with it.
Implement Cross-Entropy Loss: the training loss whose exponential is perplexity.

For more, browse the full problem catalogue.

Contents

What is perplexity?

Why Do We Need Perplexity?

The Core Idea

Simple Analogy: The Guessing Game

Perplexity Scale Reference

Intuition: Effective Choices

Low Perplexity (Good)

High Perplexity (Confused)

Perplexity as "Effective Choices"

Actual Probabilities

Perplexity Equivalent

The formula and its link to cross-entropy

Bits vs Nats

Why Exponentiate?

Entropy

Cross-Entropy

Perplexity

Key Relationship

Interactive: Word Prediction

Next Word Prediction

Worked calculation

Step 1: Get Model Probabilities

Step 2: Compute Log Probabilities

Step 3: Average Negative Log-Likelihood

Step 4: Exponentiate

Benchmark scores in context

Comparing scores fairly

Bits Per Character (BPC)

BPC to PPL

Typical Values

Limitations & Caveats

1. Not Task-Specific

2. Tokenizer Dependent

3. Domain Sensitivity

4. Memorization vs Understanding

ML Applications

LLM Evaluation

Machine Translation

t-SNE Hyperparameter

Speech Recognition

Practice this