Name: TensorTonic
Author: TensorTonic

Last updated 13 October 2026

What is KL divergence?

KL divergence (Kullback-Leibler divergence) measures how much a probability distribution P differs from a reference distribution Q. It is the expected number of extra bits, or nats, needed to encode samples from P with a code built for Q. It is never negative, is zero only when P equals Q, and is not symmetric.

In geometry, we measure distance between points with Euclidean distance. But how do we measure the "distance" between two probability distributions?

For example, how different is a Gaussian $N(0, 1)$ from $N(2, 2)$ ? Or the word distribution in Shakespeare versus a Reddit thread?

KL Divergence (Kullback-Leibler Divergence, also called Relative Entropy) quantifies this difference. It measures how much information is lost when we approximate one distribution with another.

Not a True Metric

While KL Divergence is often called a "distance," it doesn't satisfy all the properties of a mathematical distance metric (it's asymmetric and doesn't obey the triangle inequality). Think of it as a directed measure of divergence.

Intuition: Extra Bits

Imagine you're designing a compression code for English text. Optimal codes assign short representations to frequent letters ('e', 't') and long ones to rare letters ('z', 'q').

P (True Distribution): The actual frequency of letters in English.
Q (Model/Approximation): Your guess at the frequency (maybe based on French).

If you build your code based on Q but data comes from P, your code will be inefficient. You'll use long codes for letters that are actually common.

KL Divergence = Extra Bits

$D_{KL}(P || Q)$ measures the extra bits you need to transmit because you used the wrong distribution Q instead of the true P. It quantifies the cost of your approximation.

The Formula

For discrete probability distributions P and Q:

D_{KL}(P || Q) = \sum_{x} P(x) \log \left( \frac{P(x)}{Q(x)} \right)

= \sum_{x} P(x) [\log P(x) - \log Q(x)]

Critical Warning

If $P(x) > 0$ but $Q(x) = 0$ , KL divergence goes to infinity. The model Q thinks an event is impossible when it actually occurs. This is a fatal modeling error called "zero-avoiding" behavior.

For Continuous Distributions

D_{KL}(P || Q) = \int P(x) \log \left( \frac{P(x)}{Q(x)} \right) dx

For Gaussians, there's a closed-form solution, making KL Divergence computationally efficient.

Interactive: KL Between Gaussians

Explore how KL divergence changes as you move the model distribution Q away from the true distribution P.

KL Divergence

Distance between q(z|x) and prior N(0,1)

μ (mean shift)0.0

log σ² (variance)0.0 (σ = 1.00)

KL = 0.000

✓ Close to prior

KL = -½ × (1 + log σ² - μ² - σ²) = -½ × (1 + 0.0 - 0.00 - 1.00)

Asymmetry: Not a True Distance

In geometry, distance is symmetric: Distance(A to B) = Distance(B to A). KL Divergence is NOT symmetric.

D_{KL}(P || Q) \neq D_{KL}(Q || P)

This asymmetry is not a bug. It reflects a real difference: "How much does Q fail to capture P?" is a different question from "How much does P fail to capture Q?"

Worked Example

Let $P = [0.5, 0.5]$ (fair coin) and $Q = [0.9, 0.1]$ (biased coin).

Forward KL: $D_{KL}(P || Q)$

= 0.5 \cdot \ln\left(\frac{0.5}{0.9}\right) + 0.5 \cdot \ln\left(\frac{0.5}{0.1}\right)

= 0.5 \cdot (-0.588) + 0.5 \cdot (1.609)

= 0.511 \text{ nats}

Reverse KL: $D_{KL}(Q || P)$

= 0.9 \cdot \ln\left(\frac{0.9}{0.5}\right) + 0.1 \cdot \ln\left(\frac{0.1}{0.5}\right)

= 0.9 \cdot (0.588) + 0.1 \cdot (-1.609)

= 0.368 \text{ nats}

$0.511 \neq 0.368$ - The direction matters!

What are "nats"?

Nats (natural units) measure information using $\ln$ (natural log, base $e$ ). Bits use $\log_2$ . To convert: $1 \text{ nat} = \frac{1}{\ln(2)} \approx 1.44 \text{ bits}$ . Deep learning frameworks typically use nats because $\ln$ has simpler derivatives.

Forward vs Reverse KL

The choice between $D_{KL}(P || Q)$ (Forward) and $D_{KL}(Q || P)$ (Reverse) has major implications for how your model behaves.

Forward KL: D_KL(P || Q)

Mean-Seeking / Moment-Matching

Q must cover all the mass of P. If P is multimodal, Q will spread out to cover all modes, potentially with density in between.

Used in: VAEs, Maximum Likelihood, Expectation Maximization

Reverse KL: D_KL(Q || P)

Mode-Seeking / Zero-Avoiding

Q avoids placing mass where P is zero. If P is multimodal, Q will collapse to cover only ONE mode, ignoring the rest.

Used in: Variational Inference, Some GANs, Policy Gradients

Multimodal Example

If P has two peaks (bimodal), Forward KL makes Q spread across both peaks (blurry, inclusive). Reverse KL makes Q collapse to just one peak (sharp but missing mass).

Interactive: Forward vs Reverse

See how the direction of KL Divergence dramatically changes model behavior for multimodal distributions.

Forward vs Reverse KL

Two ways to fit a distribution

Mean-Seeking

Zero-Avoiding (Mass Covering)

Forward KL penalizes Q if P(x) > 0 but Q(x) ≈ 0. To avoid infinite penalty, Q stretches to cover ALL of P's support, often becoming blurry.

D_{KL}(P || Q) = \mathbb{E}_{x \sim P} [\log \frac{P(x)}{Q(x)}]

Typical Use Cases

Variational Autoencoders (VAEs)

Maximum Likelihood Estimation

P is complex (bimodal). Q is simple (unimodal).

KL vs Jensen-Shannon divergence

Jensen-Shannon divergence fixes the two properties that make KL awkward as a distance. It compares both distributions to their average $M = \tfrac{1}{2}(P + Q)$ and averages the two KL terms:

JS(P, Q) = \tfrac{1}{2} D_{KL}(P || M) + \tfrac{1}{2} D_{KL}(Q || M)

	KL divergence	Jensen-Shannon divergence
Symmetric	No	Yes
Range	$[0, \infty)$	$[0, \ln 2]$ in nats
When Q(x) = 0 but P(x) > 0	Infinite	Still finite
Typical use	VAE loss, distillation, RLHF penalties	Original GAN objective, comparing distributions

For the coin example above, KL gives 0.511 nats one way and 0.368 the other, while Jensen-Shannon gives 0.102 nats in both directions. Use KL when the direction carries meaning, such as a model approximating data, and Jensen-Shannon when you want a symmetric, bounded comparison.

Relation to Entropy

KL Divergence connects beautifully to Entropy and Cross-Entropy:

D_{KL}(P || Q) = H(P, Q) - H(P)

KL Divergence = Cross-Entropy - Entropy

Since H(P) is constant for fixed data, minimizing Cross-Entropy is equivalent to minimizing KL Divergence. This is why Cross-Entropy loss makes models learn to match the data distribution.

Why This Connection Matters

In classification, we don't actually need to compute H(P) because it's constant. We just minimize H(P,Q), which implicitly minimizes the divergence between our model and the true distribution!

Interactive: Entropy Relationship

Explore the decomposition: Cross-Entropy = Entropy + KL Divergence.

The Information Theory Identity

Visualizing $H(P, Q) = H(P) + D_{KL}(P || Q)$

True Distribution P0.70 / 0.30

Model Distribution Q0.50 / 0.50

Key Insight: Since H(P) is fixed by the data, minimizing Cross-Entropy is mathematically identical to minimizing KL Divergence.

This is why training a neural net with Cross-Entropy loss makes it learn the true distribution!

Total Cost (Bits)1.00

Cross-Entropy H(P,Q)

Min Cost0.88

Entropy H(P)

Inefficiency0.12

KL Divergence

Wasted Bits: 0.12

The model is inefficient. Cross-Entropy is higher than the theoretical minimum (Entropy).

Key Properties of KL Divergence

1. Non-Negativity (Gibbs' Inequality)

D_{KL}(P || Q) \geq 0

KL Divergence is always non-negative, and equals zero if and only if $P = Q$ everywhere.

Why? From Jensen's Inequality:

-D_{KL}(P || Q) = \sum_x P(x) \ln\left(\frac{Q(x)}{P(x)}\right)

\leq \ln\left(\sum_x P(x) \cdot \frac{Q(x)}{P(x)}\right) = \ln(1) = 0

Since $-D_{KL} \leq 0$ , we have $D_{KL} \geq 0$ .

ML implication: You can never have "negative divergence." If your loss is negative, there's a bug in your code.

2. Asymmetry (Not a Metric)

D_{KL}(P || Q) \neq D_{KL}(Q || P)

Unlike Euclidean distance, KL Divergence is not symmetric. The "distance" from P to Q differs from Q to P.

Forward KL

$D_{KL}(P || Q)$

"How well does Q explain P?"

Reverse KL

$D_{KL}(Q || P)$

"How well does P explain Q?"

Why it fails as a metric: A true metric requires symmetry ( $d(a,b) = d(b,a)$ ) and triangle inequality. KL satisfies neither.

3. Convexity

$D_{KL}(P || Q)$ is convex in the pair $(P, Q)$ . For any $\lambda \in [0, 1]$ :

D_{KL}(\lambda P_1 + (1-\lambda)P_2 \,||\, \lambda Q_1 + (1-\lambda)Q_2) \leq \lambda D_{KL}(P_1 || Q_1) + (1-\lambda) D_{KL}(P_2 || Q_2)

Mixing distributions reduces or maintains the divergence. This property ensures:

Gradient descent converges to global minimum (for fixed P)
No local minima traps in variational inference
Optimization is computationally tractable

4. Additive for Independent Variables

If $X$ and $Y$ are independent under both P and Q:

D_{KL}(P_{XY} || Q_{XY}) = D_{KL}(P_X || Q_X) + D_{KL}(P_Y || Q_Y)

Derivation:

D_{KL}(P_{XY} || Q_{XY}) = \sum_{x,y} P(x,y) \ln\frac{P(x,y)}{Q(x,y)}

Since independent:

P(x,y) = P(x)P(y)

and

Q(x,y) = Q(x)Q(y)

= \sum_{x,y} P(x)P(y) \ln\frac{P(x)P(y)}{Q(x)Q(y)}

= \sum_{x,y} P(x)P(y) \left[\ln\frac{P(x)}{Q(x)} + \ln\frac{P(y)}{Q(y)}\right]

= D_{KL}(P_X || Q_X) + D_{KL}(P_Y || Q_Y)

ML implication: For high-dimensional data with independent features, you can compute KL divergence feature-by-feature and sum them up.

5. Invariance Under Reparameterization

For any invertible transformation $y = f(x)$ :

D_{KL}(P_X || Q_X) = D_{KL}(P_Y || Q_Y)

KL Divergence only depends on the probability values, not how the space is parameterized. Scaling, rotating, or nonlinearly transforming your features does not change the divergence.

How do you compute KL divergence in Python?

The discrete formula is a one-line sum once two details are handled: terms where $P(x) = 0$ contribute nothing, and a small epsilon stops $\log 0$ when Q assigns zero probability. The listing also includes Jensen-Shannon and the closed form for two Gaussians.

Python

import numpy as np

def kl_divergence(p, q, eps=1e-12):
    """D_KL(P || Q) in nats for two discrete distributions."""
    p = np.asarray(p, dtype=float)
    q = np.asarray(q, dtype=float)
    p, q = p / p.sum(), q / q.sum()
    mask = p > 0                      # terms with P(x) = 0 contribute 0
    return float(np.sum(p[mask] * np.log(p[mask] / np.maximum(q[mask], eps))))

def js_divergence(p, q):
    """Jensen-Shannon divergence: symmetric and bounded by ln 2."""
    p, q = np.asarray(p, dtype=float), np.asarray(q, dtype=float)
    m = 0.5 * (p + q)
    return 0.5 * kl_divergence(p, m) + 0.5 * kl_divergence(q, m)

def kl_gaussians(mu1, sigma1, mu2, sigma2):
    """Closed form D_KL(N(mu1, sigma1^2) || N(mu2, sigma2^2))."""
    return float(np.log(sigma2 / sigma1)
                 + (sigma1**2 + (mu1 - mu2)**2) / (2 * sigma2**2) - 0.5)

P = [0.5, 0.5]   # fair coin
Q = [0.9, 0.1]   # biased coin

print(f"KL(P || Q) = {kl_divergence(P, Q):.3f} nats")
print(f"KL(Q || P) = {kl_divergence(Q, P):.3f} nats")
print(f"JS(P, Q)   = {js_divergence(P, Q):.3f} nats")
print(f"JS(Q, P)   = {js_divergence(Q, P):.3f} nats")
print(f"KL(N(0,1) || N(2,2)) = {kl_gaussians(0, 1, 2, 2):.3f} nats")

Output

KL(P || Q) = 0.511 nats
KL(Q || P) = 0.368 nats
JS(P, Q)   = 0.102 nats
JS(Q, P)   = 0.102 nats
KL(N(0,1) || N(2,2)) = 0.818 nats

The first two lines reproduce the worked example, and the last line is the value for $N(0, 1)$ against $N(2, 2)$ from the introduction. scipy.stats.entropy(p, q) returns the same KL value. In PyTorch, note that torch.nn.functional.kl_div expects log-probabilities for its first argument.

Where does KL divergence appear in ML?

Variational Autoencoders (VAEs)

The VAE loss has two parts: Reconstruction Loss + KL Divergence. The KL term forces the learned latent distribution q(z|x) to be close to a standard Normal prior N(0, I).

\mathcal{L} = -E_q[\log p(x|z)] + D_{KL}(q(z|x) || p(z))

This regularization ensures smooth, interpretable latent spaces for generation.

t-SNE Visualization

t-SNE minimizes the KL Divergence between the distribution of pairwise distances in high-dimensional space and low-dimensional space. This preserves local structure while reducing dimensions.

Critical for visualizing embeddings, gene expression data, and neural network activations.

Knowledge Distillation

Training a small "student" model to mimic a large "teacher" model. The loss is the KL Divergence between the teacher's softmax outputs (with temperature) and the student's outputs.

Used to compress BERT → DistilBERT, reducing size by 40% with 97% performance.

Reinforcement Learning (TRPO/PPO)

Trust Region Policy Optimization constrains policy updates so the new policy doesn't diverge too much from the old policy, measured by KL Divergence. PPO uses a clipped surrogate objective.

Powers OpenAI's robotic control, Dota 2 agents, and ChatGPT's RLHF training.

Generative Adversarial Networks (GANs)

Some GAN formulations (f-GAN) use KL or reverse KL divergence. The choice affects whether the generator covers all modes (forward KL) or focuses on high-quality single modes (reverse KL).

Explains mode collapse: reverse KL encourages sharp, single-mode outputs.

Bayesian Model Selection

KL Divergence measures how well an approximate posterior matches the true posterior in variational Bayes. Also used in Akaike Information Criterion (AIC) for model comparison.

Theoretical foundation for choosing between competing statistical models.

Practice this

On TensorTonic you implement this from scratch and test it in the browser. These problems take KL divergence from the formula to the places it shows up in real models.

Implement KL Divergence: compute KL between discrete distributions with safe handling of zero probabilities.
KL Divergence: Discrete and Continuous: work through both forms, including the closed form between Gaussians.
Implement Cross-Entropy Loss: the loss that minimizes KL to the data distribution in classification.
Variational Autoencoder Forward Pass: build the VAE whose loss includes the KL term to the prior.
PPO with Reference-Policy KL: add the KL penalty that keeps an RLHF policy close to its reference model.

For more, browse the full problem catalogue.

Contents

What is KL divergence?

Not a True Metric

Intuition: Extra Bits

KL Divergence = Extra Bits

The Formula

Critical Warning

For Continuous Distributions

Interactive: KL Between Gaussians

KL Divergence

Asymmetry: Not a True Distance

Worked Example

What are "nats"?

Forward vs Reverse KL

Forward KL: DKL(P || Q)

Reverse KL: DKL(Q || P)

Multimodal Example

Interactive: Forward vs Reverse

Forward vs Reverse KL

Zero-Avoiding (Mass Covering)

KL vs Jensen-Shannon divergence

Relation to Entropy

Why This Connection Matters

Interactive: Entropy Relationship

The Information Theory Identity

Key Properties of KL Divergence

1. Non-Negativity (Gibbs' Inequality)

2. Asymmetry (Not a Metric)

3. Convexity

4. Additive for Independent Variables

5. Invariance Under Reparameterization

How do you compute KL divergence in Python?

Where does KL divergence appear in ML?

Variational Autoencoders (VAEs)

t-SNE Visualization

Knowledge Distillation

Reinforcement Learning (TRPO/PPO)

Generative Adversarial Networks (GANs)

Bayesian Model Selection

Practice this

Forward KL: D_KL(P || Q)

Reverse KL: D_KL(Q || P)