Implement KL Divergence (Kullback-Leibler divergence) to measure how one probability distribution differs from another.
KL Divergence Formula:
DKL(P∥Q)=i∑PilogQiPiwhere P and Q are probability distributions
p: array-like - First probability distribution, shape (N,)q: array-like - Second probability distribution, shape (N,)eps: float = 1e-12 - Numerical stability epsilonInput: p=[0.4, 0.6], q=[0.5, 0.5], eps=1e-12
Output: 0.0201
Small difference between similar distributions
Input: p=[0.3, 0.7], q=[0.3, 0.7], eps=1e-12
Output: 0.0
Identical distributions have KL divergence of 0
Input: p=[0.9, 0.1], q=[0.5, 0.5], eps=1e-12
Output: 0.368
Concentrated vs uniform distribution has higher divergence
Add eps to q for stability: q_stable = q + eps before computing log ratios.
Only compute terms where p[i] > 0, since 0 * log(0/q) = 0 by convention.
Use np.log() for element-wise logarithm and np.sum() for the final result.
Sign in to take notes on this problem
Accepts: array
Accepts: array
Accepts: number
Implement KL Divergence (Kullback-Leibler divergence) to measure how one probability distribution differs from another.
KL Divergence Formula:
DKL(P∥Q)=i∑PilogQiPiwhere P and Q are probability distributions
p: array-like - First probability distribution, shape (N,)q: array-like - Second probability distribution, shape (N,)eps: float = 1e-12 - Numerical stability epsilonInput: p=[0.4, 0.6], q=[0.5, 0.5], eps=1e-12
Output: 0.0201
Small difference between similar distributions
Input: p=[0.3, 0.7], q=[0.3, 0.7], eps=1e-12
Output: 0.0
Identical distributions have KL divergence of 0
Input: p=[0.9, 0.1], q=[0.5, 0.5], eps=1e-12
Output: 0.368
Concentrated vs uniform distribution has higher divergence
Add eps to q for stability: q_stable = q + eps before computing log ratios.
Only compute terms where p[i] > 0, since 0 * log(0/q) = 0 by convention.
Use np.log() for element-wise logarithm and np.sum() for the final result.
Sign in to take notes on this problem
Accepts: array
Accepts: array
Accepts: number