Name: TensorTonic
Author: TensorTonic

Last updated 13 October 2026

What is information gain?

Information gain is how much a split reduces entropy in a decision tree. It equals the entropy of the parent node minus the size-weighted average entropy of the child nodes. A decision tree computes it for every candidate feature and splits on the one with the highest gain, since that split makes the children purest.

Building a Decision Tree involves asking a sequence of questions to split your data. But which question should you ask first?

Should you ask "Is it raining?" or "Is it humid?" first? To answer this, we need a way to measure how much "cleaner" (purer) the data becomes after a split.

Information Gain (IG) is exactly this measure. It quantifies the reduction in entropy (uncertainty) achieved by splitting on a specific feature. It is essentially Mutual Information between the feature and the label.

Intuition: The 20 Questions Game

Imagine playing "20 Questions." Your goal is to identify a secret object with yes/no questions.

Bad Question

"Is the object a banana?"

99% chance the answer is "No." You are left with almost the same uncertainty. Low Information Gain.

Good Question

"Is the object alive?"

Splits the world roughly in half. Regardless of the answer, you have eliminated 50% of possibilities. High Information Gain.

Decision Trees use Information Gain to find the "Is it alive?" questions, not the "Is it a banana?" questions.

Recap: Entropy as Impurity

Before we split, we calculate the entropy of the current dataset (the parent node). Entropy measures "impurity" or "disorder."

H(S) = -\sum_{i=1}^c p_i \log_2(p_i)

Pure (H = 0): All samples same class.

Impure (H = 1): 50/50 split.

The formula

Information Gain = Entropy Before - Entropy After.

Since a split creates multiple child nodes, "Entropy After" is the weighted average of child entropies.

IG(S, A) = H(S) - \sum_{v \in Values(A)} \frac{|S_v|}{|S|} H(S_v)

H(S)

Entropy of parent node.

Weighted Sum

Large children matter more than small ones.

H(S_v)

Entropy of child node v.

Two facts follow from the formula. Information gain is never negative, because the weighted average entropy of the children can never exceed the parent's. It reaches its maximum, $H(S)$ , when every child is pure. Information gain is also the mutual information between the feature and the label, measured on the samples at that node.

Interactive: Split Visualization

Drag the split point to see how Information Gain changes. The best split creates the purest child nodes.

Recursive Partitioning

Build a Depth-2 Decision Tree. First split the root, then split the children.

Root SplitIG: 0.001

Left Split

Right Split

Class 1

Class 0

Total Information Gain: 0.001 bits

(Sum of IG at each active node)

Worked calculation

Dataset: 10 examples, 5 Positive (+), 5 Negative (-). Testing feature "Windy" (True/False).

Step 1: Parent Entropy

H(Parent) = -0.5 \log_2(0.5) - 0.5 \log_2(0.5) = 1.0 \text{ bit}

Step 2: Split Data

Windy=False (6 samples): 4 Pos, 2 Neg
Windy=True (4 samples): 1 Pos, 3 Neg

Step 3: Child Entropies

Left:

H = -\frac{4}{6}\log_2\frac{4}{6} - \frac{2}{6}\log_2\frac{2}{6} \approx 0.918

Right:

H = -\frac{1}{4}\log_2\frac{1}{4} - \frac{3}{4}\log_2\frac{3}{4} \approx 0.811

Step 4: Information Gain

IG = 1.0 - [\frac{6}{10} \times 0.918 + \frac{4}{10} \times 0.811]

IG = 1.0 - 0.875 = 0.125 \text{ bits}

Knowing whether it is windy removes only 0.125 of the 1 bit of uncertainty, so “Windy” is a weak split. The code below repeats the calculation and compares it with a three-valued “Forecast” feature on the same 10 days, using both entropy and Gini impurity:

Python

import numpy as np

def entropy(y):
    _, counts = np.unique(y, return_counts=True)
    p = counts / counts.sum()
    return -np.sum(p * np.log2(p))

def gini(y):
    _, counts = np.unique(y, return_counts=True)
    p = counts / counts.sum()
    return 1 - np.sum(p ** 2)

def gain(y, feature, impurity):
    """Parent impurity minus the size-weighted impurity of each child."""
    children = sum((feature == v).mean() * impurity(y[feature == v])
                   for v in np.unique(feature))
    return impurity(y) - children

# 10 days: label 1 = play, 0 = don't play
y        = np.array([1, 1, 1, 1, 0, 0,   1, 0, 0, 0])
windy    = np.array([0, 0, 0, 0, 0, 0,   1, 1, 1, 1])
forecast = np.array([0, 0, 0, 0, 1, 1,   0, 1, 1, 2])  # 0 sunny, 1 rain, 2 cloudy

for name, x in [("windy", windy), ("forecast", forecast)]:
    print(f"{name:<9} info gain = {gain(y, x, entropy):.3f} bits   "
          f"gini gain = {gain(y, x, gini):.3f}")

Output

windy     info gain = 0.125 bits   gini gain = 0.083
forecast  info gain = 1.000 bits   gini gain = 0.500

Forecast separates the classes perfectly, so its gain equals the full parent entropy of 1 bit, and both criteria rank it above Windy. Keep the three-way split in mind for the limitations section below.

The ID3 Algorithm

ID3 (Iterative Dichotomiser 3) uses Information Gain to build trees greedily:

Calculate H(S) for the current node.
For every possible feature A, calculate IG(S, A).
Select the feature with the highest Information Gain.
Split the data using that feature.
Recurse on each child until nodes are pure or max depth is reached.

ID3 only handles categorical features. Its successors C4.5 and CART split a numeric feature such as temperature with a threshold: sort the values, try a cut between each pair of neighbouring values, and keep the threshold with the highest gain. The interactive split above does exactly this for one feature.

Gini vs entropy: which should you use?

Scikit-Learn's CART algorithm uses Gini Impurity by default instead of Entropy. Why?

Entropy

-\sum p \log p

Pro: Information-theoretically grounded.
Con: Slower (log computation).

Gini Impurity

1 - \sum p^2

Pro: Faster (only square).
Con: No information-theory interpretation.

	Entropy	Gini impurity
Maximum (2 classes)	1 bit at a 50/50 split	0.5 at a 50/50 split
Windy split above	Gain 0.125	Gain 0.083
Computation	Needs a logarithm per class	Only squares
Used by	ID3, C4.5; `criterion="entropy"` in scikit-learn	CART; scikit-learn's default

The two scales differ, so compare gains within one criterion, never across them. The rankings almost always agree: Raileanu and Stoffel (Annals of Mathematics and Artificial Intelligence, 2004) found the two criteria choose different splits in only about 2% of cases. Gini is the common default because it skips the logarithm, and entropy is a reasonable choice when you want gains in bits that connect to the rest of information theory. Tree depth, minimum leaf size and pruning affect accuracy far more than this choice.

Gini Impurity vs Entropy

Compare the shape of Gini Impurity and Entropy. Notice how Gini is just a quadratic approximation of Entropy.

Probability p (Class 1): 0.50

Entropy H(p)

1.000

Max at p=0.5 (1.0 bits)

Gini Impurity G(p)

0.500

Max at p=0.5 (0.5)

--- Dashed Green is Entropy scaled by 0.5. It almost perfectly overlaps Gini!

Limitations & Gain Ratio

The "User ID" Problem

Information Gain is biased towards features with many unique values.

If you split on "User ID" (unique for everyone), each child has 1 sample with H = 0. Maximum IG! But this is useless for generalization.

Solution: Gain Ratio (C4.5)

GainRatio(S, A) = \frac{IG(S, A)}{SplitInfo(A)}

Divides by the "intrinsic information" of the split itself, penalizing splits with many branches.

SplitInfo(A) = -\sum_{v} \frac{|S_v|}{|S|} \log_2 \frac{|S_v|}{|S|}

For the Forecast feature, the branches hold 5, 4 and 1 of the 10 days, so its split information is about 1.36 bits and its gain ratio is about 0.73. A User ID with one branch per row would have split information $\log_2 n$ , which pulls its gain ratio down as n grows.

Practice this

On TensorTonic you implement this from scratch and test it in the browser. These problems build up from one node's entropy to a full tree and a forest.

Compute Entropy for a Node: the parent and child entropies used in every gain calculation.
Compute Information Gain for a Split: the formula from this lesson, on a single split.
Compute Gini Impurity for a Split: the weighted Gini version, for comparison.
Decision Tree Best Split: search thresholds across features and keep the largest impurity reduction.
Decision Tree Classifier (CART): grow a full tree with Gini splits and depth limits.
Random Forest from Scratch: combine many trees with bootstrap samples and random feature subsets.

Contents