He (Kaiming) initialization is designed for networks using ReLU activations. Since ReLU zeros out negative values (roughly half the distribution), He initialization uses a larger variance than Xavier to compensate, depending only on the fan-in.
Given a raw weight matrix W with values in [0, 1] and fan_in (number of input units), scale the weights to He uniform initialization.
Input:
W = [[0.5, 0.5]], fan_in = 2
Output:
[[0.0, 0.0]]
With fan_in = 2, limit = sqrt(6/2) = sqrt(3) = 1.7321. Raw value 0.5 maps to 0.5 * 2 * 1.7321 - 1.7321 = 0.
Input:
W = [[0], [1]], fan_in = 2
Output:
[[-1.7321], [1.7321]]
Raw 0 maps to -limit and raw 1 maps to +limit. The wider range compared to Xavier compensates for ReLU killing half the activations.
Compute limit = sqrt(6 / fan_in). Then for each element: scaled = raw * 2 * limit - limit. Note that fan_out is not needed since He init only depends on fan_in.
Compare with Xavier: Xavier uses sqrt(6 / (fan_in + fan_out)) while He uses sqrt(6 / fan_in). He's limit is always larger, giving a wider initialization range for ReLU networks.
Sign in to take notes on this problem
Accepts: array
Accepts: number
He (Kaiming) initialization is designed for networks using ReLU activations. Since ReLU zeros out negative values (roughly half the distribution), He initialization uses a larger variance than Xavier to compensate, depending only on the fan-in.
Given a raw weight matrix W with values in [0, 1] and fan_in (number of input units), scale the weights to He uniform initialization.
Input:
W = [[0.5, 0.5]], fan_in = 2
Output:
[[0.0, 0.0]]
With fan_in = 2, limit = sqrt(6/2) = sqrt(3) = 1.7321. Raw value 0.5 maps to 0.5 * 2 * 1.7321 - 1.7321 = 0.
Input:
W = [[0], [1]], fan_in = 2
Output:
[[-1.7321], [1.7321]]
Raw 0 maps to -limit and raw 1 maps to +limit. The wider range compared to Xavier compensates for ReLU killing half the activations.
Compute limit = sqrt(6 / fan_in). Then for each element: scaled = raw * 2 * limit - limit. Note that fan_out is not needed since He init only depends on fan_in.
Compare with Xavier: Xavier uses sqrt(6 / (fan_in + fan_out)) while He uses sqrt(6 / fan_in). He's limit is always larger, giving a wider initialization range for ReLU networks.
Sign in to take notes on this problem
Accepts: array
Accepts: number