Compare a tanh neuron's analytic weight and bias gradients with forward finite differences.
y=tanh(i=1∑nwixi+b) ∂wi∂y=(1−y2)xi ∂b∂y=1−y2 ginum=hy(wi+h)−y(wi) gbnum=hy(b+h)−y(b)Here n is the vector length, x is inputs, w is weights, i indexes their entries, b is bias, y is the output, and h is the positive perturbation step. Each numerical gradient changes only the indicated parameter; every other value remains fixed.
Return a tuple of five PyTorch float64 tensors on the input device: analytic weight gradients, numerical weight gradients, analytic bias gradient, numerical bias gradient, and maximum absolute disagreement across all weight and bias gradients. The first two are vectors matching weights; the last three are scalars. With no weights, the maximum disagreement is the bias error.
Input: inputs = [1, -2], weights = [0.5, -0.25], bias = 0.1, h = 1e-6
Output: ([0.359201, -0.718403], [0.359201, -0.718404], 0.359201, 0.359201, 0.000001)
Explanation: Perturbing each parameter separately gives gradients close to the analytic values.
Input: inputs = [3, -4], weights = [2, -2], bias = 4, h = 1e-6
Output: ([0.0, 0.0], [0.0, 0.0], 0.0, 0.0, 0.0)
Input: inputs = [], weights = [], bias = -0.5, h = 1e-6
Output: ([], [], 0.786448, 0.786448, 0.0)
Tensor.to can create float64 working tensors.
Tensor.clone keeps separate perturbations independent.
torch.abs measures disagreement; an empty weight vector leaves only the bias error.
Accepts: array
Accepts: array
Accepts: number
Accepts: number
Compare a tanh neuron's analytic weight and bias gradients with forward finite differences.
y=tanh(i=1∑nwixi+b) ∂wi∂y=(1−y2)xi ∂b∂y=1−y2 ginum=hy(wi+h)−y(wi) gbnum=hy(b+h)−y(b)Here n is the vector length, x is inputs, w is weights, i indexes their entries, b is bias, y is the output, and h is the positive perturbation step. Each numerical gradient changes only the indicated parameter; every other value remains fixed.
Return a tuple of five PyTorch float64 tensors on the input device: analytic weight gradients, numerical weight gradients, analytic bias gradient, numerical bias gradient, and maximum absolute disagreement across all weight and bias gradients. The first two are vectors matching weights; the last three are scalars. With no weights, the maximum disagreement is the bias error.
Input: inputs = [1, -2], weights = [0.5, -0.25], bias = 0.1, h = 1e-6
Output: ([0.359201, -0.718403], [0.359201, -0.718404], 0.359201, 0.359201, 0.000001)
Explanation: Perturbing each parameter separately gives gradients close to the analytic values.
Input: inputs = [3, -4], weights = [2, -2], bias = 4, h = 1e-6
Output: ([0.0, 0.0], [0.0, 0.0], 0.0, 0.0, 0.0)
Input: inputs = [], weights = [], bias = -0.5, h = 1e-6
Output: ([], [], 0.786448, 0.786448, 0.0)
Tensor.to can create float64 working tensors.
Tensor.clone keeps separate perturbations independent.
torch.abs measures disagreement; an empty weight vector leaves only the bias error.
Accepts: array
Accepts: array
Accepts: number
Accepts: number