Evaluate a tanh neuron and manually compute the loss gradients for its inputs, weights, and bias from the supplied upstream_gradient.
a=i=1∑nwixi+b y=tanh(a) δ=g(1−y2) ∂xi∂L=δwi ∂wi∂L=δxi ∂b∂L=δHere n is the vector length, x is inputs, w is weights, i indexes their entries, b is bias, a is the preactivation, and y is the neuron output. L is the loss, g is its supplied gradient with respect to y, and delta is its gradient with respect to a.
Return a tuple of four PyTorch tensors: the scalar output, input-gradient vector, weight-gradient vector, and scalar bias gradient. Vectors have the same shape as inputs; all four tensors preserve the common input dtype and device. Empty vectors still produce an output and bias gradient.
Input: inputs = [2, 0], weights = [-3, 1], bias = 6.881373587, upstream_gradient = 1
Output: (0.707107, [-1.5, 0.5], [1.0, 0.0], 0.5)
Explanation: The preactivation gradient is approximately 0.5, so the zero input gives a zero weight gradient.
Input: inputs = [-1, 2, 0.5], weights = [0.25, -0.75, 1.5], bias = -0.2, upstream_gradient = -2
Output: (-0.833655, [-0.15251, 0.45753, -0.91506], [0.61004, -1.22008, -0.30502], -0.61004)
Input: inputs = [], weights = [], bias = 0, upstream_gradient = 3
Output: (0.0, [], [], 3.0)
torch.sum reduces an empty vector to a scalar zero.
Tensor.square can reuse the saved tanh output for its local derivative.
A scalar tensor broadcasts across a vector without changing its shape.
Accepts: array
Accepts: array
Accepts: number
Accepts: number
Evaluate a tanh neuron and manually compute the loss gradients for its inputs, weights, and bias from the supplied upstream_gradient.
a=i=1∑nwixi+b y=tanh(a) δ=g(1−y2) ∂xi∂L=δwi ∂wi∂L=δxi ∂b∂L=δHere n is the vector length, x is inputs, w is weights, i indexes their entries, b is bias, a is the preactivation, and y is the neuron output. L is the loss, g is its supplied gradient with respect to y, and delta is its gradient with respect to a.
Return a tuple of four PyTorch tensors: the scalar output, input-gradient vector, weight-gradient vector, and scalar bias gradient. Vectors have the same shape as inputs; all four tensors preserve the common input dtype and device. Empty vectors still produce an output and bias gradient.
Input: inputs = [2, 0], weights = [-3, 1], bias = 6.881373587, upstream_gradient = 1
Output: (0.707107, [-1.5, 0.5], [1.0, 0.0], 0.5)
Explanation: The preactivation gradient is approximately 0.5, so the zero input gives a zero weight gradient.
Input: inputs = [-1, 2, 0.5], weights = [0.25, -0.75, 1.5], bias = -0.2, upstream_gradient = -2
Output: (-0.833655, [-0.15251, 0.45753, -0.91506], [0.61004, -1.22008, -0.30502], -0.61004)
Input: inputs = [], weights = [], bias = 0, upstream_gradient = 3
Output: (0.0, [], [], 3.0)
torch.sum reduces an empty vector to a scalar zero.
Tensor.square can reuse the saved tanh output for its local derivative.
A scalar tensor broadcasts across a vector without changing its shape.
Accepts: array
Accepts: array
Accepts: number
Accepts: number