Implement RMSNorm independently over each final-dimension feature vector, without subtracting its mean.
RMSNorm(x)=d1∑i=1dxi2+ϵx⊙g.Here x is one feature vector, d is its width, i indexes a feature, g is the learned scale vector, epsilon is the supplied positive stabilizer, and the circled-dot operator means elementwise multiplication.
Return a PyTorch tensor with the same shape, dtype, and device as x.
Input: x = [[1,2],[3,4]], g = [1,1], epsilon = 0.00001
Output: [[0.632454,1.264909],[0.848528,1.13137]]
Explanation: Each row is divided by its own root mean square. The learned scale is [1,1], so it does not change the normalized values.
Input: x = [[3,4]], g = [2,0.5], epsilon = 0.000001
Output: [[1.697056,0.565685]]
Input: x = [[0,0,0]], g = [1,2,3], epsilon = 0.0001
Output: [[0,0,0]]
Tensor.mean with keepdim=True retains a broadcastable final axis.
torch.rsqrt computes a reciprocal square root.
Accepts: array
Accepts: array
Accepts: number
Implement RMSNorm independently over each final-dimension feature vector, without subtracting its mean.
RMSNorm(x)=d1∑i=1dxi2+ϵx⊙g.Here x is one feature vector, d is its width, i indexes a feature, g is the learned scale vector, epsilon is the supplied positive stabilizer, and the circled-dot operator means elementwise multiplication.
Return a PyTorch tensor with the same shape, dtype, and device as x.
Input: x = [[1,2],[3,4]], g = [1,1], epsilon = 0.00001
Output: [[0.632454,1.264909],[0.848528,1.13137]]
Explanation: Each row is divided by its own root mean square. The learned scale is [1,1], so it does not change the normalized values.
Input: x = [[3,4]], g = [2,0.5], epsilon = 0.000001
Output: [[1.697056,0.565685]]
Input: x = [[0,0,0]], g = [1,2,3], epsilon = 0.0001
Output: [[0,0,0]]
Tensor.mean with keepdim=True retains a broadcastable final axis.
torch.rsqrt computes a reciprocal square root.
Accepts: array
Accepts: array
Accepts: number
