SiTU-GLU smoothly caps both multiplicative branches of a gated feed-forward activation. For gate projection g=Wgx, up projection u=Wux, and positive caps β1 and β2, the activation is
[β1tanh(β1g)⊙sigmoid(g)]⊙[β2tanh(β2u)].The scaled hyperbolic tangent stays approximately linear near zero while bounding each branch at large magnitude. The gate and up caps are independent. Return the gated activation as one tensor.
Examples
Input: input_tensor = [[0]], gate_projection = [[1]], up_projection = [[1]], gate_cap = 4, up_cap = 25
Output: tensor([[0.0]])
Explanation: Both projected branches are zero, so their bounded product is also zero.
Input: input_tensor = [[1,-0.5],[0.25,2]], gate_projection = [[1,0.5],[-0.25,0.75]], up_projection = [[0.5,-1],[1.25,0.25]], gate_cap = 4, up_cap = 25
Output: tensor of shape (2, 2)
Input: input_tensor = [[[8,-6],[3,5]]], gate_projection = [[1,0],[0,1]], up_projection = [[0.5,0.5],[-0.5,1]], gate_cap = 2, up_cap = 3
Output: tensor of shape (1, 2, 2)
Hint 1
Project the input separately through the gate and up matrices.
Hint 2
Implement each smooth cap as its limit multiplied by tanh of the value divided by that limit.
Hint 3
Apply sigmoid only to the gate projection before multiplying the two capped branches.
Requirements
- Return one tensor whose leading dimensions match the input and whose last dimension matches the projections.
- Preserve the floating-point dtype and device of the input tensor.
- Apply the two cap values independently.
- Do not mutate any input tensor.
Constraints
- Both cap values are positive.
- Projection matrices have the same output width.
- Tensor inputs share a floating-point dtype and device.
- The final input dimension matches both projection input dimensions.
Accepts: array
Accepts: array
Accepts: array
Accepts: number
Accepts: number
SiTU-GLU
SiTU-GLU smoothly caps both multiplicative branches of a gated feed-forward activation. For gate projection g=Wgx, up projection u=Wux, and positive caps β1 and β2, the activation is
[β1tanh(β1g)⊙sigmoid(g)]⊙[β2tanh(β2u)].The scaled hyperbolic tangent stays approximately linear near zero while bounding each branch at large magnitude. The gate and up caps are independent. Return the gated activation as one tensor.
Examples
Input: input_tensor = [[0]], gate_projection = [[1]], up_projection = [[1]], gate_cap = 4, up_cap = 25
Output: tensor([[0.0]])
Explanation: Both projected branches are zero, so their bounded product is also zero.
Input: input_tensor = [[1,-0.5],[0.25,2]], gate_projection = [[1,0.5],[-0.25,0.75]], up_projection = [[0.5,-1],[1.25,0.25]], gate_cap = 4, up_cap = 25
Output: tensor of shape (2, 2)
Input: input_tensor = [[[8,-6],[3,5]]], gate_projection = [[1,0],[0,1]], up_projection = [[0.5,0.5],[-0.5,1]], gate_cap = 2, up_cap = 3
Output: tensor of shape (1, 2, 2)
Hint 1
Project the input separately through the gate and up matrices.
Hint 2
Implement each smooth cap as its limit multiplied by tanh of the value divided by that limit.
Hint 3
Apply sigmoid only to the gate projection before multiplying the two capped branches.
Requirements
- Return one tensor whose leading dimensions match the input and whose last dimension matches the projections.
- Preserve the floating-point dtype and device of the input tensor.
- Apply the two cap values independently.
- Do not mutate any input tensor.
Constraints
- Both cap values are positive.
- Projection matrices have the same output width.
- Tensor inputs share a floating-point dtype and device.
- The final input dimension matches both projection input dimensions.
Accepts: array
Accepts: array
Accepts: array
Accepts: number
Accepts: number