Implement the SiLU (sigmoid-weighted linear unit, also called Swish) activation in Triton. For each input element:
out[i]=x[i]⋅σ(x[i])=1+e−x[i]x[i]SiLU is the activation in LLaMA, Mistral, and most modern Transformer feedforward gates. It is the simplest activation that involves exp: one exponential, one reciprocal, one multiply. The kernel structure is the same as the other foundation maps; the new beat is calling tl.exp inside a tile.
Input: x = [0.0, 1.0, -1.0, 2.0]
Output: [0.0, 0.7311, -0.2689, 1.7616]
Input: x = [-2.0, 0.5, 3.0]
Output: [-0.2384, 0.3112, 2.8577]
SiLU is x⋅σ(x) which equals x/(1+e−x). The second form is one tl.exp, one add, one divide.
The exponent has a negative sign: tl.exp(−x), not tl.exp(x). The sign is what gives the sigmoid its shape.
Same tail-mask discipline as vector add. No reduction, no shared memory, one fused expression per lane.
atol=1e-3 against torch.nn.functional.silu.Accepts: array
Submit a correct solution to see your kernel performance.
Implement the SiLU (sigmoid-weighted linear unit, also called Swish) activation in Triton. For each input element:
out[i]=x[i]⋅σ(x[i])=1+e−x[i]x[i]SiLU is the activation in LLaMA, Mistral, and most modern Transformer feedforward gates. It is the simplest activation that involves exp: one exponential, one reciprocal, one multiply. The kernel structure is the same as the other foundation maps; the new beat is calling tl.exp inside a tile.
Input: x = [0.0, 1.0, -1.0, 2.0]
Output: [0.0, 0.7311, -0.2689, 1.7616]
Input: x = [-2.0, 0.5, 3.0]
Output: [-0.2384, 0.3112, 2.8577]
SiLU is x⋅σ(x) which equals x/(1+e−x). The second form is one tl.exp, one add, one divide.
The exponent has a negative sign: tl.exp(−x), not tl.exp(x). The sign is what gives the sigmoid its shape.
Same tail-mask discipline as vector add. No reduction, no shared memory, one fused expression per lane.
atol=1e-3 against torch.nn.functional.silu.Accepts: array
Submit a correct solution to see your kernel performance.