1 / 1

Layer Normalization as a CUDA Kernel

Neural NetworksLinear AlgebraGPU Programming

Medium

Implement layer normalization on the GPU for a 2D input matrix. Given $\texttt{input}$ of shape $(M, N)$ , normalize each row independently across the last dimension $N$ , then apply a learnable affine transform with per-feature scale $\gamma$ and shift $\beta$ (both length $N$ ).

For each row $i$ , compute the row mean and population variance (divisor $N$ ):

\mu_i = \frac{1}{N} \sum_{j=0}^{N-1} \texttt{input}[i, j]

\sigma_i^2 = \frac{1}{N} \sum_{j=0}^{N-1} \big( \texttt{input}[i, j] - \mu_i \big)^2

Then write the normalized and affined output:

\texttt{output}[i, j] = \frac{\texttt{input}[i, j] - \mu_i}{\sqrt{\sigma_i^2 + \varepsilon}} \cdot \gamma[j] + \beta[j]

$\varepsilon$ is a small positive float for numerical stability. Buffers are row-major device pointers, float32.

Loading visualization...

Examples

Input: input = [[1, 2, 3, 4]], gamma = [1, 1, 1, 1], beta = [0, 0, 0, 0], eps = 1e-5

Output: [[-1.3416, -0.4472, 0.4472, 1.3416]]

Row mean is $2.5$ , population variance is $1.25$ , so each element is shifted and scaled by $1/\sqrt{1.25}$ .

Input: input = [[1, 2, 3], [4, 5, 6]], gamma = [1, 1, 1], beta = [0, 0, 0], eps = 1e-5

Output: [[-1.2247, 0.0, 1.2247], [-1.2247, 0.0, 1.2247]]

Both rows have the same shape after centering and scaling because their variances are equal.

Hint 1

Map one CUDA block to one row. Threads inside a block cooperate via shared memory to reduce that row.

Hint 2

Accumulate both sum and sum_of_squares in the same pass, then compute var = sum_sq / N - mean * mean to avoid a second read of the row.

Hint 3

Use rsqrtf(var + eps) for the inverse standard deviation; it is faster and more accurate than 1.0f / sqrtf(...) on the GPU.

Requirements

Launch one block per row of the $(M, N)$ matrix so each row reduces independently.
Compute the row mean and variance with a single shared-memory reduction over $\sum x$ and $\sum x^2$ , using $\sigma^2 = E[x^2] - \mu^2$ .
Fuse the normalize and affine step into one element-wise write: $\texttt{output}[i,j] = (\texttt{input}[i,j] - \mu_i) \cdot \texttt{inv\_std}_i \cdot \gamma[j] + \beta[j]$ .
Add $\varepsilon$ inside the inverse square root for stability when a row has zero variance.
Do not change the $\texttt{solve}$ signature; the harness depends on the exact 7-parameter prototype.

Constraints

$1 \le M \le 4096$
$1 \le N \le 4096$
$\texttt{input}$ , $\texttt{gamma}$ , $\texttt{beta}$ , $\texttt{output}$ are device pointers, float32.
$\varepsilon$ is a positive float, typically in $[10^{-5}, 10^{-3}]$ .
Row-major flat layout: element $(i, j)$ lives at index $i \cdot N + j$ .

Try Similar Problems

RMS Normalization as a CUDA KernelMedium Matrix Multiplication as a CUDA KernelMedium Matrix-Vector Multiplication as a CUDA KernelMedium Outer Product as a CUDA KernelEasy Matrix Addition as a CUDA KernelEasy Hadamard Product as a CUDA KernelEasy Matrix Transpose as a CUDA KernelEasy L1 Normalization as a CUDA KernelMedium Cosine Similarity as a CUDA KernelMedium CUDA Dot Product: Multi-Block ReductionMedium L2 Normalization as a CUDA KernelMedium Tiled Matrix Multiplication as a CUDA KernelHard

Layer Normalization as a CUDA Kernel

Neural NetworksLinear AlgebraGPU Programming

Medium

For each row $i$ , compute the row mean and population variance (divisor $N$ ):

\mu_i = \frac{1}{N} \sum_{j=0}^{N-1} \texttt{input}[i, j]

\sigma_i^2 = \frac{1}{N} \sum_{j=0}^{N-1} \big( \texttt{input}[i, j] - \mu_i \big)^2

Then write the normalized and affined output:

\texttt{output}[i, j] = \frac{\texttt{input}[i, j] - \mu_i}{\sqrt{\sigma_i^2 + \varepsilon}} \cdot \gamma[j] + \beta[j]

$\varepsilon$ is a small positive float for numerical stability. Buffers are row-major device pointers, float32.

Loading visualization...

Examples

Input: input = [[1, 2, 3, 4]], gamma = [1, 1, 1, 1], beta = [0, 0, 0, 0], eps = 1e-5

Output: [[-1.3416, -0.4472, 0.4472, 1.3416]]

Row mean is $2.5$ , population variance is $1.25$ , so each element is shifted and scaled by $1/\sqrt{1.25}$ .

Input: input = [[1, 2, 3], [4, 5, 6]], gamma = [1, 1, 1], beta = [0, 0, 0], eps = 1e-5

Output: [[-1.2247, 0.0, 1.2247], [-1.2247, 0.0, 1.2247]]

Both rows have the same shape after centering and scaling because their variances are equal.

Hint 1

Map one CUDA block to one row. Threads inside a block cooperate via shared memory to reduce that row.

Hint 2

Accumulate both sum and sum_of_squares in the same pass, then compute var = sum_sq / N - mean * mean to avoid a second read of the row.

Hint 3

Use rsqrtf(var + eps) for the inverse standard deviation; it is faster and more accurate than 1.0f / sqrtf(...) on the GPU.

Requirements

Launch one block per row of the $(M, N)$ matrix so each row reduces independently.
Compute the row mean and variance with a single shared-memory reduction over $\sum x$ and $\sum x^2$ , using $\sigma^2 = E[x^2] - \mu^2$ .
Fuse the normalize and affine step into one element-wise write: $\texttt{output}[i,j] = (\texttt{input}[i,j] - \mu_i) \cdot \texttt{inv\_std}_i \cdot \gamma[j] + \beta[j]$ .
Add $\varepsilon$ inside the inverse square root for stability when a row has zero variance.
Do not change the $\texttt{solve}$ signature; the harness depends on the exact 7-parameter prototype.

Constraints

$1 \le M \le 4096$
$1 \le N \le 4096$
$\texttt{input}$ , $\texttt{gamma}$ , $\texttt{beta}$ , $\texttt{output}$ are device pointers, float32.
$\varepsilon$ is a positive float, typically in $[10^{-5}, 10^{-3}]$ .
Row-major flat layout: element $(i, j)$ lives at index $i \cdot N + j$ .

Try Similar Problems

input =

Accepts: array

gamma =

Accepts: array

beta =

Accepts: array

eps =

Accepts: number

Examples

Requirements

Constraints

Kernel performance

Layer Normalization as a CUDA Kernel

Examples

Requirements

Constraints

Kernel performance