Problems
Loading...
Step 4 of 7
Gated Residual Write
Easy

After a sublayer runs, Qwen4-Max's Gated Residual writes the sublayer output back into every residual branch. All branches receive the same output, and each one scales it by its own write scale between zero and two. The scales are computed from the same normalized branches the gated residual read uses, and the branches are never mixed with each other.

At one position, the residual state holds nrn_r branches R1,…,Rnr∈RdR_1, \dots, R_{n_r} \in \mathbb{R}^{d} with norm weights γi∈Rd\gamma_i \in \mathbb{R}^{d}, and the sublayer output is y∈Rdy \in \mathbb{R}^{d}. The normalized branches use zero-centered RMSNorm with ϵ=10−6\epsilon = 10^{-6}:

R^i=Ri1d∑c=1dRi,c2+ϵ⊙(1+γi)\widehat R_i=\frac{R_i}{\sqrt{\frac{1}{d}\sum_{c=1}^{d}R_{i,c}^2+\epsilon}}\odot(1+\gamma_i)

The supplied helper in the starter code returns exactly these normalized branches. Let vec⁡(R^)∈Rnrd\operatorname{vec}(\widehat R) \in \mathbb{R}^{n_r d} stack them branch-major, so branch ii (counting from zero) fills positions idid to id+d−1id+d-1. With the write projection Ww∈Rnr×nrdW_w \in \mathbb{R}^{n_r \times n_r d} and the sigmoid σ\sigma, the write scales and the updated branches are

s=2 σ(1nrWwvec⁡(R^))s=2\,\sigma\left(\tfrac{1}{n_r}W_w\operatorname{vec}(\widehat R)\right) Ri′=Ri+si yR'_i=R_i+s_i\,y

where sis_i is entry ii of s∈Rnrs \in \mathbb{R}^{n_r}. The branches tensor has shape (batch, sequence, branches, d), the sublayer output has shape (batch, sequence, d), and every position is written independently.

Return a tuple of the updated branches with shape (batch, sequence, branches, d) and the write scales with shape (batch, sequence, branches).

Loading visualization...

Examples

Input: branches = [[[[1,0],[0,2]]]], norm_weight = [[0,0],[0.5,-0.5]], write_weight = [[1,-1,0.5,0.5],[0,0,0,0]], sublayer_output = [[[0.5,-1]]]

Output: (tensor([[[[1.7076, -1.4153], [0.5000, 1.0000]]]]), tensor([[[1.4153, 1.0000]]]))

Explanation: The normalized branches stack into (1.4142, 0, 0, 0.7071). The first write row gives 1.7678, which is 0.8839 after dividing by the two branches, so its scale is 2 times the sigmoid of 0.8839, or 1.4153, and the first branch becomes (1, 0) plus 1.4153 times (0.5, -1). The second row is zero, so its scale is exactly 1 and the second branch receives a plain residual addition.

Input: branches shape (1, 2, 4, 2), norm_weight shape (4, 2), write_weight shape (4, 8), sublayer_output = [[[1,-0.5],[0.25,0.75]]]

Output: (tensor of shape (1, 2, 4, 2), tensor([[[1.2571, 0.8926, 0.7897, 1.0668], [0.7810, 1.1236, 1.1888, 1.0341]]]))

Input: branches = [[[[1,1],[2,-2],[-1,3]]]], norm_weight = [[0,0],[0,0],[0,0]], write_weight shape (3, 6), sublayer_output = [[[1,2]]]

Output: (tensor([[[[1.0000, 1.0000], [4.0000, 2.0000], [0.0000, 5.0000]]]]), tensor([[[0.0000, 2.0000, 1.0000]]]))

Hint 1

flatten(start_dim=-2)\texttt{flatten(start\_dim=-2)} on the helper's output gives the branch-major stack for every position at once.

Hint 2

unsqueeze(-1)\texttt{unsqueeze(-1)} on the scales and unsqueeze(-2)\texttt{unsqueeze(-2)} on the sublayer output line them up against the (branches, d) axes.

Requirements

  • Every write scale lies strictly between zero and two, and a zero row of the write projection gives a scale of exactly one.
  • The sublayer output is added to the raw branches, and the normalized branches are used only to compute the scales.
  • Both tensors keep the dtype and device of the branches, and the arithmetic runs in that dtype.
  • Do not modify the input tensors.

Constraints

  • Every size in the branches shape is at least 1. Qwen4-Max uses four branches, and other counts are valid inputs.
  • The norm weight has shape (branches, d) and stores the zero-centered offset.
  • The write projection has shape (branches, branches times d).
  • The sublayer output has shape (batch, sequence, d) and may be all zeros.
  • The RMSNorm epsilon is fixed at 10−610^{-6} and is not an input.
  • Inputs are float32 or float64 with finite values. Public cases and custom Run inputs are float64.