After a sublayer runs, Qwen4-Max's Gated Residual writes the sublayer output back into every residual branch. All branches receive the same output, and each one scales it by its own write scale between zero and two. The scales are computed from the same normalized branches the gated residual read uses, and the branches are never mixed with each other.
At one position, the residual state holds nr branches R1,…,Rnr∈Rd with norm weights γi∈Rd, and the sublayer output is y∈Rd. The normalized branches use zero-centered RMSNorm with ϵ=10−6:
Ri=d1∑c=1dRi,c2+ϵRi⊙(1+γi)The supplied helper in the starter code returns exactly these normalized branches. Let vec(R)∈Rnrd stack them branch-major, so branch i (counting from zero) fills positions id to id+d−1. With the write projection Ww∈Rnr×nrd and the sigmoid σ, the write scales and the updated branches are
s=2σ(nr1Wwvec(R)) Ri′=Ri+siywhere si is entry i of s∈Rnr. The branches tensor has shape (batch, sequence, branches, d), the sublayer output has shape (batch, sequence, d), and every position is written independently.
Return a tuple of the updated branches with shape (batch, sequence, branches, d) and the write scales with shape (batch, sequence, branches).
Examples
Input: branches = [[[[1,0],[0,2]]]], norm_weight = [[0,0],[0.5,-0.5]], write_weight = [[1,-1,0.5,0.5],[0,0,0,0]], sublayer_output = [[[0.5,-1]]]
Output: (tensor([[[[1.7076, -1.4153], [0.5000, 1.0000]]]]), tensor([[[1.4153, 1.0000]]]))
Explanation: The normalized branches stack into (1.4142, 0, 0, 0.7071). The first write row gives 1.7678, which is 0.8839 after dividing by the two branches, so its scale is 2 times the sigmoid of 0.8839, or 1.4153, and the first branch becomes (1, 0) plus 1.4153 times (0.5, -1). The second row is zero, so its scale is exactly 1 and the second branch receives a plain residual addition.
Input: branches shape (1, 2, 4, 2), norm_weight shape (4, 2), write_weight shape (4, 8), sublayer_output = [[[1,-0.5],[0.25,0.75]]]
Output: (tensor of shape (1, 2, 4, 2), tensor([[[1.2571, 0.8926, 0.7897, 1.0668], [0.7810, 1.1236, 1.1888, 1.0341]]]))
Input: branches = [[[[1,1],[2,-2],[-1,3]]]], norm_weight = [[0,0],[0,0],[0,0]], write_weight shape (3, 6), sublayer_output = [[[1,2]]]
Output: (tensor([[[[1.0000, 1.0000], [4.0000, 2.0000], [0.0000, 5.0000]]]]), tensor([[[0.0000, 2.0000, 1.0000]]]))
Hint 1
flatten(start_dim=-2) on the helper's output gives the branch-major stack for every position at once.
Hint 2
unsqueeze(-1) on the scales and unsqueeze(-2) on the sublayer output line them up against the (branches, d) axes.
Requirements
- Every write scale lies strictly between zero and two, and a zero row of the write projection gives a scale of exactly one.
- The sublayer output is added to the raw branches, and the normalized branches are used only to compute the scales.
- Both tensors keep the dtype and device of the branches, and the arithmetic runs in that dtype.
- Do not modify the input tensors.
Constraints
- Every size in the branches shape is at least 1. Qwen4-Max uses four branches, and other counts are valid inputs.
- The norm weight has shape (branches, d) and stores the zero-centered offset.
- The write projection has shape (branches, branches times d).
- The sublayer output has shape (batch, sequence, d) and may be all zeros.
- The RMSNorm epsilon is fixed at 10−6 and is not an input.
- Inputs are float32 or float64 with finite values. Public cases and custom Run inputs are float64.
All steps in Qwen4-Max
Related practice
Accepts: array
Accepts: array
Accepts: array
Accepts: array
Gated Residual Write
After a sublayer runs, Qwen4-Max's Gated Residual writes the sublayer output back into every residual branch. All branches receive the same output, and each one scales it by its own write scale between zero and two. The scales are computed from the same normalized branches the gated residual read uses, and the branches are never mixed with each other.
At one position, the residual state holds nr branches R1,…,Rnr∈Rd with norm weights γi∈Rd, and the sublayer output is y∈Rd. The normalized branches use zero-centered RMSNorm with ϵ=10−6:
Ri=d1∑c=1dRi,c2+ϵRi⊙(1+γi)The supplied helper in the starter code returns exactly these normalized branches. Let vec(R)∈Rnrd stack them branch-major, so branch i (counting from zero) fills positions id to id+d−1. With the write projection Ww∈Rnr×nrd and the sigmoid σ, the write scales and the updated branches are
s=2σ(nr1Wwvec(R)) Ri′=Ri+siywhere si is entry i of s∈Rnr. The branches tensor has shape (batch, sequence, branches, d), the sublayer output has shape (batch, sequence, d), and every position is written independently.
Return a tuple of the updated branches with shape (batch, sequence, branches, d) and the write scales with shape (batch, sequence, branches).
Examples
Input: branches = [[[[1,0],[0,2]]]], norm_weight = [[0,0],[0.5,-0.5]], write_weight = [[1,-1,0.5,0.5],[0,0,0,0]], sublayer_output = [[[0.5,-1]]]
Output: (tensor([[[[1.7076, -1.4153], [0.5000, 1.0000]]]]), tensor([[[1.4153, 1.0000]]]))
Explanation: The normalized branches stack into (1.4142, 0, 0, 0.7071). The first write row gives 1.7678, which is 0.8839 after dividing by the two branches, so its scale is 2 times the sigmoid of 0.8839, or 1.4153, and the first branch becomes (1, 0) plus 1.4153 times (0.5, -1). The second row is zero, so its scale is exactly 1 and the second branch receives a plain residual addition.
Input: branches shape (1, 2, 4, 2), norm_weight shape (4, 2), write_weight shape (4, 8), sublayer_output = [[[1,-0.5],[0.25,0.75]]]
Output: (tensor of shape (1, 2, 4, 2), tensor([[[1.2571, 0.8926, 0.7897, 1.0668], [0.7810, 1.1236, 1.1888, 1.0341]]]))
Input: branches = [[[[1,1],[2,-2],[-1,3]]]], norm_weight = [[0,0],[0,0],[0,0]], write_weight shape (3, 6), sublayer_output = [[[1,2]]]
Output: (tensor([[[[1.0000, 1.0000], [4.0000, 2.0000], [0.0000, 5.0000]]]]), tensor([[[0.0000, 2.0000, 1.0000]]]))
Hint 1
flatten(start_dim=-2) on the helper's output gives the branch-major stack for every position at once.
Hint 2
unsqueeze(-1) on the scales and unsqueeze(-2) on the sublayer output line them up against the (branches, d) axes.
Requirements
- Every write scale lies strictly between zero and two, and a zero row of the write projection gives a scale of exactly one.
- The sublayer output is added to the raw branches, and the normalized branches are used only to compute the scales.
- Both tensors keep the dtype and device of the branches, and the arithmetic runs in that dtype.
- Do not modify the input tensors.
Constraints
- Every size in the branches shape is at least 1. Qwen4-Max uses four branches, and other counts are valid inputs.
- The norm weight has shape (branches, d) and stores the zero-centered offset.
- The write projection has shape (branches, branches times d).
- The sublayer output has shape (batch, sequence, d) and may be all zeros.
- The RMSNorm epsilon is fixed at 10−6 and is not an input.
- Inputs are float32 or float64 with finite values. Public cases and custom Run inputs are float64.
All steps in Qwen4-Max
Related practice
Accepts: array
Accepts: array
Accepts: array
Accepts: array