Qwen4-Max widens the residual stream into four parallel branches, and every attention or MLP sublayer reads its input from them through a gated residual read. Each branch is normalized with its own weight, a low-rank network looks at all normalized branches together and produces one gate per branch and channel, and the sublayer input is the average of the gated branches. This read replaces the usual pre-norm in front of the sublayer.
At one position, the residual state holds nr branches R1,…,Rnr∈Rd, and branch i has its own norm weight γi∈Rd. Every norm is zero-centered RMSNorm with ϵ=10−6:
RMSNorm(x;γ)=d1∑c=1dxc2+ϵx⊙(1+γ) Ri=RMSNorm(Ri;γi)Let vec(R)∈Rnrd stack the normalized branches branch-major, so branch i (counting from zero) fills positions id to id+d−1, and let unvec reshape a length nrd vector back into nr rows of width d in the same order. With the down projection Wd∈Rk×nrd, the up projection Wu∈Rnrd×k, the sigmoid σ, and SiLU(z)=zσ(z), the gates and the read are
G=unvecσ(WuSiLU(nr1Wdvec(R))) x=nr1i=1∑nrGi⊙Riwhere Gi∈Rd is row i of G, k is the rank of the bottleneck, and ⊙ is the elementwise product. The branches tensor has shape (batch, sequence, branches, d), and every position is read independently.
Return a tuple of the sublayer input x with shape (batch, sequence, d) and the gates G with shape (batch, sequence, branches, d).
Examples
Input: branches = [[[[1,0],[0,2]]]], norm_weight = [[0,0],[0.5,-0.5]], down_weight = [[1,-1,0.5,0.5]], up_weight = [[1],[-1],[0.5],[2]]
Output: (tensor([[[0.4607, 0.2749]]]), tensor([[[[0.6515, 0.3485], [0.5776, 0.7775]]]]))
Explanation: The normalized branches are (1.4142, 0) and (0, 0.7071), where the second one is scaled by one plus its weight. The down projection gives 1.7678, which becomes 0.8839 after dividing by the two branches and 0.6255 after SiLU. The up projection and the sigmoid turn 0.6255 into the four gates, and the read averages 0.6515 times 1.4142 with 0.5776 times 0 in the first channel and 0.3485 times 0 with 0.7775 times 0.7071 in the second.
Input: branches shape (1, 2, 4, 2), norm_weight shape (4, 2), down_weight shape (2, 8), up_weight shape (8, 2)
Output: (tensor([[[0.3351, 0.2984], [0.1155, 0.1487]]]), tensor of shape (1, 2, 4, 2))
Input: branches = [[[[3,-1,2],[0,0,0],[0.5,0.5,-1]]]], norm_weight = [[0,0,0],[0.4,0.4,0.4],[-0.3,0.2,0]], down_weight shape (1, 9), up_weight shape (9, 1)
Output: (tensor([[[0.3006, 0.0584, -0.0905]]]), tensor of shape (1, 1, 3, 3))
Hint 1
flatten(start_dim=-2) turns the (branches, d) pair into one branch-major axis, and unflatten undoes it.
Hint 2
torch.rsqrt of the mean square with keepdim=True gives one broadcastable factor per branch vector.
Hint 3
Multiplying by down_weight.T and then up_weight.T applies both projections to every position at once.
Requirements
- Every gate lies strictly between zero and one.
- Each position of each batch item is read independently of every other position.
- Both tensors keep the dtype and device of the branches, and the arithmetic runs in that dtype.
- Do not modify the input tensors.
Constraints
- Every size in the branches shape is at least 1. Qwen4-Max uses four branches, and other counts are valid inputs.
- The norm weight has shape (branches, d) and stores the zero-centered offset, so a weight of zero leaves the normalized branch unscaled.
- The down projection has shape (rank, branches times d) and the up projection has shape (branches times d, rank), with rank at least 1.
- The RMSNorm epsilon is fixed at 10−6 and is not an input.
- A branch may be entirely zero at a position, in which case its normalized vector is zero.
- Inputs are float32 or float64 with finite values. Public cases and custom Run inputs are float64.
All steps in Qwen4-Max
Related practice
Accepts: array
Accepts: array
Accepts: array
Accepts: array
Gated Residual Read
Qwen4-Max widens the residual stream into four parallel branches, and every attention or MLP sublayer reads its input from them through a gated residual read. Each branch is normalized with its own weight, a low-rank network looks at all normalized branches together and produces one gate per branch and channel, and the sublayer input is the average of the gated branches. This read replaces the usual pre-norm in front of the sublayer.
At one position, the residual state holds nr branches R1,…,Rnr∈Rd, and branch i has its own norm weight γi∈Rd. Every norm is zero-centered RMSNorm with ϵ=10−6:
RMSNorm(x;γ)=d1∑c=1dxc2+ϵx⊙(1+γ) Ri=RMSNorm(Ri;γi)Let vec(R)∈Rnrd stack the normalized branches branch-major, so branch i (counting from zero) fills positions id to id+d−1, and let unvec reshape a length nrd vector back into nr rows of width d in the same order. With the down projection Wd∈Rk×nrd, the up projection Wu∈Rnrd×k, the sigmoid σ, and SiLU(z)=zσ(z), the gates and the read are
G=unvecσ(WuSiLU(nr1Wdvec(R))) x=nr1i=1∑nrGi⊙Riwhere Gi∈Rd is row i of G, k is the rank of the bottleneck, and ⊙ is the elementwise product. The branches tensor has shape (batch, sequence, branches, d), and every position is read independently.
Return a tuple of the sublayer input x with shape (batch, sequence, d) and the gates G with shape (batch, sequence, branches, d).
Examples
Input: branches = [[[[1,0],[0,2]]]], norm_weight = [[0,0],[0.5,-0.5]], down_weight = [[1,-1,0.5,0.5]], up_weight = [[1],[-1],[0.5],[2]]
Output: (tensor([[[0.4607, 0.2749]]]), tensor([[[[0.6515, 0.3485], [0.5776, 0.7775]]]]))
Explanation: The normalized branches are (1.4142, 0) and (0, 0.7071), where the second one is scaled by one plus its weight. The down projection gives 1.7678, which becomes 0.8839 after dividing by the two branches and 0.6255 after SiLU. The up projection and the sigmoid turn 0.6255 into the four gates, and the read averages 0.6515 times 1.4142 with 0.5776 times 0 in the first channel and 0.3485 times 0 with 0.7775 times 0.7071 in the second.
Input: branches shape (1, 2, 4, 2), norm_weight shape (4, 2), down_weight shape (2, 8), up_weight shape (8, 2)
Output: (tensor([[[0.3351, 0.2984], [0.1155, 0.1487]]]), tensor of shape (1, 2, 4, 2))
Input: branches = [[[[3,-1,2],[0,0,0],[0.5,0.5,-1]]]], norm_weight = [[0,0,0],[0.4,0.4,0.4],[-0.3,0.2,0]], down_weight shape (1, 9), up_weight shape (9, 1)
Output: (tensor([[[0.3006, 0.0584, -0.0905]]]), tensor of shape (1, 1, 3, 3))
Hint 1
flatten(start_dim=-2) turns the (branches, d) pair into one branch-major axis, and unflatten undoes it.
Hint 2
torch.rsqrt of the mean square with keepdim=True gives one broadcastable factor per branch vector.
Hint 3
Multiplying by down_weight.T and then up_weight.T applies both projections to every position at once.
Requirements
- Every gate lies strictly between zero and one.
- Each position of each batch item is read independently of every other position.
- Both tensors keep the dtype and device of the branches, and the arithmetic runs in that dtype.
- Do not modify the input tensors.
Constraints
- Every size in the branches shape is at least 1. Qwen4-Max uses four branches, and other counts are valid inputs.
- The norm weight has shape (branches, d) and stores the zero-centered offset, so a weight of zero leaves the normalized branch unscaled.
- The down projection has shape (rank, branches times d) and the up projection has shape (branches times d, rank), with rank at least 1.
- The RMSNorm epsilon is fixed at 10−6 and is not an input.
- A branch may be entirely zero at a position, in which case its normalized vector is zero.
- Inputs are float32 or float64 with finite values. Public cases and custom Run inputs are float64.
All steps in Qwen4-Max
Related practice
Accepts: array
Accepts: array
Accepts: array
Accepts: array