Qwen4-Max injects hashed N-gram memory into its multi-branch residual stream at layer two, following Engram. Each position looks up one row per hash head, the rows form one memory vector, and every residual branch decides through its own scalar gate how much of that memory to accept. A short causal convolution then mixes neighbouring memories before the result is added to the branches.
Head j has a table of Mj rows, and the tables are stacked into one embedding table E whose rows for head j start at oj=∑i<jMi. With hash ids zt,j, the memory vector concatenates the heads in order.
et=jE[oj+zt,j]For branch m with hidden state ht(m) of width d, the key uses that branch's own projection and the value uses one projection shared by every branch.
kt(m)=WK(m)et vt=WVet αt(m)=σ(dRMSNorm(ht(m))TRMSNorm(kt(m))) v~t(m)=αt(m)vtHere σ is the plain sigmoid of Engram version 2 Equation 4. Both gate norms are standard RMSNorm, x/mean(x2)+ϵ⊙w, using branch m's query or key norm weight w and the machine epsilon of the input dtype as ϵ.
Collect every gated value into V~. Normalize each branch with its own standard RMSNorm using the convolution norm weights and ϵ=10−5, flatten the branches branch-major into nrd channels so that branch m owns channels md to md+d−1, and apply a depthwise causal convolution with kernel width w, dilation δ, zero left padding and no bias. For channel c with normalized input u, the convolution output at position t is ∑i=0w−1Wc,iut−(w−1−i)δ,c, so weight index w−1 multiplies the current position and positions before the start contribute zero.
Y=SiLU(Conv1D(RMSNorm(V~)))+V~ H′=H+YThe hash ids have shape (B,T,J) for J heads, E has shape (∑jMj,dh) for head width dh, the hidden states H have shape (B,T,nr,d) for nr branches, the key projections have shape (nr,d,Jdh), the value projection has shape (d,Jdh), each norm weight has shape (nr,d), and the convolution weights have shape (nrd,w).
Return the updated hidden states H′ as a tensor of shape (B,T,nr,d) with the dtype and device of the hidden states.
Examples
Input: hash_ids = [[[1, 0], [2, 4]]], embedding_table = [[0.5, 1], [1, -1], [2, 0], [0, 1], [1, 1], [-1, 0], [0.5, 0.5], [2, 1]], prime_sizes = [3, 5], hidden_states = [[[[1, 0], [0, 1]], [[0.5, 0.5], [-1, 1]]]], key_proj = [[[0, 0, 0, 0], [0, 0, 0, 0]], [[0, 0, 0, 0], [0, 0, 0, 0]]], value_proj = [[1, 0, 0, 0], [0, 0, 1, 0]], query_norm_weight = [[1, 1], [1, 1]], key_norm_weight = [[1, 1], [1, 1]], conv_norm_weight = [[1, 1], [1, 1]], conv_weight = [[0, 0, 0, 1], [0, 0, 0, 1], [0, 0, 0, 1], [0, 0, 0, 1]], dilation = 3
Output: tensor([[[[2.6376, 0.0000], [1.6376, 1.0000]], [[2.2311, 2.2311], [0.7311, 2.7311]]]], dtype=torch.float64)
Explanation: At position 0 head 0 reads row 1 and head 1 reads row 3 + 0, so the memory is [1, -1, 0, 1] and the value is [1, 0]. The keys are zero, so both gates are sigmoid(0) = 0.5 and the gated value is [0.5, 0]. Only the current tap is nonzero, so Y = SiLU(RMSNorm([0.5, 0])) + [0.5, 0] = [1.6376, 0], which is added to both branches.
Input: hash_ids shape (1, 5, 4), embedding_table shape (17, 2), prime_sizes = [3, 7, 5, 2], hidden_states shape (1, 5, 2, 4), key_proj shape (2, 4, 8), value_proj shape (4, 8), norm weights shape (2, 4), conv_weight shape (8, 4), dilation = 3
Output: tensor of shape (1, 5, 2, 4), dtype float64
Input: hash_ids shape (2, 4, 2), embedding_table shape (16, 2), prime_sizes = [11, 5], hidden_states shape (2, 4, 2, 4), key_proj shape (2, 4, 4), value_proj shape (4, 4), norm weights shape (2, 4), conv_weight shape (8, 3), dilation = 2
Output: tensor of shape (2, 4, 2, 4), dtype float64
Hint 1
torch.cumsum over the prime sizes, shifted by one head, gives the row where each head's table starts.
Hint 2
torch.finfo(dtype).eps is the machine epsilon for the two gate norms.
Hint 3
F.conv1d with groups equal to the channel count is depthwise, and padding only on the left keeps it causal.
Requirements
- Use the plain sigmoid of the scaled dot product as each branch's gate.
- Give every branch its own key projection and gate norms, and share the value projection across branches.
- Keep the convolution causal, so no output position depends on a later position.
- Add both the gated memory and the hidden states to the convolution output.
- Leave every input unchanged.
Constraints
- The hash ids form an int64 tensor of shape (B, T, J) with B, T and J at least 1, and every id for head j lies between 0 and the size of head j minus 1.
- The prime sizes are J distinct primes in head order, and their sum equals the number of rows in the embedding table.
- The hidden width and the head width are at least 1, and there is at least one branch.
- The kernel width is at least 1 and the dilation is a positive integer.
- All floating-point inputs share one dtype, float32 or float64, and every value is finite.
All steps in Qwen4-Max
Accepts: array
Accepts: array
Accepts: array
Accepts: array
Accepts: array
Accepts: array
Accepts: array
Accepts: array
Accepts: array
Accepts: array
Accepts: number
N-gram Memory
Qwen4-Max injects hashed N-gram memory into its multi-branch residual stream at layer two, following Engram. Each position looks up one row per hash head, the rows form one memory vector, and every residual branch decides through its own scalar gate how much of that memory to accept. A short causal convolution then mixes neighbouring memories before the result is added to the branches.
Head j has a table of Mj rows, and the tables are stacked into one embedding table E whose rows for head j start at oj=∑i<jMi. With hash ids zt,j, the memory vector concatenates the heads in order.
et=jE[oj+zt,j]For branch m with hidden state ht(m) of width d, the key uses that branch's own projection and the value uses one projection shared by every branch.
kt(m)=WK(m)et vt=WVet αt(m)=σ(dRMSNorm(ht(m))TRMSNorm(kt(m))) v~t(m)=αt(m)vtHere σ is the plain sigmoid of Engram version 2 Equation 4. Both gate norms are standard RMSNorm, x/mean(x2)+ϵ⊙w, using branch m's query or key norm weight w and the machine epsilon of the input dtype as ϵ.
Collect every gated value into V~. Normalize each branch with its own standard RMSNorm using the convolution norm weights and ϵ=10−5, flatten the branches branch-major into nrd channels so that branch m owns channels md to md+d−1, and apply a depthwise causal convolution with kernel width w, dilation δ, zero left padding and no bias. For channel c with normalized input u, the convolution output at position t is ∑i=0w−1Wc,iut−(w−1−i)δ,c, so weight index w−1 multiplies the current position and positions before the start contribute zero.
Y=SiLU(Conv1D(RMSNorm(V~)))+V~ H′=H+YThe hash ids have shape (B,T,J) for J heads, E has shape (∑jMj,dh) for head width dh, the hidden states H have shape (B,T,nr,d) for nr branches, the key projections have shape (nr,d,Jdh), the value projection has shape (d,Jdh), each norm weight has shape (nr,d), and the convolution weights have shape (nrd,w).
Return the updated hidden states H′ as a tensor of shape (B,T,nr,d) with the dtype and device of the hidden states.
Examples
Input: hash_ids = [[[1, 0], [2, 4]]], embedding_table = [[0.5, 1], [1, -1], [2, 0], [0, 1], [1, 1], [-1, 0], [0.5, 0.5], [2, 1]], prime_sizes = [3, 5], hidden_states = [[[[1, 0], [0, 1]], [[0.5, 0.5], [-1, 1]]]], key_proj = [[[0, 0, 0, 0], [0, 0, 0, 0]], [[0, 0, 0, 0], [0, 0, 0, 0]]], value_proj = [[1, 0, 0, 0], [0, 0, 1, 0]], query_norm_weight = [[1, 1], [1, 1]], key_norm_weight = [[1, 1], [1, 1]], conv_norm_weight = [[1, 1], [1, 1]], conv_weight = [[0, 0, 0, 1], [0, 0, 0, 1], [0, 0, 0, 1], [0, 0, 0, 1]], dilation = 3
Output: tensor([[[[2.6376, 0.0000], [1.6376, 1.0000]], [[2.2311, 2.2311], [0.7311, 2.7311]]]], dtype=torch.float64)
Explanation: At position 0 head 0 reads row 1 and head 1 reads row 3 + 0, so the memory is [1, -1, 0, 1] and the value is [1, 0]. The keys are zero, so both gates are sigmoid(0) = 0.5 and the gated value is [0.5, 0]. Only the current tap is nonzero, so Y = SiLU(RMSNorm([0.5, 0])) + [0.5, 0] = [1.6376, 0], which is added to both branches.
Input: hash_ids shape (1, 5, 4), embedding_table shape (17, 2), prime_sizes = [3, 7, 5, 2], hidden_states shape (1, 5, 2, 4), key_proj shape (2, 4, 8), value_proj shape (4, 8), norm weights shape (2, 4), conv_weight shape (8, 4), dilation = 3
Output: tensor of shape (1, 5, 2, 4), dtype float64
Input: hash_ids shape (2, 4, 2), embedding_table shape (16, 2), prime_sizes = [11, 5], hidden_states shape (2, 4, 2, 4), key_proj shape (2, 4, 4), value_proj shape (4, 4), norm weights shape (2, 4), conv_weight shape (8, 3), dilation = 2
Output: tensor of shape (2, 4, 2, 4), dtype float64
Hint 1
torch.cumsum over the prime sizes, shifted by one head, gives the row where each head's table starts.
Hint 2
torch.finfo(dtype).eps is the machine epsilon for the two gate norms.
Hint 3
F.conv1d with groups equal to the channel count is depthwise, and padding only on the left keeps it causal.
Requirements
- Use the plain sigmoid of the scaled dot product as each branch's gate.
- Give every branch its own key projection and gate norms, and share the value projection across branches.
- Keep the convolution causal, so no output position depends on a later position.
- Add both the gated memory and the hidden states to the convolution output.
- Leave every input unchanged.
Constraints
- The hash ids form an int64 tensor of shape (B, T, J) with B, T and J at least 1, and every id for head j lies between 0 and the size of head j minus 1.
- The prime sizes are J distinct primes in head order, and their sum equals the number of rows in the embedding table.
- The hidden width and the head width are at least 1, and there is at least one branch.
- The kernel width is at least 1 and the dilation is a positive integer.
- All floating-point inputs share one dtype, float32 or float64, and every value is finite.
All steps in Qwen4-Max
Accepts: array
Accepts: array
Accepts: array
Accepts: array
Accepts: array
Accepts: array
Accepts: array
Accepts: array
Accepts: array
Accepts: array
Accepts: number