Problems
Loading...
Step 2 of 7
N-gram Memory
Hard

Qwen4-Max injects hashed N-gram memory into its multi-branch residual stream at layer two, following Engram. Each position looks up one row per hash head, the rows form one memory vector, and every residual branch decides through its own scalar gate how much of that memory to accept. A short causal convolution then mixes neighbouring memories before the result is added to the branches.

Head jj has a table of MjM_j rows, and the tables are stacked into one embedding table EE whose rows for head jj start at oj=∑i<jMio_j=\sum_{i<j}M_i. With hash ids zt,jz_{t,j}, the memory vector concatenates the heads in order.

et=∥j E[oj+zt,j]e_t=\big\Vert_{j}\,E\left[o_j+z_{t,j}\right]

For branch mm with hidden state ht(m)h_t^{(m)} of width dd, the key uses that branch's own projection and the value uses one projection shared by every branch.

kt(m)=WK(m)etk_t^{(m)}=W_K^{(m)}e_t vt=WVetv_t=W_Ve_t αt(m)=σ(RMSNorm⁡(ht(m))TRMSNorm⁡(kt(m))d)\alpha_t^{(m)}=\sigma\left(\frac{\operatorname{RMSNorm}(h_t^{(m)})^{\mathsf T}\operatorname{RMSNorm}(k_t^{(m)})}{\sqrt d}\right) v~t(m)=αt(m)vt\tilde v_t^{(m)}=\alpha_t^{(m)}v_t

Here σ\sigma is the plain sigmoid of Engram version 2 Equation 4. Both gate norms are standard RMSNorm, x/mean⁡(x2)+ϵ⊙wx/\sqrt{\operatorname{mean}(x^2)+\epsilon}\odot w, using branch mm's query or key norm weight ww and the machine epsilon of the input dtype as ϵ\epsilon.

Collect every gated value into V~\tilde V. Normalize each branch with its own standard RMSNorm using the convolution norm weights and ϵ=10−5\epsilon=10^{-5}, flatten the branches branch-major into nrdn_rd channels so that branch mm owns channels mdmd to md+d−1md+d-1, and apply a depthwise causal convolution with kernel width ww, dilation δ\delta, zero left padding and no bias. For channel cc with normalized input uu, the convolution output at position tt is ∑i=0w−1Wc,i ut−(w−1−i)δ,c\sum_{i=0}^{w-1}W_{c,i}\,u_{t-(w-1-i)\delta,c}, so weight index w−1w-1 multiplies the current position and positions before the start contribute zero.

Y=SiLU⁡(Conv1D⁡(RMSNorm⁡(V~)))+V~Y=\operatorname{SiLU}\left(\operatorname{Conv1D}\left(\operatorname{RMSNorm}(\tilde V)\right)\right)+\tilde V H′=H+YH'=H+Y

The hash ids have shape (B,T,J)(B, T, J) for JJ heads, EE has shape (∑jMj,dh)(\sum_j M_j, d_h) for head width dhd_h, the hidden states HH have shape (B,T,nr,d)(B, T, n_r, d) for nrn_r branches, the key projections have shape (nr,d,Jdh)(n_r, d, Jd_h), the value projection has shape (d,Jdh)(d, Jd_h), each norm weight has shape (nr,d)(n_r, d), and the convolution weights have shape (nrd,w)(n_rd, w).

Return the updated hidden states H′H' as a tensor of shape (B,T,nr,d)(B, T, n_r, d) with the dtype and device of the hidden states.

Loading visualization...

Examples

Input: hash_ids = [[[1, 0], [2, 4]]], embedding_table = [[0.5, 1], [1, -1], [2, 0], [0, 1], [1, 1], [-1, 0], [0.5, 0.5], [2, 1]], prime_sizes = [3, 5], hidden_states = [[[[1, 0], [0, 1]], [[0.5, 0.5], [-1, 1]]]], key_proj = [[[0, 0, 0, 0], [0, 0, 0, 0]], [[0, 0, 0, 0], [0, 0, 0, 0]]], value_proj = [[1, 0, 0, 0], [0, 0, 1, 0]], query_norm_weight = [[1, 1], [1, 1]], key_norm_weight = [[1, 1], [1, 1]], conv_norm_weight = [[1, 1], [1, 1]], conv_weight = [[0, 0, 0, 1], [0, 0, 0, 1], [0, 0, 0, 1], [0, 0, 0, 1]], dilation = 3

Output: tensor([[[[2.6376, 0.0000], [1.6376, 1.0000]], [[2.2311, 2.2311], [0.7311, 2.7311]]]], dtype=torch.float64)

Explanation: At position 0 head 0 reads row 1 and head 1 reads row 3 + 0, so the memory is [1, -1, 0, 1] and the value is [1, 0]. The keys are zero, so both gates are sigmoid(0) = 0.5 and the gated value is [0.5, 0]. Only the current tap is nonzero, so Y = SiLU(RMSNorm([0.5, 0])) + [0.5, 0] = [1.6376, 0], which is added to both branches.

Input: hash_ids shape (1, 5, 4), embedding_table shape (17, 2), prime_sizes = [3, 7, 5, 2], hidden_states shape (1, 5, 2, 4), key_proj shape (2, 4, 8), value_proj shape (4, 8), norm weights shape (2, 4), conv_weight shape (8, 4), dilation = 3

Output: tensor of shape (1, 5, 2, 4), dtype float64

Input: hash_ids shape (2, 4, 2), embedding_table shape (16, 2), prime_sizes = [11, 5], hidden_states shape (2, 4, 2, 4), key_proj shape (2, 4, 4), value_proj shape (4, 4), norm weights shape (2, 4), conv_weight shape (8, 3), dilation = 2

Output: tensor of shape (2, 4, 2, 4), dtype float64

Hint 1

torch.cumsum\texttt{torch.cumsum} over the prime sizes, shifted by one head, gives the row where each head's table starts.

Hint 2

torch.finfo(dtype).eps\texttt{torch.finfo(dtype).eps} is the machine epsilon for the two gate norms.

Hint 3

F.conv1d\texttt{F.conv1d} with groups equal to the channel count is depthwise, and padding only on the left keeps it causal.

Requirements

  • Use the plain sigmoid of the scaled dot product as each branch's gate.
  • Give every branch its own key projection and gate norms, and share the value projection across branches.
  • Keep the convolution causal, so no output position depends on a later position.
  • Add both the gated memory and the hidden states to the convolution output.
  • Leave every input unchanged.

Constraints

  • The hash ids form an int64 tensor of shape (B, T, J) with B, T and J at least 1, and every id for head j lies between 0 and the size of head j minus 1.
  • The prime sizes are J distinct primes in head order, and their sum equals the number of rows in the embedding table.
  • The hidden width and the head width are at least 1, and there is at least one branch.
  • The kernel width is at least 1 and the dilation is a positive integer.
  • All floating-point inputs share one dtype, float32 or float64, and every value is finite.