Qwen Sparse Attention decides which earlier tokens each query will read by first scoring compressed key blocks with a small indexer. The indexer is multi-query attention with H query heads and one key head shared by every head. Keys are averaged over non-overlapping blocks of r tokens before any positional encoding, so each block carries a single key placed at its start position, while every query keeps its own token position.
For token i with hidden state xi and indexer head h,
q^ih=RMSNorm(WQhxi;γq),ki=WKxi k^b=RMSNorm(r1t=pb∑pb+r−1kt;γk),pb=br,0≤b<⌊T/r⌋ qih=PRoPE(q^ih,i),kˉb=PRoPE(k^b,pb) Iib={∑h=1HReLU(⟨qih,kˉb⟩)−∞pb+r−1≤iotherwiseHere T is the sequence length, xi∈Rdmodel is row i of the hidden states, WQh∈Rdh×dmodel is the query projection of head h, WK∈Rdh×dmodel is the shared key projection, γq and γk are the query and key norm weights of length dh, and pb is the position of the first token in block b. A trailing group of fewer than r tokens is not a block.
Every norm is the zero-centered form RMSNorm(v;γ)=v/mean(v2)+10−6⊙(1+γ). PRoPE(v,p) rotates the leading rotary-dimension features of v at position p in the rotate-half layout with frequencies θ−2j/drot for the given base θ, and leaves the remaining features unchanged. The starter code supplies both operations as complete helpers.
Return the score matrix I as a tensor of shape (T,⌊T/r⌋) with the dtype of the hidden states, holding minus infinity wherever a block is not yet complete at the query.
Examples
Input: hidden_states = [[1,0],[0,1],[1,1],[-1,0.5]], query_projections = [[[1,0],[0,1]]], key_projection = [[0.5,1],[1,-0.5]], query_norm_weight = [0,0], key_norm_weight = [0,0], block_size = 2, rotary_dim = 2, rope_base = 10000
Output: tensor([[-inf, -inf], [0.0000, -inf], [0.0000, -inf], [1.2005, 0.0000]], dtype=torch.float64)
Explanation: Query 0 sits inside block 0, so both blocks are minus infinity. Block 0 becomes visible at query 1, where its rotated dot product is negative and ReLU clips it to 0. Block 1 holds tokens 2 and 3, so it appears only at query 3, which scores 1.2005 for block 0 and 0 for block 1.
Input: hidden states shape (5, 3), query projections shape (2, 4, 3), key projection shape (4, 3), zero norm weights, block_size = 2, rotary_dim = 2, rope_base = 10000
Output: tensor([[-inf, -inf], [3.2535, -inf], [0.0000, -inf], [0.0000, 0.0000], [1.4158, 2.4498]], dtype=torch.float64)
Input: hidden states shape (6, 3), query projections shape (2, 4, 3), key projection shape (4, 3), nonzero norm weights, block_size = 3, rotary_dim = 4, rope_base = 100
Output: tensor([[-inf, -inf], [-inf, -inf], [0.0000, -inf], [2.7959, -inf], [1.9751, -inf], [1.2661, 0.5275]], dtype=torch.float64)
Hint 1
Pool the first num_blocks * block_size key rows with .reshape followed by .mean(dim=1).
Hint 2
The pooled keys take the positions torch.arange(num_blocks) * block_size in the rotary helper.
Hint 3
Compare each block's last position with every query position by broadcasting, then write minus infinity with masked_fill.
Requirements
- Return one PyTorch tensor of shape sequence length by number of complete blocks, with the dtype and device of the hidden states.
- Hold minus infinity exactly where the last token of a block comes after the query, and a finite score of zero or more everywhere else.
- Keep the supplied normalization and rotary helpers unchanged.
- Leave every input tensor unchanged.
Constraints
- The hidden states have shape (T,dmodel) with 1≤T≤12.
- The query projections have shape (H,dh,dmodel) with H≥1, and the key projection has shape (dh,dmodel).
- Both norm weights have length dh, and dh is even.
- The block size is a positive integer and may exceed T, in which case there are no complete blocks and the result has zero columns.
- The rotary dimension is an even integer between 0 and dh, and the RoPE base is positive.
- All tensors share one dtype, float32 or float64, hold finite values, and may be noncontiguous.
All steps in Qwen4-Max
Accepts: array
Accepts: array
Accepts: array
Accepts: array
Accepts: array
Accepts: number
Accepts: number
Accepts: number
QSA Block Indexer
Qwen Sparse Attention decides which earlier tokens each query will read by first scoring compressed key blocks with a small indexer. The indexer is multi-query attention with H query heads and one key head shared by every head. Keys are averaged over non-overlapping blocks of r tokens before any positional encoding, so each block carries a single key placed at its start position, while every query keeps its own token position.
For token i with hidden state xi and indexer head h,
q^ih=RMSNorm(WQhxi;γq),ki=WKxi k^b=RMSNorm(r1t=pb∑pb+r−1kt;γk),pb=br,0≤b<⌊T/r⌋ qih=PRoPE(q^ih,i),kˉb=PRoPE(k^b,pb) Iib={∑h=1HReLU(⟨qih,kˉb⟩)−∞pb+r−1≤iotherwiseHere T is the sequence length, xi∈Rdmodel is row i of the hidden states, WQh∈Rdh×dmodel is the query projection of head h, WK∈Rdh×dmodel is the shared key projection, γq and γk are the query and key norm weights of length dh, and pb is the position of the first token in block b. A trailing group of fewer than r tokens is not a block.
Every norm is the zero-centered form RMSNorm(v;γ)=v/mean(v2)+10−6⊙(1+γ). PRoPE(v,p) rotates the leading rotary-dimension features of v at position p in the rotate-half layout with frequencies θ−2j/drot for the given base θ, and leaves the remaining features unchanged. The starter code supplies both operations as complete helpers.
Return the score matrix I as a tensor of shape (T,⌊T/r⌋) with the dtype of the hidden states, holding minus infinity wherever a block is not yet complete at the query.
Examples
Input: hidden_states = [[1,0],[0,1],[1,1],[-1,0.5]], query_projections = [[[1,0],[0,1]]], key_projection = [[0.5,1],[1,-0.5]], query_norm_weight = [0,0], key_norm_weight = [0,0], block_size = 2, rotary_dim = 2, rope_base = 10000
Output: tensor([[-inf, -inf], [0.0000, -inf], [0.0000, -inf], [1.2005, 0.0000]], dtype=torch.float64)
Explanation: Query 0 sits inside block 0, so both blocks are minus infinity. Block 0 becomes visible at query 1, where its rotated dot product is negative and ReLU clips it to 0. Block 1 holds tokens 2 and 3, so it appears only at query 3, which scores 1.2005 for block 0 and 0 for block 1.
Input: hidden states shape (5, 3), query projections shape (2, 4, 3), key projection shape (4, 3), zero norm weights, block_size = 2, rotary_dim = 2, rope_base = 10000
Output: tensor([[-inf, -inf], [3.2535, -inf], [0.0000, -inf], [0.0000, 0.0000], [1.4158, 2.4498]], dtype=torch.float64)
Input: hidden states shape (6, 3), query projections shape (2, 4, 3), key projection shape (4, 3), nonzero norm weights, block_size = 3, rotary_dim = 4, rope_base = 100
Output: tensor([[-inf, -inf], [-inf, -inf], [0.0000, -inf], [2.7959, -inf], [1.9751, -inf], [1.2661, 0.5275]], dtype=torch.float64)
Hint 1
Pool the first num_blocks * block_size key rows with .reshape followed by .mean(dim=1).
Hint 2
The pooled keys take the positions torch.arange(num_blocks) * block_size in the rotary helper.
Hint 3
Compare each block's last position with every query position by broadcasting, then write minus infinity with masked_fill.
Requirements
- Return one PyTorch tensor of shape sequence length by number of complete blocks, with the dtype and device of the hidden states.
- Hold minus infinity exactly where the last token of a block comes after the query, and a finite score of zero or more everywhere else.
- Keep the supplied normalization and rotary helpers unchanged.
- Leave every input tensor unchanged.
Constraints
- The hidden states have shape (T,dmodel) with 1≤T≤12.
- The query projections have shape (H,dh,dmodel) with H≥1, and the key projection has shape (dh,dmodel).
- Both norm weights have length dh, and dh is even.
- The block size is a positive integer and may exceed T, in which case there are no complete blocks and the result has zero columns.
- The rotary dimension is an even integer between 0 and dh, and the RoPE base is positive.
- All tensors share one dtype, float32 or float64, hold finite values, and may be noncontiguous.
All steps in Qwen4-Max
Accepts: array
Accepts: array
Accepts: array
Accepts: array
Accepts: array
Accepts: number
Accepts: number
Accepts: number