Qwen Sparse Attention (QSA) trains its lightweight indexer by distilling the model's dense attention into block scores. Compute both distillation losses from the teacher's attention probabilities, the indexer's block scores, and the blocks QSA selected for each query.
The teacher has H heads of causal attention probabilities P(h), each of shape (T,T), where T is the sequence length and row i is the distribution of query i over keys 0 to i. Sum the heads and L1-normalize each row:
aij=∑j′∑hPij′(h)∑hPij(h)Keys form nb=⌊T/r⌋ blocks of size r, and block b covers keys pb=br to pb+r−1. Block b is complete for query i when pb+r−1≤i, and Ci is the set of complete blocks of query i. The indexer score Iib is finite exactly for the blocks in Ci and minus infinity for every other block. Over complete blocks only, max-pool the teacher row and L1-normalize it (Equation 17):
aˉib=0≤u<rmaxai,pb+u,b∈Ci a^ib=∑b′∈Ciaˉib′aˉibTeacher mass on keys outside complete blocks, which covers the current unfinished block and any trailing keys after the last full block, is dropped by this normalization.
For a set of blocks K⊆Ci, renormalize the teacher target and the indexer softmax over K:
a~ib=∑b′∈Ka^ib′a^ib,q~ib=∑b′∈KeIib′eIib,b∈K Di(K)=b∈K∑a~iblogq~iba~ibThis is the KL divergence with the teacher distribution first, and a term with a~ib=0 contributes zero. Let Q be the set of queries with at least one complete block, N the number of such queries, and Bi⊆Ci the selected blocks of query i. The stage 1 loss (Equation 18) and the stage 2 loss (Equation 20) are
L1=N1i∈Q∑Di(Ci) L2=N1i∈Q∑Di(Bi)Return a tuple of exactly two 0-dimensional tensors, the stage 1 loss followed by the stage 2 loss, each with the dtype and device of the teacher attention.
Examples
Input: teacher_attention = [[[1, 0, 0, 0], [0.1576, 0.8424, 0, 0], [0.733, 0.0601, 0.2069, 0], [0.0253, 0.3941, 0.5189, 0.0617]]], indexer_scores = [[-inf, -inf], [0, -inf], [1.1254, -inf], [0, 0]], block_size = 2, selected_blocks = [[False, False], [True, False], [True, False], [True, False]]
Output: (tensor(0.003124, dtype=torch.float64), tensor(0.0, dtype=torch.float64))
Explanation: Query 0 has no complete block and is left out, so N is 3. Queries 1 and 2 see only block 0, so both distributions put all their weight on it and their KL is 0. Query 3 pools its row to 0.3941 and 0.5189, which normalize to 0.4317 and 0.5683, while its equal scores give the indexer 0.5 and 0.5, a KL of 0.009372. Dividing by 3 gives 0.003124. Every query keeps a single selected block, so the stage 2 loss is 0.
Input: teacher_attention = tensor of shape (2, 6, 6), indexer_scores = tensor of shape (6, 3) with -inf for incomplete blocks, block_size = 2, selected_blocks = boolean tensor of shape (6, 3) with at most two blocks per query
Output: (tensor(0.074565, dtype=torch.float64), tensor(0.048984, dtype=torch.float64))
Input: teacher_attention = tensor of shape (1, 7, 7) with some zero weights, indexer_scores = tensor of shape (7, 2) with -inf for incomplete blocks, block_size = 3, selected_blocks = boolean tensor of shape (7, 2) with one block per query
Output: (tensor(0.32871, dtype=torch.float64), tensor(0.0, dtype=torch.float64))
Hint 1
Reshape the first nbr key columns to (T, n_b, r) and reduce the last axis with amax.
Hint 2
torch.isfinite marks complete blocks, and masked_fill before torch.log_softmax restricts the indexer to any block set.
Hint 3
Wrap each KL term in torch.where on a positive teacher value so zero terms never become nan.
Requirements
- Queries with no complete block are left out of both sums and of the count N.
- Terms with zero teacher probability contribute zero, so both losses are finite.
- Use the selected blocks exactly as given, without recomputing a top-k selection.
- Leave every input unchanged.
Constraints
- The teacher attention is a float32 or float64 tensor of shape (H, T, T) with H from 1 to 4 and T from 2 to 10.
- Every teacher row is causal, with zeros above the diagonal, nonnegative, and sums to one in every head.
- The block size r is an integer from 1 to T, so at least one query has a complete block.
- The indexer scores have shape (T, floor(T / r)) and the teacher's dtype. The score of block b for query i is finite and nonnegative when block b is complete for query i, and minus infinity otherwise.
- In the Run panel, minus infinity is written as the string "-inf" inside the score list.
- The selected blocks form a boolean tensor of shape (T, floor(T / r)). They mark only complete blocks, at least one for every query that has a complete block, and none for the other queries.
- For every query with a complete block, the pooled teacher mass over its complete blocks and over its selected blocks is positive.
All steps in Qwen4-Max
Accepts: array
Accepts: array
Accepts: integer
Accepts: array
QSA Indexer Distillation
Qwen Sparse Attention (QSA) trains its lightweight indexer by distilling the model's dense attention into block scores. Compute both distillation losses from the teacher's attention probabilities, the indexer's block scores, and the blocks QSA selected for each query.
The teacher has H heads of causal attention probabilities P(h), each of shape (T,T), where T is the sequence length and row i is the distribution of query i over keys 0 to i. Sum the heads and L1-normalize each row:
aij=∑j′∑hPij′(h)∑hPij(h)Keys form nb=⌊T/r⌋ blocks of size r, and block b covers keys pb=br to pb+r−1. Block b is complete for query i when pb+r−1≤i, and Ci is the set of complete blocks of query i. The indexer score Iib is finite exactly for the blocks in Ci and minus infinity for every other block. Over complete blocks only, max-pool the teacher row and L1-normalize it (Equation 17):
aˉib=0≤u<rmaxai,pb+u,b∈Ci a^ib=∑b′∈Ciaˉib′aˉibTeacher mass on keys outside complete blocks, which covers the current unfinished block and any trailing keys after the last full block, is dropped by this normalization.
For a set of blocks K⊆Ci, renormalize the teacher target and the indexer softmax over K:
a~ib=∑b′∈Ka^ib′a^ib,q~ib=∑b′∈KeIib′eIib,b∈K Di(K)=b∈K∑a~iblogq~iba~ibThis is the KL divergence with the teacher distribution first, and a term with a~ib=0 contributes zero. Let Q be the set of queries with at least one complete block, N the number of such queries, and Bi⊆Ci the selected blocks of query i. The stage 1 loss (Equation 18) and the stage 2 loss (Equation 20) are
L1=N1i∈Q∑Di(Ci) L2=N1i∈Q∑Di(Bi)Return a tuple of exactly two 0-dimensional tensors, the stage 1 loss followed by the stage 2 loss, each with the dtype and device of the teacher attention.
Examples
Input: teacher_attention = [[[1, 0, 0, 0], [0.1576, 0.8424, 0, 0], [0.733, 0.0601, 0.2069, 0], [0.0253, 0.3941, 0.5189, 0.0617]]], indexer_scores = [[-inf, -inf], [0, -inf], [1.1254, -inf], [0, 0]], block_size = 2, selected_blocks = [[False, False], [True, False], [True, False], [True, False]]
Output: (tensor(0.003124, dtype=torch.float64), tensor(0.0, dtype=torch.float64))
Explanation: Query 0 has no complete block and is left out, so N is 3. Queries 1 and 2 see only block 0, so both distributions put all their weight on it and their KL is 0. Query 3 pools its row to 0.3941 and 0.5189, which normalize to 0.4317 and 0.5683, while its equal scores give the indexer 0.5 and 0.5, a KL of 0.009372. Dividing by 3 gives 0.003124. Every query keeps a single selected block, so the stage 2 loss is 0.
Input: teacher_attention = tensor of shape (2, 6, 6), indexer_scores = tensor of shape (6, 3) with -inf for incomplete blocks, block_size = 2, selected_blocks = boolean tensor of shape (6, 3) with at most two blocks per query
Output: (tensor(0.074565, dtype=torch.float64), tensor(0.048984, dtype=torch.float64))
Input: teacher_attention = tensor of shape (1, 7, 7) with some zero weights, indexer_scores = tensor of shape (7, 2) with -inf for incomplete blocks, block_size = 3, selected_blocks = boolean tensor of shape (7, 2) with one block per query
Output: (tensor(0.32871, dtype=torch.float64), tensor(0.0, dtype=torch.float64))
Hint 1
Reshape the first nbr key columns to (T, n_b, r) and reduce the last axis with amax.
Hint 2
torch.isfinite marks complete blocks, and masked_fill before torch.log_softmax restricts the indexer to any block set.
Hint 3
Wrap each KL term in torch.where on a positive teacher value so zero terms never become nan.
Requirements
- Queries with no complete block are left out of both sums and of the count N.
- Terms with zero teacher probability contribute zero, so both losses are finite.
- Use the selected blocks exactly as given, without recomputing a top-k selection.
- Leave every input unchanged.
Constraints
- The teacher attention is a float32 or float64 tensor of shape (H, T, T) with H from 1 to 4 and T from 2 to 10.
- Every teacher row is causal, with zeros above the diagonal, nonnegative, and sums to one in every head.
- The block size r is an integer from 1 to T, so at least one query has a complete block.
- The indexer scores have shape (T, floor(T / r)) and the teacher's dtype. The score of block b for query i is finite and nonnegative when block b is complete for query i, and minus infinity otherwise.
- In the Run panel, minus infinity is written as the string "-inf" inside the score list.
- The selected blocks form a boolean tensor of shape (T, floor(T / r)). They mark only complete blocks, at least one for every query that has a complete block, and none for the other queries.
- For every query with a complete block, the pooled teacher mass over its complete blocks and over its selected blocks is positive.
All steps in Qwen4-Max
Accepts: array
Accepts: array
Accepts: integer
Accepts: array