Problems
Loading...
Step 7 of 7
QSA Indexer Distillation
Medium

Qwen Sparse Attention (QSA) trains its lightweight indexer by distilling the model's dense attention into block scores. Compute both distillation losses from the teacher's attention probabilities, the indexer's block scores, and the blocks QSA selected for each query.

The teacher has HH heads of causal attention probabilities P(h)P^{(h)}, each of shape (T,T)(T, T), where TT is the sequence length and row ii is the distribution of query ii over keys 00 to ii. Sum the heads and L1-normalize each row:

aij=∑hPij(h)∑j′∑hPij′(h)a_{ij}=\frac{\sum_{h}P^{(h)}_{ij}}{\sum_{j'}\sum_{h}P^{(h)}_{ij'}}

Keys form nb=⌊T/r⌋n_b=\lfloor T/r\rfloor blocks of size rr, and block bb covers keys pb=brp_b=br to pb+r−1p_b+r-1. Block bb is complete for query ii when pb+r−1≤ip_b+r-1\le i, and Ci\mathcal C_i is the set of complete blocks of query ii. The indexer score IibI_{ib} is finite exactly for the blocks in Ci\mathcal C_i and minus infinity for every other block. Over complete blocks only, max-pool the teacher row and L1-normalize it (Equation 17):

aˉib=max⁡0≤u<rai,pb+u,b∈Ci\bar a_{ib}=\max_{0\le u<r}a_{i,p_b+u},\qquad b\in\mathcal C_i a^ib=aˉib∑b′∈Ciaˉib′\hat a_{ib}=\frac{\bar a_{ib}}{\sum_{b'\in\mathcal C_i}\bar a_{ib'}}

Teacher mass on keys outside complete blocks, which covers the current unfinished block and any trailing keys after the last full block, is dropped by this normalization.

For a set of blocks K⊆Ci\mathcal K\subseteq\mathcal C_i, renormalize the teacher target and the indexer softmax over K\mathcal K:

a~ib=a^ib∑b′∈Ka^ib′,q~ib=eIib∑b′∈KeIib′,b∈K\tilde a_{ib}=\frac{\hat a_{ib}}{\sum_{b'\in\mathcal K}\hat a_{ib'}},\qquad \tilde q_{ib}=\frac{e^{I_{ib}}}{\sum_{b'\in\mathcal K}e^{I_{ib'}}},\qquad b\in\mathcal K Di(K)=∑b∈Ka~iblog⁡a~ibq~ibD_i(\mathcal K)=\sum_{b\in\mathcal K}\tilde a_{ib}\log\frac{\tilde a_{ib}}{\tilde q_{ib}}

This is the KL divergence with the teacher distribution first, and a term with a~ib=0\tilde a_{ib}=0 contributes zero. Let Q\mathcal Q be the set of queries with at least one complete block, NN the number of such queries, and Bi⊆Ci\mathcal B_i\subseteq\mathcal C_i the selected blocks of query ii. The stage 1 loss (Equation 18) and the stage 2 loss (Equation 20) are

L1=1N∑i∈QDi(Ci)\mathcal L_1=\frac{1}{N}\sum_{i\in\mathcal Q}D_i(\mathcal C_i) L2=1N∑i∈QDi(Bi)\mathcal L_2=\frac{1}{N}\sum_{i\in\mathcal Q}D_i(\mathcal B_i)

Return a tuple of exactly two 0-dimensional tensors, the stage 1 loss followed by the stage 2 loss, each with the dtype and device of the teacher attention.

Loading visualization...

Examples

Input: teacher_attention = [[[1, 0, 0, 0], [0.1576, 0.8424, 0, 0], [0.733, 0.0601, 0.2069, 0], [0.0253, 0.3941, 0.5189, 0.0617]]], indexer_scores = [[-inf, -inf], [0, -inf], [1.1254, -inf], [0, 0]], block_size = 2, selected_blocks = [[False, False], [True, False], [True, False], [True, False]]

Output: (tensor(0.003124, dtype=torch.float64), tensor(0.0, dtype=torch.float64))

Explanation: Query 0 has no complete block and is left out, so N is 3. Queries 1 and 2 see only block 0, so both distributions put all their weight on it and their KL is 0. Query 3 pools its row to 0.3941 and 0.5189, which normalize to 0.4317 and 0.5683, while its equal scores give the indexer 0.5 and 0.5, a KL of 0.009372. Dividing by 3 gives 0.003124. Every query keeps a single selected block, so the stage 2 loss is 0.

Input: teacher_attention = tensor of shape (2, 6, 6), indexer_scores = tensor of shape (6, 3) with -inf for incomplete blocks, block_size = 2, selected_blocks = boolean tensor of shape (6, 3) with at most two blocks per query

Output: (tensor(0.074565, dtype=torch.float64), tensor(0.048984, dtype=torch.float64))

Input: teacher_attention = tensor of shape (1, 7, 7) with some zero weights, indexer_scores = tensor of shape (7, 2) with -inf for incomplete blocks, block_size = 3, selected_blocks = boolean tensor of shape (7, 2) with one block per query

Output: (tensor(0.32871, dtype=torch.float64), tensor(0.0, dtype=torch.float64))

Hint 1

Reshape the first nbrn_b r key columns to (T, n_b, r)\texttt{(T, n\_b, r)} and reduce the last axis with amax\texttt{amax}.

Hint 2

torch.isfinite\texttt{torch.isfinite} marks complete blocks, and masked_fill\texttt{masked\_fill} before torch.log_softmax\texttt{torch.log\_softmax} restricts the indexer to any block set.

Hint 3

Wrap each KL term in torch.where\texttt{torch.where} on a positive teacher value so zero terms never become nan\texttt{nan}.

Requirements

  • Queries with no complete block are left out of both sums and of the count N.
  • Terms with zero teacher probability contribute zero, so both losses are finite.
  • Use the selected blocks exactly as given, without recomputing a top-k selection.
  • Leave every input unchanged.

Constraints

  • The teacher attention is a float32 or float64 tensor of shape (H, T, T) with H from 1 to 4 and T from 2 to 10.
  • Every teacher row is causal, with zeros above the diagonal, nonnegative, and sums to one in every head.
  • The block size r is an integer from 1 to T, so at least one query has a complete block.
  • The indexer scores have shape (T, floor(T / r)) and the teacher's dtype. The score of block b for query i is finite and nonnegative when block b is complete for query i, and minus infinity otherwise.
  • In the Run panel, minus infinity is written as the string "-inf" inside the score list.
  • The selected blocks form a boolean tensor of shape (T, floor(T / r)). They mark only complete blocks, at least one for every query that has a complete block, and none for the other queries.
  • For every query with a complete block, the pooled teacher mass over its complete blocks and over its selected blocks is positive.