Implement causal grouped-query attention without materializing repeated key or value heads. Queries have shape (batch, query heads, sequence, head width); keys and values use fewer or equal heads. Each contiguous group of G query heads shares one key/value head, where G is the query-head count divided by the key/value-head count.
kvhead(h)=⌊Gh⌋. Attention(Q,K,V)=softmax(DQKT+M)V.Here h is a zero-based query-head index, and floor selects its shared key/value head. Q is one query head, K and V are its assigned key and value heads, D is head width, and the superscript T transposes K. The mask M is zero for keys at or before the query position and negative infinity for later keys. Softmax normalizes over key positions.
Return a PyTorch tensor with the same shape, dtype, and device as q.
Input: q = [[[[1,0],[0,1]]]], k = [[[[1,0],[0,1]]]], v = [[[[1,2],[3,4]]]]
Output: [[[[1,2],[2.339523,3.339523]]]]
Explanation: The first token can attend only to itself. The second token attends to both available positions using causal scaled dot-product weights.
Input: q = [[[[1,0],[0,1]],[[0,1],[1,0]]]], k = [[[[1,1],[1,-1]]]], v = [[[[1,0],[0,2]]]]
Output: [[[[1,0],[0.80443,0.391141]],[[1,0],[0.5,1]]]]
Input: q = [[[[0,0]],[[1,0]],[[0,1]],[[1,1]]]], k = [[[[1,0]],[[0,1]]]], v = [[[[1,2]],[[3,4]]]]
Output: [[[[1,2]],[[1,2]],[[3,4]],[[3,4]]]]
Tensor.reshape can expose a query-head group axis.
torch.matmul broadcasts over leading dimensions.
torch.triu can construct the future-position mask.
Accepts: array
Accepts: array
Accepts: array
Implement causal grouped-query attention without materializing repeated key or value heads. Queries have shape (batch, query heads, sequence, head width); keys and values use fewer or equal heads. Each contiguous group of G query heads shares one key/value head, where G is the query-head count divided by the key/value-head count.
kvhead(h)=⌊Gh⌋. Attention(Q,K,V)=softmax(DQKT+M)V.Here h is a zero-based query-head index, and floor selects its shared key/value head. Q is one query head, K and V are its assigned key and value heads, D is head width, and the superscript T transposes K. The mask M is zero for keys at or before the query position and negative infinity for later keys. Softmax normalizes over key positions.
Return a PyTorch tensor with the same shape, dtype, and device as q.
Input: q = [[[[1,0],[0,1]]]], k = [[[[1,0],[0,1]]]], v = [[[[1,2],[3,4]]]]
Output: [[[[1,2],[2.339523,3.339523]]]]
Explanation: The first token can attend only to itself. The second token attends to both available positions using causal scaled dot-product weights.
Input: q = [[[[1,0],[0,1]],[[0,1],[1,0]]]], k = [[[[1,1],[1,-1]]]], v = [[[[1,0],[0,2]]]]
Output: [[[[1,0],[0.80443,0.391141]],[[1,0],[0.5,1]]]]
Input: q = [[[[0,0]],[[1,0]],[[0,1]],[[1,1]]]], k = [[[[1,0]],[[0,1]]]], v = [[[[1,2]],[[3,4]]]]
Output: [[[[1,2]],[[1,2]],[[3,4]],[[3,4]]]]
Tensor.reshape can expose a query-head group axis.
torch.matmul broadcasts over leading dimensions.
torch.triu can construct the future-position mask.
Accepts: array
Accepts: array
Accepts: array