TensorTonic
Study Plans
Study Plan
Causal Grouped-Query Attention
Language ModelsTransformers
Hard

Implement causal grouped-query attention without materializing repeated key or value heads. Queries have shape (batch, query heads, sequence, head width); keys and values use fewer or equal heads. Each contiguous group of G query heads shares one key/value head, where G is the query-head count divided by the key/value-head count.

kvhead⁡(h)=⌊hG⌋.\operatorname{kvhead}(h)=\left\lfloor\frac{h}{G}\right\rfloor.kvhead(h)=⌊Gh​⌋. Attention⁡(Q,K,V)=softmax⁡(QKTD+M)V.\operatorname{Attention}(Q,K,V)=\operatorname{softmax}\left(\frac{QK^{\mathsf T}}{\sqrt{D}}+M\right)V.Attention(Q,K,V)=softmax(D​QKT​+M)V.

Here h is a zero-based query-head index, and floor selects its shared key/value head. Q is one query head, K and V are its assigned key and value heads, D is head width, and the superscript T transposes K. The mask M is zero for keys at or before the query position and negative infinity for later keys. Softmax normalizes over key positions.

Return a PyTorch tensor with the same shape, dtype, and device as q.

Loading visualization...

Examples

Input: q = [[[[1,0],[0,1]]]], k = [[[[1,0],[0,1]]]], v = [[[[1,2],[3,4]]]]

Output: [[[[1,2],[2.339523,3.339523]]]]

Explanation: The first token can attend only to itself. The second token attends to both available positions using causal scaled dot-product weights.

Input: q = [[[[1,0],[0,1]],[[0,1],[1,0]]]], k = [[[[1,1],[1,-1]]]], v = [[[[1,0],[0,2]]]]

Output: [[[[1,0],[0.80443,0.391141]],[[1,0],[0.5,1]]]]

Input: q = [[[[0,0]],[[1,0]],[[0,1]],[[1,1]]]], k = [[[[1,0]],[[0,1]]]], v = [[[[1,2]],[[3,4]]]]

Output: [[[[1,2]],[[1,2]],[[3,4]],[[3,4]]]]

Hint 1

Tensor.reshape can expose a query-head group axis.

Hint 2

torch.matmul broadcasts over leading dimensions.

Hint 3

torch.triu can construct the future-position mask.

Requirements

  • Use PyTorch without allocating repeated key or value heads.
  • Compute scores, probabilities, and weighted values in float32, preserving q's dtype in the result.
  • Supported tensor dtypes are float32 and bfloat16; public cases and custom Run inputs use float32.
  • Leave q, k, and v unchanged.

Constraints

  • q, k, and v are rank-four PyTorch tensors in (batch, heads, sequence, head width) order; k and v have identical shapes.
  • All three tensors share batch size, sequence length, and head width.
  • All dimensions are positive, and the key/value head count divides the query head count exactly.
  • All tensors share a float32 or bfloat16 dtype and device and contain finite values.
Try Similar Problems
RoPE: Rotary Query and Key EmbeddingsMediumDecode Attention with a KV CacheHardAssign Multimodal RoPE CoordinatesMediumParameter-Matched SwiGLU BlockMediumPatchify a Batch of ImagesEasyNamed-Dimension Batched Attention ScoresEasyTransformer Training FLOP EstimatorEasyRMSNorm Forward PassEasyVision Transformer Patch EmbeddingHardImplement Speculative Decoding and Token VerificationHardTransformer Encoder BlockHardImplement PagedAttention Block AllocationHard

Accepts: array

Accepts: array

Accepts: array

You must run your code first.

Causal Grouped-Query Attention

Language ModelsTransformers
Hard

Implement causal grouped-query attention without materializing repeated key or value heads. Queries have shape (batch, query heads, sequence, head width); keys and values use fewer or equal heads. Each contiguous group of G query heads shares one key/value head, where G is the query-head count divided by the key/value-head count.

kvhead⁡(h)=⌊hG⌋.\operatorname{kvhead}(h)=\left\lfloor\frac{h}{G}\right\rfloor.kvhead(h)=⌊Gh​⌋. Attention⁡(Q,K,V)=softmax⁡(QKTD+M)V.\operatorname{Attention}(Q,K,V)=\operatorname{softmax}\left(\frac{QK^{\mathsf T}}{\sqrt{D}}+M\right)V.Attention(Q,K,V)=softmax(D​QKT​+M)V.

Here h is a zero-based query-head index, and floor selects its shared key/value head. Q is one query head, K and V are its assigned key and value heads, D is head width, and the superscript T transposes K. The mask M is zero for keys at or before the query position and negative infinity for later keys. Softmax normalizes over key positions.

Return a PyTorch tensor with the same shape, dtype, and device as q.

Loading visualization...

Examples

Input: q = [[[[1,0],[0,1]]]], k = [[[[1,0],[0,1]]]], v = [[[[1,2],[3,4]]]]

Output: [[[[1,2],[2.339523,3.339523]]]]

Explanation: The first token can attend only to itself. The second token attends to both available positions using causal scaled dot-product weights.

Input: q = [[[[1,0],[0,1]],[[0,1],[1,0]]]], k = [[[[1,1],[1,-1]]]], v = [[[[1,0],[0,2]]]]

Output: [[[[1,0],[0.80443,0.391141]],[[1,0],[0.5,1]]]]

Input: q = [[[[0,0]],[[1,0]],[[0,1]],[[1,1]]]], k = [[[[1,0]],[[0,1]]]], v = [[[[1,2]],[[3,4]]]]

Output: [[[[1,2]],[[1,2]],[[3,4]],[[3,4]]]]

Hint 1

Tensor.reshape can expose a query-head group axis.

Hint 2

torch.matmul broadcasts over leading dimensions.

Hint 3

torch.triu can construct the future-position mask.

Requirements

  • Use PyTorch without allocating repeated key or value heads.
  • Compute scores, probabilities, and weighted values in float32, preserving q's dtype in the result.
  • Supported tensor dtypes are float32 and bfloat16; public cases and custom Run inputs use float32.
  • Leave q, k, and v unchanged.

Constraints

  • q, k, and v are rank-four PyTorch tensors in (batch, heads, sequence, head width) order; k and v have identical shapes.
  • All three tensors share batch size, sequence length, and head width.
  • All dimensions are positive, and the key/value head count divides the query head count exactly.
  • All tensors share a float32 or bfloat16 dtype and device and contain finite values.
Try Similar Problems
RoPE: Rotary Query and Key EmbeddingsMediumDecode Attention with a KV CacheHardAssign Multimodal RoPE CoordinatesMediumParameter-Matched SwiGLU BlockMediumPatchify a Batch of ImagesEasyNamed-Dimension Batched Attention ScoresEasyTransformer Training FLOP EstimatorEasyRMSNorm Forward PassEasyVision Transformer Patch EmbeddingHardImplement Speculative Decoding and Token VerificationHardTransformer Encoder BlockHardImplement PagedAttention Block AllocationHard

Accepts: array

Accepts: array

Accepts: array

You must run your code first.