TensorTonic
Study Plans
Study Plan
Reconstruct a Column-Parallel MLP Layer
Distributed ComputingNeural Networks
Medium

Reconstruct one column-parallel linear layer from rank-ordered weight shards. This is the linear projection only, with no nonlinear activation. For rank rrr:

Yr=XWr+br.Y_r=XW_r+b_r.Yr​=XWr​+br​.

Here, XXX is the input tensor with shape (B,Din)(B,D_{\mathrm{in}})(B,Din​), WrW_rWr​ is its shard with shape (Din,Dr)(D_{\mathrm{in}},D_r)(Din​,Dr​), and brb_rbr​ is the optional bias of length DrD_rDr​, broadcast across the batch. BBB is the batch size, DinD_{\mathrm{in}}Din​ is the input width, DrD_rDr​ is rank rrr's output width, and YrY_rYr​ is its local result. An absent bias contributes zero. Across RRR ranks, concatenate output features in rank order:

Y=concat⁡(Y0,…,YR−1).Y=\operatorname{concat}(Y_0,\ldots,Y_{R-1}).Y=concat(Y0​,…,YR−1​).

Return a Python dictionary with local_activations, a rank-ordered list of PyTorch tensors shaped (B, D_r), and full_activation, a PyTorch tensor shaped (B, sum of shard widths). Both preserve the input dtype and device.

Loading visualization...

Examples

Input: x = [[1,2],[3,4]], weight_shards = [[[1],[0]],[[0],[1]]], bias_shards = [[0],[0]]

Output: {"local_activations":[[[1],[3]],[[2],[4]]],"full_activation":[[1,2],[3,4]]}

Explanation: The first shard selects the first feature and the second selects the second feature. Rank-order concatenation reconstructs the input.

Input: x = [[1,2]], weight_shards = [[[1,0],[0,1]],[[1],[1]]], bias_shards = [[0.5,-0.5],[1]]

Output: {"local_activations":[[[1.5,1.5]],[[4]]],"full_activation":[[1.5,1.5,4]]}

Input: x = [[2,-1]], weight_shards = [[[1,2],[3,4]]], bias_shards = [[0,0]]

Output: {"local_activations":[[[-1,0]]],"full_activation":[[-1,0]]}

Hint 1

The @ operator multiplies two matrices.

Hint 2

A one-dimensional bias broadcasts across batch rows.

Hint 3

torch.cat joins tensors along a selected dimension.

Requirements

  • Use PyTorch and preserve input dtype and device.
  • Support unequal shard widths and optional biases without modifying inputs.
  • Preserve rank order along the output-feature axis.

Constraints

  • x is a rank-two floating-point PyTorch tensor with positive batch and input dimensions.
  • weight_shards is a nonempty list of tensors with shape (Din, Dout_r), where every Dout_r is positive.
  • bias_shards is None or a matching list of tensors with shape (Dout_r).
  • All tensors share dtype and device; supported dtypes are float32 and bfloat16.
  • Inputs contain finite values and may be noncontiguous.
Try Similar Problems
Synchronize Data-Parallel GradientsMediumSimulate Distributed CollectivesMediumCalculate Collective Effective BandwidthEasyParallel and Recurrent Linear AttentionMediumMamba 2 Gated State ScanMediumMeasure Pipeline Bubble UtilizationMediumTop-k MoE Router with Load StatisticsHardAccount for ZeRO Memory by StageEasyAutoencoder Forward PassMediumGradient AccumulationMediumDQN Loss with Target NetworkMediumWeight InitializationMedium

Accepts: array

Accepts: array

Accepts: array, any

You must run your code first.

Reconstruct a Column-Parallel MLP Layer

Distributed ComputingNeural Networks
Medium

Reconstruct one column-parallel linear layer from rank-ordered weight shards. This is the linear projection only, with no nonlinear activation. For rank rrr:

Yr=XWr+br.Y_r=XW_r+b_r.Yr​=XWr​+br​.

Here, XXX is the input tensor with shape (B,Din)(B,D_{\mathrm{in}})(B,Din​), WrW_rWr​ is its shard with shape (Din,Dr)(D_{\mathrm{in}},D_r)(Din​,Dr​), and brb_rbr​ is the optional bias of length DrD_rDr​, broadcast across the batch. BBB is the batch size, DinD_{\mathrm{in}}Din​ is the input width, DrD_rDr​ is rank rrr's output width, and YrY_rYr​ is its local result. An absent bias contributes zero. Across RRR ranks, concatenate output features in rank order:

Y=concat⁡(Y0,…,YR−1).Y=\operatorname{concat}(Y_0,\ldots,Y_{R-1}).Y=concat(Y0​,…,YR−1​).

Return a Python dictionary with local_activations, a rank-ordered list of PyTorch tensors shaped (B, D_r), and full_activation, a PyTorch tensor shaped (B, sum of shard widths). Both preserve the input dtype and device.

Loading visualization...

Examples

Input: x = [[1,2],[3,4]], weight_shards = [[[1],[0]],[[0],[1]]], bias_shards = [[0],[0]]

Output: {"local_activations":[[[1],[3]],[[2],[4]]],"full_activation":[[1,2],[3,4]]}

Explanation: The first shard selects the first feature and the second selects the second feature. Rank-order concatenation reconstructs the input.

Input: x = [[1,2]], weight_shards = [[[1,0],[0,1]],[[1],[1]]], bias_shards = [[0.5,-0.5],[1]]

Output: {"local_activations":[[[1.5,1.5]],[[4]]],"full_activation":[[1.5,1.5,4]]}

Input: x = [[2,-1]], weight_shards = [[[1,2],[3,4]]], bias_shards = [[0,0]]

Output: {"local_activations":[[[-1,0]]],"full_activation":[[-1,0]]}

Hint 1

The @ operator multiplies two matrices.

Hint 2

A one-dimensional bias broadcasts across batch rows.

Hint 3

torch.cat joins tensors along a selected dimension.

Requirements

  • Use PyTorch and preserve input dtype and device.
  • Support unequal shard widths and optional biases without modifying inputs.
  • Preserve rank order along the output-feature axis.

Constraints

  • x is a rank-two floating-point PyTorch tensor with positive batch and input dimensions.
  • weight_shards is a nonempty list of tensors with shape (Din, Dout_r), where every Dout_r is positive.
  • bias_shards is None or a matching list of tensors with shape (Dout_r).
  • All tensors share dtype and device; supported dtypes are float32 and bfloat16.
  • Inputs contain finite values and may be noncontiguous.
Try Similar Problems
Synchronize Data-Parallel GradientsMediumSimulate Distributed CollectivesMediumCalculate Collective Effective BandwidthEasyParallel and Recurrent Linear AttentionMediumMamba 2 Gated State ScanMediumMeasure Pipeline Bubble UtilizationMediumTop-k MoE Router with Load StatisticsHardAccount for ZeRO Memory by StageEasyAutoencoder Forward PassMediumGradient AccumulationMediumDQN Loss with Target NetworkMediumWeight InitializationMedium

Accepts: array

Accepts: array

Accepts: array, any

You must run your code first.