Reconstruct one column-parallel linear layer from rank-ordered weight shards. This is the linear projection only, with no nonlinear activation. For rank r:
Yr=XWr+br.Here, X is the input tensor with shape (B,Din), Wr is its shard with shape (Din,Dr), and br is the optional bias of length Dr, broadcast across the batch. B is the batch size, Din is the input width, Dr is rank r's output width, and Yr is its local result. An absent bias contributes zero. Across R ranks, concatenate output features in rank order:
Y=concat(Y0,…,YR−1).Return a Python dictionary with local_activations, a rank-ordered list of PyTorch tensors shaped (B, D_r), and full_activation, a PyTorch tensor shaped (B, sum of shard widths). Both preserve the input dtype and device.
Input: x = [[1,2],[3,4]], weight_shards = [[[1],[0]],[[0],[1]]], bias_shards = [[0],[0]]
Output: {"local_activations":[[[1],[3]],[[2],[4]]],"full_activation":[[1,2],[3,4]]}
Explanation: The first shard selects the first feature and the second selects the second feature. Rank-order concatenation reconstructs the input.
Input: x = [[1,2]], weight_shards = [[[1,0],[0,1]],[[1],[1]]], bias_shards = [[0.5,-0.5],[1]]
Output: {"local_activations":[[[1.5,1.5]],[[4]]],"full_activation":[[1.5,1.5,4]]}
Input: x = [[2,-1]], weight_shards = [[[1,2],[3,4]]], bias_shards = [[0,0]]
Output: {"local_activations":[[[-1,0]]],"full_activation":[[-1,0]]}
The @ operator multiplies two matrices.
A one-dimensional bias broadcasts across batch rows.
torch.cat joins tensors along a selected dimension.
Accepts: array
Accepts: array
Accepts: array, any
Reconstruct one column-parallel linear layer from rank-ordered weight shards. This is the linear projection only, with no nonlinear activation. For rank r:
Yr=XWr+br.Here, X is the input tensor with shape (B,Din), Wr is its shard with shape (Din,Dr), and br is the optional bias of length Dr, broadcast across the batch. B is the batch size, Din is the input width, Dr is rank r's output width, and Yr is its local result. An absent bias contributes zero. Across R ranks, concatenate output features in rank order:
Y=concat(Y0,…,YR−1).Return a Python dictionary with local_activations, a rank-ordered list of PyTorch tensors shaped (B, D_r), and full_activation, a PyTorch tensor shaped (B, sum of shard widths). Both preserve the input dtype and device.
Input: x = [[1,2],[3,4]], weight_shards = [[[1],[0]],[[0],[1]]], bias_shards = [[0],[0]]
Output: {"local_activations":[[[1],[3]],[[2],[4]]],"full_activation":[[1,2],[3,4]]}
Explanation: The first shard selects the first feature and the second selects the second feature. Rank-order concatenation reconstructs the input.
Input: x = [[1,2]], weight_shards = [[[1,0],[0,1]],[[1],[1]]], bias_shards = [[0.5,-0.5],[1]]
Output: {"local_activations":[[[1.5,1.5]],[[4]]],"full_activation":[[1.5,1.5,4]]}
Input: x = [[2,-1]], weight_shards = [[[1,2],[3,4]]], bias_shards = [[0,0]]
Output: {"local_activations":[[[-1,0]]],"full_activation":[[-1,0]]}
The @ operator multiplies two matrices.
A one-dimensional bias broadcasts across batch rows.
torch.cat joins tensors along a selected dimension.
Accepts: array
Accepts: array
Accepts: array, any