Implement a bias-free SwiGLU block with a hidden width selected to match a target parameter budget. The input's final dimension is d, the supplied weight pool has maximum hidden width H, and P is base_params.
h=min(H,max(1,⌊3dP+0.5⌋)). SiLU(z)=1+e−zz. y=(SiLU(xWg)⊙(xWv))Wo.Here h is the selected hidden width; floor rounds down, giving upward rounding at half-way budgets. The gate and value projections use the first h columns of w_g and w_v; the output projection uses the first h rows of w_o. These are the three W matrices in the equation. The circled-dot operator is elementwise multiplication, z is an activation value, e is Euler's number, and y is the block output. The three selected matrices contain 3dh parameters.
Return a Python dictionary with output as a PyTorch tensor matching x's shape, dtype, and device, hidden_width as the selected Python integer h, and parameter_count as the Python integer number of parameters used.
Input: x = [[1,2]], w_g = [[1,0,1],[0,1,1]], w_v = [[1,1,0],[1,0,1]], w_o = [[1,0],[0,1],[1,1]], base_params = 12
Output: {"output":[[2.193176,1.761594]],"hidden_width":2,"parameter_count":12}
Explanation: The target selects hidden width 2. The first two columns of the gate and value weights and the first two rows of the output weight form the budget-matched SwiGLU block.
Input: x = [[2]], w_g = [[1,0]], w_v = [[3,1]], w_o = [[2],[4]], base_params = 3
Output: {"output":[[21.13913]],"hidden_width":1,"parameter_count":3}
Input: x = [[1,-1]], w_g = [[1,0],[0,1]], w_v = [[1,0],[0,1]], w_o = [[1,0],[0,1]], base_params = 100
Output: {"output":[[0.731059,0.268941]],"hidden_width":2,"parameter_count":12}
math.floor avoids Python round's ties-to-even behavior.
A two-axis slice selects columns or rows without changing their order.
torch.nn.functional.silu provides the gate activation.
Accepts: array
Accepts: array
Accepts: array
Accepts: array
Accepts: number
Implement a bias-free SwiGLU block with a hidden width selected to match a target parameter budget. The input's final dimension is d, the supplied weight pool has maximum hidden width H, and P is base_params.
h=min(H,max(1,⌊3dP+0.5⌋)). SiLU(z)=1+e−zz. y=(SiLU(xWg)⊙(xWv))Wo.Here h is the selected hidden width; floor rounds down, giving upward rounding at half-way budgets. The gate and value projections use the first h columns of w_g and w_v; the output projection uses the first h rows of w_o. These are the three W matrices in the equation. The circled-dot operator is elementwise multiplication, z is an activation value, e is Euler's number, and y is the block output. The three selected matrices contain 3dh parameters.
Return a Python dictionary with output as a PyTorch tensor matching x's shape, dtype, and device, hidden_width as the selected Python integer h, and parameter_count as the Python integer number of parameters used.
Input: x = [[1,2]], w_g = [[1,0,1],[0,1,1]], w_v = [[1,1,0],[1,0,1]], w_o = [[1,0],[0,1],[1,1]], base_params = 12
Output: {"output":[[2.193176,1.761594]],"hidden_width":2,"parameter_count":12}
Explanation: The target selects hidden width 2. The first two columns of the gate and value weights and the first two rows of the output weight form the budget-matched SwiGLU block.
Input: x = [[2]], w_g = [[1,0]], w_v = [[3,1]], w_o = [[2],[4]], base_params = 3
Output: {"output":[[21.13913]],"hidden_width":1,"parameter_count":3}
Input: x = [[1,-1]], w_g = [[1,0],[0,1]], w_v = [[1,0],[0,1]], w_o = [[1,0],[0,1]], base_params = 100
Output: {"output":[[0.731059,0.268941]],"hidden_width":2,"parameter_count":12}
math.floor avoids Python round's ties-to-even behavior.
A two-axis slice selects columns or rows without changing their order.
torch.nn.functional.silu provides the gate activation.
Accepts: array
Accepts: array
Accepts: array
Accepts: array
Accepts: number
