Perform one SGD update using gradients accumulated across microbatches of a linear model. The parameter vector w has shape (D). Each microbatch contains an input matrix with D columns and one target per row. The full-batch mean squared error is
L=N1m∑i=1∑Nm(Xm,iw−ym,i)2. wnew=w−η∇wL.Here D is the feature count, m selects a microbatch, i selects one of its examples, N is the total example count, and its subscript m denotes a microbatch's size. X is the input, y is the target, L is the loss, eta is lr, and the gradient is taken with respect to w. Each example contributes equally, even when microbatch sizes differ.
Return a Python dictionary with new_param holding the updated parameter tensor and full_grad holding the accumulated full-batch gradient. Both values must be PyTorch tensors of shape (D), preserving param's dtype and device.
Input: param = [1,-1], microbatch_inputs = [[[1,0],[0,1]],[[1,1]]], microbatch_targets = [[0,0],[1]], lr = 0.1
Output: {"new_param":[1,-0.866667],"full_grad":[0,-1.333333]}
Explanation: The three examples produce the full-batch gradient [0,-1.333333]. One SGD update with learning rate 0.1 gives [1,-0.866667].
Input: param = [0], microbatch_inputs = [[[1],[2]],[[3],[4]]], microbatch_targets = [[1,2],[3,4]], lr = 0.1
Output: {"new_param":[1.5],"full_grad":[-15]}
Input: param = [2], microbatch_inputs = [[[1],[2]]], microbatch_targets = [[2,4]], lr = 0.25
Output: {"new_param":[2],"full_grad":[0]}
Tensor.detach and Tensor.clone can isolate a fresh gradient-tracking parameter.
Tensor.backward accumulates into a leaf tensor's grad buffer.
Tensor.shape gives the example count of each microbatch.
Accepts: array
Accepts: array
Accepts: array
Accepts: number
Perform one SGD update using gradients accumulated across microbatches of a linear model. The parameter vector w has shape (D). Each microbatch contains an input matrix with D columns and one target per row. The full-batch mean squared error is
L=N1m∑i=1∑Nm(Xm,iw−ym,i)2. wnew=w−η∇wL.Here D is the feature count, m selects a microbatch, i selects one of its examples, N is the total example count, and its subscript m denotes a microbatch's size. X is the input, y is the target, L is the loss, eta is lr, and the gradient is taken with respect to w. Each example contributes equally, even when microbatch sizes differ.
Return a Python dictionary with new_param holding the updated parameter tensor and full_grad holding the accumulated full-batch gradient. Both values must be PyTorch tensors of shape (D), preserving param's dtype and device.
Input: param = [1,-1], microbatch_inputs = [[[1,0],[0,1]],[[1,1]]], microbatch_targets = [[0,0],[1]], lr = 0.1
Output: {"new_param":[1,-0.866667],"full_grad":[0,-1.333333]}
Explanation: The three examples produce the full-batch gradient [0,-1.333333]. One SGD update with learning rate 0.1 gives [1,-0.866667].
Input: param = [0], microbatch_inputs = [[[1],[2]],[[3],[4]]], microbatch_targets = [[1,2],[3,4]], lr = 0.1
Output: {"new_param":[1.5],"full_grad":[-15]}
Input: param = [2], microbatch_inputs = [[[1],[2]]], microbatch_targets = [[2,4]], lr = 0.25
Output: {"new_param":[2],"full_grad":[0]}
Tensor.detach and Tensor.clone can isolate a fresh gradient-tracking parameter.
Tensor.backward accumulates into a leaf tensor's grad buffer.
Tensor.shape gives the example count of each microbatch.
Accepts: array
Accepts: array
Accepts: array
Accepts: number
