Apply one momentum and spectral update to a parameter matrix. Let Wt be parameter, Gt gradient, Bt−1 previous_momentum, μ momentum_coefficient and η learning_rate. The updated momentum is:
Bt=μBt−1+Gt.For a matrix with m rows and n columns, let r=min(m,n). Its compact singular value decomposition (SVD) is:
Bt=UΣVT.Here, U has shape (m,r) and V has shape (n,r); their columns are singular vectors, and diagonal matrix Σ contains the singular values. Replacing those values with one defines the spectral update Ot:
Ot=UVT.The new parameter is:
Wt+1=Wt−ηOt.Return a Python dictionary with new_parameter, new_momentum and orthogonalized_update, each a PyTorch tensor with the parameter's shape, dtype and device.
Input: parameter = [[1,0],[0,1]], gradient = [[1,0],[0,2]], previous_momentum = [[0,0],[0,0]], momentum_coefficient = 0.9, learning_rate = 0.1
Output: {"new_parameter":[[0.9,0],[0,0.9]],"new_momentum":[[1,0],[0,2]],"orthogonalized_update":[[1,0],[0,1]]}
Explanation: Momentum equals the diagonal gradient, and replacing its positive singular values with one produces the identity update.
Input: parameter = [[0,0],[0,0],[0,0]], gradient = [[1,0],[0,1],[0,0]], previous_momentum = [[0,0],[0,0],[0,0]], momentum_coefficient = 0, learning_rate = 0.1
Output: {"new_parameter":[[-0.1,0],[0,-0.1],[0,0]],"new_momentum":[[1,0],[0,1],[0,0]],"orthogonalized_update":[[1,0],[0,1],[0,0]]}
Input: parameter = [[0,0,0],[0,0,0]], gradient = [[1,0,0],[0,1,0]], previous_momentum = [[0,0,0],[0,0,0]], momentum_coefficient = 0, learning_rate = 0.2
Output: {"new_parameter":[[-0.2,0,0],[0,-0.2,0]],"new_momentum":[[1,0,0],[0,1,0]],"orthogonalized_update":[[1,0,0],[0,1,0]]}
torch.linalg.svd provides singular vectors and singular values.
The @ operator multiplies two matrices.
Tensor.to can match another tensor’s dtype.
Accepts: array
Accepts: array
Accepts: array
Accepts: number
Accepts: number
Apply one momentum and spectral update to a parameter matrix. Let Wt be parameter, Gt gradient, Bt−1 previous_momentum, μ momentum_coefficient and η learning_rate. The updated momentum is:
Bt=μBt−1+Gt.For a matrix with m rows and n columns, let r=min(m,n). Its compact singular value decomposition (SVD) is:
Bt=UΣVT.Here, U has shape (m,r) and V has shape (n,r); their columns are singular vectors, and diagonal matrix Σ contains the singular values. Replacing those values with one defines the spectral update Ot:
Ot=UVT.The new parameter is:
Wt+1=Wt−ηOt.Return a Python dictionary with new_parameter, new_momentum and orthogonalized_update, each a PyTorch tensor with the parameter's shape, dtype and device.
Input: parameter = [[1,0],[0,1]], gradient = [[1,0],[0,2]], previous_momentum = [[0,0],[0,0]], momentum_coefficient = 0.9, learning_rate = 0.1
Output: {"new_parameter":[[0.9,0],[0,0.9]],"new_momentum":[[1,0],[0,2]],"orthogonalized_update":[[1,0],[0,1]]}
Explanation: Momentum equals the diagonal gradient, and replacing its positive singular values with one produces the identity update.
Input: parameter = [[0,0],[0,0],[0,0]], gradient = [[1,0],[0,1],[0,0]], previous_momentum = [[0,0],[0,0],[0,0]], momentum_coefficient = 0, learning_rate = 0.1
Output: {"new_parameter":[[-0.1,0],[0,-0.1],[0,0]],"new_momentum":[[1,0],[0,1],[0,0]],"orthogonalized_update":[[1,0],[0,1],[0,0]]}
Input: parameter = [[0,0,0],[0,0,0]], gradient = [[1,0,0],[0,1,0]], previous_momentum = [[0,0,0],[0,0,0]], momentum_coefficient = 0, learning_rate = 0.2
Output: {"new_parameter":[[-0.2,0,0],[0,-0.2,0]],"new_momentum":[[1,0,0],[0,1,0]],"orthogonalized_update":[[1,0,0],[0,1,0]]}
torch.linalg.svd provides singular vectors and singular values.
The @ operator multiplies two matrices.
Tensor.to can match another tensor’s dtype.
Accepts: array
Accepts: array
Accepts: array
Accepts: number
Accepts: number
