Measure how activations and their one-step changes vary with model width under maximal-update parameterization (muP). activation_rms contains activation root-mean-square values; activation_change_rms contains the RMS change after one update. For each series, fit an unweighted least-squares line using natural logarithms:
logyi=a+blogwi.Here, i identifies a measurement, wi is its model width, yi is the supplied RMS value, a is the log-space intercept and b is the slope. The maximum-to-minimum ratio is:
R=miniyimaxiyi.Classify a series as width-invariant when its fitted slope satisfies:
∣b∣≤τ.Here, R measures the spread of the original values and τ is slope_tolerance. Return a Python dictionary with activation_slope, activation_intercept, change_slope, change_intercept, activation_ratio and change_ratio as Python floats, and activation_invariant and change_invariant as Python Booleans.
Input: widths = [128,256,512], activation_rms = [1,1,1], activation_change_rms = [0.5,0.5,0.5], slope_tolerance = 0.05
Output: {"activation_slope":0,"activation_intercept":0,"change_slope":0,"change_intercept":-0.693147,"activation_ratio":1,"change_ratio":1,"activation_invariant":true,"change_invariant":true}
Explanation: Constant RMS values have zero log-log slope and therefore satisfy the tolerance.
Input: widths = [1,2,4], activation_rms = [1,2,4], activation_change_rms = [1,1,1], slope_tolerance = 0.1
Output: {"activation_slope":1,"activation_intercept":0,"change_slope":0,"change_intercept":0,"activation_ratio":4,"change_ratio":1,"activation_invariant":false,"change_invariant":true}
Input: widths = [64,128,256,512], activation_rms = [1,1.01,0.99,1.02], activation_change_rms = [0.2,0.199,0.201,0.2], slope_tolerance = 0.05
Output: {"activation_slope":0.005685,"activation_intercept":-0.02463,"change_slope":0.001443,"change_intercept":-1.616944,"activation_ratio":1.030303,"change_ratio":1.01005,"activation_invariant":true,"change_invariant":true}
torch.log computes natural logarithms elementwise.
torch.linalg.lstsq returns fitted coefficients in design-column order.
Tensor.item converts a one-element tensor to a Python scalar.
Accepts: array
Accepts: array
Accepts: array
Accepts: number
Measure how activations and their one-step changes vary with model width under maximal-update parameterization (muP). activation_rms contains activation root-mean-square values; activation_change_rms contains the RMS change after one update. For each series, fit an unweighted least-squares line using natural logarithms:
logyi=a+blogwi.Here, i identifies a measurement, wi is its model width, yi is the supplied RMS value, a is the log-space intercept and b is the slope. The maximum-to-minimum ratio is:
R=miniyimaxiyi.Classify a series as width-invariant when its fitted slope satisfies:
∣b∣≤τ.Here, R measures the spread of the original values and τ is slope_tolerance. Return a Python dictionary with activation_slope, activation_intercept, change_slope, change_intercept, activation_ratio and change_ratio as Python floats, and activation_invariant and change_invariant as Python Booleans.
Input: widths = [128,256,512], activation_rms = [1,1,1], activation_change_rms = [0.5,0.5,0.5], slope_tolerance = 0.05
Output: {"activation_slope":0,"activation_intercept":0,"change_slope":0,"change_intercept":-0.693147,"activation_ratio":1,"change_ratio":1,"activation_invariant":true,"change_invariant":true}
Explanation: Constant RMS values have zero log-log slope and therefore satisfy the tolerance.
Input: widths = [1,2,4], activation_rms = [1,2,4], activation_change_rms = [1,1,1], slope_tolerance = 0.1
Output: {"activation_slope":1,"activation_intercept":0,"change_slope":0,"change_intercept":0,"activation_ratio":4,"change_ratio":1,"activation_invariant":false,"change_invariant":true}
Input: widths = [64,128,256,512], activation_rms = [1,1.01,0.99,1.02], activation_change_rms = [0.2,0.199,0.201,0.2], slope_tolerance = 0.05
Output: {"activation_slope":0.005685,"activation_intercept":-0.02463,"change_slope":0.001443,"change_intercept":-1.616944,"activation_ratio":1.030303,"change_ratio":1.01005,"activation_invariant":true,"change_invariant":true}
torch.log computes natural logarithms elementwise.
torch.linalg.lstsq returns fitted coefficients in design-column order.
Tensor.item converts a one-element tensor to a Python scalar.
Accepts: array
Accepts: array
Accepts: array
Accepts: number
