Perform one gradient descent step of a linear model with bfloat16 compute, float32 master weights and a dynamic loss scale.
The master parameters in params are w of shape (D, K) and b of shape (K,), both float32. The input x has shape (B, D) and the target y has shape (B, K). Cast w, b and x to bfloat16 and compute the predictions in bfloat16, then cast the predictions to float32 and take the mean squared error against y over all B times K entries.
y^=xbf16wbf16+bbf16,L=BK1i,j∑(y^ij−yij)2Differentiate the scaled loss S⋅L with respect to the float32 master parameters, where S is loss_scale, and divide the resulting gradients by S. The gradients count as finite only when every entry of both unscaled gradients is finite. The returned loss is the unscaled loss itself, so it stays finite on a step where only the scaled loss overflows.
When the gradients are finite, update each parameter to p minus lr times its unscaled gradient and add 1 to good_steps. If the new count reaches growth_interval, double loss_scale and reset good_steps to 0.
When any gradient entry is non-finite, keep params unchanged, halve loss_scale without letting it fall below 1.0, and reset good_steps to 0.
Your function is called under jax.jit, with growth_interval passed as a static Python int, loss_scale and lr passed as traced float32 scalars of shape (), and good_steps passed as a traced int32 scalar of shape ().
Return a dict with keys loss (the unscaled float32 loss of shape ()), params (a dict with float32 w of shape (D, K) and float32 b of shape (K,)), loss_scale (float32, shape ()), good_steps (int32, shape ()) and grads_finite (bool, shape ()), all as JAX arrays.
Input: params = {"w":[[0.5],[-1.25]],"b":[0.1]}, x = [[1.0,2.0],[0.5,-1.0]], y = [[-1.9],[1.5]], loss_scale = 1024.0, good_steps = 0, lr = 0.1, growth_interval = 2
Output: {"loss":0.005159,"params":{"w":[[0.494775],[-1.240137]],"b":[0.089697]},"loss_scale":1024.0,"good_steps":1,"grads_finite":true}
Explanation: The gradients are finite, so the master weights move by lr times the unscaled gradients and good_steps rises from 0 to 1. The new count has not reached growth_interval = 2, so loss_scale stays at 1024.
Input: params = {"w":[[1.3],[0.7]],"b":[-0.2]}, x = [[0.9,1.1],[-0.4,2.2]], y = [[1.94],[0.81]], loss_scale = 512.0, good_steps = 2, lr = 0.05, growth_interval = 3
Output: {"loss":0.021242,"params":{"w":[[1.309521],[0.709766]],"b":[-0.19043]},"loss_scale":1024.0,"good_steps":0,"grads_finite":true}
Input: params = {"w":[[0.5],[-1.25]],"b":[0.1]}, x = [[1.0,2.0],[0.5,-1.0]], y = [[40.0],[-30.0]], loss_scale = 3e+38, good_steps = 0, lr = 0.1, growth_interval = 2
Output: {"loss":1377.068848,"params":{"w":[[0.5],[-1.25]],"b":[0.1]},"loss_scale":1.5e+38,"good_steps":0,"grads_finite":false}
jax.grad(f, has_aux=True) returns (grads, aux) when f returns (value, aux).
jnp.all(jnp.isfinite(g)) checks one leaf; combine leaves with jnp.all(jnp.stack([...])).
jnp.where(finite, new_value, old_value) skips a step without a Python if.
Accepts: object
Accepts: array
Accepts: array
Accepts: number
Accepts: integer
Accepts: number
Accepts: integer
Perform one gradient descent step of a linear model with bfloat16 compute, float32 master weights and a dynamic loss scale.
The master parameters in params are w of shape (D, K) and b of shape (K,), both float32. The input x has shape (B, D) and the target y has shape (B, K). Cast w, b and x to bfloat16 and compute the predictions in bfloat16, then cast the predictions to float32 and take the mean squared error against y over all B times K entries.
y^=xbf16wbf16+bbf16,L=BK1i,j∑(y^ij−yij)2Differentiate the scaled loss S⋅L with respect to the float32 master parameters, where S is loss_scale, and divide the resulting gradients by S. The gradients count as finite only when every entry of both unscaled gradients is finite. The returned loss is the unscaled loss itself, so it stays finite on a step where only the scaled loss overflows.
When the gradients are finite, update each parameter to p minus lr times its unscaled gradient and add 1 to good_steps. If the new count reaches growth_interval, double loss_scale and reset good_steps to 0.
When any gradient entry is non-finite, keep params unchanged, halve loss_scale without letting it fall below 1.0, and reset good_steps to 0.
Your function is called under jax.jit, with growth_interval passed as a static Python int, loss_scale and lr passed as traced float32 scalars of shape (), and good_steps passed as a traced int32 scalar of shape ().
Return a dict with keys loss (the unscaled float32 loss of shape ()), params (a dict with float32 w of shape (D, K) and float32 b of shape (K,)), loss_scale (float32, shape ()), good_steps (int32, shape ()) and grads_finite (bool, shape ()), all as JAX arrays.
Input: params = {"w":[[0.5],[-1.25]],"b":[0.1]}, x = [[1.0,2.0],[0.5,-1.0]], y = [[-1.9],[1.5]], loss_scale = 1024.0, good_steps = 0, lr = 0.1, growth_interval = 2
Output: {"loss":0.005159,"params":{"w":[[0.494775],[-1.240137]],"b":[0.089697]},"loss_scale":1024.0,"good_steps":1,"grads_finite":true}
Explanation: The gradients are finite, so the master weights move by lr times the unscaled gradients and good_steps rises from 0 to 1. The new count has not reached growth_interval = 2, so loss_scale stays at 1024.
Input: params = {"w":[[1.3],[0.7]],"b":[-0.2]}, x = [[0.9,1.1],[-0.4,2.2]], y = [[1.94],[0.81]], loss_scale = 512.0, good_steps = 2, lr = 0.05, growth_interval = 3
Output: {"loss":0.021242,"params":{"w":[[1.309521],[0.709766]],"b":[-0.19043]},"loss_scale":1024.0,"good_steps":0,"grads_finite":true}
Input: params = {"w":[[0.5],[-1.25]],"b":[0.1]}, x = [[1.0,2.0],[0.5,-1.0]], y = [[40.0],[-30.0]], loss_scale = 3e+38, good_steps = 0, lr = 0.1, growth_interval = 2
Output: {"loss":1377.068848,"params":{"w":[[0.5],[-1.25]],"b":[0.1]},"loss_scale":1.5e+38,"good_steps":0,"grads_finite":false}
jax.grad(f, has_aux=True) returns (grads, aux) when f returns (value, aux).
jnp.all(jnp.isfinite(g)) checks one leaf; combine leaves with jnp.all(jnp.stack([...])).
jnp.where(finite, new_value, old_value) skips a step without a Python if.
Accepts: object
Accepts: array
Accepts: array
Accepts: number
Accepts: integer
Accepts: number
Accepts: integer