TensorTonic
Problems
Mixed Precision with Loss Scaling
JAX
Medium

Perform one gradient descent step of a linear model with bfloat16 compute, float32 master weights and a dynamic loss scale.

The master parameters in params are w of shape (D, K) and b of shape (K,), both float32. The input x has shape (B, D) and the target y has shape (B, K). Cast w, b and x to bfloat16 and compute the predictions in bfloat16, then cast the predictions to float32 and take the mean squared error against y over all B times K entries.

y^=xbf16 wbf16+bbf16,L=1BK∑i,j(y^ij−yij)2\hat y = x_{\text{bf16}} \, w_{\text{bf16}} + b_{\text{bf16}}, \qquad L = \frac{1}{BK} \sum_{i,j} \left( \hat y_{ij} - y_{ij} \right)^2y^​=xbf16​wbf16​+bbf16​,L=BK1​i,j∑​(y^​ij​−yij​)2

Differentiate the scaled loss S⋅LS \cdot LS⋅L with respect to the float32 master parameters, where SSS is loss_scale, and divide the resulting gradients by SSS. The gradients count as finite only when every entry of both unscaled gradients is finite. The returned loss is the unscaled loss itself, so it stays finite on a step where only the scaled loss overflows.

When the gradients are finite, update each parameter to p minus lr times its unscaled gradient and add 1 to good_steps. If the new count reaches growth_interval, double loss_scale and reset good_steps to 0.

When any gradient entry is non-finite, keep params unchanged, halve loss_scale without letting it fall below 1.0, and reset good_steps to 0.

Your function is called under jax.jit, with growth_interval passed as a static Python int, loss_scale and lr passed as traced float32 scalars of shape (), and good_steps passed as a traced int32 scalar of shape ().

Return a dict with keys loss (the unscaled float32 loss of shape ()), params (a dict with float32 w of shape (D, K) and float32 b of shape (K,)), loss_scale (float32, shape ()), good_steps (int32, shape ()) and grads_finite (bool, shape ()), all as JAX arrays.

Examples

Input: params = {"w":[[0.5],[-1.25]],"b":[0.1]}, x = [[1.0,2.0],[0.5,-1.0]], y = [[-1.9],[1.5]], loss_scale = 1024.0, good_steps = 0, lr = 0.1, growth_interval = 2

Output: {"loss":0.005159,"params":{"w":[[0.494775],[-1.240137]],"b":[0.089697]},"loss_scale":1024.0,"good_steps":1,"grads_finite":true}

Explanation: The gradients are finite, so the master weights move by lr times the unscaled gradients and good_steps rises from 0 to 1. The new count has not reached growth_interval = 2, so loss_scale stays at 1024.

Input: params = {"w":[[1.3],[0.7]],"b":[-0.2]}, x = [[0.9,1.1],[-0.4,2.2]], y = [[1.94],[0.81]], loss_scale = 512.0, good_steps = 2, lr = 0.05, growth_interval = 3

Output: {"loss":0.021242,"params":{"w":[[1.309521],[0.709766]],"b":[-0.19043]},"loss_scale":1024.0,"good_steps":0,"grads_finite":true}

Input: params = {"w":[[0.5],[-1.25]],"b":[0.1]}, x = [[1.0,2.0],[0.5,-1.0]], y = [[40.0],[-30.0]], loss_scale = 3e+38, good_steps = 0, lr = 0.1, growth_interval = 2

Output: {"loss":1377.068848,"params":{"w":[[0.5],[-1.25]],"b":[0.1]},"loss_scale":1.5e+38,"good_steps":0,"grads_finite":false}

Hint 1

jax.grad(f, has_aux=True) returns (grads, aux) when f returns (value, aux).

Hint 2

jnp.all(jnp.isfinite(g)) checks one leaf; combine leaves with jnp.all(jnp.stack([...])).

Hint 3

jnp.where(finite, new_value, old_value) skips a step without a Python if.

Requirements

  • Choose between the finite and non-finite results with traced operations such as jnp.where or lax.cond.
  • If the bfloat16 forward pass itself overflows, return the resulting inf or NaN loss as it is.

Constraints

  • x is float32 with shape (B, D) and y is float32 with shape (B, K), with B, D and K between 1 and 4.
  • loss_scale is a positive number no larger than 3.2e38, and lr is a positive number.
  • growth_interval is a positive integer, and good_steps is an integer from 0 to growth_interval minus 1.
  • Every input is finite, although some inputs are large enough to overflow during the step.
Try Similar Problems
Rematerialized MLP BlockMediumInt8 Matmul with Per-Channel ScalesMediumBlockwise Attention with Online SoftmaxHardPad and Mask to Bucket LengthMediumHessian-Vector ProductMediumPer-Example Gradient ClippingMediumStraight-Through Estimator with Custom VJPMediumTraced Learning Rate ScheduleMediumExponential Moving Average with ScanMediumSpectral Norm by Power IterationMediumBatched Attention with vmapMediumPairwise Distances with Nested vmapMedium

Accepts: object

Accepts: array

Accepts: array

Accepts: number

Accepts: integer

Accepts: number

Accepts: integer

You must run your code first.
Mixed Precision with Loss Scaling

Mixed Precision with Loss Scaling

JAX
Medium

Perform one gradient descent step of a linear model with bfloat16 compute, float32 master weights and a dynamic loss scale.

The master parameters in params are w of shape (D, K) and b of shape (K,), both float32. The input x has shape (B, D) and the target y has shape (B, K). Cast w, b and x to bfloat16 and compute the predictions in bfloat16, then cast the predictions to float32 and take the mean squared error against y over all B times K entries.

y^=xbf16 wbf16+bbf16,L=1BK∑i,j(y^ij−yij)2\hat y = x_{\text{bf16}} \, w_{\text{bf16}} + b_{\text{bf16}}, \qquad L = \frac{1}{BK} \sum_{i,j} \left( \hat y_{ij} - y_{ij} \right)^2y^​=xbf16​wbf16​+bbf16​,L=BK1​i,j∑​(y^​ij​−yij​)2

Differentiate the scaled loss S⋅LS \cdot LS⋅L with respect to the float32 master parameters, where SSS is loss_scale, and divide the resulting gradients by SSS. The gradients count as finite only when every entry of both unscaled gradients is finite. The returned loss is the unscaled loss itself, so it stays finite on a step where only the scaled loss overflows.

When the gradients are finite, update each parameter to p minus lr times its unscaled gradient and add 1 to good_steps. If the new count reaches growth_interval, double loss_scale and reset good_steps to 0.

When any gradient entry is non-finite, keep params unchanged, halve loss_scale without letting it fall below 1.0, and reset good_steps to 0.

Your function is called under jax.jit, with growth_interval passed as a static Python int, loss_scale and lr passed as traced float32 scalars of shape (), and good_steps passed as a traced int32 scalar of shape ().

Return a dict with keys loss (the unscaled float32 loss of shape ()), params (a dict with float32 w of shape (D, K) and float32 b of shape (K,)), loss_scale (float32, shape ()), good_steps (int32, shape ()) and grads_finite (bool, shape ()), all as JAX arrays.

Examples

Input: params = {"w":[[0.5],[-1.25]],"b":[0.1]}, x = [[1.0,2.0],[0.5,-1.0]], y = [[-1.9],[1.5]], loss_scale = 1024.0, good_steps = 0, lr = 0.1, growth_interval = 2

Output: {"loss":0.005159,"params":{"w":[[0.494775],[-1.240137]],"b":[0.089697]},"loss_scale":1024.0,"good_steps":1,"grads_finite":true}

Explanation: The gradients are finite, so the master weights move by lr times the unscaled gradients and good_steps rises from 0 to 1. The new count has not reached growth_interval = 2, so loss_scale stays at 1024.

Input: params = {"w":[[1.3],[0.7]],"b":[-0.2]}, x = [[0.9,1.1],[-0.4,2.2]], y = [[1.94],[0.81]], loss_scale = 512.0, good_steps = 2, lr = 0.05, growth_interval = 3

Output: {"loss":0.021242,"params":{"w":[[1.309521],[0.709766]],"b":[-0.19043]},"loss_scale":1024.0,"good_steps":0,"grads_finite":true}

Input: params = {"w":[[0.5],[-1.25]],"b":[0.1]}, x = [[1.0,2.0],[0.5,-1.0]], y = [[40.0],[-30.0]], loss_scale = 3e+38, good_steps = 0, lr = 0.1, growth_interval = 2

Output: {"loss":1377.068848,"params":{"w":[[0.5],[-1.25]],"b":[0.1]},"loss_scale":1.5e+38,"good_steps":0,"grads_finite":false}

Hint 1

jax.grad(f, has_aux=True) returns (grads, aux) when f returns (value, aux).

Hint 2

jnp.all(jnp.isfinite(g)) checks one leaf; combine leaves with jnp.all(jnp.stack([...])).

Hint 3

jnp.where(finite, new_value, old_value) skips a step without a Python if.

Requirements

  • Choose between the finite and non-finite results with traced operations such as jnp.where or lax.cond.
  • If the bfloat16 forward pass itself overflows, return the resulting inf or NaN loss as it is.

Constraints

  • x is float32 with shape (B, D) and y is float32 with shape (B, K), with B, D and K between 1 and 4.
  • loss_scale is a positive number no larger than 3.2e38, and lr is a positive number.
  • growth_interval is a positive integer, and good_steps is an integer from 0 to growth_interval minus 1.
  • Every input is finite, although some inputs are large enough to overflow during the step.
Try Similar Problems
Rematerialized MLP BlockMediumInt8 Matmul with Per-Channel ScalesMediumBlockwise Attention with Online SoftmaxHardPad and Mask to Bucket LengthMediumHessian-Vector ProductMediumPer-Example Gradient ClippingMediumStraight-Through Estimator with Custom VJPMediumTraced Learning Rate ScheduleMediumExponential Moving Average with ScanMediumSpectral Norm by Power IterationMediumBatched Attention with vmapMediumPairwise Distances with Nested vmapMedium

Accepts: object

Accepts: array

Accepts: array

Accepts: number

Accepts: integer

Accepts: number

Accepts: integer

You must run your code first.
Problems
Loading...
1 / 1