TensorTonic
Problems
Straight-Through Estimator with Custom VJP
JAX
Medium

Build a fake quantizer with jax.custom_vjp whose forward pass rounds values onto an integer grid and whose backward pass is a clipped straight-through estimator, then return the quantized values and the gradient that flows back through it.

With the step size scale and the integer range from qmin to qmax, the forward pass maps every entry of x to

q(x)=scale⋅clip⁡(round⁡(x/scale), qmin, qmax)q(x) = \text{scale} \cdot \operatorname{clip}\big(\operatorname{round}(x / \text{scale}),\ \text{qmin},\ \text{qmax}\big)q(x)=scale⋅clip(round(x/scale), qmin, qmax)

where round sends halves to the nearest even integer, as jnp.round does, so 2.5 becomes 2 and 3.5 becomes 4. The true derivative of q is zero almost everywhere, so the backward rule replaces it. Given an upstream gradient c with the same shape as x, called the cotangent, the backward rule returns c⊙mc \odot mc⊙m, where ⊙\odot⊙ is elementwise multiplication and the mask m is 1 for entries with qmin≤x/scale≤qmax\text{qmin} \le x / \text{scale} \le \text{qmax}qmin≤x/scale≤qmax (both ends included) and 0 for every other entry.

Return a dict with two keys, values holding q(x) and grads holding the gradient for x that the cotangent produces through your custom rule, each a float32 JAX array with the same shape as x.

Examples

Input: x = [0.3,1.25,-0.75,2.6,-3.0], scale = 0.5, qmin = -4, qmax = 3, cotangent = [1.0,1.0,1.0,1.0,1.0]

Output: {"values":[0.5,1.0,-1.0,1.5,-2.0],"grads":[1.0,1.0,1.0,0.0,0.0]}

Explanation: Dividing by 0.5 gives 0.6, 2.5, -1.5, 5.2 and -6. Rounding halves to even gives 1, 2, -2, 5 and -6, and clipping to [-4, 3] turns the last two into 3 and -4. The first three entries are inside the range, so their cotangent passes through, and the two clipped entries get zero.

Input: x = [[0.1,0.6],[3.9,-0.2]], scale = 0.25, qmin = 0, qmax = 15, cotangent = [[2.0,-1.0],[0.5,3.0]]

Output: {"values":[[0.0,0.5],[3.75,0.0]],"grads":[[2.0,-1.0],[0.0,0.0]]}

Input: x = [2.0,2.5,-2.0,-2.25,0.5], scale = 1.0, qmin = -2, qmax = 2, cotangent = [1.0,2.0,3.0,4.0,5.0]

Output: {"values":[2.0,2.0,-2.0,-2.0,0.0],"grads":[1.0,0.0,3.0,0.0,5.0]}

Hint 1

f = jax.custom_vjp(f); f.defvjp(f_fwd, f_bwd) attaches your own backward rule.

Hint 2

f_fwd returns (output, residuals) and f_bwd(residuals, g) returns a tuple like (grad_x,).

Hint 3

y, pullback = jax.vjp(f, x); (grad_x,) = pullback(cotangent).

Requirements

  • Define the quantizer as a function of x wrapped with jax.custom_vjp, attach a forward rule and a backward rule with defvjp, and compute grads by differentiating through it with jax.vjp or jax.grad, so that your backward rule produces them. Solutions whose backward rule never runs, or whose traced computation contains no custom_vjp call, are rejected, including versions built on jax.lax.stop_gradient.
  • Your function is also traced by jax.make_jaxpr with x and cotangent as traced arrays and scale, qmin and qmax as Python numbers, so never turn traced values into Python numbers or NumPy arrays.

Constraints

  • x is a float32 JAX array with 1 or 2 dimensions and at most 6 entries.
  • scale is a positive Python float.
  • qmin and qmax are Python ints with qmin < qmax.
  • cotangent is a float32 JAX array with the same shape as x.
Try Similar Problems
Hessian-Vector ProductMediumPer-Example Gradient ClippingMediumValue and Grad of a LossEasyPad and Mask to Bucket LengthMediumTraced Learning Rate ScheduleMediumExponential Moving Average with ScanMediumSpectral Norm by Power IterationMediumBatched Attention with vmapMediumPairwise Distances with Nested vmapMediumGlobal Norm Gradient ClippingMediumAdamW from ScratchMediumMixed Precision with Loss ScalingMedium

Accepts: array

Accepts: number

Accepts: integer

Accepts: integer

Accepts: array

You must run your code first.
Straight-Through Estimator with Custom VJP

Straight-Through Estimator with Custom VJP

JAX
Medium

Build a fake quantizer with jax.custom_vjp whose forward pass rounds values onto an integer grid and whose backward pass is a clipped straight-through estimator, then return the quantized values and the gradient that flows back through it.

With the step size scale and the integer range from qmin to qmax, the forward pass maps every entry of x to

q(x)=scale⋅clip⁡(round⁡(x/scale), qmin, qmax)q(x) = \text{scale} \cdot \operatorname{clip}\big(\operatorname{round}(x / \text{scale}),\ \text{qmin},\ \text{qmax}\big)q(x)=scale⋅clip(round(x/scale), qmin, qmax)

where round sends halves to the nearest even integer, as jnp.round does, so 2.5 becomes 2 and 3.5 becomes 4. The true derivative of q is zero almost everywhere, so the backward rule replaces it. Given an upstream gradient c with the same shape as x, called the cotangent, the backward rule returns c⊙mc \odot mc⊙m, where ⊙\odot⊙ is elementwise multiplication and the mask m is 1 for entries with qmin≤x/scale≤qmax\text{qmin} \le x / \text{scale} \le \text{qmax}qmin≤x/scale≤qmax (both ends included) and 0 for every other entry.

Return a dict with two keys, values holding q(x) and grads holding the gradient for x that the cotangent produces through your custom rule, each a float32 JAX array with the same shape as x.

Examples

Input: x = [0.3,1.25,-0.75,2.6,-3.0], scale = 0.5, qmin = -4, qmax = 3, cotangent = [1.0,1.0,1.0,1.0,1.0]

Output: {"values":[0.5,1.0,-1.0,1.5,-2.0],"grads":[1.0,1.0,1.0,0.0,0.0]}

Explanation: Dividing by 0.5 gives 0.6, 2.5, -1.5, 5.2 and -6. Rounding halves to even gives 1, 2, -2, 5 and -6, and clipping to [-4, 3] turns the last two into 3 and -4. The first three entries are inside the range, so their cotangent passes through, and the two clipped entries get zero.

Input: x = [[0.1,0.6],[3.9,-0.2]], scale = 0.25, qmin = 0, qmax = 15, cotangent = [[2.0,-1.0],[0.5,3.0]]

Output: {"values":[[0.0,0.5],[3.75,0.0]],"grads":[[2.0,-1.0],[0.0,0.0]]}

Input: x = [2.0,2.5,-2.0,-2.25,0.5], scale = 1.0, qmin = -2, qmax = 2, cotangent = [1.0,2.0,3.0,4.0,5.0]

Output: {"values":[2.0,2.0,-2.0,-2.0,0.0],"grads":[1.0,0.0,3.0,0.0,5.0]}

Hint 1

f = jax.custom_vjp(f); f.defvjp(f_fwd, f_bwd) attaches your own backward rule.

Hint 2

f_fwd returns (output, residuals) and f_bwd(residuals, g) returns a tuple like (grad_x,).

Hint 3

y, pullback = jax.vjp(f, x); (grad_x,) = pullback(cotangent).

Requirements

  • Define the quantizer as a function of x wrapped with jax.custom_vjp, attach a forward rule and a backward rule with defvjp, and compute grads by differentiating through it with jax.vjp or jax.grad, so that your backward rule produces them. Solutions whose backward rule never runs, or whose traced computation contains no custom_vjp call, are rejected, including versions built on jax.lax.stop_gradient.
  • Your function is also traced by jax.make_jaxpr with x and cotangent as traced arrays and scale, qmin and qmax as Python numbers, so never turn traced values into Python numbers or NumPy arrays.

Constraints

  • x is a float32 JAX array with 1 or 2 dimensions and at most 6 entries.
  • scale is a positive Python float.
  • qmin and qmax are Python ints with qmin < qmax.
  • cotangent is a float32 JAX array with the same shape as x.
Try Similar Problems
Hessian-Vector ProductMediumPer-Example Gradient ClippingMediumValue and Grad of a LossEasyPad and Mask to Bucket LengthMediumTraced Learning Rate ScheduleMediumExponential Moving Average with ScanMediumSpectral Norm by Power IterationMediumBatched Attention with vmapMediumPairwise Distances with Nested vmapMediumGlobal Norm Gradient ClippingMediumAdamW from ScratchMediumMixed Precision with Loss ScalingMedium

Accepts: array

Accepts: number

Accepts: integer

Accepts: integer

Accepts: array

You must run your code first.
Problems
Loading...
1 / 1