Place an array on a grid of 8 CPU devices with a named sharding, then report which block of the array each device holds.
A device mesh arranges the 8 devices into a grid of shape mesh_shape, for example (2, 4), and gives each grid axis the matching name from axis_names. Build the mesh so that the devices, taken in order of their ids, fill the grid in row-major order. Device d then sits at the grid position whose row-major flat index is d, so in a (2, 4) mesh device 5 sits in row 1 and column 1.
spec is a PartitionSpec with at most one entry per dimension of x, read from the first dimension onwards. An entry of None keeps that dimension whole on every device. An entry that names a mesh axis cuts the dimension into as many equal blocks as that axis has devices. An entry that is a tuple of names cuts it into as many blocks as all of those axes have together, and the blocks are numbered in row-major order over those axes with the first name changing slowest. Dimensions after the last entry stay whole, and devices that differ only along a mesh axis that spec never names hold identical copies. The examples write spec as a list, but the function receives a PartitionSpec such as PartitionSpec("data", None), and a nested list entry such as ["model", "data"] arrives as a tuple of names.
Return a dict with three keys. array is x placed on the mesh with a NamedSharding built from the mesh and spec, a float32 JAX array with the shape and values of x. shard_shape is an int32 JAX array of shape (8, n), where n is the number of dimensions of x, and its row d is the shape of the block that device d holds. shard_start is an int32 JAX array of shape (8, n), and its row d gives, for every dimension, the index at which device d's block begins.
Input: x = [[0.0,1.0,2.0,3.0],[4.0,5.0,6.0,7.0],[8.0,9.0,10.0,11.0],[12.0,13.0,14.0,15.0]], mesh_shape = [2,4], axis_names = ["data","model"], spec = ["data",null]
Output: {"array":[[0.0,1.0,2.0,3.0],[4.0,5.0,6.0,7.0],[8.0,9.0,10.0,11.0],[12.0,13.0,14.0,15.0]],"shard_shape":[[2,4],[2,4],[2,4],[2,4],[2,4],[2,4],[2,4],[2,4]],"shard_start":[[0,0],[0,0],[0,0],[0,0],[2,0],[2,0],[2,0],[2,0]]}
Explanation: The mesh puts devices 0 to 3 in its first row and devices 4 to 7 in its second row. The spec splits the 4 rows of x across the data axis, so devices 0 to 3 hold rows 0 and 1 of x and devices 4 to 7 hold rows 2 and 3. The model axis is never named, so the 4 devices in each mesh row hold identical copies of the full width of 4 columns.
Input: x = [[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0],[8.0,9.0,10.0,11.0,12.0,13.0,14.0,15.0],[16.0,17.0,18.0,19.0,20.0,21.0,22.0,23.0],[24.0,25.0,26.0,27.0,28.0,29.0,30.0,31.0]], mesh_shape = [2,4], axis_names = ["data","model"], spec = ["data","model"]
Output: {"array":[[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0],[8.0,9.0,10.0,11.0,12.0,13.0,14.0,15.0],[16.0,17.0,18.0,19.0,20.0,21.0,22.0,23.0],[24.0,25.0,26.0,27.0,28.0,29.0,30.0,31.0]],"shard_shape":[[2,2],[2,2],[2,2],[2,2],[2,2],[2,2],[2,2],[2,2]],"shard_start":[[0,0],[0,2],[0,4],[0,6],[2,0],[2,2],[2,4],[2,6]]}
Input: x = [0.0,0.5,1.0,1.5,2.0,2.5,3.0,3.5], mesh_shape = [2,4], axis_names = ["data","model"], spec = [["model","data"]]
Output: {"array":[0.0,0.5,1.0,1.5,2.0,2.5,3.0,3.5],"shard_shape":[[1],[1],[1],[1],[1],[1],[1],[1]],"shard_start":[[0],[2],[4],[6],[1],[3],[5],[7]]}
mesh = jax.make_mesh(mesh_shape, axis_names); sharding = NamedSharding(mesh, spec).
jax.device_put(x, sharding) places the array.
for s in y.addressable_shards: s.device.id, s.index, s.data; slice.indices(size) fills None bounds.
Accepts: array
Accepts: array
Accepts: array
Accepts: array
Place an array on a grid of 8 CPU devices with a named sharding, then report which block of the array each device holds.
A device mesh arranges the 8 devices into a grid of shape mesh_shape, for example (2, 4), and gives each grid axis the matching name from axis_names. Build the mesh so that the devices, taken in order of their ids, fill the grid in row-major order. Device d then sits at the grid position whose row-major flat index is d, so in a (2, 4) mesh device 5 sits in row 1 and column 1.
spec is a PartitionSpec with at most one entry per dimension of x, read from the first dimension onwards. An entry of None keeps that dimension whole on every device. An entry that names a mesh axis cuts the dimension into as many equal blocks as that axis has devices. An entry that is a tuple of names cuts it into as many blocks as all of those axes have together, and the blocks are numbered in row-major order over those axes with the first name changing slowest. Dimensions after the last entry stay whole, and devices that differ only along a mesh axis that spec never names hold identical copies. The examples write spec as a list, but the function receives a PartitionSpec such as PartitionSpec("data", None), and a nested list entry such as ["model", "data"] arrives as a tuple of names.
Return a dict with three keys. array is x placed on the mesh with a NamedSharding built from the mesh and spec, a float32 JAX array with the shape and values of x. shard_shape is an int32 JAX array of shape (8, n), where n is the number of dimensions of x, and its row d is the shape of the block that device d holds. shard_start is an int32 JAX array of shape (8, n), and its row d gives, for every dimension, the index at which device d's block begins.
Input: x = [[0.0,1.0,2.0,3.0],[4.0,5.0,6.0,7.0],[8.0,9.0,10.0,11.0],[12.0,13.0,14.0,15.0]], mesh_shape = [2,4], axis_names = ["data","model"], spec = ["data",null]
Output: {"array":[[0.0,1.0,2.0,3.0],[4.0,5.0,6.0,7.0],[8.0,9.0,10.0,11.0],[12.0,13.0,14.0,15.0]],"shard_shape":[[2,4],[2,4],[2,4],[2,4],[2,4],[2,4],[2,4],[2,4]],"shard_start":[[0,0],[0,0],[0,0],[0,0],[2,0],[2,0],[2,0],[2,0]]}
Explanation: The mesh puts devices 0 to 3 in its first row and devices 4 to 7 in its second row. The spec splits the 4 rows of x across the data axis, so devices 0 to 3 hold rows 0 and 1 of x and devices 4 to 7 hold rows 2 and 3. The model axis is never named, so the 4 devices in each mesh row hold identical copies of the full width of 4 columns.
Input: x = [[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0],[8.0,9.0,10.0,11.0,12.0,13.0,14.0,15.0],[16.0,17.0,18.0,19.0,20.0,21.0,22.0,23.0],[24.0,25.0,26.0,27.0,28.0,29.0,30.0,31.0]], mesh_shape = [2,4], axis_names = ["data","model"], spec = ["data","model"]
Output: {"array":[[0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0],[8.0,9.0,10.0,11.0,12.0,13.0,14.0,15.0],[16.0,17.0,18.0,19.0,20.0,21.0,22.0,23.0],[24.0,25.0,26.0,27.0,28.0,29.0,30.0,31.0]],"shard_shape":[[2,2],[2,2],[2,2],[2,2],[2,2],[2,2],[2,2],[2,2]],"shard_start":[[0,0],[0,2],[0,4],[0,6],[2,0],[2,2],[2,4],[2,6]]}
Input: x = [0.0,0.5,1.0,1.5,2.0,2.5,3.0,3.5], mesh_shape = [2,4], axis_names = ["data","model"], spec = [["model","data"]]
Output: {"array":[0.0,0.5,1.0,1.5,2.0,2.5,3.0,3.5],"shard_shape":[[1],[1],[1],[1],[1],[1],[1],[1]],"shard_start":[[0],[2],[4],[6],[1],[3],[5],[7]]}
mesh = jax.make_mesh(mesh_shape, axis_names); sharding = NamedSharding(mesh, spec).
jax.device_put(x, sharding) places the array.
for s in y.addressable_shards: s.device.id, s.index, s.data; slice.indices(size) fills None bounds.
Accepts: array
Accepts: array
Accepts: array
Accepts: array