Study Plan

Choose a Topology-Aware Parallel Plan

Distributed Computing

Hard

Choose data, tensor, pipeline, and context parallel degrees using the supplied memory and communication model. Let $d$ , $t$ , $p$ , and $c$ be those four positive integer degrees. Let $G$ be num_gpus, $B$ be global_batch_size, $S$ be model_state_bytes, and $A$ be activation_bytes_per_sample. A feasible plan uses every GPU:

dtpc=G.

The data degree must divide B, the context degree must divide sequence_length and meet min_context_parallel, and the product of tensor and context degrees must not exceed gpus_per_node. Per-GPU model-state memory $M_s$ and activation memory $M_a$ follow these equations, with ceiling brackets rounding up:

M_s=\left\lceil\frac{S}{tp}\right\rceil.

M_a=\left\lceil\frac{A(B/d)}{tc}\right\rceil.

Their sum must not exceed memory_per_gpu. The model's inter-node and intra-node communication estimates, $I_{\mathrm{out}}$ and $I_{\mathrm{in}}$ , are measured in bytes:

I_{\mathrm{out}}=2M_s\frac{d-1}{d}+M_a\frac{p-1}{p}.

I_{\mathrm{in}}=2M_a\left(\frac{t-1}{t}+\frac{c-1}{c}\right).

Choose the feasible plan with the smallest lexicographic score $(I_{\mathrm{out}},I_{\mathrm{in}},p,-d,t,c)$ . Return None if no plan fits; otherwise return a Python dictionary with data_parallel, tensor_parallel, pipeline_parallel, context_parallel, and estimated_memory_bytes as Python integers, plus inter_node_bytes and intra_node_bytes as Python floats.

Loading visualization...

Examples

Input: num_gpus = 4, gpus_per_node = 4, model_state_bytes = 400, activation_bytes_per_sample = 100, global_batch_size = 4, sequence_length = 8, memory_per_gpu = 300, min_context_parallel = 1

Output: {"data_parallel":1,"tensor_parallel":4,"pipeline_parallel":1,"context_parallel":1,"estimated_memory_bytes":200,"inter_node_bytes":0,"intra_node_bytes":150}

Explanation: Four-way tensor parallelism fits within one node, satisfies memory, and minimizes the ordered communication score.

Input: num_gpus = 8, gpus_per_node = 4, model_state_bytes = 800, activation_bytes_per_sample = 160, global_batch_size = 8, sequence_length = 16, memory_per_gpu = 600, min_context_parallel = 2

Output: {"data_parallel":1,"tensor_parallel":2,"pipeline_parallel":2,"context_parallel":2,"estimated_memory_bytes":520,"inter_node_bytes":160,"intra_node_bytes":640}

Input: num_gpus = 4, gpus_per_node = 2, model_state_bytes = 1000, activation_bytes_per_sample = 500, global_batch_size = 4, sequence_length = 8, memory_per_gpu = 1500, min_context_parallel = 1

Output: {"data_parallel":2,"tensor_parallel":2,"pipeline_parallel":1,"context_parallel":1,"estimated_memory_bytes":1000,"inter_node_bytes":500,"intra_node_bytes":500}

Hint 1

The % operator tests whether one integer divides another.

Hint 2

divmod gives a quotient and remainder without floating-point arithmetic.

Hint 3

Python compares tuples lexicographically.

Requirements

Satisfy all GPU, divisibility, context, node, and memory conditions.
Use the supplied model, including its pooled-byte rounding and score order.
Preserve the specified Python integer and float return types.

Constraints

num_gpus, gpus_per_node, global_batch_size, sequence_length, memory_per_gpu, and min_context_parallel are positive Python integers.
model_state_bytes and activation_bytes_per_sample are nonnegative Python integers.
gpus_per_node is at most num_gpus.
Hidden cases use at most 64 GPUs.
Communication values are returned as Python floats.

Try Similar Problems

Measure Pipeline Bubble UtilizationMedium Account for ZeRO Memory by StageEasy Synchronize Data-Parallel GradientsMedium Simulate Distributed CollectivesMedium Reconstruct a Column-Parallel MLP LayerMedium Calculate Collective Effective BandwidthEasy Decode Attention with a KV CacheHard Recover an IsoFLOP Compute-Optimal FrontierHard Calculate Tensor Parallel Communication CostHard Simulate Disaggregated Prefill and Decode ServingHard Implement Expert Parallel Routing Across GPUsHard Estimate Model Memory and Minimum GPU CountEasy

Choose a Topology-Aware Parallel Plan

Distributed Computing

Hard

dtpc=G.

M_s=\left\lceil\frac{S}{tp}\right\rceil.

M_a=\left\lceil\frac{A(B/d)}{tc}\right\rceil.

Their sum must not exceed memory_per_gpu. The model's inter-node and intra-node communication estimates, $I_{\mathrm{out}}$ and $I_{\mathrm{in}}$ , are measured in bytes:

I_{\mathrm{out}}=2M_s\frac{d-1}{d}+M_a\frac{p-1}{p}.

I_{\mathrm{in}}=2M_a\left(\frac{t-1}{t}+\frac{c-1}{c}\right).

Loading visualization...

Examples

Input: num_gpus = 4, gpus_per_node = 4, model_state_bytes = 400, activation_bytes_per_sample = 100, global_batch_size = 4, sequence_length = 8, memory_per_gpu = 300, min_context_parallel = 1

Output: {"data_parallel":1,"tensor_parallel":4,"pipeline_parallel":1,"context_parallel":1,"estimated_memory_bytes":200,"inter_node_bytes":0,"intra_node_bytes":150}

Explanation: Four-way tensor parallelism fits within one node, satisfies memory, and minimizes the ordered communication score.

Input: num_gpus = 8, gpus_per_node = 4, model_state_bytes = 800, activation_bytes_per_sample = 160, global_batch_size = 8, sequence_length = 16, memory_per_gpu = 600, min_context_parallel = 2

Output: {"data_parallel":1,"tensor_parallel":2,"pipeline_parallel":2,"context_parallel":2,"estimated_memory_bytes":520,"inter_node_bytes":160,"intra_node_bytes":640}

Input: num_gpus = 4, gpus_per_node = 2, model_state_bytes = 1000, activation_bytes_per_sample = 500, global_batch_size = 4, sequence_length = 8, memory_per_gpu = 1500, min_context_parallel = 1

Output: {"data_parallel":2,"tensor_parallel":2,"pipeline_parallel":1,"context_parallel":1,"estimated_memory_bytes":1000,"inter_node_bytes":500,"intra_node_bytes":500}

Hint 1

The % operator tests whether one integer divides another.

Hint 2

divmod gives a quotient and remainder without floating-point arithmetic.

Hint 3

Python compares tuples lexicographically.

Requirements

Satisfy all GPU, divisibility, context, node, and memory conditions.
Use the supplied model, including its pooled-byte rounding and score order.
Preserve the specified Python integer and float return types.

Constraints

num_gpus, gpus_per_node, global_batch_size, sequence_length, memory_per_gpu, and min_context_parallel are positive Python integers.
model_state_bytes and activation_bytes_per_sample are nonnegative Python integers.
gpus_per_node is at most num_gpus.
Hidden cases use at most 64 GPUs.
Communication values are returned as Python floats.

Try Similar Problems

num_gpus =

Accepts: number

gpus_per_node =

Accepts: number

model_state_bytes =

Accepts: number

activation_bytes_per_sample =

Accepts: number

global_batch_size =

Accepts: number

sequence_length =

Accepts: number

memory_per_gpu =

Accepts: number

min_context_parallel =

Accepts: number