Study Plan

Plan KV-Cache Capacity

Language ModelsInference

Medium

Estimate KV-cache capacity and decode throughput using the supplied memory-bandwidth model. Let $P$ be model parameter count, $b_p$ bytes per parameter, $L$ layer count, $S$ cached sequence length, $H_{kv}$ KV-head count, $D$ head width, and $b_e$ bytes per KV element. Model storage $M_{\mathrm{model}}$ and per-request cache storage $M_{\mathrm{request}}$ are:

M_{\mathrm{model}}=Pb_p.

M_{\mathrm{request}}=2LSH_{kv}Db_e.

The factor two accounts for keys and values. The supplied query-head count describes the attention layout; cache storage depends on KV heads, not query heads. For total GPU memory $M$ and reserved memory $R$ , the largest fitting batch $B$ is:

B=\max\left(0,\left\lfloor\frac{M-R-M_{\mathrm{model}}}{M_{\mathrm{request}}}\right\rfloor\right).

Floor brackets round down to a whole request. Assume each decode step reads the model once and every fitted request's complete cache. For bandwidth $W$ in bytes per second, step time $T$ is:

T=\frac{M_{\mathrm{model}}+BM_{\mathrm{request}}}{W}.

Throughput is $B/T$ , or zero when no request fits. The time formula still applies when $B=0$ . Return a Python dictionary with model_bytes, kv_bytes_per_request, max_batch_size as Python integers, and seconds_per_token and tokens_per_second as Python floats. seconds_per_token is the time for one batched decode step.

Loading visualization...

Examples

Input: model_parameters = 1000, bytes_per_parameter = 2, num_layers = 2, sequence_length = 4, num_attention_heads = 8, num_kv_heads = 2, head_dim = 4, bytes_per_element = 2, memory_per_gpu = 10000, reserved_memory = 1000, memory_bandwidth = 100000

Output: {"model_bytes":2000,"kv_bytes_per_request":256,"max_batch_size":27,"seconds_per_token":0.08912,"tokens_per_second":302.962298}

Explanation: After model and reserved memory, 7000 bytes hold 27 complete request caches of 256 bytes each.

Input: model_parameters = 10000, bytes_per_parameter = 2, num_layers = 2, sequence_length = 4, num_attention_heads = 8, num_kv_heads = 1, head_dim = 4, bytes_per_element = 2, memory_per_gpu = 10000, reserved_memory = 0, memory_bandwidth = 100000

Output: {"model_bytes":20000,"kv_bytes_per_request":128,"max_batch_size":0,"seconds_per_token":0.2,"tokens_per_second":0}

Input: model_parameters = 500, bytes_per_parameter = 1, num_layers = 4, sequence_length = 8, num_attention_heads = 16, num_kv_heads = 1, head_dim = 8, bytes_per_element = 2, memory_per_gpu = 20000, reserved_memory = 500, memory_bandwidth = 200000

Output: {"model_bytes":500,"kv_bytes_per_request":1024,"max_batch_size":18,"seconds_per_token":0.09466,"tokens_per_second":190.154236}

Hint 1

The // operator gives an integer quotient rounded down.

Hint 2

max can enforce a nonnegative capacity.

Requirements

Keep memory and batch counts exact using Python integer arithmetic.
Follow the supplied bandwidth model, including zero-capacity behavior.
Return the specified integer counts and floating-point rates.

Constraints

Every argument is a Python integer.
model_parameters and reserved_memory are nonnegative; all other size, count, capacity, and bandwidth arguments are positive.
num_attention_heads is divisible by num_kv_heads.
All byte arithmetic precedes floating-point rate calculations.
A zero fitting batch is valid.
Input integers, model_bytes, kv_bytes_per_request, and max_batch_size are at most 9,007,199,254,740,991.

Try Similar Problems

Decode Attention with a KV CacheHard Perform Exact Speculative SamplingHard Find Reusable Prefixes in a KV TrieMedium Route Cache-Aware Prefill RequestsHard Simulate Continuous Decode BatchingHard Train a BPE Tokenizer From Scratch in PythonMedium Canonicalize Pairwise Preference DataMedium Compute Temperature-Based Data MixturesMedium Implement Prefix Cache Matching and ReuseMedium Beam Search DecodingMedium Text Generation SamplingMedium Implement Temperature, Top-k, and Top-p SamplingMedium

Plan KV-Cache Capacity

Language ModelsInference

Medium

M_{\mathrm{model}}=Pb_p.

M_{\mathrm{request}}=2LSH_{kv}Db_e.

B=\max\left(0,\left\lfloor\frac{M-R-M_{\mathrm{model}}}{M_{\mathrm{request}}}\right\rfloor\right).

Floor brackets round down to a whole request. Assume each decode step reads the model once and every fitted request's complete cache. For bandwidth $W$ in bytes per second, step time $T$ is:

T=\frac{M_{\mathrm{model}}+BM_{\mathrm{request}}}{W}.

Loading visualization...

Examples

Output: {"model_bytes":2000,"kv_bytes_per_request":256,"max_batch_size":27,"seconds_per_token":0.08912,"tokens_per_second":302.962298}

Explanation: After model and reserved memory, 7000 bytes hold 27 complete request caches of 256 bytes each.

Output: {"model_bytes":20000,"kv_bytes_per_request":128,"max_batch_size":0,"seconds_per_token":0.2,"tokens_per_second":0}

Output: {"model_bytes":500,"kv_bytes_per_request":1024,"max_batch_size":18,"seconds_per_token":0.09466,"tokens_per_second":190.154236}

Hint 1

The // operator gives an integer quotient rounded down.

Hint 2

max can enforce a nonnegative capacity.

Requirements

Keep memory and batch counts exact using Python integer arithmetic.
Follow the supplied bandwidth model, including zero-capacity behavior.
Return the specified integer counts and floating-point rates.

Constraints

Every argument is a Python integer.
model_parameters and reserved_memory are nonnegative; all other size, count, capacity, and bandwidth arguments are positive.
num_attention_heads is divisible by num_kv_heads.
All byte arithmetic precedes floating-point rate calculations.
A zero fitting batch is valid.
Input integers, model_bytes, kv_bytes_per_request, and max_batch_size are at most 9,007,199,254,740,991.

Try Similar Problems

model_parameters =

Accepts: number

bytes_per_parameter =

Accepts: number

num_layers =

Accepts: number

sequence_length =

Accepts: number

num_attention_heads =

Accepts: number

num_kv_heads =

Accepts: number

head_dim =

Accepts: number

bytes_per_element =

Accepts: number

memory_per_gpu =

Accepts: number

reserved_memory =

Accepts: number

memory_bandwidth =

Accepts: number