Study Plan

Recover an IsoFLOP Compute-Optimal Frontier

Scaling LawsLanguage ModelsOptimization

Hard

Estimate the compute-optimal model size from fixed-compute training sweeps. Each row gives parameter counts $N$ and terminal losses at compute budget $C$ . Its unweighted least-squares quadratic is defined over $x=\log N$ , using natural logarithms:

L_C(x)=a_Cx^2+b_Cx+c_C.

Here, $L_C$ is fitted loss, and $a_C$ , $b_C$ , $c_C$ are the fitted quadratic coefficients. Positive curvature and an interior minimum are guaranteed. The optimal log parameter count $x_C^*$ , parameter count $N_C$ , and token count $D_C$ satisfy:

x_C^*=-\frac{b_C}{2a_C}.

N_C=e^{x_C^*}.

D_C=\frac{C}{6N_C}.

The supplied compute model is six FLOPs per parameter per training token. The frontier is an unweighted least-squares line across all recovered optima:

\log N_C=\log k+\alpha\log C.

Here, $k$ is frontier scale and $\alpha$ is frontier exponent. Return a Python dictionary with optima, a budget-ordered list of dictionaries containing compute, parameters, tokens, loss as Python floats, plus frontier_scale and frontier_exponent as Python floats. Each loss is the quadratic's value at its fitted minimum.

Loading visualization...

Examples

Input: compute_budgets = [1000000,4000000], parameter_counts = [[250,500,2000,4000],[500,1000,4000,8000]], terminal_losses = [[1.5765436167,1.1441359042,1.1441359042,1.5765436167],[1.6765436167,1.2441359042,1.2441359042,1.6765436167]]

Output: {"optima":[{"compute":1000000,"parameters":1000,"tokens":166.666667,"loss":1},{"compute":4000000,"parameters":2000,"tokens":333.333333,"loss":1.1}],"frontier_scale":1,"frontier_exponent":0.5}

Explanation: The two quadratic vertices recover 1000 and 2000 parameters, which lie on the exact square-root compute frontier.

Input: compute_budgets = [10000,1000000,100000000], parameter_counts.shape = (3, 4), terminal_losses.shape = (3, 4)

Output: {"optima":[{"compute":10000,"parameters":100,"tokens":16.666667,"loss":1},{"compute":1000000,"parameters":1000,"tokens":166.666667,"loss":1.1},{"compute":100000000,"parameters":10000,"tokens":1666.666667,"loss":1.2}],"frontier_scale":1,"frontier_exponent":0.5}

Input: compute_budgets = [100000,1000000,10000000], parameter_counts.shape = (3, 4), terminal_losses.shape = (3, 4)

Output: {"optima":[{"compute":100000,"parameters":100,"tokens":166.666667,"loss":1},{"compute":1000000,"parameters":251.188644,"tokens":663.51195,"loss":1.1},{"compute":10000000,"parameters":630.957344,"tokens":2641.488655,"loss":1.2}],"frontier_scale":1,"frontier_exponent":0.4}

Hint 1

torch.log computes natural logarithms elementwise.

Hint 2

torch.stack combines feature vectors along a new axis.

Hint 3

torch.linalg.lstsq solves both quadratic and linear least-squares fits.

Requirements

Use PyTorch float64 arithmetic for both least-squares fits.
Preserve compute-budget order in optima.
Do not modify input tensors.

Constraints

compute_budgets is a one-dimensional float64 PyTorch tensor with at least two unique positive entries.
parameter_counts and terminal_losses are aligned float64 PyTorch tensors, one row per budget and at least three columns.
Parameter counts are positive and unique within each row.
Every fitted quadratic has positive curvature and an interior minimum.
All inputs are finite, share a device, and may be noncontiguous.

Try Similar Problems

Fit Learning-Rate and Batch-Size ScalingHard Fit a Data Power Law with a Noise FloorHard Extrapolate Data-Mixture Scaling CurvesHard Estimate Critical Batch SizeEasy Generate a Warmup-Stable-Decay ScheduleMedium Diagnose muP Width InvariantsMedium Orthogonalize a Muon Momentum UpdateEasy Mixed-Precision Training Memory AccountantEasy GRPO Group-Relative AdvantageHard Hungarian MatchingHard Implement Speculative Decoding and Token VerificationHard Gradient ClippingHard

Recover an IsoFLOP Compute-Optimal Frontier

Scaling LawsLanguage ModelsOptimization

Hard

L_C(x)=a_Cx^2+b_Cx+c_C.

x_C^*=-\frac{b_C}{2a_C}.

N_C=e^{x_C^*}.

D_C=\frac{C}{6N_C}.

The supplied compute model is six FLOPs per parameter per training token. The frontier is an unweighted least-squares line across all recovered optima:

\log N_C=\log k+\alpha\log C.

Loading visualization...

Examples

Output: {"optima":[{"compute":1000000,"parameters":1000,"tokens":166.666667,"loss":1},{"compute":4000000,"parameters":2000,"tokens":333.333333,"loss":1.1}],"frontier_scale":1,"frontier_exponent":0.5}

Explanation: The two quadratic vertices recover 1000 and 2000 parameters, which lie on the exact square-root compute frontier.

Input: compute_budgets = [10000,1000000,100000000], parameter_counts.shape = (3, 4), terminal_losses.shape = (3, 4)

Input: compute_budgets = [100000,1000000,10000000], parameter_counts.shape = (3, 4), terminal_losses.shape = (3, 4)

Hint 1

torch.log computes natural logarithms elementwise.

Hint 2

torch.stack combines feature vectors along a new axis.

Hint 3

torch.linalg.lstsq solves both quadratic and linear least-squares fits.

Requirements

Use PyTorch float64 arithmetic for both least-squares fits.
Preserve compute-budget order in optima.
Do not modify input tensors.

Constraints

compute_budgets is a one-dimensional float64 PyTorch tensor with at least two unique positive entries.
parameter_counts and terminal_losses are aligned float64 PyTorch tensors, one row per budget and at least three columns.
Parameter counts are positive and unique within each row.
Every fitted quadratic has positive curvature and an interior minimum.
All inputs are finite, share a device, and may be noncontiguous.

Try Similar Problems

compute_budgets =

Accepts: array

parameter_counts =

Accepts: array

terminal_losses =

Accepts: array