TensorTonic
Study Plans
Study Plan
Fit Elo Ratings from Pairwise Votes
Metrics & EvaluationLanguage Models
Hard

Fit centered Elo ratings from pairwise votes. There are NNN models and MMM comparisons. In comparison nnn, ana_nan​ and bnb_nbn​ identify the models; outcome yny_nyn​ is one for an A win, zero for a B win and one half for a tie. For ratings rir_iri​, the predicted A-win probability is:

pn=11+10(rbn−ran)/400.p_n=\frac{1}{1+10^{(r_{b_n}-r_{a_n})/400}}.pn​=1+10(rbn​​−ran​​)/4001​.

The full-batch rating gradient is:

gi=ln⁡10400M∑n(yn−pn)(1[an=i]−1[bn=i]).g_i=\frac{\ln 10}{400M}\sum_n(y_n-p_n)\left(\mathbf{1}[a_n=i]-\mathbf{1}[b_n=i]\right).gi​=400Mln10​n∑​(yn​−pn​)(1[an​=i]−1[bn​=i]).

Here, iii is a model index, ln⁡\lnln is the natural logarithm, and 1\mathbf{1}1 is one when its condition holds and zero otherwise. Starting from zero ratings, perform exactly steps simultaneous updates with learning rate η\etaη:

ri←ri+ηgi.r_i\leftarrow r_i+\eta g_i.ri​←ri​+ηgi​.

Subtract the global mean rating after each update. With no comparisons, ratings remain zero. Rank by descending final rating, breaking exact ties by lower model index. Treat comparisons as undirected edges; isolated models count as components. Number components from zero in increasing order of their smallest model index.

Return a Python dictionary with ratings (float64, shape (N,)), rankings (int64, shape (N,)), fitted_probabilities (float64, shape (M,), in comparison order), and component_ids (int64, shape (N,), in model order), all PyTorch tensors on the outcomes device, plus connected as a Python Boolean indicating whether there is one component.

Loading visualization...

Examples

Input: num_models = 2, model_a = [0], model_b = [1], outcomes = [1], steps = 20, learning_rate = 100

Output: {"ratings":[5.666765,-5.666765],"rankings":[0,1],"fitted_probabilities":[0.516304],"connected":true,"component_ids":[0,0]}

Explanation: Each positive residual raises model 0 and lowers model 1 while centering keeps their ratings opposite.

Input: num_models = 3, model_a = [0,1,2], model_b = [1,2,0], outcomes = [0.5,0.5,0.5], steps = 20, learning_rate = 100

Output: {"ratings":[0,0,0],"rankings":[0,1,2],"fitted_probabilities":[0.5,0.5,0.5],"connected":true,"component_ids":[0,0,0]}

Input: num_models = 4, model_a = [0,2], model_b = [1,3], outcomes = [1,0], steps = 10, learning_rate = 80

Output: {"ratings":[1.147865,-1.147865,-1.147865,1.147865],"rankings":[0,3,1,2],"fitted_probabilities":[0.503304,0.496696],"connected":false,"component_ids":[0,0,1,1]}

Hint 1

Tensor.index_add_ accumulates contributions at repeated indices.

Hint 2

Python sorting supports a secondary key for deterministic ties.

Hint 3

A visited set can track graph components during a traversal.

Requirements

  • Use PyTorch float64 arithmetic for fitting.
  • Report probabilities from the final ratings, including when steps is zero.
  • Preserve repeated comparisons and do not modify inputs.

Constraints

  • num_models is a positive Python integer; model IDs range from zero to num_models minus one.
  • model_a and model_b are aligned one-dimensional int64 PyTorch tensors; the two IDs in each comparison differ.
  • outcomes is an aligned float64 PyTorch tensor containing only 0, 0.5 or 1.
  • The comparison list may be empty, repeated pairs are allowed, and some models may be unseen.
  • steps is a nonnegative Python integer; learning_rate is a positive finite Python number.
  • Tensor inputs share a device and may be noncontiguous.
Try Similar Problems
Remove Response-Length Bias from Judge ScoresHardCompute Conditional PerplexityMediumScore Multiple-Choice ContinuationsMediumDecode Attention with a KV CacheHardRecover an IsoFLOP Compute-Optimal FrontierHardFit a Data Power Law with a Noise FloorHardTop-k MoE Router with Load StatisticsHardRoute Cache-Aware Prefill RequestsHardImplement Speculative Decoding and Token VerificationHardBeam Search DecodingHardEvaluation Metrics (BLEU, ROUGE)MediumAUC-ROC from ScratchMedium

Accepts: number

Accepts: array

Accepts: array

Accepts: array

Accepts: number

Accepts: number

You must run your code first.

Fit Elo Ratings from Pairwise Votes

Metrics & EvaluationLanguage Models
Hard

Fit centered Elo ratings from pairwise votes. There are NNN models and MMM comparisons. In comparison nnn, ana_nan​ and bnb_nbn​ identify the models; outcome yny_nyn​ is one for an A win, zero for a B win and one half for a tie. For ratings rir_iri​, the predicted A-win probability is:

pn=11+10(rbn−ran)/400.p_n=\frac{1}{1+10^{(r_{b_n}-r_{a_n})/400}}.pn​=1+10(rbn​​−ran​​)/4001​.

The full-batch rating gradient is:

gi=ln⁡10400M∑n(yn−pn)(1[an=i]−1[bn=i]).g_i=\frac{\ln 10}{400M}\sum_n(y_n-p_n)\left(\mathbf{1}[a_n=i]-\mathbf{1}[b_n=i]\right).gi​=400Mln10​n∑​(yn​−pn​)(1[an​=i]−1[bn​=i]).

Here, iii is a model index, ln⁡\lnln is the natural logarithm, and 1\mathbf{1}1 is one when its condition holds and zero otherwise. Starting from zero ratings, perform exactly steps simultaneous updates with learning rate η\etaη:

ri←ri+ηgi.r_i\leftarrow r_i+\eta g_i.ri​←ri​+ηgi​.

Subtract the global mean rating after each update. With no comparisons, ratings remain zero. Rank by descending final rating, breaking exact ties by lower model index. Treat comparisons as undirected edges; isolated models count as components. Number components from zero in increasing order of their smallest model index.

Return a Python dictionary with ratings (float64, shape (N,)), rankings (int64, shape (N,)), fitted_probabilities (float64, shape (M,), in comparison order), and component_ids (int64, shape (N,), in model order), all PyTorch tensors on the outcomes device, plus connected as a Python Boolean indicating whether there is one component.

Loading visualization...

Examples

Input: num_models = 2, model_a = [0], model_b = [1], outcomes = [1], steps = 20, learning_rate = 100

Output: {"ratings":[5.666765,-5.666765],"rankings":[0,1],"fitted_probabilities":[0.516304],"connected":true,"component_ids":[0,0]}

Explanation: Each positive residual raises model 0 and lowers model 1 while centering keeps their ratings opposite.

Input: num_models = 3, model_a = [0,1,2], model_b = [1,2,0], outcomes = [0.5,0.5,0.5], steps = 20, learning_rate = 100

Output: {"ratings":[0,0,0],"rankings":[0,1,2],"fitted_probabilities":[0.5,0.5,0.5],"connected":true,"component_ids":[0,0,0]}

Input: num_models = 4, model_a = [0,2], model_b = [1,3], outcomes = [1,0], steps = 10, learning_rate = 80

Output: {"ratings":[1.147865,-1.147865,-1.147865,1.147865],"rankings":[0,3,1,2],"fitted_probabilities":[0.503304,0.496696],"connected":false,"component_ids":[0,0,1,1]}

Hint 1

Tensor.index_add_ accumulates contributions at repeated indices.

Hint 2

Python sorting supports a secondary key for deterministic ties.

Hint 3

A visited set can track graph components during a traversal.

Requirements

  • Use PyTorch float64 arithmetic for fitting.
  • Report probabilities from the final ratings, including when steps is zero.
  • Preserve repeated comparisons and do not modify inputs.

Constraints

  • num_models is a positive Python integer; model IDs range from zero to num_models minus one.
  • model_a and model_b are aligned one-dimensional int64 PyTorch tensors; the two IDs in each comparison differ.
  • outcomes is an aligned float64 PyTorch tensor containing only 0, 0.5 or 1.
  • The comparison list may be empty, repeated pairs are allowed, and some models may be unseen.
  • steps is a nonnegative Python integer; learning_rate is a positive finite Python number.
  • Tensor inputs share a device and may be noncontiguous.
Try Similar Problems
Remove Response-Length Bias from Judge ScoresHardCompute Conditional PerplexityMediumScore Multiple-Choice ContinuationsMediumDecode Attention with a KV CacheHardRecover an IsoFLOP Compute-Optimal FrontierHardFit a Data Power Law with a Noise FloorHardTop-k MoE Router with Load StatisticsHardRoute Cache-Aware Prefill RequestsHardImplement Speculative Decoding and Token VerificationHardBeam Search DecodingHardEvaluation Metrics (BLEU, ROUGE)MediumAUC-ROC from ScratchMedium

Accepts: number

Accepts: array

Accepts: array

Accepts: array

Accepts: number

Accepts: number

You must run your code first.