TensorTonic
Study Plans
Study Plan
Score Multiple-Choice Continuations
Metrics & EvaluationLanguage Models
Medium

Score multiple-choice continuations by their summed conditional log likelihood. Logits and target IDs are already aligned. For batch item bbb, choice ccc and token position ttt, let zbctz_{bct}zbct​ be the vocabulary-logit vector, xbctx_{bct}xbct​ the target ID and mbctm_{bct}mbct​ the Boolean selection mask. The score is:

sbc=∑tmbctlog⁡softmax⁡(zbct)xbct.s_{bc}=\sum_t m_{bct}\log\operatorname{softmax}(z_{bct})_{x_{bct}}.sbc​=t∑​mbct​logsoftmax(zbct​)xbct​​.

The selected choice is:

cb∗=arg max⁡csbc.c_b^*=\operatorname*{arg\,max}_c s_{bc}.cb∗​=cargmax​sbc​.

Here, softmax is over vocabulary entries and log is the natural logarithm. Masked positions contribute zero; do not divide by continuation length. A choice with no selected tokens has score zero, and exact ties select the lowest choice index.

Return a Python dictionary with choice_scores, a PyTorch tensor of shape (batch, choices) matching the logits dtype and device, and predicted_indices, an int64 PyTorch tensor of shape (batch,) on that device.

Loading visualization...

Examples

Input: logits = [[[[2,0]],[[0,2]]]], token_ids = [[[0],[0]]], continuation_mask = [[[true],[true]]]

Output: {"choice_scores":[[-0.126928,-2.126928]],"predicted_indices":[0]}

Explanation: The first choice assigns more probability to its target token and therefore has the larger summed log likelihood.

Input: logits = [[[[1,0],[1,0]],[[2,0],[0,0]]]], token_ids = [[[0,0],[0,1]]], continuation_mask = [[[true,true],[true,false]]]

Output: {"choice_scores":[[-0.626523,-0.126928]],"predicted_indices":[1]}

Input: logits = [[[[0,0]],[[2,-2]],[[-2,2]]]], token_ids = [[[0],[0],[1]]], continuation_mask = [[[false],[false],[false]]]

Output: {"choice_scores":[[0,0,0]],"predicted_indices":[0]}

Hint 1

torch.log_softmax computes log probabilities over a chosen dimension.

Hint 2

Tensor.gather selects each target token from its vocabulary axis.

Hint 3

torch.argmax returns the first index when maximum values tie.

Requirements

  • Use PyTorch and preserve the logits dtype for scores.
  • Rank choices independently for each batch item.
  • Do not modify input tensors.

Constraints

  • logits is a float32 or float64 PyTorch tensor of shape (batch, choices, tokens, vocabulary).
  • token_ids and continuation_mask have shape (batch, choices, tokens), with int64 and bool dtypes respectively.
  • There is at least one choice and one vocabulary entry; target IDs are valid vocabulary indices.
  • Inputs share a device, contain finite values and may be noncontiguous.
Try Similar Problems
Compute Conditional PerplexityMediumFit Elo Ratings from Pairwise VotesHardRemove Response-Length Bias from Judge ScoresHardPerplexity ComputationEasyTrain a BPE Tokenizer From Scratch in PythonMediumCanonicalize Pairwise Preference DataMediumCompute Temperature-Based Data MixturesMediumApply Ranked BPE MergesMediumEvaluation Metrics (BLEU, ROUGE)MediumAUC-ROC from ScratchMediumImplement Prefix Cache Matching and ReuseMediumN-gram Language ModelMedium

Accepts: array

Accepts: array

Accepts: array

You must run your code first.

Score Multiple-Choice Continuations

Metrics & EvaluationLanguage Models
Medium

Score multiple-choice continuations by their summed conditional log likelihood. Logits and target IDs are already aligned. For batch item bbb, choice ccc and token position ttt, let zbctz_{bct}zbct​ be the vocabulary-logit vector, xbctx_{bct}xbct​ the target ID and mbctm_{bct}mbct​ the Boolean selection mask. The score is:

sbc=∑tmbctlog⁡softmax⁡(zbct)xbct.s_{bc}=\sum_t m_{bct}\log\operatorname{softmax}(z_{bct})_{x_{bct}}.sbc​=t∑​mbct​logsoftmax(zbct​)xbct​​.

The selected choice is:

cb∗=arg max⁡csbc.c_b^*=\operatorname*{arg\,max}_c s_{bc}.cb∗​=cargmax​sbc​.

Here, softmax is over vocabulary entries and log is the natural logarithm. Masked positions contribute zero; do not divide by continuation length. A choice with no selected tokens has score zero, and exact ties select the lowest choice index.

Return a Python dictionary with choice_scores, a PyTorch tensor of shape (batch, choices) matching the logits dtype and device, and predicted_indices, an int64 PyTorch tensor of shape (batch,) on that device.

Loading visualization...

Examples

Input: logits = [[[[2,0]],[[0,2]]]], token_ids = [[[0],[0]]], continuation_mask = [[[true],[true]]]

Output: {"choice_scores":[[-0.126928,-2.126928]],"predicted_indices":[0]}

Explanation: The first choice assigns more probability to its target token and therefore has the larger summed log likelihood.

Input: logits = [[[[1,0],[1,0]],[[2,0],[0,0]]]], token_ids = [[[0,0],[0,1]]], continuation_mask = [[[true,true],[true,false]]]

Output: {"choice_scores":[[-0.626523,-0.126928]],"predicted_indices":[1]}

Input: logits = [[[[0,0]],[[2,-2]],[[-2,2]]]], token_ids = [[[0],[0],[1]]], continuation_mask = [[[false],[false],[false]]]

Output: {"choice_scores":[[0,0,0]],"predicted_indices":[0]}

Hint 1

torch.log_softmax computes log probabilities over a chosen dimension.

Hint 2

Tensor.gather selects each target token from its vocabulary axis.

Hint 3

torch.argmax returns the first index when maximum values tie.

Requirements

  • Use PyTorch and preserve the logits dtype for scores.
  • Rank choices independently for each batch item.
  • Do not modify input tensors.

Constraints

  • logits is a float32 or float64 PyTorch tensor of shape (batch, choices, tokens, vocabulary).
  • token_ids and continuation_mask have shape (batch, choices, tokens), with int64 and bool dtypes respectively.
  • There is at least one choice and one vocabulary entry; target IDs are valid vocabulary indices.
  • Inputs share a device, contain finite values and may be noncontiguous.
Try Similar Problems
Compute Conditional PerplexityMediumFit Elo Ratings from Pairwise VotesHardRemove Response-Length Bias from Judge ScoresHardPerplexity ComputationEasyTrain a BPE Tokenizer From Scratch in PythonMediumCanonicalize Pairwise Preference DataMediumCompute Temperature-Based Data MixturesMediumApply Ranked BPE MergesMediumEvaluation Metrics (BLEU, ROUGE)MediumAUC-ROC from ScratchMediumImplement Prefix Cache Matching and ReuseMediumN-gram Language ModelMedium

Accepts: array

Accepts: array

Accepts: array

You must run your code first.