Compute conditional perplexity using only the response tokens selected by the mask. Logits and target IDs are already aligned at each batch and sequence position; no token shift is needed. Let i range over those positions, xi be the target ID, mi be its mask value (one for selected, zero otherwise), and pi(xi) be its probability under the vocabulary softmax. With M=∑imi selected tokens, the negative log likelihood is:
NLL=−M1i∑milogpi(xi).Perplexity is:
PPL=exp(NLL).Here, log is the natural logarithm and exp its inverse. Average across selected tokens, not across sequences. If none are selected, use NLL zero and perplexity one. Return a Python dictionary with negative_log_likelihood and perplexity as Python floats, and selected_token_count as a Python integer.
Input: logits = [[[0,0],[0,0],[0,0]]], target_ids = [[0,1,0]], response_mask = [[false,true,true]]
Output: {"negative_log_likelihood":0.693147,"perplexity":2,"selected_token_count":2}
Explanation: Each selected token has probability one half, so the average negative log likelihood is log 2 and perplexity is 2.
Input: logits = [[[4,0],[0,4],[9,-9],[-9,9]]], target_ids = [[0,1,1,0]], response_mask = [[false,true,false,false]]
Output: {"negative_log_likelihood":0.01815,"perplexity":1.018316,"selected_token_count":1}
Input: logits = [[[1,-1],[2,-2]]], target_ids = [[0,1]], response_mask = [[false,false]]
Output: {"negative_log_likelihood":0,"perplexity":1,"selected_token_count":0}
torch.log_softmax computes stable log probabilities.
Tensor.gather selects entries using an index tensor.
Boolean indexing selects only the positions marked true.
Accepts: array
Accepts: array
Accepts: array
Compute conditional perplexity using only the response tokens selected by the mask. Logits and target IDs are already aligned at each batch and sequence position; no token shift is needed. Let i range over those positions, xi be the target ID, mi be its mask value (one for selected, zero otherwise), and pi(xi) be its probability under the vocabulary softmax. With M=∑imi selected tokens, the negative log likelihood is:
NLL=−M1i∑milogpi(xi).Perplexity is:
PPL=exp(NLL).Here, log is the natural logarithm and exp its inverse. Average across selected tokens, not across sequences. If none are selected, use NLL zero and perplexity one. Return a Python dictionary with negative_log_likelihood and perplexity as Python floats, and selected_token_count as a Python integer.
Input: logits = [[[0,0],[0,0],[0,0]]], target_ids = [[0,1,0]], response_mask = [[false,true,true]]
Output: {"negative_log_likelihood":0.693147,"perplexity":2,"selected_token_count":2}
Explanation: Each selected token has probability one half, so the average negative log likelihood is log 2 and perplexity is 2.
Input: logits = [[[4,0],[0,4],[9,-9],[-9,9]]], target_ids = [[0,1,1,0]], response_mask = [[false,true,false,false]]
Output: {"negative_log_likelihood":0.01815,"perplexity":1.018316,"selected_token_count":1}
Input: logits = [[[1,-1],[2,-2]]], target_ids = [[0,1]], response_mask = [[false,false]]
Output: {"negative_log_likelihood":0,"perplexity":1,"selected_token_count":0}
torch.log_softmax computes stable log probabilities.
Tensor.gather selects entries using an index tensor.
Boolean indexing selects only the positions marked true.
Accepts: array
Accepts: array
Accepts: array
