Problems

1 / 1

BLEU Score

NLPMetrics & Evaluation

Hard

BLEU (Bilingual Evaluation Understudy) is the standard metric for evaluating machine translation quality. It measures how many n-grams in the candidate translation appear in the reference, with a penalty for translations that are too short.

Given a candidate translation, a reference translation (both as token lists), and a maximum n-gram order, compute the BLEU score.

Algorithm

For each n-gram order from 1 to max_n, compute modified precision by clipping candidate n-gram counts by reference counts:

p_n = \frac{\sum_{ng} \min(C_{ng},\; R_{ng})}{\sum_{ng} C_{ng}}

Compute the brevity penalty to penalize short translations:

BP = \begin{cases} 1 & \text{if } c \ge r \\ e^{1 - r/c} & \text{if } c < r \end{cases}

Combine into the BLEU score using the geometric mean of precisions:

\text{BLEU} = BP \cdot \exp\left(\frac{1}{N} \sum_{n=1}^{N} \log p_n\right)

If any precision is zero, BLEU is zero.

Return the BLEU score as a float between zero and one.

Loading visualization...

Examples

Input: candidate = ["the", "cat", "sat", "on", "the", "mat"], reference = ["the", "cat", "sat", "on", "the", "mat"], max_n = 4

Output: 1.0

Explanation: Every modified precision equals one and no brevity penalty applies.

Input: candidate = ["the", "cat", "is", "here"], reference = ["the", "cat", "was", "here"], max_n = 2

Output: 0.5

Hint 1

Use tuples as n-gram keys and clip each candidate count by its reference count.

Hint 2

Return zero for any zero precision; otherwise combine log precisions and the brevity penalty.

Requirements

Compute modified n-gram precision for each order from 1 to max_n, clipping counts by the reference
Apply the brevity penalty when the candidate is shorter than the reference
Combine precisions using a uniform-weight geometric mean
Return 0.0 if the candidate is empty or any precision is zero
Return the BLEU score as a float

Constraints

candidate and reference are lists of string tokens
max_n >= 1
Return a float between 0.0 and 1.0
Time limit: 300 ms

Try Similar Problems

Bag Of Words Tfidf Vectorizer Perplexity Computation Edit Distance Bigram Probabilities

Prev Next

BLEU Score

NLPMetrics & Evaluation

Hard

Given a candidate translation, a reference translation (both as token lists), and a maximum n-gram order, compute the BLEU score.

Algorithm

For each n-gram order from 1 to max_n, compute modified precision by clipping candidate n-gram counts by reference counts:

p_n = \frac{\sum_{ng} \min(C_{ng},\; R_{ng})}{\sum_{ng} C_{ng}}

Compute the brevity penalty to penalize short translations:

BP = \begin{cases} 1 & \text{if } c \ge r \\ e^{1 - r/c} & \text{if } c < r \end{cases}

Combine into the BLEU score using the geometric mean of precisions:

\text{BLEU} = BP \cdot \exp\left(\frac{1}{N} \sum_{n=1}^{N} \log p_n\right)

If any precision is zero, BLEU is zero.

Return the BLEU score as a float between zero and one.

Loading visualization...

Examples

Input: candidate = ["the", "cat", "sat", "on", "the", "mat"], reference = ["the", "cat", "sat", "on", "the", "mat"], max_n = 4

Output: 1.0

Explanation: Every modified precision equals one and no brevity penalty applies.

Input: candidate = ["the", "cat", "is", "here"], reference = ["the", "cat", "was", "here"], max_n = 2

Output: 0.5

Hint 1

Use tuples as n-gram keys and clip each candidate count by its reference count.

Hint 2

Return zero for any zero precision; otherwise combine log precisions and the brevity penalty.

Requirements

Compute modified n-gram precision for each order from 1 to max_n, clipping counts by the reference
Apply the brevity penalty when the candidate is shorter than the reference
Combine precisions using a uniform-weight geometric mean
Return 0.0 if the candidate is empty or any precision is zero
Return the BLEU score as a float

Constraints

candidate and reference are lists of string tokens
max_n >= 1
Return a float between 0.0 and 1.0
Time limit: 300 ms

Try Similar Problems

Bag Of Words Tfidf Vectorizer Perplexity Computation Edit Distance Bigram Probabilities

Case 1

Case 2

candidate =

Accepts: array

reference =

Accepts: array

max_n =

Accepts: number

Given a candidate translation, a reference translation (both as token lists), and a maximum n-gram order, compute the BLEU score.

Algorithm

For each n-gram order from 1 to max_n, compute modified precision by clipping candidate n-gram counts by reference counts:

p_n = \frac{\sum_{ng} \min(C_{ng},\; R_{ng})}{\sum_{ng} C_{ng}}

Compute the brevity penalty to penalize short translations:

BP = \begin{cases} 1 & \text{if } c \ge r \\ e^{1 - r/c} & \text{if } c < r \end{cases}

Combine into the BLEU score using the geometric mean of precisions:

\text{BLEU} = BP \cdot \exp\left(\frac{1}{N} \sum_{n=1}^{N} \log p_n\right)

If any precision is zero, BLEU is zero.

Return the BLEU score as a float between zero and one.