A fraud model gives every transaction a score between 0 and 1, and someone has to decide the score above which a transaction is blocked. The classification metrics topic of the ML System Design course explains that the model does not make this decision, the team does, by choosing a threshold that fits the cost of each kind of mistake. A common way to state the business rule is a floor on precision, such as at least nine out of ten blocked transactions must really be fraud, and then to catch as much fraud as that floor allows. This problem asks you to find that threshold.
You are given a list y_true of labels, each 0 or 1, a list scores of the same length with the model's score for each row, and a number min_precision. A candidate threshold is any value that appears in scores. At a candidate threshold t, every row with a score greater than or equal to t is predicted positive. Precision is the number of predicted positives whose label is 1 divided by the number of predicted positives, and recall is the number of predicted positives whose label is 1 divided by the number of rows whose label is 1.
Among the candidate thresholds whose precision is greater than or equal to min_precision, choose the one with the highest recall. If two candidates tie on recall, choose the one with the higher precision. Return a dictionary with the keys threshold, precision and recall, where threshold is the chosen score as a float. If no candidate meets the precision floor, return the same three keys with every value equal to None.
Worked example. The labels are 1, 0, 1, 1, 0, 0 and the scores are 0.9, 0.8, 0.7, 0.4, 0.3, 0.1, with a precision floor of 0.6. At threshold 0.9 one row is predicted positive and it is a true positive, so precision is 1.0 and recall is one third. At 0.8 precision drops to 0.5, which fails the floor, and at 0.7 precision is two thirds and recall is two thirds.
At 0.4 precision is 0.75 and recall is 1.0, at 0.3 precision is 0.6 and recall is 1.0, and at 0.1 precision is 0.5, which fails. The candidates that pass are 0.9, 0.7, 0.4 and 0.3. The highest recall is 1.0, shared by 0.4 and 0.3, and 0.4 has the higher precision, so the answer is threshold 0.4 with precision 0.75 and recall 1.0.
Input: y_true = [1, 0, 1, 1, 0, 0], scores = [0.9, 0.8, 0.7, 0.4, 0.3, 0.1], min_precision = 0.6
Output: {"threshold": 0.4, "precision": 0.75, "recall": 1.0}
Explanation: Thresholds 0.4 and 0.3 both reach full recall while meeting the floor, and 0.4 has the higher precision.
Input: y_true = [0, 1, 0], scores = [0.9, 0.5, 0.2], min_precision = 0.75
Output: {"threshold": null, "precision": null, "recall": null}
Explanation: The top scoring row is a negative, so every threshold has precision of one half or less and none meets the floor.
Loop over sorted(set(scores)) and for each candidate count true positives and predicted positives with a comparison of score >= candidate.
Track the best candidate as you go and only replace it when recall is strictly higher, or equal with higher precision.
Accepts: array
Accepts: array
Accepts: number
A fraud model gives every transaction a score between 0 and 1, and someone has to decide the score above which a transaction is blocked. The classification metrics topic of the ML System Design course explains that the model does not make this decision, the team does, by choosing a threshold that fits the cost of each kind of mistake. A common way to state the business rule is a floor on precision, such as at least nine out of ten blocked transactions must really be fraud, and then to catch as much fraud as that floor allows. This problem asks you to find that threshold.
You are given a list y_true of labels, each 0 or 1, a list scores of the same length with the model's score for each row, and a number min_precision. A candidate threshold is any value that appears in scores. At a candidate threshold t, every row with a score greater than or equal to t is predicted positive. Precision is the number of predicted positives whose label is 1 divided by the number of predicted positives, and recall is the number of predicted positives whose label is 1 divided by the number of rows whose label is 1.
Among the candidate thresholds whose precision is greater than or equal to min_precision, choose the one with the highest recall. If two candidates tie on recall, choose the one with the higher precision. Return a dictionary with the keys threshold, precision and recall, where threshold is the chosen score as a float. If no candidate meets the precision floor, return the same three keys with every value equal to None.
Worked example. The labels are 1, 0, 1, 1, 0, 0 and the scores are 0.9, 0.8, 0.7, 0.4, 0.3, 0.1, with a precision floor of 0.6. At threshold 0.9 one row is predicted positive and it is a true positive, so precision is 1.0 and recall is one third. At 0.8 precision drops to 0.5, which fails the floor, and at 0.7 precision is two thirds and recall is two thirds.
At 0.4 precision is 0.75 and recall is 1.0, at 0.3 precision is 0.6 and recall is 1.0, and at 0.1 precision is 0.5, which fails. The candidates that pass are 0.9, 0.7, 0.4 and 0.3. The highest recall is 1.0, shared by 0.4 and 0.3, and 0.4 has the higher precision, so the answer is threshold 0.4 with precision 0.75 and recall 1.0.
Input: y_true = [1, 0, 1, 1, 0, 0], scores = [0.9, 0.8, 0.7, 0.4, 0.3, 0.1], min_precision = 0.6
Output: {"threshold": 0.4, "precision": 0.75, "recall": 1.0}
Explanation: Thresholds 0.4 and 0.3 both reach full recall while meeting the floor, and 0.4 has the higher precision.
Input: y_true = [0, 1, 0], scores = [0.9, 0.5, 0.2], min_precision = 0.75
Output: {"threshold": null, "precision": null, "recall": null}
Explanation: The top scoring row is a negative, so every threshold has precision of one half or less and none meets the floor.
Loop over sorted(set(scores)) and for each candidate count true positives and predicted positives with a comparison of score >= candidate.
Track the best candidate as you go and only replace it when recall is strictly higher, or equal with higher precision.
Accepts: array
Accepts: array
Accepts: number