Problems

1 / 1

Monitoring Metrics Selection

MLOps

Hard

Compute monitoring metrics for one of three ML system types. Return a dictionary whose keys depend on system_type.

For "classification", return accuracy, precision, recall, and F1. Use zero when a precision, recall, or F1 denominator is zero.

\mathrm{accuracy}=\frac{TP+TN}{n}

\mathrm{precision}=\frac{TP}{TP+FP}

\mathrm{recall}=\frac{TP}{TP+FN}

F_1=\frac{2PR}{P+R}

For "regression", return mean absolute error and root mean squared error.

\mathrm{MAE}=\frac{1}{n}\sum_i |y_i-\hat y_i|

\mathrm{RMSE}=\sqrt{\frac{1}{n}\sum_i(y_i-\hat y_i)^2}

For "ranking", sort items by descending predicted score and return precision at 3 and recall at 3. Tied scores retain input order. Precision at 3 always divides by 3. Here, $P$ is precision, $R$ is recall, $y_i$ is a target, and $\hat y_i$ is a prediction.

Loading visualization...

Examples

Input: system_type = "classification", y_true = [1, 0, 1, 1, 0, 1, 0, 0], y_pred = [1, 0, 0, 1, 0, 1, 1, 0]

Output: {"accuracy": 0.75, "precision": 0.75, "recall": 0.75, "f1": 0.75}

Explanation: There are three true positives, three true negatives, one false positive, and one false negative.

Input: system_type = "regression", y_true = [3, 5, 2.5, 7], y_pred = [2.5, 5.5, 2, 8]

Output: {"mae": 0.625, "rmse": 0.661438}

Hint 1

For classification, count tp, fp, fn, and tn in one pass over paired values.

Hint 2

For ranking, sort indices with key=lambda i: y_pred[i], reverse=True and inspect the first three.

Requirements

Select metrics from system_type
Compute every metric from the supplied targets and predictions
Handle zero metric denominators by returning 0.0
Return the named metric dictionary specified for that system type

Constraints

system_type is classification, regression, or ranking
y_true and y_pred have equal nonzero length
Ranking inputs contain at least three items

Try Similar Problems

Data Drift Detection Train Serving Skew Shadow Deployment Evaluation Retraining Trigger Design Model Versioning Basics

Prev Next

Monitoring Metrics Selection

MLOps

Hard

Compute monitoring metrics for one of three ML system types. Return a dictionary whose keys depend on system_type.

For "classification", return accuracy, precision, recall, and F1. Use zero when a precision, recall, or F1 denominator is zero.

\mathrm{accuracy}=\frac{TP+TN}{n}

\mathrm{precision}=\frac{TP}{TP+FP}

\mathrm{recall}=\frac{TP}{TP+FN}

F_1=\frac{2PR}{P+R}

For "regression", return mean absolute error and root mean squared error.

\mathrm{MAE}=\frac{1}{n}\sum_i |y_i-\hat y_i|

\mathrm{RMSE}=\sqrt{\frac{1}{n}\sum_i(y_i-\hat y_i)^2}

Loading visualization...

Examples

Input: system_type = "classification", y_true = [1, 0, 1, 1, 0, 1, 0, 0], y_pred = [1, 0, 0, 1, 0, 1, 1, 0]

Output: {"accuracy": 0.75, "precision": 0.75, "recall": 0.75, "f1": 0.75}

Explanation: There are three true positives, three true negatives, one false positive, and one false negative.

Input: system_type = "regression", y_true = [3, 5, 2.5, 7], y_pred = [2.5, 5.5, 2, 8]

Output: {"mae": 0.625, "rmse": 0.661438}

Hint 1

For classification, count tp, fp, fn, and tn in one pass over paired values.

Hint 2

For ranking, sort indices with key=lambda i: y_pred[i], reverse=True and inspect the first three.

Requirements

Select metrics from system_type
Compute every metric from the supplied targets and predictions
Handle zero metric denominators by returning 0.0
Return the named metric dictionary specified for that system type

Constraints

system_type is classification, regression, or ranking
y_true and y_pred have equal nonzero length
Ranking inputs contain at least three items

Try Similar Problems

Data Drift Detection Train Serving Skew Shadow Deployment Evaluation Retraining Trigger Design Model Versioning Basics

Case 1

Case 2

system_type =

Accepts: string

y_true =

Accepts: array

y_pred =

Accepts: array

Compute monitoring metrics for one of three ML system types. Return a dictionary whose keys depend on system_type.

For "classification", return accuracy, precision, recall, and F1. Use zero when a precision, recall, or F1 denominator is zero.

\mathrm{accuracy}=\frac{TP+TN}{n}

\mathrm{precision}=\frac{TP}{TP+FP}

\mathrm{recall}=\frac{TP}{TP+FN}

F_1=\frac{2PR}{P+R}

For "regression", return mean absolute error and root mean squared error.

\mathrm{MAE}=\frac{1}{n}\sum_i |y_i-\hat y_i|

\mathrm{RMSE}=\sqrt{\frac{1}{n}\sum_i(y_i-\hat y_i)^2}