Problems

1 / 1

Monitoring Metrics Selection

MLOps

Hard

You are setting up monitoring for deployed ML systems. Different system types require different metrics. Accuracy alone does not capture the full picture.

Given a system type and ground truth / prediction arrays, compute the appropriate set of monitoring metrics.

Metric Sets

classification (binary, y_true and y_pred are 0 or 1):

\text{accuracy} = \frac{TP + TN}{n}

\text{precision} = \frac{TP}{TP + FP} \quad\quad \text{recall} = \frac{TP}{TP + FN}

\text{f1} = \frac{2 \cdot \text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}}

regression (y_true and y_pred are floats):

\text{mae} = \frac{1}{n} \sum_{i} |y_i - \hat{y}_i| \quad\quad \text{rmse} = \sqrt{\frac{1}{n} \sum_{i} (y_i - \hat{y}_i)^2}

ranking (y_true is binary relevance, y_pred is predicted scores):

\text{precision\_at\_3} = \frac{\text{relevant items in top 3}}{3} \quad\quad \text{recall\_at\_3} = \frac{\text{relevant items in top 3}}{\text{total relevant}}

Return a sorted list of (metric_name, value) tuples, sorted alphabetically by name. Handle division by zero by returning 0.0.

Loading visualization...

Examples

Input:

system_type = "classification" y_true = [1, 0, 1, 1, 0, 1, 0, 0] y_pred = [1, 0, 0, 1, 0, 1, 1, 0]

Output:

[("accuracy", 0.75), ("f1", 0.75), ("precision", 0.75), ("recall", 0.75)]

TP=3, FP=1, FN=1, TN=3. All four classification metrics are computed and returned sorted by name.

Input:

system_type = "regression" y_true = [3.0, 5.0, 2.5, 7.0] y_pred = [2.5, 5.5, 2.0, 8.0]

Output:

[("mae", 0.625), ("rmse", 0.6614)]

Errors are [0.5, 0.5, 0.5, 1.0]. MAE = 0.625, RMSE is approximately 0.6614. Regression uses different metrics than classification.

Hint 1

Use if/elif to branch on system_type and compute only the relevant metrics for each.

Hint 2

For ranking, pair up predictions with labels using zip(), then sort by predicted score descending before slicing the top k.

Requirements

Route to the correct metric set based on the system_type argument
Compute all metrics for that system type
Return 0.0 for any metric that would involve division by zero
Return a list of (name, value) tuples sorted alphabetically by name

Constraints

system_type is one of "classification", "regression", "ranking"
len(y_true) == len(y_pred), and len >= 3
For classification: y_true and y_pred contain only 0 and 1
For ranking: y_true is binary (0/1), y_pred contains distinct float scores
Time limit: 300 ms

Try Similar Problems

Data Drift Detection Train Serving Skew Shadow Deployment Evaluation Retraining Trigger Design Model Versioning Basics

Prev Next

Monitoring Metrics Selection

MLOps

Hard

You are setting up monitoring for deployed ML systems. Different system types require different metrics. Accuracy alone does not capture the full picture.

Given a system type and ground truth / prediction arrays, compute the appropriate set of monitoring metrics.

Metric Sets

classification (binary, y_true and y_pred are 0 or 1):

\text{accuracy} = \frac{TP + TN}{n}

\text{precision} = \frac{TP}{TP + FP} \quad\quad \text{recall} = \frac{TP}{TP + FN}

\text{f1} = \frac{2 \cdot \text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}}

regression (y_true and y_pred are floats):

\text{mae} = \frac{1}{n} \sum_{i} |y_i - \hat{y}_i| \quad\quad \text{rmse} = \sqrt{\frac{1}{n} \sum_{i} (y_i - \hat{y}_i)^2}

ranking (y_true is binary relevance, y_pred is predicted scores):

\text{precision\_at\_3} = \frac{\text{relevant items in top 3}}{3} \quad\quad \text{recall\_at\_3} = \frac{\text{relevant items in top 3}}{\text{total relevant}}

Return a sorted list of (metric_name, value) tuples, sorted alphabetically by name. Handle division by zero by returning 0.0.

Loading visualization...

Examples

Input:

system_type = "classification" y_true = [1, 0, 1, 1, 0, 1, 0, 0] y_pred = [1, 0, 0, 1, 0, 1, 1, 0]

Output:

[("accuracy", 0.75), ("f1", 0.75), ("precision", 0.75), ("recall", 0.75)]

TP=3, FP=1, FN=1, TN=3. All four classification metrics are computed and returned sorted by name.

Input:

system_type = "regression" y_true = [3.0, 5.0, 2.5, 7.0] y_pred = [2.5, 5.5, 2.0, 8.0]

Output:

[("mae", 0.625), ("rmse", 0.6614)]

Errors are [0.5, 0.5, 0.5, 1.0]. MAE = 0.625, RMSE is approximately 0.6614. Regression uses different metrics than classification.

Hint 1

Use if/elif to branch on system_type and compute only the relevant metrics for each.

Hint 2

For ranking, pair up predictions with labels using zip(), then sort by predicted score descending before slicing the top k.

Requirements

Route to the correct metric set based on the system_type argument
Compute all metrics for that system type
Return 0.0 for any metric that would involve division by zero
Return a list of (name, value) tuples sorted alphabetically by name

Constraints

system_type is one of "classification", "regression", "ranking"
len(y_true) == len(y_pred), and len >= 3
For classification: y_true and y_pred contain only 0 and 1
For ranking: y_true is binary (0/1), y_pred contains distinct float scores
Time limit: 300 ms

Try Similar Problems

Data Drift Detection Train Serving Skew Shadow Deployment Evaluation Retraining Trigger Design Model Versioning Basics

Case 1

Case 2

system_type =

Accepts: string

y_true =

Accepts: array

y_pred =

Accepts: array

You are setting up monitoring for deployed ML systems. Different system types require different metrics. Accuracy alone does not capture the full picture.

Given a system type and ground truth / prediction arrays, compute the appropriate set of monitoring metrics.

Metric Sets

classification (binary, y_true and y_pred are 0 or 1):

\text{accuracy} = \frac{TP + TN}{n}

\text{precision} = \frac{TP}{TP + FP} \quad\quad \text{recall} = \frac{TP}{TP + FN}

\text{f1} = \frac{2 \cdot \text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}}

regression (y_true and y_pred are floats):

\text{mae} = \frac{1}{n} \sum_{i} |y_i - \hat{y}_i| \quad\quad \text{rmse} = \sqrt{\frac{1}{n} \sum_{i} (y_i - \hat{y}_i)^2}

ranking (y_true is binary relevance, y_pred is predicted scores):

\text{precision\_at\_3} = \frac{\text{relevant items in top 3}}{3} \quad\quad \text{recall\_at\_3} = \frac{\text{relevant items in top 3}}{\text{total relevant}}

Return a sorted list of (metric_name, value) tuples, sorted alphabetically by name. Handle division by zero by returning 0.0.