Problems

1 / 1

Shadow Deployment Evaluation

MLOps

Medium

A shadow model runs beside the production model on the same ordered requests, but its predictions are not served. Compare both logs and decide whether the shadow model satisfies every promotion criterion.

\mathrm{accuracy}=\frac{\text{correct predictions}}{n}

\mathrm{accuracy\ gain}=\mathrm{shadow\ accuracy}-\mathrm{production\ accuracy}

For shadow latency, sort latencies and use nearest-rank P95 at index $\lceil0.95n\rceil-1$ . Agreement rate is the fraction of positions where both models predict the same value. Promote only when accuracy gain is at least min_accuracy_gain, P95 latency is at most max_latency_p95, and agreement rate is at least min_agreement_rate. Return {"promote": bool, "metrics": {...}} with all five named metrics shown in the examples.

Loading visualization...

Examples

Input: production_log = [{"input_id": 1, "prediction": 1, "actual": 1, "latency_ms": 15}, {"input_id": 2, "prediction": 0, "actual": 1, "latency_ms": 20}, {"input_id": 3, "prediction": 1, "actual": 1, "latency_ms": 18}, {"input_id": 4, "prediction": 0, "actual": 0, "latency_ms": 22}], shadow_log = [{"input_id": 1, "prediction": 1, "actual": 1, "latency_ms": 10}, {"input_id": 2, "prediction": 1, "actual": 1, "latency_ms": 25}, {"input_id": 3, "prediction": 1, "actual": 1, "latency_ms": 20}, {"input_id": 4, "prediction": 0, "actual": 0, "latency_ms": 30}], criteria = {"min_accuracy_gain": 0, "max_latency_p95": 50, "min_agreement_rate": 0.5}

Output: {"promote": true, "metrics": {"shadow_accuracy": 1, "production_accuracy": 0.75, "accuracy_gain": 0.25, "shadow_latency_p95": 30, "agreement_rate": 0.75}}

Explanation: The shadow model satisfies the gain, latency, and agreement thresholds.

Input: production_log = [{"input_id": 1, "prediction": 1, "actual": 1, "latency_ms": 15}, {"input_id": 2, "prediction": 0, "actual": 1, "latency_ms": 20}, {"input_id": 3, "prediction": 1, "actual": 1, "latency_ms": 18}, {"input_id": 4, "prediction": 0, "actual": 0, "latency_ms": 22}], shadow_log = [{"input_id": 1, "prediction": 1, "actual": 1, "latency_ms": 40}, {"input_id": 2, "prediction": 1, "actual": 1, "latency_ms": 45}, {"input_id": 3, "prediction": 1, "actual": 1, "latency_ms": 50}, {"input_id": 4, "prediction": 0, "actual": 0, "latency_ms": 200}], criteria = {"min_accuracy_gain": 0, "max_latency_p95": 100, "min_agreement_rate": 0.5}

Output: {"promote": false, "metrics": {"shadow_accuracy": 1, "production_accuracy": 0.75, "accuracy_gain": 0.25, "shadow_latency_p95": 200, "agreement_rate": 0.75}}

Hint 1

Use math.ceil(0.95 * n) - 1 after sorting shadow latencies.

Hint 2

Build promote by joining all three threshold comparisons with and.

Requirements

Compute production and shadow accuracy
Compute accuracy gain, nearest-rank shadow P95 latency, and agreement rate
Require all three criteria for promotion
Return promote and the five named metrics

Constraints

Both logs have equal nonzero length and matching order
Corresponding entries describe the same input_id and actual value
Latencies are nonnegative numbers

Try Similar Problems

Monitoring Metrics Selection Data Drift Detection Train Serving Skew Model Versioning Basics Retraining Trigger Design

Prev Next

Shadow Deployment Evaluation

MLOps

Medium

\mathrm{accuracy}=\frac{\text{correct predictions}}{n}

\mathrm{accuracy\ gain}=\mathrm{shadow\ accuracy}-\mathrm{production\ accuracy}

Loading visualization...

Examples

Output: {"promote": true, "metrics": {"shadow_accuracy": 1, "production_accuracy": 0.75, "accuracy_gain": 0.25, "shadow_latency_p95": 30, "agreement_rate": 0.75}}

Explanation: The shadow model satisfies the gain, latency, and agreement thresholds.

Input: production_log = [{"input_id": 1, "prediction": 1, "actual": 1, "latency_ms": 15}, {"input_id": 2, "prediction": 0, "actual": 1, "latency_ms": 20}, {"input_id": 3, "prediction": 1, "actual": 1, "latency_ms": 18}, {"input_id": 4, "prediction": 0, "actual": 0, "latency_ms": 22}], shadow_log = [{"input_id": 1, "prediction": 1, "actual": 1, "latency_ms": 40}, {"input_id": 2, "prediction": 1, "actual": 1, "latency_ms": 45}, {"input_id": 3, "prediction": 1, "actual": 1, "latency_ms": 50}, {"input_id": 4, "prediction": 0, "actual": 0, "latency_ms": 200}], criteria = {"min_accuracy_gain": 0, "max_latency_p95": 100, "min_agreement_rate": 0.5}

Output: {"promote": false, "metrics": {"shadow_accuracy": 1, "production_accuracy": 0.75, "accuracy_gain": 0.25, "shadow_latency_p95": 200, "agreement_rate": 0.75}}

Hint 1

Use math.ceil(0.95 * n) - 1 after sorting shadow latencies.

Hint 2

Build promote by joining all three threshold comparisons with and.

Requirements

Compute production and shadow accuracy
Compute accuracy gain, nearest-rank shadow P95 latency, and agreement rate
Require all three criteria for promotion
Return promote and the five named metrics

Constraints

Both logs have equal nonzero length and matching order
Corresponding entries describe the same input_id and actual value
Latencies are nonnegative numbers

Try Similar Problems

Monitoring Metrics Selection Data Drift Detection Train Serving Skew Model Versioning Basics Retraining Trigger Design

Case 1

Case 2

production_log =

Accepts: array

shadow_log =

Accepts: array

criteria =

Accepts: any

Output: {"promote": true, "metrics": {"shadow_accuracy": 1, "production_accuracy": 0.75, "accuracy_gain": 0.25, "shadow_latency_p95": 30, "agreement_rate": 0.75}}

Explanation: The shadow model satisfies the gain, latency, and agreement thresholds.