A shadow model runs beside the production model on the same ordered requests, but its predictions are not served. Compare both logs and decide whether the shadow model satisfies every promotion criterion.
accuracy=ncorrect predictions accuracy gain=shadow accuracy−production accuracyFor shadow latency, sort latencies and use nearest-rank P95 at index ⌈0.95n⌉−1. Agreement rate is the fraction of positions where both models predict the same value. Promote only when accuracy gain is at least min_accuracy_gain, P95 latency is at most max_latency_p95, and agreement rate is at least min_agreement_rate. Return {"promote": bool, "metrics": {...}} with all five named metrics shown in the examples.
Input: production_log = [{"input_id": 1, "prediction": 1, "actual": 1, "latency_ms": 15}, {"input_id": 2, "prediction": 0, "actual": 1, "latency_ms": 20}, {"input_id": 3, "prediction": 1, "actual": 1, "latency_ms": 18}, {"input_id": 4, "prediction": 0, "actual": 0, "latency_ms": 22}], shadow_log = [{"input_id": 1, "prediction": 1, "actual": 1, "latency_ms": 10}, {"input_id": 2, "prediction": 1, "actual": 1, "latency_ms": 25}, {"input_id": 3, "prediction": 1, "actual": 1, "latency_ms": 20}, {"input_id": 4, "prediction": 0, "actual": 0, "latency_ms": 30}], criteria = {"min_accuracy_gain": 0, "max_latency_p95": 50, "min_agreement_rate": 0.5}
Output: {"promote": true, "metrics": {"shadow_accuracy": 1, "production_accuracy": 0.75, "accuracy_gain": 0.25, "shadow_latency_p95": 30, "agreement_rate": 0.75}}
Explanation: The shadow model satisfies the gain, latency, and agreement thresholds.
Input: production_log = [{"input_id": 1, "prediction": 1, "actual": 1, "latency_ms": 15}, {"input_id": 2, "prediction": 0, "actual": 1, "latency_ms": 20}, {"input_id": 3, "prediction": 1, "actual": 1, "latency_ms": 18}, {"input_id": 4, "prediction": 0, "actual": 0, "latency_ms": 22}], shadow_log = [{"input_id": 1, "prediction": 1, "actual": 1, "latency_ms": 40}, {"input_id": 2, "prediction": 1, "actual": 1, "latency_ms": 45}, {"input_id": 3, "prediction": 1, "actual": 1, "latency_ms": 50}, {"input_id": 4, "prediction": 0, "actual": 0, "latency_ms": 200}], criteria = {"min_accuracy_gain": 0, "max_latency_p95": 100, "min_agreement_rate": 0.5}
Output: {"promote": false, "metrics": {"shadow_accuracy": 1, "production_accuracy": 0.75, "accuracy_gain": 0.25, "shadow_latency_p95": 200, "agreement_rate": 0.75}}
Use math.ceil(0.95 * n) - 1 after sorting shadow latencies.
Build promote by joining all three threshold comparisons with and.
Accepts: array
Accepts: array
Accepts: any
A shadow model runs beside the production model on the same ordered requests, but its predictions are not served. Compare both logs and decide whether the shadow model satisfies every promotion criterion.
accuracy=ncorrect predictions accuracy gain=shadow accuracy−production accuracyFor shadow latency, sort latencies and use nearest-rank P95 at index ⌈0.95n⌉−1. Agreement rate is the fraction of positions where both models predict the same value. Promote only when accuracy gain is at least min_accuracy_gain, P95 latency is at most max_latency_p95, and agreement rate is at least min_agreement_rate. Return {"promote": bool, "metrics": {...}} with all five named metrics shown in the examples.
Input: production_log = [{"input_id": 1, "prediction": 1, "actual": 1, "latency_ms": 15}, {"input_id": 2, "prediction": 0, "actual": 1, "latency_ms": 20}, {"input_id": 3, "prediction": 1, "actual": 1, "latency_ms": 18}, {"input_id": 4, "prediction": 0, "actual": 0, "latency_ms": 22}], shadow_log = [{"input_id": 1, "prediction": 1, "actual": 1, "latency_ms": 10}, {"input_id": 2, "prediction": 1, "actual": 1, "latency_ms": 25}, {"input_id": 3, "prediction": 1, "actual": 1, "latency_ms": 20}, {"input_id": 4, "prediction": 0, "actual": 0, "latency_ms": 30}], criteria = {"min_accuracy_gain": 0, "max_latency_p95": 50, "min_agreement_rate": 0.5}
Output: {"promote": true, "metrics": {"shadow_accuracy": 1, "production_accuracy": 0.75, "accuracy_gain": 0.25, "shadow_latency_p95": 30, "agreement_rate": 0.75}}
Explanation: The shadow model satisfies the gain, latency, and agreement thresholds.
Input: production_log = [{"input_id": 1, "prediction": 1, "actual": 1, "latency_ms": 15}, {"input_id": 2, "prediction": 0, "actual": 1, "latency_ms": 20}, {"input_id": 3, "prediction": 1, "actual": 1, "latency_ms": 18}, {"input_id": 4, "prediction": 0, "actual": 0, "latency_ms": 22}], shadow_log = [{"input_id": 1, "prediction": 1, "actual": 1, "latency_ms": 40}, {"input_id": 2, "prediction": 1, "actual": 1, "latency_ms": 45}, {"input_id": 3, "prediction": 1, "actual": 1, "latency_ms": 50}, {"input_id": 4, "prediction": 0, "actual": 0, "latency_ms": 200}], criteria = {"min_accuracy_gain": 0, "max_latency_p95": 100, "min_agreement_rate": 0.5}
Output: {"promote": false, "metrics": {"shadow_accuracy": 1, "production_accuracy": 0.75, "accuracy_gain": 0.25, "shadow_latency_p95": 200, "agreement_rate": 0.75}}
Use math.ceil(0.95 * n) - 1 after sorting shadow latencies.
Build promote by joining all three threshold comparisons with and.
Accepts: array
Accepts: array
Accepts: any