Retraining Trigger Design
Retraining Trigger Design
You are managing an ML system in production. Retraining is expensive but necessary to maintain performance. Design a retraining policy that triggers retraining based on multiple conditions while respecting cost and operational constraints.
Given daily monitoring stats and a configuration, decide which days to trigger retraining.
Trigger Conditions
Retrain if any of the following is true:
-
Drift trigger: drift_score strictly exceeds drift_threshold
-
Performance trigger: performance drops strictly below performance_threshold
-
Staleness trigger: days_since_retrain reaches or exceeds max_staleness
Constraints
A triggered retrain only happens if both constraints are satisfied:
-
Cooldown: at least cooldown days must have passed since the last retrain
-
Budget: remaining budget must be at least retrain_cost
State Management
-
days_since_retrain starts at 0 and increments by 1 each day. It resets to 0 after a retrain.
-
Budget depletes by retrain_cost after each retrain.
-
Cooldown is initially satisfied (the model was trained before the monitoring period).
Return a sorted list of day numbers on which retraining was triggered.
Examples
Input:
daily_stats = [ {"day": 1, "drift_score": 0.1, "performance": 0.95}, {"day": 2, "drift_score": 0.3, "performance": 0.93}, {"day": 3, "drift_score": 0.6, "performance": 0.90}, {"day": 4, "drift_score": 0.2, "performance": 0.94}, ] config = {"drift_threshold": 0.5, "performance_threshold": 0.7, "max_staleness": 30, "cooldown": 1, "retrain_cost": 100, "budget": 500}
Output:
[3]
Day 3: drift_score 0.6 > 0.5 triggers retraining. Cooldown and budget are satisfied.
Input:
daily_stats = [ {"day": 1, "drift_score": 0.8, "performance": 0.90}, {"day": 2, "drift_score": 0.8, "performance": 0.90}, {"day": 3, "drift_score": 0.8, "performance": 0.90}, {"day": 4, "drift_score": 0.8, "performance": 0.90}, ] config = {"drift_threshold": 0.5, "performance_threshold": 0.7, "max_staleness": 30, "cooldown": 3, "retrain_cost": 100, "budget": 500}
Output:
[1, 4]
Day 1: drift triggers retrain. Days 2-3: drift triggers but cooldown blocks (only 1-2 days since last retrain, need 3). Day 4: cooldown satisfied (4 - 1 = 3 >= 3), retrain again.
Hint 1
Track days_since_retrain as a counter that increments each day and resets to 0 on retrain.
Hint 2
Initialize last_retrain_day so that the cooldown constraint is already satisfied on day 1.
Requirements
- Check all three trigger conditions each day (drift, performance, staleness)
- Enforce cooldown and budget constraints before allowing a retrain
- Reset days_since_retrain to 0 and deduct cost after each retrain
- Process days sequentially since state carries across days
Constraints
- 1 ≤ len(daily_stats) ≤ 365
- Days are sequential and 1-indexed
- 0 ≤ drift_score ≤ 1, 0 ≤ performance ≤ 1
- cooldown ≥ 1, max_staleness ≥ 1, retrain_cost > 0, budget ≥ 0
- Time limit: 300 ms
Log in to take notes on this problem
Accepts: array
Accepts: any
Retraining Trigger Design
Retraining Trigger Design
You are managing an ML system in production. Retraining is expensive but necessary to maintain performance. Design a retraining policy that triggers retraining based on multiple conditions while respecting cost and operational constraints.
Given daily monitoring stats and a configuration, decide which days to trigger retraining.
Trigger Conditions
Retrain if any of the following is true:
-
Drift trigger: drift_score strictly exceeds drift_threshold
-
Performance trigger: performance drops strictly below performance_threshold
-
Staleness trigger: days_since_retrain reaches or exceeds max_staleness
Constraints
A triggered retrain only happens if both constraints are satisfied:
-
Cooldown: at least cooldown days must have passed since the last retrain
-
Budget: remaining budget must be at least retrain_cost
State Management
-
days_since_retrain starts at 0 and increments by 1 each day. It resets to 0 after a retrain.
-
Budget depletes by retrain_cost after each retrain.
-
Cooldown is initially satisfied (the model was trained before the monitoring period).
Return a sorted list of day numbers on which retraining was triggered.
Examples
Input:
daily_stats = [ {"day": 1, "drift_score": 0.1, "performance": 0.95}, {"day": 2, "drift_score": 0.3, "performance": 0.93}, {"day": 3, "drift_score": 0.6, "performance": 0.90}, {"day": 4, "drift_score": 0.2, "performance": 0.94}, ] config = {"drift_threshold": 0.5, "performance_threshold": 0.7, "max_staleness": 30, "cooldown": 1, "retrain_cost": 100, "budget": 500}
Output:
[3]
Day 3: drift_score 0.6 > 0.5 triggers retraining. Cooldown and budget are satisfied.
Input:
daily_stats = [ {"day": 1, "drift_score": 0.8, "performance": 0.90}, {"day": 2, "drift_score": 0.8, "performance": 0.90}, {"day": 3, "drift_score": 0.8, "performance": 0.90}, {"day": 4, "drift_score": 0.8, "performance": 0.90}, ] config = {"drift_threshold": 0.5, "performance_threshold": 0.7, "max_staleness": 30, "cooldown": 3, "retrain_cost": 100, "budget": 500}
Output:
[1, 4]
Day 1: drift triggers retrain. Days 2-3: drift triggers but cooldown blocks (only 1-2 days since last retrain, need 3). Day 4: cooldown satisfied (4 - 1 = 3 >= 3), retrain again.
Hint 1
Track days_since_retrain as a counter that increments each day and resets to 0 on retrain.
Hint 2
Initialize last_retrain_day so that the cooldown constraint is already satisfied on day 1.
Requirements
- Check all three trigger conditions each day (drift, performance, staleness)
- Enforce cooldown and budget constraints before allowing a retrain
- Reset days_since_retrain to 0 and deduct cost after each retrain
- Process days sequentially since state carries across days
Constraints
- 1 ≤ len(daily_stats) ≤ 365
- Days are sequential and 1-indexed
- 0 ≤ drift_score ≤ 1, 0 ≤ performance ≤ 1
- cooldown ≥ 1, max_staleness ≥ 1, retrain_cost > 0, budget ≥ 0
- Time limit: 300 ms
Log in to take notes on this problem
Accepts: array
Accepts: any