08 · Monitoring and improvement
Retraining and model updates
3 min read
Monitoring can reveal changes that require a model update after data and service failures have been ruled out. Retraining uses new or revised examples to produce a candidate through the training pipeline established earlier. Deciding when to start that work is separate from deciding whether the resulting model is good enough to replace the current version.
Separate the training trigger from promotion
A scheduled job is easy to reason about and provides predictable freshness. Event-driven training can respond to enough new labels, a detected quality regression, or a meaningful data change. Either trigger creates a candidate; neither should imply that the candidate is automatically better.
The update frequency depends on label delay, how quickly behavior changes, training cost, and release risk. A daily job using labels that take two months to mature can only learn from outcomes old enough to be confirmed, even though the job runs frequently. Relating the schedule to that observation delay explains what kind of adaptation the system can realistically provide.
Balance recency with coverage
A recent rolling window follows current behavior but can forget infrequent patterns or seasonality. A longer window gives more coverage while retaining outdated behavior. You can compare windows or weight examples by recency, but validate the choice on a future period rather than assuming newer is always better.
Each new dataset needs a check of which examples are present, since an incident, review-policy change, or logging outage can alter its composition. A stable evaluation set supports comparison with earlier versions, while a separate recent set helps assess current conditions. Giving each set a clear role makes changes in the model easier to distinguish from changes in the evaluation data.
Warm-starting continues training from the parameters of an earlier model, which can reduce computation but also carries forward what that model learned. Starting a fresh run may be easier to compare when features or data have changed substantially. Evaluating the time needed to reach acceptable quality, along with compatibility and reproducibility, helps determine which approach suits the update.
Promote a complete, compatible version
Compare the candidate with the incumbent on the same data and operating constraints. Include important slices, calibration where needed, serving latency, and compatibility with feature transforms and indexes. A registry should retain the dataset and code lineage so an unexpected change can be investigated.
Use a staged release with explicit stop conditions. Monitor immediate health and later quality, and preserve the previous compatible artifact set. A rollback changes what is served; it does not erase data already collected under the candidate, so keep version identifiers in the logs.
More frequent updates also increase the importance of understanding how the new data was produced. If the current model influenced which outcomes could be observed, retraining may reinforce its previous choices instead of correcting them. The next topic follows this connection between the system’s decisions and the data available for its next update.