When and how to retrain

A printed city map goes out of date as roads are built and shops change hands. The publisher can print a new edition every January, which is predictable and sometimes wasteful, or print one whenever enough changes have piled up, which follows the city more closely and makes the next edition date hard to plan. Either way, a proofreader checks the new edition against the old one before it goes to the shops, and a new edition with a misprinted motorway stays in the warehouse while the old one keeps selling.

A model is the map, the retraining trigger is the printing schedule, and the release gate is the proofreader. The analogy breaks on how the publisher learns about changes, since a surveyor reports a new road the week it opens, while a fraud model learns that a transaction was fraudulent only when the dispute arrives up to 60 days later. That delay in learning the truth, more than the speed of the world, is what sets how often retraining can help.

A scheduled trigger starts a training run at a fixed time, such as every night at two, and it is easy to reason about because everyone knows when the next model will exist. An event trigger starts a run when something happens, such as 100,000 new labelled examples arriving, a confirmed drop in quality on predictions old enough for their labels to have arrived, or a large and verified change in the inputs. Either trigger produces a candidate, which is a trained model that has not yet earned the right to serve, and neither trigger says anything about whether the candidate is better.

Choosing between them is a question about the product. A news feed whose items are stale within a day needs frequent scheduled runs, while a model that predicts house prices can wait for a quarter of new sales and a large move in the market. The schedule or the event rule is written down with the reason, so that a year later nobody wonders why the job runs at two in the morning.

The gap between a prediction and its confirmed outcome caps how current any model can be. Assume fraud disputes take up to 60 days to arrive. A model trained tonight can only use examples whose outcomes are known, so its newest trustworthy example is 60 days old, and a fraud pattern that began last week will not be in any training set for about two months, however often the job runs. Running the job nightly in that setting buys little, and the design effort goes instead into fast monitoring signals and rules that cover the gap.

A product whose labels arrive in seconds, such as whether a user clicked a recommendation, is the opposite case. There the training set can include yesterday, and a daily or even hourly run tracks changing taste, which is why recommendation systems retrain far more often than fraud models do. The sentence to say in a design is how long the labels take and therefore how old the newest example in the training set will be, since that one number explains the whole schedule.

The training window is how far back the examples go. A short window of the last 90 days follows current behavior closely and forgets anything rarer than that, such as the December shopping pattern it has not seen since last year. A long window of two years covers every season and also keeps behavior that no longer happens, such as a product line that was discontinued. The choice is tested by training on each window and checking on a later period that neither saw, rather than by assuming newer is better.

Every new training set is checked for what it contains before training starts, because an outage in the logging, a change in the review policy, or a fraud attack during the window all change its mix without anyone deciding to. Two evaluation sets have two jobs. A fixed set that never changes lets a new model be compared fairly with the ones before it, and a recent set shows how it does on current conditions. Keeping the roles separate means a change in the score can be traced to the model rather than to the test.

Warm-starting means beginning training from the learned numbers, called weights, of the previous model rather than from scratch, which saves computing time and carries forward what the old model learned, including anything it learned wrongly. A fresh start is cleaner when the features or the data have changed substantially and costs more. The choice is made on the time to reach acceptable quality and on whether the result can be reproduced, which matters when a model has to be explained months later.

The release gate is the set of checks a candidate passes before it is allowed to serve, and it compares the candidate with the model in production on the same examples under the same conditions. The checks include the main quality metric, the slices the product cares about, meaning groups such as new users or small markets, serving latency, and compatibility with the current feature definitions and index. Where a threshold depends on the scores, the gate also checks that they are calibrated, so that a score of 0.8 comes true about 80 percent of the time. A candidate that fails any check stays in the registry, which is the catalog of trained models with the data and code that produced each, and the production model keeps serving.

A rejected candidate is a normal outcome and the pipeline records why, since a run of rejections is itself a signal that the data or the code changed. A passing candidate is then put in front of users in stages, in shadow mode and then as a canary, with a stop rule and a compatible set kept for rollback. That process is covered in the fallbacks and rollouts topic and is the same for a retrained model as for a new one.

Frequent retraining raises one more question, which is where the new examples came from. If the production model decided which items users saw, the clicks it collected describe what it showed and not what users wanted, and a model trained on them can repeat the old choices with more confidence. The next topic follows that loop from the model’s decisions to its next training set.

Retraining more often is believed to make the model more current. It cannot make the model more current than the newest confirmed label, so for a fraud model with 60-day disputes a nightly run and a weekly run learn from the same two-month-old world, and the extra runs cost money for nothing.

The other confusions are about the gate. The training job is allowed to promote its own output, when a trigger says when to train and a gate says whether to ship, and a pipeline that joins them ships the model trained on the night the logging broke. The candidate also gets compared with the production model on different test sets, and a higher number on a newer test set says nothing about which model is better, so the fair comparison uses one fixed set for both, with a separate recent set to answer the separate question of how either does today.

At the end of a design, monitoring leads into retraining, and two to three minutes cover it. Say how long the labels take to arrive, choose a schedule or an event trigger with that number as the reason, and say in one sentence that a candidate passes a gate against the production model before it serves. The window choice and warm-starting are deep-dive material and worth a sentence only if the interviewer steers there.