Monitoring and drift

The manager of a clothing shop keeps an eye on four kinds of thing. The tills and the doors work. Deliveries arrive and the stock on the shelves is what the labels say. The staff make the recommendations they were trained to make, and the returns desk reports what came back, which it can only do weeks after the sale. A broken till shows within minutes, a bad delivery within a day, and a season of poor recommendations only when the returns pile up in March.

A prediction service has the same four layers, the service, the data, the predictions, and the outcomes, and the same gap between the fast signals and the confirmed ones. Where the shop and the service differ is visibility, because the manager can walk the floor and see a shelf of the wrong size. A model that has started returning the same ten items to everyone looks exactly like a healthy one until someone measures it, so every one of the four layers has to be measured on purpose.

Service metrics say whether the machinery is running, and they include the response time, measured as the p99, meaning the time the slowest one in a hundred requests took, the error rate, how full the machines are, and how often the fallback path was used. Data metrics say whether the inputs look like the inputs the model was trained on, and they include missing fields, the age of each feature, values outside their allowed range, and shifts in the spread of values each input takes. Prediction metrics look at the outputs before anyone knows whether they were right, such as the spread of scores and the number of items flagged or shown. Outcome metrics are the ones the business cares about, such as precision, recall, click rate, or revenue, and they arrive only when the truth does.

Reading the layers side by side is what turns an alert into a diagnosis. A fast service with features that are 30 hours old points to a stalled data pipeline. A doubled review queue right after a threshold change points to the decision rule, and a fall in fraud recall with healthy machines and fresh data points to new behavior the model has not seen. The table below is the version of that reasoning a team writes on the wall.

Drift is the word for the inputs changing shape over time, such as the average basket size rising from 40 to 55 dollars across a year. A second kind, sometimes called concept drift, is the relationship between inputs and outcomes changing, such as a basket of 55 dollars no longer being a sign of a returning customer. Seeing the first does not prove the second. Holiday traffic looks different from an ordinary week in every input and the model may still be doing fine.

A drift alert is a reason to look, and the first look is for breakage rather than for a changed world. A field that is suddenly empty, a price that switched from dollars to cents, or a category list with new spellings each produce a dramatic shift and a trivial fix, and retraining on that data would teach the model the bug. Statistical tests on millions of rows will flag a change too small to matter, so the alert threshold is set on the size of the shift and on whether it touches a feature the model leans on. The comparison is against a sensible reference week rather than against yesterday.

Outcomes arrive late, and the delay sets what monitoring can promise. A fraud label arrives when the cardholder disputes the charge, which can take up to 60 days, so a model that broke on the first of the month has its confirmed recall drop show up at the end of the second month. In those 60 days the only warnings are the fast layers, which is why a service and data check that fires within an hour matters more to the team than a perfect quality metric that fires in two months.

The predictions layer gives the earliest hint about quality. If 2 percent of transactions scored above 0.9 last week and 8 percent do today, four times as many customers are being blocked and someone should know before lunch, whatever the eventual labels say. When the labels do arrive, a cohort, meaning the group of predictions made in one week, is only judged when its labels are complete. Measuring recall on a week whose disputes have not yet come in makes the model look better than it is, and a broken join that drops outcome records makes it look worse.

Each prediction is logged with its model version, its feature versions, and the experiment it belonged to, so a change in any metric can be traced to the release or the pipeline that caused it. An alert is only useful when it names an action, such as rerunning a delayed job or switching back to the previous version. A threshold that fires twice a day with nothing to do is turned off by the people it was meant to help. Slow changes also need a weekly look, since fallback usage or the share of the catalog ever shown can creep for months without a single alert. When that look shows the model no longer fits the task, the next topic covers when and how to train a replacement.

A drift alert gets treated as a retraining trigger. A shift in an input is a reason to look, and the look usually finds a broken pipeline or an expected seasonal change, neither of which a new model fixes. Retraining on data from a broken pipeline teaches the model the breakage.

Measuring quality on the most recent week is the next habit to lose. The labels for that week are incomplete, so recall looks higher than it will be, and a team that celebrates the number on Monday is disappointed in two months. Quality is measured on cohorts whose labels have finished arriving.

Adding alerts for everything defeats the ones that matter. An alert with no action attached trains the team to ignore the channel, and the one that mattered is missed. Each alert names the action, and the number of alerts per week is itself something the team watches.

Interviewers use monitoring to see whether the candidate has thought past launch day, and it takes three to five minutes near the end of a design. Name the four layers in one sentence, then say which one warns fastest for this product and how long the outcome labels take to arrive. If asked how you would respond to an alert, walk the detect, localize, diagnose, respond row and offer the retraining question for the next topic rather than answering it here.