08 · Monitoring and improvement
Monitoring and drift
3 min read
The serving topics established how to return predictions and handle immediate failures. Monitoring checks whether that behavior remains acceptable as traffic, data, and user needs change. By observing the service, its inputs, its predictions, and their eventual outcomes, we can begin to distinguish an infrastructure failure from a problem with data or model quality.
Monitor four layers of the system
Service metrics include latency, errors, saturation, and fallback rate. Data metrics include missing fields, feature age, schema violations, and distribution changes. Prediction metrics include score distributions and action volume. Outcome metrics include precision, recall, business results, and important user segments once labels are available.
Comparing these layers helps narrow down the reason for a change. A fast endpoint with stale features suggests a stalled data pipeline, while rising review volume after a threshold update points toward the decision policy. A decline in fraud recall with healthy inputs and infrastructure may instead require investigating new behavior that the model does not recognize.
| Layer | Example signal | Possible response |
|---|---|---|
| Service | p99 latency or fallback rate rises | Inspect capacity and dependencies. |
| Data | Feature age or missingness rises | Inspect ingestion and materialization. |
| Predictions | Alert volume suddenly doubles | Check threshold, version, and traffic mix. |
| Outcomes | Recall falls on mature cohorts | Investigate labels, drift, and failure slices. |
A distribution shift is evidence to investigate
Input drift means the distribution of inputs has changed. A relationship between inputs and outcomes can also change, sometimes called concept drift. Observing the first does not establish the second. Holiday traffic may legitimately differ from an ordinary week while model quality remains acceptable.
Compare sensible reference windows and segments, and account for seasonality. Missing values, changed units, or a vocabulary mismatch should trigger a data-integrity investigation before retraining. Statistical tests can flag tiny, harmless changes in huge datasets, so pair them with magnitude and product relevance.
Separate fast warnings from confirmed quality
When outcomes arrive weeks after a prediction, immediate service and data checks provide earlier warnings than confirmed quality metrics. Evaluating mature cohorts later lets those warnings be checked against observed outcomes. Label completeness and arrival delays also need monitoring, because a broken join can make prediction quality appear to decline when the outcome records are actually missing.
Keeping the model version, feature definitions, and experiment assignment with the relevant records makes a change easier to trace to its origin. An alert is useful when someone can investigate it and take a defined action, such as repairing a delayed pipeline or returning to an earlier version. Choosing thresholds with that response in mind helps avoid repeated alerts that provide little useful information.
Slower changes also need review, because fallback usage or gaps in item coverage can grow without a sudden outage. Once an investigation shows that the current model no longer fits the task well, creating an updated candidate becomes one possible response. The next topic examines how to make that update without treating a monitoring signal as automatic approval for a release.