06 · Evaluation and experiments
Error analysis and evaluation slices
3 min read
The metrics introduced so far summarize performance, but the same score can result from very different mistakes. Error analysis examines failed predictions and the circumstances around them to understand what needs to change. Grouping errors by affected users and likely cause helps distinguish a data problem from a model limitation or an unsuitable decision policy.
Inspect examples from the deployed decision
For an ETA system, sample large underestimates and overestimates, then inspect route, traffic freshness, geography, trip length, and the point at which the estimate was made. A missing traffic feature and a genuinely unusual road closure can create similar residuals but require different fixes.
The direction of an error can matter as much as its size. Mean absolute error summarizes how far estimates are from the outcome, while signed errors can reveal a tendency to predict too early or too late. When arriving later than promised has a different cost from arriving early, examining the directions separately gives a more useful account of the failure.
Inspect slices and their sample sizes
A slice is a subset of examples that share a relevant characteristic, such as a language, device type, or missing input. Choosing slices from the application’s known risks helps reveal failures that an overall average can conceal. Each comparison needs its sample size and baseline, because a large percentage change in a small group can be much less reliable than a consistent change across many examples.
Do not search hundreds of slices and treat the worst fluctuation as a confirmed discovery. Use exploratory findings to form a hypothesis, then validate on separate data or a subsequent period. Keep a stable set of important slices in the release checks so known vulnerabilities are not forgotten.
| Observed pattern | Possible explanation | Next check |
|---|---|---|
| Only new users regress | Missing history or weak cold-start handling | Compare feature defaults and baseline fallback. |
| One client version regresses | Logging or preprocessing change | Inspect schema and matched requests. |
| Offline improves, online falls | Exposure shift, skew, or wrong proxy | Check experiment validity and serving values. |
Separate data, model, and policy failures
The appropriate fix depends on where the error originates. Incorrect labels call for a review of how outcomes were assigned, while missing predictive information may require a new feature or better logging. If the necessary information is present but the representation cannot distinguish relevant cases, a modeling change may help. A well-estimated probability followed by an unsuitable threshold instead points to the decision policy.
For a retrieval system, trace an error through eligibility, candidate generation, ranking, and final filtering. An unavailable item leaking through a policy check should not be fixed by teaching the ranker to dislike unavailable items. Enforce hard requirements directly and evaluate the component responsible.
A proposed fix should be checked on targeted failures and the broader evaluation set so an improvement in one area does not conceal a regression elsewhere. Repeatedly used failure examples become development data and still need a separate final evaluation. Once that evidence supports a candidate, the next step is to test whether changing the live application produces the intended benefit for users.