05 · Models and training
Choosing a model and loss
3 min read
The baseline gives us a reference and evidence about what the current approach misses. Model selection uses that evidence to choose a structure capable of learning the relevant patterns, while the loss defines how training penalizes errors. Both choices need to fit the available data and the time and resources allowed for prediction.
Match the model to the information
For structured payment features, a linear classifier can provide an interpretable starting point, while a boosted-tree model can learn nonlinear interactions between those inputs. Text or images may benefit from a pretrained representation that captures patterns difficult to describe with hand-built features. These possibilities need comparison on the actual data because the input structure suggests candidates without establishing a winner.
For recommendation retrieval, a two-tower model allows item representations to be computed in advance. A model that jointly processes a user-item pair may capture richer interactions, but evaluating it against every item can be too expensive. That motivates a retrieval stage followed by a smaller ranking stage.
| Choice | Reason to consider it | Question to resolve |
|---|---|---|
| Linear model | Fast baseline with simple feature effects | Are important interactions represented? |
| Boosted trees | Nonlinear structured-data patterns | Does serving meet the latency budget? |
| Two-tower retrieval | Precomputed item vectors | Do candidates retain relevant items? |
| Joint ranker | Richer query-item interactions | How many pairs can be scored? |
Explain the target and loss together
A binary event such as a click can use binary cross-entropy to fit probabilities. A continuous ETA target needs a regression loss whose error penalties fit the task. Mean squared error emphasizes large residuals; absolute error is less sensitive to very large errors. If the product needs an interval or a high-percentile estimate, a point prediction alone may not be enough.
Ranking objectives can learn from individual relevance labels, preferences between item pairs, or entire lists. State how examples are constructed and why the loss should improve the ranking you evaluate. The objective cannot compensate for a dataset that treats unexposed items as confirmed negatives without justification.
Evaluate the complete operating point
Suppose a larger ranker improves an offline metric by a small amount but doubles serving cost. Check whether the gain persists across time and important user segments, whether it changes the online outcome, and whether a smaller candidate set, distillation, or better features can recover most of the benefit.
Model size also affects deployment time, hardware choice, cold starts, and rollback speed. A model that depends on a new real-time feature introduces an additional service even if the model artifact itself is small. Compare the whole request path and training process rather than the model benchmark in isolation.
A model choice should therefore remain a hypothesis until it has been tested against the baseline under the same conditions. State which limitation the alternative is intended to address and which measurements would justify adopting it. To make that comparison repeatable, we next need a training pipeline that records how each candidate was produced.