Building a baseline

A town wants to hire a weather forecaster. Before paying anyone, the mayor writes down the cheapest rule she can think of, which is that tomorrow will be the same as today. Assume that in this town the rule is right, within a couple of degrees, on about seven days in ten, and that it costs nothing to run.

Any forecaster the town interviews now has a number to beat. A candidate with expensive instruments who is right on seven and a half days in ten has bought the town half a day a week for a salary, and the mayor can decide whether that is worth paying. The learned model is the forecaster, the same-as-today rule is the baseline, and the comparison only counts when both are scored on the same days.

A forecaster works alone, while a model sits inside a system that collects data, prepares inputs and delivers answers, and that is where the analogy stops matching. The baseline is the first thing to run through all of those pipes, and that second job, testing the plumbing, has no counterpart in the weather story, so it gets its own section below.

The reference should match the task and, where possible, the way the decision is made today. For a recommendation system the reference is the list of the most popular items the person is allowed to see. For a delivery time estimate it is the average time this route took in the past. Comparing a learned model against that rule shows whether the extra information the model uses improves a decision the business cares about.

A model that always predicts the most common category is the weakest baseline, and it is useful as a sanity check when one class is rare. If two percent of payments are fraud, a rule that calls every payment honest is 98 percent accurate and catches nothing, which is why accuracy alone is a poor yardstick on rare events. Where humans make the decision today, measure them instead. If a review team already checks payments, record how many they clear in an hour and how often they are wrong, because that is the number the model has to beat.

Train the simple model on a proper split of the data, keeping the examples it is judged on apart from the ones it learns from, run the same input preparation when it answers live requests, and record what it predicted and what then happened. Doing this forces decisions about data formats, version numbers, missing inputs, response time and measurement before a complicated model arrives and hides them. A baseline that takes a week to wire up has told you where the week went, and the learned model will need the same pipes.

The baseline also reads the data for you. If a plain linear model, which adds up weighted inputs, cannot learn a signal that anyone can see in a spreadsheet, the labels, the joins or the transformations are broken somewhere. The opposite result is a warning too. A baseline that is nearly perfect has usually been handed the answer through a leaked input, which is a value that was recorded after the outcome it is meant to predict. A small model makes both problems easy to find because its few inputs can be inspected one by one.

Suppose a logistic regression spam model, which weighs each word and phrase on its own, handles the common phrases but misses the combination of a brand-new sender and a message full of links. A tree model, which sorts examples with a series of yes-or-no questions and can learn such combinations, is a reasonable next experiment. State the limitation the new model is meant to fix before naming the model, because the limitation is the hypothesis and the model is one way to test it.

Compare the two on the same examples and at the same operating point, which is the cutoff where the system starts to act. Suppose the rules baseline flags 1,000 payments a day and 300 of them turn out to be fraud, so three in ten flagged payments are fraud. A candidate model that also flags 1,000 payments but finds 450 cases has caught 150 more frauds a day at the same review workload. If instead it finds 450 cases by flagging 2,000 payments, the gain came from doubling the review team, which is a different decision with a different cost.

The comparison also has to include response time, memory, training cost and the work of maintaining any new inputs. Keep the baseline running after the new model wins, because it remains an understandable reference on the same task and, once tested in that role, a safe thing to fall back on when the new model fails.

Newcomers tend to treat the baseline as a lesser answer to be apologised for. Interviewers hear it the other way round, because a candidate who names a yardstick has shown how every later claim will be measured. The baseline is the first result, and the model is the second.

Beating random guessing is also believed to mean something. On rare events random guessing and the always-honest rule are both easy to beat. The only comparison that matters is against the decision the business makes today, whether that is a rule, a review team or nothing at all.

Dropping the baseline once a better model ships is the confusion that shows up months later. Without it, the next comparison has no fixed reference, and a slow decline in data quality can hide behind two models that both got worse together.

In a 45-minute design round the baseline takes about two minutes, inside the ten minutes given to modelling. Bring it up with one sentence that names the reference and promises that every model you propose will be measured against it on the same data, then move to the model that fixes the baseline’s first failure.

If the interviewer wants to hear the sophisticated model first, name it and promise the baseline comparison in the evaluation phase, because that comparison is what the follow-ups come back to when it is missing.