Data splits and leakage

A football pundit who writes a prediction for Saturday’s match on Sunday morning, after reading the result, will be right every week. The column looks brilliant and it is worthless, because the information that made it right did not exist when the prediction was supposed to be made. Leakage is this mistake happening inside a data pipeline, where nobody notices because nobody is reading the Sunday paper on purpose.

Everything the pundit knew on Saturday morning is allowed, including every past result, the team sheets and the weather. Everything that happened after kick-off is off limits, however tempting. The dividing line is a moment in time. The discipline of this topic is drawing that line for every training example and checking that nothing from the wrong side crossed it.

The pundit knows what they read, while a pipeline does not know what it joined, and that is why the analogy breaks. A table of customer statistics updated every Friday carries no sign that it includes Thursday’s refund, so the check has to be built into how training rows are constructed rather than left to someone noticing. The running example from here on is the payment fraud system of the earlier topics.

The question to ask of every input is whether its value was known at the moment the prediction would have been made. A payment made on Monday must be described with what the system knew on Monday, and a customer summary such as ‘disputes in the last 90 days’ that was computed on Friday may already include the dispute this very payment caused. Joining Friday’s summary onto Monday’s row hands the model the answer, even though the payments table itself was split cleanly by date.

There is a second, subtler boundary, which is the gap between when something happened and when the system could see it. A card payment at 9.59 am that reaches the feature pipeline at 10.05 am was not available to a prediction made at 10.00 am, so a training row for that prediction must leave it out even though it happened first. Rebuilding history faithfully means using the arrival time, the moment the value became visible, alongside the time the event occurred.

The practical tool is a point-in-time join, which means that for each training row you look up the version of each feature that was current at that row’s prediction time rather than the latest version in the table. It is more expensive than a plain join because the feature tables have to keep their history, and that cost is the reason feature history is a design requirement rather than a nicety.

A split is the division of the dataset into the part the model learns from and the parts used to tune and judge it. For a system that predicts the future, split by time, so the model trains on the oldest months, is tuned on the month after, and is judged on the month after that, because that is the situation the live model will be in. A random shuffle lets Tuesday’s row sit in the training set while Monday’s row is in the test set, so the model is judged on a past it has already partly seen, which is the pundit reading Sunday’s paper by another route.

Leave a gap between the training and test periods when labels take time to arrive or when features are built over windows. With a 60-day dispute window, the last 60 days of the training period contain payments whose labels are still unknown, and a feature counting the last seven days of activity for the first test row reaches back into training data. A gap the width of the longest window keeps the two sides from touching.

The split also has to match who the model will meet. If the live model will serve returning customers, their earlier history in the training set is fair, and a split by time is enough. If it must also serve brand-new customers, keep a set of customers entirely out of training so you can measure how it does with no history, and check for near-duplicates such as the same image under two ids, which cross a split unnoticed.

Preprocessing steps also learn from data, and they leak when they learn from the test set. Scaling a feature to a standard range uses the mean of the column, filling missing values uses an average, and building a vocabulary of categories uses the set of values seen. Each of these must be computed on the training part only and then applied unchanged to the tuning and test parts. Target encoding, which replaces a category by the average label of its rows, is the riskiest, because it writes the label into a feature, and it needs the training set itself to be split further so that no row sees its own label.

The test set leaks in a second way, through the people using it. Every time you try a change and keep it because the test score went up, the test set has shaped the model a little, and after fifty such decisions it is part of the training process. Use the tuning set for choosing cutoffs, settings and calibration, and open the test set once at the end, or better still, judge the final model on a period that arrived after all the decisions were made.

A single new feature that lifts a metric from 0.80 to 0.97 is more likely a leak than a discovery. Look at where the feature comes from and when its value is written, remove it and measure on a later period, and trace a handful of rows back to the raw events to see what the model was shown. A field that is only filled in after a human review, such as a review decision code, is a classic case, and so is a missing value that is itself a signal, because the field stays empty until someone has investigated.

The explanation an interviewer wants names the information that could cross the line and the check that stops it, which for a historical join is a point-in-time lookup plus a few sample rows inspected by hand against the raw events. With that boundary in place, the next question is which of the inputs that are available in time are worth computing, which is the subject of the features topic.

The two confusions about timing are both about where the boundary sits. A time-based split is believed to remove leakage, when a split separates rows from each other and most leaks live inside a row, in a feature that was computed with knowledge of that row’s own future, so the split and the point-in-time join solve different problems and both are needed. The time an event happened also gets treated as the time it was available. The pipeline that aggregates events runs on a schedule, so a value that happened at 9.59 am may not be visible until 10.05 am, and a training row that uses it at 10.00 am is using the future.

The remaining confusion is about results rather than timing, and it is celebrating a jump in offline numbers. Honest improvements from one feature are usually small, and a large jump should start an investigation rather than a deployment.

Whenever you describe a feature built from history, interviewers test it with a feature that sounds plausible and is computed too late, and that exchange takes two or three minutes. Say the rule once, out loud, when you first mention features. For example, ‘every feature is computed with a point-in-time join so it only uses what was known at the prediction moment, and I would split by time with a gap for the label delay’. From then on, each feature you name can be checked against that rule.