Data splits and leakage

A payment from Monday must be represented using what the service knew on Monday. A customer aggregate computed on Friday may include a later refund or dispute. Joining that aggregate to Monday’s training row introduces future information even if the raw payment table is correctly split by date.

The time an event occurred may differ from the time it became available to the model. A transaction could occur at 9.59 am and reach the feature pipeline at 10.05 am, making it unavailable to a prediction at 10 am. Reconstructing the historical input therefore requires considering arrival time and when processing made the value accessible, as well as the original event time.

For a system repeatedly predicting future events, a chronological split is usually a better starting point than a random shuffle. Train on older data, tune on a later validation period, and hold out a final future period. Leave a gap when labels or overlapping windows would otherwise leak across the boundary.

The split also depends on whether the application will serve familiar or previously unseen users and items. Earlier activity from returning users may be a realistic input, while evaluating generalization to new users requires keeping those users separate between datasets. Near-duplicate images or copied text can still cross the boundary under different identifiers, so checking content overlap matters as well as separating records.

Fit normalization, imputation, vocabulary construction, feature selection, and target encoding on training data only, with additional fold separation where target encoding requires it. Apply the fitted transforms to validation and test data. A pipeline abstraction helps prevent accidental fitting on the full dataset, but it cannot fix an outcome-derived raw feature.

Treat the test set as a final check, not a tuning dashboard. Repeatedly selecting models based on the same test result gradually makes it part of development. Maintain separate validation data for threshold selection, hyperparameters, and calibration, then use a held-out test or forward evaluation for the final comparison.

If adding one feature produces an enormous improvement, inspect its provenance and timestamp before celebrating. Remove it, evaluate on a later period, and trace a few individual rows back to source events. Check whether missingness itself reveals an outcome, such as a field populated only after manual review.

An explanation of leakage should identify the information that could cross the prediction boundary and the check that prevents it. For a historical join, that means limiting values to those available before the request and inspecting sample rows to verify the result. Once this boundary is clear, we can decide which of the available inputs are useful features and how to compute them.