Logging training examples

A pizza shop pushes leaflets through letterboxes on a Tuesday and wants to know which streets are worth leafleting again. Orders arrive over the following week, some by phone and some through the app, and a few customers order twice. To learn anything, the shop has to have written down which doors got a leaflet, on which day, and with which offer printed on it, and then match each order to a door.

Two mistakes ruin the exercise. If the shop forgot to record which doors it covered, it cannot tell a street that ignored the leaflet from a street that never received one, and both look the same in the order book. If it closes the books on Thursday, it misses the weekend orders and concludes that the leaflet failed. An ML system that logs its decisions faces the same two problems, with millions of letterboxes and a model deciding which leaflet each one gets.

A leaflet is one offer through one door, while an app shows ten items at once in a ranked list, and the item at the top is seen far more than the item at the bottom, which is where the leaflet stops matching the feed. That is why the record of an exposure has to include where on the screen the item appeared, which the leaflet shop never needed to write down.

An impression is the record that an item appeared on a user’s screen, and it is the unit everything else attaches to. The model may have scored two hundred items for one request and shown ten, and the one hundred and ninety that were never displayed tell you nothing about the user’s taste. A missing click on an item the user never saw is not a rejection, so the dataset can only use exposures that were visible. The design has to decide what visible means, for example whether an item drawn below the bottom of the screen counts.

Each impression gets its own identifier and carries the context of the moment. That context is the request id, a user id that does not reveal who the person is, the item id, the position on the screen and the time it was shown. It also records which test group the user was in when two versions of the product are being compared, which version of the model produced the ranking, and the inputs the model used or a pointer to a saved copy of them. When a click or a purchase arrives later, it refers to that impression id, so the system knows it was this appearance of the item, in this position, under this model, that led to the action.

Three kinds of record result, and they are written at different moments. The impression is written when the screen is drawn, the prediction record is written when the model scores, and the outcome record is written whenever the user acts, which may be seconds or days later. Keeping them as separate records joined by ids, instead of one row that is edited as events arrive, lets the dataset builder redo the join when a late event shows up.

Events do not arrive in the order they happened, and some arrive twice. A phone that lost signal uploads a click an hour later, and a client that did not get an acknowledgement sends the same event again. The pipeline, meaning the chain of jobs that builds training rows, has to tolerate both, which it does with two small design choices made at logging time rather than afterwards.

The first choice is that every event carries a stable identifier, so the second copy of a retried event can be recognised and dropped. The second is that every event carries two timestamps, the time it happened on the device and the time it reached the servers. A click that happened at 8.02 pm and arrived at 9.15 pm must count as an 8.02 pm click for the label, and the pipeline must also be able to see that it arrived late. That is the pizza shop keeping its order book open past Thursday. Without the second timestamp, the builder cannot tell a late event from a new one.

The waiting window is the agreed time the builder waits before calling an exposure a negative, meaning an example where the user saw the item and did nothing. For a video feed you might say that a displayed video is a positive if a watch of at least thirty seconds is recorded within the same session, meaning the same visit to the app. It becomes a negative only after the session has ended and a further day has passed for stragglers. The window is part of the definition of the label, so changing it from one day to three produces a different dataset and must be recorded as a new version.

The join itself, the step that matches each outcome to its impression, needs a health check, because a broken join looks exactly like a change in user behaviour. Count the outcomes that found no impression, the impressions that have no prediction record, and the duplicate ids per day. If an app release stops sending the impression id with clicks, the match rate falls from, say, 98 percent to 60 percent overnight, the dataset fills with missing positives, and a model trained on it looks worse although nobody’s behaviour changed.

Logging everything forever is the wrong default, because it costs money and it stores personal information you may be obliged to delete. Decide what the records are for, which is training, evaluation, investigating an incident, and honouring a deletion request, and keep the fields those four uses need. Assume a feed that shows 50 million impressions a day at roughly 500 bytes each, which is about 25 gigabytes a day and 9 terabytes a year before compression. That is large enough that a one-year retention rule and a smaller long-term sample are a design decision rather than an afterthought.

Give the event schema, meaning the agreed list of fields and their types, a named owner and a version number. Check required fields and plausible ranges when events are received, so a position of minus one or a timestamp from 1970 is rejected at the door, and watch the rate of missing values downstream. A schema check catches a field that disappears, and a check on the distribution of values catches a field that is still present but has changed meaning, such as a duration that switched from milliseconds to seconds.

The test of a logging design is to trace one impression by hand through the three records and see whether you can rebuild what the model saw and what happened next. If you can, the system has training data. What it does not yet have is a decision about what the outcome means when it arrives late, is ambiguous, or never arrives at all, and that decision is the subject of the next topic, labels.

Treating every item the user did not click as a rejection is the confusion that does the most damage. The model only learns about items that were shown, and a catalogue of a million items contains 999,990 that this user never saw. Training on those as negatives teaches the model that unseen items are unwanted. In practice it learns which items were unpopular with the previous model.

The click table also gets mistaken for the training set. A click table holds positives and nothing else, and a model needs to see the exposures that led nowhere to learn the difference. Without impressions there are no negatives, and without positions there is no way to tell a click earned by a good match from a click earned by being first on the screen.

Events are assumed to arrive once and on time, until the first Friday evening of clicks shows up on Monday and the weekend’s dataset has already been built. Two timestamps per event and a stated waiting window cost almost nothing when designed in and are painful to add later.

Of all the data topics, logging is the one candidates skip most, and it deserves two or three minutes inside the data part of a 45-minute design interview. Bring it up yourself the moment you name the label, with a sentence such as ‘to build that label I would log every impression with its position and model version, and join later actions to it by id’. Interviewers then tend to ask about negatives, late events, or the join rate, and the sections above are the answers.