Choosing features

A librarian recommending a book to a regular visitor on a Friday afternoon draws on five kinds of knowledge without thinking about it. She knows the book, its subject and its length. She knows the visitor, roughly their age and that they read in English. She remembers what they borrowed before and whether they finished it, she knows their book group has been reading Scandinavian crime, and she knows it is the Friday before a long weekend, when people want something thick.

Those five kinds of knowledge are the five sources of features in almost every ML system. They are the item, the person, the person’s past behaviour, the people and things they are connected to, and the context of the request. Organising features by source stops the two common failures, which are forgetting a whole source and reciting fifty features with no reason attached to any of them.

The librarian can ask a question and get an answer in two seconds, and that is where the analogy breaks, because the model gets no conversation and must have every value fetched from stored records inside its time budget, which for a fraud check at checkout is about 100 milliseconds. The technical sections use the payment fraud system from the data topics, where the model has that long to decide whether to allow a card payment.

Item features describe the thing being decided about, which for a payment is the amount, the currency, the merchant and the type of goods. They are available in the request itself and cost nothing to fetch. They are also the easiest for a fraudster to choose, so they are necessary and never enough. The librarian’s equivalent is knowing the book without knowing the reader.

Actor features describe who is acting, such as how old the account is, whether the email address is at a disposable provider, and how many cards are on file. Behaviour features describe what the actor has done over time, and the clearest example is the number of payment attempts on this card in the last five minutes, which must be computed from recorded events and kept fresh to the minute. These two sources are where most of the predictive power in fraud systems lives, and they are also where the leakage questions from the previous topic have to be asked.

Network features describe connections, such as how many other accounts have used this device or this card. Context features describe the moment, such as the hour of the day, whether the shipping country matches the card’s country, and which device type made the request. Two or three features per source, each with the sentence that says why it should matter, is a complete answer in a design discussion, and the table below is that answer for the payment example.

A feature definition is a short written contract that says whose information it describes, how it is calculated, over what time window, and what to do when the value is missing. For ‘attempts in the last five minutes’ the contract has to say whether failed attempts count, whether the current attempt is included, and whether the window ends at the request time or at the last run of the job that computes it. Each of those choices changes the number, and training and serving have to make the same choice or the model is reading a different quantity from the one it learned.

A missing value is a value too, and it is not zero. A customer with no purchase history and a customer with zero purchases this week look identical if both become zero, yet the first is a stranger and the second is a regular having a quiet week. The librarian would never confuse a new visitor with a regular who skipped a month. Store a default plus a flag that says the value was missing, so the model can tell them apart. Check too whether missingness differs across groups of users, since a feature that is missing for all customers in one country is a country feature in disguise.

A representation is the form a value takes when it reaches the model. Amounts spanning 1 to 10,000 dollars are usually squashed with a logarithm so that a 10 dollar difference at the low end is not drowned out by the high end. Categories such as the merchant are turned into numbers either by a lookup table of known values, by hashing, a fixed recipe that turns the text into one of a set number of numbered slots, or by a learned list of numbers, which the embeddings topic covers. Tree-based models, which sort examples with a series of yes-or-no questions, cope with raw numbers, while neural networks usually need the scaling, so the representation choice follows the model choice.

Every category list has a first day for a new value, such as a merchant that signed up this morning. The representation has to say what happens then, whether the value maps to an ‘unknown’ bucket, is hashed like any other, or triggers a default. A model that has never seen the value will otherwise receive something arbitrary, and the failure will not show up in any log.

Every feature that is fetched from somewhere has a cost in time and reliability. Suppose a lookup to an outside reputation service about the card’s history lifts recall, the share of frauds the model catches, by three points but takes 30 milliseconds of a 100 millisecond budget and times out one request in two hundred. The design options are to compute the slowly changing part of that reputation overnight and store it locally, or to use a stored copy that is allowed to be up to an hour old. A third option is to train a second model without the feature and serve it whenever the lookup fails.

An ablation is the measurement that justifies the cost, and it means training and evaluating the model with and without a feature or a group of features and recording the difference. Do it by group and by user segment, because related features stand in for each other, so a single feature’s importance score understates a group’s value and overstates one member’s. A feature that helps on new accounts and does nothing on old ones is a feature worth keeping and worth knowing about.

Features also need a review for sensitive information and unequal treatment. Removing an explicit attribute such as nationality does not remove it from the model if postcode and language carry the same information, so the check is on outcomes across groups rather than on the feature list. With the features chosen and justified, the next topic is how to keep each definition identical in the two separate programs that compute it, one for training and one for serving.

More features are believed to mean a better model. Each feature is a dependency that has to be computed in two places, kept fresh, and defended against leakage, and a feature that adds nothing still adds all of that. A design with twelve justified features is stronger than one with eighty.

Listing features without sources or reasons is what interviewers hear as a feature dump. Naming the five sources and giving two or three per source with a sentence each shows that you know where data comes from and what it costs to fetch.

Missing values get treated as zero and never looked at again. Missingness is a pattern of its own, it differs by segment, and it is one of the first things to break when a system that feeds the data changes.

Inside the ten-minute data and features block, about five minutes belong to features, and the way to open is with the sources. For example, ‘I would take features from the item, the actor, their behaviour, their network and the context. For this payment that is the amount and merchant, account age, attempts on the card in the last five minutes, accounts sharing the device, and the hour and country match’. Then stop and let the interviewer pick the one they want to probe, which is usually freshness or leakage.