Feature stores and consistency

Two people in a household both track ‘what we spent this week’. One resets the count every Monday morning and the other counts the last seven days from today, so on a Thursday one says 180 dollars and the other says 310, and both are right by their own definition. They argue about whether they are over budget while using the same words for two different quantities.

A model is in a worse position than either of them, because it cannot argue. If it learned from the Monday-reset number and is served the rolling seven-day number, it reads 310 where it expects something like 180 and raises the alarm, with no sign anywhere that the definition changed. The remedy in the household is to agree on one definition and write it down, and the remedy in the system is the same, with a test that proves both programs follow it.

At the moment of contact the household and the system differ, because the two people meet at dinner and discover the mismatch in a sentence, while the training program runs overnight over months of stored records and the serving program runs in milliseconds at checkout, and they never meet. The running example from here on is the fraud model’s ‘purchases in the last seven days’ feature from the previous topic.

Training needs the value of a feature as it was at thousands of past moments, while serving needs the value as it is right now, for one customer, in a few milliseconds. The first need is served by an offline store, which is a large historical table that can be joined and scanned over months of data at leisure. The second is served by an online store, which is a fast lookup keyed by customer id that holds only the current values. They are different storage because the access patterns are opposite, one wide and slow and the other narrow and fast.

The definition has to be the same across both or the names lie. A training pipeline that reads a nightly snapshot of purchases in the last seven days and a serving pipeline that computes a rolling window at request time can both call the column purchases in the last seven days. One counts to midnight and the other counts to now, which is the household argument again. For a customer who bought twice this morning, the training value is 3 and the serving value is 5, and the model, trained on midnight numbers, reads a quiet week as a busy one.

A feature store is a system that holds each feature definition once, computes it into both the historical table and the fast lookup from the same code, and serves it to training and serving alike. It removes the duplicated pipeline and the duplicated definition, and it gives several models a shared set of features instead of each team rebuilding ‘account age’ in its own way. It does not fix wrong source events, missing timestamps, or a refresh job that stalled, so it moves the consistency problem to one place rather than making it disappear.

The decision to build or buy one depends on how many models and how many history-based features there are. A single model with five request-time features and one count can keep consistency with shared transformation code and a small table of counts, and a feature store would be overhead. Three models sharing twenty history-based features, each needing values as they stood at each past moment for training and millisecond reads for serving, is the case where a feature store pays for itself. Saying which requirement it satisfies is what makes it a design decision rather than a product name.

The test is to log the feature values the serving program used for a sample of real requests, together with the request id and time. Then rebuild the same values with the training pipeline as of those times and compare them row by row. Comparing averages is not enough, because a serving default of zero for some customers and a training value that is too high for others can cancel out in the average while every single row is wrong. Skew is the name for the difference between the two paths, and a skew check that compares matched rows finds a mismatched window boundary in an afternoon.

Versions have to travel together, because the model, the feature definitions it was trained with, and the vocabulary that turns categories into numbers are one package. A model that learned merchant number 17 means a grocery store will happily accept an index 17 from a new vocabulary where it means a casino. A compatibility check at deployment that refuses a model whose feature versions do not match the serving versions, plus a kept copy of the last compatible set to roll back to, is the design that prevents it.

Freshness is a separate check from correctness, because a correctly computed count that is six hours old after a stalled job is wrong for the purpose. Record the time of the last successful update for every feature and have the serving layer compare it to the age limit in the feature definition, falling back to a default or to a model without that feature when the limit is exceeded. The same discipline of versioned pairs applies to learned representations, which the embeddings topic takes up next.

Two beliefs about what proves consistency are both wrong. Sharing the transformation code is assumed to guarantee the same values, when the code can be identical while the two programs feed it different source events, different timestamps, or counts of different ages, so shared code removes one cause of skew and leaves the others. A dashboard showing that the average of a feature matches offline and online gets read the same way, and it hides errors that cancel, which is why the matched-request comparison is the one that counts.

The feature store also gets treated as the design. It is a place to put definitions and a way to serve them. The design questions are which features need history, how fresh they must be, and how the two paths will be tested, and those stay yours whether or not a store is involved.

The follow-up ‘how do you make sure training and serving see the same features’ is where consistency comes up, and it takes two or three minutes to answer well. Pre-empt it when you first mention a history-based feature, for example ‘the count of attempts in the last five minutes would come from one feature definition, computed into an offline table for training and an online lookup for serving. I would log serving features and compare them with offline rebuilds on matched requests’. Mentioning the matched-request check is what separates a candidate who has seen skew from one who has read about it.