Batch, online, and streaming inference

A bakery has the same decision to make every morning. It can bake 200 loaves before opening, which is cheap and quick to sell but leaves unsold bread at closing time and nothing fresh for the customer who wanted rye. It can bake each loaf to order, which gives every customer exactly what they asked for and makes them wait 40 minutes. Or it can bake a stock before opening and adjust the next batch through the day as it watches what sells, which is the pattern a busy bakery settles on.

A recommendation system faces the same trade. Scoring every user overnight is the baked-ahead shelf, scoring at the moment a request arrives is baking to order, and updating a stored result as a user clicks through the day is the adjusted batch. The analogy breaks on speed, because a computer can bake to order in a tenth of a second, so the choice is about freshness and cost rather than about whether the customer will wait. That is why the sections below keep returning to two questions, how stale the answer may be and how much computation is wasted on people who never ask.

Batch inference scores a whole collection ahead of use, such as preparing a list of ten recommendations for every user during the night. When a user opens the app the next morning, the service reads the prepared list, which takes a few milliseconds, and the model does no work at that moment. The costs are freshness, since the list cannot know what the user did five minutes ago, and waste, since most of the scored users will not open the app tomorrow.

Online inference computes the prediction for the current request, at the moment it arrives. It can use the last thing the user did, but everything has to happen inside the response deadline, which for a web page is usually a few hundred milliseconds. That includes the lookup of the inputs the model needs, which are called features, the model itself, and every other service it has to call. Streaming inference sits between the two. It reacts to arriving events, such as a purchase or a skip, and updates a stored prediction or a stored summary of the user, so the next read sees something recent without running the full model on every request.

The table below is the comparison a designer makes in the first minute. The useful question for each row is which inputs change between one request and the next, because an input that does not change can be computed ahead and an input that changes every second cannot.

Assume a service with 2 million registered users and a model that takes 50 milliseconds to score one user on one machine. Those are assumptions for this exercise, so state them out loud in an interview. Scoring everyone takes 100,000 seconds of machine time, which is nearly 28 hours, so a single machine cannot finish before morning and the job needs at least four machines running in parallel to fit in a seven-hour night.

Assume further that only 400,000 of those users, one in five, open the app on a given day. Then four fifths of the night’s computation produced lists nobody read, and a design that scores only users active in the last 30 days, or that scores on request, spends far less. The number that decides between batch and online is this ratio of users scored to users served, and a designer who says it aloud has shown the trade-off without naming a single tool.

A music home page combines the patterns instead of choosing one. During the night it prepares the slow-changing parts, such as a numeric summary of each listener’s long-term taste and a numeric summary of each song, and stores them. When a request arrives, a small model reads those stored summaries together with what the listener did in the last few minutes, such as skipping two songs, and orders the page. The listener gets a page that reflects the current session without the service having to read their whole listening history on every request.

The design question in a hybrid is which pieces may be stale and by how much. A song’s summary can be a week old without harm, while a flag that says the song has been removed from the catalog must be current. One rule follows from that, which is that permission and deletion checks never hide behind a long-lived recommendation cache, a store of recent answers kept so a repeated request is answered without recomputing, even when the ranking itself is allowed to be a day old.

A serving contract is the written agreement between the product and the model service about what a request contains, what comes back, how long it may take, and what happens when something is wrong. The service checks that a request has the fields it expects and rejects a malformed one with a clear error, because a model handed a missing field will still return a number and that number means nothing. Each response carries the model version and the feature version that produced it, so that a complaint next week can be traced to the exact thing that was running.

Batching means scoring several items in one call to the model instead of one at a time, and the contract should say which kind is meant. Batching the 500 candidates of one request together is almost free, because they are already waiting. Batching requests from different users together makes the hardware more efficient but forces the first request to wait for the others to arrive, and that waiting comes out of the response deadline.

Before trusting the chosen pattern, a team tests it on realistic traffic with an empty cache and with one dependency switched off, and measures waiting time separately from computing time. For a product that chooses among millions of items, the amount of model work also depends on how the candidates were narrowed down first, which is the subject of the next topic.

Newcomers hear that online inference is more accurate and treat batch as the inferior option. Batch is the better design whenever the inputs change slowly, because the same prediction costs less and the page loads faster, and a nightly list of films to watch loses nothing by being eight hours old. Online wins only when something in the current request, such as a search phrase or the last click, changes the answer.

Streaming inference is also taken to mean running the model on every event. In most designs the events update a stored summary, such as the count of items viewed in the last hour, and the model runs later, either on request or on a schedule. The streaming part is the bookkeeping, and the model work can still be batch or online.

The cache gets treated as part of the model, when it has its own rules about how old an entry may be and when it must be thrown away, and those rules belong to the product, since a stale film list is harmless and a stale permission check is a security bug.

Right after the pipeline sketch and before the data discussion, the inference pattern takes two to three minutes. Name the two questions, how fresh the answer must be and how much of the scored work will be used, and then choose a pattern with one sentence of reasoning, such as scoring active users nightly and reranking on request. If the interviewer wants the detail of the hybrid or the cache rules, offer to come back to it in the deep dive rather than spending the data minutes on it.