07 · Serving and scale
Batch, online, and streaming inference
2 min read
Evaluation establishes whether a model is suitable to try in the application, while serving determines how its predictions become available. The first serving decision is when to compute them. Batch, online, and streaming inference place that computation at different points, depending on how quickly inputs change and when the application needs the result.
Batch, online, and streaming inference
Batch inference scores a collection ahead of use, such as generating nightly recommendations for users with enough history. Serving can then read a prepared result quickly. The tradeoff is freshness, unused computation for inactive users, and the need to handle new items or users between runs.
Online inference computes a prediction for the current request. It can use recent context, but feature lookup, model execution, and dependencies all sit within the response deadline. Streaming inference reacts to arriving events and updates predictions or state continuously; the user-facing service may read those results later.
| Pattern | Useful when | Main constraint |
|---|---|---|
| Batch | Inputs change slowly and results can be prepared | Staleness and work for unused results. |
| Online | The current request materially changes the answer | Tail latency and dependency failures. |
| Streaming | Events should update state promptly | Late events, duplicates, and processing lag. |
A hybrid recommendation request
A music home page can combine these patterns by preparing long-term user preferences and item vectors ahead of time, then using them to retrieve candidates for the current request. A later ranking step can incorporate recent skips or a newly selected genre. This preserves current session information without requiring every request to process the user’s complete listening history.
Specify which pieces can be stale and by how much. A stable item representation may tolerate a slower refresh than a catalog availability flag. Do not put authorization or deletion behavior behind a long-lived recommendation cache just because the ranking itself is allowed to be stale.
Define the serving contract
Validate the request schema, identify the model and feature versions, and specify output shape and timeout behavior. The service should reject malformed input predictably rather than returning an arbitrary score. Include enough version and timing information in logs to investigate a response later, without unnecessarily storing sensitive payloads.
Distinguish batching candidates within one request from batching requests across users. The former often reduces overhead without waiting for unrelated requests. Cross-request dynamic batching can improve accelerator utilization, but queueing to assemble a batch consumes part of the latency budget.
Testing representative traffic, cold caches, and dependency failures shows whether the selected pattern can meet the response deadline. Measuring time spent waiting separately from computation helps identify where it cannot. For systems that choose among many items, the amount of prediction work also depends on how candidates are selected, which leads to the retrieval and ranking pipeline in the next topic.