Latency, throughput, and cost

A request may spend time waiting for capacity, fetching features, retrieving candidates, running the model, and applying the final decision rules. Preparing data for network transfer and fetching values missing from a cache also contribute to the delay. A model that executes in 15 ms may therefore still wait for 80 ms before it runs when the service is overloaded.

Latency percentiles describe how the request times are distributed. A p99 of 200 ms means that 99% of measured requests finished within 200 ms, while the slowest 1% took longer. Those slower requests form the tail and may be affected by cache misses, large inputs, or contention, so measuring complete requests under a stated workload is necessary to understand the user-facing delay.

Precompute stable features and item vectors, parallelize independent lookups, batch candidates within a request, and reduce unnecessary network round trips. A cache can help when requests or representations repeat, but define its key, invalidation policy, freshness limit, and behavior after deletion or permission changes.

Cross-request batching can increase throughput while adding queue delay. Set a bounded batching wait and benchmark the tradeoff at realistic traffic. Distillation, quantization, and smaller models can reduce compute or memory, but require quality checks on the actual workload rather than an assumption that compression is free.

Bounded queues limit the work waiting for capacity, while passing deadlines to downstream services lets them avoid processing requests that can no longer finish in time. Retries also need limits because repeating timed-out work can increase the overload. When a retry could repeat an action, such as recording a decision, the operation needs a way to recognize duplicates and avoid applying that action twice.

Autoscaling has a delay, especially when large model artifacts must load or accelerator capacity is scarce. Maintain headroom and a validated degraded path, such as a cheaper ranker or cached eligible results. Measure fallback frequency and quality so acceptable latency does not conceal a permanently degraded experience.

Comparing cost at a shared quality and latency target shows whether an optimization actually benefits the application. Even with adequate capacity, individual features, models, or indexes can become unavailable. The next topic considers which alternative responses are acceptable in those cases and how to return safely to a working version.