Latency, throughput, and cost

The time to get a coffee is the time in the queue plus the time ordering plus the time the barista spends making it plus the handover. A shop that buys a faster espresso machine saves 20 seconds on the making step and nothing on the queue, so on a busy morning customers still wait eight minutes. The customer who notices most is the one behind a group ordering six complicated drinks, and that unlucky customer in a hundred is the one a serving engineer designs for.

A prediction request has the same pieces, waiting for a free worker, fetching the inputs, running the model, and applying the final rules. A coffee shop cannot hand a waiting customer a default coffee when their patience runs out, while a service can decide in advance that a request past 200 milliseconds gets a cheaper answer, and that is where the coffee analogy stops. That decision is the subject of the fallbacks topic, and this topic is about keeping most requests from reaching it.

A request spends its time in several places, and the model is only one of them. It waits in a queue, which is the line of requests holding for a free worker, it fetches features from a store on another machine, it runs the model, and it applies the final rules and writes the response. A model that runs in 15 milliseconds can therefore sit behind 80 milliseconds of waiting when the service is busy, and the user feels 95 milliseconds.

Engineers describe these times with percentiles rather than averages. A p99 of 200 milliseconds means that 99 of every 100 requests finished within 200 milliseconds and the slowest one took longer, and that slowest one in a hundred is called the tail. The tail is where requests with no stored answer ready, unusually large requests, and crowded machines show up, so a service is measured on complete requests under realistic load, because an average hides exactly the users who are about to leave.

A latency budget is the deadline for the whole request split into an allowance for each stage, written down before anything is built. For a 200 millisecond page element a designer might allow 20 milliseconds for the queue, 40 for feature lookups running in parallel, 30 for retrieval, 60 for the ranker, and 50 for everything else, and each team then knows what it has to hit. The budget is an allocation rather than a measurement, and because the p99 of the whole request is not the sum of the p99s of its stages, the end-to-end figure still has to be measured under load.

The budget also decides how many machines are needed, which connects it to the request estimate from the scale topic. If a worker handles one request at a time and each takes 15 milliseconds of model work, one worker finishes about 66 requests a second, and the 2,315 peak requests per second from that estimate need about 35 workers with no spare room. A design adds headroom, meaning spare machines, for bursts and for the work a failed machine hands to the others, so a sensible plan is about 50 workers, and that count is what the cost conversation starts from.

The first savings come from moving work off the request path. Stable features and item embeddings are computed ahead and stored, independent lookups run at the same time instead of one after another, and the candidates of one request are scored in a single batch rather than one call each. A cache is a store of recent answers kept so a repeated request is answered without recomputing, and it helps when the same user or item is asked about again within minutes. It needs a written rule for how old an entry may be and when it is thrown away.

The second kind of saving changes the model itself. Distillation trains a smaller model to copy a larger one, and quantization stores the model’s numbers with less precision so it runs faster and uses less memory, and both cut the model bar in the budget. Each needs a quality check on the live workload, because a compressed model that loses two points of accuracy on rare users has moved the cost rather than removed it.

Batching requests from different users together is the one change that can make the budget worse. It raises throughput, because the hardware processes 32 requests in little more time than one, but the first request in the batch waits for the other 31 to arrive, and that wait lands in the queue bar. A bounded wait, such as at most 5 milliseconds before the batch is sent whatever its size, keeps the gain without letting the tail grow.

Overload is the state where requests arrive faster than the service finishes them, and left alone it makes everything slow. A bounded queue holds at most a set number of waiting requests and turns the rest away at once, so the ones it keeps still finish in time. Passing the deadline downstream lets a feature store skip a lookup for a request that can no longer finish, and retries get a limit, because a client that retries every timed-out request doubles the load on a service that is already drowning.

Adding machines takes time, especially when each new one must load a model of several gigabytes, so a service keeps headroom and a cheaper path ready for the minutes before new capacity arrives. The cheaper path might be a smaller ranker or a cached popular list. How often it is used is a number to watch, because a service that reports healthy latency while serving a quarter of its users the fallback has hidden the problem rather than solved it.

Every saving is compared at the same quality and the same deadline, since a change that halves cost by returning worse results has changed the product rather than optimized it. Even with enough capacity, a single feature store or index can still fail, and the next topic covers what the service returns in that case and how a new model version is released without the same risk.

A faster model is expected to mean a faster page. In the budget figure the model is 15 of 100 milliseconds, so a model twice as fast saves less than eight, and the queue and the feature lookups are where the time goes. A designer who asks for the time breakdown before proposing a smaller model has avoided the commonest wasted month.

Reporting the average is the second habit to drop. An average of 60 milliseconds can hide a p99 of 900, and the users at the p99 are the ones who close the page. Percentiles with a stated load are the only honest description of a service.

Busier hardware gets read as a win in itself. Cross-request batching keeps the hardware busier and lowers cost per request, and it also makes the tail longer. The target is the deadline at the required throughput, and busier hardware is only good when it is reached inside that.

In a serving-heavy question latency is worth three to five minutes, and otherwise one or two sentences. State the deadline as a p99, such as 200 milliseconds for the page element, split it across the stages in one breath, and name the stage that worries you. If the interviewer wants the overload behavior or the batching trade-off, say you will cover it in the deep dive and move on.