02 · Framing the problem
Scale and latency estimates
9 min read
A scale estimate is a two-minute calculation that turns a product’s expected use into numbers a designer can build from. Those numbers are the request rate per second, the work inside each request and the bytes to store. It tells you whether the design needs one machine or a fleet, and whether an expensive model can run on every request or only on a shortlist. The estimate comes before the design because the numbers decide its shape.
An everyday example
A bakery expects to sell 2,000 loaves a day and has to decide how many ovens to buy. Two thousand a day sounds like a quiet trickle until the owner asks when people buy bread. About 1,200 of those loaves go out between seven and nine in the morning, so the shop needs 600 loaves an hour at its peak. An oven that bakes 60 loaves an hour means ten ovens running flat out at breakfast time and sitting cold for most of the afternoon.
The owner has used the whole method in one breath. Take the daily total, find the busy hour, divide by what one unit of equipment can do, and add a little room for a broken oven. The same four steps size a prediction service, with requests in place of loaves and serving machines in place of ovens. The bakery example stops matching at one point, because a loaf is a loaf while a request can ask for one prediction or five hundred, and that difference is the subject of the second step below.
Count work rather than users
Start from the daily volume and work down to a per-second rate, because machines are bought by the second and the day hides the busy hour. Take an imagined service with two million people using it each day, each asking for recommendations 20 times. That is 40 million requests a day, and a day has 86,400 seconds, so the average is about 463 requests a second. Traffic bunches in the evening, so assume the peak is five times the average, which gives about 2,315 requests a second. Neither the activity figure nor the peak multiplier is a fact about any particular company, and in an interview you say both out loud and ask whether they sound right.
Requests are still not work. If each request scores 500 possible recommendations, called candidates, then peak demand is 2,315 times 500, or about 1.16 million candidate scores a second. The model’s cost follows the number of scores rather than the number of requests. A request can hand its 500 candidates to the model together in one batch, so the number of scores does not mean the same number of network calls. The thing to measure is how many candidates one machine can score a second in batches of that size.
40,000,000 requests / 86,400 seconds ≈ 463 requests a second
463 × 5 peak factor ≈ 2,315 requests a second
2,315 × 500 candidates ≈ 1.16 million candidate scores a secondTurn the work into machines
The last step divides the work by what one machine can do. Assume a benchmark, a timed test of one machine, shows that one serving machine scores 20,000 candidates a second in batches of 500 while staying inside the response deadline. Then 1.16 million scores a second needs about 58 machines at peak. Add room for one or two machines failing and for a peak that arrives higher than planned, and the fleet is around 70 machines, running mostly idle at four in the morning.
The number forces a design choice. Seventy machines for one recommendation feature is a visible cost, and the lever that moves it most is the 500. Cutting the candidates to 100 with a cheap first pass drops the fleet to about 14 machines with the same spare room for failures, which is why large systems put a cheap stage in front of the expensive model. The serving topics explain how that stage works, and the estimate is what justifies building it.
The response deadline enters the estimate as a given. The requirements say how long a person will wait, and for a page that is loading that leaves around 100 milliseconds for the model’s share. The benchmark has to be run at that deadline, because a machine that scores 20,000 candidates a second when nobody is waiting may manage half that when every answer must come back inside 100 milliseconds. How to split the deadline across the steps of a request is the subject of the latency topic.
Estimate the data the system holds
The same arithmetic sizes storage. If each of five million items is represented by a list of 128 numbers, and each number takes four bytes, the lists alone take 5 million times 128 times 4 bytes, which is about 2.56 gigabytes. Lists of numbers like these are called embeddings, and the features topics explain where they come from. For the estimate, 2.56 gigabytes fits in the memory of one ordinary machine, so this design can keep every item’s embedding in memory and skip a trip to disk on each request. A search structure for finding similar items, the item identifiers and spare copies add to that figure, so it is a floor to build from and never a budget.
Event data grows instead of sitting still. At 40 million requests a day and an assumed 500 bytes per logged event, the log grows by about 20 gigabytes a day. That is 600 gigabytes a month and about 7 terabytes a year before any copies, indexes or compression. One year fits on a handful of disks and ten years fits nowhere cheap, so the estimate forces a decision about how long raw events are kept and when they are boiled down to summaries.
5,000,000 items × 128 numbers × 4 bytes ≈ 2.56 gigabytes
40,000,000 events × 500 bytes ≈ 20 gigabytes a day
20 gigabytes × 30 days ≈ 600 gigabytes a monthWhere people get confused
Most of the confusion here is about which number sizes the fleet. A fleet built for the average of 463 requests a second falls over every evening at 2,315, so the peak is the number that buys machines and the average is the number that pays the electricity bill. Users and requests are the wrong unit for a related reason, because the model is paid in scores. Two million users, 40 million requests and 1.16 million scores a second are three different numbers, and the one that sizes the model fleet is the last. Saying which one you mean, every time, avoids an argument later about a fleet that is five hundred times too small.
The other confusion is treating the estimate as a measurement. The 20,000 scores a second per machine was an assumption, and a benchmark on production hardware, with the feature lookups and the deadline included, can come back at half of that. The estimate tells you which measurement to run before any other, and it stops being useful the moment that measurement exists.
In the interview
Right after the requirements are agreed and before any boxes are drawn, there are two or three minutes for the estimate, and the interviewer is listening for whether you reach a per-second number and say what it means for the design. A plain way to start is to say that you would like a quick estimate to see whether this is one machine or a fleet. Then ask for the daily volume, or state an assumption, and round aggressively.
Say the assumptions out loud, say the peak factor, and land on one sentence about what the number forces, such as a cheap stage before the expensive model or a decision about retention. Anything about splitting the deadline into stages belongs in the serving discussion, and saying that you would return to it there is a good answer.
Interview questions
The interviewer says the platform receives one billion posts a day and each one must be checked by a model. What does that tell you about the design?
One billion a day is about 11,600 posts a second on average, and if the peak is three times that, around 35,000 a second. If a large model takes 50 milliseconds of accelerator time per post, where an accelerator is the chip that runs large models, that is 35,000 times 0.05, or 1,750 accelerator-seconds of work every second. That means roughly 1,750 accelerators busy at peak. These are assumptions I would state and let the interviewer correct.
A number that size tells me the big model cannot see every post. I would put a cheap check in front, such as simple rules and a small model that clears the obviously fine posts, and send only the uncertain few percent to the expensive model. The estimate is what forces that two-stage shape, and I would say so.
A ranking model handles 2,000 batches a second in a benchmark. Are two machines enough for 3,000 requests a second?
Two machines are not enough on those numbers alone. Together they handle 4,000 batches a second on paper, which is above 3,000, but that leaves no room for one machine failing, and a single machine would then face 3,000 requests on its own and make them wait in line. I would want at least three, and I would check that each benchmark batch had the same size and the same feature lookups as a production request.
I would also ask what the benchmark did to the response time. A machine handling as many requests as it possibly can usually answers slowly. I would measure how many requests a second each machine sustains while still answering inside the deadline, and size the fleet from that number with a third of it in reserve.
Estimate how much training data this recommendation system would collect in three months.
At 40 million logged requests a day and about 500 bytes each, the log grows by roughly 20 gigabytes a day, so three months is about 1.8 terabytes of raw events. That is far too large to load into memory and comfortable on disk, so training would read it in pieces, and I would keep a smaller summarized copy for quick experiments.
I would also ask how far back the model needs to see. If the catalog changes weekly, the last few weeks matter most and older events can be thinned out, which shrinks both the storage and the training time, and I would state that as an assumption to confirm.
What to practice next
Design these systems on the canvas and get your design graded.
This problem lets you compute percentiles from a list of response times, such as the time the slowest one in a hundred requests took, which is the measurement that replaces the estimate once the service exists.